Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Side inputs not working in CombineGlobally #19851

Open
damccorm opened this issue Jun 4, 2022 · 3 comments
Open

Side inputs not working in CombineGlobally #19851

damccorm opened this issue Jun 4, 2022 · 3 comments

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

Side inputs are not working in CombineGlobally. They do work as expected in CombinePerKey and CombineValues.

 

The function argument still has the value of AsSingleton rather than being resolved into the real value.

 

Here is a Notebook where you can reproduce the issue.

https://colab.research.google.com/drive/149By0ZKJjb_JdDOsFywdT_OLj1hMEPBa

Imported from Jira BEAM-8400. Original Jira may contain additional context.
Reported by: dcavazos.

@tvalentyn
Copy link
Contributor

tvalentyn commented Mar 8, 2024

From mailing list thread a while back (https://lists.apache.org/thread/zbzzd1w9wjcdbff6nppjp2o079x1xng8):

You can work around this by specifying
without_defaults() on the global combine (as the default is computed
at pipeline construction time). Note that even in the cases where it
works, it disables combiner lifting, so side inputs in combiners is
generally discouraged.

@hjtran
Copy link
Contributor

hjtran commented Mar 26, 2024

I haven't put much thought into whether this will work with windowed pcollections. If not, maybe we can just have this implement side inputs for the batch case and error for the streaming case and punt on the streaming case

@hjtran
Copy link
Contributor

hjtran commented Sep 30, 2024

I tried using the workaround but it didn't seem to work
https://play.beam.apache.org/?sdk=python&shared=WNAXIOXX8h1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants