Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] 3.1.0 pos_explode tests are failing #953

Closed
revans2 opened this issue Oct 15, 2020 · 4 comments · Fixed by #1412
Closed

[BUG] 3.1.0 pos_explode tests are failing #953

revans2 opened this issue Oct 15, 2020 · 4 comments · Fixed by #1412
Assignees
@tgravescs
Labels
bug Something isn't working P1 Nice to have for release

Comments

@revans2
Copy link
Collaborator

revans2 commented Oct 15, 2020

Describe the bug
A lot of pos_explode and explode integration tests are failing in the 3.1.0 CI.

From looking at the exception it appears to be that spark is inserting a filter into the plan somewhere now that checks the length of the array, and this results in the data not being on the GPU any longer.

11:22:48      def deco(*a, **kw):
11:22:48          try:
11:22:48              return f(*a, **kw)
11:22:48          except py4j.protocol.Py4JJavaError as e:
11:22:48              converted = convert_exception(e.java_exception)
11:22:48              if not isinstance(converted, UnknownException):
11:22:48                  # Hide where the exception came from that shows a non-Pythonic
11:22:48                  # JVM exception message.
11:22:48  >               raise converted from None
11:22:48  E               pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.FilterExec
11:22:48  E               Filter (size(array(b#222501, c#222502, d#222503), true) > 0)
11:22:48  E               +- Scan ExistingRDD[a#222500,b#222501,c#222502,d#222503]

This appears to be a needless step because we are creating the array right then. This also will make things more compilcated to optimize and keep the same functionality in place. I also don't want to have to materialize the arrays so this is going to require some optimizations in the planning of the query, and possibly pushing those back into spark itself.

11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-String-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-String-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Byte-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Byte-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Short-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Short-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Integer-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Integer-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Long-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Long-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Float-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Float-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Double-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Double-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Boolean-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Boolean-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Date-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Date-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Timestamp-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-INT96-Timestamp-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-String-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-String-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Byte-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Byte-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Short-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Short-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Integer-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Integer-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Long-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Long-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Float-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Float-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Double-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Double-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Boolean-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Boolean-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Date-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Date-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Timestamp-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MICROS-Timestamp-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-String-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-String-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Byte-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Byte-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Short-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Short-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Integer-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Integer-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Long-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Long-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Float-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Float-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Double-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Double-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Boolean-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Boolean-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Date-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Date-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Timestamp-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[true-TIMESTAMP_MILLIS-Timestamp-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-String-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-String-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Byte-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Byte-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Short-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Short-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Integer-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Integer-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Long-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Long-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Float-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Float-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Double-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Double-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Boolean-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Boolean-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Date-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Date-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Timestamp-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-INT96-Timestamp-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-String-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-String-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Byte-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Byte-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Short-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Short-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Integer-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Integer-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Long-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Long-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Float-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Float-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Double-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Double-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Boolean-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Boolean-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Date-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Date-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Timestamp-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MICROS-Timestamp-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-String-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-String-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Byte-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Byte-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Short-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Short-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Integer-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Integer-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Long-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Long-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Float-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Float-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Double-CORRECTED][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Double-LEGACY][INCOMPAT, ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Boolean-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Boolean-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Date-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Date-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Timestamp-CORRECTED][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/cache_test.py::test_cache_posexplode_makearray[false-TIMESTAMP_MILLIS-Timestamp-LEGACY][ALLOW_NON_GPU(DataWritingCommandExec)]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[String][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Byte][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Short][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Integer][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Long][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Float][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Double][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Boolean][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Date][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_posexplode_makearray[Timestamp][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[String][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Byte][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Short][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Integer][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Long][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Float][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Double][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Boolean][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Date][IGNORE_ORDER({'local': True})]
11:22:48  FAILED src/main/python/generate_expr_test.py::test_explode_makearray[Timestamp][IGNORE_ORDER({'local': True})]
@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Oct 15, 2020
@revans2
Copy link
Collaborator Author

revans2 commented Oct 15, 2020

It looks like https://issues.apache.org/jira/browse/SPARK-32295 caused this to happen

@revans2
Copy link
Collaborator Author

revans2 commented Oct 15, 2020

Looking at the code change it is supposed to skip this if the children are foldable. We are hitting this in a special case where the child is not foldable. We cannot make make_array(a, b, c, d) foldable because it is not, so we would have to add in a special case rule here.

@revans2 revans2 mentioned this issue Oct 16, 2020
Merged
@sameerz sameerz added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Oct 20, 2020
@sameerz sameerz added this to the Nov 9 - Nov 20 milestone Nov 6, 2020
@tgravescs
Copy link
Collaborator

jira in spark: https://issues.apache.org/jira/browse/SPARK-33544?filter=-2

@tgravescs
Copy link
Collaborator

spark PR for the jira is merged, we need to wait for a build and then we can add our tests back

@sameerz sameerz removed this from the Nov 23 - Dec 4 milestone Dec 5, 2020
@revans2 revans2 mentioned this issue Dec 16, 2020
Merged
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to e4ffcbb3b67fafd54ee4759da87aab9f98eac5cf (NV…
a927681 
…IDIA#953)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgravescs tgravescs

Labels
bug Something isn't working P1 Nice to have for release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix a lot of tests marked with xfail for Spark 3.1.0 that no longer fail revans2/spark-rapids
3 participants
@revans2 @tgravescs @sameerz