You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@GaryShen2008 found the test_from_utc_timestamp failed on Cloudera Env when TZ is Iran
Navin found:
the timezone Iran has transition rules in my JVM,
we should not recommend it as non-UTC timezone for testing as this means it supports DST.
test_from_utc_timestamp can pass when TZ is Asia/Shanghai, but failed on Iran TZ.
Note:
This error happends on Cloudera when global TZ is UTC. The TZ here refers to the TZ parameter in from_utc_timestamp function.
Steps/Code to reproduce bug
The error on Cloudera CI
[2023-12-28T16:15:37.579Z] =================================== FAILURES ===================================
[2023-12-28T16:15:37.579Z] ___________________ test_from_utc_timestamp[Timestamp-Iran] ____________________
[2023-12-28T16:15:37.579Z]
[2023-12-28T16:15:37.579Z] data_gen = Timestamp, time_zone = 'Iran'
[2023-12-28T16:15:37.579Z]
[2023-12-28T16:15:37.579Z] @pytest.mark.parametrize('time_zone', ["Asia/Shanghai", "Iran", "UTC", "UTC+0", "UTC-0", "GMT", "GMT+0", "GMT-0"], ids=idfn)
[2023-12-28T16:15:37.579Z] @pytest.mark.parametrize('data_gen', [timestamp_gen], ids=idfn)
[2023-12-28T16:15:37.579Z] @tz_sensitive_test
[2023-12-28T16:15:37.579Z] @allow_non_gpu(*non_utc_allow)
[2023-12-28T16:15:37.579Z] def test_from_utc_timestamp(data_gen, time_zone):
[2023-12-28T16:15:37.579Z] > assert_gpu_and_cpu_are_equal_collect(
[2023-12-28T16:15:37.579Z] lambda spark: unary_op_df(spark, data_gen).select(f.from_utc_timestamp(f.col('a'), time_zone)))
[2023-12-28T16:15:37.579Z]
[2023-12-28T16:15:37.579Z] ../../src/main/python/date_time_test.py:295:
[2023-12-28T16:15:37.579Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2023-12-28T16:15:37.579Z] ../../src/main/python/asserts.py:595: in assert_gpu_and_cpu_are_equal_collect
[2023-12-28T16:15:37.579Z] _assert_gpu_and_cpu_are_equal(func, 'COLLECT', conf=conf, is_cpu_first=is_cpu_first, result_canonicalize_func_before_compare=result_canonicalize_func_before_compare)
[2023-12-28T16:15:37.579Z] ../../src/main/python/asserts.py:503: in _assert_gpu_and_cpu_are_equal
[2023-12-28T16:15:37.579Z] from_gpu = run_on_gpu()
[2023-12-28T16:15:37.579Z] ../../src/main/python/asserts.py:496: in run_on_gpu
[2023-12-28T16:15:37.579Z] from_gpu = with_gpu_session(bring_back, conf=conf)
[2023-12-28T16:15:37.579Z] ../../src/main/python/spark_session.py:164: in with_gpu_session
[2023-12-28T16:15:37.579Z] return with_spark_session(func, conf=copy)
[2023-12-28T16:15:37.579Z] /usr/local/lib/python3.8/contextlib.py:75: in inner
[2023-12-28T16:15:37.579Z] return func(*args, **kwds)
[2023-12-28T16:15:37.579Z] ../../src/main/python/spark_session.py:131: in with_spark_session
[2023-12-28T16:15:37.579Z] ret = func(_spark)
[2023-12-28T16:15:37.579Z] ../../src/main/python/asserts.py:205: in <lambda>
[2023-12-28T16:15:37.579Z] bring_back = lambda spark: limit_func(spark).collect()
[2023-12-28T16:15:37.579Z] /opt/cloudera/parcels/SPARK3-3.3.0.3.3.7180.0-274-1.p0.31212967/lib/spark3/python/lib/pyspark.zip/pyspark/sql/dataframe.py:817: in collect
[2023-12-28T16:15:37.579Z] sock_info = self._jdf.collectToPython()
[2023-12-28T16:15:37.579Z] /opt/cloudera/parcels/SPARK3-3.3.0.3.3.7180.0-274-1.p0.31212967/lib/spark3/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py:1321: in __call__
[2023-12-28T16:15:37.579Z] return_value = get_return_value(
[2023-12-28T16:15:37.579Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2023-12-28T16:15:37.579Z]
[2023-12-28T16:15:37.579Z] a = ('xro21350', <py4j.clientserver.JavaClient object at 0x7f484578e520>, 'o21349', 'collectToPython')
[2023-12-28T16:15:37.579Z] kw = {}, converted = IllegalArgumentException()
[2023-12-28T16:15:37.579Z]
[2023-12-28T16:15:37.579Z] def deco(*a: Any, **kw: Any) -> Any:
[2023-12-28T16:15:37.579Z] try:
[2023-12-28T16:15:37.579Z] return f(*a, **kw)
[2023-12-28T16:15:37.579Z] except Py4JJavaError as e:
[2023-12-28T16:15:37.579Z] converted = convert_exception(e.java_exception)
[2023-12-28T16:15:37.579Z] if not isinstance(converted, UnknownException):
[2023-12-28T16:15:37.579Z] # Hide where the exception came from that shows a non-Pythonic
[2023-12-28T16:15:37.579Z] # JVM exception message.
[2023-12-28T16:15:37.579Z] > raise converted from None
[2023-12-28T16:15:37.579Z] E pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
[2023-12-28T16:15:37.579Z] E Project [from_utc_timestamp(a#1873, Iran) AS from_utc_timestamp(a, Iran)#1875]
[2023-12-28T16:15:37.579Z] E +- Scan ExistingRDD[a#1873]
[2023-12-28T16:15:37.579Z]
[2023-12-28T16:15:37.579Z] /opt/cloudera/parcels/SPARK3-3.3.0.3.3.7180.0-274-1.p0.31212967/lib/spark3/python/lib/pyspark.zip/pyspark/sql/utils.py:196:
The text was updated successfully, but these errors were encountered:
Describe the bug
@GaryShen2008 found the test_from_utc_timestamp failed on Cloudera Env when TZ is Iran
Navin found:
test_from_utc_timestamp
can pass when TZ isAsia/Shanghai
, but failed onIran
TZ.Note:
This error happends on Cloudera when global TZ is UTC. The TZ here refers to the TZ parameter in
from_utc_timestamp
function.Steps/Code to reproduce bug
The error on Cloudera CI
The text was updated successfully, but these errors were encountered: