[BUG] arithmetic ops tests failing on Spark 3.3.0

[jlowe]

Many arithmetic ops tests failed in the recent Spark 3.3.0 nightly tests against 22.06:

10:24:14  =========================== short test summary info ============================
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(7,3)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(12,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(20,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(30,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(36,5)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(38,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(a, cast(0 as {}))-Decimal(38,10)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(7,3)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(12,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(20,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(30,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(36,5)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(38,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[pmod(cast(-12 as {}), cast(0 as {}))-Decimal(38,10)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(7,3)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(12,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(20,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(30,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(36,5)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(38,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(38,10)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(7,3)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(12,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(20,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(30,2)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(36,5)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(38,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[cast(-12 as {}) % cast(0 as {})-Decimal(38,10)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_cast_neg_to_decimal_err
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_floor_scale_zero[Double]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_floor_scale_zero[Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_ceil_scale_zero[Double]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_ceil_scale_zero[Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(7,3)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(12,2)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(18,0)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(20,2)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(30,2)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(36,5)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(38,0)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(38,10)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(7,-3)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(36,-5)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_bround[Decimal(38,-10)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(7,3)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(12,2)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(18,0)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(20,2)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(30,2)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(36,5)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(38,0)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(38,10)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(7,-3)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(36,-5)][INCOMPAT, APPROXIMATE_FLOAT]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_decimal_round[Decimal(38,-10)][INCOMPAT, APPROXIMATE_FLOAT]

[nartal1]

test_mod_pmod_by_zero - Looks like simple fix of changing error message to "Division by zero"

round/bround tests are the ones which might need more work. Looks like bug fix when DecimalType with negative scale is provided. Spark commit - apache/spark@ee0aecc. I missed this in audit thinking it's only for RoundFloor and RoundCeil (which we currently fall back to CPU for scales other than 0). It affects Round and Bround as well. Not sure how much of the code needs to be shimmed here.

I haven't looked into RC of these test failures yet - test_floor_scale_zero, test_ceil_scale_zero and test_cast_neg_to_decimal_err

[firestarman]

test_mod_pmod_by_zero should be fixed by #5781 .

[sperlingxx]

#5786 attempts to fix round problems

[tgravescs]

we think the following tests are still not fixed:

10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_floor_scale_zero[Double]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_floor_scale_zero[Decimal(18,0)]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_ceil_scale_zero[Double]
10:24:14  FAILED ../../src/main/python/arithmetic_ops_test.py::test_ceil_scale_zero[Decimal(18,0)]

checking this

[GaryShen2008]

Close this issue, since we merged all the PRs and created a following issue #5797.