Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] test_struct_groupby_count failed #2406

Closed
jlowe opened this issue May 13, 2021 · 0 comments · Fixed by #2413
Closed

[BUG] test_struct_groupby_count failed #2406

jlowe opened this issue May 13, 2021 · 0 comments · Fixed by #2413
Assignees
@gerashegalov
Labels
bug Something isn't working
Milestone
May 10 - May 21

Comments

@jlowe
Copy link
Member

jlowe commented May 13, 2021

On last night's Dataproc integration test build, test_struct_groupby_count failed. From the failure log:

06:40:46  cpu = 2, gpu = 3
06:40:46  float_check = <function get_float_check.<locals>.<lambda> at 0x7f3960bddd30>
06:40:46  path = [0, 'a', 'a', 'aa']

Full log for reference:

06:40:46  �[31m�[1m_ test_struct_groupby_count[Struct(not_null)(('a', Struct(('aa', Integer),('ab', Integer))))] _�[0m
06:40:46  
06:40:46  key_data_gen = Struct(not_null)(('a', Struct(('aa', Integer),('ab', Integer))))
06:40:46  
06:40:46      �[37m@allow_non_gpu�[39;49;00m(�[96many�[39;49;00m = �[94mTrue�[39;49;00m)
06:40:46      �[37m@pytest�[39;49;00m.mark.parametrize(�[33m'�[39;49;00m�[33mkey_data_gen�[39;49;00m�[33m'�[39;49;00m, [
06:40:46          StructGen([
06:40:46              (�[33m'�[39;49;00m�[33ma�[39;49;00m�[33m'�[39;49;00m, StructGen([
06:40:46                  (�[33m'�[39;49;00m�[33maa�[39;49;00m�[33m'�[39;49;00m, IntegerGen(min_val=�[94m0�[39;49;00m, max_val=�[94m9�[39;49;00m))
06:40:46              ]))], nullable=�[94mFalse�[39;49;00m),
06:40:46          StructGen([
06:40:46              (�[33m'�[39;49;00m�[33ma�[39;49;00m�[33m'�[39;49;00m, StructGen([
06:40:46                  (�[33m'�[39;49;00m�[33maa�[39;49;00m�[33m'�[39;49;00m, IntegerGen(min_val=�[94m0�[39;49;00m, max_val=�[94m4�[39;49;00m)),
06:40:46                  (�[33m'�[39;49;00m�[33mab�[39;49;00m�[33m'�[39;49;00m, IntegerGen(min_val=�[94m5�[39;49;00m, max_val=�[94m9�[39;49;00m)),
06:40:46              ]))], nullable=�[94mFalse�[39;49;00m),
06:40:46      ], ids=idfn)
06:40:46      �[94mdef�[39;49;00m �[92mtest_struct_groupby_count�[39;49;00m(key_data_gen):
06:40:46          �[94mdef�[39;49;00m �[92mgroup_by_count�[39;49;00m(spark):
06:40:46              df = two_col_df(spark, key_data_gen, IntegerGen())
06:40:46              �[94mreturn�[39;49;00m df.groupBy(df.a).count()
06:40:46  >       assert_gpu_and_cpu_are_equal_collect(group_by_count)
06:40:46  
06:40:46  �[1m�[31mintegration_tests/src/main/python/hash_aggregate_test.py�[0m:624: 
06:40:46  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:364: in assert_gpu_and_cpu_are_equal_collect
06:40:46      _assert_gpu_and_cpu_are_equal(func, �[33m'�[39;49;00m�[33mCOLLECT�[39;49;00m�[33m'�[39;49;00m, conf=conf, is_cpu_first=is_cpu_first)
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:356: in _assert_gpu_and_cpu_are_equal
06:40:46      assert_equal(from_cpu, from_gpu)
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:93: in assert_equal
06:40:46      _assert_equal(cpu, gpu, float_check=get_float_check(), path=[])
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:41: in _assert_equal
06:40:46      _assert_equal(cpu[index], gpu[index], float_check, path + [index])
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:34: in _assert_equal
06:40:46      _assert_equal(cpu[field], gpu[field], float_check, path + [field])
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:34: in _assert_equal
06:40:46      _assert_equal(cpu[field], gpu[field], float_check, path + [field])
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:34: in _assert_equal
06:40:46      _assert_equal(cpu[field], gpu[field], float_check, path + [field])
06:40:46  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
06:40:46  
06:40:46  cpu = 2, gpu = 3
06:40:46  float_check = <function get_float_check.<locals>.<lambda> at 0x7f3960bddd30>
06:40:46  path = [0, 'a', 'a', 'aa']
06:40:46  
06:40:46      �[94mdef�[39;49;00m �[92m_assert_equal�[39;49;00m(cpu, gpu, float_check, path):
06:40:46          t = �[96mtype�[39;49;00m(cpu)
06:40:46          �[94mif�[39;49;00m (t �[95mis�[39;49;00m Row):
06:40:46              �[94massert�[39;49;00m �[96mlen�[39;49;00m(cpu) == �[96mlen�[39;49;00m(gpu), �[33m"�[39;49;00m�[33mCPU and GPU row have different lengths at �[39;49;00m�[33m{}�[39;49;00m�[33m CPU: �[39;49;00m�[33m{}�[39;49;00m�[33m GPU: �[39;49;00m�[33m{}�[39;49;00m�[33m"�[39;49;00m.format(path, �[96mlen�[39;49;00m(cpu), �[96mlen�[39;49;00m(gpu))
06:40:46              �[94mif�[39;49;00m �[96mhasattr�[39;49;00m(cpu, �[33m"�[39;49;00m�[33m__fields__�[39;49;00m�[33m"�[39;49;00m) �[95mand�[39;49;00m �[96mhasattr�[39;49;00m(gpu, �[33m"�[39;49;00m�[33m__fields__�[39;49;00m�[33m"�[39;49;00m):
06:40:46                  �[94mfor�[39;49;00m field �[95min�[39;49;00m cpu.__fields__:
06:40:46                      _assert_equal(cpu[field], gpu[field], float_check, path + [field])
06:40:46              �[94melse�[39;49;00m:
06:40:46                  �[94mfor�[39;49;00m index �[95min�[39;49;00m �[96mrange�[39;49;00m(�[96mlen�[39;49;00m(cpu)):
06:40:46                      _assert_equal(cpu[index], gpu[index], float_check, path + [index])
06:40:46          �[94melif�[39;49;00m (t �[95mis�[39;49;00m �[96mlist�[39;49;00m):
06:40:46              �[94massert�[39;49;00m �[96mlen�[39;49;00m(cpu) == �[96mlen�[39;49;00m(gpu), �[33m"�[39;49;00m�[33mCPU and GPU list have different lengths at �[39;49;00m�[33m{}�[39;49;00m�[33m CPU: �[39;49;00m�[33m{}�[39;49;00m�[33m GPU: �[39;49;00m�[33m{}�[39;49;00m�[33m"�[39;49;00m.format(path, �[96mlen�[39;49;00m(cpu), �[96mlen�[39;49;00m(gpu))
06:40:46              �[94mfor�[39;49;00m index �[95min�[39;49;00m �[96mrange�[39;49;00m(�[96mlen�[39;49;00m(cpu)):
06:40:46                  _assert_equal(cpu[index], gpu[index], float_check, path + [index])
06:40:46          �[94melif�[39;49;00m (t �[95mis�[39;49;00m pytypes.GeneratorType):
06:40:46              index = �[94m0�[39;49;00m
06:40:46              �[90m# generator has no zip :( so we have to do this the hard way�[39;49;00m
06:40:46              done = �[94mFalse�[39;49;00m
06:40:46              �[94mwhile�[39;49;00m �[95mnot�[39;49;00m done:
06:40:46                  sub_cpu = �[94mNone�[39;49;00m
06:40:46                  sub_gpu = �[94mNone�[39;49;00m
06:40:46                  �[94mtry�[39;49;00m:
06:40:46                      sub_cpu = �[96mnext�[39;49;00m(cpu)
06:40:46                  �[94mexcept�[39;49;00m �[96mStopIteration�[39;49;00m:
06:40:46                      done = �[94mTrue�[39;49;00m
06:40:46      
06:40:46                  �[94mtry�[39;49;00m:
06:40:46                      sub_gpu = �[96mnext�[39;49;00m(gpu)
06:40:46                  �[94mexcept�[39;49;00m �[96mStopIteration�[39;49;00m:
06:40:46                      done = �[94mTrue�[39;49;00m
06:40:46      
06:40:46                  �[94mif�[39;49;00m done:
06:40:46                      �[94massert�[39;49;00m sub_cpu == sub_gpu �[95mand�[39;49;00m sub_cpu == �[94mNone�[39;49;00m, �[33m"�[39;49;00m�[33mCPU and GPU generators have different lengths at �[39;49;00m�[33m{}�[39;49;00m�[33m"�[39;49;00m.format(path)
06:40:46                  �[94melse�[39;49;00m:
06:40:46                      _assert_equal(sub_cpu, sub_gpu, float_check, path + [index])
06:40:46      
06:40:46                  index = index + �[94m1�[39;49;00m
06:40:46          �[94melif�[39;49;00m (t �[95mis�[39;49;00m �[96mdict�[39;49;00m):
06:40:46              �[90m# TODO eventually we need to split this up so we can do the right thing for float/double�[39;49;00m
06:40:46              �[90m# values stored under the map some where, especially for NaNs�[39;49;00m
06:40:46              �[94massert�[39;49;00m cpu == gpu, �[33m"�[39;49;00m�[33mGPU and CPU map values are different at �[39;49;00m�[33m{}�[39;49;00m�[33m"�[39;49;00m.format(path)
06:40:46          �[94melif�[39;49;00m (t �[95mis�[39;49;00m �[96mint�[39;49;00m):
06:40:46  >           �[94massert�[39;49;00m cpu == gpu, �[33m"�[39;49;00m�[33mGPU and CPU int values are different at �[39;49;00m�[33m{}�[39;49;00m�[33m"�[39;49;00m.format(path)
06:40:46  �[1m�[31mE           AssertionError: GPU and CPU int values are different at [0, 'a', 'a', 'aa']�[0m
06:40:46  
06:40:46  �[1m�[31mintegration_tests/src/main/python/asserts.py�[0m:70: AssertionError
06:40:46  ----------------------------- Captured stdout call -----------------------------
06:40:46  ### CPU RUN ###
06:40:46  ### GPU RUN ###
06:40:46  ### COLLECT: GPU TOOK 0.24631977081298828 CPU TOOK 0.35289764404296875 ###

cc: @gerashegalov
@jlowe jlowe added bug Something isn't working ? - Needs Triage Need team to review and classify labels May 13, 2021
@jlowe jlowe mentioned this issue May 13, 2021
Closed
@gerashegalov gerashegalov self-assigned this May 14, 2021
@gerashegalov gerashegalov added this to the May 10 - May 21 milestone May 14, 2021
@gerashegalov gerashegalov removed the ? - Needs Triage Need team to review and classify label May 14, 2021
gerashegalov added a commit to gerashegalov/spark-rapids that referenced this issue May 14, 2021
@gerashegalov
Ignore group order in test_struct_groupby_count
3c5e3d2 
Contributes to NVIDIA#2406

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this issue May 14, 2021
Merged
@gerashegalov gerashegalov linked a pull request May 14, 2021 that will close this issue
Ignore group order in test_struct_groupby_count #2413
Merged
jlowe pushed a commit that referenced this issue May 14, 2021
@gerashegalov
Ignore group order in test_struct_groupby_count (#2413)
d473970 
Contributes to #2406

Signed-off-by: Gera Shegalov <gera@apache.org>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this issue Jun 9, 2021
@gerashegalov
Ignore group order in test_struct_groupby_count (NVIDIA#2413)
8094052 
Contributes to NVIDIA#2406

Signed-off-by: Gera Shegalov <gera@apache.org>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this issue Jun 9, 2021
@gerashegalov
Ignore group order in test_struct_groupby_count (NVIDIA#2413)
26cd62d 
Contributes to NVIDIA#2406

Signed-off-by: Gera Shegalov <gera@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerashegalov gerashegalov

Labels
bug Something isn't working
Projects
None yet
Milestone
May 10 - May 21
Development

Successfully merging a pull request may close this issue.

Ignore group order in test_struct_groupby_count gerashegalov/spark-rapids
2 participants
@jlowe @gerashegalov