Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] test_str_to_map_expr_random_delimiters with DATAGEN_SEED=1702166057 hangs #10012

Closed
revans2 opened this issue Dec 11, 2023 · 1 comment · Fixed by #10016
Closed

[BUG] test_str_to_map_expr_random_delimiters with DATAGEN_SEED=1702166057 hangs #10012

revans2 opened this issue Dec 11, 2023 · 1 comment · Fixed by #10016
Assignees
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Dec 11, 2023

Describe the bug
I was able to reproduce this locally on Spark 3.3.0

SPARK_HOME=spark_3.3.0/ TEST_PARALLEL=0 PYSP_TEST_spark_rapids_sql_explain=ALL DATAGEN_SEED=1702166057 TZ=UTC ./run_pyspark_from_build.sh --test_oom_injection_mode=always -k 'test_str_to_map_expr_random_delimiters' -s

and git version ff19012

It appears to hang in python. No queries were submitted to Spark, from what I could see.

When I tried to use gdb on it I got some great stack traces, but they were all in C, in what appeared to be creating error messages, and I ma not enough of a python expert to really dig into what is happening here.

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Dec 11, 2023
@jlowe jlowe self-assigned this Dec 11, 2023
@jlowe
Copy link
Member

jlowe commented Dec 11, 2023

The test hangs because the test has a bug in it:

    while pair_delim == keyval_delim:
        (pair_delim, keyval_delim) = with_cpu_session(
            lambda spark: gen_scalars_for_sql(delim_gen, 2, force_no_nulls=True))

The data generation will reset the generator to the beginning each time, so if it happened to generate two identical scalars it will infinitely loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants