[BUG] test_no_fallback_when_ansi_enabled failed in databricks

[abellina]

@razajafri found this in one of his PRs, the results from the CPU and GPU do not match for test_no_fallback_when_ansi_enabled, which I added here #3597:

First few rows from the CPU:

 [Row(a=None, first(b)=118, last(b)=507, min(b)=118, max(b)=507), Row(a=1, first(b)=507, last(b)=507, min(b)=507, max(b)=507),  Row(a=2, first(b)=848, last(b)=848, min(b)=848, max(b)=848),

First few rows from the GPU:

[Row(a=1, first(b)=507, last(b)=507, min(b)=507, max(b)=507), Row(a=2, first(b)=848, last(b)=848, min(b)=848, max(b)=848)

Last row on the GPU:

Row(a=None, first(b)=118, last(b)=507, min(b)=118, max(b)=507

I am not entirely sure how this is happening, after a coalesce(1) and orderBy(every column).

[jlowe]

I am not entirely sure how this is happening, after a coalesce(1) and orderBy(every column)

But the orderBy is not the last thing in the query?

        df = gen_df(spark, [('a', data_gen), ('b', data_gen)], length=100)
        # coalescing because of first/last are not deterministic
        df = df.coalesce(1).orderBy("a", "b")
        return df.groupBy('a').agg(f.first("b"), f.last("b"), f.min("b"), f.max("b"))

What's preventing a Spark implementation from hash-aggregating the grouping and therefore the resulting order of the output rows being non-deterministic because it's dumping hash table contents?

[abellina]

Ok @jlowe is absolutely right. Just adding the ignore order marker here should do it.

[razajafri]

Do you want me to do this as part of my #3330? Adding skip is the same amount of work as adding ignore_order