Remove InMemoryTableScanExec support for Spark 3.5+ [databricks]

[razajafri]

This PR removes support for InMemoryTableScanExec from Spark-3.5+.

Changes

• Disabled GpuInMemoryTableScanExec support by default for Spark 3.5+

Tests

• Ran cache_tests

contributes towards #8678

[revans2]

Instead of removing it, and we just have it be off by default in 3.5.x, until we get a fix into Spark? That lets us and customers experiment with it. It also makes it so that we can run tests on it and see if they are mostly working or not.

[razajafri]

The only thing I don't like about this implementation is that the GPU plan will show

! <InMemoryTableScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.columnar.InMemoryTableScanExec

which is the default message we print when an Exec isn't supported.

If others also think this should be improved then I can add an override in the shim which always returns with an error message with more info about why we aren't supporting it by calling the willNotWorkOnGpu method

[revans2]

if you use disabledByDefault it lets you put in a comment about why it is disabled.

[razajafri]

I like this approach better as it has less code churn. We may have to resort to adding shims after we reenable it.
@revans2 thanks for the suggestion. PTAL

[razajafri]

I am using an unconventional way in this PR to avoid code duplication, I hope it's OK

[revans2]

I was not concerned with the shim code duplication. There are other things we can do to work around that. I was more concerned with the impact that there would be to tests and also our ability to try and work on getting this to work properly on 3.5.0.

Have you run the cache integration tests on 3.5.0? I have to assume that they are going to fail, unless we explicitly enable/disable the InMemoryTableScanExec replacement.

[razajafri]

I was not concerned with the shim code duplication. There are other things we can do to work around that. I was more concerned with the impact that there would be to tests and also our ability to try and work on getting this to work properly on 3.5.0.

Have you run the cache integration tests on 3.5.0? I have to assume that they are going to fail, unless we explicitly enable/disable the InMemoryTableScanExec replacement.

Yes, I did run them on 311 and 350 and they all passed.
I don't remember if this was deliberate but we are never looking for InMemoryTableScanExec to be on the GPU, possibly because the cache_test.py can be run without PCBS in which case the said Exec will not be on the GPU. We can see that on this line

spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuTransitionOverrides.scala

Line 635 in a56da0c

case imts: InMemoryTableScanExec =>

that it will not check for the plan on GPU and move on.

[razajafri]

build