Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add spark312 and spark320 versions of cache serializer #2264

Merged
merged 2 commits into from
Apr 26, 2021

Conversation

jlowe
Copy link
Member

@jlowe jlowe commented Apr 26, 2021

After #2204 the nightly tests are failing with this error:

16:52:03  E                   : java.lang.ClassNotFoundException: com.nvidia.spark.rapids.shims.spark312.ParquetCachedBatchSerializer
16:52:03  E                   	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
16:52:03  E                   	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
16:52:03  E                   	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
16:52:03  E                   	at java.lang.Class.forName0(Native Method)
16:52:03  E                   	at java.lang.Class.forName(Class.java:348)
16:52:03  E                   	at org.apache.spark.util.Utils$.classForName(Utils.scala:207)
16:52:03  E                   	at org.apache.spark.sql.execution.columnar.InMemoryRelation$.getSerializer(InMemoryRelation.scala:273)
16:52:03  E                   	at org.apache.spark.sql.execution.columnar.InMemoryRelation$.apply(InMemoryRelation.scala:301)
16:52:03  E                   	at org.apache.spark.sql.execution.CacheManager.$anonfun$cacheQuery$2(CacheManager.scala:102)
16:52:03  E                   	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
16:52:03  E                   	at org.apache.spark.sql.execution.CacheManager.cacheQuery(CacheManager.scala:97)
16:52:03  E                   	at org.apache.spark.sql.Dataset.persist(Dataset.scala:3165)
16:52:03  E                   	at org.apache.spark.sql.Dataset.cache(Dataset.scala:3175)
16:52:03  E                   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:52:03  E                   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:52:03  E                   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:52:03  E                   	at java.lang.reflect.Method.invoke(Method.java:498)
16:52:03  E                   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
16:52:03  E                   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
16:52:03  E                   	at py4j.Gateway.invoke(Gateway.java:282)
16:52:03  E                   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
16:52:03  E                   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
16:52:03  E                   	at py4j.GatewayConnection.run(GatewayConnection.java:238)
16:52:03  E                   	at java.lang.Thread.run(Thread.java:748)

The scripts are asking for a shim-specific cache serializer but there's currently only the spark311 version. This adds spark312 and spark320 versions of the cache batch serializer which are trivial derivations of the spark311 version.

@jlowe jlowe requested a review from razajafri April 26, 2021 15:02
@jlowe jlowe self-assigned this Apr 26, 2021
Signed-off-by: Jason Lowe <jlowe@nvidia.com>
@jlowe jlowe changed the title Add spark312 and spar320 versions of cache serializer Add spark312 and spark320 versions of cache serializer Apr 26, 2021
@jlowe
Copy link
Member Author

jlowe commented Apr 26, 2021

build

@sameerz sameerz added the bug Something isn't working label Apr 26, 2021
razajafri
razajafri previously approved these changes Apr 26, 2021
@razajafri
Copy link
Collaborator

Sorry, I forgot that I had reverted the change that added the files https://github.com/NVIDIA/spark-rapids/pull/2208/files.

@razajafri
Copy link
Collaborator

Can you also add the doc change? Or do you want me to add to this?

@razajafri
Copy link
Collaborator

build

@jlowe jlowe merged commit 37ef220 into NVIDIA:branch-0.5 Apr 26, 2021
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
* Add spark312 and spark320 versions of cache serializer

Signed-off-by: Jason Lowe <jlowe@nvidia.com>

* Update serializer docs
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
* Add spark312 and spark320 versions of cache serializer

Signed-off-by: Jason Lowe <jlowe@nvidia.com>

* Update serializer docs
@jlowe jlowe deleted the shim-serializer branch September 10, 2021 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants