Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] IDE complains about duplicate ShimBasePythonRunner instances #3462

Closed
revans2 opened this issue Sep 13, 2021 · 2 comments
Closed

[BUG] IDE complains about duplicate ShimBasePythonRunner instances #3462

revans2 opened this issue Sep 13, 2021 · 2 comments
Labels
bug Something isn't working P1 Nice to have for release

Comments

@revans2
Copy link
Collaborator

revans2 commented Sep 13, 2021

Describe the bug
I want to debug a test in 3.2.0 using the IDE, but I cannot because the IDE is not interpreting maven profiles correctly. It says that there are duplicate versions of ShimBasePythonRunner in the source tree.

$ find . -type f -iname ShimBasePythonRunner.scala
./sql-plugin/src/main/320/scala/org/apache/spark/rapids/shims/v2/api/python/ShimBasePythonRunner.scala
./sql-plugin/src/main/311db/scala/org/apache/spark/rapids/shims/v2/api/python/ShimBasePythonRunner.scala
./sql-plugin/src/main/301db/scala/org/apache/spark/rapids/shims/v2/api/python/ShimBasePythonRunner.scala
./sql-plugin/src/main/301+-nondb/scala/org/apache/spark/rapids/shims/v2/api/python/ShimBasePythonRunner.scala

When I manually go digging through the IDE's directories it looks like it thinks 301+-nondb is the other version.

From the name of the directory 301+-nondb it should be included in the build for 3.2.0, but it is not.

        <profile>
            <id>release320</id>
            <activation>
                <property>
                    <name>buildver</name>
                    <value>320</value>
                </property>
            </activation>
...
                                    <sources>
                                        <source>${project.basedir}/src/main/320/scala</source>
                                        <source>${project.basedir}/src/main/311+-all/scala</source>
                                    </sources>
...
        </profile>

Which I verified in the maven build.

[INFO] --- build-helper-maven-plugin:3.2.0:add-source (add-profile-src-31+) @ rapids-4-spark-udf-examples_2.12 ---
[INFO] Source directory: /home/roberte/src/rapids-plugin-4-spark/udf-examples/src/main/320/scala added.
[INFO] Source directory: /home/roberte/src/rapids-plugin-4-spark/udf-examples/src/main/311+-all/scala added.
[INFO] 

I will file a separate issue about the name of the directory. But as it is right now I cannot run/debug tests in the IDE, and I have no idea how to manually make this work either. I tried changing the profiles selected in the IDE's maven tab, but that didn't work at all.

Steps/Code to reproduce bug
Import your project into IntelliJ with -Dbuildver=320 and then go to AdaptiveQueryExecSuite right click and try to run it.

Expected behavior
It should be able to run

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Sep 13, 2021
@Salonijain27 Salonijain27 added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Sep 14, 2021
@revans2
Copy link
Collaborator Author

revans2 commented Sep 17, 2021

I am getting different errors now, but I still cannot do the simple thing of going to a test in the IDE, right clicking and saying run. I have to do remote debugging locally if I want to debug something

@Salonijain27 Salonijain27 added this to the Sep 27 - Oct 1 milestone Sep 24, 2021
@revans2
Copy link
Collaborator Author

revans2 commented Sep 27, 2021

Works now thanks @gerashegalov

@revans2 revans2 closed this as completed Sep 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P1 Nice to have for release
Projects
None yet
Development

No branches or pull requests

2 participants