Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Qualification tool checks if there is any "Scan JDBCRelation" and count it as "problematic" #3432

Closed
viadea opened this issue Sep 9, 2021 · 1 comment · Fixed by #3937
Assignees
Labels
feature request New feature or request P1 Nice to have for release tools

Comments

@viadea
Copy link
Collaborator

viadea commented Sep 9, 2021

Is your feature request related to a problem? Please describe.
I wish qualification tool can detect the "Scan JDBCRelation" from SQL plan and count it as "problematic".

Describe the solution you'd like
A clear and concise description of what you want to happen.

The qualification tool can detect the "Scan JDBCRelation" from SQL plan.

@viadea viadea added feature request New feature or request ? - Needs Triage Need team to review and classify tools labels Sep 9, 2021
@Salonijain27 Salonijain27 removed the ? - Needs Triage Need team to review and classify label Sep 14, 2021
@viadea viadea added the P0 Must have for release label Sep 21, 2021
@tgravescs tgravescs added P1 Nice to have for release and removed P0 Must have for release labels Sep 24, 2021
@viadea
Copy link
Collaborator Author

viadea commented Oct 19, 2021

For example, I generate event log with Scan JDBC key words by using below sample code which is to scan a mysql table(used by hive metastore).

spark-shell --jars /usr/share/java/mysql.jar

val jdbcDF = spark.read.format("jdbc").
             option("driver", "com.mysql.jdbc.Driver").
             option("url", "jdbc:mysql://localhost:3306/hive?useSSL=false").
             option("dbtable", "TBLS").option("user", "hive").
             option("password", "xxx").
             load()

jdbcDF.createOrReplaceTempView("t")

spark.sql("select count(distinct TBL_ID) from t").show()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request P1 Nice to have for release tools
Projects
None yet
4 participants