Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automated self-profile tests #10893

Open
jlowe opened this issue May 24, 2024 · 0 comments
Open

Add automated self-profile tests #10893

jlowe opened this issue May 24, 2024 · 0 comments
Labels
feature request New feature or request

Comments

@jlowe
Copy link
Member

jlowe commented May 24, 2024

Is your feature request related to a problem? Please describe.
The self-profiler in #10870 needs automated testing to ensure it keeps working. However this cannot be done as a simple integration test since it requires special startup-only configs.

Describe the solution you'd like
Add an explicit section to the spark-tests script that will launch sample self-profiling sessions. The tests should minimally startup the Spark session with the profiling path prefix config pointing to a filesystem (distributed would be best, but local could be used if no distributed filesystem is available/configured in the test environment). A simple, sample query can be run in the Spark session, and the test script can verify a profile is created at the expected destination. It can then further verify the profile data by using the profile converter in NVIDIA/spark-rapids-jni#2066 to convert the data to JSON and grep the results for lkey profiling landmarks, e.g.: kernel names, Acquire GPU or other common NVTX ranges, etc.

@jlowe jlowe added feature request New feature or request ? - Needs Triage Need team to review and classify labels May 24, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label May 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants