Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qualification tool: update SQL Df value used and look at jobs in SQL #5612

Merged
merged 141 commits into from
May 24, 2022
Merged

Qualification tool: update SQL Df value used and look at jobs in SQL #5612

merged 141 commits into from
May 24, 2022

Conversation

tgravescs
Copy link
Collaborator

Fixes #5570

Ok this has a couple of changes.

  1. Change to use the spark reported SQL DF duration instead of the stage duration since we might not have all stages.
  2. I found that the sql mapping to stages can miss quite a few stages, especially in the case an entire SQL query doesn't have any mappings. In these cases look at the SQL to Job mapping and get the stages from those and just average everything in that SQL query across those times.
  3. Fixed a couple bugs.
  4. removed a decimal check we should have removed a while.

This change makes the DF times much closer then what they were in previous version of the tool.

amahussein and others added 30 commits May 10, 2022 15:59
@amahussein
QualificationTool. Add speedup information to AppSummaryInfo
55c8384 
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
@amahussein
address review comments
82a5a9b 
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
@tgravescs
debug
9d0cbc3
@tgravescs
check for dataset
af060fb
@tgravescs
change dataset check for all
ed74657 
Signed-off-by: Thomas Graves <tgraves@apache.org>
@tgravescs
test unsupported time
17b39c7
@tgravescs
more changes
6b2d419
@tgravescs
fix to string
c14e2e3
@tgravescs
fix including wholestage codegen
da9665d
@tgravescs
put unsupported Dur
a68deae
@tgravescs
calculate duration of non sql stages
c965c1c
@tgravescs
change to get stage task time
53668a4
@tgravescs
combine
480ddee
@tgravescs
hooking up final output
c7aa64b
@tgravescs
initial scores changes
9b09b23
@tgravescs
logging
5ad9ec4
@tgravescs
logging
5af6d70
@tgravescs
update factor
3abed14
@tgravescs
gturn off some logging:
c0f3bb5
@tgravescs
debug
b8b0ad9
@tgravescs
Merge remote-tracking branch 'origin/branch-22.06' into qualStageMetrics
e941ca4
@tgravescs
track execs without stages
718a32a
@tgravescs
Add in exec info output
14419e8
@tgravescs
fix output
1a1b5d1
@tgravescs
add sorting
f715975
@tgravescs
fix output
5fbe980
@tgravescs
fix output sizes
aaf3873
@tgravescs
use plan infos without execs removed
ad365cd
@tgravescs
output children node ids
e5c3551
@tgravescs
output stages info
851b712
@tgravescs tgravescs added this to the May 23 - Jun 3 milestone May 24, 2022
@tgravescs tgravescs self-assigned this May 24, 2022
@tgravescs
Copy link
Collaborator Author

build

@tgravescs
Copy link
Collaborator Author

build

@tgravescs
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit 808bcc2 into NVIDIA:branch-22.06 May 24, 2022
@tgravescs tgravescs deleted the qualDFTime branch May 24, 2022 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nartal1 nartal1 nartal1 approved these changes

@amahussein amahussein Awaiting requested review from amahussein

Assignees

@tgravescs tgravescs

Labels
tools
Projects
None yet
Milestone
May 23 - Jun 3
Development

Successfully merging this pull request may close these issues.

Qualification tool investigate SQL Df value reported
3 participants
@tgravescs @nartal1 @amahussein