Add in reporting of time taken to transition plan to GPU #3315
Conversation
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
revans2
added
the
task
|
build |
| val ret = applyOverrides(updatedPlan, conf) | ||
| val end = System.nanoTime() | ||
| if (conf.shouldExplain) { | ||
| val timeMs = (end - start) / 1000000.0 |
There was a problem hiding this comment.
Consider NANOSECONDS.toMillis(end - start)
Also consider a utility function to measure duration to avoid duplicating code
def withDurationLog[T](block: => T, conf: RapidsConf, format: String): T There was a problem hiding this comment.
I started off using NANOSECONDS.toMillis, but it returns a long, not a double. If I wanted just millisecond precision I would have measured the values in milliseconds to begin with. I also looked at MILLISECONDS.toNanos(1) to produce the magic number, but decided against it. Happy to move back to that if you think it is better.
I also thought about writing a utility, but the code is only in 2 places and it is 6 lines of code in each place. So best case I would save 3 lines of code total. 12 lines as it is now vs 6 lines for the utility body + 1 line for the utility def + 2 lines to call the utility from each place. It just didn't feel like it was worth it at this time. But for code cleanliness if you want me to I will do it.
There was a problem hiding this comment.
I see your point about fractional millis.
It's easily conceivable that we want to instrument more and more durations in code and this PR already needs it twice. Thus I'd favor a util method making it easy.
|
build |
|
build |
|
build |
|
The previous build failed with what appeared to be running out of host memory. |
|
Need to resolve the merge conflict otherwise this looks ready to go. |
|
build |
|
Sorry I had to upmerge |
@jlowe asked me earlier about how long it takes to take a CPU plan and convert it into a GPU enabled plan. I had dug into this early on, but I have not done anything with it recently. Sadly there is no good way to report this type of a metric query wide. Nor is there a good way to measure how long it takes compared to how long Spark takes in total to parse and optimize the query. But this should provide users with a little bit of knowledge if they ask for it.
I also cleaned up a few small code warnings while I was at it.