-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Report equivilant stages/sql ids as a part of compare #2793
Conversation
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
|
||
import SparkPlanInfoWithStage._ | ||
|
||
def debugEquals(other: Any, depth: Int = 0): Boolean = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left this in, because our plugin is constantly changing and we have not really tested this with databricks or others, so if we run into issues, this is a quick way to track down what parts of a plan need to be normalized for the comparison to work properly.
+-----------------------+-----------------------+ | ||
|app-20210329165943-0103|app-20210329170243-0018| | ||
+-----------------------+-----------------------+ | ||
|0 |0 | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it be helpful to also add time each took? I guess we can always add more later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ya I thought about that, but I didn't want to make it too complicated to start out with.
tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/CollectInformation.scala
Show resolved
Hide resolved
tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileMain.scala
Show resolved
Hide resolved
tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileMain.scala
Show resolved
Hide resolved
tools/src/main/scala/org/apache/spark/sql/rapids/tool/profiling/ApplicationInfo.scala
Outdated
Show resolved
Hide resolved
build |
For the Profile tool this helps a lot when trying to compare large complicated queries to each other.
This refactored dot generation because I wanted to reused the stage matching part of it, but I needed a better way to do it. I also wanted to be able to output dot graphs as a apart of compare, even if they were not producing a graph that compared the different applications.