-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
openlineage: adjust log levels #34801
Conversation
2e30896
to
0f9fe36
Compare
ff708c5
to
50cd52e
Compare
@@ -82,7 +82,7 @@ def extract(self) -> OperatorLineage | None: | |||
self.operator.__class__.__module__ + "." + self.operator.__class__.__name__ | |||
) | |||
if fully_qualified_class_name in self.disabled_operators: | |||
self.log.warning( | |||
self.log.info( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be DEBUG too, no? i.e user performed an action to disable the lineage for specific operators
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch.
@@ -144,10 +144,10 @@ def _get_openlineage_facets(self, get_facets_method, *args) -> OperatorLineage | | |||
job_facets=facets.job_facets, | |||
) | |||
except ImportError: | |||
self.log.exception( | |||
self.log.error( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.log.error( | |
self.log.exception( |
exception seems fine here
return | ||
self.executor.submit(self.adapter.dag_success, dag_run=dag_run, msg=msg) | ||
|
||
@hookimpl | ||
def on_dag_run_failed(self, dag_run: DagRun, msg: str): | ||
if not self.executor: | ||
self.log.error("Executor have not started before `on_dag_run_failed`") | ||
self.log.warning("Executor have not started before `on_dag_run_failed`") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.log.warning("Executor have not started before `on_dag_run_failed`") | |
self.log.debug("Executor have not started before `on_dag_run_failed`") |
@@ -194,14 +194,14 @@ def on_dag_run_running(self, dag_run: DagRun, msg: str): | |||
@hookimpl | |||
def on_dag_run_success(self, dag_run: DagRun, msg: str): | |||
if not self.executor: | |||
self.log.error("Executor have not started before `on_dag_run_success`") | |||
self.log.warning("Executor have not started before `on_dag_run_success`") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.log.warning("Executor have not started before `on_dag_run_success`") | |
self.log.debug("Executor have not started before `on_dag_run_success`") |
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
50cd52e
to
080e05d
Compare
Thanks @kaxil ! I've commited changes as suggested. |
OpenLineage provider is reported to be a bit too verbose. This PR attempts to lower logging levels of some messages and also cleans up the code.