-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Airflow agent #1725
Airflow agent #1725
Conversation
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw What's the status of this PR? |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #1725 +/- ##
==========================================
+ Coverage 55.04% 55.24% +0.19%
==========================================
Files 296 300 +4
Lines 22242 22382 +140
Branches 3357 3359 +2
==========================================
+ Hits 12244 12365 +121
- Misses 9835 9855 +20
+ Partials 163 162 -1
☔ View full report in Codecov by Sentry. |
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few comments. Looks really close.
TL;DR
Airflow agent allows you to seamlessly run Airflow tasks in the Flyte workflow without changing code. All the airflow tasks will be run on an airflow agent (long-running server) instead of launching a new pod to run it, which dramatically reduces overhead.
Type
Are all requirements met?
Complete description
DataprocOperator
bashSensor
HiveSensor
FileSensor
TimeSensor
PythonSensor
Tracking Issue
NA
Follow-up issue
NA