Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dependencies needed for some ml integration tests #25548

Closed
wants to merge 2 commits into from

Conversation

damccorm
Copy link
Contributor

@damccorm damccorm commented Feb 17, 2023

Right now, our inference postcommit tests aren't running correctly (context - #25496 (comment)), this gets them running.

Fixes #25547


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@damccorm
Copy link
Contributor Author

Run Python 3.8 PostCommit

'sqlalchemy>=1.3,<2.0',
'tensorflow>=2.11.0,<3',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add these in the test?

Initially, there was a disagreement of adding ML libraries to the apache_beam[test], as it could install lot of unncessary dependencies and also big in size(tensorflow, torch) on to the user machine.

Also, I am removing tensorflow from the python/containers/base_image_requirements_manual.txt since it is blocking the upgrade of protobuf.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am in favor of adding these dependencies to the respective gradle/tox tasks.

@AnandInguva
Copy link
Contributor

AnandInguva commented Feb 18, 2023

I think they are not getting skipped. They are not visible in Jenkins UI Test Result.

the tests are actually running but not visible in the Jenkins UI. You can look in the console log and search for InferencePostCommitIT in beam_PostCommit_Python37

dependsOn(":sdks:python:test-suites:direct:py37:inferencePostCommitIT")
and [beam_PostCommit_Python37](https://ci-beam.apache.org/job/beam_PostCommit_Python39/) test suite
dependsOn(":sdks:python:test-suites:direct:py39:inferencePostCommitIT")
.

Console log: https://ci-beam.apache.org/job/beam_PostCommit_Python39/1480/consoleFull

The tests we see in the Test Results UI of jenkins is of the gradle task

dependsOn(":sdks:python:test-suites:dataflow:py39:postCommitIT")
. In this gradle task, we skip all the ML tests since they are run on different gradle task InferencePostCommitIT

@damccorm damccorm closed this Feb 21, 2023
@damccorm damccorm deleted the users/damccorm/ml-integration-tests branch February 21, 2023 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task]: Add pytorch, onnx dependencies for integration tests
2 participants