[Task]: Add pytorch, onnx dependencies for integration tests

[riteshghorse]

What needs to happen?

Currently the python post commits are skipping the integration tests for pytorch, onnx model handlers. Add the dependencies to enable these tests.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[AnandInguva]

I think they are not getting skipped. They are not visible in Jenkins UI Test Result.

Commented at #25548 (comment) but adding here as well for reference

the tests are actually running but not visible in the Jenkins UI. You can look in the console log and search for InferencePostCommitIT in beam_PostCommit_Python37

beam/build.gradle.kts

Line 457 in 977e531

dependsOn(":sdks:python:test-suites:direct:py37:inferencePostCommitIT")

and [beam_PostCommit_Python37](https://ci-beam.apache.org/job/beam_PostCommit_Python39/) test suite

beam/build.gradle.kts

Line 476 in 977e531

dependsOn(":sdks:python:test-suites:direct:py39:inferencePostCommitIT")

.

Console log: https://ci-beam.apache.org/job/beam_PostCommit_Python39/1480/consoleFull

The tests we see in the Test Results UI of jenkins is of the gradle task

beam/build.gradle.kts

Line 470 in 977e531

dependsOn(":sdks:python:test-suites:dataflow:py39:postCommitIT")

. In this gradle task, we skip all the ML tests since they are run on different gradle task InferencePostCommitIT

[AnandInguva]

Also, for ONNX there is no gradle task to run IT tests so we can't see them in console logs. I will add it next week.

torch can be seen

[AnandInguva]

Seems like tensorflow tests are failing.

apache_beam/ml/inference/tensorflow_inference_it_test.py:62: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/examples/inference/tensorflow_mnist_classification.py:111: in run
    result = pipeline.run()
apache_beam/pipeline.py:550: in run
    return Pipeline.from_runner_api(
apache_beam/pipeline.py:577: in run
    return self.runner.run_pipeline(self, self._options)
apache_beam/runners/direct/test_direct_runner.py:42: in run_pipeline
    self.result = super().run_pipeline(pipeline, options)
apache_beam/runners/direct/direct_runner.py:131: in run_pipeline
    return runner.run_pipeline(pipeline, options)
apache_beam/runners/portability/fn_api_runner/fn_runner.py:202: in run_pipeline
    self._latest_run_result = self.run_via_runner_api(
apache_beam/runners/portability/fn_api_runner/fn_runner.py:224: in run_via_runner_api
    return self.run_stages(stage_context, stages)
apache_beam/runners/portability/fn_api_runner/fn_runner.py:455: in run_stages
    bundle_results = self._execute_bundle(
apache_beam/runners/portability/fn_api_runner/fn_runner.py:783: in _execute_bundle
    self._run_bundle(
apache_beam/runners/portability/fn_api_runner/fn_runner.py:1012: in _run_bundle
    result, splits = bundle_manager.process_bundle(
apache_beam/runners/portability/fn_api_runner/fn_runner.py:1348: in process_bundle
    result_future = self._worker_handler.control_conn.push(process_bundle_req)
apache_beam/runners/portability/fn_api_runner/worker_handlers.py:379: in push
    response = self.worker.do_instruction(request)
apache_beam/runners/worker/sdk_worker.py:624: in do_instruction
    return getattr(self, request_type)(
apache_beam/runners/worker/sdk_worker.py:655: in process_bundle
    bundle_processor = self.bundle_processor_cache.get(
apache_beam/runners/worker/sdk_worker.py:486: in get
    processor = bundle_processor.BundleProcessor(
apache_beam/runners/worker/bundle_processor.py:906: in __init__
    op.setup()
apache_beam/runners/worker/operations.py:882: in setup
    self.dofn_runner.setup()
apache_beam/runners/common.py:1472: in setup
    self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
apache_beam/runners/common.py:1468: in _invoke_lifecycle_method
    self._reraise_augmented(exn)
apache_beam/runners/common.py:1508: in _reraise_augmented
    raise new_exn.with_traceback(tb)
apache_beam/runners/common.py:1466: in _invoke_lifecycle_method
    lifecycle_method()
apache_beam/runners/common.py:552: in invoke_setup
    self.signature.setup_lifecycle_method.method_value()
apache_beam/ml/inference/base.py:509: in setup
    self._model = self._load_model()
apache_beam/ml/inference/base.py:487: in _load_model
    model = self._shared_model_handle.acquire(load, tag=side_input_model_path)
apache_beam/utils/shared.py:305: in acquire
    return _shared_map.acquire(self._key, constructor_fn, tag)
apache_beam/utils/shared.py:246: in acquire
    result = control_block.acquire(constructor_fn, tag)
apache_beam/utils/shared.py:139: in acquire
    result = constructor_fn()
apache_beam/ml/inference/base.py:476: in load
    model = self._model_handler.load_model()
apache_beam/ml/inference/base.py:196: in load_model
    return self._unkeyed.load_model()
apache_beam/ml/inference/tensorflow_inference.py:116: in load_model
    return _load_model(self._model_uri, self._model_type)
apache_beam/ml/inference/tensorflow_inference.py:59: in _load_model
    return tf.keras.models.load_model(hub.resolve(model_uri))
../../build/gradleenv/1398941893/lib/python3.9/site-packages/tensorflow_hub/module_v2.py:47: in resolve
    return registry.resolver(handle)
../../build/gradleenv/1398941893/lib/python3.9/site-packages/tensorflow_hub/registry.py:51: in __call__
    return impl(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tensorflow_hub.resolver.PathResolver object at 0x7f28770f90a0>
handle = 'apache-beam-ml/models/tensorflow/mnist/'

    def __call__(self, handle):
      if not tf.compat.v1.gfile.Exists(handle):
>       raise IOError("%s does not exist." % handle)
E       OSError: apache-beam-ml/models/tensorflow/mnist/ does not exist. [while running 'RunInference/BeamML_RunInference']

Fix: #25553

[AnandInguva]

https://ci-beam.apache.org/job/beam_PostCommit_Python310/496/

The postcommits are green now. Thanks @Abacn for merging the fix.

[riteshghorse]

Oops I missed the gs:// in older pr for path and had it https://github.com/apache/beam/pull/25496/files. Thanks for fixing it @AnandInguva

[riteshghorse]

This task is obsolete as it was a problem with UI but tests were actually running.