[Bug]: Benchmark tool raises exception #1109

marcodiri · 2023-05-26T12:23:54Z

Describe the bug

I'm trying to use the benchmark script but it results in the error below.

Dataset

N/A

Model

N/A

Steps to reproduce the behavior

I'm running the benchmark notebook from this repo, but it seems to be outdated because I had to add !pip install -r requirements/loggers.txt and use the config from the tutorial in the docs.

OS information

I'm using Colab with the following:

OS: Ubuntu 20.04.5 LTS
Python version: 3.10.11
Anomalib version: 1.0.0dev
PyTorch version: 2.0.1+cu118
CUDA/cuDNN version: 11.8
GPU models and configuration: 1x Tesla T4

Expected behavior

No errors.

Screenshots

No response

Pip/GitHub

GitHub

What version/branch did you use?

No response

Configuration YAML

seed: 42
compute_openvino: false
hardware:
  - gpu
writer:
  - tensorboard
grid_search:
  dataset:
    name: hazelnut
    format: folder
    path: /content/anomalib/datasets/hazelnut_toy
    normal_dir: good # name of the folder containing normal images.
    abnormal_dir: colour # name of the folder containing abnormal images.
    normal_test_dir: null
    task: segmentation # classification or segmentation
    mask: /content/anomalib/datasets/hazelnut_toy/mask/colour
    extensions: .jpg
    split_ratio: 0.2
    category:
      - colour
      - crack
    image_size: [128, 256]
  model_name:
    - padim
    - stfpm

Logs

2023-05-26 12:22:47.200570: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-26 12:22:48.684323: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Benchmarking started 🏃‍♂️. This will take a while ⏲ depending on your configuration.
2023-05-26 12:22:58.393493: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Global seed set to 42
2023-05-26 12:23:00,660 - anomalib.data - INFO - Loading the datamodule
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/content/anomalib/tools/benchmarking/benchmark.py", line 69, in wrapper
    value = func(*args, **kwargs)
  File "/content/anomalib/tools/benchmarking/benchmark.py", line 92, in get_single_model_metrics
    datamodule = get_datamodule(model_config)
  File "/usr/local/lib/python3.10/dist-packages/anomalib/data/__init__.py", line 117, in get_datamodule
    root=config.dataset.root,
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 355, in __getattr__
    self._format_and_raise(
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 480, in _get_node
    raise ConfigKeyError(f"Missing key {key!s}")
omegaconf.errors.ConfigAttributeError: Missing key root
    full_key: dataset.root
    object_type=dict

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/content/anomalib/tools/benchmarking/benchmark.py", line 170, in compute_on_gpu
    model_metrics = sweep(run_config, device, seed, compute_openvino)
  File "/content/anomalib/tools/benchmarking/benchmark.py", line 285, in sweep
    model_metrics = get_single_model_metrics(model_config=model_config, openvino_metrics=convert_openvino)
  File "/content/anomalib/tools/benchmarking/benchmark.py", line 71, in wrapper
    raise Exception(buf.getvalue()) from exp
Exception
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 201, in distribute_over_gpus
    job.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 315, in <module>
    distribute(_sweep_config)
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 232, in distribute
    distribute_over_gpus(config, folder=runs_folder)
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 203, in distribute_over_gpus
    raise Exception(f"Error occurred while computing benchmark on GPU {job}") from exc
Exception: Error occurred while computing benchmark on GPU <Future at 0x7f2c088a4d90 state=finished raised Exception>

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

pirnerjonas · 2023-06-05T13:12:28Z

Hi :)

if I look at the newer benchmarking notebook in the repo it looks like one of the dataset keywords changed from path to root:

folder_padim = """
dataset:
  name: hazelnut
  format: folder
  path: /content/anomalib/datasets/hazelnut_toy

from omegaconf import OmegaConf

dataset_configuration_str = f"""
  name: hazelnut
  format: folder
  root: {str(root_directory / "datasets" / "hazelnut_toy")}

does it help if you change it accordingly?

marcodiri · 2023-06-14T08:18:23Z

Another problem is the docs say:

By default, compute_openvino is set to False to support instances where OpenVINO requirements are not installed in the environment

But if you don't install it you get

Traceback (most recent call last):
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 30, in <module>
    from anomalib.deploy import export
  File "/usr/local/lib/python3.10/dist-packages/anomalib/deploy/__init__.py", line 7, in <module>
    from .inferencers import Inferencer, OpenVINOInferencer, TorchInferencer
  File "/usr/local/lib/python3.10/dist-packages/anomalib/deploy/inferencers/__init__.py", line 7, in <module>
    from .openvino_inferencer import OpenVINOInferencer
  File "/usr/local/lib/python3.10/dist-packages/anomalib/deploy/inferencers/openvino_inferencer.py", line 24, in <module>
    raise ImportError("OpenVINO is not installed. Please install OpenVINO to use OpenVINOInferencer.")
ImportError: OpenVINO is not installed. Please install OpenVINO to use OpenVINOInferencer.

even if compute_openvino: false in the config.

marcodiri · 2023-06-14T08:32:51Z

Hi :)

if I look at the newer benchmarking notebook in the repo it looks like one of the dataset keywords changed from path to root:
folder_padim = """
dataset:
  name: hazelnut
  format: folder
  path: /content/anomalib/datasets/hazelnut_toy
from omegaconf import OmegaConf

dataset_configuration_str = f"""
  name: hazelnut
  format: folder
  root: {str(root_directory / "datasets" / "hazelnut_toy")}
does it help if you change it accordingly?

Thanks for the reply.
I also had to change mask to mask_dir to get something running, but it seems to totally ignore the custom dataset and instead uses the default in the model config, indeed at the end I get a results folder for mvtec bottle which is the default in padim config.
My guess is that in the model default config they use dataset.path while in the benchmark config they use dataset.root so it's not overwritten correctly.
Also it terminates early with an error

Traceback (most recent call last):
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 315, in <module>
    distribute(_sweep_config)
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 232, in distribute
    distribute_over_gpus(config, folder=runs_folder)
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 203, in distribute_over_gpus
    raise Exception(f"Error occurred while computing benchmark on GPU {job}") from exc
Exception: Error occurred while computing benchmark on GPU <Future at 0x7ff326a32770 state=finished raised Exception>

So probably the whole benchmark script is to be updated for the latest changes.

samet-akcay · 2023-07-22T13:32:28Z

Another problem is the docs say:

By default, compute_openvino is set to False to support instances where OpenVINO requirements are not installed in the environment

But if you don't install it you get

Traceback (most recent call last):
  File "/content/anomalib/./tools/benchmarking/benchmark.py", line 30, in <module>
    from anomalib.deploy import export
  File "/usr/local/lib/python3.10/dist-packages/anomalib/deploy/__init__.py", line 7, in <module>
    from .inferencers import Inferencer, OpenVINOInferencer, TorchInferencer
  File "/usr/local/lib/python3.10/dist-packages/anomalib/deploy/inferencers/__init__.py", line 7, in <module>
    from .openvino_inferencer import OpenVINOInferencer
  File "/usr/local/lib/python3.10/dist-packages/anomalib/deploy/inferencers/openvino_inferencer.py", line 24, in <module>
    raise ImportError("OpenVINO is not installed. Please install OpenVINO to use OpenVINOInferencer.")
ImportError: OpenVINO is not installed. Please install OpenVINO to use OpenVINOInferencer.

even if compute_openvino: false in the config.

@marcodiri, regarding your OpenVINO issue, PR #1182 addressed this. Rather than throwing import error, it only throws a warning message.

samet-akcay · 2023-07-22T13:41:16Z

For the rest of the stuff, we are working on a new benchmarking script that we would like to release in v1.1.0

samet-akcay assigned ashwinvaidya17 Jun 9, 2023

blaz-r mentioned this issue Jun 27, 2023

[Bug] Benchmarking tool running immediately and not producing any output #1116

Closed

samet-akcay added the Bug Something isn't working label Jul 21, 2023

samet-akcay added this to the v1.0.0 milestone Jul 21, 2023

samet-akcay added the Documentation Improvements or additions to documentation label Jul 22, 2023

samet-akcay modified the milestones: v1.0.0, v1.1.0 Jul 22, 2023

samet-akcay mentioned this issue Aug 7, 2023

Update benchmarking notebook #1242

Merged

9 tasks

samet-akcay closed this as completed in #1242 Aug 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Benchmark tool raises exception #1109

[Bug]: Benchmark tool raises exception #1109

marcodiri commented May 26, 2023

pirnerjonas commented Jun 5, 2023

marcodiri commented Jun 14, 2023

marcodiri commented Jun 14, 2023

samet-akcay commented Jul 22, 2023

samet-akcay commented Jul 22, 2023

[Bug]: Benchmark tool raises exception #1109

[Bug]: Benchmark tool raises exception #1109

Comments

marcodiri commented May 26, 2023

Describe the bug

Dataset

Model

Steps to reproduce the behavior

OS information

Expected behavior

Screenshots

Pip/GitHub

What version/branch did you use?

Configuration YAML

Logs

Code of Conduct

pirnerjonas commented Jun 5, 2023

marcodiri commented Jun 14, 2023

marcodiri commented Jun 14, 2023

samet-akcay commented Jul 22, 2023

samet-akcay commented Jul 22, 2023