Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate PreProcessor #795

Merged
merged 25 commits into from
Dec 19, 2022
Merged

Deprecate PreProcessor #795

merged 25 commits into from
Dec 19, 2022

Conversation

djdameln
Copy link
Contributor

@djdameln djdameln commented Dec 16, 2022

Description

  • This PR deprecates the PreProcessor class, which was already reduced to a wrapper around an albumentations transform object. With these changes we now just use albumentations transforms directly.
  • Adds some configurability options for the default transforms:
    • Previously we always normalized to ImageNet stats if no transform_config was supplied, which was a bit problematic for models (e.g. DRAEM) that do not require this as a preprocessing step. The Normalization is now optional and can be disabled by setting config.dataset.normalize to false.
    • Center cropping can now be enabled from the config. This is useful to reproduce the results of models like PatchCore which have this in the original implementation. Center cropping can be enabled by setting config.dataset.center_crop to the desired crop size.

Changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which refactors the code base)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • My code follows the pre-commit style and check guidelines of this project.
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes
  • I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).

@github-actions github-actions bot added the Docs label Dec 16, 2022
train_batch_size: 16
eval_batch_size: 16
inference_batch_size: 16
num_workers: 8
image_size: 256 # dimensions to which images are resized (mandatory)
center_crop: null # dimensions to which images are center-cropped after resizing (optional)
normalize: true # when true, images will be normalized to ImageNet statistics
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ImageNet statistics

Using the torchfx extractor, the pre-trained model is using the new model versioning system in torchvision, no?

If that's the case, I think the preprocessing statistics necessary to each pre-trained model are encapsulated in the version object (?), so it shouldn't need to be assumed anywhere that it's from imagenet (which is often but not always the case).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another comment: I think consts/enums should be used instead of bools; if a new option for normalization shows up, it's natural to add a value in the enum (while the bool would need a refactor,).

Do you think there could be a use case where the normalize needs more than 2 values?
By contradiction, is it impossible that normalize needs more options than a bool?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We default to ImageNet normalization because it's by far the most used. If a user needs custom normalization statistics they can configure it through the transform_config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another comment: I think consts/enums should be used instead of bools; if a new option for normalization shows up, it's natural to add a value in the enum (while the bool would need a refactor,).

Good idea. I'll replace the bool with an enum :)

Copy link
Collaborator

@ashwinvaidya17 ashwinvaidya17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

@@ -198,7 +199,7 @@ def __init__(
task: TaskType = TaskType.SEGMENTATION,
image_size: Optional[Union[int, Tuple[int, int]]] = None,
center_crop: Optional[Union[int, Tuple[int, int]]] = None,
normalize: bool = True,
normalization: Union[InputNormalizationMethod, str] = InputNormalizationMethod.IMAGENET,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

picky comment: could you define InputNormalizationType = Union[InputNormalizationMethod, str] somewhere?

to be consistent with

FeatureExtractorParams = Union[TimmFeatureExtractorParams, TorchFXFeatureExtractorParams]

in #748

i'm actually wondering if this is necessary since InputNormalizationMethod inherits from str and Enum, but in any case it would be nice to avoid repetitive Union[...] for maintenance purposes

Copy link
Contributor Author

@djdameln djdameln Dec 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm actually wondering if this is necessary since InputNormalizationMethod inherits from str and Enum

This is a good point, actually. This works fine:

>>> isinstance(InputNormalizationMethod.IMAGENET, InputNormalizationMethod)
True
>>> isinstance(InputNormalizationMethod.IMAGENET, str)
True

I guess the only question is if we want to allow users to instantiate the class with a string instead of the enum. So, do we want to allow something like this:

mvtec = MVTec(..., normalize="imagenet", ...)

If we do, then we need the Union typing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, do we want to allow something like this:

mvtec = MVTec(..., normalize="imagenet", ...)

Unless there it makes handling things harder somehow, I'd say yes :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case we need to keep the str typing. For now we'll merge it as, so with the Union typing. We could consider switching to custom types later if needed.

@djdameln djdameln merged commit 663692e into feature/datamodules Dec 19, 2022
@djdameln djdameln deleted the da/transforms branch December 19, 2022 14:51
djdameln added a commit that referenced this pull request Jan 6, 2023
* New datamodules design (#572)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* fix typo

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Video Datamodules (#676)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add pedestrian and avenue datasets and video utils

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* add basic visualization for video datasets

* simplify ucsdped implementation

* add ucsd and avenue to __all__

* add default value for task

* add tests for ucsd and avenue

* add tests for video dataset and utils

* add download info for avenue dataset

* add download info for ucsd pedestrian dataset

* more consistent naming

* fix path to masks folder in gt dir

* pass original image in batch to facilitate visualization

* convert mask files for avenue

* suppress warning due to torchvision bug

* fix bug in avenue masks

* store visualizations for each video in separate folder

* rename parameters

* add warning for clip_length > 1

* fix dataset tests

* fix labels tensor shape bug

* add pyav to requirements

* add description for avenue dataset

* use pathlib

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/utils/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/ucsd_ped.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* import video dataset from base

* fix bug when collecting ucsd samples

* clean up datamodules tests

* fix tests

* remove redundant test cases

* retrieve masks as numpy array

* use pathlib

* variable name

* pathlib

* use preprocesser from arguments

* fix indexing bug

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update lightning_inference.py

* Make `val split ratio` configurable (#760)

* make val split ratio configurable

* use DeprecationWarning, update config key

* Add support for Detection task type (#732)

* add basic support for detection task

* use enum for task type

* formatting

* small bugfix

* add unit tests for bounding box conversion

* update error message

* use as_tensor

* typing and docstring

* explicit keyword arguments

* simplify bbox handling in video dataset

* docstring consistency

* add missing licenses

* add whitespace for readability

* add missing license

* Update anomalib/data/utils/boxes.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Revert "Update anomalib/data/utils/boxes.py"

This reverts commit cec6138.

* add test case for custom collate function

* docstring

* add integration tests for detection dataloading

* extend and clean up datamodules tests

* add detection task type to visualizer tests

* only show pred_boxes during inference

* add detection support for torch inference

* add detection support for openvino inference

* test inference for all task types

* pylint

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* [Datamodules] Update deprecation messages (#764)

* update deprecation messages

* raise warnings as DeprecationWarning

* Improve image source parsing for Folder dataset (#784)

* mask -> mask_dir

* properly handle absolute and relative paths

* make root path parameter optional

* formatting

* path -> root

* update docs

* remove options hint for name parameter

* refactor function

* Update anomalib/config/config.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/config/config.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* make root and abnormal_dir optional

* Update anomalib/data/folder.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Synthetic anomaly for testing and validation (#634)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add synthetic dataset class

* move augmenter to data directory

* add base classes

* update docstring

* use synthetic dataset in base datamodule

* fix imports

* clean up synthetic anomaly dataset implementation

* fix mistake in augmenter

* change default split ratio

* remove accidentally added file

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* add logging message

* use val_split_ratio for synthetic validation set

* pathlib

* make synthetic anomaly available for test set

* update configs

* add tests

* simplify test set splitting logic

* update docstring

* add missing licence

* split_normal_and_anomalous -> split_by_label

* VideoAnomalib -> AnomalibVideo

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Bugfixes for Datamodules feature branch (#800)

* properly handle NoneType mask_dir and add test case

* fix wrong deprecation handling

* Deprecate PreProcessor (#795)

* deprecate PreProcessor

* update configs

* update deprecation messages

* update video dataset

* update inference dataset

* move transforms to data module

* update and extend transform tests

* fix cyclic import

* add validity checks for image size and center crop

* pass image size as tuple

* update path to get_transforms

* update error message

* fix center crop tuple conversion

* update inferencers

* remove draem transform config

* update changelog

* fix cyclic import

* add crop size vs image size check

* improve readability

* mypy

* use enum to configure input normalization

* update lightning inference

* update inference dataset

* [Datamodules] Fix bug in bbox score to image score conversion (#803)

handle empty box predictions

* Improve handling of `test_split_mode='none'` and `val_split_mode='none'` (#801)

* enable none as split mode

* use get to retrieve config keys

* update deprecation message and config key

* fix to float transform

* Detection improvements (#820)

* apply pixel threshold to bbox detections

* allow visualizing normal boxes

* normalize box scores

* fix bbox logic in base anomaly module

* boxes_scores -> box_scores

* fix inferencers

* update changelog

* update csflow config to new format

* remove unused imports

* line length

* suppress bandit warnings

* use torch rng in augmenter

* use tuple instead of list

* add missing params to dosctring

* add missing licence information

* COLS -> COLUMNS

* typing and variable naming

* remove duplicate parameter in docstring

* im_dir -> image_dir

* typing and docstring

* typing

* ValSplitMode -> ValidationSplitMode

* add missing licence

* rename variable

* remove empty comment

* remove unused class attribute

* [Detection] Compute box score when generating boxes from masks (#828)

* infer box scores from anomaly maps

* discard single pixel boxes

* revert discard single pixel boxes

* add test case for bbox scores

* update torch inferencer

* minor refactor

* revert val_split_mode -> validation_split_mode

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>
djdameln added a commit that referenced this pull request Jan 6, 2023
* New datamodules design (#572)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* fix typo

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Video Datamodules (#676)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add pedestrian and avenue datasets and video utils

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* add basic visualization for video datasets

* simplify ucsdped implementation

* add ucsd and avenue to __all__

* add default value for task

* add tests for ucsd and avenue

* add tests for video dataset and utils

* add download info for avenue dataset

* add download info for ucsd pedestrian dataset

* more consistent naming

* fix path to masks folder in gt dir

* pass original image in batch to facilitate visualization

* convert mask files for avenue

* suppress warning due to torchvision bug

* fix bug in avenue masks

* store visualizations for each video in separate folder

* rename parameters

* add warning for clip_length > 1

* fix dataset tests

* fix labels tensor shape bug

* add pyav to requirements

* add description for avenue dataset

* use pathlib

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/utils/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/ucsd_ped.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* import video dataset from base

* fix bug when collecting ucsd samples

* clean up datamodules tests

* fix tests

* remove redundant test cases

* retrieve masks as numpy array

* use pathlib

* variable name

* pathlib

* use preprocesser from arguments

* fix indexing bug

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update lightning_inference.py

* Make `val split ratio` configurable (#760)

* make val split ratio configurable

* use DeprecationWarning, update config key

* Add support for Detection task type (#732)

* add basic support for detection task

* use enum for task type

* formatting

* small bugfix

* add unit tests for bounding box conversion

* update error message

* use as_tensor

* typing and docstring

* explicit keyword arguments

* simplify bbox handling in video dataset

* docstring consistency

* add missing licenses

* add whitespace for readability

* add missing license

* Update anomalib/data/utils/boxes.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Revert "Update anomalib/data/utils/boxes.py"

This reverts commit cec6138.

* add test case for custom collate function

* docstring

* add integration tests for detection dataloading

* extend and clean up datamodules tests

* add detection task type to visualizer tests

* only show pred_boxes during inference

* add detection support for torch inference

* add detection support for openvino inference

* test inference for all task types

* pylint

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* [Datamodules] Update deprecation messages (#764)

* update deprecation messages

* raise warnings as DeprecationWarning

* Improve image source parsing for Folder dataset (#784)

* mask -> mask_dir

* properly handle absolute and relative paths

* make root path parameter optional

* formatting

* path -> root

* update docs

* remove options hint for name parameter

* refactor function

* Update anomalib/config/config.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/config/config.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* make root and abnormal_dir optional

* Update anomalib/data/folder.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Synthetic anomaly for testing and validation (#634)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add synthetic dataset class

* move augmenter to data directory

* add base classes

* update docstring

* use synthetic dataset in base datamodule

* fix imports

* clean up synthetic anomaly dataset implementation

* fix mistake in augmenter

* change default split ratio

* remove accidentally added file

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* add logging message

* use val_split_ratio for synthetic validation set

* pathlib

* make synthetic anomaly available for test set

* update configs

* add tests

* simplify test set splitting logic

* update docstring

* add missing licence

* split_normal_and_anomalous -> split_by_label

* VideoAnomalib -> AnomalibVideo

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Bugfixes for Datamodules feature branch (#800)

* properly handle NoneType mask_dir and add test case

* fix wrong deprecation handling

* Deprecate PreProcessor (#795)

* deprecate PreProcessor

* update configs

* update deprecation messages

* update video dataset

* update inference dataset

* move transforms to data module

* update and extend transform tests

* fix cyclic import

* add validity checks for image size and center crop

* pass image size as tuple

* update path to get_transforms

* update error message

* fix center crop tuple conversion

* update inferencers

* remove draem transform config

* update changelog

* fix cyclic import

* add crop size vs image size check

* improve readability

* mypy

* use enum to configure input normalization

* update lightning inference

* update inference dataset

* [Datamodules] Fix bug in bbox score to image score conversion (#803)

handle empty box predictions

* Improve handling of `test_split_mode='none'` and `val_split_mode='none'` (#801)

* enable none as split mode

* use get to retrieve config keys

* update deprecation message and config key

* fix to float transform

* Detection improvements (#820)

* apply pixel threshold to bbox detections

* allow visualizing normal boxes

* normalize box scores

* fix bbox logic in base anomaly module

* boxes_scores -> box_scores

* fix inferencers

* update changelog

* update csflow config to new format

* remove unused imports

* line length

* refactor make_mvtec_dataset to improve flexibility

* add visa dataset

* move download and extract functionality to shared location

* move visa subset splitting to separate method

* update changelog

* add tests for visa dataset

* suppress bandit url warning

* update test

* address PR comments

* suppress bandit warnings

* use torch rng in augmenter

* fix logic in prepare_data

* add comments

* cleaner zipfile import

* address PR comments

* use tuple instead of list

* add missing params to dosctring

* add missing licence information

* COLS -> COLUMNS

* typing and variable naming

* remove duplicate parameter in docstring

* im_dir -> image_dir

* typing and docstring

* typing

* ValSplitMode -> ValidationSplitMode

* add missing licence

* rename variable

* remove empty comment

* remove unused class attribute

* [Detection] Compute box score when generating boxes from masks (#828)

* infer box scores from anomaly maps

* discard single pixel boxes

* revert discard single pixel boxes

* add test case for bbox scores

* update torch inferencer

* minor refactor

* revert val_split_mode -> validation_split_mode

* use empty string instead of nan as empty mask path

* typing

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>
djdameln added a commit that referenced this pull request Jan 6, 2023
* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add pedestrian and avenue datasets and video utils

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* Created rbad directory

* Keep refactoring region-extractor

* rename new_image_sizes to transformed_image_sizes

* Renamed the variables in region extractor

* post-process function in region extractor

* Refactored tile-boxes function

* Added feature extractor

* Add main.py

* Added feature extractor to tests

* Update the jupyter notebook

* Uncomment loa weights from region.py

* Add feature and region extractors

* Finished feature-extractor implementation

* Rename the algo as rkde

* New datamodules design (#572)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* fix typo

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add basic visualization for video datasets

* simplify ucsdped implementation

* TODO: Investigate torch_model

* add ucsd and avenue to __all__

* add default value for task

* add tests for ucsd and avenue

* add tests for video dataset and utils

* add download info for avenue dataset

* add download info for ucsd pedestrian dataset

* more consistent naming

* fix path to masks folder in gt dir

* pass original image in batch to facilitate visualization

* convert mask files for avenue

* suppress warning due to torchvision bug

* fix bug in avenue masks

* store visualizations for each video in separate folder

* rename parameters

* add warning for clip_length > 1

* fix dataset tests

* fix labels tensor shape bug

* add pyav to requirements

* Add TODO notes

* add todo notes

* add description for avenue dataset

* use pathlib

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/utils/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/ucsd_ped.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* import video dataset from base

* fix bug when collecting ucsd samples

* clean up datamodules tests

* fix tests

* remove redundant test cases

* add test case for normality model

* retrieve masks as numpy array

* use pathlib

* variable name

* pathlib

* use preprocesser from arguments

* fix indexing bug

* Video Datamodules (#676)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add base classes

* update docstring

* fix imports

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add pedestrian and avenue datasets and video utils

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* add basic visualization for video datasets

* simplify ucsdped implementation

* add ucsd and avenue to __all__

* add default value for task

* add tests for ucsd and avenue

* add tests for video dataset and utils

* add download info for avenue dataset

* add download info for ucsd pedestrian dataset

* more consistent naming

* fix path to masks folder in gt dir

* pass original image in batch to facilitate visualization

* convert mask files for avenue

* suppress warning due to torchvision bug

* fix bug in avenue masks

* store visualizations for each video in separate folder

* rename parameters

* add warning for clip_length > 1

* fix dataset tests

* fix labels tensor shape bug

* add pyav to requirements

* add description for avenue dataset

* use pathlib

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/avenue.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/utils/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/base/video.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/data/ucsd_ped.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* import video dataset from base

* fix bug when collecting ucsd samples

* clean up datamodules tests

* fix tests

* remove redundant test cases

* retrieve masks as numpy array

* use pathlib

* variable name

* pathlib

* use preprocesser from arguments

* fix indexing bug

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* properly handle batch processing

* include batch index in rois tensor

* return rkde results as lists

* update default rkde config

* add basic support for detection task

* use enum for task type

* formatting

* small bugfix

* add unit tests for bounding box conversion

* update error message

* use as_tensor

* typing and docstring

* explicit keyword arguments

* simplify bbox handling in video dataset

* docstring consistency

* add missing licenses

* add whitespace for readability

* add missing license

* Update anomalib/data/utils/boxes.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Revert "Update anomalib/data/utils/boxes.py"

This reverts commit cec6138.

* add test case for custom collate function

* docstring

* add integration tests for detection dataloading

* extend and clean up datamodules tests

* add detection task type to visualizer tests

* Update lightning_inference.py

* only show pred_boxes during inference

* add detection support for torch inference

* add detection support for openvino inference

* test inference for all task types

* pylint

* Make `val split ratio` configurable (#760)

* make val split ratio configurable

* use DeprecationWarning, update config key

* Add support for Detection task type (#732)

* add basic support for detection task

* use enum for task type

* formatting

* small bugfix

* add unit tests for bounding box conversion

* update error message

* use as_tensor

* typing and docstring

* explicit keyword arguments

* simplify bbox handling in video dataset

* docstring consistency

* add missing licenses

* add whitespace for readability

* add missing license

* Update anomalib/data/utils/boxes.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Revert "Update anomalib/data/utils/boxes.py"

This reverts commit cec6138.

* add test case for custom collate function

* docstring

* add integration tests for detection dataloading

* extend and clean up datamodules tests

* add detection task type to visualizer tests

* only show pred_boxes during inference

* add detection support for torch inference

* add detection support for openvino inference

* test inference for all task types

* pylint

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* [Datamodules] Update deprecation messages (#764)

* update deprecation messages

* raise warnings as DeprecationWarning

* update rkde

* Improve image source parsing for Folder dataset (#784)

* mask -> mask_dir

* properly handle absolute and relative paths

* make root path parameter optional

* formatting

* path -> root

* update docs

* remove options hint for name parameter

* refactor function

* Update anomalib/config/config.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Update anomalib/config/config.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* make root and abnormal_dir optional

* Update anomalib/data/folder.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Synthetic anomaly for testing and validation (#634)

* move sample generation to datamodule instead of dataset

* move sample generation from init to setup

* remove inference stage and add base classes

* replace dataset classes with AnomalibDataset

* move setup to base class, create samples as class method

* update docstrings

* refactor btech to new format

* allow training with no anomalous data

* remove MVTec name from comment

* raise NotImplementedError in base class

* allow both png and bmp images for btech

* use label_index to check if dataset contains anomalous images

* refactor getitem in dataset class

* use iloc for indexing

* move dataloader getters to base class

* refactor to add validate stage in setup

* implement alternative datamodules solution

* small improvements

* improve design

* remove unused constructor arguments

* adapt btech to new design

* add prepare_data method for mvtec

* implement more generic random splitting function

* update docstrings for folder module

* ensure type consistency when performing operations on dataset

* change imports

* change variable names

* replace pass with NotImplementedError

* allow training on folder without test images

* use relative path for normal_test_dir

* fix dataset tests

* update validation set parameter in configs

* change default argument

* use setter for samples

* hint options for val_split_mode

* update assert message and docstring

* revert name change dataset vs datamodule

* typing and docstrings

* remove samples argument from dataset constructor

* val/test -> eval

* remove Split.Full from enum

* sort samples when setting

* update warn message

* formatting

* use setter when creating samples in dataset classes

* add tests for new dataset class

* add test case for label aware random split

* update parameter name in inferencers

* move _setup implementation to base class

* address codacy issues

* fix pylint issues

* codacy

* update example dataset config in docs

* fix test

* move base classes to separate files (avoid circular import)

* add synthetic dataset class

* move augmenter to data directory

* add base classes

* update docstring

* use synthetic dataset in base datamodule

* fix imports

* clean up synthetic anomaly dataset implementation

* fix mistake in augmenter

* change default split ratio

* remove accidentally added file

* validation_split_mode -> val_split_mode

* update docs

* Update anomalib/data/base/dataset.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* get length from self.samples

* assert unique indices

* check is_setup for individual datasets

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove assert in __getitem_\

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update anomalib/data/btech.py

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* clearer assert message

* clarify list inversion in comment

* comments and typing

* validate contents of samples dataframe before setting

* add file paths check

* add seed to random_split function

* fix expected columns

* fix typo

* add seed parameter to datamodules

* set global seed in test entrypoint

* add NONE option to valsplitmode

* clarify setup behaviour in docstring

* add logging message

* use val_split_ratio for synthetic validation set

* pathlib

* make synthetic anomaly available for test set

* update configs

* add tests

* simplify test set splitting logic

* update docstring

* add missing licence

* split_normal_and_anomalous -> split_by_label

* VideoAnomalib -> AnomalibVideo

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Bugfixes for Datamodules feature branch (#800)

* properly handle NoneType mask_dir and add test case

* fix wrong deprecation handling

* Deprecate PreProcessor (#795)

* deprecate PreProcessor

* update configs

* update deprecation messages

* update video dataset

* update inference dataset

* move transforms to data module

* update and extend transform tests

* fix cyclic import

* add validity checks for image size and center crop

* pass image size as tuple

* update path to get_transforms

* update error message

* fix center crop tuple conversion

* update inferencers

* remove draem transform config

* update changelog

* fix cyclic import

* add crop size vs image size check

* improve readability

* mypy

* use enum to configure input normalization

* update lightning inference

* update inference dataset

* expose more parameters and fix wrong return format

* fix tdd tests

* update config

* [Datamodules] Fix bug in bbox score to image score conversion (#803)

handle empty box predictions

* update config

* apply pixel threshold to bbox detections

* remove confidence threshold parameter from rkde

* hardcode steepness and offset

* rename variable

* remove unused parameters from config

* Improve handling of `test_split_mode='none'` and `val_split_mode='none'` (#801)

* enable none as split mode

* use get to retrieve config keys

* update deprecation message and config key

* update config with new keys

* remove unused parameter

* set device in rpn stage

* move prediction format conversion to lightning model

* clean up torch model

* move region- and feature-extractor to separate files

* allow visualizing normal boxes

* refactor

* WIP: simplify region extractor

* simplify region extractor

* cleanup and docstrings

* typing

* expose max detections per image parameter

* explain configurable parameters

* fix wrong config value

* remove unnecessary squeeze

* box_likelihood -> rcnn_box_threshold

* update comments

* remove unnecessary typing

* separate density estimation stage from torch model

* improve readability

* change default transform settings

* fix to float transform

* simplify feature extractor

* normalize box scores

* further simplify region extractor

* update comment

* improve prn configurability

* remove unnecessary check

* use enum for roi stage options

* use enum for feature scaling method

* re-order parameters

* clean up model dir

* fix bbox logic in base anomaly module

* update key in output dict

* boxes_scores -> box_scores

* remove notebook

* add comments and todo

* Detection improvements (#820)

* apply pixel threshold to bbox detections

* allow visualizing normal boxes

* normalize box scores

* fix bbox logic in base anomaly module

* boxes_scores -> box_scores

* fix inferencers

* add readme

* update changelog

* update changelog

* update csflow config to new format

* initialize max_length as empty tensor

* include RKDE in model tests

* remove unused imports

* line length

* move kde classifier to shared location

* fix import

* re-use RKDE classifier in DFKDE

* remove old imports

* docstrings

* fix codacy issues

* load feature extractor weights from url

* suppress bandit warnings

* use torch rng in augmenter

* typing

* add fit method to torch model

* fix typo

* use enum when checking stage

* use tuple instead of list

* add missing params to dosctring

* add missing licence information

* COLS -> COLUMNS

* typing and variable naming

* remove duplicate parameter in docstring

* im_dir -> image_dir

* typing and docstring

* typing

* ValSplitMode -> ValidationSplitMode

* add missing licence

* rename variable

* remove empty comment

* remove unused class attribute

* [Detection] Compute box score when generating boxes from masks (#828)

* infer box scores from anomaly maps

* discard single pixel boxes

* revert discard single pixel boxes

* add test case for bbox scores

* update torch inferencer

* minor refactor

* revert val_split_mode -> validation_split_mode

Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet <samet.akcay@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants