Predboxes of CsFlow, Draem and DSR always return 0.5 #2122
Replies: 1 comment 13 replies
-
|
What is your testing dataset accuracy after training? Can you show the Full Training and inferencing Code? |
Beta Was this translation helpful? Give feedback.
-
|
ahh ok it's working with other models, thats strange. what do the test result images look like? They should have been saved in a dir called |
Beta Was this translation helpful? Give feedback.
-
|
I cannot share these images, but the results on mvtec are identical. |
Beta Was this translation helpful? Give feedback.
-
|
Somehow the image upload doesnt work. Therefore here as pdf |
Beta Was this translation helpful? Give feedback.
-
|
This the code for the mvtec dataset: Import libraries# general
import torch
# anomalib
from anomalib import TaskType
from anomalib.data import Folder
from anomalib.engine import Engine
from anomalib.models import Csflow# setting device on GPU if available, else CPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
if device == 'cuda':
torch.set_float32_matmul_precision(precision= 'high')Configurationepochs = 1
num_workers = 15
batch_size = 64
model_architecture = "Csflow"
data = "/home/azureuser/cloudfiles/code/Users/weu4rt/MVTec/bottle"
image_size = [128,128]
# dataset usage
train_ratio = 1.0
val_ratio = 1.0
test_ratio = 1.0Import Data# Set up Training Datamodule
anomalies = ['test/broken_large', 'test/broken_small', 'test/contamination']
datamodule = Folder("mvtec",
"train/good",
data,
anomalies,
task=TaskType.CLASSIFICATION,
image_size=image_size,
num_workers=num_workers,
train_batch_size=batch_size,
eval_batch_size=batch_size,
seed = 42
)
datamodule.setup()Create model and setup training/testingmodel = Csflow()
engine = Engine(task=TaskType.CLASSIFICATION, max_epochs = epochs)Train/Test model# train
engine.fit(model=model, datamodule=datamodule)
# test
test_results = engine.test(
model=model,
datamodule=datamodule,
ckpt_path=engine.trainer.checkpoint_callback.best_model_path,
)
# inference
prediction = engine.predict(model = model, datamodule= datamodule)
print("Finished!")┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ image_AUROC │ 0.6607142686843872 │ │ image_F1Score │ 0.738095223903656 │ └───────────────────────────┴───────────────────────────┘ print('anomaly score of first image: ', prediction[0]['pred_scores'][0])
print('anomaly scores of the first 10 images are all excactly 0.5: ', prediction[0]['pred_scores'][0:10])
print('anomaly map of first image: ', prediction[0]['anomaly_maps'][0])
print('maximum anomaly score in anomaly map: ', prediction[0]['anomaly_maps'][0].max())
print('min anomaly score in anomaly map: ', prediction[0]['anomaly_maps'][0].min()) |
Beta Was this translation helpful? Give feedback.
-
|
I have the following theory for this: In the models you describe, anomaly maps and pred_scores are on a very different scale in their absolute values (e.g. for Csflow the pred_scores are in a range of 10-20 and anomaly maps in 4,000 to 300,000). For both anomaly maps and pred scores the values are normalized. However, with the current implementation of normalization, the normalization min and max values are only calculated for anomaly maps if they exist: anomalib/src/anomalib/utils/callbacks/min_max_normalization.py Lines 54 to 61 in 5cadb65 The problem is, the normalization metrics for anomaly maps and pred scores are not separately calculated, they just use the single same metric object! So the min and max values of the anomaly maps are used to normalize the pred scores, which results in 0.5000 values, as the anomaly map values are way larger than pred score values (for Csflow) anomalib/src/anomalib/callbacks/normalization/min_max_normalization.py Lines 97 to 109 in 41d8cf9 @samet-akcay can you have a look into this, because if its correct it means a general flaw in anomalib with all pred scores being calculated are at least slightly off |
Beta Was this translation helpful? Give feedback.
-
Hey all,
I am currently testing out different models on my dataset. I have noticed that for
CsFlow,DraemandDsrI get decent results, although the interpretability is difficult. The heatmaps which are created duringengine.predict()always have a continuous color, which would be the case for ideal normal images. Sadly, this is happening for ALL images. Additionally the pred_boxes always return 0.5, independent of predicting the image as normal or anomalous. Due to the good performance of the models I assume that the prediction cant be random.Has anyone faced similar issues? Or has an idea how to fix this?
Thanks in advance!
Edit: Adding an example (of 1 epoch training of Csflow).
Beta Was this translation helpful? Give feedback.
All reactions