Add Augmentations for Robustness Experiments

[SakshayMahna]

The robustness experiments include:

• Translations of the Camera
• Rotations of the Camera

Currently, we use the albumentations library to introduce the different augmentations. The appropriate function for this use case would be the Affine function.

https://albumentations.ai/docs/api_reference/augmentations/geometric/transforms/

However, the camera rotation operation is a Homography Operation, which does not seem to be available in the library.

[sergiopaniego]

I have applied Affine using Albumentations to one of the images in the dataset, so we can discuss if it enough with this function or we need to implement something more complex.

Normal image

Normal image with affine (Rotation and translation)

Cropped image

Cropped image with affine (rotation and translation)

What do you think? Could you share image examples of applying the transformation on a model level so we can compare and decide?

The code that I have used is quite simple:

# Examples made with x=-0.1, x=0.1, y=-0.1, y=0.1
augmentation_option = Compose([Affine(translate_percent={'x': -0.1, 'y': 0}, always_apply=True)])

[SakshayMahna]

That's a nice trick. Here are the original model images:

Translation

Original

Cropped

Rotation

Original

Cropped

[SakshayMahna]

I think the Affine Transform does approximate the translation and rotation, but in some cases, when we crop the image, it would capture some black background as well. Any idea how to avoid that?

Conventional methods seem to use Reflection and other techniques, should we use that?

[sergiopaniego]

I'm sharing the results for the different types of available border_modes so we can decide if any fits out needs.

mode=cv2.BORDER_REPLICATE

[sergiopaniego]

mode=cv2.BORDER_REFLECT

[sergiopaniego]

mode=cv2.BORDER_WRAP

[sergiopaniego]

mode=cv2.BORDER_REFLECT_101

[sergiopaniego]

These are the available options.
IMO, mode=cv2.BORDER_REPLICATE or mode=cv2.BORDER_REFLECT are the best two options. What do you think @SakshayMahna ?

The images are not exactly the same as the ones you've shared. I'm suggesting this solution because of its simplicity of integration and implementation but we can still explore the raw opencv option you suggested.

[SakshayMahna]

Yes, these two seem to be the best options. Let's go with REPLICATE. I'll try training the model with the same. In the meantime, study the opencv option as well.

[SakshayMahna]

I also did some experiments with the homography operation. Here are the results:

Original Rotation

Homography Operation

Homography with Adjusted Background

The accuracy of these operations may be much higher. But, the problem with this approach is that they are extremely hardcoded. We have to input specific points (4 for each image) to transform the image. Due to this, we will only have specific rotation values, like -20, -10, +10, +20.