e2scrub finds corruption immediately after mounting

[brianjmurrell]

I am trying to migrate from lvcheck to using the officially supported e2scrub[_all] kit.

I am finding that e2scrub very often (much more than lvcheck even) finds corruption and wants me to do an offline e2fsck. Not only does it do this immediately after booting a system that includes filesystem checks (that were caused by e2scrub previously setting a filesystem to be checked on next boot), but it happens immediately after I run an e2fsck and then mount the filesystem, even without any activity on it. Observe:

# umount /opt
# e2fsck -y /dev/rootvol_tmp/almalinux8_opt 
e2fsck 1.45.6 (20-Mar-2020)
/dev/mapper/rootvol_tmp-almalinux8_opt: clean, 1698/178816 files, 482404/716800 blocks
# e2scrub /dev/rootvol_tmp/almalinux8_opt 
  Logical volume "almalinux8_opt.e2scrub" created.
e2fsck 1.45.6 (20-Mar-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/rootvol_tmp/almalinux8_opt.e2scrub: 1698/178816 files (86.9% non-contiguous), 482404/716800 blocks
/dev/rootvol_tmp/almalinux8_opt: Scrub succeeded.
tune2fs 1.45.6 (20-Mar-2020)
Setting current mount count to 0
Setting time filesystem last checked to Wed Jan  3 11:37:04 2024

  Logical volume "almalinux8_opt.e2scrub" successfully removed.
# mount /opt
# e2scrub /dev/rootvol_tmp/almalinux8_opt 
  Logical volume "almalinux8_opt.e2scrub" created.
e2fsck 1.45.6 (20-Mar-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/rootvol_tmp/almalinux8_opt.e2scrub: 1698/178816 files (86.9% non-contiguous), 482404/716800 blocks
/dev/rootvol_tmp/almalinux8_opt: Scrub FAILED due to corruption!  Unmount and run e2fsck -y.
tune2fs 1.45.6 (20-Mar-2020)
Setting filesystem error flag to force fsck.
  Logical volume "almalinux8_opt.e2scrub" successfully removed.

So as you can see, I unmount /opt, run an e2fsck -y on it to clean it and then before mounting run e2scrub and it finds the filesystem clean. Good so far.

I then mount it and then immediately run another e2scrub on it and that finds it dirty and wants me to unmount and run another e2fsck -y on it. But how can that be? Surely an e2scrub on a freshly cleaned and mounted filesystem (with no activity on it in between) should be clean, yes?