Enable L2 cache of all (MRU+MFU) metadata but MFU data only

[shodanshok]

Fixes #16343

l2arc_mfuonly was added to avoid wasting L2 ARC on read-once MRU data and metadata. However it can be useful to cache as much metadata as possible while, at the same time, restricting data cache to MFU buffers only.

This patch allow for such behavior by setting l2arc_mfuonly to 2 (or higher). The list of possible values is the following:
0: cache both MRU and MFU for both data and metadata;
1: cache only MFU for both data and metadata;
2: cache both MRU and MFU for metadata, but only MFU for data.

Motivation and Context

Provide as much metadata L2 cache as possible, while at the same time caching (hopefully) useful MFU data.

Description

See above.

How Has This Been Tested?

Minimal, on local virtual machine only.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

The logic of the values looks ugly to me, but I don't have better thoughts.

[tkittich]

@shodanshok Thank you very much for working on this feature. ^^

Maybe it could be possible to check a dataset property and mark a buffer as not eligible for the L2ARC in function l2arc_write_eligible(). It would be great if this feature could be enabled per dataset as a new setting, i.e. metaandmfu, for primarycache and secondarycache properties.

[shodanshok]

@shodanshok Thank you very much for working on this feature. ^^

Maybe it could be possible to check a dataset property and mark a buffer as not eligible for the L2ARC in function l2arc_write_eligible(). It would be great if this feature could be enabled per dataset as a new setting, i.e. metaandmfu, for primarycache and secondarycache properties.

I tried something similar during my tests, however it seems to me that it is not possible to that without altering the arc_buf_hdr structure.

More specifically, arc_buf_hdr can be used to distinguish between data and metadata, but not between MRU and MFU status, which are determined by the underlying buffer list. Adding such a field to arc_buf_hdr would mean to a) increase the size of a core structure and b) insert some kind of function to update the new field whenever the buffer switches from MRU to MFU.

So while I agree that would be great to control such thing via dataset property, the patch as-merged provide an immediate workaround without changing the header struct.