FreeBSD ZFS node exporter no longer works on 14.x

[peterjeremy]

Host operating system: output of uname -a

FreeBSD gce1.rulingia.com 14.0-CURRENT FreeBSD 14.0-CURRENT #1238 current-c282710-gee8f5c2a31ec-dirty: Sun Apr 9 04:15:20 AEST 2023 root@gce1.rulingia.com:/usr/obj/usr/src/amd64.amd64/sys/GCE amd64

node_exporter version: output of node_exporter --version

node_exporter, version 1.5.0 (branch: release-1.5, revision: 3)
  build user:       peter
  build date:       
  go version:       go1.20.1
  platform:         freebsd/amd64

node_exporter command line flags

--web.listen-address=192.168.122.5:9100 --collector.textfile.directory=/var/tmp/node_exporter --collector.devstat --collector.ntp

node_exporter log output

Apr  9 15:27:28 gce1 node_exporter[1018]: ts=2023-04-09T05:27:28.977Z caller=collector.go:169 level=error msg="collector failed" name=zfs duration_seconds=0.0001047 err="couldn't get sysctl: no such file or directory"
Apr  9 15:27:43 gce1 node_exporter[1018]: ts=2023-04-09T05:27:43.982Z caller=collector.go:169 level=error msg="collector failed" name=zfs duration_seconds=0.00010304 err="couldn't get sysctl: no such file or directory"
Apr  9 15:27:58 gce1 node_exporter[1018]: ts=2023-04-09T05:27:58.976Z caller=collector.go:169 level=error msg="collector failed" name=zfs duration_seconds=0.000114849 err="couldn't get sysctl: no such file or directory"
Apr  9 15:28:13 gce1 node_exporter[1018]: ts=2023-04-09T05:28:13.975Z caller=collector.go:169 level=error msg="collector failed" name=zfs duration_seconds=0.00012275 err="couldn't get sysctl: no such file or directory"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Ran node_exporter

What did you expect to see?

No node_exporter ZFS errors.

What did you see instead?

The above error messages.

Having investigated, the problem relates to the sysctl kstat.zfs.misc.arcstats.p. This was removed in https://cgit.freebsd.org/src/commit?id=a8d83e2a24de6419dc58d2a7b8f38904985726cb, which was commited to FreeBSD main in https://cgit.FreeBSD.org/src/commit/?id=2a58b312b62f908ec92311d1bd8536dbaeb8e55b Based on my reading of the commit, there's no longer a single "ARC MRU target size", instead there are a set of ratios between data & metadata for MRU and MFU

Note that not including the failing sysctl name in the error message makes debugging the issue more difficult.

[dongjiang1989]

Need to install sysctl like: https://command-not-found.com/sysctl
@peterjeremy

[peterjeremy]

Need to install sysctl like: https://command-not-found.com/sysctl

That is not a helpful response because:

• sysctl is part of the FreeBSD base system so it's always installed.
• node_exporter doesn't depend on the sysctl command, instead it uses the sysctl system call.

As per my original report, the problem is that the sysctl kstat.zfs.misc.arcstats.p has been removed in recent OpenZFS and that removal has been imported into FreeBSD-14

[discordianfish]

At least overlap with #2068 - still unclear what we can do about this.

[dongjiang1989]

 #define	arc_no_grow	ARCSTAT(arcstat_no_grow) /* do not grow cache size */
-#define	arc_p		ARCSTAT(arcstat_p)	/* target size of MRU */
+#define	arc_meta	ARCSTAT(arcstat_meta)	/* target frac of metadata */
+#define	arc_pd		ARCSTAT(arcstat_pd)	/* target frac of data MRU */
+#define	arc_pm		ARCSTAT(arcstat_pm)	/* target frac of meta MRU */
 #define	arc_c		ARCSTAT(arcstat_c)	/* target size of cache */
 #define	arc_c_min	ARCSTAT(arcstat_c_min)	/* min target cache size */
 #define	arc_c_max	ARCSTAT(arcstat_c_max)	/* max target cache size */

review new code， delete kstat.zfs.misc.arcstats.p and add kstat.zfs.misc.arcstats.meta , kstat.zfs.misc.arcstats.pd, kstat.zfs.misc.arcstats.pm

I want fix this case.