macOS 11 (Big Sur beta 3) further locks down /

[ryanbooker]

The existing work around to install into /nix via a new APFS volume, and mounting as rw in vifs, no longer appears to work. The volume simply won't mount.

[shepting]

Thanks for testing this.

[tnytown]

This can be temporarily worked around by disabling SIP "filesystem protections".

Reboot to Recovery OS and execute the following:

csrutil enable --without fs

[sycured]

In my case, I needed to do: sudo rm /etc/synthetic.conf && echo "nix\tVolumes/Nix" | sudo tee -a /etc/synthetic.conf
and removing the line about nix inside /etc/fstab

[tnytown]

In my case, I needed to do: sudo rm /etc/synthetic.conf && echo "nix\tVolumes/Nix" | sudo tee -a /etc/synthetic.conf
and removing the line about nix inside /etc/fstab

I originally considered this approach. This turns /nix into a symlink to /Volumes/Nix, which works for accessing things already in your nix store but breaks on adding packages (try nix-shell -p hello --run hello or something similar.) It's mentioned in the docs as an unsupported configuration. Messing with SIP definitely sucks but there are downsides IRT reproducibility when the realpath of the store isn't rooted at /nix.

Alternatively, you can stick with the symlink and set the environment variable NIX_IGNORE_SYMLINK_STORE, but keep the consequences in mind :)

[sycured]

No problem about mounting on Beta 1 and 2, so wait and see for the final release because Apple can revert this change.

Otherwise, next step will be moving back to homebrew because I refuse to degrade SIP if NIX_IGNORE_SYMLINK_STORE doesn't work well but also because nix doesn't provide a way to install it at custom location.

One possible way is what I did, having a macOS Catalina in vm on another computer and syncing the /nix across the network but it's not very elegant.

[Mic92]

@arianvp said that we might get a firmlink for /nix, which would solve this issue.

[Mic92]

Otherwise it might be time for nix to switch to a different store path on macOS?

[arianvp]

My exact words were that somebody at Apple poked me on the firmlink issue on twitter. I don't know whether they will actually be able to help; but I did sent them a link to this issue.

Lets hope if they can help us =)

[arianvp]

Tweet in question for the record: https://twitter.com/ProgrammerDude/status/1285858829155151873

[arianvp]

Could someone file an issue at https://feedbackassistant.apple.com with their Apple Developer account on the firmlink issue, and then share with the kind people in this twitter thread? https://twitter.com/rballard/status/1287786882688880651?s=20

[dhess]

Could someone file an issue at https://feedbackassistant.apple.com with their Apple Developer account on the firmlink issue, and then share with the kind people in this twitter thread? https://twitter.com/rballard/status/1287786882688880651?s=20

I have done so, as FB8179445. I doubt you'll be able to see the URL, but it's here:

https://feedbackassistant.apple.com/feedback/8179445

However, the Apple engineer who's been replying has requested that I file a separate Feedback for the Big Sur beta 3 behavior as described in this GitHub issue. I'll try to do that if nobody beats me to it, but I'm not currently running Big Sur, so it might take me a few days.

[lilyball]

The fact that mounting a volume using synthetic.conf in Big Sur isn't working seems like a bug, and Apple is aware of it. It remains to be seen whether it will be fixed, but I'm hopeful, as this was one of the two primary use-cases for synthetic.conf and it would be pretty bad for Apple to introduce this mechanism in one release only to break it in the next.

[ryanbooker]

In Big Sur everything on the system volume is protected, paths that are user editable like home, etc, tmp etc, are links out to another volume. Maybe Apple will let this through the new wall, or provide a mechanism to create a link that works the same way as home etc, or maybe it is just a bug in Beta 3, and synthetic.conf mounted volumes should be rw. But it does seem at face value to make sense given the tighter restriction they've added to Big Sur.

[lilyball]

@ryanbooker What you describe sounds like Catalina. It's unclear to me what's changed in this regard for Big Sur, besides the volume mounting bug.

[ryanbooker]

https://eclecticlight.co/2020/06/25/big-surs-signed-system-volume-added-security-protection/

[abathur]

I read through the eclectic light post a few weeks ago, and again now.

I won't try to proclaim, having not tested, but both times it didn't seem to me like the signing change it focuses on should have any impact on the Nix install approach (because it isn't modifying the system volume).

[lilyball]

I just read that post, and it sounds like the changes here aren't actually related to being unable to mount a volume rw on a path listed in synthetic.conf, as none of this touches the system volume. It definitely just seems like a bug.

[ryanbooker]

I hope you're right and it's just a bug. I was assuming it's because the mount point is on the system volume and nothing on the system volume is allowed to be rw (at least that was my understanding). Whether that's just a mistake in B3 or an intended change is an open question, I suppose.

[arianvp]

@ryanbooker do you have an Apple Developer account, and would you want to file an FB for this specific issue and share it here for documentation-sake?

In the FB you can say it's a duplicate of radar 66170165 which is the internal number under which this issue is being tracked at apple. (See https://twitter.com/rballard/status/1287809402385207296)

[ryanbooker]

Yep. I had already filed one, but I've marked it as a dupe. FB8112166

[tnytown]

FWIW, the (probable?) regression is still present on Beta 4.

$ csrutil status
System Integrity Protection status: enabled.
$ cat /etc/synthetic.conf /etc/fstab 
nix
run	private/var/run
#
# Warning - this file should only be modified with vifs(8)
#
# Failure to do so is unsupported and may be destructive.
#
LABEL=Nix /nix apfs rw,nobrowse
$ diskutil ap unlockVolume disk1s5
Unlocking any cryptographic user on APFS Volume disk1s5
Error unlocking APFS Volume: Couldn't mount disk (-69842)
$ diskutil info disk1s5
...
   Volume Name:               Nix
   Mounted:                   No
...
   FileVault:                 Yes
   Sealed:                    No
   Locked:                    No

[abathur]

If anyone listening/reading is in the DTK program, I'm curious if there are other reports/discussion about this in the DTK forum that supposedly exists?

[arianvp]

Note that I and @grahamc have a contact person at Apple's developer relations that can help us approve DTK access requests for companies with nix contributors. If you're interested please reach out to us. Another option is for NixOS organization to request access to MacStadium cloud machines (free of charge)

I think @grahamc also requested one for the NixOS Foundation but I am not sure

The original email that I got:

Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:

https://developer.apple.com/programs/universal/

If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.

Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email opensource@macstadium.com.

The process for qualified OSS developers will be:

Developer signs up for a general MacStadium account so the MSA can be accepted.
MacStadium will manually add the DTK asset to their account at no cost.
Developer will have access to screen sharing on a DTK and can log in right away

.

[ryanbooker]

The drive mounts now.

[kloenk]

Heise (German Tech newspaper) just reported that apple will only allow signed code. Can that work with binary caches?

(I'm not yet on apple and I think of waiting to go to apple untill the arm is available)

Source: https://www.heise.de/news/Apple-ARM-Macs-fuehren-nur-noch-signierten-Code-aus-4875048.html

[Mic92]

Heise (German Tech newspaper) just reported that apple will only allow signed code. Can that work with binary caches?

(I'm not yet on apple and I think of waiting to go to apple untill the arm is available)

Source: heise.de/news/Apple-ARM-Macs-fuehren-nur-noch-signierten-Code-aus-4875048.html

We likely cannot add signing keys to the nix-build in a reproducible manner. Also the clang used in nixpkgs won't have those keys as well. Everyone on that platform would need to compile packages themselves. What will better work is to run NixOS in their VM similiar to WSL - it may even be faster for certain workloads - macOS system calls are not known to be very performant.

[dhess]

The drive mounts now.

Do you mean that in the latest beta, this is no longer an issue?

[arianvp]

Shall we open a new issue for ARM compatibility? It seems to be a bit of a separate discussion

[rbvermaa]

Note that I and @grahamc have a contact person at Apple's developer relations that can help us approve DTK access requests for companies with nix contributors. If you're interested please reach out to us. Another option is for NixOS organization to request access to MacStadium cloud machines (free of charge)

I think @grahamc also requested one for the NixOS Foundation but I am not sure

The original email that I got:

Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:
https://developer.apple.com/programs/universal/
If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.
Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email opensource@macstadium.com.
The process for qualified OSS developers will be:
Developer signs up for a general MacStadium account so the MSA can be accepted.
MacStadium will manually add the DTK asset to their account at no cost.
Developer will have access to screen sharing on a DTK and can log in right away

.
@arianvp
Note that the NixOS foundation got ahold of a ARM mac mini now (DTK), and I am in the process of arranging some access for @edolstra initially. Let's discuss this further in a new issue. Did you create one already?

[arianvp]

@rbvermaa excellent news :) I have opened an issue NixOS/nixpkgs#95903

[ryanbooker]

The drive mounts now.

Do you mean that in the latest beta, this is no longer an issue?

Yes. The Nix drive will mount again, and nix will work as it did in Catalina. :)

[Mic92]

I think we can close this issue than.

[ryanbooker]

Still working in the latest beta.