Darwin builders are stuck

[r-burns]

The x86_64-darwin builders on hydra.nixos.org seem to be stuck in some way. Most builders I have checked have hung on the "Sending inputs" phase.

For example, threefish (https://hydra.nixos.org/build/154946284) was cancelled several days ago but is still stuck in "Sending inputs" phase. Mhost (https://hydra.nixos.org/build/155222382) has not been cancelled but has been stuck in the same phase for over two days.

Apologies if this is a known issue. I'm aware the darwin builders are typically slower but it seems especially bad right now.

[domenkozar]

Afaik only @grahamc and @edolstra have access to Hydra to check what state they're at.

[r-burns]

PTAL at https://hydra.nixos.org/status. Every single x86_64-darwin builder is stuck on a job which has been running for over 6 days. We have effectively zero build capacity for x86_64-darwin.

[domenkozar]

Note that this is also blocking macOS work for getting llvm bump and macOS SDK bump into 21.11, potentially also delaying the release.

I'm happy to take a look if I'm given access, if noone else has the time.

[r-burns]

The builders are unstuck 🎉

[tomberek]

• mac7-guest and mac3-guest are working. mac2-guest and mac5-guest are still stuck. (and potentially the other mac#-guest's).
• There are max-m1-# aarch64-darwin builders currently running
• New builders: 0469fb3d, 299a3510, dcd22189 are running. But I'm not sure what they are. (it looks like instance id's?)

[r-burns]

It looks like this is still happening, so I'll reopen until we can figure out the root cause.

[domenkozar]

I'm not sure anyone is looking into the root cause.

[grahamc]

It is some time later, so I'm going to close this. The macs are in Prometheus now, too: https://monitoring.nixos.org/grafana/d/hkRCcV0mk/instance-metrics?orgId=1&refresh=30s&var-role=mac