ZERO Hydra Failures 20.03

[worldofpeace]

ZERO Hydra Failures 20.03

Jobsets:

• https://hydra.nixos.org/jobset/nixos/release-20.03
• https://hydra.nixos.org/jobset/nixos/release-20.03-aarch64
• https://hydra.nixos.org/jobset/nixpkgs/nixpkgs-20.03-darwin
• https://hydra.nixos.org/jobset/nixpkgs/staging-20.03

Mission

Every time we branch off a release we stabilize the release branch.
Our goal here is to get as little as possible jobs failing on the release-20.03 jobset.
I'd like to heighten, while it's great to focus on zero as our goal, it's essentially to
have all deliverables that worked in the previous release work here also.

How many failing jobs are there?

At the opening of this issue we have the main jobset at 1204 failing jobs, x86_64-darwin at 1384, and aarch64-linux at 7482.

The first evaluation of 19.09 had 1654 failing jobs.
So we're actually starting off with slightly less failing jobs compared to the last release.

How to help

1. Select an evaluation of the release-20.03 jobset by #id
   

2. Find a failed job ❌️
   

3. Work out why it's failing and fix it

4. Pull Request the fix

Generally the job fails on master also, you can verify that on Hydra - example URL: https://hydra.nixos.org/job/nixpkgs/trunk/bash.x86_64-linux.
That means most PR's should be targeted on master or staging for mass rebuilding changes.

Always reference this issue in the body of your PR:

ZHF: #80379

5. Backport it to release-20.03 or staging-20.03.

Details on that are in CONTRIBUTING.
Please ping @NixOS/nixos-release-managers on the PR.

Packages that don't get fixed

The remaining packages will be marked as broken before the release (on the failing platforms).
You can do this like:

meta = {
  # ref to issue/explanation
  # `true` is for everything
  broken = stdenv.isDarwin 
};

These are the utility flags used to test the type of platform.

Closing

This is a great way to help NixOS, and it was some of my earliest contributions.
Let's go ✌️

✨️ worldofpeace

cc @NixOS/nixpkgs-committers @NixOS/nixpkgs-maintainers

[jonringer]

IT BEGINS!!!!

[worldofpeace]

This was actually delayed by #79907, but we just got this sorted.
Many thanks to everyone who fixed it ✨

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-20-03-beta/5935/1

[rnhmjoj]

It looks like all nixosTests on i686 are failing because of a test in python aiohttp: https://hydra.nixos.org/build/112800138/nixlog/390/tail.

The problem is

>> int(4575744000.0).bit_lenght()
33

which can't be represented on i686 time_t.

Should we silence the test on 32bit platforms?

[worldofpeace]

@rnhmjoj They already disable some tests on that platform https://github.com/aio-libs/aiohttp/pull/4021/files#diff-484462fced51d1a06b1d93b4a44dd535R34, sounds good.

[samueldr]

Even better Somewhat preferred, patch it to disable only the failing tests.

[asymmetric]

Can I just say that this is an awesome ZHF description, with lots of valuable info in it.

Great job @worldofpeace! 👏

[worldofpeace]

Can I just say that this is an awesome ZHF description, with lots of valuable info in it.

Great job @worldofpeace! clap

Thank you 😊 I tried to include all the useful information from my predecessors and my own stylized information.

[tbenst]

Awesome! Anyone know if there's a way to show failing builds by maintainer? Also, if I want to test my config, should I use https://github.com/NixOS/nixpkgs/tree/20.03-beta?

Thanks for the great work!

[infinisil]

@tbenst Here's a simple expression you can put in your nixpkgs checkout (say mypkgs.nix), which you can then nix-build mypkgs.nix to build all top-level packages with you as a maintainer (adjust the maintainer there to yourself):

{ pkgs ? import ./. {}
, maintainer ? "infinisil"
}: with pkgs.lib;
filterAttrs (name: value:
 (builtins.tryEval value).success &&
 elem maintainers.${maintainer} (value.meta.maintainers or [])
) pkgs

Or nix-build mypkgs.nix -A <TAB> to see and build individual ones

[worldofpeace]

Also, if I want to test my config, should I use https://github.com/NixOS/nixpkgs/tree/20.03-beta?

No, that's the tag, you want to use the release-20.03 branch. But I would wait a few days because I'm not sure everything is built.

[bhipple]

I have this extremely primitive script:

#!/usr/bin/env bash
# Simple script to check nix attrs for build status on Hydra
while [ -n "$1" ]; do
    url="https://hydra.nixos.org/job/nixos/release-20.03/nixpkgs.$1.x86_64-linux"
    printf "Last status of $1 = "
    res=$(curl -s "$url"  | rg -o 'title="[A-Za-z]+"' | sed 's|title=||' | sed 's|"||g' | head -1)
    echo "$res"

    if [ "$res" = "Failed" ]; then
        xdg-open "$url" >/dev/null 2&>1
    fi

    shift
done

which can then be called like this:

$ hydra python37Packages.numpy dnnl s4cmd
Last status of python37Packages.numpy = Succeeded
Last status of dnnl = Succeeded
Last status of s4cmd = Succeeded

It'll open the Hydra page in your web browser for details if there's a failure.

I've been chaining it together with ripgrep, but I think chaining together with a nix expression similar to @infinisil's could result in something more robust.

I've also noticed Hydra is responding very slowly to page loads, so I'm wondering if there's a faster/better way to hit it; I'm not calling in high volume, but please wait for someone who knows more to chime in before you hammer the server with a huge load. It's possible Hydra's REST API has something better here than curl'ing and grepping the status page.
https://nixos.org/hydra/manual/#chap-api

[orivej]

The real problem is mitmproxy in 20.03 is one major release behind.

Do you mean cryptography rather than mitmproxy?

[rnhmjoj]

Do you mean cryptography rather than mitmproxy?

No, I really meant mitmproxy, which is at 4.0.4, latest is 5.1.1. cryptography is only slightly outdate: 2.8, latest 2.9.

I'm trying to build 5.1.1 right now, if I manage to get it working without changing too many dependencies I could port it to 20.03, otherwise I'll just add yet another patch.

[orivej]

Aha, OK, mitmproxy is not broken on master since master has not yet updated OpenSSL from 1.1.1d to 1.1.1f.

[rnhmjoj]

I've fixed mitmproxy in 20.03 with 2e08e8c. The update to 5.1.1 requires a bump of cryptography, which is probably better not to backport, see #85458.

[sbourdeauducq]

@disassembler Why was python3Packages.svgwrite marked as broken in c6be4c1 ? It seems to build fine, and Python can import the resulting package.

[worldofpeace]

If anyone has a package that is wrongly marked as broken or can be fixed, please open a PR.
There's no way us as RMs are going to be able to follow that after the fact, it's just not possible.

[worldofpeace]

20.03 is out https://discourse.nixos.org/t/nixos-20-03-release/6785.

Thanks to everyone who helped with 20.03. We accept backports into the stable release during its lifetime, bugfixes and security updates. Minor updates also. If certain exceptions are needed we can work with it. 👋

[jonringer]

I just want to thank @worldofpeace @disassembler and many others for the work they put forward to make this release happen. It makes me happy to see how much the nix community desires to see NixOS be successful :).

May NixOS be the way of the future :)