[staging] gnugrep: 3.7 -> 3.11 #228714

dasJ · 2023-04-28T10:23:12Z

Also replace myself as a maintainer

Description of changes

Things done

dasJ · 2023-04-28T10:29:57Z

Draft for now because this breaks PCRE

dasJ · 2023-04-28T10:48:26Z

Should work now, @trofi can you verify that I didn't break the bootstrap tools?

trofi · 2023-04-28T11:26:44Z

At least on linux (and I think on darwin as well) we will need to add pcre2 to allowed references (or substitute from pcre?). Otherwise stdenv fails to build as:

$ nix build --no-link -f. stdenv
error: output '/nix/store/jy1wr873b4ib5gplrg3118z8gjs2bpa1-stdenv-linux' is not allowed to refer to the following paths:
         /nix/store/grd3jawsc8xchi0zg92k91m1n5piasbj-pcre2-10.42

dasJ · 2023-04-28T12:34:46Z

Do I need them in persistent as well?

Edit: I just added it as well because it made sense. Also switched from pcre to gnugrep.pcre2 to be more consistent

trofi · 2023-04-28T13:00:51Z

pkgs/stdenv/linux/bootstrap-tools/scripts/unpack-bootstrap-tools.sh

-for i in $out/lib/librt-*.so $out/lib/libpcre*; do
+for i in $out/lib/librt-*.so $out/lib/libpcre2*; do


This script runs against current bootstrapTools binaries that nixpkgs uses. It will have to have two worlds simultaneously:

grep with pcre

grep with pcre2

As it it fails to unpack current bootstrapTools:

$ nix build --no-link github:NixOS/nixpkgs/pull/228714/merge#stdenv -L ... bootstrap-tools> patching /nix/store/l4xssxrlz7mdpqr4blfh774q6v56849k-bootstrap-tools/lib/libpcre2* bootstrap-tools> stat: No such file or directory

trofi · 2023-04-28T13:01:49Z

pkgs/stdenv/linux/bootstrap-tools-musl/scripts/unpack-bootstrap-tools.sh

-for i in $out/lib/libpcre* $out/lib/libc.so; do
+for i in $out/lib/libpcre2* $out/lib/libc.so; do


Similar to glibc case it has to handle our current bootstrapTools as well.

dasJ · 2023-04-28T15:10:46Z

I have successfully built stdenv from this branch now for x86_64-linux

trofi

LGTM. We can fix the

./pkgs/stdenv/linux/bootstrap-tools-musl/scripts/unpack-bootstrap-tools.sh
./pkgs/stdenv/linux/bootstrap-tools/scripts/unpack-bootstrap-tools.sh

a while after (hopefully before anyone tries to use newer bootstrapTools.

Artturin · 2023-04-28T20:57:26Z

pkgs/tools/text/gnugrep/default.nix


-  # Perl is needed for testing
-  nativeBuildInputs = [ perl ] ++ lib.optional stdenv.hostPlatform.isLoongArch64 autoreconfHook;
+  checkInputs = [ perl ];


Suggested change

checkInputs = [ perl ];

nativeCheckInputs = [ perl ];

vcunat · 2023-05-29T10:58:25Z

musl is in trouble, but fortunately only on aarch64:
https://hydra.nixos.org/build/221679820

risicle · 2023-05-29T16:42:27Z

This appears to break pkgsStatic.gnugrep for me on x86_64 linux, with 4 tests failing: test-nl_langinfo-mt, test-setlocale_null-mt-one, test-setlocale_null-mt-all, test-thread_create.

wegank · 2023-05-30T02:57:14Z

So, add !stdenv.hostPlatform.isMusl to doCheck?

vcunat · 2023-05-30T05:34:39Z

Yes, I'd use that as the simplest workaround, in case nothing better is submitted. It's surely better than to have no musl-based stdenv. For example, we could conditionally drop just the tests failing now (a tiny minority).

dasJ · 2023-05-30T06:06:44Z

I'm currently investigating

vcunat · 2023-06-02T20:30:11Z

Hmm, on x86_64-darwin it always gets stuck during tests (four times at least now). That's on Hydra: https://hydra.nixos.org/build/222612843#tabs-buildsteps
Locally it did pass for me on the first try – so maybe something about rosetta. I don't know, but I expect we'd conditionally drop tests again?

dasJ · 2023-06-02T20:31:24Z

so doCheck = !stdenv.hostPlatform.isDarwin?

wegank · 2023-06-02T20:32:46Z

IIRC, @reckenrode also mentioned the x86_64-darwin issue, albeit for sandbox builds.

reckenrode · 2023-06-02T20:51:24Z

I encountered it trying to work on the Darwin stdenv bump on x86_64-darwin. It always hangs on my Apple Silicon MBP, but it builds on our Intel iMac. The MBP is running 13.4 while the iMac is still on 13.3.1(a). I don’t know if OS version makes a difference, or if it’s a Rosetta 2 issue the test is triggering.

vcunat · 2023-06-03T07:29:04Z

Apparently it succeeded in the end

nix build /nix/store/cprxh7g44qgvygmc2hv3l6xg3qkyl6sx-gnugrep-3.11 --dry-run

so I guess I'll let it be for now and reconsider doCheck = !stdenv.hostPlatform.isDarwin once it gets stuck again? (which I expect to happen in some future staging-next)

reckenrode · 2023-06-04T13:37:50Z

How long did it take to run? I left mine going for hours, and it never finished.

vcunat · 2023-06-04T13:52:05Z

I don't know how to find that. The log is easy to get, as the URL doesn't change on retries, but I can't see how to find which job built it (and see the time and machine).

Artturin · 2023-06-05T01:11:35Z

I don't know how to find that. The log is easy to get, as the URL doesn't change on retries, but I can't see how to find which job built it (and see the time and machine).

checkPhase completed in 2 minutes 32 seconds

reckenrode · 2023-06-05T13:22:25Z

Are any of the builders actually x86_64-darwin hardware? It’s still stuck for me (Rosetta 2 on an M1 Max).

[1/158/203 built, 0.0 MiB DL] building gnugrep-3.11 (checkPhase): FAIL: test-c-stack.sh

It’s been stuck at that for hours. I even tried to rework my stdenv stuff to build gnugrep later (with clang 16), but it didn’t make a difference.

vcunat · 2023-06-05T13:33:11Z

~~I believe all are aarch64-darwin currently.~~ @cole-h should have a better grasp on this.

cole-h · 2023-06-05T13:36:56Z

ofborg has 3 x86 and 2 aarch64 macs, with the aarch64 macs enabling Rosetta and taking x86 builds whenever The Scheduler desires.

dasJ · 2023-06-05T13:38:37Z

Can anyone with Rosetta test this patch?

diff --git a/pkgs/tools/text/gnugrep/default.nix b/pkgs/tools/text/gnugrep/default.nix
index 75c6f695073..2494ed1e49b 100644
--- a/pkgs/tools/text/gnugrep/default.nix
+++ b/pkgs/tools/text/gnugrep/default.nix
@@ -17,7 +17,8 @@ stdenv.mkDerivation {
   };

   # Some gnulib tests fail on Musl: https://github.com/NixOS/nixpkgs/pull/228714
-  postPatch = if stdenv.hostPlatform.isMusl then ''
+  # They also seem to fail on Rosetta: https://github.com/NixOS/nixpkgs/pull/228714#issuecomment-1574300624
+  postPatch = if stdenv.hostPlatform.isMusl || stdenv.hostPlatform.system == "x86_64-darwin" then ''
     sed -i 's:gnulib-tests::g' Makefile.in
   '' else null;

vcunat · 2023-06-05T13:52:55Z

@cole-h: this was too big for OfBorg, I meant Hydra where it did succeed after several attempts (but I don't know on which machine).

cole-h · 2023-06-05T13:55:50Z

According to https://hydra.nixos.org/machines, we only have 1 "real" x86_64-darwin machine (destined-gannet). (There's stirred-oryx too, but it's been dead basically since forever.)

cole-h · 2023-06-05T14:52:42Z

Can anyone with Rosetta test this patch?

diff --git a/pkgs/tools/text/gnugrep/default.nix b/pkgs/tools/text/gnugrep/default.nix
index 75c6f695073..2494ed1e49b 100644
--- a/pkgs/tools/text/gnugrep/default.nix
+++ b/pkgs/tools/text/gnugrep/default.nix
@@ -17,7 +17,8 @@ stdenv.mkDerivation {
   };

   # Some gnulib tests fail on Musl: https://github.com/NixOS/nixpkgs/pull/228714
-  postPatch = if stdenv.hostPlatform.isMusl then ''
+  # They also seem to fail on Rosetta: https://github.com/NixOS/nixpkgs/pull/228714#issuecomment-1574300624
+  postPatch = if stdenv.hostPlatform.isMusl || stdenv.hostPlatform.system == "x86_64-darwin" then ''
     sed -i 's:gnulib-tests::g' Makefile.in
   '' else null;

Worked for me:

vin@catacendre nixpkgs % file result/bin/grep
result/bin/grep: Mach-O 64-bit executable x86_64
vin@catacendre nixpkgs % realpath ./result
/nix/store/vpcabk9qbxssqj3qcy9c3nag465mi3zw-gnugrep-3.11

dasJ requested review from trofi and m00wl April 28, 2023 10:23

dasJ linked an issue Apr 28, 2023 that may be closed by this pull request

Update request: gnugrep 3.7 → 3.10 #228535

Closed

1 task

dasJ marked this pull request as draft April 28, 2023 10:30

github-actions bot added the 6.topic: stdenv Standard environment label Apr 28, 2023

dasJ marked this pull request as ready for review April 28, 2023 10:48

ofborg bot added the 2.status: merge conflict label Apr 28, 2023

dasJ force-pushed the upd/gnugrep branch 2 times, most recently from b37eece to e6b87ff Compare April 28, 2023 12:44

trofi reviewed Apr 28, 2023

View reviewed changes

ofborg bot removed the 2.status: merge conflict label Apr 28, 2023

dasJ force-pushed the upd/gnugrep branch from e6b87ff to 5a848e7 Compare April 28, 2023 15:10

trofi approved these changes Apr 28, 2023

View reviewed changes

ofborg bot added 10.rebuild-darwin-stdenv 10.rebuild-linux-stdenv 11.by: package-maintainer 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Apr 28, 2023

Artturin reviewed Apr 28, 2023

View reviewed changes

figsoda added the 12.approvals: 1 label Apr 30, 2023

dasJ force-pushed the upd/gnugrep branch from d4a1fce to 10396d0 Compare May 15, 2023 08:22

ofborg bot requested a review from m00wl May 15, 2023 10:11

pogobanane approved these changes May 16, 2023

View reviewed changes

m00wl approved these changes May 17, 2023

View reviewed changes

dasJ requested a review from mweinelt May 17, 2023 18:58

wegank merged commit 91b69f8 into NixOS:staging May 24, 2023

dasJ deleted the upd/gnugrep branch May 30, 2023 06:06

dasJ mentioned this pull request May 30, 2023

[staging-next] gnugrep: Fix build on Musl #234943

Merged

12 tasks

dasJ mentioned this pull request Jun 6, 2023

gnugrep: enable parallel build and tests #236229

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[staging] gnugrep: 3.7 -> 3.11 #228714

[staging] gnugrep: 3.7 -> 3.11 #228714

dasJ commented Apr 28, 2023

dasJ commented Apr 28, 2023

dasJ commented Apr 28, 2023

trofi commented Apr 28, 2023

dasJ commented Apr 28, 2023 •

edited

Loading

trofi Apr 28, 2023

trofi Apr 28, 2023

dasJ commented Apr 28, 2023

trofi left a comment

Artturin Apr 28, 2023

vcunat commented May 29, 2023

risicle commented May 29, 2023

wegank commented May 30, 2023

vcunat commented May 30, 2023

dasJ commented May 30, 2023

vcunat commented Jun 2, 2023

dasJ commented Jun 2, 2023

wegank commented Jun 2, 2023 •

edited

Loading

reckenrode commented Jun 2, 2023 •

edited

Loading

vcunat commented Jun 3, 2023

reckenrode commented Jun 4, 2023

vcunat commented Jun 4, 2023

Artturin commented Jun 5, 2023

reckenrode commented Jun 5, 2023 •

edited

Loading

vcunat commented Jun 5, 2023 •

edited

Loading

cole-h commented Jun 5, 2023

dasJ commented Jun 5, 2023

vcunat commented Jun 5, 2023

cole-h commented Jun 5, 2023

cole-h commented Jun 5, 2023

		for i in $out/lib/librt-.so $out/lib/libpcre; do
		for i in $out/lib/librt-.so $out/lib/libpcre2; do

		for i in $out/lib/libpcre* $out/lib/libc.so; do
		for i in $out/lib/libpcre2* $out/lib/libc.so; do

[staging] gnugrep: 3.7 -> 3.11 #228714

[staging] gnugrep: 3.7 -> 3.11 #228714

Conversation

dasJ commented Apr 28, 2023

Description of changes

Things done

dasJ commented Apr 28, 2023

dasJ commented Apr 28, 2023

trofi commented Apr 28, 2023

dasJ commented Apr 28, 2023 • edited Loading

trofi Apr 28, 2023

Choose a reason for hiding this comment

trofi Apr 28, 2023

Choose a reason for hiding this comment

dasJ commented Apr 28, 2023

trofi left a comment

Choose a reason for hiding this comment

Artturin Apr 28, 2023

Choose a reason for hiding this comment

vcunat commented May 29, 2023

risicle commented May 29, 2023

wegank commented May 30, 2023

vcunat commented May 30, 2023

dasJ commented May 30, 2023

vcunat commented Jun 2, 2023

dasJ commented Jun 2, 2023

wegank commented Jun 2, 2023 • edited Loading

reckenrode commented Jun 2, 2023 • edited Loading

vcunat commented Jun 3, 2023

reckenrode commented Jun 4, 2023

vcunat commented Jun 4, 2023

Artturin commented Jun 5, 2023

reckenrode commented Jun 5, 2023 • edited Loading

vcunat commented Jun 5, 2023 • edited Loading

cole-h commented Jun 5, 2023

dasJ commented Jun 5, 2023

vcunat commented Jun 5, 2023

cole-h commented Jun 5, 2023

cole-h commented Jun 5, 2023

dasJ commented Apr 28, 2023 •

edited

Loading

wegank commented Jun 2, 2023 •

edited

Loading

reckenrode commented Jun 2, 2023 •

edited

Loading

reckenrode commented Jun 5, 2023 •

edited

Loading

vcunat commented Jun 5, 2023 •

edited

Loading