Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0081] Show unmaintained packages #81

Closed
wants to merge 5 commits into from

Conversation

Lassulus
Copy link
Member

@ryantm
Copy link
Member

ryantm commented Dec 10, 2020

The "fancy website" basically already exists.

@Lassulus
Copy link
Member Author

The "fancy website" basically already exists.

this is indeed quite cool. But I would still prefer a integration in nix/nixpkgs/nixos.

On the repology page I can't see more than 200 packages at once (so creating a complete list seems not possible?). And I'm not entirely sure which packages I really use.

@garbas
Copy link
Member

garbas commented Dec 14, 2020

The "fancy website" basically already exists.

this is indeed quite cool. But I would still prefer a integration in nix/nixpkgs/nixos.

On the repology page I can't see more than 200 packages at once (so creating a complete list seems not possible?). And I'm not entirely sure which packages I really use.

This information is already in elasticsearch backend of search.nixos.org. We just need to find the right way to show it.

@Lassulus
Copy link
Member Author

This information is already in elasticsearch backend of search.nixos.org. We just need to find the right way to show it.

This also sounds lika a good idea, but would be orthogonal to the "feature" proposed in this RFC.

@garbas
Copy link
Member

garbas commented Dec 16, 2020

When evaluating a system (for example with nixos-rebuild) unmaintained packages should show up, so people are encouraged to maintain them.

Wouldn't this be too verbose? I think that exposing unmaintained package is going to help people maintaining them, but displaying them would mostly annoy users.

Maybe a middle ground would be to show a summary. eg. Your system is using X unmaintained packaged. We're looking for volunteers to maintain them. Follow this link https://....

But that might also rub new users the wrong way: Oh, "half" of NixOS is unmaintained. I wonder what would be a good way to do this.

@Ma27
Copy link
Member

Ma27 commented Dec 16, 2020

But that might also rub new users the wrong way: Oh, "half" of NixOS is unmaintained. I wonder what would be a good way to do this.

Just a random, probably not so well-thought, idea during lunch: what about something like nix why-depends, but for maintainers? This could e.g. be embedded into nixos-rebuild, so that interested people can see where help is needed when they run this dedicated command.

@Lassulus
Copy link
Member Author

Wouldn't this be too verbose? I think that exposing unmaintained package is going to help people maintaining them, but displaying them would mostly annoy users.

Maybe a middle ground would be to show a summary. eg. Your system is using X unmaintained packaged. We're looking for volunteers to maintain them. Follow this link https://....

The important part would be to have a list of which packages I use are unmaintained. If I get a number and a list of not maintained packages I don't know how I'm afffected.

I would be happy to know if a package I use is unmaintained. I want to have an option to disable this, so if people get annoyed by it, could disable it with this option. We could of course also make this opt-in instead of opt-out. But I guess most people don't know how many packages in their system are maintained and will never find out if not shown to them.

Yes, showing people, that "half" of NixOS is unmaintained is exactly the purpose of this. Hiding it would not be a good solution IMHO :D

@FRidh
Copy link
Member

FRidh commented Dec 16, 2020

Just a random, probably not so well-thought, idea during lunch: what about something like nix why-depends, but for maintainers? This could e.g. be embedded into nixos-rebuild, so that interested people can see where help is needed when they run this dedicated command.

With flakes it becomes more straightforward to do because you know of a derivation what inputs are used. Right now you could do, in case of your system, nix show-derivation -r $(readlink -f /run/current-system) to get the build-time closure of your system, however, maintainers are not part of the derivation and show-derivation shows a path, not an attribute, so you need to match paths with an evaluated Nixpkgs.

Anyway, users should not be bothered with these kinds of internal things; a user just wants to use the software. Whenever you use free and open source software you should be aware that things may not be maintained or can be unmaintained from one moment to the next, and that you're on your own. If you feel you only want to use software that is maintained, then use an evaluation test.

@Lassulus
Copy link
Member Author

Anyway, users should not be bothered with these kinds of internal things; a user just wants to use the software. Whenever you use free and open source software you should be aware that things may not be maintained or can be unmaintained from one moment to the next, and that you're on your own. If you feel you only want to use software that is maintained, then use an evaluation test.

If users are to be expected, that there is unmaintained software, why not show it to them?
I think a warning like unmaintained packages used: bzip2, curl, firefox, .... would not be very annoying. But maybe we can disable the warning on stable systems by default. Fresh users probably use a stable channel anyway.

I also think a warning like this would encourage users to adopt unmaintained packages (at least I would). Maybe we can link to some introductory material of how to become a package maintainer or something along those lines.

@Zimmi48
Copy link
Member

Zimmi48 commented Dec 16, 2020

It is my impression that the meta.maintainers field does not really reflect any "reality". What does it mean for a package to be maintained? In this RFC, it seems that you define it as "has a maintainer". But if the maintainer hasn't been active (hasn't updated or reviewed updates) recently, then is the package still actually maintained? And if there is no listed maintainer, does it really mean that the package is not maintained?

IMHO this RFC is incomplete as is and should consider the policy to implement on having this field reflect the "reality".

@Lassulus
Copy link
Member Author

It is my impression that the meta.maintainers field does not really reflect any "reality". What does it mean for a package to be maintained? In this RFC, it seems that you define it as "has a maintainer". But if the maintainer hasn't been active (hasn't updated or reviewed updates) recently, then is the package still actually maintained? And if there is no listed maintainer, does it really mean that the package is not maintained?

Yes, I suggest here, that a package is maintained when is has an entry in the maintainers field. I don't want to talk about how to update this field here by other tools or manually (that would be another RFC IMHO). I believe the reality and the maintainers field will align when the effects of having people there or not are better visible, like with a warning during evaluation.

Your questions are good questions, but I don't want to answer them in this RFC, because I fear we will go down a rabbit hole from which we will not come back.

@Zimmi48
Copy link
Member

Zimmi48 commented Dec 16, 2020

I don't want to talk about how to update this field here by other tools or manually (that would be another RFC IMHO). I believe the reality and the maintainers field will align when the effects of having people there or not are better visible, like with a warning during evaluation.

I'm fine with this being a separate RFC, but I don't share your belief of a magical alignment once this field becomes more visible. In particular, your proposal might make more maintained packages get an official maintainer, but it definitely won't help clearing the inactive maintainers that are currently listed.

Your questions are good questions, but I don't want to answer them in this RFC, because I fear we will go down a rabbit hole from which we will not come back.

I share your concern, but I also think that addressing those questions (possibly as part of another RFC) is a prerequisite for this one.

@Mic92 Mic92 added status: open for nominations Open for shepherding team nominations and removed status: new labels Dec 17, 2020
@Lassulus
Copy link
Member Author

I share your concern, but I also think that addressing those questions (possibly as part of another RFC) is a prerequisite for this one.

I don't think they are so entangled. This RFC is about transparency and showing our current status to the users/maintainers/interested parties. The current state is the maintainers field. Letting maintainers decay is another topic which is also important but not required for showing this information.

also:

I nominiate @kloenk as a shepherd
Does anyone else want to participate? please speak up :)

@garbas
Copy link
Member

garbas commented Dec 17, 2020

I'd like to nominate myself for shepherding.

@infinisil
Copy link
Member

infinisil commented Dec 17, 2020

Proof-of-concept NixOS module by searching for unmaintained packages in environment.systemPackages (so no transitive dependencies, which is probably for the better). Uses NixOS/nixpkgs#97023

{ config, ... }: {
_module.checks.unmaintainedPackages =
  let unmaintained = lib.filter (v: v ? pname && v ? meta.maintainers && v.meta.maintainers == []) config.environment.systemPackages;
  in {
    check = unmaintained == [];
    type = "warning";
    message = "These packages you're using don't have a maintainer: ${lib.concatMapStrings (v: "\n- ${v.pname}") unmaintained}";
  };
}

Since some maintainer-less packages are in the default environment.systemPackages, with an empty config this shows as

❯ nix-instantiate '<nixpkgs/nixos>' --arg configuration '{ fileSystems."/".device = "/"; boot.loader.grub.device = "/"; }' -A system
trace: warning: [unmaintainedPackages] These packages you're using don't have a maintainer: 
- bzip2
- gnused
- gnutar

Can be disabled by the user with

{
  _module.checks.unmaintainedPackages.enable = false;
}

@edolstra edolstra changed the title [RFC-0081] Show unmaintained packages [RFC 0081] Show unmaintained packages Dec 18, 2020
@spacekookie
Copy link
Member

spacekookie commented Jan 7, 2021

@Zimmi48 would you want to Shepherd this RFC?

@Zimmi48
Copy link
Member

Zimmi48 commented Jan 20, 2021

Sorry, I don't have enough available time at the moment to be part of a shepherd team.

@Mic92
Copy link
Member

Mic92 commented Jan 21, 2021

@Ma27 @mweinelt @infinisil Does any of you want to be shepherd on this RFC?

@Ma27
Copy link
Member

Ma27 commented Jan 21, 2021

Unfortunately not, but this is only due to a time limitation on my end.

@infinisil
Copy link
Member

I can be shepherd

@Mic92
Copy link
Member

Mic92 commented Jan 21, 2021

Ok. We have enough shepherds than: @infinisil @garbas @kloenk

@Mic92
Copy link
Member

Mic92 commented Feb 18, 2021

Can we have another meeting for this one?

@kloenk
Copy link
Member

kloenk commented Feb 18, 2021

Best for me would be after the fifth of march. As then School is much less for me.

@Lassulus
Copy link
Member Author

Lassulus commented Mar 3, 2021

lets plan another date: https://when2meet.com/?11230345-jwDak

@Lassulus
Copy link
Member Author

Lassulus commented Mar 9, 2021

alright, so we will meet 2021-03-14 19:00 UTC (which is 20:00 in Europe/Berlin timezone). I guess we will do it on jitsi again.

@Lassulus
Copy link
Member Author

meeting link: https://jitsi.lassul.us/rfc81

@Mic92
Copy link
Member

Mic92 commented Mar 14, 2021

RFC 81 Meeting #2 Notes

Opt-in or not

Under the assumption that this feature is expensive, we should make it opt-in by default.

In case this feature can be implemented more efficiently in the future, we can change this to opt-out.

Say max 1% eval time increase if opt-out.

An opt-in prototype could be implemented and included into nixpkgs without this RFC beeing done/accepted

Limitations of current prototype

The current prototype is incapabale of finding packages which are "hidden" inside a string. example "${pkgs.firefox}/bin/firefox"

However it can find all buildInputs, and co. which should be good enough as a start.

Interface proposals

Opt-in

nixos-rebuild --show-unmaintained

Additionally we could inform the user about this flag at the end of a normal nixos-rebuild, like:

Did you know:
You can pass --show-unmaintained to nixos-rebuild 
to show unmaintained packages used in your system

This could be one of many "Did you know"s

Opt-out

At the end of nixos-rebuild, display a count of unmaintaned packages like

Your system depends on 5 unmaintained packages
Run `nixos-rebuild rebuild --show-unmaintained` to see the full list

Implement it by propagated this info in stdenv

stdenv can fill out a $out/nix-support/unmaintained which contains all the buildInputs and co. that don't have a maintainer, plus the $out/nix-support/unmaintained of them (making it recursive). This could be very efficient.

Future work: CI checks for maintainership

Since the goal of this RFC is to have fewer unmaintained packages, in the future we could have a CI check that ensures all packages have a maintainer. Packages would then have to be removed when the last maintained removes themselves.

We can have the next meeting when we have a nixos-rebuild prototype as a PR.

@infinisil
Copy link
Member

I discussed and worked a bit on this with @garbas today. Some points that were brought up:

  • If we can have it be opt-out, we should only do so for people not using the stable channel. Reason being that stable could be outdated by many months, giving outdated information on which packages are maintained. In addition, people on unstable are more likely to step up to maintain packages.
  • We should maybe show the date/commit of the maintainer state that is used in the output, so we can know how up-to-date it is
  • With the search.nixos.org overhaul, backed by elasticsearch, it's now possible to do a single curl query to get up-to-date information on which packages don't have any maintainers, assuming you have an attribute path to search for

@garbas then created a simple example of how such a search.nixos.org query can be done. With a query.json file with contents

{
  "from": 0,
  "size": 10000,
  "sort": [
    {
      "_score": "desc",
      "package_attr_name": "desc",
      "package_pversion": "desc"
    }
  ],
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": {
              "value": "package",
              "_name": "filter_packages"
            }
          }
        }
      ],
      "must": [
        {
          "terms": {
            "package_attr_name": [
              "hello",
              "haskellPackages.hello",
              "haskellPackages.gtk2hs-hello"
            ]
          }
        },
        {
          "term": {
            "package_maintainers_set": {
              "value": "No maintainers"
            }
          }
        }
      ]
    }
  }
}

You can do a query for the given attribute paths with the following (the authorization key is public, don't worry)

curl 'https://nixos-search-5886075189.us-east-1.bonsaisearch.net/latest-19-20.09/_search' \
  -H 'Authorization: Basic ejNaRko2eTJtUjpkczhDRXZBTFBmOXB1aTdYRw==' \
  -H 'Content-Type: application/json' \
  -H 'Pragma: no-cache' \
  -H 'Cache-Control: no-cache' \
  --data @query.json

Meanwhile, I worked on a prototype for the idea of having an $out/nix-support/unmaintained be created by stdenv. However I implemented it a bit differently: To have a $out/nix-support/attribute-deps which contains a list of attribute paths that the derivation depends on (recursive build time deps, not runtime deps, for now). I implemented this with the commits 857efbc2 and d33a27b2. It's a bit messy with the whole splicing part, but it basically works (requires stdenv rebuild of course):

$ nix-build -A mbuffer
/nix/store/n3kx0839798la0k8sakvzwravb8grhsi-mbuffer-20200929
$ cat result/nix-support/attribute-deps | sort -u
binutils
pkgsHostTarget.openssl
pkgsHostTarget.perl

Once you have this information, it's very easy to query maintainer information for those packages. Doing it this way is very flexible, as you can use this list of attribute paths for not only maintainer information, but also e.g. for licenses or other things. It also allows integration with the above search.nixos.org query, therefore getting more up-to-date information than the user might have in their nixpkgs version.

However, the code is also easily adjustable to query maintainer information in the same evaluation.

@infinisil
Copy link
Member

infinisil commented Mar 19, 2021

A tiny problem with this approach is that attribute changes can now influence derivations. So e.g. renaming pkgs.foo to pkgs.bar requires rebuilding all reverse-dependencies. Not sure if that's actually a problem, but something to keep in mind.

Here's another example:

$ nix-build -A python3
/nix/store/vipj0kvgnwg22gypla36lhj63ajcck63-python3-3.8.8
$ cat result/nix-support/attribute-deps | sort -u
binutils
gcc
pkgs.gettext
pkgsHostTarget.autoconf
pkgsHostTarget.autoreconfHook
pkgsHostTarget.bzip2
pkgsHostTarget.expat
pkgsHostTarget.gdbm
pkgsHostTarget.gettext
pkgsHostTarget.help2man
pkgsHostTarget.libffi
pkgsHostTarget.lzma
pkgsHostTarget.m4
pkgsHostTarget.ncurses
pkgsHostTarget.nukeReferences
pkgsHostTarget.openssl
pkgsHostTarget.perl
pkgsHostTarget.perlPackages.LocaleGettext
pkgsHostTarget.perlPackages.perl
pkgsHostTarget.pkg-config
pkgsHostTarget.readline
pkgsHostTarget.sqlite
pkgsHostTarget.xz
pkgsHostTarget.zlib
pkgs.perl532

@7c6f434c
Copy link
Member

7c6f434c commented Mar 20, 2021 via email

@infinisil
Copy link
Member

Oh yeah you're right, and that seems like a dealbreaker:

$ nix-instantiate --eval -A linux_latest.__path
[ "linux_latest" ]
$ nix-instantiate --eval -A linux_5_11.__path
[ "linux_5_11" ]

@7c6f434c
Copy link
Member

7c6f434c commented Mar 21, 2021 via email

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-9/12357/1

@Mic92
Copy link
Member

Mic92 commented Jun 24, 2021

Closing for now until @Lassulus has time to work on a prototype.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/get-all-unmaintained-nixpkgs-attributes-that-are-part-of-my-nixos-system/15472/2

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-get-package-maintainers-of-all-installed-packages/27307/8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.