Build code as packages/PPAs

[zerebubuth]

During the discussion of a PR on the chef repo, the subject of whether it would be better to have an internal APT repo or use PPAs for built code (as opposed to building code within the cookbook as we currently do) came up. Rather than continue the discussion there, it seemed better to have it here where it was more visible (and appropriately titled).

@zerebubuth: Might it be better to build these into packages, either hosted internally or using something like launchpad?

@tomhughes: Oh that would be ideal yes, and if it was an RPM based distro then I would do it in a heartbeat. I still barely understand dpkg packaging however and as best I can tell it's incredibly painful to work with and likewise for building your own repos (hence using a PPA).

If you're up for doing that then I'm certainly not going to stop you however!

@gravitystorm: I don't like packages/PPAs for this stuff - it makes it much harder for an observer to figure out how the system works, and the creation of packages is often not code-driven, so it becomes hard to reproduce and/or handover in the future.

@tomhughes: Checking out and building from chef is not at all robust though. I'd much rather have packages which can be independently built and validated.

You just have a git repo with the packing stuff, or at least that would be easy for RPMs. Mostly just one spec file for each package and maybe the odd patch or auxiliary file.

As I say however I don't think it's something we can do right now unless we have a much more experienced debian person wanting to do the work of dealing with it.

[zerebubuth]

I don't like packages/PPAs for this stuff - it makes it much harder for an observer to figure out how the system works, and the creation of packages is often not code-driven, so it becomes hard to reproduce and/or handover in the future.

I don't understand. It seems to me that it would be easier for someone to figure out how the system works, as much of the complexity of the cookbook and many of the packages it installs as dependencies would go away. The dependencies would be specified in the built .deb package, which seems pretty logical to me.

I get the point about reproducibility... but we're already missing out on that by specifying a 'live' tag rather than a revision hash in the cookbook. 😔

[gravitystorm]

Let's take osm2pgsql for example. https://launchpad.net/~osmadmins/+archive/ubuntu/ppa shows it as 0.81.0-1~precise1 , indicating (by version number alone i.e. not exactly reliably) that it's a patched version of the upstream osm2pgsql codebase. What does the patch contain? How would I find out? Does anyone other than the person who uploaded the package have the patch? Does that person still have the patch kicking around on their laptop ~2 years later?

If it had been built via a chef recipe, then the git checkout and/or other chef code would indicate exactly what's being used and what's in the patches.

With packages/ppas it's possible to avoid this situation of having binaries of unknown origin, but involves a lot more tooling. For example, the ppa could only contain artifacts from a build process (like jenkins/travis etc), and nobody has permissions to upload packages directly.

[zerebubuth]

What does the patch contain? How would I find out? Does anyone other than the person who uploaded the package have the patch? Does that person still have the patch kicking around on their laptop ~2 years later?

Launchpad does keep that information, so it's slightly better than "random upload from someone's laptop".

If it had been built via a chef recipe, then the git checkout and/or other chef code would indicate exactly what's being used and what's in the patches.

Not the way we're using Chef. Take a look at the recipes - at least two that I looked at have a revision "live" parameter on the git checkout, which means at a later date we have no idea which revision was used. Unless we look at the logs, but who knows if the logs are kicking around on the servers ~2 years later?

With packages/ppas it's possible to avoid this situation of having binaries of unknown origin, but involves a lot more tooling.

Yup, this is a good argument in favour of hosting our build & APT repo internally, so that we can ensure that any extra patches included are checked in somewhere and referenced from the package.

A more general point: Knowing the exact revision of the code we're running quickly turns into a bit of a nightmare. Many build packages install their dependencies (gcc, -dev packages) and we have no idea what version those are. The goal of perfectly reproducible builds is a good one, but would require more radical changes.

[gravitystorm]

at least two that I looked at have a revision "live" parameter on the git checkout

There's a few possibilities (use either live or master below):

• action :checkout / revision 'live' - this is a one-off checkout of whatever is live at the time of the chef run. I think we should avoid this and I'd be surprised if we're using it at all.
• action :sync / revision 'live' - reasonable if we're pretty confident we want to deploy a moving target. Mostly this will be where whoever is writing the code is also responsible for deploying it (e.g. rails port) but there's a separation of concerns issue here. Also useful where things move quickly upstream.
• action :sync / revision 'x.y.z' - this should be the most normal case (e.g. openstreetmap-carto) where we're explicit about what version is running. Version would normally be as an attribute. Needs some sort of upstream release process though, and is a bit of make-work when it's either the same person or when releases happen continuously.

at a later date we have no idea which revision was used

That's not so much of an issue for me as making it easy and clear to figure out which code is (modulo chef-run timeframes) being currently used. Thanks for the launchpad links, I didn't realise that the source of the packages was in fact available.

[eighthave]

Debian source packages contain all of the patches needed to build the binary package. And Debian, Ubuntu, launchpad.net PPAs, etc all keep the source packages for all of their binary packages. So while the Debian package format is not the most readable format out there, the information is all there. As someone who has not used Chef, I would have all of the same complaints as @gravitystorm about Chef scripts ;-)

Standard Debian packages are based around the original upstream tarball, i.e. the official source release. Then there is an accompanying debian tarball with all the stuff needed to build the package, including patches, for example:
https://launchpad.net/~osmadmins/+archive/ubuntu/ppa/+files/boost1.49_1.49.0-2ubuntu1~precise3.debian.tar.gz

I'm a Debian Developer (i.e. I have upload privs) and I'm happy to help with this where I can. It seems to me that the ideal sitaution would be to get the OSM fixes into the official Debian packages so that no patches are needed.

[Firefishy]

Link #862 #807 #999