Enable declarative projects.

[shlevy]

This allows fully declarative project specifications. This is best
illustrated by example:

• I create a new project, setting the declarative spec file to
  "spec.json" and the declarative input to a git repo pointing
  at git://github.com/shlevy/declarative-hydra-example.git
• hydra creates a special ".jobsets" jobset alongside the project
• Just before evaluating the ".jobsets" jobset, hydra fetches
  declarative-hydra-example.git, reads spec.json as a jobset spec,
  and updates the jobset's configuration accordingly:

{
    "enabled": 1,
    "hidden": false,
    "description": "Jobsets",
    "nixexprinput": "src",
    "nixexprpath": "default.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}

• When the "jobsets" job of the ".jobsets" jobset completes, hydra
  reads its output as a JSON representation of a dictionary of
  jobset specs and creates a jobset named "master" configured
  accordingly (In this example, this is the same configuration as
  .jobsets itself, except using release.nix instead of default.nix):

{
    "enabled": 1,
    "hidden": false,
    "description": "js",
    "nixexprinput": "src",
    "nixexprpath": "release.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}

[shlevy]

@rickynils @aszlig

[shlevy]

Example at https://github.com/shlevy/declarative-hydra-example

[aszlig]

Very cool! 👍

[shlevy]

In case anyone happens to be reviewing this right now, I have a significant improvement coming shortly.

[shlevy]

OK, I've force-pushed the new approach and updated the description.

[rickynils]

I've tested this and it is working nicely. I'm going to migrate our production builds to it, and might have more feedback after that.

[shlevy]

@edolstra ping

[rbvermaa]

@shlevy would you be able to add a simple test?

[shlevy]

Hmm, I'm not sure the best way to do that. A proper test for this feature would involve creating a declarative project through the UI, waiting for the evaluator to evaluate, waiting for the queue runner to build, and then waiting for the evaluator to evaluate the actual jobset. Is there a good way to implement an integrated test like that currently?

[edolstra]

I've been thinking about declarative projects jobsets, but I'd prefer to implement it in Nix, by having some builtin fetchgit / fetchhg functions. That way, you can have a release.nix expression that does something like:

with import (fetchgit https://github.com/NixOS/nixpkgs.git) {};
{
  job = stdenv.mkDerivation { ... }; 
}

The advantage over doing it in Nix is that nix-build release.nix -A foo will do the right thing, i.e., you can build Hydra jobs outside of Hydra.

[shlevy]

But then we can't add new inputs with simple plugins. I want to e.g. add a way to check a github repo for new PRs as a new kind of input, so that hydra can automatically build pull requests and replace travis, are we really going to accept that kind of thing as a nix builtin?

[shlevy]

(not to mention the fact that these builtins will likely need to manage caching somehow, we'll probably lose the benefit of cached evals, etc.)

[copumpkin]

A happy medium between the two might be NixOS/nix#520. I've actually been toying recently with simple schemes (would require a new primop in Nix, but not too much) to make that idea happen. The advantage there is that we could write new input types in Nix without a separate plugin mechanism, and without hardcoding them into the core of Nix. It could also unify a lot of other similar sources of "lockable" nondeterminism throughout Nix...

[shlevy]

Can you give an example of how, say, git inputs would work there?

[copumpkin]

Think of a nix-flavored generalization of Gemfile.lock.

The nondetDerivation (or whatever the name ends up being) would take a builder script, just like a regular derivation. That builder script is treated roughly like a fixed-output derivation from a sandboxing perspective, but we don't require a fixed hash output.

Instead, we require the following of nondetDerivation builders:

1. It should produce $out, which is the actual thing your builder is producing
2. It should produce $anchor (could be a distinct output or we could combine with 3 below into a structured file), which is some abstract notion of a long-lived hook that should let you retrieve the thing you asked for again
3. It should produce $anchorHash, which is the sha256 (or whatever) of the thing it fetched
4. It should provide some interface to accept previous anchors and behaves like a fixed-output derivation in such cases
5. It should provide some indication of how often it should be run (possibly incorporated into the hash)

Nix, when fed one of these things:

1. Runs the script
2. Produces a "hashed witness" that acts like an assertion from Nix saying "hey, I ran the builder with hash X, and it produced fixed-output with hash Y, at time T".
3. The witness produced in 2 gets spit out as a special "side channel" during normal execution

A user then:

1. Runs a Nix script that uses nondetDerivation, passing some flag into nix-build telling it that you're allowing nondeterminism, and possibly where to link the anchors
2. It fetches the e.g., latest git ref from some URL, and builds as usual (this needs to happen before other stuff so we can resolve the hash)
3. Along with result, it spits out symlinks into the witnesses from your build in the folder you specify

The witnesses can then be used (in a couple of different ways including NixOS/nix#709) to get the same output.

That's the ideal, at least. I don't have all the details worked out but I think something like the above is possible in principle.

We could then use this for a bunch of VCS fetchers, as well as channel fetching (the anchor in that case would be the full URL post-redirect), and possibly even resolvers for other systems (ivy can produce a resolution report that "locks down" the result of a lookup), and generally allow us to ask for "latest" things in a semi-deterministic manner.

[shlevy]

Hm, cool! I'd love to see something like that get merged in eventually. (though in the meantime I think the approach in this PR is pretty solid 😉 )

[copumpkin]

I agree that it is! I just want my thing for a bunch of other reasons and
figured I'd mention it :)

On Wed, Apr 6, 2016 at 07:48 Shea Levy notifications@github.com wrote:

Hm, cool! I'd love to see something like that get merged in eventually.
(though in the meantime I think the approach in this PR is pretty solid [image:
😉] )

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#277 (comment)

[shlevy]

Rebased and added a declInput input to the project evaluation that contains revision info etc. for the declarative project input ( @rickynils )

@edolstra ping.

[rickynils]

@edolstra This PR has been working great for me since its inception. It allows me to keep all CI stuff in a single version-controlled place, instead of manually updating mutable Hydra projects. Any chance of merging this?

[Ericson2314]

NixOS/nix@38539b9

[domenkozar]

cc @aszlig for FYI

[shlevy]

@edolstra I will close this by the end of the week if I haven't heard otherwise. I would appreciate at least some response here, even if confirming that this is not wanted.

[edolstra]

Some comments:

• I don't understand the need for a jobsets job. Why not just have a single .json file listing all the jobsets? That way you don't need to build anything.
• Having a .jobsets jobset seems ugly from a UI perspective. It would be cleaner to give the Projects table a repo URL that hydra-evaluator would fetch periodically.
• Do jobsets get disabled or deleted if they're removed from the declarative spec?

[rickynils]

• Having a nix expression for the jobsets is more flexible. For example, I pick up the inputs from a json-like file paths.nix. This file is also used by other tools in our build/deploy toolchain. The jobsets job does some mangling of the inputs, and can also inject some more stuff into the jobs based on other assets in our repo (like emailoverride etc). If the jobsets were a static json file, I would have to duplicate various pieces of information.
• I agree the .jobset jobset is ugly from a UI perspective and would prefer a nicer implementation. I do not regard it a merge-blocker since it will not appear if you don't use this feature.
• The jobsets get disabled. I would like them to get removed instead. @shlevy: Any reasoning behind that behavior?

All in all, yeah, this feature seems a little "tacked-on" to the rest of Hydra, but from a functional standpoint it is golden. I suggest merging it, collect feedback from users, and start thinking about a nicer (possibly more intrusive) way of implementing a similar feature.

[Ericson2314]

@edolstra My point of reference is https://github.com/codepath/android_guides/wiki/Building-Gradle-Projects-with-Jenkins-CI, and this seems strictly less jank :). For the record, I believe this also

I do like the idea of a single nix-derivation providing hydra with all of its state. Down the road perhaps we can make the hydra UI for "imperative jobs" actually modify the git repo which is (aka should be) fetched in the jobset job? That would make this less of a tack-on and move all state out of hydra -- a nice if not pressing goal.

Also, given that builtin.fetchGit now exists, can we leverage it here? inputs could be a json object / attrset of of derivations somehow? (I haven't taking a hard enough look at the implementation to know how much sense that makes.)

[shlevy]

I don't understand the need for a jobsets job. Why not just have a single .json file listing all the jobsets? That way you don't need to build anything.

And you can't do anything dynamic at all without some extra functionality to automatically update the repo containing the file or whatever. As @rickynils points out, this is a lot of information duplication.

Having a .jobsets jobset seems ugly from a UI perspective. It would be cleaner to give the Projects table a repo URL that hydra-evaluator would fetch periodically.

Agreed that the UI isn't the cleanest here. This was just the quickest way to get the functionality working, and I'd be happy to iterate more on it (though I agree with @rickynils that there's no need for that to block this merge).

Do jobsets get disabled or deleted if they're removed from the declarative spec?

Currently disabled, though this is easy to change of course.

[shlevy]

The jobsets get disabled. I would like them to get removed instead. @shlevy: Any reasoning behind that behavior?

I wasn't sure what would happen to old builds etc of deleted jobsets, and it seems to be the practice on hydra.nixos.org to disable jobsets, not delete them. FWIW disabled jobsets don't show up to non-logged in users, I think.

[shlevy]

@Ericson2314 I think your first paragraph was cut off 😄

I like the idea of the imperative UI eventually using this as a backend!

Not sure what you want to do with fetchgit exactly here?

[Ericson2314]

@shlevy No missing info, just forgot to remove that sentence fragment :).

for fetchGit I am kinda thinking back to #277 (comment) . Like it or not, we now have fetchBuiltin, so might as well use it. For example, whenever I see a json file, I think we can always generalize to an arbitrary nix expression (which could just fromJSON a file if the user chooses). Perhaps better

Just before evaluating the ".jobsets" jobset, hydra fetches declarative-hydra-example.git, reads spec.json as a jobset spec, and updates the jobset's configuration accordingly:

It seems like that step could use fetchGit. I guess overall think how this should work with @copumpkin's deterministic derivations, and then try to shoe-horn in fetchgit instead.

[edolstra]

This is currently giving a merge conflict.

[shlevy]

Rebased

[domenkozar]

Awesome - I'll setup a playground hydra tomorrow and test this out. It's going to be super useful.

@shlevy could I ask you to add a section to hydra manual about this feature?

[shlevy]

@domenkozar will do

[domenkozar]

@shlevy I'd love to use this asap, any eta? :)

[shlevy]

#316