Skip to content

Commit

Permalink
Add fetchgit builtin
Browse files Browse the repository at this point in the history
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
  • Loading branch information
edolstra committed Apr 29, 2016
1 parent 8325822 commit 38539b9
Show file tree
Hide file tree
Showing 2 changed files with 78 additions and 1 deletion.
77 changes: 77 additions & 0 deletions src/libexpr/primops/fetchgit.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#include "primops.hh"
#include "eval-inline.hh"
#include "download.hh"
#include "store-api.hh"

namespace nix {

static void prim_fetchgit(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
// FIXME: cut&paste from fetch().
if (state.restricted) throw Error("‘fetchgit’ is not allowed in restricted mode");

std::string url;
std::string rev = "master";

state.forceValue(*args[0]);

if (args[0]->type == tAttrs) {

state.forceAttrs(*args[0], pos);

for (auto & attr : *args[0]->attrs) {
string name(attr.name);
if (name == "url")
url = state.forceStringNoCtx(*attr.value, *attr.pos);
else if (name == "rev")
rev = state.forceStringNoCtx(*attr.value, *attr.pos);
else
throw EvalError(format("unsupported argument ‘%1%’ to ‘fetchgit’, at %3%") % attr.name % attr.pos);
}

if (url.empty())
throw EvalError(format("‘url’ argument required, at %1%") % pos);

} else
url = state.forceStringNoCtx(*args[0], pos);

if (!isUri(url))
throw EvalError(format("‘%s’ is not a valid URI, at %s") % url % pos);

Path cacheDir = getCacheDir() + "/nix/git";

if (!pathExists(cacheDir)) {
createDirs(cacheDir);
runProgram("git", true, { "init", "--bare", cacheDir });
}

Activity act(*logger, lvlInfo, format("fetching Git repository ‘%s’") % url);

std::string localRef = "pid-" + std::to_string(getpid());
Path localRefFile = cacheDir + "/refs/heads/" + localRef;

runProgram("git", true, { "-C", cacheDir, "fetch", url, rev + ":" + localRef });

std::string commitHash = chomp(readFile(localRefFile));

unlink(localRefFile.c_str());

debug(format("got revision ‘%s’") % commitHash);

// FIXME: should pipe this, or find some better way to extract a
// revision.
auto tar = runProgram("git", true, { "-C", cacheDir, "archive", commitHash });

Path tmpDir = createTempDir();
AutoDelete delTmpDir(tmpDir, true);

runProgram("tar", true, { "x", "-C", tmpDir }, tar);

Path storePath = state.store->addToStore("git-export", tmpDir);

mkString(v, storePath, PathSet({storePath}));
}

static RegisterPrimOp r("__fetchgit", 1, prim_fetchgit);

}
2 changes: 1 addition & 1 deletion src/libstore/download.cc
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ bool isUri(const string & s)
size_t pos = s.find("://");
if (pos == string::npos) return false;
string scheme(s, 0, pos);
return scheme == "http" || scheme == "https" || scheme == "file" || scheme == "channel";
return scheme == "http" || scheme == "https" || scheme == "file" || scheme == "channel" || scheme == "git";
}


Expand Down

5 comments on commit 38539b9

@domenkozar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really need to be consistent here and pick a name that:

  • hints the fetching is done at eval time
  • matches fetchTarball alternative that just uses a different format

@shlevy
Copy link
Member

@shlevy shlevy commented on 38539b9 May 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nix-exec has a fetchgit builtin we may want to pull from here https://github.com/shlevy/nix-exec/blob/master/src/fetchgit.cc https://github.com/shlevy/nix-exec/blob/master/scripts/fetchgit.sh.in

FWIW I'm a bit worried about using one local cache repo for everything. It doesn't seem to gain much (seems unlikely that a lot of big files will be shared between repos) and just costs us a higher chance of collision. nix-exec has a separate local repo for each repo.

@lheckemann
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shlevy I think significant savings can be made when sharing e.g. nixpkgs and nixpkgs-channels, or different linux kernel repos, for example.

@shlevy
Copy link
Member

@shlevy shlevy commented on 38539b9 Mar 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lheckemann On the other hand, we see ridiculously huge slowdowns (to the point of unusability) when using builtins.fetchGit on multiple actually-distinct repos. I'm fine with a way to give a common cache name to disparate repos (e.g. "nixpkgs-channels should live with nixpkgs"), but the current situation is effectively broken for the multi-repo usecase.

@lheckemann
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I've never seen that. I suppose I don't use fetchGit enough!

Please sign in to comment.