FileHistory Feature #963

ThomasBarnekow · 2015-02-17T00:58:03Z

Based on my research, this feature was discussed in the past but I could not find the functionality in libgit2sharp. As I needed it as well, I implemented it and would like to share it.

The FileHistory class provides a history of commits in which a given file was created, changed, or renamed. Based on that commit history, it also provides the collection of Blobs representing a change of the file's contents, i.e., ignoring renames.

I've also implemented a FileHistoryExtensionsclass that provides extension methods for the Repositoryclass. These could potentially be moved to the RepositoryExtensionsclass.

Usage would be as follows:

using (Repository repo = new Repository(repoPath))
{
    FileHistory history = repo.GetFileHistory(relativePath);
    IEnumerable<FileHistoryEntry> relevantCommits = history.RelevantCommits();
    IEnumerable<Blob> relevantBlobs = history.RelevantBlobs();
}

Therzok · 2015-02-17T01:07:11Z

.nuget/NuGet.targets

@@ -0,0 +1,144 @@
+<?xml version="1.0" encoding="utf-8"?>


This shouldn't have landed here. 🍊

ThomasBarnekow · 2015-02-17T08:55:20Z

I reverted the undesired changes to the solution and project files and also deleted the NuGet files that were added. The question is whether git settings should ignore these files.

nulltoken · 2015-02-17T09:09:27Z

Hey @ThomasBarnekow thanks for this. It looks like the CI servers cringe a bit 😉 In order oavoid this, please specify some Signatures in your Commit calls. Otherwise, the signature is going to be guessed from the ambient .gitconfig configuration store (which doesn't exist on the CI servers).

nulltoken · 2015-02-17T09:10:10Z

Related topics #893 and #89

ThomasBarnekow · 2015-02-17T09:12:36Z

Hey @nulltoken, you are welcome. Can you be more specific on "specify[ing] some Signatures in your Commit calls)?

nulltoken · 2015-02-17T09:15:12Z

@ThomasBarnekow I think leveraging this overload would do the trick.

ThomasBarnekow · 2015-02-17T10:10:44Z

I also saw that the MetaFixture test complained about missing constructors to support abstraction in test contexts. It also wanted the public methods and properties to be virtual. In addition to adding the Signature to the test commits, I also made those required changes.

When I ran the tests on my machine, my test cases and the MetaFixture test cases all passed. However, other test cases failed but I can't see any connection between my stuff and those test cases.

ThomasBarnekow · 2015-02-17T12:12:54Z

The last change fixes an error I've seen on the CI system.

One challenge is that my tests passed nicely on my machine but failed on the CI server. Thus, it would be helpful to have some guide saying how tests should be set up to make sure they also run on the CI server, i.e., they should also fail on my machine if they would fail on the CI server.

BTW, a few of the tests, e.g., LibGit2Sharp.Tests.RefSpecFixture.SettingInvalidRefSpecsThrows fail on my machine due to an UnauthorizedAccessException. For example:

System.UnauthorizedAccessException : Access to the path 'C:\Visual Studio 2010\Projects\GitHub\LibGit2Sharp\LibGit2Sharp.Tests\bin\Debug\TestRepos\1xokwaiz.xg3\testrepo_wd\.git\objects\pack\pack-a81e489679b7d3418f9ab594bda8ceb37dd4c695.idx' is denied.

How do I deal with that?

nulltoken · 2015-02-17T13:35:45Z

BTW, a few of the tests, e.g., LibGit2Sharp.Tests.RefSpecFixture.SettingInvalidRefSpecsThrows fail on my machine due to an UnauthorizedAccessException. For example:
System.UnauthorizedAccessException : Access to the path 'C:\Visual Studio
2010\Projects\GitHub\LibGit2Sharp\LibGit2Sharp.Tests\bin\Debug\TestRepos\1xokwaiz.xg3\testrepo
_wd\.git\objects\pack\pack-a81e489679b7d3418f9ab594bda8ceb37dd4c695.idx' is denied.
How do I deal with that?

Hmmm. This is unexpected. Can you please share the whole stack trace? Would it possible that a local service (Antivirus, Search indexer, ...) may hold a temporary lock of those files?

nulltoken · 2015-02-17T13:50:37Z

The last change fixes an error I've seen on the CI system.

One challenge is that my tests passed nicely on my machine but failed on the CI server. Thus, it
would be helpful to have some guide saying how tests should be set up to make sure they also run
on the CI server, i.e., they should also fail on my machine if they would fail on the CI server.

The only case I can think of relies on the methods/tests that would depend on configuration. The bulletproof approach to cope with this is to isolate the tests from ambient configuration by injecting some fake configuration stores. Another approach (as hinted here, is to pass explicit parameters to the methods and not let the library retrieve the missing required piece of data from config.

ThomasBarnekow · 2015-02-17T15:06:11Z

Here's a full stack trace:

LibGit2Sharp.Tests.DiffTreeToTreeFixture.CanCompareTwoVersionsOfAFileWithATrailingNewlineDeletion(contextLines: 4, expectedPatchLength: 193) [FAIL]
   System.IO.IOException : The process cannot access the file 'C:\Visual Studio 2010\Projects\GitHub\LibGit2Sharp\LibGit2Sharp.Tests\bin\Debug\TestRepos\eodajdlt.kwj\testrepo_wd\README' because it is being used by another process.
   Stack Trace:
      at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
      at System.IO.File.InternalDelete(String path, Boolean checkHost)
      C:\Visual Studio 2010\Projects\GitHub\LibGit2Sharp\LibGit2Sharp.Tests\TestHelpers\DirectoryHelper.cs(63,0): at LibGit2Sharp.Tests.TestHelpers.DirectoryHelper.DeleteDirectory(String directoryPath)
      C:\Visual Studio 2010\Projects\GitHub\LibGit2Sharp\LibGit2Sharp.Tests\TestHelpers\DirectoryHelper.cs(68,0): at LibGit2Sharp.Tests.TestHelpers.DirectoryHelper.DeleteDirectory(String directoryPath)
      C:\Visual Studio 2010\Projects\GitHub\LibGit2Sharp\LibGit2Sharp.Tests\TestHelpers\BaseFixture.cs(212,0): at LibGit2Sharp.Tests.TestHelpers.BaseFixture.Dispose()
      at Xunit.Sdk.LifetimeCommand.Execute(Object testClass)
      at Xunit.Sdk.ExceptionAndOutputCaptureCommand.Execute(Object testClass)

It is possible, though, that this is related to a local service. I've not found out which one that might be, because I still had 2-3 errors like the one above after deactivating my antivirus and malware services.

One further thing I noted is that 42 "temporary" folders are left in the TestRepos subdirectory after one test run and a previously empty TestRepos folder. My folder contained a few hundred of these, though. Not sure whether that's expected.

nulltoken · 2015-02-17T15:27:17Z

@ThomasBarnekow The stack trace shows that the failure happened upon the removal of the test repo.

Running a full test suite would create a lot more than 42 temporary folders. So I'd say most of them have been successfully removed. Take a look at the DirectoryHelper type. We guard against IOExceptions and output a message when such race conditions happen. Adding a clause for your specific exception may help you while you hunt for the evil process.

ThomasBarnekow · 2015-02-17T16:11:37Z

@nulltoken Indeed, I saw that many test folders were created and successfully removed. The only "issue" I'd have is that, after a while, the leftovers might become a problem.

Your guarding against IOException does not mean the test will be passed, though. They are reported as a failure. Not sure whether that should be considered a failed test case.

Would it make sense to include a "housekeeping" test case that is run at the very end to remove whatever folders are left? I would not have any issue with some leftover folders. But during the limited time I've been using this, hundreds of folders have been created that way.

nulltoken · 2015-02-17T18:24:11Z

after a while, the leftovers might become a problem.

I agree. But to my best knowledge, this should only happen when an external process (anti-virus, ...) races (and wins) against the test runner.

Your guarding against IOException does not mean the test will be passed, though. They are reported as a failure. Not sure whether that should be considered a failed test case.

Actually, when an IOException occur, it's swallowed and a Trace message is output to the console. In your case, it looks like you're fighting against a different exception that doesn't derive from IOException.

Not sure whether that should be considered a failed test case.

IMHO I don't think that failing to clean the test repo once the test is done should be seen as a failure.

Would it make sense to include a "housekeeping" test case that is run at the very end to remove whatever folders are left?

Why not? But I think this may be a little out of the scope of this PR. How about logging a new issue in order to keep track of it? As it may take some time to iron in a way that's sensible (For instance, NCrunch and XUnit v2 run tests in parallel. In that perspective, it may be tricky to determine the last test. Another option would be to add a finalizer to BaseFixture, but if it takes too long to execute, it will be shut down before it's done.)

ThomasBarnekow · 2015-02-17T19:01:40Z

All agreed. And these failures are totally unrelated to my code. I'd be interested to have it pulled as soon as possible and don't see any issue standing in its way.

Mit freundlichen Grüßen / Kind regards

Dr. Thomas Barnekow

Sent from my iPhone

On 17.02.2015, at 19:24, nulltoken notifications@github.com wrote:

after a while, the leftovers might become a problem.

I agree. But to my best knowledge, this should only happen when an external process (anti-virus, ...) races (and wins) against the test runner.

Your guarding against IOException does not mean the test will be passed, though. They are reported as a failure. Not sure whether that should be considered a failed test case.

Actually, when an IOException occur, it's swallowed and a Trace message is output to the console. In your case, it looks like you're fighting against a different exception that doesn't derive from IOException.

Not sure whether that should be considered a failed test case.

IMHO I don't think that failing to clean the test repo once the test is done should be seen as a failure.

Would it make sense to include a "housekeeping" test case that is run at the very end to remove whatever folders are left?

Why not? But I think this may be a little out of the scope of this PR. How about logging a new issue in order to keep track of it? As it may take some time to iron in a way that's sensible (For instance, NCrunch and XUnit v2 run tests in parallel. In that perspective, it may be tricky to determine the last test. Another option would be to add a finalizer to BaseFixture, but if it takes too long to execute, it will be shut down before it's done.)

—
Reply to this email directly or view it on GitHub.

nulltoken · 2015-02-17T19:42:03Z

I'd be interested to have it pulled as soon as possible and don't see any issue standing in its way.

I'll start to review the code in the following hours. I also think some other contributors may chime in as this is a very interesting feature (@Aimeast, @Jameson, ...).

Therzok · 2015-02-17T20:05:23Z

LibGit2Sharp/FileHistory.cs

+                if (lastSha != null)
+                {
+                    CommitFilter endOfPreviousRangeFilter = GetCommitFilter(filter, commitRange.Last());
+                    Commit nextCommit = commitLog.QueryBy(endOfPreviousRangeFilter).Skip(1)


Nit: Can we have these like one linq chain each line?

We probably could and I actually thought about that (because I've written a good number of pure functional transforms for XML and like that style). But I decided to go down this route because I felt it would be easier to read.

What benefits do you see in "having these like one line chain each line"?

nulltoken · 2015-02-17T20:25:57Z

Thanks a lot for this contribution!

I'm actually not sure about the implementation (as the git native --follow option copes with many corner cases). So if that's ok with you, I'd like to start by improving a bit the test coverage of this feature.

Could you please take a look at the very basic use cases that are described here and create some additional unit tests that would verify them? You can take a peek at CloneFixture.cs to see how we usually work with remote test repositories.

nulltoken · 2015-02-17T20:29:22Z

LibGit2Sharp.Tests/FileHistoryFixture.cs

+            return repo.Commit("Changed " + relativePath, repo.Config.BuildSignature(DateTimeOffset.Now));
+        }
+
+        protected string CreateFile(string repoPath, string relativePath, string text)


We've already got a very similar helper method named Touch(). Could you please rather leverage it?

ThomasBarnekow · 2015-02-18T11:35:50Z

@nulltoken I've just cloned your follow-test repo and quickly run the three test cases using my interactive test program (so no unit tests yet). With the exception of the first one, the results are the same.

The first one is currently not handled by my feature because it includes a rename AND a change in one commit. Thus, it only shows the top-most commit but not the second one. The question is how I can safely establish that these two commits or objects are part of the latest object's (so-renamed.txt) history? Currently, I'm only using the file name and the SHA, both of which are changed when renaming AND updating.

ThomasBarnekow · 2015-02-18T12:13:23Z

@nulltoken Another question would be whether this feature should do exactly what git log --follow <filename> does. For example, that does not show merge commits. If, however, the objective is to see all commits in which a file was indeed changed, which can include merge commits, these should be included as well.

My application, for example, would be to identify all commits in which a Word document's contents were changed and then show potentially all document revisions side by side (with track changes highlighting changes between commits). If a merge commit included manual merge activities, that would be relevant as well. git log --follow would ignore that (and my feature ignores these as well but probably shouldn't or should at least be configurable to that extent).

nulltoken · 2015-02-18T22:30:12Z

The first one is currently not handled by my feature because it includes a rename AND a change in one commit. Thus, it only shows the top-most commit but not the second one. The question is how I can safely establish that these two commits or objects are part of the latest object's (so-renamed.txt) history? Currently, I'm only using the file name and the SHA, both of which are changed when renaming AND updating.

The Diff.Compare() method could be quite handy to do this. See the DiffTreeToTreeFixture.cs suite for a better grasp. It's far possible leveraging it may be a bit more efficient than recursively harvesting the Trees

ThomasBarnekow · 2015-02-18T22:36:24Z

Thanks, that's currently the most important part for me. Does that also work for binaries? I'd be specifically interested in Word documents, which are essentially zipped collections of XML files.

ThomasBarnekow · 2015-04-13T21:21:18Z

@nulltoken Squashed all commits into one and pushed.

Quick question. I have ReSharper installed and this always updates the existing LibGit2Sharp.sln.DotSettings. I've never committed those changes. Do you have an idea why that happens? Would it be safe to add the changes? It adds the following line:

<s:Boolean x:Key="/Default/Environment/SettingsMigration/IsMigratorApplied/=JetBrains_002EReSharper_002EPsi_002ECSharp_002ECodeStyle_002ESettingsUpgrade_002EMigrateThisQualifierSettings/@EntryIndexedValue">True</s:Boolean>

FileHistory Feature

nulltoken · 2015-04-14T09:12:11Z

It's in. Thanks for this amazing job!

✨ ✨ ✨ ✨ ✨ ✨ ✨

ThomasBarnekow · 2015-04-14T09:14:50Z

Thanks! And you are most welcome.

nulltoken · 2015-04-14T17:49:58Z

@nulltoken Squashed all commits into one and pushed.

Quick question. I have ReSharper installed and this always updates the existing LibGit2Sharp.sln.DotSettings. I've never committed those changes. Do you have an idea why that happens? Would it be safe to add the changes? It adds the following line:
<s:Boolean x:Key="/Default/Environment/SettingsMigration/IsMigratorApplied/=JetBrains_002EReSharper_002EPsi_002ECSharp_002ECodeStyle_002ESettingsUpgrade_002EMigrateThisQualifierSettings/@EntryIndexedValue">True</s:Boolean>

I'm also regularly bothered by this. It's happening when a new version of Resharper is installed.

Let's merge the change it. Could you please send a PR to take care of this?

ThomasBarnekow · 2015-04-14T18:13:59Z

Sure, will do.

On 14.04.2015, at 19:50, nulltoken notifications@github.com wrote:

@nulltoken https://github.com/nulltoken Squashed all commits into one and pushed.

Quick question. I have ReSharper installed and this always updates the existing LibGit2Sharp.sln.DotSettings. I've never committed those changes. Do you have an idea why that happens? Would it be safe to add the changes? It adds the following line:

<s:Boolean x:Key="/Default/Environment/SettingsMigration/IsMigratorApplied/=JetBrains_002EReSharper_002EPsi_002ECSharp_002ECodeStyle_002ESettingsUpgrade_002EMigrateThisQualifierSettings/@EntryIndexedValue">True/s:Boolean
I'm also regularly bothered by this. It's happening when a new version of Resharper is installed http://stackoverflow.com/questions/29096939/resharper-dotsettings-file-getting-modified-every-time.

Let's merge the change it. Could you please send a PR to take care of this?

—
Reply to this email directly or view it on GitHub #963 (comment).

nulltoken · 2015-04-15T17:58:29Z

Published as NuGet pre-release package LibGit2Sharp.0.22.0-pre20150415174523

linkerro · 2015-07-09T13:43:10Z

Has the usage for this feature changed from the first comment?

quisse · 2015-07-14T08:19:07Z

@linkerro
I think the usage now is:

using(var repo = new Repository(somePath))
{
    IEnumerable<LogEntry> history = repo.Commits.QueryBy(filepath);
}

(Please correct me if I'm wrong)

linkerro · 2015-07-14T08:40:00Z

That is correct. I started reading the tests and got it working, but this should stay here for reference, if not in the wiki :)

ThomasBarnekow · 2015-07-14T11:01:44Z

@linkerro and @quisse Sorry for not commenting earlier. The usage described by @quisse is essentially correct (would have new Repository(somePath)). And the wiki should be updated with at least a short description.

quisse · 2015-07-14T11:03:32Z

@ThomasBarnekow edited

deepak-arya · 2016-07-04T08:21:22Z

Hi @ThomasBarnekow thanks this gr8 library. Unfortunately i didn't find FileHistory class. i looked into code and found this call as "internal" access. Have this class been removed from public interface? i am creating a VS addin which requires to have history of files. i would like to know if this class is still there.
Thanks...

txdv · 2016-07-04T08:49:29Z

His last comment literally points out that @quisse last comment is the current way of achieving that.

deepak-arya · 2016-07-05T06:15:38Z

yahh.. thanks @txdv . i got it. i am using below code which always return zero length LogEntry enumerator. Am i using it correct way @quisse

public static void GetFileHistory(string filepath,string repopath)
{
using (var repo = new Repository(repopath))
{
// here i am using absolute file path
IEnumerable history = repo.Commits.QueryBy(filepath);
//problem is here. it does not go into loop.
foreach(LogEntry e in history)
{
Console.WriteLine(e.Commit.Message);
}
}
}

/// here is what i get
GetEnumerator() result view is always null

txdv · 2016-07-13T17:23:52Z

Are you passing the entire path? If you have a file in a subdirectory, you need to pass Path.Combine("diectory", "file") to the QueryBy

mirsaeedi · 2018-01-05T21:49:31Z

As @deepak-arya said Repository.Commits.QueryBy always returns an empty collection. What is the resolution to that? I pass the absolute path to the method.

ethomson · 2018-01-05T23:06:32Z

@mirsaeedi Please ask a question on StackOverflow for support.

Therzok reviewed Feb 17, 2015
View reviewed changes

.nuget/NuGet.targets

@@ -0,0 +1,144 @@

<?xml version="1.0" encoding="utf-8"?>

Copy link

Member

Therzok Feb 17, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't have landed here. 🍊

Therzok reviewed Feb 17, 2015
View reviewed changes

nulltoken reviewed Feb 17, 2015
View reviewed changes

nulltoken mentioned this pull request Apr 13, 2015

Question: How to get the last commit that affected a given file #89

Closed

nulltoken added this to the v0.22 milestone Apr 13, 2015

nulltoken added a commit that referenced this pull request Apr 14, 2015

Merge pull request #963 from ThomasBarnekow/file-history

0a0dce7

FileHistory Feature

nulltoken merged commit 0a0dce7 into libgit2:vNext Apr 14, 2015

nulltoken added the nuget-pre label Apr 15, 2015

ThomasBarnekow deleted the file-history branch April 15, 2015 22:14

carlosmn mentioned this pull request Feb 15, 2016

Support "git log --path" #463

Closed

carlosmn mentioned this pull request Apr 19, 2016

Getting Started #44

Closed

This was referenced Jan 18, 2020

Repository.Commits.QueryBy is missing changes #1752

Open

Return all changes to a file when using Repository.Commits.QueryBy(path) #1753

Open

Shnatsel mentioned this pull request Jun 6, 2021

Is there any way to get last commit of a certain file? rust-lang/git2-rs#588

Open

extrawurst mentioned this pull request Jun 7, 2021

View log/history of a file extrawurst/gitui#381

Closed

gukoff mentioned this pull request May 8, 2024

KeyNotFoundException when trying to iterate through QueryBy result set. #1410

Open

FileHistory Feature #963

FileHistory Feature #963

Conversation

ThomasBarnekow commented Feb 17, 2015

Therzok Feb 17, 2015

Choose a reason for hiding this comment

ThomasBarnekow commented Feb 17, 2015

nulltoken commented Feb 17, 2015

nulltoken commented Feb 17, 2015

ThomasBarnekow commented Feb 17, 2015

nulltoken commented Feb 17, 2015

ThomasBarnekow commented Feb 17, 2015

ThomasBarnekow commented Feb 17, 2015

nulltoken commented Feb 17, 2015

nulltoken commented Feb 17, 2015

ThomasBarnekow commented Feb 17, 2015

nulltoken commented Feb 17, 2015

ThomasBarnekow commented Feb 17, 2015

nulltoken commented Feb 17, 2015

ThomasBarnekow commented Feb 17, 2015

nulltoken commented Feb 17, 2015

Therzok Feb 17, 2015

Choose a reason for hiding this comment

ThomasBarnekow Feb 17, 2015

Choose a reason for hiding this comment

nulltoken commented Feb 17, 2015

nulltoken Feb 17, 2015

Choose a reason for hiding this comment

ThomasBarnekow Feb 17, 2015

Choose a reason for hiding this comment

ThomasBarnekow commented Feb 18, 2015

ThomasBarnekow commented Feb 18, 2015

nulltoken commented Feb 18, 2015

ThomasBarnekow commented Feb 18, 2015

ThomasBarnekow commented Apr 13, 2015

nulltoken commented Apr 14, 2015

ThomasBarnekow commented Apr 14, 2015

nulltoken commented Apr 14, 2015

ThomasBarnekow commented Apr 14, 2015

nulltoken commented Apr 15, 2015

linkerro commented Jul 9, 2015

quisse commented Jul 14, 2015

linkerro commented Jul 14, 2015

ThomasBarnekow commented Jul 14, 2015

quisse commented Jul 14, 2015

deepak-arya commented Jul 4, 2016 • edited Loading

txdv commented Jul 4, 2016

deepak-arya commented Jul 5, 2016 • edited Loading

txdv commented Jul 13, 2016

mirsaeedi commented Jan 5, 2018

ethomson commented Jan 5, 2018

deepak-arya commented Jul 4, 2016 •

edited

Loading

deepak-arya commented Jul 5, 2016 •

edited

Loading