<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">tl;dr of the below: We’re migrating a large, single Subversion repository into 857 Git repositories, and it went fine with some warnings [3], but some branches of some repositories are missing some individual files, and investigating that revealed some commits have a blank diff / no changed files even though revision metadata in the commit message points to revisions containing diffs and changed files.<div class=""><br class=""></div><div class="">*Background*</div><div class="">We have a large (~30GB) single Subversion repository with about 78,000 revisions. It is not OSS/publicly available. Its contents started out as a monolith in 2003, and over the years 857 “components” were extracted from the monolith and put in their own locations in the same Subversion repository, with their own version/branch tracking. We’re beyond ready to make the switch to Git (each of the 857 components will get its own Git repository) and plan on using svn2git/svn-all-fast-export [1] (we did initial testing with three different tools and this tool was the only one that was even close to being up to the task). </div><div class=""><br class=""></div><div class="">One of the first things we did was spend about two weeks writing and running various Python scripts to help us fully understand the revision history of the repository, primarily using `svn log —stop-on-copy`. With this, we were able to figure out what mapping rules we needed to write.</div><div class=""><br class=""></div><div class="">We spent another week writing and proofreading about 7,000 lines of rules, and determined that, because some of the components were extracted from other components, we actually had to run the migration in four passes: One with the rules for 854 of the components/repositories and then three more times with the rules for the three oldest components from which any other components can trace their lineage. This is to prevent rules for some of the 854 repositories from short-circuiting the rules for those other three repositories. And, by the way, this is the full command we are running:</div><div class=""><br class=""></div><div class="">/path/to/svn-all-fast-export --debug-rules --rules /path/to/rules.config --identity-map /path/to/authors.config --identity-domain “<a href="http://example.org" class="">example.org</a>" --add-metadata --stats /path/to/svn_repo</div><div class=""><br class=""></div><div class="">We then spent the last week debugging the rules with dry-run after dry-run and then wet-run after wet-run until we had a fully-completed “dress rehearsal” of sorts, giving us all of the migrated repositories to push to Bitbucket so that we could begin analyzing the results and look for any problems before we schedule the real/final migration. And boy did we find problems…</div><div class=""><br class=""></div><div class="">The first thing we did was perform recursive diffs on some (not all) strategic repositories and their more modern branches. We still have more to inspect, but what the first four showed us set off some alarm bells. The first indicator of problems were than some individual files were missing from some branches on these repositories and we’re having a really hard time figuring out how.</div><div class=""><br class=""></div><div class="">So, here’s the list of (obscured) filenames in question from one example repository:</div><div class=""><br class=""></div><div class="">README.txt</div><div class="">Foo/<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Somefile</span></div><div class="">Foo/Bar/<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Somefile</span></div><div class="">Foo/Baz/<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Somefile</span></div><div class="">Foo/Baz/Qux/Somefile</div><div class=""><br class=""></div><div class="">In Subversion, those Somefiles exist in branches 6.0.5, 6.0.6, and 6.0.7, but they do not exist in branches 6.0.8, 6.0.9, and 6.1.0 (they were deleted in a revision in 6.0.8). In Git, those Somefiles do not exist in any of those six branches.</div><div class=""><br class=""></div><div class="">In Subversion, README.txt exists in all six branches. In Git, README.txt exists only in 6.1.0 and does not exist in any of the other five branches.</div><div class=""><br class=""></div><div class="">Those are the only differences. All other files exist and all file contents match.</div><div class=""><br class=""></div><div class="">In trying to track down the issue, the first thing we discovered is that there was a revision about 11 years ago in branch 6.0.2 that did a *replacement” of README.txt. It wasn’t an M (modification), and it wasn’t a D (delete) in one revision followed by an A (add) in another revision. It was an R (replacement) in a single revision. So looking at even older branches in Git, README.txt exists in 6.0.1 through 6.0.4 and disappears in 6.0.5 only to re-appear in 6.1.0. But that file has never been deleted in Subversion in any of those branches, yet it’s mysteriously missing in those branches on Git. We tried searching Git history for the deleting commit using every single technique suggested on this StackOverflow question [2], but none of them gave us any results.</div><div class=""><br class=""></div><div class="">So we moved on to tracking down the issue with the four Somefiles. Grep-ing the svn2git gitlog file for the repository yielded the following (obscured) commit:</div><div class=""><br class=""></div><div class="">commit refs/heads/main-6.0.8<br class="">mark :15516<br class="">committer Some Person <<a href="mailto:some.person@example.org" class="">some.person@example.org</a>> 1624637813 +0000<br class="">data 96<br class="">MW-0 Prune cruft.<br class=""><br class="">svn path=/all/source/Library/ComponentName/trunk-6.0.8/; revision=73604<br class=""><br class="">D Foo/Somefile<br class="">D Foo/Bar/Somefile<br class="">D Foo/Baz/Somefile<br class="">D Foo/Baz/Qux/Somefile<br class=""><br class="">progress SVN r73604 branch main-6.0.8 = :15516</div><div class=""><br class=""></div><div class="">This wasn’t a surprise. We knew these files had been deleted in branch 6.0.8, so it made sense to find that in the gitlog. A sanity check `svn log -r` confirms the committer, message, and deleted files match the gitlog. But, again, all of the techniques for finding when a file was deleted in Git failed, even on branch 6.0.8 where we knew it had been deleted. So we used `git log --grep=73604` to find the commit with that revision metadata, and then `git show` for that commit and … nothing. The commit diff is blank, the commit stats show no files changed. (Which, I’m sure, is why the git commits for finding when a file was deleted weren’t working.)</div><div class=""><br class=""></div><div class="">So we're at a point where we don’t know where else to look or what else to try. Keep in mind that it takes several hours to run the migration again, so if you have several suggestions that I can reasonably try simultaneously, that could ease things along.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Nick Williams</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">[1] <a href="https://github.com/svn-all-fast-export/svn2git" class="">https://github.com/svn-all-fast-export/svn2git</a></div><div class="">[2] <a href="https://stackoverflow.com/questions/6839398/find-when-a-file-was-deleted-in-git" class="">https://stackoverflow.com/questions/6839398/find-when-a-file-was-deleted-in-git</a></div><div class=""><br class=""></div><div class="">[3] There were some warnings during the migration. The vast majority looked like this:</div><div class=""><br class=""></div><div class="">WARN: Branch “foo" in repository “bar" doesn't exist at revision 12345 -- did you resume from the wrong revision?</div><div class=""><br class=""></div><div class="">But we were not stopping/resuming. I’m unsure if this could be a problem.</div><div class=""><br class=""></div><div class="">A couple other warnings I saw:</div><div class=""><br class=""></div><div class="">WARN: repository foo branch bar has some files copied from baz@12345</div><div class="">WARN: backing up branch</div><div class=""><br class=""></div></body></html>