Repository Editing and Conversion With Reposurgeon

Reposurgeon is more difficult to use than any of dozens of fully-automated conversion tools out there; you have to make choices and compose a recipe. This section explains why it’s worth the bother.

In brief, it’s because fully-automated converters don’t work very well. They are very poor at dealing with the ontological mismatches between the data models of different version-control systems. For detailed discussion of the technical flaws in many common converters, see Appendix A.

In particular: reposurgeon is the only conversion tool that handles multi-branch Subversion repositories in full generality. It can even correctly translate Subversion commits that alter multiple branches.

But even automated converters that are relatively good at bridging data-model differences tend to produce crude, jackleg, unidiomatic conversions that make the seam between the pre-conversion and post-conversion parts of the repository very obvious.

A central example of this is commit references in change comments. These references convey important information to anyone reading the comments, and it is correspondingly important to change them from using the reference format of the old system to one that is intelligible in whatever your new one is.

As another example, git has a convention about the form of change comments; they’re supposed to consist of a standalone summary line, followed optionally by a spacer blank line and running text. Git relies on this convention to produce log summaries that are easy to take in at a glance.

Older version-control systems don’t have this convention. An ideal conversion changes as many comments as possible to be in Git-like form so that the Git summary tools see the data regularity they want. But this kind of editing can’t be fully automated. The best you can hope for, if you want to do it right, is that your tool automates as much of this fixup as it can and it assists a human operator in applying fixups.

Neither reference-lifting nor patching comments for Git-friendliness is a process that can be fully automated. Both require human judgment; accordingly, fully-automated converters don’t even try to do the right thing. The result is often a history that is full of unpleasant little speedbumps and distractions. These induce wasted developer effort and, correspondingly, higher defect rates.

On the other hand, a skilled operator of reposurgeon can produce a conversion that is fully idiomatic in the target system, significantly lowering future friction costs for developers browsing the history.

One fully automated reposurgeon feature of some significance that no other importer supports is that it can parse ChangeLogs in histories which use that Free Software Foundation convention, and use the attributions in them to fill in Git author fields. This recovers better information about the provenance of changesets corresponding to patches committed by a project developer (who continues to be recorded as the committer of that changeset).

If you are an organization that pays programmers and has a requirement to do a repository conversion, the author can be engaged to perform or assist with the transition. You are likely to find this is more efficient than paying someone in-house for the time required to learn the tools and procedures. I (the author) have been very open about my methods here, but nothing substitutes for experience when you need it done quickly and right.

If you are wondering why it’s worth spending any money at all for a real history conversion, as opposed to just starting a new repository with a snapshot of the old head revision, the answer comes down to two words: risk management.

Suppose you do a snapshot conversion, head revision only. Then you get a regression report with a way to reproduce the problem. What you want to do is bisect in the new history to identify the revision where the bug was introduced, because knowing what the breaking change was makes a fix far easier. Bzzzt! You can’t. That history is missing in the new system.

Yes, in theory you could run a manual bisection using bracketing builds in new and old repositories. Until you have tried this, you will have no comprehension of how easy it is to get that process slightly but fatally wrong, and (actually more importantly) how difficult it is to be sure you haven’t gotten it wrong. This is the kind of friction cost that sounds minor until the first time it blows up on you and eats man-weeks of NRE.

So congratulations, tracing the bug just got an order of magnitude more expensive in engineer time, and your expected time to fix changed proportionally. It typically only takes one of these incidents to justify the up-front cost of having had the conversion done right.

If you go the snapshot-conversion route, maybe you’ll get lucky and never need visibility further back. Or maybe you’ll have a disaster because you increased the friction costs of debugging just enough that you, say, miss a critical ship date. The more experienced with in-the-trenches software development you are, the more plausible that second scenario will sound.

A subtler issue is that by losing the old change comments you have thrown away a great deal of hard-won knowledge about why your code is written the way it is. Again, this may never matter – but if it does, it’s going to bite you on the butt, hard, probably when you least expect it.

And if you’re thinking "No problem, the old repository will still be around"…​heh. Repositories that have become seldom-accessed are like other kinds of dead storage in that they have a way of quietly disappearing because after a few job turnovers the knowledge of why they’re important is lost. Typically you don’t find out this has happened until you have an unanticipated urgent need, at which point whatever trouble you were in gets deeper.

Spending the relatively small amount it takes to have a proper full conversion done right is a way of bounding your downside risk. If you aren’t a software engineer and had trouble following the preceding argument, propose a snapshot conversion to the engineer you trust the most and watch that person reaching for a diplomatic way to tell you it’s a stupid, shortsighted idea.

Make sure the tools in the reposurgeon suite (especially reposurgeon and repotool) are on your $PATH.

Create a scratch directory for your conversion work.

Run "repotool initialize" in the scratch directory; it requires that you follow the initialize verb with a project name. This will create a Makefile designed to sequence your conversion, and an empty lift script. Then set the variables near the top appropriately for your project.

This Makefile will help you avoid typing a lot of fiddly commands by hand, and ensure that later products of the conversion pipeline are always updated when earlier ones have been modified or removed.

The most important variables to set in the Makefile are the ones that set up local mirroring of your repository. The repotool command has a mode that handles the details of making (and, when necessary, updating) a local mirror. To enable this you need to fill in either REMOTE_URL or CVS_HOST and CVS_MODULE; read the header comment of the conversion makefile for details.

If you’re lifting a Subversion repository you can specify the repository URL in either of two ways: as a standard Subversion repository URL (service prefix "svn:") or as an rsync URL pointing at the same repository master directory (service prefix "rsync:"). Usually rsync mirroring is faster, but it depends on sshd running at the server end and may being in complications around its security features. Use a "svn:" URL, which uses mirroring by svnsync, when you can’t make rsync work.

Note by the way that repotool mirror’s "rsync:" URLs do not have the requirement for a remote rsyncd that an rsync URL fed directly to rsync itself does; internally, repotool turns them into a single-colon host plus path rsync source specification.

If your first access attempt failed with ssome message abourt access control, you may need to uncomment the RUSERNAME and RPASSWORD and set them to a valid credential pair.

Later, you will put your custom commands in the lift script file. Doing this helps you not lose older steps as you experiment with newer ones, and it documents what you did.

Doing a high-quality repository conversion is not a simple job, and the odds that you will get it perfectly right the first time are close to zero. By packaging your lift commands in a repeatable script and using the Makefile to sequence repetitive operations, you will reduce the overhead of experimenting.

In the rest of the steps we describe below, when we write "make foo" that means the step can be sequenced by the "foo" production in the Makefile. Replace $(PROJECT) in these instructions with your project name.

You may find it instructive to type "make -n" to see what the entire conversion sequence will look like.

Subversion and CVS identify users by a Unix login name local to the repository host; DVCSes use pairs of fullnames and email addresses. Before you can finish your conversion, you’ll need to put together an author map that maps the former to the latter; the Makefile assumes this is named $(PROJECT).map. The author map should specify a full name and email address for each local user ID in the repo you’re converting. Each line should be in the following form:

You can optionally specify a third field that is a timezone description, either an ISO8601 offset (like "-0500") or a named entry in the Unix timezone file (like "America/Chicago"). If you do, this timezone will be attached to the timestamps on commits made by this person.

Using the generic Makefile for Subversion, "make stubmap" will generate a start on an author-map file as $(PROJECT).map. Edit in real names and addresses (and optionally offsets) to the right of the equals signs.

How best to get this information will vary depending on your situation.

If you are converting the repository for an open-source project, it is good courtesy and good practice after the above first step to email the contributors and invite them to supply a preferred form of their name, a preferred email address to be used in the mapping, and a timezone offset. The reason for this is that some sites, like OpenHub, aggregate participation statistics (and thus, reputation) across many projects, using developer name and email address as a primary key.

Your authors file does not have to be final until you ship your converted repo, so you can chase down authors' preferred identifications in parallel with the rest of the work.

Install whatever front end reposurgeon needs to read your repository. That will be cvs-fast-export for CVS, or the VCS tool itself for Subversion and other systems.

The generic-workflow Makefile will call reposurgeon for you, interpreting your $(PROJECT).lift file, when you type "make". You may have to watch the baton spin for a few minutes. For very large repositories it could be more than a few minutes.

This will convert your repository to git. If you need to export to something else, reposurgeon has write support for a couple of other modern VCSes.

Before putting in further effort on polishing your conversion and putting it into production, you should check it for basic correctness.

Pay attention to error messages emitted during the lift. Most of these, and remedial actions to take, are described in this guide.

For Subversion lifts, use the "headcompare", "tagscompare" and "branchescompare" productions to compare the converted with the unconverted repository. If you didn’t use the cvsconvert wrapper for your CVS lift, these productions have a similar effect. Be aware that these operations may be extremely slow on large Subversion repositories.

The only differences you should see are those due to keyword expansion and ignore-file lifting. If this is not true, you may have found a serious bug in either reposurgeon or the front end it used, or you might just have a misassigned tag that can be manually fixed. Consult How to report bugs for information on how to usefully report bugs.

Use reposurgeon’s ‘lint’ command to find anomalies like detached branches that may need manual correction.

If you are converting from CVS, use reposurgeon’s ‘graph’ command to examine the conversion, looking (in particular) for misplaced tags or branch joins. Often these can be manually repaired with little effort. These flaws do 'not' necessarily imply bugs in cvs-fast-export or reposurgeon; they may simply indicate previously undetected malformations in the history. However, reporting them may help improve cvs-fast-export.

Warning: As of September 2016, stock CVS is known buggy in ways which may affect checking the correctness of conversions. For best results, use a CVS version with the MirOS patches. These are carried by Debian Linux and derivatives; you can check by Looking for "MirDebian" in the output of cvs --version.

You should now have a git repository, but it is likely to have a lot of cruft and conversion artifacts in it. Here are some common forms of cruft:

Here’s a checklist of cleanup steps. If you’re using the makefile generated by repotool, most of these will be done by commands in your lift script.

Most of the work will be in the comment-fixup and reference-lifting stages. These normally take only a couple of hours even on very large repos with thousands of commits. An entire conversion is usually less than two days of work.

You can use the authors read command to perform the author-ID mapping operation with reposurgeon.

You can find empty commits as the =Z visibility set set and clean them up with the command tagify. Consult the reposurgeon manual page for usage details.

If your repository is small, a good way to spot junk commits is to eyeball the picture of the commit DAG created by the reposurgeon 'graph' command - they tend to stand out visually as leaf nodes in odd places. Be aware that the graph command outputs DOT, the language interpreted by the graphviz suite; you will need a DOT rendering program and an image viewer. Unfortunately, for large repositoris the sheer size of the graph image makes this imoractical.

See the documentation of the references command for details on how fix up Subversion and CVS changeset references in comments so they’re still meaningful.

The command =L msgout is good for extracting commits with multi-line comments in a form that can be deleted and merged back in with the msgin command.

The reposurgeon command =H inspect will show you tip commits which may contain only deletes and deletealls.

Tags can be inspected with =T inspect. Junk tags can be removed with the delete command. Tag comments can be modified with a msgin/msgout sequence. Check that the creation date of tags matches what you see in the source repository; this is the easiest way to spot when one has been attached to the wrong commit, something that can be manually fixed by editing its "from" field.

The command =I will select all commits with attributions and comments that don’t decode to UTF-8 in both the commit comment and attribution parts. You can eyeball those to figure out what the encoding is and apply the transcode command to fix things.

Reposurgeon has a merge command specifically for performing branch merges. The msgin command will also allow you to add a parent mark to a commit.

One minor feature you lose in moving from SCCS, CVS, Subversion, or BitKeeper to a DVCS is keyword expansion. You should go through the last revision of the code and remove $Id$, $Date$, $Revision$, and other keyword cookies lest they become unhelpful fossils. The full Subversion set is $Date$, $Revision$, $Author$, $HeadURL$ and $Id$. CVS uses $Author$, $Date$, $Header$, $Id$, $Log$, $Revision$, also (rarely) $Locker$, $Name$, $RCSfile$, $Source$, and $State$. A command like grep -R '$[A-Z]' . may be helpful.

After conversion of a branchy repository, look to see if there is a 'root' branch. If there are any commits with a sufficiently pathological structure that reposurgeon can’t figure out what branch they belong to, they’ll wind up there.

It’s good practice to leave an annotated tag at the conversion point noting the date and time of the repo lift. See the next section on conversion comments for discussion. Here’s an example of how to make a tag:

Experiments with reposurgeon suggest that git import doesn’t try to pack or otherwise optimize for space when it populates a repo from a dump file; this produces large repositories. Running git repack and git gc --aggressive can slim them down quite a lot.

Developers who are already git fans and know how to use a git client will, of course, have no particular trouble using a git repository.

Windows users accustomed to working through TortoiseSVN can move to TortoiseGIT.

Developers who like hg can use the hg-git mercurial plugin. There is an Ubuntu package "mercurial-git" for this, and other distributions are likely to carry it as well. It installs a Mercurial plugin called hg-git.

There are some hg-git limitations to be aware of. In order to simulate git behavior, hg-git keeps some local state in the .hg directories; a map from git branch names to Mercurial commits, a list of Mercurial bookmarks describing git branches (which have bookmark-like behavior different from a Mercurial named branch) and a file mapping git SHA1 hashes to hg SHA1 hashes (both systems use them as commit IDs). The problem is that hg doesn’t copy any of this local state when it clones a repo, so clones of hg-git repos lose their git branches and tags.

If you have developers attached to the CVS interface, it is possible (and in fact relatively easy) to set up a gateway interface that lets them continue using their CVS client tools. Consult the documentation for git-cvsserver.

Educate your developers in the following good practices:

As the quick-start example shows, you’re typically going to do three steps when you use reposurgeon: (1) read in one (or more) repositories, (2) do surgical things on them, and (3) write out one (or more) repositories.

To keep reposurgeon simple and flexible, it normally does not do its own repository reading and writing. Instead, it relies on being able to parse and emit the command streams created by git-fast-export and read by git-fast-import. This means that it can be used on any version-control system that has both fast-export and fast-import utilities. The git-import stream format also implicitly defines a common language of primitive operations for reposurgeon to speak.

In order to deal with version-control systems that do not have fast-export equivalents, reposurgeon can also host extractor code that reads repositories directly. For each version-control system supported through an extractor, reposurgeon uses a small amount of knowledge about the system’s command-line tools to (in effect) replay repository history into an input stream internally. Repositories under systems supported through extractors can be read by reposurgeon, but not modified by it. In particular, reposurgeon can be used to move a repository history from any VCS supported by an extractor to any VCS supported by a normal importer/exporter pair.

Mercurial repository reading is implemented with an extractor class; writing is handled with the "hg-git-fast-import" command. A test extractor exists for git, but is normally disabled in favor of the regular exporter.

Subversion is an important exception. Its exporter is ‘svnadmin dump’, which doesn’t ship a git-fast-import stream, but rather the unique dump format supported by Subversion. Reposurgeon contains an interpreter for this stream format.

As a matter of historical interest, some old versions of reposurgeon had the ability to build a Subversion repository on output by synthesizing a Subversion dump stream and feeding it to ‘svnadmin load’. This feature was a cute stunt, but was abandoned during translation to Go for a couple of reasons. Most importantly, there is zero demand for moving histories to Subversion - and supposing there were, moving content and metadata from git’s DAG representation to a Subversion stream is very lossy. Subversion to Git to Subversion wouldn’t even have round-tripped well.

Between reads and writes, reposurgeon can usefully be thought of as a structure editor for directed acyclic graphs with a pre-defined set of attributes on their nodes.

To get a feel for what that graph is like, it’s helpful to have seen a git-fast-import stream file. Here is a trivial example from the reposurgeon test suite, describing a history with two commits to a single file:

A git-fast-import stream consists of a sequence of commands which must be executed in the specified sequence to build the repo; to avoid confusion with reposurgeon commands we will refer to the stream commands as events in this documentation. These events are implicitly numbered from 1 upwards. Most commands require specifying a selection of event sequence numbers so reposurgeon will know which events to modify or delete.

For all the details of event types and semantics, see the git-fast-import(1) manual page; the rest of this paragraph is a quick start for the impatient. The most prominent events in a stream are commits describing revision states of the repository; these group together under a single change comment one or more fileops (file operations), which usually point to blobs that are revision states of individual files. A fileop may also be a delete operation indicating that a specified previously-existing file was deleted as part of the commit; there are a couple of other special fileop types of lesser importance.

Reposurgeon’s internal representation of a repository history is basically a deserialized git fast-import stream. A few extra attributes are supported; most notably, commits and resets have a legacy-id attribute that carries over the object’s ID from whatever version-control system exported the stream, in particular a Subversion or CVS revision number.

The program can be run in one of two modes, either as an interactive command interpreter or in batch mode to execute commands given as arguments on the reposurgeon invocation line.

The only differences between these modes are (1) the interactive one begins by turning on the ‘interactive’ option, (2) in batch mode all errors (including normally-recoverable errors in selection-set syntax) are fatal, and (3) each command-line argument beginning with ‘--’ has that stripped off (which in particular means that --help and --version will work as expected).

Also, in interactive mode, Ctrl-P and Ctrl-N will be available to scroll through your command history, and tab completion of both command keywords and name arguments (wherever that makes semantic sense) is available.

It is expected that interactive mode will be used mainly for exploring repository metadata, while conversion experiments will be captured in a script that is gradually improved until the day final cutover can be performed and the old repository decommissioned.

Note that this means the old repository can be left in service while the conversion recipe is under development. Recipe development should be treated as a serious project with its own change tracking.

To give you the flavor of the command language, here are some simple examples:

Each command description begins with a syntax summary. Mandatory parts are in {}, optional in [], and …​ says the element just before it may be repeated. Parts in ALL-CAPS are expected to be filled in by the user.

Commands are distinguished by a command keyword. Most take a selection set immediately before it; see "help selection" for details. Some commands have a following subcommand verb. Many commands take additional modifier arguments after the command keyword(s).

The syntax of following arguments is variable according to the requirements of individual commands, but there are a couple of general rules.

When a command changes repository state, it will usually so indicate in a response.

Help is always available.

Command history is always available.

You don’t need to exit the interpreter to run quick shell commands.

Here’s how to bail out:

Regular expressions are an important building block of the command language, used both in event selections and various command arguments.

The pattern expressions used in event selections and various commands (attribution, expunge, filter, msgout, path) are those of the Go language, with one exception. Due to a conflict with the use of $ for arguments in the "script" command, we retain Python’s use of backslashes as a leader for references to group matches.

Normally patterns intended to be interpreted as regular expressions are wrapped in slashes (e.g. /foobar/ matches any text containing the string "foobar"), but any punctuation character other than single quote will work as a delimiter in place of the /; this makes it easier to use an actual / in patterns. Matched single quote delimiters mean the literal should be interpreted as plain text, suppressing interpretation of regexp special characters and requiring an anchored, entire match.

Pattern expressions following the command verb may not contain literal whitespace; use \s or \t if you need to. Event-selection regexps may contain literal whitespace.

Regular expressions are not anchored. Use ^ and $ to anchor them to the beginning or end of the search space, when appropriate.

Some commands support regular expression flags, and some even add additional flags over the standard set. The documentation for each individual command will include these details.

Commands to reposurgeon consist of a command keyword, usually preceded by a selection set, sometimes followed by whitespace-separated arguments. It is often possible to omit the selection-set argument and have it default to something reasonable. For commands that are considered safe (no side effects) the default is all events; for risky commands the default is no events.

A selection set is ordered; that is, any given element may occur only once, and the set is ordered by when its members were first added.

The selection-set specification syntax is an expression-oriented minilanguage. The most basic term in this language is a location. The following sorts of primitive locations are supported:

These may be grouped into sets in the following ways:

There are some other ways to construct event sets:

A quick example-centered reference for selection-set syntax.

First, these ways of constructing singleton sets:

You can select commits and tags by date, or by date and committer:

More ways to construct event sets:

Other special functions are available: do 'help functions' for more.

You can compose sets as follows:

Selection sets are ordered: elements remain in the order they were added, unless sorted by the ? suffix.

Sets can be composed with | (union) and & (intersection). | has lower precedence than &, but set expressions can be grouped with ( ). Postfixing a ? to a selection expression widens it to include all immediate neighbors of the selection and sorts it; you can do this repeatedly for effect. Do set negation with prefix ~; it has higher precedence than & | but lower than ?.

The selection-expression language has named special functions. The syntax for a named function is "@" followed by a function name, followed by an argument in parentheses. Presently the following functions are defined:

Set expressions may be combined with the operators ‘|’ and ‘&’ which are, respectively, set union and intersection. The | has lower precedence than intersection, but you may use parentheses ‘(’ and ‘)’ to group expressions in case there is ambiguity.

Any set operation may be followed by ‘?’ to add the set members' neighbors and referents. This extends the set to include the parents and children of all commits in the set, and the referents of any tags and resets in the set. Each blob reference in the set is replaced by all commit events that refer to it. The ? can be repeated to extend the neighborhood depth. The result of a ? extension is sorted so the result is in ascending order.

Do set negation with prefix ‘~’; it has higher precedence than & and | but lower than ?.

An optional command argument prefixed by "<" indicates that the command accepts input redirection; an optional argument prefixed by ">" indicates that the command accepts output redirection. There must be whitespace before the "<" or ">" so that the command parser won’t falsely match uses of these characters in regular expressions.

Commands that support output redirection can also be followed by a pipe bar and a normal Unix command. For example, "list | more" directs the output of a list command to more(1). Some whitespace around the pipe bar is required to distinguish it from uses of the same character as the alternation operator in regular expressions.

The command line following the first pipe bar, if present, is passed to a shell and may contain a general shell command line, including more pipe bars. The SHELL environment variable can set the shell, falling back to /bin/sh.

Beware that while the reposurgeon CLI mimics these simple shell features, many things you can do in a real shell won’t work until the right-hand side of a pipe-bar output redirection, if there is one. String-quoting arguments will fail unless the specific, documented syntax of a command supports that. You can’t redirect standard error (but see the "log" command for a rough equivalent). And you can’t pipe input from a shell command.

In general you should avoid trying to get cute with the command parser. It’s stupider than it looks.

Each event has an associated "Q" bit that says whether it was touched by the last Q-aware command. Almost all commands that modify the history set Q bits in some interesting way; so do some report generators, such as the "lint" command.

Thus, "=Q count" and "=Q list" can provide you with a check on the scope of modifications while you are trying things out. Or you can use "=Q list" and "=Q inspect" to zero in on anomalies reported by lint.

reposurgeon can rebuild an altered repository in place. Untracked files are normally saved and restored when the contents of the new repository are checked out (but see the documentation of the ‘preserve’ command for a caveat).

This section will become relevant only if reposurgeon or something underneath it in the software and hardware stack crashes while in the middle of writing out a repository, in particular if the target directory of the rebuild is your current directory.

The tool has two conflicting objectives. On the one hand, we never want to risk clobbering a pre-existing repo. On the other hand, we want to be able to run this tool in a directory with a repo and modify it in place.

We resolve this dilemma by playing a game of three-directory monte.

During the critical region, all signals that can be ignored are ignored.

When the repository type you are working with has a "lister" method, it can tell which files in a repository directory are not checked in and will copy them into the edited repository made by a rebuild.

The following commands are required only if there is no lister method and you have to set preservations by hand. Under systems with such a command (which include git and hg), all files that are neither beneath the repository dot directory nor under reposurgeon temporary directories are preserved automatically.

When converting a legacy repository, it sometimes happens that there are archived releases of the project surviving from before the date of the repository’s initial commit. It may be desirable to insert those releases at the front of the repository history. Do do this, use this command:

Reposurgeon can have several repositories loaded at once. The following commands operate on the repository list.

Modifying a repository so every commit in it has a unique timestamp is often a useful thing to do, in order for every commit to have a unique action stamp that can be referred to in surgical commands.

The ‘lint’ command will tell you if you have timestamp collisions.

Those of you twitchy about "rewriting history" should bear in mind that the commit stamps in many older repositories were never very reliable to begin with.

+ CVS in particular is notorious for shipping client-side timestamps with timezone and DST issues (as opposed to UTC) that don’t necessary compare well with stamps from different clients of the same CVS server. Thus, inducing a timequake in a CVS repo seldom produces effects anywhere near as large as the measurement noise of the repository’s own timestamps.

+ Subversion was somewhat better about this, as commits were stamped at the server, but older Subversion repositories often have sections that predate the era of ubiquitous NTP time.

Some commands automate fixing various kinds of artifacts associated with repository conversions from older systems.

reposurgeon recognizes how supported VCSes represent file ignores (CVS .cvsignore files lurking untranslated in older Subversion repositories, Subversion ignore properties, .gitignore/.hgignore/.bzrignore/.brzignore file in other systems) and moves ignore declarations among these containers on repo input and output. This will be sufficient if the ignore patterns are exact filenames.

Translation may not, however, be perfect when the ignore patterns are Unix glob patterns or regular expressions. This compatibility table describes which patterns will translate; "plain" indicates a plain filename with no glob or regexp syntax or negation, "no !" means no negated regexps, and "no RE:" means the RE prefix for a regular expression does not work.

RCS has no ignore files or patterns and is therefore not included in the table.

For bzr (Breezy) see the bzr entries.

The hg rows and columns of the table describe compatibility to hg’s glob syntax rather than its default regular-expression syntax. When writing to an hg repository from any other kind, reposurgeon prepends to the output .hgignore a ‘syntax: glob’ line.

For dealing with unusual cases, there’s this:

This group of commands is meant for fixing up references in commits that are in the format of older version-control systems. The general workflow is this: first, go over the comment history and change all old-fashioned commit references into machine-parseable cookies. Then, automatically turn the machine-parseable cookie into action stamps. The point of dividing the process this way is that the first part is hard for a machine to get right, while the second part is prone to errors when a human does it.

A Subversion cookie is a comment substring of the form ‘[[SVN:ddddd]]’ (example: ‘[[SVN:2355]]’) with the revision read directly via the Subversion exporter, deduced from git-svn metadata, or matching a $Revision$ header embedded in blob data for the filename.

A CVS cookie is a comment substring of the form ‘[[CVS:filename:revision]]’ (example: ‘[[CVS:src/README:1.23]]’) with the revision matching a CVS $Id$ or $Revision$ header embedded in blob data for the filename.

A mark cookie is of the form ‘[[:dddd]]’ and is simply a reference to the specified mark. You may want to hand-patch this in when one of the previous forms is inconvenient.

An action stamp is an RFC3339 timestamp, followed by a ‘!’, followed by an author email address (author is preferred rather than committer because that timestamp is not changed when a patch is replayed on to a branch, but the code to make a stamp for a commit will fall back to the committer if no author field is present). It attempts to refer to a commit without being VCS-specific. Thus, instead of “commit 304a53c2” or “r2355”, “2011-10-25T15:11:09Z!fred@foonly.com”.

The following git aliases allow git to work directly with action stamps. Append it to your ~/.gitconfig; if you already have an [alias] section, leave off the first line.

There is a rare case in which an action stamp will not refer uniquely to one commit. It is theoretically possible that the same author might check in revisions on different branches within the one-second resolution of the timestamps in a fast-import stream. There is nothing to be done about this; tools using action stamps need to be aware of the possibility and throw a warning when it occurs.

In order to support reference lifting, reposurgeon internally builds a legacy-reference map that associates revision identifiers in older version-control systems with commits. The contents of this map come from three places: (1) cvs2svn:rev properties if the repository was read from a Subversion dump stream, (2) $Id$ and $Revision$ headers in repository files, and (3) the .git/cvs-revisions created by ‘git cvsimport’.

The detailed sequence for lifting possible references is this: first, find possible CVS and Subversion references with the references or =N visibility set; then replace them with equivalent cookies; then run references lift to turn the cookies into action stamps (using the information in the legacy-reference map) without having to do the lookup by hand.

It is not guaranteed that every such reference will be resolved, or even that any at all will be. Normally all references in history from a Subversion repository will resolve, but CVS references are less likely to be resolvable.

A legacy-reference file maps reference cookies to (committer, commit-date, sequence-number) triplets; these in turn (should) uniquely identify a commit. The format is two whitespace-separated fields: the cookie followed by an action stamp identifying the commit.

+ It should not normally be necessary to use this command. The legacy map is automatically preserved through repository reads and rebuilds, being stored in the file legacy-map under the repository subdirectory.

CVS, Subversion and Mercurial do not have separated notions of committer and author for changesets; when lifted to a VCS that does, like git, their one author field is used for both.

However, if the project used the FSF ChangeLog convention, many changesets will include a ChangeLog modification listing an author for the commit. In the common case that the changeset was derived from a patch and committed by a project maintainer, but the ChangeLog entry names the actual author, this information can be recovered.

The Co-Author convention described in the Linux kernel’s co-author message conventions is observed: If an attribution header is followed by a whitespace-led line containing only a valid email address, that name becomes the payload of a "Co-Author" header that is appended to the change comment for the containing commit.

The command reports statistics on how many commits were altered.

When lifting a history from a version-control system that lacks changesets, it is useful to have a way to recognize cliques of per-file changes that ought to be grouped into changesets.

You won’t need this for CVS because cvs-fast-export does clique coalescence itself.

Occasionally you will need to issue a large number of complex surgical commands of very similar form, and it’s convenient to be able to package that form so you don’t need to do a lot of error-prone typing. For those occasions, reposurgeon supports simple forms of named variables and macro expansion.

Here are some more advanced examples of scripting:

List the last commit that refers to a ChangeLog file containing a specified string. (The trick here is that ? extends the singleton set consisting of the last eligible ChangeLog blob to its set of referring commits, and list only notices the commits.)

Mass-rename tags, replacing "sample" on the basename with "example". Illustrates a general technique of generating reposurgeon commands via shell that you then execute with the ‘script’ command. Enabling this technique is the reason as many commands as possible support redirects.

These commands control the level and destination of diagnostic logging.

A few commands have been implemented primarily for debugging and regression-testing purposes, but may be useful in unusual circumstances.

The hg extractor does not attempt to recursively handle subrepos. Rather, it will extract the history of the top-level repo, in which .hgsub and .hgsubstate will be treated as regular files. If you wish to translate these into the semantics of your target VCS, you will need to do so with surgical primitives after reading the history into reposurgeon.

Note that the Subversion dump reader only supports versions 1 and 2 of the dump file format, not version 3 with diff-based file changes. This shouldn’t be a problem with normal use of reposurgeon, which calls svnadmin dump in its default mode generating version 2.

If a stream file has gaps in the revision sequence, branch copy references to nonexistent revisions will be patched to refer to the most recent existing revision before the missing one.

Certain optional modifiers on the read command change its behavior when reading Subversion repositories:

These modifiers can go anywhere in any order on the command line after the read verb. They must be whitespace-separated.

As stacking up read options can result in a very long read invocation line, it’s useful to know that backslash is accepted as a continuation character.

It is also possible to embed a magic comment in a Subversion stream file to set these options (this is mainly useful in test loads). Prefix a space-separated list of them with the magic comment ‘ # reposurgeon-read-options:’; the leading space is required. This may be useful when synthesizing test loads; in particular, a stream file that does not set up a standard trunk/branches/tags directory layout can use this to perform a mapping of all commits onto the master branch that the git importer will accept.

Here are the rules used for mapping subdirectories in a Subversion repository to branches:

Branch-creation operations with no following commits are usually tagified. However, this is done to preserve comment/committer data entered by users; when reposurgeon can detect that a branch-creation comment was automatically generated (as often happens in cvs2svn conversions) the commit will simply be discarded so as not to create clutter that has to be manually removed by the operator. (That discard action is prevented by the --preserve option.)

Otherwise, each commit that only creates or deletes directories (in particular, copy commits for tags and branches, and commits that only change properties) will be transformed into a tag named after the tag or branch, containing the date/author/comment metadata from the commit.

Subversion branch deletions are turned into deletealls, clearing the fileset of the import-stream branch. When a branch finishes with a deleteall at its tip, the deleteall is transformed into a tag. This rule cleans up after aborted branch renames.

Occasionally (and usually by mistake) a branchy Subversion repository will contain revisions that touch multiple branches. These are handled by partitioning them into multiple import-stream commits, one on each affected branch. The Legacy-ID of such a split commit will have a 1-origin split number separated by a dash - for example, if Subversion revision 2317 touches three branches, the three generated commits will have IDs 2317-1, 2317-2, and 2317-3.

The svn:executable and svn:special properties are translated into permission settings in the input stream; svn:executable becomes 100755 and svn:special becomes 120000 (indicating a symlink; the blob contents will be the path to which the symlink should resolve).

Any cvs2svn:rev properties generated by cvs2svn are incorporated into the internal map used for reference-lifting, then discarded.

Normally, per-directory svn:ignore properties (and svn:global-ignores properties, e.g. in a site configuration file) become .gitignore files. Actual .gitignore files in a Subversion directory are presumed to have been created by git-svn users separately from native Subversion ignore properties and discarded with a warning. It is up to the user to merge the content of such files into the target repository by hand. But this behavior is changed by the --user-ignores option which disables filtering of in-tree .gitignore files and instead merges them with .gitignore files generated from Subversion properties. On the other hand, the --no-automatic-ignores option discards Subversion svn:ignore and svn:global-ignores properties without translation.

Normally, .cvsignore files left over from a Subversion repository’s ancient history as a CVS repository are deleted. The assumption is that the repository users want the (presumably more up-to-date) Subversion ignore properties to be translated. However, this deletion can be prevented with the --cvsignores read option.

svn:mergeinfo properties are interpreted. Any svn:mergeinfo property on a revision A with a merge source containing all revisions on a branch from the forking point (or the branch start if the histories are independent) up to revision B produces a merge link such that the branch tip at revision B becomes a parent of A. The "svnmerge-integrated" properties produced by Subversion’s svnmerge.py script are handled the same way.

All other Subversion properties are discarded. (This may change in a future release.) The property for which this is most likely to cause semantic problems is svn:eol-style. However, since property-change-only commits get turned into annotated tags, the translated tags will retain information about setting changes.

The sub-second resolution on Subversion commit dates is discarded; Git wants integer timestamps only. Normally Subversion timestamps are rounded down, but when two adjacent timestamps have the same seconds part and the later one is in the top half-second of the interval, the later one is rounded up instead. This does much to reduce collisions while guaranteeing that no timestamp is ever shifted to a non-adjacent second mark.

Because fast-import format cannot represent an empty directory, empty directories in Subversion repositories will be lost in translation.

Normally, Subversion local usernames are mapped in the style of git cvs-import; thus user ‘foo’ becomes ‘foo <foo>’, which is sufficient to pacify git and other systems that require email addresses. With the option --use-uuid, usernames are mapped in the git-svn style, with the repository’s UUID used as a fake domain in the email address. Both forms can be remapped to real addresses using the authors read command.

Reading a Subversion stream enables writing of the legacy map as 'legacy-id' passthroughs when the repo is written to a stream file.

Reposurgeon tries hard to silently do the right thing, but there are Subversion edge cases in which it emits warnings because a human may need to intervene and perform fixups by hand. Here are the less obvious messages it may emit:

When a branch A is deleted and a branch B is copied to the name A, the Subversion intent is to replace the contents of branch A with the contents of branch B, keeping the A name. This is a poor man’s merge from before "svn merge" existed. Many Subversion users who formed their habits before svn merge existed still operate this way.

In git terms, this almost corresponds to a merge of A into B followed by a rename of B to A. Branch B continues to exist, however, so we can’t do that in translation. The reposurgeon logic does not try to be clever about this, because "clever" would have rebarbative edge cases; the sequence is translated into a deleteall followed by a commit operation that recreates the B files under corresponding A names. No merge link is created. The commit filling A with a branch copy from B will have B as its first parent, though, so all that would be needed is to create a merge link from the old A before the delete to the commit recreating A.

This case is mentioned here because it is likely to confuse the merge-tracking algorithms used, e.g., by git diff, or if you ever try to merge a branch that forked off the old A to a branch spun off the new (and expect git to know that you do not want to incorporate old A’s changes).

By convention, subversion repository is supposed to have a regular structure, what we’ll call "standard layout". This has three top-level directories, trunk/ and tags/ and branches/. In standard layout, directories beneath tags are never modified after the copy that creates them.

A repository in standard layout can be converted nearly losslessly into Git and other modern DVCSes, with minor exceptions around deleted and later recreated branches and some Subversion-specific file properties.

But standard layout is only a convention; the Subversion tools don’t actually enforce it. One very simple non-standard layout is a "flat" repository. We’ll discuss these in detail in a later section.

In a more common non-standard layout, Subversion repositories are organized to hold multiple projects, with the root directory containing one subdirectory per project and each project subdirectory having a standard layout underneath it.

We’ll call this kind of layout "conformable" when it is perfectly regular, not containing any of the following exceptions:

These sorts of irregularities do not fit well into reposurgeon’s internal data model. The fallback behaviors in reposurgeon that attempt to handle them are fragile and kludgy. Attempting to read such a stream may confuse or even crash reposurgeon’s reader for Subversion dumps.

Unfortunately, it is rare for a multiproject repository not to have defects of this kind. There’s an easy way to check; feed the stream to repocutter swapcheck. If you get an empty report, the stream is perfectly conformable; otherwise, each report line flags a defect you need to fix or remove. Look at the output of "repocutter help swapcheck" for more details.

Conversions of non-conformable multiproject Subversion repositories are messy and tricky. If you work for an organization with a budget, you should seriously consider engaging the author to do yours - that’s likely to save you time and money relative to what you’d pay someone in-house to climb the learning curve.

The general strategy in these cases has three steps:

The first step is the tricky one. There are two general tactics you can pursue towards this: (1) expunging badly-structured content, and/or (2) using repocutter pathrename to massage non-conformable directories into conformable ones.

Before you think about doing expunges, be aware that they can be complicated by cross-project copy operations. If you delete a directory that’s ever a source for a copy into conformable structure, havoc will ensue; a very common case of this is copies from obsolete branches to trunk. That’s why it’s generally best to defer trimming unwanted history until you have it inside reposurgeon.

Generally you’re going to do the heavy lifting with repocutter pathrename operations. The author has seen conversions that literally required hundreds of these. You may also need to do repocutter filecopy and skipcopy operations - but here be dragons; by the time you need to do those you should reconsider not hiring the author.

It cannot be emphasized enough that conversions of non-conformable multi-project repositories are difficult; expect your timeframe to completion to be measured not in hours or days but months. Improving to tools to make these less painful is the reposurgeon project’s main area of ongoing research.

If the repository has no branch structure at all, you might want to filter the stream with "repocutter push trunk" and then individually turning subdirectories into branches using the branchlift command. The pattern for that looks like this:

This will turn two subdirectories into branches named "release-1.0" and "release-2.0", then pops "trunk" off all the paths where it occurs (leaving those commits on the master branch).

This method has the disadvantage that you have to enumerate all branches you want to lift. Still, it may be useful on repositories that consist of one big unbranched file tree not conforming to a standard layout.

If for some reason you want to treat a branchy repository as though it’s flat, using repocutter to rename the "trunk" directory to something else first (like, say, "main") will foil the logic in the reposurgeon stream reader that wants to do branch and tag analysis.

When you’re experimenting with mass changes based on pattern matching (as in filter, transcode, changelogs, or even msgin) bear in mind that commits and other objects actually modified get their Q bits set. You can use this as a selection with "list" or "inspect" commands to check your work.

In CVS repositories of projects that use the GNU ChangeLog convention, a very common pre-conversion artifact is a commit with the comment “*** empty log message ***” that modifies only a ChangeLog entry explaining the commit immediately previous to it. The following

is equivalent to the more verbose

but you are less likely to make difficult-to-notice errors typing the first version.

(Also note how the text regexp acts as a failsafe against the possibility of typing a wrong date that doesn’t refer to a commit with an empty comment. This was a real-world example from the CVS-to-git conversion of groff.)

Occasionally when processing a CVS or Subversion repository you will encounter a case where, during two different spans of time, the same user ID was used by two different people for which you have full name-and-email attributions you want to apply.

As an example, suppose your history has two committers named "fred", one active before March 15 2010 and one after. The name and email fields in their commit attributions all look like "fred <fred>" with variable data following.

There’s no way you can discriminate by time in an authors file. But there’s a different technique that will work. Like this:

The third command allows you to check that the previous two produced a sane-looking result.

Part of the trick here is knowing that if you only set name and email fields with an attribution command the date field is left unchanged.

If you needed to handle more than one of these collisions, you might consider a slightly different approach that illustrates how to write and use a multiline macro:

This depends on the fact that the the script arguments $1 and $2 get substituted before the regular-expression compiler interprets $. You’d follow it by applying an attribution file containing entries for fred1 and fred2.

An Emacs mode for editing msgout dumps ships with the reposurgeon distribution. It includes various useful commands for Gitifying comments and reference-lifting.

Subversion histories that have been touched by git-svn often have subtle damage in the metadata. One common class of problems comes from git-svn creating .gitignore and .git histories in the Subversion repository. Commits on these paths are best treated as junk and removed.

While they could be removed with a reposurgeon expunge command after the repository has been translated to Git, sometimes one sees a worse problem resulting from git-svn somehow creating invalid copy nodes involving .git/.gitignore. This can produce alarming log messages, or even cause a Subversion stream read to fail spuriously despite the actual Subversion part of the history being undamaged.

If you suspect this is happening, the following repocutter expunge command is your friend:

If you use this to pre-filter your repository dump it will remove the garbled copy nodes along with every other modification to matching path, without affecting anything in the actual native Subversion part of the history.

If reposurgeon crashes on a Subversion stream dump that has not been altered or filtered in any way since being emitted by svnadmin dump, that is definitely a bug; please report it as such. To speed up the fix, follow the instructions on preparing a reduced test case.

You are more likely to see a crash when processing a stream that has beem modified by filtering out unwanted commits, e.g. via select or expunge or sift commands. The most common problem is copyfrom operations referring to a source revision that has been removed. In theory reposurgeon should fall back to referencing the latest revision before the gap, but there are still occasional crashes the the reposurgeon maintainers have not successfully characterized.

To do its job, reposurgeon needs to hold all of your history’s metadata in memory. That doesn’t mean the content part, but does mean all of the changeset attributions, comments, and tags. Given a large enough repository, this will overrun the RAM of a small machine. If this happens to you, your reposurgeon instance will die abruptly with an OOM (Out Of Memory) error while attempting to read in your repository.

It is extremely unlikely that this is due to a bug in reposurgeon. Before filing an issue about it, there is a procedure you should try. It consists of bisecting on the parameter the Go language runtime uses to control the frequency of garbage collection. You can set this using the environment variable GOGC or reposurgeon "gc" command.

GOGC defaults to 100, which instructs the runtime to garbage-collect when the heap size is 100% bigger than (i.e., twice) the size of the reachable objects after the last garbage collection. To increase the frequency of GC, usually resulting in a lower memory high-water mark, decrease that percentage threshold. To decrease gc frequency, increase the threshold so the runtime tolerates a larger heap.

To troubleshoot your OOM problem, bisect on this threshold to find the highest value that will avoid OOM. Start by cutting it to 50, then to 25, then to 12, then to 6. If you find a value that allows you to read to completion, you may want to try increasing it by a half interval (e,g. 50 to 75, 25 to 37, etc.) to get back some throughput.

If your repository won’t read in at GOGC=6 you have a real problem. Unfortunately, it’s not one the reposurgeon devteam can help you with; the correct solution to it is to do your conversion on a machine with more RAM and/or more swap configured. 64GB should be sufficient. The largest repository the reposurgeon devteam has ever seen (the history of GCC, 280K Subversion commits) fit on a 128GB machine with GOGC=30.

If you can’t read your history onto a 128GB machine with GOGC=30, then maybe the reposurgeon devteam ought to hear about it. That said, if you can find ways to make reposurgeon more efficient, we are eager to accept those patches, or even just a bug report with the details. It’s probable that there are some efficiency gains yet to be made.

Occasionally it will happen that a conversion on a particularly large or malformed repository seems to stall out, grinding endlessly without completing a conversion phase.

Reposurgeon’s execution time is dominated by cycles spent in the memory allocator and garbage collector. Thus, you can pay RAM to decrease running time - push GOGC up from its default of 100. If your conversion completes in reasonable time before your memory usage increases enough that reposurgeon gets killed by OOM, you win. Otherwise see the previous section about adding RAM and swap space.

A stallout is more difficult to troubleshoot than an OOM, and more likely to indicate an actual bug or algorithmic problem in reposurgeon. There are a couple of things you can do to make a good resolution more likely:

Be aware that bugs can be due to problems in any of (a) reposurgeon itself, (b) the front end it’s using to read your repository, or (c) the back end that writes the modified repository. For example, when you have a bug in a CVS conversion it is more than likely to be a problem with the cvs-fast-export front end. In that case you need to read Reporting bugs in cvs-fast-export and follow those directions.

It is generally not possible to reproduce reposurgeon/repocutter bugs without a copy of the history on which they occurred. When you find a bug, please send the maintainers:

(a) An exact description of how observed behavior differed from expected behavior. If reposurgeon/repocutter died, include the backtrace.

(b) A git fast-import or Subversion dump file of the repository you were operating on, or in the case of CVS the whole repository. Alternatively a pointer to where it can be pulled from - though a static copy submitted with the bug is better is better because commits added afterwards can confuse diagnostics.

(c) A script containing the sequence of reposurgeon or repocutter commands that revealed the bug. If you were exploring interactively, remember that the "history" command exists and can dump your command history to a file.

(d) If you were using the standard-workflow Makefile generated by "repotool initialize", mention that in your bug report. If you modified the Makefile, include a copy with the bug report.

Please use the reposurgeon project’s issue tracker and attach these files. It’s helpful.

Are you seeing git die with a complaint about an unknown --show-original-IDs option? Upgrade your git; reposurgeon needs 1.19.2 or later.

If your target VCS’s importer dies during a rebuild, try writing the repository content to a stream instead and importing the stream by hand. If the latter does not fail, the target VCS’s importer may be slightly buggy - but you have a workaround.

(This has been observed under git 2.5.0 with the result of a 'unite' operation on two repositories. The cause is unknown, as git dies suddenly enough to not leave a crash report.)

A fair amount of effort has been expended to keep the run-time performance of reposurgeon as linear as possible. This is not an easy state to stay in; it is unfortunately quite simple to accidentally regress this without noticing.

To that end, there are some fairly simple scripts in the bench directory of the source distribution that can be used to check for this type of problem. repobench runs reposurgeon multiple times with a different readlimit each time, recording the run time and memory allocated at each iteration. Supply arguments specifying the svn dump file to read and the readlimit values to use like this:

This reads your-dump-file.svn 10 times, with the readlimit set first to 1,000, then 3,000, etc, stepping up until it reaches 20,000.

This produces a .dat file which you can use with repobench -p, or repobench -o to produce graphs.

For an example, see oops.svg. This shows a graph made using a good revision that had linear performance, several made with revisions that introduced a regression that made performance quite non-linear, and the fix. You can easily tell the difference visually.

Reposurgeon scripts are effectively never reused. Thus, incompatible changes to the command language don’t have a high cost in pain to users, and the maintainers feel free to make them whenever improvement seems possible. But just in case, such changes are recorded here.

In versions 4.30 and earlier, the read command had a --branchify option. This has been retired, you are now expected to do the equivalent manipulations with repocutter.

The "tip" command present in 4.24 and earlier has been removed, because the concept it’s trying to compute is not clearly defined at any commit upstream of a fork point. There’s no evidence this was ever used in the wild.

In versions 4.24 and earlier the behavior of the "unite" command was incompatibly different in that it took master branch from the last repo in the list, renaming others. Now it leaves master from the first.

The blob command now takes an explicit mark, rather than creating a new blob :1 and renumbering all others as in 4.23 and earlier.

Filter syntax now uses subcommand verbs rather than the options of 4.23 and earlier.

In versions 4.23 and earlier, the stats command operated on named repositories rather than the currently-selected one.

In versions 4.23 and earlier, it was possible to redirect output from msgin to capture a mailbox of changed entries. This feature has been removed; msgin now sets =Q bits which can be used to generate the same report.

In versions 4.23 and earlier, "[branchify_opt]" read option was a separate command that needed to be invoked before the read and set hidden global context. There was a related branchmap command that has been retired because branch rename covers all its cases in a simpler way.

In versions 4.23 and earlier, the behavior of the branch command was incompatibly different; it did not require a "heads/" or "tags/" prefix on its operand and, because of that, could only operate on heads/ branches and not lightweight tags.

In versions 4.23 and earlier, several commands that now have the form "object verb selection" had the form "object selection verb". This includes branch, tag, reset, and path.

In versions 4.23 and earlier, the syntax of the expunge command was different; it used "~" instead of "--not" and took multiple patterns.

The "paths sup" and "paths sub" commands of versions 4.23 and earlier have been retired and replaced by the enhanced path rename command.

As of 4.23, branch references as used in "branch rename" and elswhere now require a namespace prefix - heads/ or tags/. This was done to eliminate a number of special cases that were difficult to document and remember.

In versions before 4.10, the "reduce" and "blob" options of the "strip" command were bare keywords. Also the options of the "ignores" command were bare keywords. There was a command to set the prompt string that has been retired.

In versions before 4.8, the expunge command run on a repository named "foo" tried to keep deleted fileops in a new synthetic repository named "foo-expunges". This feature has been replaced by the "~" negation operator on expunge selections.

In versions before 4.1, the index command did not see blobs by default.

In versions before 4.0, msgin and msgout were named "mailbox_in" and "mailbox_out:"; --branchify was "branchify_map". Previous versions used the Python variant of regular expressions; some of the more idiosyncratic features of these are not replicated in the Go implementation.

In versions before 3.23, ‘prefer’ changed the repository type as well as the preferred output format. Since then, do this with ‘sourcetype’.

In versions before 3.0, the general command syntax put the command verb first, then the selection set (if any) then modifiers (VSO). It has changed to optional selection set first, then command verb, then modifiers (SVO). The change made parsing simpler, allowed abolishing some noise keywords, and recapitulates a successful design pattern in some other Unix tools - notably sed(1).

In versions before 3.0, path expressions only matched commits, not commits and the associated blobs as well. The names of the "a" and "c" flags were different.

In reposurgeon versions before 3.0, the delete command had the semantics of squash; also, the policy flags did not require a ‘--’ prefix. The ‘--delete’ flag was named "obliterate".

In reposurgeon versions before 3.0, read and write optionally took file arguments rather than requiring redirects (and the write command never wrote into directories). This was changed in order to allow these commands to have modifiers. These modifiers replaced several global options that no longer exist.

In reposurgeon versions before 3.0, the earliest factor in a unite command always kept its tag and branch names unaltered. The new rule for resolving name conflicts, giving priority to the latest factor, produces more natural behavior when uniting two repositories end to end; the master branch of the second (later) one keeps its name.

In reposurgeon versions before 3.0, the tagify command expected policies as trailing arguments to alter its behavior. The new syntax uses similarly named options with leading dashes, that can appear anywhere after the tagify command.

In versions before 2.9. the syntax of authors, legacy, list, and what are now msgin and msgout was different (and legacy was named fossils). They took plain filename arguments rather than using redirect < and >.

In versions so old that the changeover point is now lost in the mists of time, curly brackets (not parens) performed subexpression grouping.

If you need emergency help, go to the #reposurgeon IRC on irc.oftc.net. Be aware, however, that the maintainer is too busy to babysit difficult repository conversions unless he has explicitly volunteered for one or someone is paying him to care about it. For explanation, see Your money or your spec.

http://cvs2svn.tigris.org/features.html

Just after the turn of the 21st century, when Subversion was the new thing in version control, most projects that were using version control were using CVS, and cvs2svn was about the only migration path.

Early cvs2svn had problems on every level, only some of which have been fixed by more recent releases. It tended to spew junk commits into the translated history, and produced strange combinations of Subversion internal operations that most later translation tools would cope with only poorly. Sometimes the resulting translations are actually malformed; more often they contains noisy commits or commit duplications that made little sense under Subversion and make even less under the new target system.

!LINEAR, ?MIXEDBRANCH, DOCUMENTATION

http://www.catb.org/~esr/cvs-fast-export

Formerly named parsecvs. Originally written by Keith Packard to port the X.org repositories, which it did a good job on. Now maintained by me; reposurgeon uses it to read CVS repositories. It is extremely fast and can thus be productively used even on huge repositories.

!ABANDONED, !LINEAR, !NO_IGNORES, !DOCUMENTATION, !CONFIGURATION

http://www.catb.org/~esr/cvsps

Don’t use this. Just plain don’t. The author maintained version 3.x until end-of-lifing it in favor of cvs-fast-export due to fundamental, unfixable problems. It gets branch topology wrong in ways that are difficult to detect.

git-svn, the Subversion converter in the git distribution, is really designed to be a two-way live gateway enabling git users to push and pull commits from a Subversion server. It operates by creating a git repository that is effectively a local mirror of the Subversion history, then performing Subversion client commands to synchronize the two in a git-like way.

This choice of mission means that git-svn’s translation of history into git uses a compromise between Subversion idioms and git ones that is more designed to make transactions back to the Subversion server easy and safe to generate than it is to make full use of the git capabilities that Subversion doesn’t have. This is pidgin translation for a reason better than laziness or failure of nerve, but it’s still pidgin.

Worse, git-svn has bugs that severely compromise it for full translations. It tends to stumble over common repository malformations in Subversion, producing history damage that is significant but evades superficial scrutiny. The author has written about this problem in detail at Don’t do svn-to-git repository conversions with git-svn!

For a straight linear history with no tags or branches, the difference between git-svn’s Subversion-emulating behavior and the way a git repository would most naturally be structured is minimal. But for conformability with Subversion, git-svn cannot (practically speaking) use git’s annotated-tag facility in the local mirror; instead, Subversion tags have to be represented in the local mirror as git branches even if they have no changes after the initial branch copy.

Another thing the live-gatewaying use case prevents is reference-lifting. Subversion references like "r1234" in commit comments have to be left as-is to avoid creating pain for users of the same Subversion remote not going through git-svn.

git-svn was used by both Google Code’s exporter and is used in GitHub’s importer web services. Depending on the latter is not recommended.

!ABANDONED, MIXEDBRANCH, NO_TAGS, NO_IGNORES.

Formerly part of the git suite; what they had before git-svn, and inferior to it. Among other problems, it can only handle Subversion repos with a "standard" trunk/tags/branches layout. Now deprecated.

MIXEDBRANCH, NO_TAGS, NO_IGNORES, ABANDONED.

https://github.com/stevenharman/git-svn-import

A trivial wrapper around git-svn. All the reasons not to use git-svn apply to it as well.

MIXEDBRANCH, NO_TAGS, NO_IGNORES, !ABANDONED.

https://github.com/git/git/tree/master/contrib/svn-fe

svn-fe was a valiant effort to produce a tool that would dump a Subversion repository history as a git fast-import stream. It made it into the git contrib directory and lingers there still.

LINEAR, NO_TAGS, NO_IGNORES, ABANDONED.

http://progetti.arstecnica.it/tailor/

Tailor aimed to be an any-to-any repository translator.

LINEAR, ?NO_IGNORES, ABANDONED.

https://github.com/fragglet/agito

This is a Subversion-to-git tool that was written to handle some cases that git-svn barfs on (but reposurgeon doesn’t - the reposurgeon test suite contains a case sent by agito’s author to check this). It even handles mixed-branch commits correctly.

!LINEAR, !NO_TAGS, !MIXEDBRANCH, CONFIGURATION.

If you cannot use reposurgeon for some reason, this is one of the best alternatives.

https://github.com/nirvdrum/svn2git

A batch-conversion wrapper around git-svn that creates real tag objects. This is the one written in Ruby. Has all pf git-svn’s problems, alas.

!ABANDONED, !NO_TAGS, NO_IGNORES.

If you cannot use reposurgeon for some reason, this is another alternative that is not too horrible. But beware of possible history damage if your Subversion repo has malformations that confuse git-svn.

http://python.ca/nas/python/svn2git.py

Native Python. More a proof of concept than a production tool.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

http://gitorious.org/svn2git

Written in C++. Says it’s based on svn-fast-export by Chris Lee. Not easy to figure out what it actually does, as there is no documentation at all and no test cases. May be genetically related to svn-all-fast-export, but if so they diverged in 2008. Notable for having a configuration language for specifying rules that guide the conversion, allowing the conversion to be incrementally developed.

As Gitorious has shut down, this repository is now available from the Software Heritage Archive.

Now apparently hosted on GitHub. This version has been extended and modified somewhat, and some documentation has been added.

A later version turned up in FreeBSD’s SVN repository. The git history has been removed from this version, but the SVN log indicates that it is a fork of the GitHub repository.

Both the FreeBSD repository and the kde-ruleset repository on GitHub contain rules used by actual conversions, making them a useful source of inspiration for your own.

CONFIGURATION, NO_DOCUMENTATION.

http://c133.org/code/svn-fast-export.c

Written in C. More a proof of concept than a production tool.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

https://github.com/barrbrain/svn-dump-fast-export/

Written in C. Documentation is so lacking that there isn’t even a README. However, it’s possible to deduce what isn’t there by reading the code.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION.

http://repo.or.cz/w/svn-all-fast-export.git

May be genetically related to the Nyblom svn2git, but if so they diverged in 2008.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

There is a script called sccs2git on CPAN which attempts to lift SCCS collections direct to Git. It is not recommended, as it is poorly documented and makes no attempt to group commits into changesets.

NO_DOCUMENTATION, ABANDONED.

http://www.subgit.com/

Nearly unique for this category of software in being closed-source. Beyond an evaluation period, users have to register, possibly for a cost (it’s supposed to be free-of-charge for certain uses: open source projects, education, and ``startups'' — history with BitKeeper shows that such exemptions should probably not be trusted).

The intended outcome of this program is to provide a server with support for both Subversion and Git users to interact at once. This may be of little value overall, as new developers are frequently unfamiliar with Subversion (and old ones forget the usage patterns!), fundamental differences in design of the two VCSes interfering with the quality of both views, and increased confusion with preferred modes of contribution arise.

The quality of SubGit’s conversion is rather poor. It fails to properly translate at least half of the reposurgeon *.svn regression tests, even some of the simpler ones - although trickier cases such as agito.svn it does translate correctly. Large real-world Subversion repos will exhibit multiple issues that SubGit may, silently or otherwise, trip over.

This program will forever contain compromises for the same reasons git-svn does. The non-open source nature leaves little hope of having such issues repaired by skilled community members.

Atlassian’s BitBucket service relies on this for Subversion-to-Git migration. Depending on this service is not recommended.

!MIXEDBRANCH, !LINEAR, CONFIGURATION, DOCUMENTATION

The core classes in inner.go support deserializing and reserializing import streams. In between these two operations the repo state lives in a fairly simple object, Repository. The main part of Repository is just a list of objects implementing the Event interface - Commits, Blobs, Tags, Resets, and Passthroughs. These are straightforward representations of the command types in a Git import stream, with Passthrough as a way of losslessly conveying lines the parser does not recognize.

The general theory of reposurgeon is: you deserialize, you do stuff to the event list that preserves correctness invariants, you reserialize. The "do stuff" is mostly not in the core classes, but there is one major exception. The primitive to delete a commit and squash its fileops forwards or backwards is seriously intertwined with the core classes and actually makes up almost 50% of Repository by line count.

The rest of the surgical code lives outside the core classes. Most of it lives in the RepoSurgeon class (the command interpreter) or the RepositoryList class (which encapsulated by-name access to a list of repositories and also hosts surgical operations involving multiple repositories). A few bits, like the repository reader and builder, have enough logic that’s independent of these classes to be factored out of it.

In designing new commands for the interpreter, try hard to keep them orthogonal to the selection-set code. As often as possible, commands should all have a similar form with a (single) selection set argument.

VCS is not a core class. The code for manipulating actual repos is bolted on the the ends of the pipeline, like this:

The Import and Export boxes call methods in VCS.

Extractor classes build the deserialized internal representation directly. Each extractor class is a set of VCS-specific methods to be used by the RepoStreamer driver class. Key detail: when a repository is recognized by an extractor it sets the repository type to point to the corresponding VCS instance.

All code that knows about the DSL syntax should live in reposurgeon.go along with the program main and the functions for reporting errors, logging, handling signals and aborts, etc.

This is the reader for Subversion dumpfiles. It is the only exception to the rule that reads support for version control systems is implemented by front ends that read them and emit a fast-import stream.

The reason it’s an exception is that Subversion has its own serialization format, and the total complexity of embedding support for those streams was estimated to be lower than that if writing a a completely separate front end.

The code was translated from Python. It retains, for internal documentation purposes, the Python convention of using leading underscores on field names to flag fields that should never be referenced outside a method of the associated struct.

The capitalization of other fieldnames looks inconsistent because the code tries to retain the lowercase Python names and compartmentalize as much as possible to be visible only within the declaring package. Some fields are capitalized for backwards compatibility with the setfield command in the Python implementation, others (like some members of FileOp) because there’s an internal requirement that they be settable by the Go reflection primitives.