= Hacker's guide to cvs-fast-export =

`cvs-fast-export` is a complex program doing an intrinsically difficult
job.  Because analyzing CVS repositories has lots of strange edge
cases, it is likely to need modification in the future.  This document
is a collection of notes intended to make that less intimidating.

== History ==

This program was originally written as a one-off hack by Keith Packard
in early 2006, when he was working on migrating the X repositories
from CVS to git, and was not originally intended to survive that
conversion.  It was called `parsecvs` then.  It called git tools
directly to create translated repositories.

The code was briefly maintained by Bart Massey before passing to Eric
S. Raymond in late 2012. ESR wrote the fast-export output stage and
renamed the program to reflect its new function. Later, he made the
parser re-entrant and parallelized the ingestion of masters. He
collected a rogue's caller of test histories to verify correct operation

Until the code was ported from C to Golang with AI assistance in 2026,
significant portions of this code were a trackless jungle of complex
algorithms with poorly documented assumptions. Particularly weird CVS
histories triggered buffer overruns and heap-corruption errors. The
decade of maintenance in C was tedious and difficult.

The Go port, and AI assistance in analyzing CVS edge cases, resolved
the code's chronic issues.

== Description ==

To understand this program, you need to understand the problem
it solves: lifting CVS repositories to git-compatible fast-import
streams.

What makes this problem difficult is that CVS records deltas (and
tags) per-file, but what we want for the fast-import representation is
changesets - that is, coherent groups of per-file deltas that capture
multiple per-file changes made with the same intention at the same
time.  The fundamental thing `cvs-fast-export` does is identify cliques
of per-file deltas that should be coalesced into changesets.

To do this, it relies on the fact that the CVS command-line tools
fake supporting changesets by replicating the comment that the user
supplied to `cvs commit` into every individual file delta that the
commit creates.

Under relatively recent implementations, CVS also embeds a common (and
unique) commit-ID field in each file delta of the group.  These
cliques can be unambiguously identified.  Groups recorded by older
implementations that don't generate commit-IDs must be identified by
the fact that they have the same change comment, author, and change
date.

Actually, commit-date comparison has to be fuzzy because each file
commit is actually done as a separate operation and may not complete
in the same clock second as the previous one (this is why
`cvs-fast-export` has the `-w` option).  Timestamp matching is further
complicated by clock-skew effects; for historical reasons, deltas are
committed with a timestamp generated on the client side rather than
the server.  Thus, clock drift between different client machines can
cause some odd effects, including child revisions with dates before
their parents.

But timestamp issues are actually the least part of the problem.  A
much bigger issue is per-file branch structures and tags that aren't
conformable with each other. The CVS tools have few safeguards against
creating such, and it is easy to end up with a situation where file-delta
cliques can be resolved but the right way to build them into a DAG of
parent-child links is unclear or ill-defined.  Inconsistent or
incomplete tagging can cause interpretation problems as well.

Now you should read "RCS/CVS LIMITATIONS" in the `cvs-fast-export(1)`
manual page.

== Conformable branch structure ==

Below is a simple example of conformable branch structure involving
two files.

In this diagram, down is the arrow of time.  Each box on the left-hand
side represents a CVS file delta, each box on the right a changeset.
In each box the top line is a list of files modified and the bottom
line a change comment.  The branch labels at the bottom are HEAD
for the main branch in each CVS file and master for the main branch
in the gitspace DAG.

-----------------------------------------------------------------

 +--------------+                          +================+
 |  foo.c 1.1   | +---------------+        | foo.c, bar.c   |
 |First revision| |   bar.c 1.1   |        | First revision |
 +--------------+ |First revision |        +================+
        |         +---------------+                 |
        |                 |                         |
        |         +---------------+        +=================+
        |         |   bar.c 1.2   |        |      bar.c      |
        |         |Second revision|        | Second revision |
        |         +---------------+        +=================+
        |                 |                         |
 +--------------+         |                         |
 |  foo.c 1.2   | +---------------+        +===============+
 |Third revision| |   bar.c 1.3   |        | foo.c, bar.c  |
 +--------------+ |Third revision |        |Third revision |
        |         +---------------+        +===============+
      HEAD                |                       |
                        HEAD                    master
-----------------------------------------------------------------

Here's an elaboration of that example, a conformant pair of CVS
masters with branching:

-------------------------------------------------------------------------

 +--------------+                             +===============+
 |  foo.c 1.1   |   +--------------+          | foo.c, bar.c  |
 |First revision|   |   bar.c 1.1  |          |First revision |
 +--------------+   |First revision|          +===============+
        |           +--------------+                 |
        |                   |                        |
        |           +---------------+         +===============+
        |           |   bar.c 1.2   |         |    bar.c      |
        |           |Second revision|         |Second revision|
        |           +---------------+         +===============+
        |                   |                        |
 +--------------+           |                        |
 |  foo.c 1.2   |   +---------------+         +===============+
 |Third revision|   |   bar.c 1.3   |         | foo.c, bar.c  |
 +--------------+   |Third revision |         | Third revision|
        |   \       +---------------+         +===============+
        |    \              |       \1.3.1           |  \
        |     \1.2.1        |        \               |   \
        |      \            |    +---------------+   |  +===============+
        | +---------------+ |    |bar.c 1.3.1.1  |   |  | foo.c, bar.c  |
        | |foo.c 1.2.1.1  | |    |Fourth revision|   |  |Fourth revision|
        | |Fourth revision| |    +---------------+   |  +===============+
        | +---------------+ |           |            |          |
        |       |           |           |            |          |
      HEAD   alternate    HEAD       alternate     master     alternate
-------------------------------------------------------------------------

Note that the branch point and branch ID (the three-part label on the
branch) for `alternate` are different in the two CVS masters, so
`cvs-fast-export` cannot rely on them matching to figure out the
topology.

It also has to deal with this case correctly:
---------------------------------------------------------------------------

 +--------------+                               +===============+
 |  foo.c 1.1   |   +---------------+           | foo.c, bar.c  |
 |First revision|   |   bar.c 1.1   |           |First revision |
 +--------------+   |First revision |           +===============+
        |           +---------------+                  |
        |                   |                          |
        |           +---------------+           +===============+
        |           |   bar.c 1.2   |           |    bar.c      |
        |           |Second revision|           |Second revision|
        |           +---------------+           +===============+
 +--------------+           |                          |
 |  foo.c 1.2   |   +---------------+           +===============+
 |Third revision|   |   bar.c 1.3   |           | foo.c, bar.c  |
 +--------------+   |Third revision |           |Third revision |
        |   \       +---------------+           +===============+
        |    \1.2.1         |        \                 |   \
        |     \             |         \1.3.1           |  +===============+
        | +---------------+ |          \               |  |     foo.c     |
        | |foo.c 1.2.1.1  | |           |              |  |Fourth revision|
        | |Fourth revision| |           |              |  +===============+
        | +---------------+ |           |              |         |
        |       |           |    +--------------+      |  +===============+
        | +--------------+  |    |bar.c 1.3.1.1 |      |  | foo.c, bar.c  |
        | |foo.c 1.2.1.2 |  |    |Fifth revision|      |  |Fifth revision |
        | |Fifth revision|  |    +--------------+      |  +===============+
        | +--------------+  |           |              |         |
        |       |           |           |              |         |
        |       |           |           |              |         |
      HEAD   alternate    HEAD       alternate      master    alternate
---------------------------------------------------------------------------

That is, after any branch there may be a delta that *doesn't* make
a changeset with any delta on matching branches.

The previous diagrams elide some important details, which is how tags and
branches are actually represented in CVS.  First: there are no
per-changeset tags, only per-file ones.  When CVS fakes tagging a
changeset, what it actually does is add the same tag symbol to
every file master in the changeset.

Various kinds of operator error and/or CVS bug can cause the creation
of incomplete tagged sets, which *don't* annotate every master in
existence at tag creation time.  These are a headache for any
conversion tool.  The old C version of `cvs-fast-export` dealt with
them by creating tagged branchlets containing exactly one commit; the
new one accepts incomplete tag sets but scans for them backward from
the end of the history. So if copies of a tag were attached CVS
revisions that ended up in more than one commit, it gets attached
to the latest one.

Named CVS branches are represented by adding a "sticky tag" to every
file in the branch. In the above examples, the branch beginning with
1.2.1.1 would have been created with a command sequence like this done
while 1.2 is checked out:

------------------------------------------------------------------------------
cvs tag alternate_0                  # Create a symbolic name for 1.2
cvs tag -r alternate_0 -b alternate  # Give 'alternate' a magic sticky value
------------------------------------------------------------------------------

The magic sticky value for the first (1.2.1.x) branch is 1.2.0.1.  If
a second, 1.2.2.x branch were created, its magic sticky tag would have
the value 1.2.0.2.  The sticky tag is treated as a name for its
corresponding branch, whatever the tip revision happens to be.

== Vendor branches ==

Vendor branches are a poorly-documented feature which has been a
source of great confusion for programs attempting to convert or
data-mine CVS repositories.  This section describes the assumptions
`cvs-fast-export` uses in dealing with them in painstaking detail,
because it is not unlikely they will be a continuing source of
correctness issues.

The CVS manual documents two key behaviors that matter for conversion:
imports create a vendor branch (default 1.1.1), and *until a local
trunk commit exists* the vendor branch is treated as HEAD. Once a local
trunk commit is made, HEAD moves to trunk. The import command also
allows overriding the vendor branch with `-b`, and CVS does *not* detect
inconsistencies between the vendor tag and the branch number specified
by `-b`. This can result in irreconcilable ambiguity in later history.

Here's what the CVS documentation has to say under the documentation
of the "import" command: "The [vendor] source is saved in a
first-level RCS branch, by default 1.1.1. Updates are leaves of this
branch; for example, files from the first imported collection of
source will be revision 1.1.1.1, then files from the first imported
update will be revision 1.1.1.2, and so on."

It is possible to have imports from multiple vendors using the -b
option of "import" to set a base branch other than 1.1.1.

In "CVS II: Parallelizing Software Development" (1990) Brian Berliner,
one of the principal CVS developers, write a major section 2.2 titled
"Tracking Third-Party Source Distributions". It begins:

____
Currently, a large amount of software is based on source
distributions from a third-party distributor. It is often the case
that local modifications are to be made to this distribution, and that
the vendor's future releases should be tracked. Rolling your local
modifications forward into the new vendor release is a time-consuming
task, but cvs can ease this burden somewhat. The checkin program of
cvs initially sets up a source repository by integrating the source
modules directly from the vendor's release, preserving the directory
hierarchy of the vendor's distribution.  The branch support of RCS is
used to build this vendor release as a branch of the main RCS trunk.
Figure 2 shows how the "head" tracks a sample vendor branch when no
local modifications have been made to the file.
____

The following diagram reproduces the topology of Berliner's figure 2
using the same conventions as the diagrams in the previous section
(these revisions have no change comments):

-------------------------------------------------------------------------

 +---------------+    1.1.1   +-------------------+
 | rcsfile.c 1.1 |------------| rcsfile.c 1.1.1.1 | 'SunOS_4_0'
 +---------------+   'SunOS'  +-------------------+
                        A               |
                        |     +-------------------+
                        |     | rcsfile.c 1.1.1.2 | 'SunOS_4_0_1'
                        |     +-------------------+
                        |               |
                        |     +-------------------+
                        |     | rcsfile.c 1.1.1.3 | 'YAPT_5_5C'
                        |     +-------------------+
                        |               |
                        |     +-------------------+
             "HEAD"-----+---->| rcsfile.c 1.1.1.4 | 'SunOS_4_0_3'
                              +-------------------+

-------------------------------------------------------------------------

(The intended meaning of the arrow from "HEAD" to the vendor branch
label 1.1.1 is not explained in the paper.)

Berliner continues: 
____
Once this is done, developers can check out files
and make local changes to the vendor's source distribution. These
local changes form a new branch to the tree which is then used as the
source for future check outs. Figure 3 shows how the "head" moves to
the main RCS trunk when a local modification is made.
____

-------------------------------------------------------------------------

 +---------------+    1.1.1   +-------------------+
 | rcsfile.c 1.1 |------------| rcsfile.c 1.1.1.1 | 'SunOS_4_0'
 +---------------+   'SunOS'  +-------------------+
         |                              |
 +---------------+            +-------------------+
 | rcsfile.c 1.2 |            | rcsfile.c 1.1.1.2 | 'SunOS_4_0_1'
 +---------------+            +-------------------+
         A                              |
         |                    +-------------------+
         |                    | rcsfile.c 1.1.1.3 | 'YAPT_5_5C'
         |                    +-------------------+
         |                              |
         |                    +-------------------+
       "HEAD"                 | rcsfile.c 1.1.1.4 | 'SunOS_4_0_3'
                              +-------------------+

-------------------------------------------------------------------------

Berliner continues:
____
When a new version of the vendor's source
distribution arrives, the checkin program adds the new and changed
vendor's files to the already existing source repository. For files
that have not been changed locally, the new file from the vendor
becomes the current "head" revision. For files that have been modified
locally, checkin warns that the file must be merged with the new
vendor release. The cvs "join" command is a useful tool that aids this
process by performing the necessary RCS merge, as is done above when
performing an "update."
____

Berliner concludes:
____
There is also limited support for "dual"
derivations for source files. See Figure 4 for a sample dual-derived
file. This example tracks the SunOS distribution but includes major
changes from Berkeley. These BSD files are saved directly in the RCS
file off a new branch.
____

----------------------------------------------------------------------------

 +---------------+       1.1.1                         +-------------------+
 | rcsfile.c 1.1 |----+--------------------------------| rcsfile.c 1.1.1.1 |
 +---------------+    |                                +-------------------+
         |            |  1.1.2  +-------------------+            |
 +---------------+    +---------| rcsfile.c 1.1.2.1 |  +-------------------+
 | rcsfile.c 1.2 |              +-------------------+  | rcsfile.c 1.1.1.2 |
 +---------------+                        |            +-------------------+
                                +-------------------+            |
                                | rcsfile.c 1.1.2.2 |  +-------------------+
                                +-------------------+  | rcsfile.c 1.1.1.3 |
                                                       +-------------------+

----------------------------------------------------------------------------

Note that the paper does not actually describe how CVS should behave
if the 1.2 revision were absent from this diagram.

Historically, `cvs-fast-export`'s behavior with respect to vendor
branches (from when it was `parsecvs`) was described by the following
comment due to Keith Packard:

"Vendor branches" (1.1.x) are created by importing sources from an
external source. In X.org, this was from XFree86 and DRI. When these
trees are imported, cvs sets the 'default' branch in each `,v` file to
point along this branch. This means that tags made between the time
the vendor branch is imported and when a new revision is committed to
the head branch are placed on the vendor branch In addition, any files
without such a delta appear to adopt the vendor branch as 'head'. We
fix this by merging these two branches together as if they were the
same."

All that is consistent with the Berliner paper except, crucially, the
last sentence (" merging these two branches together as if they were
the same").  Consider the following revision diagram, which
corresponds to `Changelog,v` in the `oldhead` test repository:

----------------------------------------------------------------------------
 +---------------------+            +---------------------+
 |    Changelog 1.1    |            |  Changelog 1.1.1.1  |
 | 1994-12-03T06:09:14 |----------->| 1994.12.03.06.09.14 |
 +---------------------+            +---------------------+
           |                                   |
 +---------------------+                       |
 |    Changelog 1.2    |                       |
 | 1995-02-08T11:54:21 |                       |
 +---------------------+                       |
                                    +---------------------+
                                    |   Changelog 1.1.1.2 |
                                    | 1995-07-27T20:23:14 |
                                    +---------------------+

----------------------------------------------------------------------------

The actual `oldhead` repo has revisions up to 1.8 on the master branch
and 1.1.1.3, but this subgraph illustrates the problem.  Under the
merge rule, the tip content will be that of 1.1.1.2 than 1.2.
This does not match CVS's observed behavior.

The behavior now implemented is to prefer the branch header (when
present) as the indicator of which vendor branch is HEAD before the
first trunk commit. When no branch header is present or it does not
identify an existing vendor branch, we fall back to the heuristic of
choosing the highest-numbered vendor branch, point the "master" named
reference at it, and splice the existing master branch to the end of
that vendor branch.

In addition, when a branch header is present in the RCS admin section,
`cvs-fast-export` uses it as a hint for which vendor branch to treat as
HEAD before the first trunk commit. If the header is missing or points
to a vendor branch that does not exist, we fall back to the heuristic
and emit a warning. If a non-standard vendor branch (anything other
than 1.1.1) is used *and* trunk commits exist, CVS itself no longer
records enough information to resolve history unambiguously; this is
treated as a data-quality warning.

== Operation ==

This program operates in three stages.  The first (analysis) digests a
collection of RCS masters into a collection of linked lists and
structures representing per-file revision trees.  The second
(resolution) massages the revision trees into a DAG (directed acyclic
graph) of changesets.  The third stage (export) emits a report on the
DAG structure, either a fast-export stream expressing it or DOT code
for a visualization that can be rendered by graphviz.

=== Analysis stage ===

The main function of this stage is `pathToMaster()`. It uses the golang-rcs
library by Arran Ubels; this replaced a lot of gnarly code (which was
sadly prone to heap corruption in odd cases) in the C version.

Parsing of masters is done concurrently. The point of this is to avoid
allowing I/O waits for one master read or snapshot export to stall
compute-intensive processing of other masters (that is, mainly, delta
assembly).

=== Resolution stage ===

The main function of this stage is `Build()`.  All the really black
magic happens inside it.  Nobody understands all of this code; it is
recommended that you not attempt to modify it without AI assistance

For a focused description of branch/tag collation behavior and edge
cases (used as compatibility spec for the Go port), see
link:cvs-collation-spec.adoc[CVS Collation and Symbol Resolution Spec].

=== Export stage ===

Most of the export third stage is relatively easy to understand.
t takes the annotated DAG produced by the second stage and emits either a
fast-import stream or a DOT representation of the DAG.

== Good practice ==

When modifying this code, run the regression tests (`make check`)
early and often.  It is very easy to break even with apparently
innocuous changes.  You will want to have `pylint`, and `shellcheck`
installed for full code validation.

If you find a bug and fix it, please try to create a toy repo exhibiting
the problem - or, better yet, a minimal set of operations to reproduce
it. Then add that to the regression tests.

Likewise, when adding a feature, add a test for it as well.

This code is entirely enough of a comprehension problem without the
extra drag created by style variations in whitespace. The "make
reflow" production uses go fmt to enforce canonical Go style. Please
always use braces even on single-statement blocks. Before shipping a
patch, please run "make reflow" and test the code again.

If you figure out something about the code that isn't documented here
- or, especially, if it's documented wrongly - please include an
explanation with your patch.

== Historical credits

A lot of the original C codebase evaporated when this program was
ported to Go, obsolesced by the fact that Go gas first-class
dictionaries and also by the existence of the golang-rcs library. But
thanks is still owed to people who worked on the C code.

Most notably: Jens Bethkowsky <jens.bethkowsky@rwth-aachen.de> added a
red-black-tree implementation to speed up symbol search; Aidan Hobson
Sayers <aidanhs@cantab.net> replaced an O(n**3) sort with an O(n log
n) sort; David Leonard <david.leonard@opengear.com> sped up
`compute_parent_links()` and wrote several other significant
optimizations.  Laurence Hygate <loz@flower.powernet.co.uk> wrote many
sort and hash optimizations. Alan Barrett wrote the improved progress
meter.  Tom Enterline sped up snapshot generation.

// end
