shipper — automated shipping of project releases


shipper [-d] [-v] [-w] [-x destinations] [-n profile] [-i] [-N] [var=val...]


shipper is a tool that tries to reduce the friction cost of shipping software releases to as near zero as possible.

Its job is to make it possible for you to run the command shipper version=NNN | sh -e -x in the top-level directory of a project and have a release be properly exported to all the places that you normally deliver it — your personal website, Linux source code archive sites, forge sites, and distribution submission queues; also, to send appropriate notifications to, mailing lists, project IRC channels, and Twitter.

Another goal is to arrange your shipping process in such a way that any one piece of metadata (like your project name, version, or synoptic description) only has to be kept in one place and modified once per release. Metadata is collected from several locations that naturally own pieces of it.

In normal use, you only absolutely need to set only two variables: destinations which is the list of destinations to ship to, and version which is the release version of the software you are shipping. The default package name is the basename of the directory from which shipper is run.

When adding as a destination, you may also want to add Project-Tags headers to your project specfiles that will become Project tags.

Once you have shipper up and running, you can experiment with more advanced features such as: having the program generate project web pages for you using a simple templating system, and automatically tagging your shipped releases.

The output of shipper, run in your top-level project source directory, is a shellscript. To ship your project, feed the script to a shell (of course you can review it before you run it). The only shell features required are command execution and here-documents. No wildcard expansion, shell control syntax, or shell variable evaluation will be required. In practical use you will probably want to pass the shell the -e option (any command failure aborts the script, so notifications sent later don't go out if an early upload fails) and the -x option (to echo shipping commands as they are performed).

Theory of Operation

shipper generates commands to push deliverables out to destinations. Deliverables include: source tarballs, tarball checksums, zip archives, ChangeLog files, README files,and various other project metadata files. Destinations include both private destinations like websites, FTP archive sites and mailing lists, and public destinations like,, and

shipper goes through the following steps each time it runs:

  1. Perform pre-shipping checks

    1. Exit with a complaint if run from a repository with uncommitted changes.

    2. Run the validation check (if any) aborting if it fails.

  2. Collect metadata, stage 1.

    1. Read ~/.shipper.

    2. Read ~/.config/shipper.

  3. Deduce your identity, if it can.

  4. Collect metadata, stage 2.

    1. Read ./control.

    2. Read an RPM spec, if there is just one.

    3. Read any [shipper] section in your git configuration.

    4. Apply command-line overrides..

  5. Find deliverables

  6. Generate shipping commands

    1. File and directory uploads.

    2. Repository release tagging.

    3. announcement.

    4. Mailing-list announcements.

    5. Notifications to IRC channels.

First, shipper performs pre-ship checks (this may be disabled by the -N or --no-check). If the current directory is a version-control repository (git, hg, and svn are supported) it complains and bails out if the repository has uncommitted local changes. Then, if the project metadata includes a validation check command, that is run and shipper bails out with a complaint if it returns a nonzero status.

Next, shipper reads the .shipper file in your home directory and the current directory (if you like XDG conventions, make a ~/.config/shipper instead). These can set various control variables and templates that aren't project specific. These are regular Python files; see the section called “Configuration in More Detail” for more detail.

If that step didn't find your identity (a nonempty value of whoami), the program's second step is to try to figure out what your name and email address are. It snoops the profiles associated with git, bzr, and hg looking for these. If it can't deduce this information, it will complain and exit.

Next, shipper reads per-project information from a project metadata file in the current directory, either a Debian control file or an RPM spec. This is where it will normally find the project name, a short text description, its declared homepage location, and other similar things.

The program then reads .shipper for project-specific configuration settings.

Next, if there is a .git/config file, it looks for project-specifuc settings in a [shipper] section. This is where you put settings that you don't want to export in the tarball, like hosting-site links.

Finally, any override settings specified on the shipper command line are applied. Normally version, the release version designation, will be set here.

The -d (dump) option causes shipper to dump all configuration values and exit immediately after collecting them in step 3.

The next phase is finding deliverables. These are either unversioned project metadata (like a README or NEWS file) or versioned product files such as source tarballs.

For some unversioned metadata - README, COPYING, AUTHORS, NEWS, BUGS, and TODO - the program looks for a corresponding asciidoc (.adoc) file and renders it to HTML, which will be used in generating the resource table that goes in the default templated web page.

The last step before generating upload commands may be to generate an HTML file for uploading, if you have specified the html_target variable. (Under some circumstances informational files for a remote Gitweb repository may be generated at this stage as well.)

The option -w (web-only) tells shipper to generate upload commands for web pages and updates for gitweb views only, rather than for shipping (and possibly tagging) a release.

The option -p (phase) tells shipper to commands for some phases only. The phases are:


Tarballs and other code products.


Web content.


Gitweb repository decorations


Repository tagging.


Notifications to Freshcode.


Email, IRC and other broadcast notifications.

The -w option is equivalent to "-p web". Any -p option overrides -w.

Otherwise, shipper then generates commands to upload deliverables. The -x may be used to suppress delivery to specified destinations that would normally be performed (e.g., due to a destination variable set in one of your .shipper files); it may be a comma-separated list.

After deliverables have shipped successfully shipper determines if the directory it is in is a working copy in a version-control system that supports release tagging. If so, and the tag_template variable is set to a nonempty value, it generates commands to tag the release just shipped. See the section called “After Uploads” for discussion of what version-control systems are supported and what actions will be taken.

Note that shipper makes one important assumption about the structure of your website(s). Beneath each directory in your destinations list, there will be one subdirectory for each project, with the directory leaf name being the same as the project. Thus, for example, if you have three projects named ruby, diamond and sapphire, and your personal site is at, shipper will expect to be able to drop deliverables in three directories,, and Note that shipper will not create these project directories for you if they're missing; this is deliberate, so that uploads to sites that are not prepared for them will fail noisily.

After generating commands to do uploads, shipper may generate additional commands to perform project release notifications to, email lists, and IRC channels.

Configuration in More Detail

The behavior of shipper depends on a handful of internal variables. Some of these variables have defaults computed at startup time. All can be set or overridden in the per-user ~/.config/shipper and ~/.shipper files, and overridden again in any per-project control file, specfile, or .shipper file. All .shipper files are Python code and the syntax of variable settings is Python's; control and RPM files are parsed according to their native conventions.

If a variable is set in a config file, that value is locked in (except for the destinations variable which can be appended to from a specfile, see below) Variables that are not set in a config file may be set by the values of fields in your project specfile.

Any variable, locked or not, can be overidden from the command line by an argument of the form "name=value".

For basic use, it is only necessary to set one variable in your ~/.config/shipper or ~/.shipper file: destinations, the list of destinations to ship to. Normally you'll set this globally, pointing all your projects at your main distribution website and public drop sites. It is also possible to add destinations on a per-project basis by giving a comma-separated list in a #Destinations: comment in one of the specfile. Finally, you can set the variable in a per-project .shipper to ignore your global destination list.

Here are the configuration variables you will normally set in your ~/.config/shipper or ~/.shipper file:

VariableRPM specfile fieldDebian specfile fieldgit config entryMeaning

A list of destinations to ship to. Each location is a place to drop deliverables: either a [user@]site:path destination that scp(1) can use, or an FTP url that lftp(1) can use, or a mail URL, or an IRC URL, or one of the special public destination names.

Note that when the destination is a web or FTP site actual project directory is computed by appending the value of project to the destination you're shipping to.

A destination entry of ~ can be used to clear the previously-set value, so an individual project control file can ignore destinations set in your home-directory ~/.config/shipper or ~/.shipper file. Additionally, ~ followed by a destination causes that individual destination to be removed.

There is no default.. If you do not set this variable, shipper won't generate any shipping commands.


A pattern which, if a Developer-Clone is present and matches it, tells shipper that the repository is manages by gitolite. This in turn tells shipper how to upload description and README.html to the repository where a Gitweb instance will see and use them.


A name and email address for the user; should be RFC-822 form, e.g. "J. Random User <>. If you don't configure this, shipper will snoop the configuration files of other programs attempting to deduce it.

Your ID on Savannah. Defaults to the username part of whoami.

Your ID on SourceForge. Defaults to the username part of whoami.


Your ID on If present, generated web pages will include a link to your patreon page where people can leave tips.


The program's startup time. This can be used in the web page and email announcement templates.

You can use the Python function time.strftime("...") in your ~/.shipper file to format this date to your taste. If you don't set this in the config file, the program will set a default for you.


If present, a validation command to run before shipping anything; if it returns a nozero status shipping is aborted. Standard output of the command is redirected to standard error. If your project uses standard convebtions, "make check" is likely to be a good value for this.


Template HTML from which to generate the page_file for shipping. There is a default which generates a very simple page containing a title, a date, and a table listing downloadable resources. This template is used when shipping to a web directory if the html_target is set. If this variable is None or an empty string, index page generation is suppressed.


Attributes to be inserted in the body tag of the templated web page.


Background image to use for web page.


Template HTML from which to generate a Gitweb-style README.html into the .git subdirectory (if that directory is present). There is a default which generates a very simple page section containing a home=page link, Git clone URL, and description. If this variable is None or an empty string, README.html generation is suppressed.


Template text from which to generate the message shipped to destinations that are mailto URLs. There is a default which generates a very simple email containing a subject, a pointer to the project web page, and the last entry in the project changelog.

Here are the variables you will normally set in a project control file or git config:

VariableRPM specfile fieldDebian specfile fieldgit config entryMeaning

Project name, used to generate the stem part of the names deliverables that shipper builds. If the specfile is a Debian control file, the Debian-specific part of the version number (after the dash) is removed.


The one-line project summary field from the first line of description in your control file, or the Summary field in your specfile.


The Description field from your specfile.


File to which a templated web page generated from the release data should be generated. If this variable is an empty string (the default), templated page generation is suppressed. The most likely value for this, if you set it, is "index.html"

Project website URL. Used when generating project announcements. When your upload destination is SourceForge, this will be generated for you if you don't specify it.


Topic tags, used on Freshcode.


Freshcode shortname, used in generating announcements. If this isn't present, it defaults to the project name; you only need to set it if they differ.

Savannah shortname, used in generating announcements and computing the location of your project web directory on (no support for gnu projects yet). If this isn't present, it defaults to the project name; you only need to set it if they differ.

SourceForge shortname, used in generating announcements and computing the location of your project directories on SourceForge. If this isn't present, it defaults to the project name; you only need to set it if they differ.

If your SourceForge-Name has a '@' in it, the prefix before that is removed and interpreted as your SourceForge user ID. If no such prefix is present, the user ID from whoami will be assumed.


Path componenent to be appended to the destination for file updates. Useful when you want to organize release tarballs into subdirectories for historical or other reasons.


URL pointing at a repository web page for the project.


Reference to a git repository from which the project can be cloned via ssh, for push privileges.


Reference to a git repository from which the project can be cloned read-only.


URL pointing at OpenHub statistics.


URL pointing at one IRC chat channel for the project, or a comma-separated list of such channels.


A comma-separated list of Debian pachage names.


A logo or icon for the project. The default web page template will embed this in the page header.


The relative path of a web directory to be mirrored to project webspace at a forge location. If this is defined it overrides the normal web page templating mechanism associated with the html_template and html_target variables.


You can change tagging behavior by setting the variable tagtemplate. If you set it to None, release tagging will be suppressed entirely. If you set it to a nonempty string, that will become the template for the name of the tag. The "%(version)s" in the string will be replaced with the version number, and other globals are available for substitution in the same way.


You can also customize the actual log message by setting the variable tag_message. See the default with shipper -N; the "%(version)s" in it will be replaced with the version, and other globals are available for substitution in the same way.

bzr, however, cannot annotate tags. The tag_message variable will be ignored in this case.


You can customize the message(s) sent over IRC and Twitter on release by setting the variable instant_message. See the default with shipper -n; variable cookies in it will be replaced in obvious ways.

There is one variable you will normally set as a shipper command-line option:

VariableRPM specfile fieldDebian specfile fieldMeaning

Project version, used in generating the names of deliverables that shipper builds. In order to run, shipper requires that version be set somewhere, in a profile or spec file or as a command-line override.

Finally, these variables are computed and available for substitution:


The most-recent-changes entry from your NEWS, NEWS.adoc, HISTORY, Changelog, or RPM spec file. In a specfile, all lines up to and including %changelog are skipped. Then the first nonblank line (which is assumed to be the date/release information) is skipped. Then all lines before the next blank one are grabbed. The rules used on NEWS and HISTORY files are described in a later section.

This is used in the default template for announcement mail.


The HTML table of links to downloadable resources. This variable is only computed if the you specified html_template. Any setting of it in the startup files is ignored.


Lines to be inserted after the table of resources. Shipper will fill this with generated links to, e.g., Freshcode, OpenHub, and your project IRC channel. If you set it in a profile, the generated links will be appended to your text.

All these variables are available for substitution at the time a web page or email announcement is generated. In general, any variable you set in your ~/.shipper file will be available at the time the web page or email announcement is generated. Use the Python "%(variable)s" syntax, not shell-substitution syntax.

Finding Local Deliverables

The following files are considered stock deliverables and may be shipped if they are present when shipper starts up:


Project roadmap file.


Current source tarballs, that is any files named ${project}.*-${version}.tar.gz or ${project}.*-${version}.tgz or ${project}.*-${version}.tar.bz2.

checksum files

Current source tarball checksums, that is any files named ${project}.*-${version}.md5 or ${project}.*-${version}.sh{224,356,384,512}.


Current RPMs, that is any files named ${project}.*-${version}.*.rpm.


Current deb files, that is any files named ${project}.*-${version}.*.deb.


Current source zip archives, that is any file named ${project}.*-${version}.zip.


Project license file.

NEWS or NEWS.adoc

Project news file.


Project change log.


Project history file.

BUGS or BUGS.adoc

Project bug list.

TODO or TODO.adoc

Current to-do list.


Any files with an .html, .js, or .css extension will normally be shipped to all website destinations. This is overridden if you have declared a web directory to mirror to website destinations.

Stock deliverables which are textfiles and contain the string "shipper: ignore this." will not be shipped by the generated script. This is especially useful for archival GNU-style ChangeLogs, which you may want to avoid shipping but which need to be present to pacify autotools.

Additionally, if the html_template and html_target variables are set, a templated HTML page named by it will be composed and shipped to websites by the generated script.

In the resource-table element of the generated web page, shipper generates explanatory text to go with each deliverable. For web pages, it tries to extract the contents of the title element. Anything else gets a canned description of its type.

Shipping to Destinations

In operation, shipper walks through a list of destinations, finding the required deliverables for each one and generating the required shipping actions to push them out to the destination. Here are the destination types shipper knows about:

Destination TypeDeliverablesSpecified byExplanation
sourceforgetarball, zipfile, RPMs, debs, README, COPYING, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO.-

If the sourceforge destination is on your list, shipper will generate commands to ship a tarball and source and binary RPMs to the prject's frs directory via rsync; the deliverables will be placed in a subdirectory named after the version. It will also attempt to upload all web deliverables (README, CHANGES, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO or their .adoc-suffixed variants) to the project's SourceForge web directory.

savannahtarball, zipfile, RPMs, debs, BUGS, TODO, BUGS.adoc, TODO.adoc.-

If the savannah destination is on your list, shipper will generate commands to ship a tarball and source and binary RPMs to your project's download directory. It does not attempt to do anything with web content, as Savannah's web-update protocol involves CVS and is too painful to cope with.


If the freshcode destination is on your list, shipper will generate a command to post a release announcement on using freshcode-submit(1). The announcement will include URLs for whichever of the following deliverables are shipped, using the URL field from your specfile: tarballs, zipfiles, RPMs, debs, ChangeLog.

Generic Web siteREADME[.adoc], COPYING, tarball, zipfile, RPMs, debs, ChangeLog, NEWS[.adoc], HISTORY, *.{html,css,js}, BUGS[.adoc], TODO[.adoc].scp destination ([user@]host:dir)

This destination type represents a website. shipper generates commands using scp(1) to put deliverables on websites. If the user part of the scp destination is absent, it will be taken from the environment variable USERNAME.

No generic Web sites are shipped to by default. You must declare them by putting scp destinations in the destinations variable.

Generic FTP sitetarball, RPMs, debsFTP URL

Old-fashioned FTP site with no metadata. The FTP URL is parsed to get the sitename and directory where deliverables should be dropped. The FTP username to be used will be taken from the environment variable USERNAME. The FTP password will be looked up in your ~/.netrc file.

No generic FTP sites are shipped to by default. You must declare them by putting FTP urls in the destinations variable.

Email address-mailto URL

A message expanded from the mail_template is emailed to each email address specified as a destination.

No email destinations are set up by default. You must declare them by putting mailto: URLs in the destinations variable.


A generated announcement is sent to a specified IRC channel.

No IRC destinations are set up by default. You must declare them by putting IRC urls in the destinations variable. IRC URLs specified in the IRC-Channel variable are announced to as well; they're also listed on the templated web page, if templating has been selected.

informational files for a remote Gitweb repository may also be shipped at this stage. These are description and README.html, and are generated on the theory that a gitolite instance is likely to have a Gitweb view as well.

For this update to work, the repository must have the writer-is-owner option set. Additionally, in .gitolite.rc on the server, the readme must be uncommented.

After Uploads

After upload commands have been generated, shipper looks at where it is running. If it can recognize that the current directory is under version control, and the tag_template option has been enabled, it will generate additional commands to tag the just-shipped tree as an external release.

Currently, the version-control systems supported for tagging after shipping are Subversion, git, hg, and bzr. Due to variations in tagging behavior in these systems, this feature behaves slightly differently depending on which one is in use.

If the run is in the trunk of a Subversion repository (that is, the current directory is name trunk and has a subdirectory named .svn) and there is a tags peer directory, and the tags directory does not have a subdirectory named for the release version, then shipper generates commands to copy and tags the contents of trunk.

If it is in a git, hg, or bzr repository (that is, there is a subdirectory named .git, .bzr, or .hg) it will generare commands to tag the release. If there is a remote repo to be pushed to, local changes and tags wil be pushed. In all three of these cases the tag attempt will fail if the tag previously existed. See also the discussion of the tagtemplate variable.

After tagging, commands to ship email notifications are generated.

The last command generated, if applicable, will be the IRC and Twitter broadcasts.

Configuring Twitter broadcasts

To enable Twitter broadcasters, yio first need to get developer access on Twitter, then register an app. It doesn't matter what you call the app; what you need are the four credentials that Twitter calls an API key, an API secret key, an access token, and an access token secret.

Once you have these, put them in your shipper config file by setting for variables like this (with the actual values filled in):

twitter_consumer_key="AAAAAAAAAAAAAAAAAAAAAAAAAAAAA" # (API key)

Then make sure the Twython library is installed on your system. The commnd "sudo pip install twython" should do this. Once this is set up, shipper will generate tweets with the same subsitutions it uses for IRC notifications: the relevant config variable is "Instant-Message".

Twitter broacasting is performed only when shipper is passed the -i option. This is not intended to be used manually; rater, it's there so that another instance can generate a "shipper -i" command as the last part of a series of shipping and notification actions.

NEWS/HISTORY/ChangeLog format

In order to generate its various kinds of announcements, shipper needs to extract the version number and relase notes particular to the current release. In order to accomplish this, it tries to parse your NEWS, HISTORY, or ChangeLog files, in that order. It also recognizes these with the subset .adoc in case they are in asciidoc markup in conformance with GitHub and GitLab conventions

The file is analyzed as a list of stanzas - continuous stretches of nonblank lines separated by blank lines. If the first line of the first stanza begins with a digit, it is selected. Otherwise the first stanza is assumed to be a title header and skipped; then the second stanza is selected.

The first line of the selected stanza is examined. If it begins with a decimal digit, the leading part of the line consisting of decimal digits and periods is interpreted as the version number of the release. (This will be overridden if the invocation of shipper specifies a version.)

That line is then scanned for a date of the form yyyy-mm-dd. If no such date is found, or the date does not match today's date in local time, shipper aborts.

The first line of the selected stanza is discarded. All subsequent lines are treated as release notes.

These rules match conventions that are very widely used in NEWS files. If you normally use a different format, you can play nicely with both human viewers and the parser by adding a title line and following stanza with a first line something like Recent Changes.

Command-line Options

The -d option generates a configuration dump to standard output.

The -x option specifies a destination, or a comma-separated list of destinations, not to ship to. This overrides the setting of the destinations variable by shippper config files and control or spec files.

The -v option makes shipper chatty about what it's doing. The -h option prints a usage message and exits.

The -n option disables normal evaluation of the user's profile under $HOME in faver of a profile specified as the argument. It is intended to support regression testing of shipper and is not likely to be of general interest.

Arguments, if present, must be of the form name=value and can be used to override earlier settings from your profiles. The most frequently useful override will probably be "version=".

Hints and Tips

The following variable definition in your makefile will ensure that the makefile version is derived from (and thus always consistent with) the specfile version.

VERSION=$(shell sed <*.spec control -n -e '/Version: \(.*\)/s//\1/p')

A makefile production like the following will allow you to type make release and be sure that all the deliverables shipper knows about will be rebuilt before being shipped.

release: project-$(VERS).tar.gz project-$(VERSION).md5 index.html
	shipper version=$(VERS) | sh -e -x

The -e option is recommended to make all errors fatal. This way if an upload fails for some reason, the later notifications won't happen. The -x option makes the script echo its commands as they're performed.

You will want to change project to your project name. You will need to write a production that builds your tarball and checksum files.

It is also a good idea to have a similar production that just refreshes web pages and other public metadata without shipping tarballs or touching repositories.

refresh: index.html
	shipper -N -w version=$(VERS) | sh -e -x

In an SConstruct file, you may want something like this:

version = version

and something like

env.Alias("release", [tarball], 'shipper version=%s | sh -e -x' % version)

with 'tarball' being a production that makes a release tarball (and checksum file, if desired).


This program requires the following helpers for shipping to specific destination types:


For shipping release notifications to The freshcode-submit program is separately maintained.


For shipping notices to IRC channels. The irkerd program is separately maintained.


For shipping to ftp sites. The lftp program is separately maintained.


For shipping to websites. There are several different implementations of scp; one widely-used version is shipped by the OpenSSH project.


For shipping email announcements. Either classic sendmail or Postfix sendmail will work. All it has to be able to do is ship mail presented on standard input to a list of addresses specified as arguments.


Used to put size attributes in the page's logo inclusion, if there is one. Without this, large logos will slop out of the table generated for the page title and date.


For shipping release announcemts to Twitter.

The first four must be on your executable $PATH at the time this program is invoked, or shipping to the relevant destination types will fail. If the Twython library is not importable, shipping announcements to Twitter will fail silently.


This is most of my .config/shipper file. I have omitted a bulky customized web page template.

# Set defaults for shipper(1)

# Identify myself
whoami = "Eric S. Raymond <>"

# My ID on patreon
patreon_id = "esr"

# By default, ship to my primary website
destinations = [""]

# Customize date format
date = time.strftime("%d %b %Y")

Here is the control file for the shipper project:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
 shipper is a power distribution tool for developers with multiple
 projects who do frequent releases.  It automates the tedious process
 of shipping a software release and (if desired) templating a project
 web page. It can deliver releases in correct form to SourceForge,
 and Savannah, and knows how to post a release announcement via


XBS-IRC-Channel: irc://

XBS-Logo: shipper-logo.png

#XBS-Project-Tags: packaging, distribution

XBS-Validate: make check

And here is the shipper config section in its repository:

	destinations =
	htmltarget = index.html
	repositoryurl =
	developerclone =
	anonymousclone = git://
	tagtemplate = %(version)s

Note that by putting these entries in the git configuration rather than the control file you make it possible to (a) hide potentially senstive email addresses, and (b) rehost the project repository without having to ship a release to update the control-file contents. Instead, just update these entries and use shipper to regenerate the displayed URLs on the project web page.

This shipper project does not have a .shipper in its directory.


The program cannot update Savannah webspace, which is only accessible through a CVS checkin procedure.

The program can put downloadables in place on SourceForge, but not set the "Preferred Download". Usually this will point to the most recent download subdirectory by default, which is the right thing.

The shipper framework is extensible and it is relatively easy to add new destination types and new deliverables; it is a bug that we do not yet support all major forge sites as destinations.


$HOME/.shipper; $HOME/.config/shipper; $PWD/.shipper. $PWD/.git/config.


Eric S. Raymond .

There is a project web page at

See Also

freecode-submit(1), lftp(1), scp(1), ssh(1).