Name

shipper — automated shipping of project releases

Synopsis

shipper [-d] [-v] [-w] [-x destinations] [-n profile] [var=val...]

Description

shipper is a tool that tries to reduce the friction cost of shipping releases to as near zero as possible.

Its job is to make it possible for you to run the command shipper | sh in the top-level directory of a project and have a release be properly exported to all the places that you normally deliver it — your personal website, Linux source code archive sites, forge sites, and distribution submission queues; also, to send appropriate notfications to freecode.net, mailing lists, and project IRC channels.

Another goal is to arrange your shipping process in such a way that any one piece of metadata (like your project name, version, or synoptic description) only has to be kept in one place and modified once per release. Metadata is collected from several locations that naturally own pieces of it.

In normal use, you absolutely need to set only two variables: destinations which is the list of destinations to ship to, and version which is the release version of the software you are shipping. The default package name is the basename of the directory from which shipper is run.

When shipping to freecode.com, you may also want to add Freecode-Tags headers to your project specfiles that will become Freecode tags.

Once you have shipper up and running, you can experiment with more advanced features such as: having the program generate project web pages for you using a simple templating system, and automatically tagging your shipped releases.

The output of shipper, run in your top-level project source directory, is a shellscript. To ship your project, feed the script to a shell (of course you can review it before you run it). The only shell features required are command execution and here-documents. No wilcard expansion, shell control syntax, or shell variable evaluation will be required. In practical use you will probably want to pass the shell the -e option (any command failure aborts the script, so notifications sent later don't go out if an early upload fails) and the -x option (to echo shipping commands as they are performed).

Theory of Operation

shipper generates commands to push deliverables out to destinations. Deliverables include: source tarballs, source zip archives, ChangeLog files, README files,and various other project metadata files. Destinations include both private destinations like websites, FTP archive sites and mailing lists, and public destinations like soureceforge.net, freecode.com, savannah.nongnu.org and berlios.de.

shipper goes through the following steps each time it runs:

  1. Collect metadata, stage 1.

    1. Read ~/.shipper.

    2. Read ~/.config/shipper.

  2. Deduce your identity, if it can.

  3. Collect metadata, stage 2.

    1. Read ./control.

    2. Read an RPM spec, if there is just one.

    3. Apply command-line overrides..

  4. Find deliverables

  5. Generate shipping commands

    1. File and directory uploads.

    2. Repository release tagging.

    3. freenode.com announcement.

    4. Mailing-list announcements.

    5. Sends to IRC channels.

First, shipper it reads the .shipper file in your home directory and the current directory (if you like XDG conventions, make a ~/.config/shipper instead). These can set various control variables and templates that aren't project specific.

If that first step didn't find your identity (a nonempty value of whoami), the program's second step is to try to figure out what your name and email address are. It snoops the profiles asociated with git, bzr, and hg looking for these. If it can't deduce this information, it will complain and exit.

Next, shipper reads per-project information from a project metadata file in the current directory, either a Debian control file or an RPM spec. This is where it will normally find the project name, a short text description, its declared homepage location, and other similar things.

The program then reads .shipper for project-specific configuration settings.

Finally, any override settings specified on the shipper command line are applied. Normally version, the release version designation, will be set here.

The -d (dump) option causes shipper to dump all configuration values and exit immediately after collecting them in step 3.

The next phase is finding deliverables. These are either unversioned project metadata (like a README or NEWS file) or versioned product files such as source tarballs.

The last step before generating upload commands may be to generate an HTML file for uploading, if you have specified the html_target variable.

The option -w (web-only) tells shipper to generate upload commands for web pages only rather than for shipping (and possibly tagging) a release.

Otherwise, shipper then generates commands to upload deliverables. The -x may be used to suppress delivery to specified destinations that would normally be performed (e.g., due to a destination variable set in one of your .shipper files); it may be a comma-separated list.

After deliverables have shipped successfully shipper determines if the directory it is in is a working copy in a version-control system that supports release tagging. If so, and the tag_template variable is set to a nonempty value, it generates commands to tag the release just shipped. See the section called “After Uploads” for discussion of what version-control systems are supported and what actions will be taken.

Note that shipper makes one important assumption about the structure of your website(s). Beneath each directory in your destinations list, there will be one subdirectory for each project, with the directory leaf name being the same as the project. Thus, for example, if you have three projects named ruby, diamond and sapphire, and your personal site is at gemstones.net:/public/www/precious/, shipper will expect to be able to drop deliverables in three directories gemstones.net:/public/www/precious/ruby, gemstones.net:/public/www/precious/diamond/, and gemstones.net:/public/www/precious/sapphire/. Note that shipper will not create these project directories for you if they're missing; this is deliberate, so that uploads to sites that are not prepared for them will fail noisily.

After generating commands to do uploads, shipper may generate additional commands to perform project release notifications to freecode.com, email lists, and IRC channels.

Configuration in More Detail

The behavior of shipper depends on a handful of internal variables. Some of these variables have defaults computed at startup time. All can be set or overridden in the per-user ~/.config/shipper and ~/.shipper files, and overridden again in any per-project control file, specfile, or .shipper file. All .shipper files are Python code and the syntax of variable settings is Python's; control and RPM files are parsed according to their native conventions.

If a variable is set in a config file, that value is locked in (except for the destinations variable which can be appended to from a specfile, see below) Variables that are not set in a config file may be set by the values of fields in your project specfile.

Any variable, locked or not, can be overidden from the command line by an argument of the form "name=value".

For basic use, it is only necessary to set one variable in your ~/.config/shipper or ~/.shipper file: destinations, the list of destinations to ship to. Normally you'll set this globally, pointing all your projects at your main distribution website and public drop sites. It is also possible to add destinations on a per-project basis by giving a comma-separated list in a #Destinations: comment in one of the specfile. Finally, you can set the variable in a per-project .shipper to ignore your global destination list.

Here are the configuration variables you will normally set in your ~/.config/shipper or ~/.shipper file:

VariableRPM specfile fieldDebian specfile fieldMeaning
destinations#Destinations:XBS-Destinations:

A list of destinations to ship to. Each location is a place to drop deliverables: either a [user@]site:path destination that scp(1) can use, or an FTP url that lftp(1) can use, or a mail URL, or an IRC URL, or one of the special public destination names.

Note that when the destination is a web or FTP site actual project directory is computed by appending the value of project to the destination you're shipping to.

A destination entry of ~ can be used to clear the previously-set value, so an individual project control file can ignore destinations set in your home-directory ~/.config/shipper or ~/.shipper file. Additionally, ~ followed by a destination causes that individual destination to be removed.

There is no default.. If you do not set this variable, shipper won't generate any shipping commands.

whoami--

A name and email address for the user; should be RFC-822 form, e.g. "J. Random User <user@fubar.com>. If you don't configure this, shipper will snoop the configuration files of other programs attempting to deduce it.

berlios.id--

Your ID on Berlios. Defaults to the username part of whoami.

savannah.id--

Your ID on Savannah. Defaults to the username part of whoami.

sourceforge.id--

Your ID on SourceForge. Defaults to the username part of whoami.

gittip_id--

Your ID on gittip.com. If present, generated web pages will include a link to your gittip page where people can leave tips.

date--

The program's startup time. This can be used in the web page and email announcement templates.

You can use the Python function time.strftime("...") in your ~/.shipper file to format this date to your taste. If you don't set this in the config file, the program will set a default for you.

html_template--

Template HTML from which to generate the page_file for shipping. There is a default which generates a very simple page containing a title, a date, and a table listing downloadable resources. This template is used when shipping to a web directory if the html_target is set. If this variable is None or an empty string, index page generation is suppressed.

mail_template--

Template text from which to generate the message shipped to destinations that are mailto URLs. There is a default which generates a very simple email containing a subject, a pointer to the project web page, and the last entry in the project changelog.

Here are the variables you will normally set in a project control file:

VariableRPM specfile fieldDebian specfile fieldMeaning
projectName:Project:

Project name, used to generate the stem part of the names deliverables that shipper builds. If the specfile is a Debian control file, the Debian-specific part of the version number (after the dash) is removed.

summarySummaryDescription:

The one-line project summary field from the first line of description in your control file, or the Summary field in your specfile.

description%descriptionDescription:

The Description field from your specfile.

html_target#HTML-TargetXBS-HTML-Target

File to which a templated web page generated from the release data should be generated. If this variable is an empty string (the default), templated page generation is suppressed. The most likely value for this, if you set it, is "index.html"

websiteURL:Website:

Project website URL. Used when generating project announcements. When your upload destination is Berlios or SourceForge, this will be generated for you if you don't specify it.

freecode_tags#Freecode-Tags:XBS-Freecode-Tags:

Topic tags, used on Freecode.

freecode_name#Freecode-Name:XBS-Freecode-Name:

Freecode shortname, used in generating freecode.com announcements. If this isn't present, it defaults to the project name; you only need to set it if they differ.

savannah.name#Savannah-Name:XBS-Savannah-Name:

Savannah shortname, used in generating announcements and computing the location of your project web directory on savannah.nongnu.org (no support for gnu projects yet). If this isn't present, it defaults to the project name; you only need to set it if they differ.

berlios.name#Berlios-Name:XBS-Berlios-Name:

Berlios shortname, used in generating berlios.de announcements and computing the location of your project web directory. If this isn't present, it defaults to the project name; you only need to set it if they differ.

sourceforge.name#SourceForge-Name:XBS-SourceForge-Name:

SourceForge shortname, used in generating announcements and computing the location of your project directories on SourceForge. If this isn't present, it defaults to the project name; you only need to set it if they differ.

If your SourceForge-Name has a '@' in it, the prefix before that is removed and interpreted as your SourceForge user ID. If no such prefix is present, the user ID from whoami will be assumed.

sourceforge.folder#SourceForge-Folder:XBS-SourceForge-Folder:

Path componenent to be appended to the destination for file updates. Useful when you want to organize release tarballs into subdirectories for historical or other reasons.

gitorious_url#Gitorious-URL:XBS-Gitorious-URL:

URL pointing at Gitorious hosting space for the project. If it exists, it should point at a git repository for the source code.

github_url#Github-URL:XBS-Github-URL:

URL pointing at GitHub hosting space for the project. If it exists, it should point at a git repository for the source code.

ohloh_url#Ohloh-URL:XBS-Ohloh-URL:

URL pointing at Ohloh statistics.

irc_channel#IRC-Channel:XBS-IRC-Channel:

URL pointing at one IRC chat channel for the project, or a comma-separated list of such channels.

logo#LogoXBS-Logo

A logo or icon for the project. The default web page template will embed this in the page header.

webdir#Web-Directory#XBS-Web-Directory

The relative path of a web directory to be mirrored to project webspace at a forge location. If this is defined it overrides the normal web page templating mechanism associated with the html_template and html_target variables.

tag_template#VC-Tag-Template#XBS-VC-Tag-Template

You can change tagging behavior by setting the variable tagtemplate. If you set it to None, release tagging will be suppressed entirely. If you set it to a nonempty string, that will become the template for the name of the tag. The "%(version)s" in the string will be replaced with the version number, and other globals are available for substitution in the same way.

tag_message#VC-Tag-Message#XBS-VC-Tag-Message

You can also customize the actual log message by setting the variable tag_message. See the default with shipper -N; the "%(version)s" in it will be replaced with the version, and other globals are available for substitution in the same way.

bzr, however, cannot annotate tags. The tag_message variable will be ignored in this case.

irc_message#IRC-Message#XBS-IRC-Message

You can customize the IRC message(s) sent on release by setting the variable ircmessage. See the default with shipper -n; variable cookies in it will be replaced in obvious ways.

There is one variable you will normally set as a shipper command-line option:

VariableRPM specfile fieldDebian specfile fieldMeaning
versionVersion:Version:

Project version, used in generating the names of deliverables that shipper builds. In order to run, shipper requires that version be set somewhere, in a profile or spec file or as a command-line override.

Finally, these variables are computed and available for substitution:

VariableMeaning
lastchange

The most-recent-changes entry from your NEWS, HISTORY, Changelog, or RPM spec file. To extract this information from a NEWS or HISTORY file, first any header lines (detected by leading spaces) are skipped; in a specfile, all lines up to and including %changelog are skipped. Then the first nonblank line (which is assumed to be the date/release information) is skipped. Then all lines before the next blank one are grabbed.

This is used to generate Freecode announcements and in the default template for announcement mail.

resourcetable

The HTML table of links to downloadable resources. This variable is only computed if the you specified html_template. Any setting of it in the startup files is ignored.

extralines

Lines to be inserted after the table of resources. Shipper will fill this with generated links to, e.g., Freecode, Ohloh, and your project IRC channel. If you set it in a profile, the generated links will be appended to your text.

All these variables are available for substitution at the time a web page or email announcement is generated. In general, any variable you set in your ~/.shipper file will be available at the time the web page or email announcement is generated. Use the Python "%(variable)s" syntax, not shell-substitution syntax.

Finding Local Deliverables

The following files are considered stock deliverables and may be shipped if they are present when shipper starts up:

FileExplanation
README or READ.ME

Project roadmap file.

tarballs

Current source tarballs, that is any files named ${project}.*-${version}.tar.gz or ${project}.*-${version}.tgz or ${project}.*-${version}.tar.bz2.

checksum files

Current source tarball checksums, that is any files named ${project}.*-${version}.md5 or ${project}.*-${version}.sh{224,356,384,512}.

RPMs

Current RPMs, that is any files named ${project}.*-${version}.*.rpm.

debs

Current deb files, that is any files named ${project}.*-${version}.*.deb.

zipfiles

Current source zip archives, that is any file named ${project}.*-${version}.zip.

COPYING

Project license file.

NEWS

Project news file.

ChangeLog

Project change log.

HISTORY

Project history file.

BUGS

Project bug list.

TODO

Current to-do list.

*.{html,xhtml,css,js}

Any files with an .html, .js, or .css extension will normally be shipped to all website destinations. This is overridden if you have declared a web directory to mirror to website destinations.

Stock deliverables which are textfiles and contain the string "shipper: ignore this." will not be shipped by the generated script. This is especially useful for archival GNU-style ChangeLogs, which you may want to avoid shipping but which need to be present to pacify autotools.

Additionally, if the html_template and html_target variables are set, a templated HTML page named by it will be composed and shipped to websites by the generated script.

In the resource-table element of the generated web page, shipper generates explanatory text to go with each deliverable. For web pages, it tries to extract the contents of the title element. Anything else gets a canned description of its type.

Shipping to Destinations

In operation, shipper walks through a list of destinations, finding the required deliverables for each one and generating the required shipping actions to push them out to the destination. Here are the destination types shipper knows about:

Destination TypeDeliverablesSpecified byExplanation
sourceforgetarball, zipfile, RPMs, debs, README, COPYING, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO.-

If the sourceforge destination is on your list, shipper will generate commands to ship a tarball and source and binary RPMs to the prject's frs directory via rsync; the deliverables will be placed in a subdirectory named after the version. It will also attempt to upload all web deliverables (README, CHANGES, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO) to the project's SourceForge web directory.

berliostarball, zipfile, RPMs, debs, README, COPYING, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO.-

If the berlios destination is on your list, shipper will generate commands to ship a tarball and source and binary RPMs to the berlios.de submission directory via FTP. It will also attempt to upload all web deliverables (README, CHANGES, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO) to the project's Berlios web directory.

savannahtarball, zipfile, RPMs, debs, BUGS, TODO.-

If the savannah destination is on your list, shipper will generate commands to ship a tarball and source and binary RPMs to your project's download directory. It does not attempt to do anything with web content, as Savannah's web-update protocol involves CVS and is too painful to cope with.

freecode--

If the freecode destination is on your list, shipper will generate a command to post a release announcement on freecode.com using freecode-submit(1). The announcement will include URLs for whichever of the following deliverables are shipped, using the URL field from your specfile: tarballs, zipfiles, RPMs, debs, ChangeLog.

Generic Web siteREADME, COPYING, tarball, zipfile, RPMs, debs, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO.scp destination ([user@]host:dir)

This destination type represents a website. shipper generates commands using scp(1) to put deliverables on websites. If the user part of the scp destination is absent, it will be taken from the environment variable USERNAME.

No generic Web sites are shipped to by default. You must declare them by putting scp destinations in the destinations variable.

Generic FTP sitetarball, RPMs, debsFTP URL

Old-fashioned FTP site with no metadata. The FTP URL is parsed to get the sitename and directory where deliverables should be dropped. The FTP username to be used will be taken from the environment variable USERNAME. The FTP password will be looked up in your ~/.netrc file.

No generic FTP sites are shipped to by default. You must declare them by putting FTP urls in the destinations variable.

Email address-mailto URL

A message expanded from the mail_template is emailed to each email address specified as a destination.

No email destinations are set up by default. You must declare them by putting mailto: URLs in the destinations variable.

IRC-IRC URL

A generated announcement is sent to a specified IRC channel.

No IRC destinations are set up by default. You must declare them by putting IRC urls in the destinations variable. IRC URLs specified in the IRC-Channel variable are announced to as well; they're also listed on the templated web page, if templating has been selected.

After Uploads

After upload commands have been generated, shipper looks at where it is running. If it can recognize that the current directory is under version control, and the tag_template option has been enabled, it will generate additional commands to tag the just-shipped tree as an external release.

Currently, the version-control systems supported for tagging after shipping are Subversion, git, hg, and bzr. Due to variations in tagging behavior in these systems, this feature behaves slightly differently depending on which one is in use.

If the run is in the trunk of a Subversion repository (that is, the current directory is name trunk and has a subdirectory named .svn) and there is a tags peer directory, and the tags directory does not have a subdirectory named for the release version, then shipper generates commands to copy and tags the contents of trunk.

If it is in a git, hg, or bzr repository (that is, there is a subdirectory named .git, .bzr, or .hg) it will generare commands to tag the release. If there is a remote repo to be pushed to, local changes and tags wil be pushed. In all three of these cases the tag attempt will fail if the tag previously existed. See also the discussion of the tagtemplate variable.

After tagging, commands to ship email notifications are generated.

The last command generated, if applicable, will be the IRC broadcast.

Command-line Options

The -d option generates a configuration dump to standard output.

The -x option specifies a destination, or a comma-separated list of destinations, not to ship to. This overrides the setting of the destinations variable by shippper config files and control or spec files.

The -v option makes shipper chatty about what it's doing. The -h option prints a usage message and exits.

The -n option disables normal evaluation of the user's profile under $HOME in faver of a profile specified as the argument. It is intended to support regression testing of shipper and is not likely to be of general interest.

Arguments, if present, must be of the form name=value and can be used to override earlier settings from your profiles. The most frequently useful override will probably be "version=".

Hints and Tips

The following variable definition in your makefile will ensure that the makefile version is derived from (and thus always consistent with) the specfile version.

VERSION=$(shell sed <*.spec control -n -e '/Version: \(.*\)/s//\1/p')

A makefile production like the following will allow you to type make release and be sure that all the deliverables shipper knows about will be rebuilt before being shipped.

release: project-$(VERS).tar.gz project-$(VERSION).md5
	shipper version=$(VERS) | sh -e -x

The -e option is recommended to make all errors fatal. This way if an upload fails for some reason, the later notifications won't happen. The -x option makes the script echo its commands as they're performed.

You will want to change project to your project name. You will need to write a production that builds your tarball and checksum files.

In an SConstruct file, you may want something like this:

version = version

and something like

env.Alias("release", [tarball], 'shipper version=%s | sh -e -x' % version)

with 'tarball' being a production that makes a release tarball (and checksum file, if desired).

Requirements

This program requires the following helpers for shipping to specific destination types:

freecode-submit

For shipping release notifications to freecode.com. The freecode-submit program is separately maintained.

irkerd

For shipping notices to IRC channels. The irkerd program is separately maintained.

lftp

For shipping to ftp sites. The freecode-submit program is separately maintained.

scp

For shipping to websites. There are several different implementations of scp; one widely-used version is shipped by the OpenSSH project.

sendmail

For shipping email announcements. Either classic sendmail or Postfix sendmail will work. All it has to be able to do is ship mail presented on standard input to a list of addresses specified as arguments.

These must be on your executable $PATH at the time this program is invoked, or shipping to the relevant destination types will fail.

EXAMPLES

This is most of my .config/shipper file. I have omitted a bulky customized web page template.

# Set defaults for shipper(1)

# Identify myself
whoami = "Eric S. Raymond <esr@thyrsus.com>"

# My ID on gittip
gittip_id = "esr"

# By default, ship to my primary website
destinations = ["login.ibiblio.org:/public/html/catb/esr/"]

# Customize date format
date = time.strftime("%d %b %Y")

Here is the control file for the shipper project:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
 shipper is a power distribution tool for developers with multiple
 projects who do frequent releases.  It automates the tedious process
 of shipping a software release and (if desired) templating a project
 web page. It can deliver releases in correct form to SourceForge,
 Berlios, and Savannah, and knows how to post a release announcement
 to freecode.com via freecode-submit.

XBS-Destinations: freecode, mailto:esr@thyrsus.com

Homepage: http://www.catb.org/~esr/shipper

XBS-HTML-Target: index.html

XBS-Gitorious-URL: https://gitorious.org/shipper

XBS-IRC-Channel: irc://chat.freenode.net/#shipper

XBS-Logo: shipper-logo.png

XBS-Freecode-Tags: packaging, distribution

XBS-VC-Tag-Template: %(version)s

This shipper project does not have a .shipper in its directory.

Bugs

The program can deliver downloadables to Berlios incoming, but not script a Berlios release. That has to be done through the Berlios web interface.

The program cannot update Savannah webspace, which is only accessible through a CVS checkin procedure.

The program can put downloadables in place on SourceForge, but not set the "Preferred Download". Usually this will point to the most recent download subdirectory by default, which is the right thing.

The shipper framework is extensible and it is relatively easy to add new destination types and new deliverables; it is a bug that we do not yet support all major forge sites as destinations.

Files

$HOME/.shipper; $HOME/.config/shipper; $PWD/.shipper.

Author

Eric S. Raymond .

There is a project web page at http://www.catb.org/~esr/shipper/.

See Also

freecode-submit(1), lftp(1), scp(1), ssh(1).