shipper is a tool for shipping project releases. Its job is to make it possible for you to run the command shipper in the top-level directory of a project and have a release be properly exported to all the places that you normally deliver it — your personal website, Linux source code archive sites, and distribution submission queues. A second goal is to arrange your shipping process in such a way that metadata like your project version only have to be kept in one place and modified once per release. The overall goal is to reduce the friction cost of shipping releases to as near zero as possible.
As much as possible, shipper tries to deduce what it should do rather than requiring you to tell it. In order to do this, it relies on your project obeying standard GNU-like naming conventions. It also relies on being able to mine project metadata out of a package specfile. A package specfile may be either an RPM spec or a Debian control file; for both, shipper expects to see a small handful of extension fields that tell it what to do.
In normal use, you need set only one header, “Destinations:” which is the list of destinations to ship to. You may also want to add some magic “Project-Tag-List” headers to your project specfiles. Once you have shipper up and running, you can experiment with more advanced features such as having the program generate project web pages for you.
shipper pushes deliverables out to destinations. Deliverables include: source tarballs, source zip archives, ChangeLog files, README files,and various other project metadata files. Destinations include both private destinations like websites, FTP archive sites and mailing lists, and public destinations like freecode.net and berlios.de. The shipper framework is extensible and it is relatively easy to add new destination types and new deliverables; in the future, we hope to support all major forge sites as destinations.
The shipper program's first step is
to read the .shipper files in your home directory
and the current directory. These set various control variables and
templates shipper will use later.
Second, shipper reads per-project information from a specfile, either an RPM spec or a Debian control file. The project name, version, and various other pieces of project-related metadata are extracted from the specfile.
The -n (nobuild) option causes
shipper to dump all configuration values
and exit immediately after collecting them.
The first real work that gets done is finding deliverables. These are either unversioned project metadata (like a README or NEWS file) or versioned product files such as source tarballs. Finding deliverables is separated from uploading because it means that you can stop and inspect what you're going to ship before committing to an upload.
The last step before uploading may be to generate an
index.html file for uploading, if no such file
alreay exists and you have specified the -m
option.
Without the -u (upload) option,
shipper stops just before uploading,
displaying the exact upload commands that would have been used to ship
deliverables. Thus, the command shipper with no
options will show you exactly what shipper
would do for a real upload.
Otherwise, shipper then uploads
deliverables and posts all announcements. The -x may
be use to suppress delivery to specified destinations that would
normally be performed (e.g., due to a destination variable set in one
of your .shipper files>); it may be a
comma-separated list.
After deliverables have shipped successfully and notifications
posted, shipper determines if the directory
it is in is a working copy in a version-control system that supports
release tagging. If so, and the -t has been
selected, it tries to tag the release just shipped. See the section called “After Deliverables have Shipped” for discussion of what version-control systems
are supported and what actions will be taken.
Finally, note that shipper makes one
important assumption about the structure of your website(s). Beneath
each directory in your destinations list, there
will be one subdirectory for each project, with the directory leaf
name being the same as the project. Thus, for example, if you have
three projects named ruby, diamond and sapphire, and your personal
site is at gemstones.net:/public/www/precious/,
shipper will expect to be able to drop
deliverables in three directories
gemstones.net:/public/www/precious/ruby,
gemstones.net:/public/www/precious/diamond/, and
gemstones.net:/public/www/precious/sapphire/.
Note that shipper will not create these
project directories for you if they're missing; this is deliberate, so
that uploads to sites that are not prepared for them will fail
noisily.
The behavior of shipper depends on a handful of internal
variables. Some of these variables have defaults computed at startup
time. All can be set or overridden in the per-user
~/.shipper file, and overridden again in any
per-project .shipper file. Both files are Python
code and the syntax of variable settings is Python's.
If a variable is set in a config file, that value is locked in
(except for the destinations variable which can be
appended to from a specfile, see below) Variables that are
not set in a config file may be set by the values
of fields in your project specfile.
For basic use, it is only necessary to set one such variable:
destinations, the list of destinations to ship to.
Normally you'll set this globally, pointing all your projects at your
main distribution website and public drop sites, in your
~/.shipper file. It is also possible to add
destinations on a per-project basis by giving a comma-separated list
in a #Destinations: comment in the specfile. You can set the variable
in a per-project .shipper to ignore your global
destination list.
The first thing shipper looks for is a specfile in the current
directory; there must be exactly one. It extracts the project name
from the Name field. Next step is to find the project version (the
variable package). This is extracted from the
Version field of the specfile, or by looking for a makefile macro with
a name beginning with VERS; if the value of that macro is a shell
command wrapped in $(shell ...), it is executed and the output is
captured to yield the version. If both versions are present, they are
consistency-checked.
shipper gets most of the rest of the data it uses to decide what to do from headers in the specfile. The following table lists all the variables and their corresponding specfile fields.
| Variable | RPM specfile field | Debian specfile field | Meaning |
|---|---|---|---|
destinations | #Destinations: | XBS-Destinations: |
A list of destinations to ship to using
scp(1). Each location is a place to drop deliverables:
either a [user@]site:path destination that
scp(1) can use, or an FTP url that
lftp(1)
can use, or one of the special public destination names. Note that when
the destination is a web or FTP site
actual project directory is computed by
appending the value of A destination entry of “~” can be used to clear the
previously-set value, so an individial project control file can ignore
desrinations set in your home-directory
There is no default.. If you do not set this variable, shipper won't ship anything. |
whoami | - | - |
A name and email address for the user; should be RFC-822 form, e.g. "J. Random User <user@fubar.com>. If you don't configure this, shipper will snoop the configuration files of other programs attempting to deduce it. |
date | - | - |
The program's startup time. This can be used in the web page and email announcement templates. You can use the Python function time.strftime("...") in your
|
indextemplate | - | - |
Template HTML from which to generate index.html for shipping.
There is a default which generates a very simple page containing a
title, a date, and a table listing downloadable resources. This is
used when shipping to a web directory, if the |
mailtemplate | - | - |
Template text from which to generate the file SHIPPER.EMAIL to be shipped to destinations that are mailto URLs. There is a default which generates a very simple email containing a subject, a pointer to the project web page, and the last entry in the project changelog. |
package | Name: | Package: |
Project name, used to generate the stem part of the names deliverables that shipper builds. If the specfile is a Debian control file, the Debian-specific part of the version number (after the dash) is removed. |
version | Version: | Version: |
Project version, used in generating the names of deliverables that shipper builds. |
website | URL: | Website: |
Project website URL. Used when generating project announcements. When your upload destination is Berlios or SourceForge, this will be generated for you if you don't specify it. |
project_tags | #Project-Tag-List: | XBS-Project-Tag-List: |
Topic tags, used on Freecode. |
freecode_name | #Freecode-Name: | XBS-Freecode-Name: |
Freecode shortname, used in generating freecode.net announcements. If this isn't present, it defaults to the project name; you only need to set it if they differ. |
berlios_name | #Berlios-Name: | XBS-Berlios-Name: |
Berlios shortname, used in generating berlios.de announcements and computing the location of your project web directory. If this isn't present, it defaults to the project name; you only need to set it if they differ. |
sourceforge_name | #SourceForge-Name: | XBS-SourceForge-Name: |
SourceForge shortname, used in generating announcements and computing the location of your project directories on SourceForge. If this isn't present, it defaults to the project name; you only need to set it if they differ. If your SourceForge-Name has a '@' in it, the prefix before that is removed an interpreted as your SourceForge user ID. If no such prefix is present, the user ID from whoami will be assumed. |
gitorious_url | #Gitorious-URL: | XBS-Gitorious-URL: |
URL pointing at Gitorious hosting space for the project. If it exists, it should point at a git repository for the source code. |
summary | Summary | Description: |
The one-line project summary field from your specfile. |
description | %description | Description: |
The Description field from your specfile. |
lastchange | - | - |
The last-change entry from your NEWS, Changelog, or RPM spec file. To extract this information from a NEWS or HISTORY file, first any header lines (detected by leading spaces) are skipped; in a secfile, all lines up to and including %changelog are skipped. Then the first nonblank line (which is assumed to be the date/release information) is skipped. Then all lines before the next blank one are grabbed. |
resourcetable | - | - |
The HTML table of links to downloadable resources. This variable is only computed if the index page is built. Any setting of it in the startup files is ignored. |
extralines | - | - |
Lines to be inserted after the table of resources. You can set this; shipper may add to it to, for example, create a link to the project's Freecode page. |
logo | #Logo | XBS-Logo |
A logo or icon for the project. The default web page template will embed this in the page header. |
All these variables are available for substitution at the time a
web page or email announcement is generated. In general, any variable
you set in your ~/.shipper file will be available
at the time the web page or email announcement is generated. Use the
Python "%(variable)s" syntax, not shell-substitution syntax.
The following files are considered stock deliverables and may be shipped if they are present when shipper starts up:
| File | Explanation |
|---|---|
| README or READ.ME |
Project roadmap file. |
| tarballs |
Current source tarballs, that is any files named ${package}.*-${version}.tar.gz or ${package}.*-${version}.tgz or ${package}.*-${version}.tar.bz2. |
| RPMs |
Current RPMs, that is any files named ${package}.*-${version}.*.rpm. |
| debs |
Current deb files, that is any files named ${package}.*-${version}.*.deb. |
| zipfiles |
Current source zip archives, that is any file named ${package}.*-${version}.zip. |
| COPYING |
Project license file. |
| NEWS |
Project news file. |
| ChangeLog |
Project change log. |
| HISTORY |
Project history file. |
| BUGS |
Project bug list. |
| TODO |
Current to-do list. |
| *.{html,xhtml,css,js} |
Any files with an .html, .js, or .css extension will be shipped to all website destinations. |
Here are the generated deliverables that shipper will build and ship, if they don't exist when it starts up. Any of these that are created will be deleted after a successful upload.
| Type | Explanation |
|---|---|
| index.html |
An index web page, to be shipped to any website destination.
Only generated if the |
| SHIPPER.FREECODE |
If freecode is in the user's destination list,
shipper will generate a file called
SHIPPER.FREECODE, unless that file already exists (this is soo you
can geberate the file with shipper -n and edit it
before running without |
| SHIPPER.EMAIL |
This what shipper
generates to be emailed to destinations that are mailto URLs.
This file mill not be generated if it already exists (this is soo you
can geberate the file with shipper -n and edit it
before running without |
In operation, shipper walks through a list of destinations, building the required deliverables for each one and performing the required shipping actions to push them out to the destination. Here are the destination types shipper knows about:
| Destination Type | Deliverables | Specified by | Explanation |
|---|---|---|---|
| sourceforge | tarball, zipfile, RPMs, debs, README, COPYING, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO. | - |
If the sourceforge destination is on your list, shipper will attempt to ship a tarball and source and binary RPMs to the prject's frs directory via rsync; the deliverables will be placed in a subdirectory named after the version. It will also attempt to upload all web deliverables (README, CHANGES, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO) to the project's SourceForge web directory. |
| berlios | tarball, zipfile, RPMs, debs, README, COPYING, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO. | - |
If the berlios destination is on your list, shipper will attempt to ship a tarball and source and binary RPMs to the berlios.de submission directory via FTP. It will also attempt to upload all web deliverables (README, CHANGES, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO) to the project's Berlios web directory. |
| freecode | SHIPPER.FREECODE | - |
If the freecode destination is on your list, shipper will attempt to post a release announcement on freecode.net using freecode-submit(1). The announcement will include URLs for whichever of the following deliverables are shipped, using the URL field from your specfile: tarballs, zipfiles, RPMs, debs, ChangeLog. The user will be prompted for a Freecode release-focus. This announcement is generated into the local deliverable SHIPPER.FREECODE. |
| Generic Web site | README, COPYING, tarball, zipfile, RPMs, debs, ChangeLog, NEWS, HISTORY, *.{html,css,js}, BUGS, TODO. | scp destination ([user@]host:dir) |
This destination type represents a website.
shipper uses
scp(1)
to put deliverables on websites. If the user part of the scp
destination is absent, it will be taken from the environment variable
No generic Web sites are shipped to by default. You must declare
them by putting scp destinations in the |
| Generic FTP site | tarball, RPMs, debs | FTP URL |
Old-fashioned FTP site with no metadata. The FTP URL is parsed
to get the sitename and directory where deliverables should be dropped. The
FTP username to be used will be taken from the environment variable
No generic FTP sites are shipped to by default. You must
declare them by putting FTP urls in the
|
| Email address | SHIPPER.EMAIL | mailto URL |
The contents of the generated SHIPPER.EMAIL file is emailed to each email address specified as a destination. No email destinations are set up by default. You must
declare them by putting mailto: urls in the
|
When uploads are complete, shipper
looks at where it is running. If it can recognize that the current
directory is under version control, and the -t option
has been anabled, it will try to tag the just-shipped tree as an
external release.
Currently, the version-control systems supported for tagging after shipping are Subversion, git, hg, and bzr. Due to variations in tagging behavior in these systems, this feature behaves slightly differently depending on which one is in use.
If it is in the trunk of a Subversion repository (that is, the
current directory is name trunk and has a
subdirectory named .svn) and there is a
tags peer directory, and the tags directory does
not have a subdirectory named "$version", then
shipper copies and tags the contents of
trunk.
If it is in a git, hg, or bzr repository (that is, there is a
subdirectory named .git,
.bzr, or .hg) it will tag
the release. In all three of these cases the tag attempt will fail if
the tag previously existed.
You can change this action by setting the variable
tag_template. If you set it to None,
release tagging will be suppressed entirely. If you set it to a
nonempty string, that will become the template for the name of the
tag. "%s" in the string will be replaced with the
version number.
You can also customize the actual log message by setting the variable
tag_message. See the default with shipper
-N; the %s in it will be replaced with the version.
bzr, however, cannot annotate tags. The tag_message
variable will be ignored in this case.
The -n generates a
configuration dump to standard output.
The -u option enables uploading.
The -m option tells
shipper to generate a web page an upload it
to website destinations.
The -t option tells shipper to
attempt to tag the local repo after shipping.
The -x option specifies a destination, or a
comma-separated list of destinations, not to ship to. This overrides
the setting of the destinations variable by shippper config files and
control or spec files.
The -v option makes shipper chatty about what it's doing. The -h option prints a usage message and exits.
The following variable definition in your makefile will ensure that the makefile version is derived from (and thus always consistent with) the specfile version.
VERS=$(shell sed <*.spec control -n -e '/Version: \(.*\)/s//\1/p')
A makefile production like the following will allow you to type make release and be sure that all the deliverables shipper knows about will be rebuilt before being shipped.
release: package-$(VERS).tar.gz package.html shipper -u -m -t
You will want to change package to your project name.
The program can deliver downloadables to Berlios incoming, but not script a Berlios release. That has to be done through the web interface.
The program can put downloadables in place on SourceForge, but not set the "Preferred Download". Usually this will point to the most recent download subdirectory by default, which is the right thing.
Eric S. Raymond <esr@thyrsus.com>.
There is a project web page at http://www.catb.org/~esr/shipper/.
In late 2011 freshmeat.net changed its name to freecode.net. For backward compatibility, Freshmeat-Name is accepted as a synonym for Freecode-Name.
Support for LSM files and shipping to the ibiblio submissions directory has been removed, as that portion of its archives is now mainly historical and no longer used for distribution (the last index compilation was in 2007). Also, the "redhat" public destination is gone, as they no longer accept RPM submissions.
Trying to deduce a default whoami gave bad results and has been abandoned.
The old variable subversion_tag_template is now
tag_template.
The keywords variable is gone along with LSM
support.
The tool no longer tries to build RPMs itself, but still ships them if present.
It is no longer default behavior to build and ship a web page, or to tag after shipping.