Practical Python porting for systems programmers

Use functional Python-3-style print, and float-valued division.

2to3 will try to move your code to functional print. Beware that if you apply 2to3 patches against your sources more than once, it’s going to try to add more parens than you want.

2to3 will not try to fix up your division operations. Anywhere an API counts on a result being integral you need to change to //. An important case of this is array indices; however, these are easy to spot because Python will throw a TypeError when you try to index with a float.

This import is a good idea even if your program does not presently contain print operations or divisions. In the future, someone hacking on it might introduce them and cause subtle breakage under whichever major version of Python isn’t immediately being tested.

In Python 3, the argument of an exception raise must be an object instance of a class derived from Exception. Python 2 is more relaxed. If your Python 2 code is raising something like a tuple (a not uncommon thing in older code) you’ll need to create a trivial wrapper class to hold that data and throw it instead That is, something like

needs to become

where MyException is something like this:

of course, catcher code that previously looked somewhat like this

must now look like this:

This is a no-op under 2.x in Unix; under Windows it has implications for newline handling. As previously noted, the effect under Unix is that binary I/O returns and requires byte-buffers rather than Unicode.

Most importantly, this will prevent encode/decode errors or data mangling when you do I/O on binary data.

Also note a gotcha: pickles are binary data. Change your pickle file opens to "rb" and "wb", too.

In regular expression, the semantics of some character classes (\w, \W, \b, \B, \d, \D, \s and \S) change when the string to be searched is Unicode. In Python 3 you can can defeat this by passing in the re.ASCII flag, but that flag does not exist in Python 2.

One change you can’t disable that way is interpretation of \u and \U escapes, always interpreted in Unicode literals and always not in byte-buffers.

To address all these issues, encode() the Unicode RE literal to a byte-buffer before matching or compiling, and also be sure that the data being evaluated is a byte-buffer (encoding it if necessary). Doing this keeps the semantics of your program the same across the transition, and is good practice even though systems programs are unlikely to do the sort of multi-lingual processing for which the change is an issue.

The str() and bytes() constructors do different things in Python 2 and Python 3. In Python 2 the bytes type is a synonym for str; in Python 3 it is not.

Under Python 2, bytes(n) behaves like str(), returning a string representation of an integer; under Python 3 it returns a n-byte byte-buffer. This is less likely to trip you up than the next problem…

Under Python 2, passing a byte-buffer object to str() just gives you the object back. Under Python 3 you get a string representation of the byte-buffer: thus str(b’23') == b'23'.

The insidious thing about the Python 3 stringization difference is that it also applies to the implicit stringization applied by % on strings and the format method. Suspect this is happening if your regression tests fail with extraneous instances of b appearing under Python 3.

If you were coding in Python 2 it is likely you weren’t using byte-buffer objects at all (and if you were they are identical to strings), but this problem can still pop up in Python 3 because input from binary file opens and subprocess pipes comes in as byte-buffers.

Because a Python 2 object returned by bytes() is just a string, indexing it yields a string. A Python 3 byte-buffer, in the other hand, is a sequence of integers, and indexing it yields an integer.

This can cause various obscure errors if, for examples, you index a string innocently read from a binary file or subprocess and try to combine it with another string via + or %. Under Python 2 this will work; under Python 3 you might get an error from trying to concatenate an integer with a string, or a % operation might produce an integer literal where you expected a character.

In Python 2, x / y is truncating division that yields an integer. In Python 3 it’s float-valued. In both dialects // is truncating division. Change your / divisions to //.

You can also use this:

In this case, / will always be float-valued, even on Python 2.

In Python 2 it’s still possible to do:

The code can be rewritten as:

and will work the same in Python 2 and Python 3

The reduce builtin is gone in Python 3. Import functools and use functools.reduce; this works in late versions of Python 2, as well.

In Python 3, the StringIO module is gone - it’s imported from io instead. To run in both versions you need to do this:

Python 2’s built-in "string_escape" codec does not exist in Python 3. Instead of

write

The second version works under both Python 2 and Python 3.

The pygtk bindings used by Python 2 were deprecated in 2011, and at least some Linux distributions (including Ubuntu) are not packaging them for Python 3. This means that to be polyglot you need to migrate to the new GTK bindings based on object introspection, python-GI.

Follow the general directions you’ll find here. That is, begin by making these three changes if your code is not already up to date with pygtk 2.24:

Here’s one the page doesn’t list.

You can and should test these in place before going further.

Next, apply the pygi-convert.sh script. As the directions say, this does a surprisingly good job of source conversion. However, there is one place it seriously falls down. If you had an expose_event handler, the signature it requires has changed incomatibly and needs to be fixed.

The name of the signal has changed from "expose_event" to "draw". Where it used to take a Gdk event object it now takes a Cairo context. If you were counting on that event to give you the expose-area size, you lose. What you need to do is add another handler for the allocation event. That is, your code needs to go from a setup something like this:

to something like this:

Another, minor point is that while the pygi-convert.sh script will automatically set up a Gtk import for you, it doesn’t do likewise for Gdk. So if you have, say, gtk.gdk.color_parse() calls in your code, they’ll mutate to Gdk.color_parse() but you need to add "from gi.repository import Gdk" to the import list yourself.

All these changes can be tested before you move to Python 3.

If you’re writing a systems programming tool, it probably doesn’t need to use Python’s email package; but you should be aware that the Python 3 version of this package insists on transforming your data in various ways to make it compliant with the email-related RFCs. The Python 2.7 version of this package, while it has a similar module layout, does not transform your data in the same way as the Python 3 version; so code that ran happily under Python 2.7 can break in various hard-to-debug ways under Python 3. If for some reason you need to use this package, proceed with extreme caution when trying to port your project to Python 3.

Given the same seed value, the Mersenne Twister implementations in Python 2 and Python 3 will yield different sequences of pseudorandom numbers.

This means that if you have a program that relies on reproducability of random-number sequences from a seed, you will need to roll your own random number generator.

First, have a decent regression- and functional-test suite for your program. If you don’t have one, write one now. This may sound like a step you can skip, but if you do…count on it that the Dread God Finagle and his mad prophet Murphy will cause you much more pain later in the process than you think you’re avoiding now.

This procedure assumes you are starting with your shebang line as #!/usr/bin/env python2 so you nail down what version you are testing with.

The first step is to run through the porting-issues checklist in the previous section. Doing these changes while the code is still running under Python 2 should not cause any issues under Python 3, but to be extra sure you should finish this step by temporarily changing your shebang line to #!/usr/bin/env python3 and running your tests.

Run 2to3 on your program, apply the patch it generates, and (this is important) partially revert what it does so the result runs correctly under Python 2.

For example, if your program uses the Python 2 ConfigParser library, 2to3 is going to change this to the Python 3 name configparser. What you need to do is yank that out and add this code snippet just after your general imports:

Apply a similar pattern to all the other simple library name changes; try to import the Python 3 version, and if that fails substitute in the Python 2 version.

A similar piece of magic autoadapts to the name change between Python 2 raw_input and Python 3 input. The only difference here is that Python 2 also defines input as a builtin, so to avoid colliding with it, you have to pick your own name that will point to the right function in both Python versions, and write, for example

and then replace all your calls to raw_input with calls to my_input.

As another example, the intern builtin function in Python 2 becomes sys.intern in Python 3. So:

It’s good practice to make names look like Python 3’s where possible, as in the above example; this will minimize code churn if you ever decide to leave Python 2 support behind. However, in some cases, the spelling you have to use to keep your code running on both Python 2 and Python 3 will look like the Python 2 spelling instead of the Python 3 one. For example:

The xrange builtin exists in Python 2 and has the same behavior as the range builtin in Python 3; but you can’t use the range spelling because range also exists as a builtin in Python 2, but it returns a list, not an opaque, immutable sequence. A similar strategy can be used with generators:

map and zip are builtins in Python 2 as well as Python 3, but in Python 2 they return lists, not generators. So you have to use the itertools spelling to get the generator behavior in both versions.

Sometimes you need to be a bit more fine-grained. For example, in Python 2 getstatusoutput() is a method of the commands library module; in Python 3 there is no commands and the method moves to subprocess. You can get around this by writing

and then replacing all your commands.getstatusoutput calls with getstatusoutput.

Warning: In some Python 3 versions getstatusoutput() returns status incorrectly so that a nonzero exit looks like the subprocess was signaled! (Observed under 3.4.3; Debian bug #764848) It is likely this will not affect your program unless you are trying to distinguish between these cases.

Your objective at this stage is not yet to move fully to 3, so it’s possible you might need to back out some 2to3 patchbands that make incompatible changes relating to strings and unicode. Save these, you’ll want them for a later stage.

At the end of this step, you should have a kind of amphibian - a working 2.7 program, passing your regression tests, that does Python 3 imports when run under 3 and does binary (implicitly byte-buffer) I/O. However, this amphibian probably will not run correctly under Python 3.

The reason you wanted this as a separate step is so that you can do the next bit - actually making it run under 3 - with the serious string-vs.-unicode issues separated from the syntax tweaks and import munging that 2to3 does for you.

In Python 3, the next method of iterators is renamed to __next__. 2to3 does this renaming. For compatibility to Python 2, you should add a method alias

immediately after each transformed __next__ definition, e.g.

should become

Now it’s time to tweak your shebang line to #!/usr/bin/env python3 and make that work.

This is going to consist mostly of adding encode() and decode() calls to change data between string and unicode types. This is the heavy lifting in your Python 3 port. Because of the ASCII-compatible, 0x80..0xff-preserving encoding you’ve chosen, these will be no-ops under Python 2.

The art here is in doing as little work as possible. Your encode() and decode() calls should intercept your binary I/O close to where it happens, so the bulk of your code is just seeing Unicode strings.

This is also the stage at which you may need to tag some literals with a prefix b for byte-buffer. Beware, if you have a lot of these it may mean you have not put encode/decode calls near enough to the natural choke points where your binary I/O is happening.

You may have to fix up string-to-byte concatenations as well. Again, you’ll minimize effort by moving these conversions as close to the I/O source or sink of the data as possible so that the interior computations are always done with Unicode strings.

Here is an error message you may see during conversion:

It is worth noting that the above strategy, using encode() and decode() calls with no other checking, relies on two key properties of Python 2’s handling of byte-buffer and Unicode strings:

If the above makes you nervous, however, there is a trick that avoids having to use Unicode at all under Python 2. Consider this snippet:

Under Python 2, str and bytes both refer to the same type object, so all that happens is that polystr and polybytes are aliased to that type object. But under Python 3, the polystr and polybytes functions are the equivalent of the encode and decode calls described above. So if you use polystr whenever you want to decode incoming data, and polybytes whenever you want to encode outgoing data, then under Python 2 your code will be using byte strings everywhere; it will only do Unicode conversions under Python 3. The only thing you need to decide is what to do if these functions receive an argument that isn’t a string at all. The above functions raise an exception, which is probably what you want if you want to make sure the functions only get used for the specific purpose of string data conversion. But there might be use cases where it makes sense to do something else.

(There are also polyord() and polychr() function in this wrapper; polyord() prevents the lossage that otherwise happens when calling ord() on an element of a byte buffer in Python 3, while polychr() prevents problems going in the opposite direction.)

Another item in this code snippet is worth noting: under Python 3, when constructing the alternate I/O streams (note that the above snippet doesn’t do anything to them under Python 2), you have to set the newline parameter to \n, as shown, or you will have problems with the way Python handles line breaks in your data. There are actually two issues here. The first is newline translation: by default, Python 3 opens text files in “universal newlines” mode, in which it automatically translates all non-Unix newline markers it finds (i.e., DOS-style \r\n newlines and MAC-style \r newlines) into its chosen newline marker for internal operations, which is the Unix newline, \n. Once the translation is done, there’s no way to recover the original newlines. Obviously you don’t want this default behavior.

You can stop Python from translating line breaks when reading files by passing any value for the newline parameter except None. However, when writing files, Python will translate newlines if you pass anything but a blank string '' or \n as the newline parameter. If you pass None, or accept Python’s default behavior, any \n characters will get translated to the system default line separator, os.linesep, on writing. If you pass the DOS or MAC newline, any \n characters will get translated to that newline. This behavior is rather counterintuitive; you might think that, if your data has all DOS newlines, you would want to tell Python that by passing newline="\r\n" when writing a file. In fact, what that will do is make Python translate every \r\n to \r\r\n when writing the file! This is because, when writing, Python just looks at the \n, interprets it as a newline (since that’s its internal newline character, as above), and translates it to \r\n.

Further, there is the second issue, which is string operations. If you pass Python anything but \n as the newline parameter to a file you open for reading, then line-related operations on that file, such as readlines() or for line in file, will break lines at markers other than \n. However, once the incoming data from that file is stored as a Unicode string, any line-related operations on that string, such as splitlines(), will only break lines at \n. So using anything other than \n as the newline parameter creates a mismatch between the way Python processes text files and the way it processes text strings.

You could conceivably try to work through all this, but it’s much better to just avoid the problem by using \n as the newline parameter for all files, and accepting that all your program’s internal data will be using \n as the newline marker, in accordance with Python’s internal data model.

Finally, if your program is going to be used interactively, you will want to set line_buffering=True, as shown in the code above, so that interactive sessions will work as expected.

Remember that in Python 2 the sort() method takes a two-argument comparison function, but in Python 3 it takes a "sort key" function which must return a comparison value and be passed in as the value of the keyword argument "key=".

If you’re sorting numbers or strings or anything else for which there is a a strict ordering in Python, your key function can just be lambda: x x. (This is actually the default behavior if you don’t specify a key.)

If you’re sorting on tuples and want the usual behavior (earliest tuple member is most significant for sort value) remember that Python uses a stable sort, so sorting on each tuple member in turn will give the desired effect.

Let’s say you already had a sortvalue() function for the objects you’re sorting. Then polyglot code will, at worst, look like this:

Actually, since Python 2.4, the sort builtin accepts a named "key=" argument and "sequence.sort(key=sortvalue)" will do.

Let’s say you have some code like this.

By default, when you run 2to3, your code will be changed to:

This is because in Python 3, keys() returns a dictionary view, which is different from the list you get in Python 2, and is also different from the iterator you get with iterkeys() on Python 2

But in most cases, you just want to iterate over the keys, so we recommend using 2to3 with --nofix=dict.

Be careful though, code will blow up if you have something like:

That’s because dictionary views do not have a sort() method.

Instead, write something like:

Another gotcha is when you change the dictionary:

Here there’s no choice but converting to a list:

In some older (pre-2.6) versions of Python 2, in order for a class to be thrown or caught as an exception value, it had to inherit from exceptions.Exception, and the exceptions module had to be imported to support that.

In Python 3, the exceptions module does not exist. Instead, exception classes must inherit from a builtin class called BaseException.

In Python 2 at versions 2.6 and later, it was possible to inherit from BaseException as in 3, but the older way of inheriting from exceptions.Exception also worked.

If you have exceptions in your imports remove it. Then, make your exception classes inherit from BaseException.

Now you can hack the shebang line so it says #!/usr/bin/env python confident that the program will work whether the environment defaults to Python 2 or Python 3.

Final step: tweak your regression tests so that they run twice, once under python2 and once under Python 3. This way you’ll avoid unpleasant surprises during later modifications.

The setpython script can be some help with this:

This helps you redirect a #!/usr/bin/env python shebang to your choice of a Python 2 or 3 version without actually modifying the Python code. To use it, ensure that the $PATH seen by your test scripts includes . (the current directory) so ./python will intercept the python invocation in the shebang.