There are actually two layers within a running X server itself: a device-dependent display driver and a device-independent protocol engine. On startup, the X server determines which device driver to load by looking at its configuration file. Thereafter, from the point of view of anything above the X server in the stack, the display driver is almost invisible; all it does is supply a small set of calls that manipulate the display bitmap.
The X library linked to the client, Xlib, abstracts away the presence of the X protocol connection to the server and the individual characteristics of the underlying hardware. From the point of view of any program calling the client X library, the hardware is represented by a C structure called a graphics context; library operations manipulate that context, and there is an update call that tells the library to ship back to the X server the operations necessary to make the physical screen state match that of the graphics context. (Some library operations also finish by forcing an update.)
The X design uses several tricks to cut latency in screen updates. One is the Shared Memory Extension: if application and X server are running on the same hardware, X detects this and they communicate using a shared-memory mailbox, bypassing the TCP/IP stack entirely. Another is that the X protocol design is contrary to the normal Unix practice of loose, textual application protocols optimized for debuggability; instead, it uses a tightly-packed binary protocol to reduce datagram sizes per transaction to a bare minimum. Finally, the X server also bypasses the operating-system kernel; at startup time it requests access to I/O ports that go direct to the graphics card's video RAM, rather than using the Unix read/write. system calls. Thus, X avoids context-switching on every update.
The Xlib API is unpleasantly complex (less charitable but not infrequent descriptions run along the lines of “a baroque nightmare”). There are entire geological strata inside it full of fossils of optimization hacks that made sense in an environment of expensive RAM and slow hardware but are no longer necessary. For example, many of the Xlib calls are duplicated in clipped and unclipped variants, a leftover from days when clipping was an expensive software operation rather than a standard feature of graphics-card firmware.
Other complexities address problems that are still live but have better solutions independent of X than existed when the X design was laid down in the 1980s. Representative of these are the X security and access-control features. Under modern Unixes this notoriously messy and poorly-documented part of the system has been largely superseded by tunnelling the X protocol through ssh(1), the Secure Shell. ssh(1), an open-source tool for encrypting network connections. ssh(1)specializes in encryption and access control, and is easier to configure and administer than the native X features.
A few of the nasty spots are complex and unsatisfactory solutions to problems for which nobody has yet devised simpler and more satisfactory ones. Prominent among these is the X resource system. This allows users of X applications to set up name/property pairs which the X server digests and can use to alter the behavior of the applications running on it, (Common per-application resources include, for example, foreground and background colors for screen displays.) The problem with the resource system is that the server can collect name/value pairs from so many different sources that it is often difficult to tell where a given setting came from or what the effect of changing it will be. Modern practice tends to avoid using the resource facility entirely in favor of specialized per-application configuration files.
Despite these flaws, X has had no serious challenger for the role of Unix graphics engine in the last decade. A handful of attempts, such as the Fresco project (originally the “Berlin” project) have come to nothing. The Aqua libraries layered over the Unix Foundation of Mac OS X are X's only functional competition — but as single-vendor proprietary code they are a non-starter anywhere but on Apple machines, and can never expect to attract X's volume of cross-platform developer support. In recognition of this fact, Apple has added an X server compatibility feature to Aqua so that “legacy” X applications can run under it.
After several years of political and technical stagnation during which many Unix insiders worried that the codebase might have aged and complexified beyond salvage, X server development is undergoing something of a renaissance at time of writing in 2004. A new rendering engine based on Porter-Duff composition rather than the traditional BitBlt has been successfully implemented, and has been used to demonstrate dramatic new capabilities which should soon make their way into production versions.
The most visually obvious of these is full alpha-channel support. This means that upcoming X servers will make it easy for toolkit libraries to stack transparent and partially-transparent widgets on the screen, with lower layers correctly blending with the layers above them. This feature will make X fully competitive with Aqua, which has used alpha transparency to beautiful effect on recent Macintoshes.
Just as importantly, though less visibly, the re-architecting of the X server carries the original X concept a step forward by breaking the X server into communicating pieces. Image-composition logic will be separated from the server core that drives the graphics hardware. The next generation of X servers will be both more capable and less complex than today's, while still retaining backward compatibility with the X application base.