Ceci est une ancienne révision du document !
Long ago and far away, compiling the kernel of a GNU/Linux system used to be what is called a rite of passage. You could not really call yourself a Linux enthusiast until you had painstakingly obtained the several different bits that form a working system - the kernel, the C compiler, perhaps also an X11 windowing system and several user programs - from different places on the Internet. The various bits and pieces would almost never work well together at first, so you needed to go through a process of compilation: i.e. transforming the source code of each program (in the C language) into an executable “binary file” - and the biggest and most complicated program was the system kernel itself.
It should also be said that in those good old days, most if not all Linux users were in fact computer people by trade or by interests.
Then came along several important steps towards making a GNU/Linux system more readily accessible to the average user. The first was the distribution, that collected all these software packages, compiled them in a coherent fashion and served them nicely bundled up in a CD image together with an installation program to make the process much more automatic. Slackware and RedHat were among the first distributions to make it to the general public, though many more came along afterwards.
A second important step that also sets the GNU/Linux world at an advantage from other operating systems - at least to my mind - is the package manager. Being able to install software packages written by different authors or projects, all directly from a common repository, definitely makes system software management easier on the administrator – be it of a single machine, and all the more so when a park of several hundred computers must be set up in the same fashion.
Finally, a third step happened with the Ubuntu distribution, when it changed the way the game was played and focused more closely on what in Apple's parlance is called the “user experience”: making it not only possible but easy and even enjoyable for non-technical users to set up a working box. This is not to say that previous distributions had not obtained some progress in this direction, but Ubuntu took the process a step further, with a streamlined installation program that did not require a PhD in computer science to understand, and a large selection of world languages available straight away during the installation process itself. The importance of being able to install a system with all the messages and dialogue boxes in your own language cannot be overstated. Although most computer technicians all over the world are quite capable of understanding technical information given in English, this may or may not be the case for normal people who must contend not only with the technical barrier, but also with a language they do not always fully grasp.
At this point where we stand today, each and every GNU/Linux distribution offers at least one Linux kernel, or a main default kernel plus different optional kernels for those who need them. It has been years since most of us have actually needed to compile a kernel in anger, just to make a system work. So the question can be posed: is there really any more a valid reason for a user of a modern distribution to know how to do so?
This is the point I will try to reply to in this first part of the series. We will give some insight into what a kernel is, what it does and why it may in some cases be necessary to change it. In a second installment, we will see what we need to get in order to compile a kernel, and take a first glance at the source code itself. Further along, we will actually configure and compile a kernel, and see how the result may be installed and used on our system. We will then be able to go through some simple kernel tweaks, among which the different processor options (e.g. PAE) will be discussed. This will bring us to performing some simple changes in the existing source code, and seeing what they do. Finally, we will go through creating some of our own code, in the form of a kernel module.
WHAT IS THE LINUX KERNEL?
On of the first diagrams that operating system students will see is the “onion” representing the different operating system layers. In this - very much simplified - version of the “onion”, we see the kernel at the center of the diagram. Surrounding it, we find a layer of libraries and system utility programs. Finally, the third and outermost layer is formed by user programs.
It is important to understand the purpose of each layer. The kernel itself is a very low-level piece of software; that is to say, it is in immediate relationship with the hardware and manages the most basic functionalities of the operating system. These include:
1. Managing the processes and their access to CPUs.
In a modern multiprocessing environment, computers have one or more CPUs available. Contrary to popular belief, each CPU core can in fact perform only simple tasks, and furthermore can execute only one single task in a given time-step. On the other hand, we wish to run more than one concurrent program at the same time: for example, we could very well be listening to some music with Exaile while we peruse a PDF issue of our favorite FullCircle in Evince and run an instance of Hexchat in the background. This means some part of our system needs to be available to segment each running program into short steps. Each step in then executed in turn on a CPU for a short period of time, after which it goes to sleep while other programs get access to the CPU. The process is then woken up once more and the next step is executed, and so on and so forth. The same system component that manages this will need to make sure each process gets a fair share of CPU time, that “zombie” processes are terminated, and so forth. This process management component - or “scheduler” - is part of the kernel.
2. Managing memory.
Once more, in a multiprocessing environment, each process will, at some point, require the usage of a certain amount of direct-access core memory (RAM). If we left memory management to the processes themselves, we could expect a fair amount of competition between them: who gets access to that last page of available RAM? So we need a centralized memory management system, to which individual processes apply for access to RAM. This is also a function of the kernel, which furthermore ensures each process accesses only the memory that has been assigned to it. If it should access a page of memory assigned to another process, something visibly has gone wrong and the kernel shall immediately terminate the offending process.
3. Managing access to Input/Output devices.
Much in the same way as CPUs and memory, hardware devices must be shared between many processes. For example, we could consider a USB port to which a printer has just been connected. Which process gets to manage this? It is the kernel that must recognize the type of driver required for this model of printer, activate it and grant it exclusive access to the USB port while the hardware remains connected.
All this can get quite involved as modern computers include new hardware types with passing time. So it is understandable that operating system kernels are just about the largest and most complex type of computer program the average user will encounter.
On the other hand, a kernel that works really well is a necessity for any computing device. Otherwise, conflicts between different running programs could not be solved, hardware would cease to be available to the software, the hard drives themselves could not be accessed, etc.
Going back to the “onion” diagram, each successive layer can request the services of layers situated inwards of themselves. System libraries and programs are respectively formed of libraries that contain much-used routines on one hand, and simple programs that any operating system would need on the other. An example of the first is the glibc library, that most if not all programs in a GNU/Linux will require. This contains many often-used routines such as writing a character string on screen, accessing a file, or writing to a network port. An example of a system program could be the mkfs.ext4 utility to format an ext4 partition. These libraries and programs will at some point need to access physical system resources, be it memory or a hardware device. At that point they will request this service from the inner kernel layer, using what is called a “system call”.
This request may or may not succeed, depending on whether the requested resource is at that time available to the kernel. Certain security policies may also be in place, restricting access to resources depending on the type of program on the identity of the user on behalf of whom it is executing. In any case, the program making the request cannot directly access the resource without checking whether the kernel has granted access, although some programs have been seen to do so. The ‘not taking access for granted’ is one of the differences between well-written and that less well-implemented software.
Many libraries and system programs will be needed on all computers running a given version of the operating system.
Continuing outwards in the diagram, we find the user programs. These may vary from installation to installation, depending on the specific use given to the system. They will also require the service of the layers inwards of them, both of the kernel itself and also of the system libraries. For example, a web browser will request some free memory from the kernel when it starts up, in which to store pages accessed on the Internet. But if the user should access a web page through the encrypted HTTPS protocol, the browser will also require the services of the openssl library and its routines to set up a secure channel to the server – to encrypt and decrypt data.
This explains many of the package dependencies that arise when installing new software: the maintainers of the web browser will have introduced a dependency on the openssl package, to make sure that the openssl is installed and with an appropriate version number when the browser fires up a HTTPS connection.
Some readers may have noticed that purists - such as myself - tend to refer to our operating system as the “GNU/Linux” system, instead of the more abbreviated “Linux”. This is the terminology used by the Free Software Foundation and the Debian Project, among others. It recognizes the fact that in our operating system the kernel is developed by one project, started by Linus Torvalds and hosted at www.kernel.org. This kernel is in fact the only part of the system that can be called “Linux”.
On the other hand, some of the most important bits of the operating system software have been developed in conjunction with the GNU Project at www.gnu.org, which is now sponsored by the Free Software Foundation (FSF). This includes the C language compiler, gcc. The GNU Project also has its own kernel, the GNU Hurd, which is quite different from the Linux kernel, and in some respects perhaps more advanced. So, by combining different kernels and keeping the rest of the operating system software, we can obtain our well-known GNU/Linux, but also GNU/FreeBSD with the FreeBSD kernel or GNU/Hurd that combines the GNU system software with the also GNU Hurd kernel. What does not help clarify the situation is that, for some time now, the Linux kernel is also released under the very same GNU General Public License (GPL) as the GNU Project software. So let us just take away that the kernel and the accompanying software of the GNU/Linux are released by different teams and leave it at that. Needless to say, many user programs have been developed in further projects, unrelated either to the Linux project or to GNU. Their software may be released under GPL, or other licenses such as the Apache license, the BSD license, or others - even commercial licenses.
WHY COMPILE YOUR OWN KERNEL?
Now that we know what the Linux kernel is, we can discuss a bit why it could be interesting for the user of a modern system to compile his or her very own kernel.
There are several reasons for this. A first point that must be made is that not all processors are equal. If we stay within the Intel product line, basically we can find two different CPU families. The first is based on the 80386 (or “i386”) model released in 1985. This was a 32-bit processor, meaning that numerical operations could be executed on operands 32-bits in length. It also means that memory addresses could use 32-bits, so each process could “address” (use) up to 2^32 address locations. This translates to a memory space of up to 4GBytes, which seemed extraordinarily large at the time.
Over the years, succeeding derivatives of the i386 (the i486, Pentium, Pentium Pro, Pentium II and III, the Pentium IV and finally the Atom) incorporated more and more features. However, these processors of the “Intel Architecture, 32-bits” or IA32 family maintained the backwards compatibility of their instruction set. That is to say that the i486, for example, introduced new functionality compared to the i386, and the corresponding new instructions were added. However, it could understand perfectly all the i386's instructions, so a program compiled for the i386 would use just the i386's instruction set, and work on both processors – just a mite faster on the i486.
This backwards compatibility was also maintained when AMD developed the 64-bit architecture that is now used in 64-bit personal computer systems. Processors include both AMD's own line of processors, but also Intel's Core Duo, Core i3, i5 and i7 offerings. These can work in 32-bit mode, just like a 32-bit processor – which explains why, for example, the 32-bit Windows XP could be used until recently with modern processors. However, in order to take advantage of the 64-bit instruction set, we need to compile our programs and kernel explicitly for this architecture. They will then be able to execute numerical operations on operands that are 64-bits long, and use longer memory addresses to access larger memory sizes.
GNU/Linux distributions contain kernels that are compiled for a certain model of processor. Nowadays, most 32-bit kernels are compiled using the “i686” instruction set of the Pentium Pro CPU model.
At the time of writing, the two kernel packages available for Ubuntu 14.04 are: http://archive.ubuntu.com/ubuntu/pool/main/l/linux/md-modules-3.13.0-31-generic-di_3.13.0-31.55_i386.udeb for the IA32 i686 instruction set – even though the “i386” indication has been maintained in the package names;
http://archive.ubuntu.com/ubuntu/pool/main/l/linux/md-modules-3.13.0-31-generic-di_3.13.0-31.55_amd64.udeb for the amd64 (also known as x86-64) instruction set.
This means two things: • a i686 kernel will not work on earlier models, since a i386, i486 or Pentium lack some of the instructions used. This will either simply not work at all, or may freeze during execution. • a i686 kernel will work on later models, but will not be optimized since the more recent instructions available on an Atom processor (that was released in 2008) will not be used by the kernel.
An example of this is the famous “Physical Address Extension” (PAE) instruction set. This extension of the original IA32 instruction set allowed processors to connect to and use larger memory address sizes than the 32-bit limit set by the i386.
Originally presented in the Pentium Pro generation of Intel processors, PAE became standard in most Pentium-III desktop and all Pentium-IV and Core series. This should include most personal computers that have been sold during the last ten years. So most people will not need to worry if our favorite distribution (Ubuntu) is activating PAE by default in its kernels since version 12.10, and so making PAE presence mandatory in the CPU. Ubuntu 14.04 will no longer work on processors without it, though other (earlier) distributions may run.
Even if we exclude users of really old hardware, a certain class of laptop that is still current in terms of usability suffers from the lack of PAE. Laptops built on Intel’s Pentium M (“M” for “Mobile”) processors present several advantages on later Pentium IV, M or Core CPUs. This processor class is based on the Pentium III, which is known to be internally less complex that later Pentium IVs. In practice, they compute faster when run at the same clock speed, and so are more energy efficient and manage laptop battery life better.
So it makes sense for owners of computers, such as the original eeePC or some of the first 17” laptops, to try and keep them going - especially since, with a lightweight distribution such as Lubuntu or Xubuntu, they are still well up to most normal web browsing or office tasks.
Several solutions can be found on the web, for example those described in “Enabling PAE” (https://help.ubuntu.com/community/EnablingPAE) and “Lubuntu-fake-PAE” (https://help.ubuntu.com/community/Lubuntu-fake-PAE) on the Ubuntu community documentation server. However, simply compiling and using a kernel with PAE disabled solves the problem once and for all.
The same could be said for earlier processors. The Debian project still supported i386 kernel until recently, though the new base line is the i486 instruction set (see http://www.debian.org/releases/sarge/i386/release-notes/ch-upgrading.en.html). It makes sense for distribution maintainers to concentrate their efforts on newer architectures that are more abundant in actual use, even though this means support for earlier models will slowly but surely disappear. In this case, too, running a recent distribution on older machines will imply even more often compiling your own kernel.
As for more recent machines, there are also arguments in favor of compiling your own kernel. The standard i686 kernel will work quite well on modern hardware, but will not be able to use the more recent architecture developments. This is the point of view of the Gentoo distribution, that allows the user to compile each and every software package installed (http://wiki.gentoo.org/wiki/FAQ), leading to a globally more efficient and leaner installation.
Even if we do not need a complete new kernel, in some cases, when a user needs to use relatively new hardware, it becomes necessary to compile at least the relevant driver. Graphics controllers and wireless communication devices are among the potential candidates. The new driver is a modular part of the kernel, that plugs into the existing kernel to give it the capabilities to handle the hardware.
And finally, perhaps the best reason for compiling a kernel is simply because it can be done. Few other mainstream operating system users can say they have compiled a main part of their systems – but we can. Along the way, we will also be learning a lot about how our computer and its software actually work.
THE NECESSARY HARDWARE
In the next few episodes, we will be going through the process of first obtaining the source code, and then actually compiling and installing a kernel. I will be using a fresh installation of Ubuntu 14.04 on a Core i5 laptop to perform example operations. The reader is encouraged to go ahead and do the same. However, the usual caveats apply: installing a new kernel is a major operation on your system. Though things should usually go well, there is some potential for breaking stuff, and needing to reinstall the system from scratch. So this is definitely a process that you should not do on a production machine.
On the other hand, compiling a kernel will need some raw CPU power. Though it should be possible on a low consumption processor (such as a small netbook), it will benefit greatly from a heavyweight laptop or desktop CPU. An Intel Core Duo, Core i3 or similar is probably the slowest processor that could be recommended for this purpose. You should also be aware that the source code itself and the kernel files once compiled can take up to 20GB of disk space (mostly in the /usr directory), and plan accordingly.
Whatever route you choose to go, please do make sure your own data is backed up before proceeding.