In a recent statement (commented here: http://fullcirclemagazine.org/2016/05/04/no-ubuntu-back-doors-windows-and-mac-migrations/) Mark Shuttleworth spoke about increasing the size of Ubuntu desktop images - those ISO Live CD files we download to install our system. The current limit is nominally at 1 GByte, and should go up to 2 GBytes or more. Is this a rational decision – or not, and what does it mean for us users?

Dans un récent article (commenté ici : http://fullcirclemagazine.org/2016/05/04/no-ubuntu-back-doors-windows-and-mac-migrations/) Mark Shuttleworth parlait d'une augmentation de la taille des images d'Ubuntu desktop [version de bureau] - ces fichiers ISO Live CD que nous téléchargeons pour installer notre système. La limite actuelle est de 1 Go et pourrait passer à 2 Go ou plus. Est-ce une décision rationnelle - ou pas, et qu'est-ce que ça signifie pour nous, utilisateurs ?

Modern partition schemes In the first place, perhaps it may be useful to take a look at the hard drive layout used by the system, once installed. Most desktop and laptop computers - those used directly by individuals as opposed to servers - have a hard drive with several partitions. Ubuntu can be installed in a single partition (e.g.: /dev/sda1), especially if the btrfs filesystem and subvolumes are used. In most systems, however, partitioning schemes use several partitions. One partition for the system, and a second for user data is commonplace: /dev/sda1 → mounted on / = system partition /dev/sda2 → mounted on /home = user data On servers, typically other directories (such as /var and /opt) will reside on further partitions, making system administration easier and more foolproof since errors that affect one partition are more easily confined to it. An example would be running out of space, which can happen with /var if a daemon runs away on us.

Schémas modernes de partitionnement

En premier, il est peut-être utile de regarder la disposition d'un disque dur utilisé le système, une fois installé. La plupart des ordinateurs de bureau et portables - -ceux qui sont utilisés directement par des individus, à la différence des serveurs - ont un disque dur avec plusieurs partitions. Ubuntu peut installer dans une seule partition (par ex. : /dev/sda1), spécialement si le système de fichiers et les sous-volumes btrfs sont utilisés. Cependant, dans la plupart des systèmes, les schémas de partitionnement utilisent plusieurs partitions. Très souvent, ce sera une partition pour le système et une seconde pour les données de l'utilisateur.

/dev/sda1 → monté comme / = partition système

/dev/sda2 → monté comme /home = données utilisateur

Sur les serveurs, d'autres répertoires (comme /var et /opt) résideront dans des partitions supplémentaires, rendant l'administration du système plus facile et plus sécurisée car les erreurs qui n'affectent qu'une partition sont plus facilement restreintes à celle-ci. Un exemple pourrait être une dépassement de capacité, ce qui peut arriver si un démon perd ses repères.

This tendency to split the file system up into distinct partitions is also applicable to devices. On a modern mobile device, the internal Flash-based storage space is organized in a similar fashion to a computer’s hard drive, with a partition table and separate partitions for different purposes. For example, a typical Android-based device with 32 GBytes capacity was structured as follows: $ df Filesystem Size Used Free Blksize /system 343M 290M 52M 4096 → APP = /dev/block/mmcblk0p3 /data 27G 13G 14G 4096 → UDA = /dev/block/mmcblk0p10 /system/vendor 343M 157M 185M 4096 → FLX = /dev/block/mmcblk0p6 Here, we can identify the Flash device used as /dev/block/mmcbk0 (Memory Block device 0), where partition 3 is taken up by the operating system itself, partition 10 is user data, and partition 6 holds vendor-specific additions such as additional applications.

Cette tendance à découper le système de fichiers en partitions distinctes est aussi applicable aux dispositifs. Sur un mobile moderne, l'espace de stockage en mémoire Flash est organisé de manière similaire au disque dur d'un ordinateur, avec une table de partition et des partitions séparées pour les différents besoins. Par exemple, un appareil typique sous Android avec une capacité de 32 Go était structuré ainsi :

$ df Filesystem Size Used Free Blksize /system 343M 290M 52M 4096 → APP = /dev/block/mmcblk0p3 /data 27G 13G 14G 4096 → UDA = /dev/block/mmcblk0p10 /system/vendor 343M 157M 185M 4096 → FLX = /dev/block/mmcblk0p6

Voici. Nous pouvons identifier que le module flash était utilisé comme /dev/block/mmcbk0 (Dispositif de bloc mémoire 0), où la partition 3 est prise par le système d'exploitation, la partition 10 par les données de l'utilisateur et la partition 6 contient les compléments propres au fournisseur comme des applications complémentaires.

Such a layout may seem excessively complex at first glance. However, it does retain the advantage of being able to alter just one of the partitions without affecting the others. A system upgrade, for instance, may alter only data in partition 3 (/system), while affecting neither user data nor vendor-specific applications. Ubuntu Snappy and Touch installations are organized along the same lines. For example, a 4 GByte Ubuntu Snappy layout was as follows: /dev/sda1 (8M) = GRUB area, flagged bios_grub, “grub” /dev/sda2 (63M, 52M used, fat32) → /boot/efi = boot partition, “system-boot” /dev/sda3 (976M, 612M used, ext4) → / = system area, “system-a” /dev/sda4 (976M, 1.3M used, ext4) → /writable/cache/system, “system-b” /dev/sda5 (1.6G, 40M used, ext4) → /oem = application area, “writable” Here, the main system hard drive /dev/sda contains two system partitions (2 and 4), while partition 5 is for user-installed applications.

Une telle disposition peut sembler extrêmement complexe à première vue. Cependant, elle conserve l'avantage d'être capable de supporter une altération d'une des partitions sans affecter les autres. Une montée de version du système, par exemple, peut dégrader uniquement les données dans la partition 3 (/system), sans affecter, ni les données utilisateur, ni les applications propres au fournisseur.

Les installations par Snappy et Touch sont organisées suivant les mêmes orientations. Par exemple, une disposition de 4 Go avec Snappy d'Ubuntu était comme suit :

/dev/sda1 (8M) = zone de GRUB, marquée bios_grub, “grub”

/dev/sda2 (63M, 52M utilisés, fat32) → /boot/efi = partition boot, “system-boot”

/dev/sda3 (976M, 612M utilisés, ext4) → / = zone système, “system-a”

/dev/sda4 (976M, 1.3M utilisés, ext4) → /writable/cache/system, “system-b” (inscriptible, cache)

/dev/sda5 (1.6G, 40M utilisés, ext4) → /oem = zone des applications, “writable” (inscriptible)

Ubuntu system images What, precisely, is an Ubuntu image? In essence, we are speaking of a file system, similar to what would be found on a computer’s system hard drive, but packaged into a single file instead of occupying sectors spread over a physical hard drive. Basically, we have a lot of files within a single file. This has a few advantages. A single file may be downloaded, instead of having to synchronize a complete file system with all its directories and subdirectories. It is also easy to compress this file, taking up less space. This is what already happens within a ZIP or RAR compressed file. In the traditional UNIX compressed file, two distinct tools are used: tar to group together directory contents into a single file, and then either gzip or the more modern bzip to compress it, giving the .TAR.GZ (.TGZ) or .TAR.BZ extension combinations. Typically nowadays, a compressed system disk occupies about 500 MBytes to 1.5 GBytes, which decompresses to 2 - 4 GBytes. Naturally, the actual numbers depend on the file system contents. Thus, the rationale behind Canonical’s decision is mostly the wish to include more software packages than previously available on the installation image itself.

Images du système Ubuntu

Qu'est-ce qu'une image Ubuntu, précisément ? En gros, nous parlons d'un système de fichiers, similaire à ce que nous trouvons dans le disque dur d'un ordinateur, mais empaqueté dans un seul fichier plutôt qu'occupant des secteurs divers sur un disque dur physique. De base, nous avons un grand nombre de fichiers dans un seul fichier. Ceci a quelques avantages. Un fichier unique peut être téléchargé, plutôt que de devoir synchroniser un système complet de fichiers avec tous ses répertoires et sous-répertoires. Ce la facilite aussi la compression du fichier, prenant moins de place. C'est ce qui arrive déjà à un fichier compressé en ZIP ou en TAR. Dans le fichier compressé UNIX traditionnel, deux outils distincts sont utilisés : tar pour regrouper les contenus des répertoires en un seul fichier et ensuite, soit gzip, soit le plus moderne bzip, pour le compresser, donnant les combinaisons d'extensions .TAR.GZ (.TGZ) ou .TAR.BZ.

Typiquement de nos jours, un disque d'un système compressé occupe de 500 Mo à 1,5 Go, qui se décompresse en 2 à 4 Go. Naturellement, la taille réelle dépend du contenu du système de fichiers. Par cons&quent, le rationnel derrière la décision de Canonical est principalement le souhait d'inclure plus de paquets logiciels que précédemment disponibles dans l'image d'installation.

One may ask for what reason an Ubuntu system image is not distributed in one of these compressed file formats. This would actually be possible, at least as far as the file system itself is concerned. However, the end result, once downloaded, would somehow need to be made bootable so we can actually start the computer from it, and launch the installation process. Further information must be included within the compressed file so that the BIOS can detect and boot the image, once written to a physical medium (optical disk or USB drive). The simplest tool to do this is an IMG file. This can be seen as a byte-for-byte clone of an existing hard disk. The distribution manager creates a working system or “Gold Master” (in Apple’s terms), which is then cloned into a file for distribution, usually in compressed form. On the receiving side, the user will decompress and copy back the bytes from the IMG file onto the chosen support. This works well when working with USB drives as a boot medium for a computer, or when flashing a mobile device such as a tablet or phone.

Pour quelle raison, peut-on se demander, une image du système Ubuntu n'est pas distribuée dans un de ces formats de fichier compressé ? Cela serait vraiment possible, au moins pour autant que le système d'exploitation est concerné. Cependant, le résultat final, une fois téléchargé, devrait être rendu amorçable de façon à démarrer à partir de lui, et lancer le processus d'installation. Des informations supplémentaires doivent être incluses dans le fichier compressé de sorte que le BIOS puisse détecter et démarrer depuis l'image, une fois écrite sur un support physique (un disque optique ou une clé USB).

L'outil le plus simple pour le faire un fichier IMG. il peut être vu comme un clone octet-par-octet d'un disque dur existant. Le gestionnaire de la distribution crée un système fonctionnel ou “Gold Master” (dans les termes d'Apple) [Ndt : le système-père], qui est ensuite cloné en un fichier de distribution, habituellement sous forme compressée. A la réception, l'utilisateur décompressera et copiera les octets du fichier IMG vers le support choisi. Ceci fonctionne bien en tavaillant avec des clés USB comme support intermédiaire pour un ordinateur ou en flashant un appareil mobile tel qu'une tablette ou un téléphone.

The IMG file can be an image either of a single partition, or of a complete drive with several partitions and a partition table to identify each. In most cases, an IMG file for an Ubuntu bootable USB contains a single partition, unlike OpenSUSE images which contain two. IMG files for flashing devices with Ubuntu Touch have their own particularities, as evidenced here: # file vivid-preinstalled-boot-i386+generic_x86.img vivid-preinstalled-boot-i386+generic_x86.img: Android bootimg, kernel (0x10008000), ramdisk (0x11000000), page size: 2048 # file vivid-preinstalled-system-i386+generic_x86.img vivid-preinstalled-system-i386+generic_x86.img: Android sparse image, version: 1.0, Total of 51200 4096-byte output blocks in 1698 input chunks.

Le fichier IMG peut être l'image soit d'une partition unique, ou d'un disque complet avec plusieurs partitions et une table de partitions pour les identifier. Dans la plupart des cas, un fichier IMG pour un clé USB bootable avec Ubuntu contient une partition unique, alors que les images OpenSUSE en contiennent deux. Les fichiers IMG pour les dispositifs flash avec Ubuntu Touch ont leurs propres particularités, comme vous pouvez le voir :

# fichier vivid-preinstalled-boot-i386+generic_x86.img

vivid-preinstalled-boot-i386+generic_x86.img: image bootable Android, noyau (0x10008000), ramdisk (0x11000000), taille de page: 2048

# fichier vivid-preinstalled-system-i386+generic_x86.img

vivid-preinstalled-system-i386+generic_x86.img: image clairsemée Android, version: 1.0, total de 51200 blocs de sortie de 4096 octets en 1698 chunks d'entrée.

ISO files are another beast altogether. ISO files are designed as a byte-by-byte copy of the contents of an optical disk, be it CD or DVD, which gives them several particularities. One is size limits: once upon a time, CDs were limited in capacity to 700 MBytes. The ISO 9660 standardized the filesystem used, with quirks that harken back to the the MS-DOS era such as filenames limited to the 8.3 format (eight characters for the file name, three characters for the extension); extensions of the standard allowed longer filenames. The “El Torito” extension, for example, allowed CDs to be used as a boot medium for computers. It should be noted that this was not a feature of the original filesystem, but rather an addition that at first inserted a floppy-disk image of 1440 kBytes that computers detected and booted from. Later DVD formats evolved somewhat, with capacities ranging from 4.37 GBytes (DVD-5 single-layer) to 7.95 GBytes (DVD-9 double-layer). Disks with higher capacities have been produced, but have not really gone mainstream for computer applications. The more modern Universal Disk Format (UDF) filesystem is now commonplace.

Les fichiers ISO sont tout-à-fait autre chose. Les fichiers ISO sont conçus comme des copie octet-par-octet des contenus d'un disque optique, CD ou DVD, ce qui leur donner certaines particularités. L'une est la limite de taille : autrefois, les CD étaient limités à 700 Mo. La norme ISO 9660 avait standardisée le système de fichier utilisé, avec des bizarreries qui remontent à l'époque du MS-DOS où les noms de fichiers étaient limités au format 8.3 (huit caractères pour le nom de fichier, trois caractères pour l'extension) ; des extensions de la norme ont autorisé des noms de fichiers plus longs. L'extension « El Torito », par exemple, permettait aux CD d'être utilisés comme support de démarrage sur les ordinateurs. Notons que ce n'était pas une fonctionnalité du système de fichier d'origine, mais plutôt un ajout qui, au début, insérait une image pour disquette de 1440 Ko que les ordinateurs détectaient pour démarrer de là.

Plus tard, les formats des DVD ont évolué un peu, avec des capacités s'étendant de 4,37 Go (DVD-5 à simple couche) à 7,95 Go (DVD-9 à double couche). Des disques avec des capacités supérieures ont été fabriqués mais n'ont pas rencontré un grand succès dans les applications d'ordinateurs. Le système de fichiers plus récent Universal Disk Format (UDF - Format universel de disque) est maintenant d'usage courant.

These restrictions on the physical medium and file system -with the desire to remain compatible with older computers- have limited the freedom of movement of the producers of GNU/Linux system images. The image file needs to be produced in precisely such a way as to be bootable on a variety of systems - and the more recent EFI BIOS has complicated matters just a tad more. To simplify things a bit, what one finds in a recent Ubuntu ISO file is the following: # mount -o loop ubuntu-16.04-desktop-amd64.iso /mnt mount: /dev/loop0 is write-protected, mounting read-only # mount | grep ubuntu ubuntu-16.04-desktop-amd64.iso on /mnt type iso9660 (ro,relatime) # df -lh | grep loop /dev/loop0 1.4G 1.4G 0 100% /mnt # ls /mnt boot dists install md5sum.txt pool README.diskdefines casper EFI isolinux pics preseed ubuntu

The basic file system is the ISO9660 compatible with the original CD format, though grown in size from the original 700 MByte limit, up to about 1.4 GBytes. The directory structure contains several specific directories such as isolinux for the Isolinux boot system for older BIOS, and EFI for the newer EFI-capable computers. The one that really concerns us, however, is the /casper directory. It contains the main contents of the ISO image such as the Linux kernel (vmlinux.efi), the initial RAM disk (initrd.lz), and also a very large file, filesystem.squashfs. # ls -lh /mnt/casper/ -r–r–r– 1 root root 1.4G apr 21 00:25 filesystem.squashfs -r–r–r– 1 root root 26M apr 21 00:25 initrd.lz -r–r–r– 1 root root 6.7M apr 21 00:25 vmlinuz.efi This SquashFS file is the actual file system that is decompressed in RAM and mounted on / within the Live CD upon boot. We can see this from within the Live CD itself: ubuntu@ubuntu:~$ mount /dev/sr0 on /cdrom type iso9660 (ro,noatime) /dev/loop0 on /rofs type squashfs (ro,noatime) /cow on / type overlay (rw,relatime,lowerdir=//filesystem.squashfs,upperdir=/cow/upper,workdir=/cow/work)

The first entry here is the ISO image itself, giving access to all its files - including the /pool directory. Naturally, it is mounted read-only “ro”, and no changes can be made to its contents from within the newly booted Live CD. /rofs is the SquashFS, still in compressed form. This is also read-only, by design as well as the fact that it resides in an ISO image. Finally, the /cow bit is the fascinating piece of technology that makes it possible for the complete Live CD construction to work. In a GNU/Linux system, some parts of the file system need to be mounted read-write - otherwise, a modern system simply will not boot up completely. /cow implements an overlay file system, in which two separate file systems are combined. The lower part is comprised of files and directories that form the base of the file system, and that will not be modified during normal execution. These reside in the SquashFS. On the other hand, an upper layer holds the files and directories of the root file system that have changed during execution. This can be seen as similar to a delta backup, where only changes from an initial state need to be recorded. The overlay system makes the fact that a file resides in the lower or the upper layers completely transparent to the user. On a system booted from a Live CD that is, in fact, a physical DVD or an ISO image booted in a virtual machine, the boot medium cannot be altered. In these cases, the upper part of the overlay file system is maintained in RAM, and will be lost when the machine is shut down. If, on the other hand, the boot medium is a writable device such as an USB stick, persistency can be implemented – by storing the upper layer in a special file on the device, and changes can be preserved from one boot to another.

System installation Once the Live CD has been booted, the user can go on to install the system onto the computer’s hard drive. What actually happens is quite different if the system image is a server image, or a standard desktop image. In the case of a server image, Ubuntu Server goes the route of the traditional Debian install ISO. In this case, a minimal bootstrap environment is transferred onto the computer’s hard drive, and then software packages in the DEB format are installed on that drive using apt-get commands much in the way an administrator could do by hand. The only difference is that many of these packages are files located on the CD itself, within the /pool directory. These may be supplemented by other package files brought down from the repositories on the Internet if a network connection is available.

On the other hand, an Ubuntu desktop image contains a large selection of software, already installed in the SquashFS file system: applications such as Firefox, Libreoffice and the desktop environment itself (Unity, Gnome or whatever). It would be wasteful to have these programs in software packages on the CD as well, so a different installation strategy is preferred. The main root file system is simply cloned over on to the computer’s hard drive, and then specific changes are performed - such as creating new user profiles, tweaking GRUB, etc. Even so, when installing a desktop system, an attentive user can see how the installation process finishes up by downloading software packages with newer versions of some applications if available, and also removing some software packages such as language packs that the user has not chosen.

Some final thoughts When Mark Shuttleworth was speaking about having large installation images (ISO files) for future versions of Ubuntu, the basic aim seems to be to make a larger collection of software available straight from the installation medium. This would have some consequences, depending on the precise way in which the ISO file is laid out. On the one hand, the amount of applications within the Live system can be increased. This would mean a larger SquashFS file - but also a larger amount of space used in a freshly installed system’s hard drive. In this day and age, this is probably not a concern for most people; going up from Ubuntu’s 3.5 GByte disk usage on a new system to -let’s say- about 5 or 6 GBytes would be feasible nowadays even on machines with restrictions on disk space, such as SSD hard drives or virtual machines. These applications would be available from the Live CD environment, and also on the new system once installed, which would make things easier both for users who wish to explore software choices that they are perhaps not familiar with, or for users with constraints on downloading software packages from the Internet. However, it could also be an inconvenience for “normal” users, whose computers would be updating a larger selection of software on a regular basis, unless some of these applications were promptly uninstalled. Imagine, for instance, having both the Mozilla Firefox and Google Chrome browsers installed: both these programs are rather large, and regularly updated. Most of us opt for either one or the other. Is there a real reason to download upgrades for both on a monthly basis?

Another option would be a hybrid approach. The basic SquashFS would contain just the basic system and desktop manager, with a larger selection of software packages available in DEB files within the CD image’s /pool directory. Some sort of selection procedure would then need to be put in place during installation - as in the current Ubuntu Server. This would mean only software that the user really wishes to install ends up on the final system. However, it would also mean that some of this software would not be available from within the Live environment, and could not be tested out before installation to the hard drive. The installation procedure would also require a supplementary stage, which may or may not be agreeable to first-time installers. Which way Canonical goes on this one will obviously depend on their take on which alternative is best for most users. The beauty of the ISO file structure that has evolved to create the Live CD we all know is its flexibility. Different software choices in the SquashFS give us, today, the means to build the installation images for the different flavors of Ubuntu: Ubuntu itself with Unity, Kubuntu, Xubuntu, Ubuntu-gnome, etc. Remastering the SquashFS with a different set of applications installed is, in itself, not a big deal. So it would be conceivable for Canonical to supply not one, but several desktop installation ISO images: one with a bare system that the user will deck out with applications from the repositories online, another with all the bells and whistles already installed and ready to test, a third with several desktop manager options (Unity + Gnome, Unity + XFCE + MATE), etc. In any case, it seems rather certain that one of these images will be chosen over the others and proposed as a “main” image, the go-to ISO file to download for users unsure of their choice and its implications. The Ubuntu distribution has always been about making it simple for new users; it would perhaps be best to continue on this track.