Ceci est une ancienne révision du document !
Like most users, I do change my main working computer from time to time. When doing so, it may be a good time to start out afresh and give some thought to our hard disk setup. Even if cloud storage is widely used, limits on amounts of storage available (or its monthly fee), and network bandwidth to get our files up into the cloud, mean most users actually keep a sizable portion of their files on their local drives. Security considerations may also be a factor to consider, even with smaller files. In this article, I would like to set out several ideas that a more or less technical user could consider, strategies that may be of help both for data deduplication on your local drive, or for extending a drive with a limited capacity we are growing out of.
Why mirror data?
Making multiple copies of all files that are written to our computer’s hard drives is a technique used in all server environments; every time a file is modified, alterations are written to more than one disk using one of several different schemes. The thinking being that, should one of these drives stop working suddenly: the second copy would not only ensure no data is lost, but, moreover, that service could continue immediately with no interruptions. Even when making a more pedestrian use of our personal computers, at some point we need to recognise that hard drives can, and do, fail.
Perhaps things are not as critical with modern SSD technology as they were at one point with rotational drives and their moving disks. The Mean Time Between Failures (MTBF) is a commonly-used metric that seems to have gone up with the switch to solid-state technology. But still, common sense tells us it is perhaps best not to entrust all our data to a single copy on our internal drive without making any backups. Fine… but there still are obstacles. In practice, not all backups are made on a schedule and kept up-to-date. At times, large files may never even get backed up at all – neither to the cloud because they are too large, nor to a local external drive because we would need to think of plugging it in and performing the backup. Since, in the immortal words of Murphy’s Law, anything that can go wrong will do so and at the worst possible time, it can make sense to have multiple copies of all our data, made automatically, on our daily driver computer.
Clearly, making several copies of our files on a single disk would not be very useful at all. In the event of that disk having a catastrophic physical failure, we would lose access to all data it contained and the fact that it held several copies of each file would be moot. So, our goal must include the use of more than one physical hard drive on our computer. And, since messing around with the hard drives on our daily driver is probably not a great idea, what better time to take a look at the possibilities than when acquiring a new (or new-to-us) machine?
Why just mirror data?
The term “mirrored drive” implies the use of exactly two drives of a certain capacity. All data is written to disk in two copies, one on each disk. So, in essence, we are buying (and probably paying for) twice the disk space, and then using only the capacity of a single unit giving us a disk use efficiency of a mere 50%.
In server rooms, this would be considered wasteful, since techniques such as RAID-5, RAID-6 or others allow higher efficiencies – given a sufficient number of disks. For instance, with an 8-disk setup and using single parity, a RAID-5 would give us a useful space of 7 times the capacity of a single disk, while buying eight times that worth of space. 7/8 or 87,5% disk usage efficiency, while retaining the ability to survive the death of any single disk without losing any data is not a bad result. However, the immediate objection is that while it may be practical to house a large number of disks in a server rack, things are a lot more complicated in a typical laptop, or even some desktop computers with slim form factors. Even the smaller grade of commercially-available Network Attached Storage (NAS) units tend to have receptacles or bays for only two hard drives, thus limiting us to mirroring or – in technical parlance – RAID-1.
On most desktop machines, the motherboard will have at least two SATA interfaces, where two rotational drives or two SSD drives with SATA interfaces may be plugged in. Even on many laptops, a second SATA hard drive may often be installed. If the laptop comes with an optical drive, a possibility would be to replace this little-used piece of equipment with a hard drive carrier that just plugs in the same space the DVD used to take. This is actually quite useful, mainly with laptops with the larger form factors.
Modern technology can also give us the possibility of using M2 interface SSD drives. These take up quite a bit less space than rotational drives with the SATA interface. Many laptop motherboards either come equipped with a disk drive with this interface, or an available M2 slot on the motherboard. However, most laptops will only have a single M2 interface, so some creativity may be necessary to combine, for instance, one SSD on the M2 interface with a second one on a SATA interface. This setup would not be advisable from the standpoint of speed, but should be doable if data redundancy is the primary objective and maximizing throughput is not (an SSD drive should be plenty fast, even on a SATA interface).
On the other hand, desktop machines can use a PCI-Express adapter (extension card) to add an M2 interface to an existing machine. Some adapters are for a single M2 drive, while others even have space for two M2 interfaces thus allowing the user to build a two-drive system on a single adapter. What is more, the cost can be quite moderate, with some dual M2 interface adapters available for as little as 17 USD. However, maybe ensure your computer can boot off this extension card before acquiring one.
As a final thought, it may be useful to remember that using two partitions situated on the same physical disk would not be a good idea: not only will access times be lengthened since all data needs to be written twice to the very same disk thus creating a bottleneck, but if that unit goes out physically, both partitions will be lost at once thus rendering mirroring completely useless as implemented. To sum up: mirroring requires the use of (at least) two physical disks or drives.
Why extend an existing disk (striping) instead of mirroring?
For some people, mirroring data may not even be necessary for the user’s workflow. This would be the case when the user actually generates most data in the cloud – or on a company server, and the local hard drive is seen more as an archive to have a local copy of files in case the remote server is no longer available or some space needs to be freed up. Another scenario would be the “perfect” user who has one copy of each file locally, one copy on a remote server, and a third on cold storage, for instance on an archived external disk or optical media. Finally, we may at some times need a large amount of disk space to store files for a relatively short period of time, after which this large amount of data will be discarded. This is usually the scenario found while editing video, for instance, where the final product would certainly be kept, but perhaps not most of the original material if it is no longer relevant.
In these cases, would it not make more sense to use several drives to extend the total amount of space available, while retaining just a single copy of each file. In essence, we are trading some data security for available space. In the same way, the concept of mirroring deals with combining several hard drives into a single unit in which files are stored in two copies, striping is the analogous concept of combining several drives into one unit with a larger capacity but without duplication. In technical terms, where mirroring is often spoken of as RAID-1, striping is often known as RAID-0. In very large setups, these two techniques may be combined to create RAID-10 disk arrays, but this would probably be pertinent only on larger desktop computers with many hard drive bays.
With some of the technologies used, striping may allow an existing disk drive to be extended – in some cases, even while it is mounted and working – using a second drive.
What about the file system?
To my knowledge, there are at least four different ways of setting up either mirroring or striping on a personal computer. The first includes the use of dedicated physical hardware (thus, logically called “hardware RAID”) to which the two disks are plugged in, and which manages the complete set and presents it as a single unit to the user. However, issues include difficulties in using such a system on a laptop, the fact that not all desktop motherboards include this feature, and the hassle of using proprietary software (that often is not available under Linux) to recuperate data in the event of a problem. Needless to say, I would not advise using this way of going about it in a domestic or small business setting. It should also be noted that such setups may or may not allow “hot-plugging” new drives and associating the new disks to existing arrays. Some reading may be necessary if looking into using such a system.
The second way would be to set up a RAID array using the Linux kernel’s built-in md (multiple-disk) subsystem. Utilities such as mdadm give us access to the system, which is quite mature and works well. But the interface and commands are not always user-friendly, which is a bit of a pity and also why I will not get further into this system. Interested users’ are referred to Jan Mussche’s “Install Mint On A Two Disk Raid 0 Set” in FullCircle issue number 104 – switch from using RAID 0 (striping) to RAID 1 (mirroring) and the basics are about the same. In this case, “hot-plugging” new drives will be possible if your computer’s hardware allows it, which is in general not the case for consumer-grade computers and their drives. Adding new drives to existing arrays is possible in software, but will require some familiarity with the commands.
Mirroring or striping with BTRFS
This leaves us with two systems that are not based on managing individual disk drives, but whole file systems. In FCM#94, I wrote about the B-tree File System (BTRFS) and some of its applications. This file system has evolved even further since then, to the point that I consider it mature and use it on all my own computers (yes, I do eat my own dog food!). More to the point, Ubuntu’s installer, and that of most other modern Linux distributions, supports installing our system on a btrfs volume out of the box. GRUB can also use btrfs volumes, so very little fiddling is needed to set up mirroring. Basically, if our computer has two hard drives, we can use both disks to set up a mirrored or striped system. The usual caveats exist, including the fact that all existing data may be erased – so, begin by making sure you have a full backup of all your data, and also that the computer can be safely formatted with no ill-effects (i.e. please do not try this out on a computer you need to work on within the next hour or two…).
In the following examples, I will be using two rotational hard drives connected via USB to demonstrate the principles. In all cases, a single partition of 100 GByte size will be created on each drive.
Boot from the USB live image as usual, install the operating system using ubiquity on the first hard disk. When you get to the hard drive selection, create a partition. In my example, I will be using Kubuntu 20.04, and the disk to install to is /dev/sdc. Please do verify you are using the correct disk – perhaps twice – better than just once!
Boot from our new system, as usual. Now, using our tools of choice for partitioning (fdisk, gfdisk, KDE Partition Manager…), create a single partition on your second hard drive. It is not necessary to format it as btrfs, since whatever format we use will be overwritten.
We then add the second drive’s partition to the existing partition (containing our operating system). This can be done with the computer actually running the system. Supposing our main partition on the first drive with the operating system is /dev/sda1, and the new partition on the second drive is /dev/scd1, we would issue commands such as these:
$ sudo bash
# btrfs dev add /dev/sdc1 / -f
# btrfs fil show Label: none uuid: aeb12e81-f5b1-4a48-b80d-d64624867456 Total devices 2 FS bytes used 8.13GiB devid 1 size 93.13GiB used 11.02GiB path /dev/sda1 devid 2 size 97.66GiB used 0.00B path /dev/sdc1
At this point, the two separate partitions have been combined into a single BTRFS file system. With these two commands, we have a striped unit with a total capacity of about 200 GBytes. If striping (RAID-0) is desired, stop here.
Mirroring (or RAID-1) has not yet been implemented, and we will proceed to do so by converting our striped unit into a mirrored one. We will be using the fact that the file system is currently mounted on / (i.e. is the root file system). Be aware this process may take some time, especially if partitions’ sizes are in the TeraByte range:
# btrfs balance start -dconvert=raid1 -mconvert=raid1 /
This converts both our data (-dconvert) and filesystem metadata (-mconvert) into RAID-1, effectively writing a second copy of each item on the other disk. Our two partitions should not report a similar amount of space being used. That’s it; we have a mirrored unit and our data will from this point on be effectively duplicated with one copy on each partition.
Please note that both partitions are not quite the exact same size. With BTRFS, this is not an issue for striping. With mirroring, however, some space will be lost with unequal partitions. As a general rule of thumb, the final capacity of our mirrored drive will be about equal to that of the smallest of our two partitions.
Before continuing with our final option, let’s consider recovery from a failed disk. With BTRFS, if we are booting from a mirrored filesystem, the boot process will not continue at the point at which the Linux kernel mounts the drive. To get past that, we need to boot from another medium – an Ubuntu live image on a USB thumb drive is fine – and fix the bad BTRFS volume. We have two choices, both of which include removing the offending bad drive. On the one hand, if we have a spare physical drive to plug in, we can physically replace the bad unit and then run the “btrfs replace” command to restore the array to operation. On the other, if we have no physical spares available at the time, we can re-balance the BTRFS volume to single-disk mode, and remove the bad drive using the “btrfs balance” and “btrfs remove” commands. At a later date – better sooner than later – a second, working disk may be partitioned and added to the system as described above.
Mirroring and striping with ZFS
Although ZFS (at this time, the acronym is given no precise meaning) is an old player in the server market with Sun Microsystems behind it, it took some time for this technology to come to Linux. OpenZFS is gaining traction in the Linux kernel and Ubuntu distribution worlds, though perhaps not as swiftly as some server administrators would like.
ZFS file systems share many characteristics with BTRFS, though some details are different. For example, with ZFS, a virtual device (vdev) may be a single write unit, or may implement mirroring or other forms of data duplication – but it cannot change from one scheme to another as can BTRFS. Thus, when a vdev is created, one must specify from the onset if we want mirroring to happen, or not. A second feature is that ZFS units are usually mounted automatically by the ZFS subsystem, without needing any entries in the system configuration. In brief, file /etc/fstab may contain zero entries on a system that uses only ZFS units. Finally, in current versions of Ubuntu, ZFS drivers are not part of the basic install, but need to be added through various apt packages.
For interested readers, there is a nice writeup by Jim Salter on ArsTechnica. Though server-focussed, it should give you a good initial presentation to fully understand the specific strengths (and quirks) of ZFS over other file systems: https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/ .
On Debian-based distributions such as Ubuntu, things have come on to the point that there has been talk of offering to install recent versions on a root ZFS file system, at least as a beta feature. As a matter of personal preference, for the time being I am steering away from this option, at least until ZFS tools mature slightly (they are now shown to be at version 0.8.4) and are included out-of-the-box on all Ubuntu and derivatives’ live boot media. The reasoning here is that, while ZFS does have some very nifty features, the path to recovering data from a crashed boot volume – and please note we are speaking only of a boot volume – is a tad complex for regular users. For this reason, in the following example I will be working with: • A root file system on /dev/sda1 (first partition of our main hard drive), with an existing Kubuntu 20.04 installation. This is actually the very same BTRFS unit used previously. Which actual filesystem is used is not important, any familiar options such as ext4 or btrfs should work well.
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 46G 7,7G 36G 18% /
• Two partitions, /dev/sda2 and /dev/sdc1. These will be combined in a ZFS pool to hold user data.
The purpose of the root file system is just to hold our operating system, and to do so separately from user data. This means it can be formatted easily without affecting user data, or even replaced by a partition on another disk (or even a live media) at a pinch. The ZFS pool will be mounted on /home, to hold user data.
Before beginning, let us install the required packages. BTRFS is integrated into current Linux kernels, but ZFS is not. So, we will need:
# apt update ; apt install zfs-dkms zfsutils
As previously, begin by creating the appropriate partitions. Since we will not be using the root partition for ZFS, we will leave it alone and create /dev/sda2 and /dev/sdc1, both with about the same capacity. In our case, I will be using 400 GBytes each.
Now, let us create a zpool that uses striping to combine the two partitions. It is traditional to call this zpool the tank, but feel free to choose your own name.
# zpool create tank /dev/sda2 /dev/sdc1 -f # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 46G 7,7G 36G 18% / tank 756G 128K 756G 1% /tank
Each partition is of about 400 GBytes, but in combination they yield 756 GBytes. Note that the new zpool has automatically been mounted as /tank.
Within the zpool, we will now create a filesystem (vdev) called home, which may expand to use up all the space available.
# zfs create tank/home # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 46G 7,7G 36G 18% / tank 756G 128K 756G 1% /tank tank/home 756G 128K 756G 1% /tank/home
The new filesystem has also been mounted automatically, on /tank/home.
We now need to transfer over the contents of our /home directory, since this is where we will be mounting our new filesystem.
# mv /home/* /tank/home/
We actually need to move the files (and not merely make a copy) since the /home directory will need to be empty. Otherwise, the ZFS filesystem will refuse to mount on a non-empty directory.
Finally, we program the new filesystem to mount automatically on /home.
# zfs set mountpoint=/home tank/home
Now, reboot. The system should come up with tank/home mounted on /home:
$ mount /dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro) tank/home on /home type zfs (rw,xattr,noacl) tank on /tank type zfs (rw,xattr,noacl)
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 46G 5,6G 38G 13% / tank/home 756G 36M 756G 1% /home tank 756G 36M 756G 1% /tank
As can be seen, the new ZFS vdev is now mounted in its place on /home, with the two striped volumes giving us the expected combined space.
With ZFS, to create a mirrored zpool instead of striped, the appropriate syntax would simply have been:
# zpool create tank mirror /dev/sda2 /dev/sdc1 -f
Please note the “mirror” keyword. Other commands would be the same in either case, especially the creation of the ZFS filesystem (vdev) within the zpool.
Some final thoughts
A decade ago, mirroring and striping were options that would have seemed exotic, or quite frankly difficult to use, to many users. Even today, systems administrators can have cause to rely on dedicated hardware to set up such arrays. However, the spread of BTRFS and ZFS gives regular users – albeit those with a rather technical bent – the option of creating mirrored or striped units using only software that is available with the Linux kernel.
Both technical solutions have their proponents, with some very good reasons on either side – and a tendency to be quite vocal about it at times. Since our purpose is not to provoke an all-out flame war, let us merely suggest that BTRFS has perhaps a tad more flexibility as regards converting striped units into mirrored and vice-versa, can more easily extend an existing unit – even on the fly in a working system – and is also slightly better integrated into standard kernels. Setting up a mirrored root partition with BTRFS is also marginally easier than with ZFS. On the other hand, the strong points of ZFS include more flexibility setting up and mounting both zpools and vdevs in various configurations, and is more similar to the setup used in many large servers.
Additionally, both ZFS and BTRFS feature snapshots, which facilitates rolling back system changes in the event of a disaster, which is perhaps better integrated into the GRUB bootloader under OpenSuSE. They may also be used as a backup mechanism, since snapshots can be transferred between systems using the send-receive process.
All this opens interesting perspectives for the interested student of systems management – or for the advanced user. However, we would suggest to begin by testing out either or both systems described here on a spare computer, not using them as your main system (i.e. on data you depend upon) until you feel thoroughly at home with their syntax and quirks.