Ceci est une ancienne révision du document !
Quick Look At Linux in Industry
For the big one-hundred, I thought I’d do something a bit different. I emailed over a dozen large scale Linux users – companies, businesses, and educational establishments. I was interested in knowing what distro (or distros) they were using, what kind of hardware it was running on, how they were using it, and were they using open source or proprietary software.
From over a dozen, only a few replied. Here’s my report from the information I received.
Amazon
Amazon’s most well-known use of Linux is its Amazon Web Services (AWS - http://aws.amazon.com). Basically, it’s where you can build a scalable site using Amazon’s infrastructure. It’s used by everyone from Reddit to Netflix, Dow Jones to Vodafone.
AWS runs on the Amazon Linux AMI which is an Amazon supported Linux image for use on Amazon Elastic Compute Cloud (Amazon EC2). The image is maintained by AWS and is free to EC2 users.
The Amazon Linux AMI uses YUM for packages and repositories, and (from what I can tell) is a lightweight distro.
The source code for the distro can be viewed using the get_reference_source command line tool which is provided.
For those of you looking to install Amazon Linux AMI on your laptop, it’s available for use only inside of Amazon EC2.
EPFL
The EPFL (École Polytechnique Fédérale de Lausanne) uses Linux in everything from large networks to even drones. Their Laboratory of Intelligent Systems (LIS) specialises in bio-inspired AI, autonomous robots, and the like.
From messing around with cheap, simple, quadcopters, I can tell you that flying them in close quarters is just asking for trouble. This is something the EPFL’s LIS is looking into. Their GimBall drone has a protective cage which keeps it safe even in collisions.
The AirBurr drone is even more impressive in that it can right itself, if landed/fallen upside down, and can even stick itself to surfaces using a fibre-based adhesive.
But it doesn’t end there for the EPFL. Oh, no. We’ve looked at the small things running Linux, now it’s time to wheel out the big guns.
Aries
Aries is one of four IBM AMD Opteron clusters used by the EPFL. You think your desktop PC is pretty beefy? Take a look at what’s inside Aries:
• a front end consisting of two dodeca-cores processors AMD Opteron 6176 (Magny-Cours) 2.3 GHz, i.e. 24 cores, with 24GB of memory;
• 2 racks including 44 pizza nodes; each node contains four processors dodeca-cores AMD Opteron 6176 (Magny-Cours) 2.3 GHz (i.e. 48 cores per node), each with 192 GB of memory i.e. 4 GB RAM per core; peak performance: 19.43 TFlops;
• nodes are interconnected by an InfiniBand fast network QDR at 40 Gb/s, (fully non-blocking);
• a GPFS dedicated storage of effective 87 TB – 14 TB for home and 73 TB for scratch.
Bellatrix
Another EPFL cluster machine, this time Sandy Bridge based, and has:
• Peak performance: 119 TFLOPs
• Total RAM: 14TB
• Storage: 200TB
The cluster is composed of:
• a frontend, • a master node in charge of administration and backup, • 424 compute nodes, each with:
• 2 Sandy Bridge processors running at 2.2 GHz, with 8 cores each, • 32 GB of RAM,
• for a total of 6784 cores
• Infiniband QDR 2:1 connectivity
• GPFS filesystem.
Castor
Castor is a Dalco Intel Ivy Bridge cluster specifically tailored for sequential calculations which is comprised of:
• a front-end;
• an administration and backup node;
• 50 standard nodes with 64 GB of RAM each;
• 2 large memory nodes with 256 GB of RAM each;
• an NFS storage server of 22 TB (4 TB for home, 18 TB for scratch);
• the nodes are interconnected by a 10 GbE network.
Each node has 2 Ivy Bridge processors with 8 cores each, running at 2.6GHz (model name: Intel Xeon E5-2650 v2).
Daneb
Last, but not least, is Daneb. This is an Intel Xeon based cluster which has:
• Peak performance: 293 TFLOPs (211 in CPUs, 92 in GPUs) • Total RAM: 37TB • Storage: 350TB
The cluster is composed of:
• two front ends, • two master nodes in charge of administration and backup, • 376 compute nodes, each with: • 2 Ivy Bridge processors running at 2.6 GHz, with 8 cores each, • 64 GB of DDR3 RAM,
• 144 compute nodes, each with: • 2 Haswell processors running at 2.5 GHz, with 12 cores each • 64 GB of DDR4 RAM
• 16 GPU accelerated nodes, each with 4 K40 NVIDIA cards • 8 large memory nodes, each with 256 GB of RAM • 2 NUMA nodes, each with 4 processors and 512 GB of RAM • Infiniband QDR 2:1 connectivity, • GPFS filesystem.
All of the above clusters run RHEL (Red Hat Enterprise Linux)
If you think they’re impressive, you ain’t seen nothing yet!
BlueGene/Q
This behemoth is the Lemanicus BG/Q supercomputer. The specs will bring tears to your eyes:
• IBM Blue Gene/Q Massively Parallel Supercomputer • 1 rack, wired as a 4x4x4x8x2 5D torus • 1024 sixteen-core nodes, PowerA2, 1.6 GHz • Energy efficient, water cooled • 209 Tflops peak, 172 Tflops LINPACK • 16 TB of memory (16 GB per compute node, 1 GB per core)
Storage
• 2.1 PB of disk space • GPFS Native Raid (GNRx) disk management solution • GPFS parallel file system
If you really want to geek out on this beast, I’ll let you download and flip through the Getting Started guide: http://bluegene.epfl.ch/pdf/GettingStarted.pdf
I’ll let Vicky from the EPFL tell you more about them:
“As standard HPC platforms, our clusters run a mix of Open-Source and proprietary software. There are some specific drivers for the InfiniBand network and the GPUs (for the nodes with GPUs), and proprietary compilers and parallelization libraries (Intel). Of course, we also provide GNU compilers and Open Source parallelization libraries, so the users who want to, can use free software on our machines. Actually you can go fully Open Source on the supercomputers. But, as I say, those are beowulf clusters following the beowulf philosophy: commodity hardware, free and open source software.”
What’s a Beowolf cluster? Basically: • a distributed memory machine. • a cluster built from inexpensive personal computer hardware. • a group of identical, commercially available computers (compute nodes). • running Free and Open Source Software (FOSS). • nodes are networked into a small TCP/IP LAN, and have libraries and programs installed which allow processing to be shared among them. • behaves more like a single machine rather than many workstations. • compute nodes are accessed only via the master nodes. • appears to the network as a unique entity. • Beowulf nodes can be thought of as a CPU + memory package which can be plugged into the cluster. • to have a faster cluster, just add nodes.
Some users access the cvlab cluster with just a shell. Computations are done using Python and C++ to give maximum speed. Some, though, use Matlab.
The cluster in question is made up of: • 6 x IBM x3650 M3 • 4 x Dell PowerEdge R720 • 3 x IBM x3650 M4 • 2 x Supermicro X8DTU • 1 x Supermicro X7DWU
Most of the Computing Science courses in first or second year are encouraged to use Linux. Quite a few IT rooms dual-boot Linux with Windows. Some can even be accessed using SSH.
Many thanks to Vicky and Axel for the info.
I’m sure, after all that, your head is spinning. Speaking of spinning…
CERN
The European Organization for Nuclear Research, or CERN, is probably most famous for the Large Hadron Collider (LHC). Every second, 600 million particles collide within the LHC. Every year physicists sift through 30 petabytes (more on them later) of data.
CERN uses several Linux distributions on approximately 25,000 servers and desktops at the laboratory’s sites in Geneva and Budapest. The majority are running Scientific Linux CERN 5 and 6 (ie: Scientific Linux from Red Hat Enterprise Linux 5 and 6 respectively), along with some Red Hat Enterprise Linux. New installations are now starting to use CERN CentOS 7 (CC7).
A wide range of applications are used, including standard infrastructure components such as web services or development build services, but the bulk of the resources are used to support physics analysis and simulation by the 11,000 physicists who use the facilities at CERN.
Open source software is used extensively at CERN – from the infrastructure such as Linux and OpenStack, through to the applications written by the physicists themselves to analyse the 27 PB/year (PB = petabyte = 1,000 terabytes, or 1,000,000 gigabytes) which is produced by the Large Hadron Collider.
There are around 11,000 ‘white box’ servers in the CERN computer centres with between 8 and 32 cores and an average of 2 GB/core. They are generally connected using 1GB/s Ethernet. These servers are part of the Worldwide LHC grid which runs over 2 million jobs per day.
Many thanks to Tim Bell and the Press Office from CERN for that info.
To learn more about the LHC, there’s an excellent PDF brochure from: http://cds.cern.ch/record/1165534/files/CERN-Brochure-2009-003-Eng.pdf
To learn more about CC7, SLC5/6, and more, see: http://linuxsoft.cern.ch/
Supercomputers
Needless to say, almost all of the world's top supercomputers run Linux in some form. The TOP500 is a table showing the top 500 most powerful supercomputers. Starting in 1993 the list is published twice per year.
The top five as of November 2014 are:
Number one on the list is Tianhe-2 from China. It runs Kylin Linux, which is a version of Ubuntu. In second place is Titan, which runs the Cray Linux Environment. Sequoia, in third, runs Linux. Just ‘Linux’ according to the TOP500 listing. Same with K in fourth place and Mira in fifth.
Having looked through the top ten, they all run Linux of some sort.