Ever since the shift to multi-core processors began, I have always had a nagging question – Does more cores mean less nodes? I have wrestled with this question and finally realized that there is no simple answer. I should preface this post by stating that I am talking about HPC and not the general market place where multi-core and virtualization are all the rage.

To understand why I ask this, consider that back when Linux clustering began, the single core Pentium Pro and Dec Alpha where the two processors of choice. Large clusters were maybe 64 or 128 nodes (often tower cases), which translated into 64 or 128 cores. Today you can easily pack 128 cores into 16 nodes – not even a full rack chassis. Given this trend, is the node count for clusters getting smaller?

From my anecdotal evidence, I seem to notice several trends. First, the HPC market, after a downturn, seems to be growing again and is projected to reach $11.7 billion by 2012 (IDC). This revenue is up from a $9.6 billion figure for 2008. Thus, the hunger for nodes is increasing and not deceasing, which begs a further question. Are node counts increasing or are more people buying clusters? (i.e. instead of a few people buying larger clusters, are there a lot of people buying smaller manageable clusters.) For the marketing types out there, maybe you know the answer. If not, that is a good question to ask. Leave a comment and give us a clue.

Second, from my experience there is plenty of HPC work to go around. Whether nodes are in a large data center cluster or in a small local blade system, it seems the cores are busy. Perhaps we are seeing the rise of the Closet Cluster?

Platform HPC Workgroup Manager v1.1 is sold through Platform Computing partners and is for use in clusters containing 32-nodes or less. The bundle includes the following:

Platform Cluster Manager (PCM) v1.2 – Standard Edition
Platform LSF Workgroup 7 Update 5
Platform MPI v5.6.6

The offering includes the following capabilities:

Dynamic Provisioning
Dynamic provisioning provides the ability to dynamically re-purpose nodes within a Platform LSF Workgroup cluster managed by Platform HPC Workgroup Manager.  The re-purposing is driven by LSF Workgroup (or manually) to match the OS configuration of the compute nodes according to the workload requirements. Support configurations include compute nodes configured as dual/multi-boot with Red Hat Enterprise Linux 5.3 and Microsoft Windows Server operating systems.

Firstboot scripts
The Firstboot script functionality provides a simple means for IHVs to configure a series of scripts to be executed upon the initial boot of a PCM cluster headnode to prompt for key site specific information including network configuration, Platform LSF Workgroup and Platform MPI licensing.

For more details, please contact partnersales@platform.com (1-877-710-4522).

Every so often I am asked about books on Beowulf/HPC clustering. The good news is there are books available at several levels. The bad news is no one book covers everything because the number of topic areas is so vast. To make your search easier, I have surveyed all of the cluster books I know about. I have not included some that are out of print or don’t really discuss HPC. Some of the books are dated, but still contain good general information. Others are just plain old, but are included here for completeness. If you know of any other books, please add a comment. My list is as follows:

Engineering a Beowulf-Style Compute Cluster by Robert G. Brown
A freely available book (pdf) that discusses building and designing Beowulf style clusters. Robert Brown is a long time contributor to the Beowulf/HPC cluster community.

Building Clustered Linux Systems by Robert W. Lucke
A very good overview of cluster computing methods and hardware. The book provides a rather wide coverage of options, but does not dive too deep into any one approach. It is somewhat Hewlett Packard focused as author works for HP. (648 Pages) ISBN: 0-13-144853-6

HPC For Dummies (an ebook) by Douglas Eadline
A very short introduction to HPC, but not for dummies either! The ebook is available for free from Sun Microsystems after registration. The book provides backgound and best practices. (authored by yours truly)

Beowulf Cluster Computing with Linux, Second Edition by William Gropp, Ewing Lusk and Thomas Sterling
Updated edition, now edited by William Gropp, Ewing Lusk (in addition to Sterling). Provides a good, but high level view of Linux clustering. This edition includes ROCKS and OSCAR coverage plus other important issues. (504 pages) ISBN 0-262-69292-9

Beowulf Cluster Computing with Linux by Thomas Sterling
The next book after the original “How to Build a Beowulf” by Tom Sterling (see below). Coverage is much expanded, but now very high level as chapters are written by key players in the cluster community. (536 pages) ISBN 0-262-69274-0

Beowulf Cluster Computing with Windows by Thomas Sterling
Approximately 75% of the material is the same as the Linux book by Sterling (no file system coverage). Newer versions of Windows HPC Server 2008 make this a bit dated. If you need Windows clusters, you may want to start here. (488 pages) ISBN 0-262-69275-9

How to Build a Beowulf by Thomas Sterling, John Salmon, Donald J. Becker and Daniel F. Savarese
The first book on Beowulf cluster computing. Published in 1999, it is now quite dated. Both hardware and software have moved past the text. It does provide good coverage of the issues facing the cluster builder. (261 pages) ISBN 0-262-69218-X

In Search of Clusters by Gregory Pfister
A good (and very technical) book about the advantages of cluster architectures. Also provides a very detailed analysis of programming models This book seems to be out of print. Worth reading if you can find a copy. It does not cover Linux Beowulf clusters. (608 pages) ISBN 0138997098

High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI by Joseph D. Sloan
O’Reilly’s second attempt at a Linux cluster book. Many feel this second attempt has missed the mark (again). Also note, the The OpenMosix Project has officially closed as of March 1, 2008. (367 pages) ISBN: 0-596-00570-9

Linux HPC Cluster Installation An IBM Redbooks publication
This document focus on xCAT (xCluster Administration Tools) for installation and administration. All nodes and components of the cluster, such as compute nodes and management nodes, are installed with xCAT. Dated material.

In the processor world, that means we talk about not just multi-core, but many-core, and how users will take advantage of these vast arrays of computational power. But in our enthusiasm to embrace this abundance, it is important not to lose sight of the fact that the transition from single-core is far from complete.

Our 2009 High Performance Computing Site Census Survey found that as of April this year, 43% of the HPC installed base is single-core systems, making them still the leading processor type, ahead of dual-core (34%) and quad-core (20%).

To be sure, part of the reason for this is that some HPC users keep their systems in some form of employ for many years. The average life span of a cluster node is under three years, but many systems persist. We found about a quarter of the systems in our current census survey were installed before 2005, and over four-fifths of these were single-core.

Natural longevity isn’t the only contributing factor. First of all, some HPC users are still buying single-core systems. In our survey, one out of every seven systems installed since the beginning of 2008 was single core.

One potential reason for this might be that many users have not optimized their codes – or entered into ISV licensing agreements – to take multiple cores into account. This same phenomenon could lead HPC users to prolong the life of their existing single-core systems.

This is a significant, enduring issue for the HPC industry. The data points to a large population of the HPC market that may still help optimizing their environments for even dual-core, let alone quad, eight, and beyond.

Getting ahead of the competition is one part of the battle. Making sure your customers can keep up with you is the other.

The use of virtualization and multi-core processors has made cloud computing an option for many users. The ability to buy cloud time as you need it and not purchase hardware is certainly attractive from a financial standpoint. The concept is not new and has its roots in time shared mainframes and grid computing. One might assume the the vast amount of computing resources in clouds may make them ideal candidates for HPC clustering. Unfortunately, it is not as simple as collecting cores.

One of the issues facing clouds is I/O. Basically, I/O is often not predictable or repeatable. From a storage standpoint read and write times can be fast, but not always fast. In terms of messages between servers, most clouds do not support high performance interconnects and similarly make no guarantees as to latency or bandwidth consistency. While grids paid attention to certain HPC performance guarantees in terms of I/O, clouds, in order to offer ease of use, have declined such guarantees. Unless a cloud has been specifically designed for HPC, the user cannot expect consistent and/or high performance. There are two papers which discuss this very idea. The first paper looks at Benchmarking Amazon EC2 for High-performance Scientific Computing and the second paper asks, Can Cloud Computing Reach The TOP500?. Both papers conclude that the cloud is not mature enough for HPC applications.

The limitations of the cloud become more apparent when one looks a little deeper at HPC applications. First, many applications rely on user space communication (i.e. high performance MPI programs transfer data directly from one node to another without using kernel services.) Such a close to the wire operation runs counter to the virtualization model. Secondly, as reported in the first paper (above), the performance of OpenMP applications was reduced by 7-21% when running in the EC2 cloud.

Recently Penguin Computing began offering POD (Penguin on Demand) for HPC cloud computing. The POD cloud offers both Ethernet and InfiniBand connections between nodes thus providing a dedicated high performance computing environment. This service can be considered a specialized HPC cloud.

There are some other other important issues to consider with cloud computing — security and reliability. When data leaves your domain over the Internet it is virtually impossible to guarantee 100% security. If your organization can live with this situation, using the cloud may be an option. If on the other hand, you need to keep a tight reign on your data, then you may not want to be injecting it into the cloud. The other issue is reliability. If your day to day operations are based on using a cloud, then a contingency plan is a must. Interruptions in Internet traffic due to congestion or hardware failures can be common in some areas. In addition, the cloud provider may have issues (even go out of business) and thus not meet the service requirements.

I believe the cloud is an interesting model, but it is not a real solution for HPC (in its current form). My issue with clouds is that they are often categorized as “grid like” and then are somehow (incorrectly) considered “HPC like.” Cloud offers utility computing like grid promised, but has pushed the application layer further away from the hardware. HPC practitioners spend a lot of time making sure the application is as close to the hardware as possible. At this point in time, HPC in the cloud is more of a curiosity than a solution. When examining HPC benchmarks it becomes clear that clouds are not the best means to provide HPC cycles. Whether efforts like POD can meet the HPC users needs in the cloud is still unknown.

To be fair, there are some HPC applications that lend themselves to clouds quite well. (i.e. those that do not require predictable I/O) Folding@home and Seti@home are two good examples. These applications could easily run in a cloud (in a sense they do run in the Internet cloud). Keep in mind they have been designed to work in a robust distributed fashion and are not virtualized. Clouds can be enticing and even enabling for some applications, but remember a collection of servers (in the cloud or in a rack) does not a cluster make.

Many of the HPC people I know are what you would call rugged individualists. The have been around since the beginning and were responsible for moving the market/community along when commodity HPC was less than fashionable. This group consists mostly of developers, implementers, and administrators. Many of these people developed, by way of discussion on the Beowulf Mailing List, the best practices used today. The Beowulf Mailing list is a true resource if there ever was one. A newbie can ask a question and get polite (and lengthy) answers by list members. The list holds a large amount of open community knowledge because all the HPC plumbing is open source. Unhindered discussions can take place at any level between any number of people.

There is also the false notion that open source software is “free as in beer.” This idea is not quite true because software unlike toasters have a usage cost. Once you install, configure, and study any software you have already made an “investment” in the package. Continued use furthers this investment. The size of the investment is up to you. And, because the software is open, in theory you can fix any problem. Thus, you have the choice to decide how much time you can invest in a particular software package before it becomes “expensive” to you. At some point, the cost (or dilution) of your time may come into play. Spending weeks fine tuning a single application at the expense of other responsibilities is probably not going to play out very well.

In terms of support responsibilities, many clusters have minimal issues once they are configured correctly. There are, of course, hardware failures, but in general, once everything is booted, things often work quite well. There are two areas that need attention, however. The first is software updates. Updates are needed for several reasons including security, bug fix, or feature updates. These types of updates are usually easy to manage unless they have dependencies, which means there may be a whole raft of packages that need updating. If you don’t get the dependencies right, then there can be problems with the entire cluster.

The other issue is local integration. This is what I consider the “last mile problem” for clusters. Very often local file system issues need to be worked out and managed in addition to creating job submission policies. There is usually some end-user assistance needed as well as questions on how to compile and submit jobs to the queue. Of course, “Why is my job sitting in the queue?” is probably the question that gets asked the most.

If your job includes time for installing, integrating, and updating software and you happen to be one of those rugged individualist HPC people, then you probably have no interest in professional support. If on the other hand, you are new to clustering (or Linux) and already have many responsibilities, then you may want to consider using professional support services. As with all open source software, the choice is yours. In terms of commercial software, or commercial support of open software, there are many options. In any case, purchasing support for business critical applications is always a good idea. As is the use of a Intel Cluster Ready (ICR) solution. By adhering to the ICR specification support is much easier — for you and/or a vendor. That is, a reference platform allows you and your vendors to work from the same page. Without a common framework, support vendors and others in your organization may have to decipher/debug how you configured the cluster.

In conclusion, cluster support can be commercial or it can be institutional. In either case, there is a cost. If you do it on your own, it will cost time and if you hire a consultant or company, it will cost money. To supplement either effort, there is a large amount of information on the web that can be useful when identifying and solving problems. Support is an important part of any successful HPC cluster,  just ask the old-timers. They figured it out, so you don’t have to.

I think I do not have to state that setting up an HPC cluster sometimes is an adventure of compiling and installing different program versions and libraries. Nevertheless, when having all up and running, the Intel Cluster Checker is a really nice tool to verify that all services needed for your HPC cluster are set up correctly and you haven’t forgotten one of those tiny details during the installation of your cluster.

My last experience with the Intel Cluster Checker and the newly installed HPC cluster were a bit different from the ones before. The first steps ran smoothly and everything seemed to work fine, with me having learned from my previous first Intel Cluster Checker experience. The tests were successful, I altered the benchmark thresholds and all tests ended with a “passed”. But in the end, when checking the dmidecode output I got a “failed” on this test. First I thought this was due to some BIOS specific mismatches, which you can exclude. But when I had a closer look on the output file of the Intel Cluster Checker I saw that there seemed to be installed different RAM modules on the compute nodes.  Hmm…the sizes of the memories looked fine, all were 2GB and I double checked the product numbers of the DIMMs that the Intel Cluster Checker reported. Here we got two different types of memories installed. I was astonished on the one hand as I would not have checked this without the Intel Cluster Checker and on the other hand I hardly could believe that there were two different types of memory DIMMs installed. Well, the servers I had installed were Intel Nehalem based and I remembered the days when AMD started with the CPU built-in memory controller and the problems with memories that arose during those times…

I took a closer look at the compute nodes and opened the chassis of  those which were affected. And indeed, there were different types of memory modules built in, but they nearly looked the same and also had the same product number written down on them. After some investigation with our purchasing department I found out that those memory types have been mixed up. Unfortunately it was not possible to pin down the real cause for this confusion.

The DIMMs were swapped to the correct ones in the end and I ran the Intel Cluster Checker again. This time all tests were passed and the results were sent to Intel to verify the Cluster Ready Certificate, which now proudly resides beside the other ICRs we have scored.

The Solid State Drive (SSD) has had a break-out year. Unlike traditional mechanical hard disk drives that use spinning platters with movable read/write heads, SSDs have no moving parts. The SSDs is made entirely out of a special type of flash memory — the same kind of NAND flash memory found in thumb-drives and memory sticks. Overall, SSDs are faster, quieter, more energy efficient, but less dense than the traditional spinning platter drive.

The staple of the storage industry has been the mechanical drives, where I/O rates are limited to the mechanical properties of a drive. Unlike other semiconductor trends, the sustained write rate has barely doubled from 50 to 90 MB/second over the past 8-10 years. All this is about to change as the use of flash memory will allow stroage to take advantage of a semiconductor growth curve similar to that of processors and memory. (i.e. mechanical drives are limited by the physical motion of spinning disks).

Perhaps the most important feature offered by SSDs is the read and write performance. The IOPS (I/Os per second) rate for an SSD is usually two to five times that of a traditional mechanical hard drive. When reading, performance is mostly constant because the seek time is virtually instantaneous and does not depend on the physical location of the data on a platter. As a result, file fragmentation has almost no impact on read performance. In addition, because there are no moving parts, SSDs use as little as one-fifth the power of a mechanical drive. Another interesting feature is the SSD failure mode. Most SSD failures tend to happen when writing. In contrast, mechanical drives tend to have most failures when reading. Thus, once data is written, it is more likely it can be read from a failed SSD.

SSDs do suffer from “degradation” over time that results in reduced performance and limited lifetimes (i.e. there are a limited amount of read/write cycles avaliable for NAND memory). Vendors have taken this into account and include wear leveling algorithms in SSDs that spread write access evenly over the entire device.

If you are interested in the exploring SSDs, there are some key points to consider. First, SSDs are not the best solution in every case. Currently, their capacity is much less than that of traditional mechanical drives and as such may not be suitable for some of the large HPC data sets. In addition, the cost is per MB is higher and are more susceptible to data loss from energy and power surges.

Second, the very fast read times offered by SSDs has made them a good candidate for improving OLTP (Online Transaction Processing) systems where frequently read tables and indexes can be accelerated. Check both the read and write IOPS, as there is usually a big difference between these values. Typically, the read speed is 10 times the write speed resulting in asymmetric performance. In terms of clusters, using SSDs for NFS mounts or read-only data may be helpful.

Third, because SSDs are new, questions still remain about how much of that speed they can deliver for the long haul (due to degradation). Typically, an SSD will show an initial decrease in performance and then level off. Even with a performance drop over time, SSD drives are almost always faster than traditional hard drives. The JEDEC standards organization plans to publish two standards by the end of this year for SSD endurance metrics.

Finally, pay particular attention to “write endurance,” this number should pertain to random writes. For instance, an Intel® X25-E Extreme 64 GB SATA Solid-State Drive is rated for 2 petabytes of lifetime random writes.

In terms of software, one difficulty facing the industry are the legacy assumptions built into file systems. These assumptions will need to be challenged in order to take advantage of SSD technology. For instance user applications and file systems will need to account for the asymmetric read/write performance of SSDs. Many computer applications rely on synchronous patterns of read/write operations, wherein a given write or update must be completed and the write confirmed before additional application read requests can be issued. With SSDs this process may need to be reconsidered.

There is no doubt that SSDs are the future of storage. Indeed, SSDs are even changing the way we compute. For example, CAE (Computer Aided Engineering) applications can use the speed advantage of SSDs in their out-of-core algorithms. The power of semiconductor manufacturing technology combined the speed of NAND flash memory are about to make the storage market stand still!

The thirst for better file system technology is not new to the Unix/Linux world. There is a rich history of trying to optimize the balance between the storage system and the process improvements in a demanding user environment. The efforts to build a better file system are quite numerous and are built on the efforts of many people. For example Kirk McKusicks’ original Berkeley Fast File System improved on the original V7 release. Steven Tweedie’s ext3 took ideas from the database logs and Margo Seltzer’s LFS and added them to Linux’s implementation of UFS — Ted Tsao’s ext2. In the mean time, DEC released Megasafe, SGI released XFS and Sun released ZFS all to the wild. And now Oracle has developed Btrfs for Linux.

So why do we as users care? There better be a good reason to change a file system because a new file system usually means converting to and trusting a new format with your data. Thus, any new format or change must provide a compelling reason or solve a big problem. Otherwise what is “good enough and works” is often better than that which is “new and fancy.”

If you follow the details of file systems development, then this quick update may not be of interest to you. For the rest of us, who just choose whatever file system the installer offers, you may want to read further because changes are afoot.

If you are like me, you probably are running Linux with the ext3 file system. There is nothing wrong with ext3 as it is stable, robust, and a standard Linux file systems. And, one other thing, it is old. Even if you are running the newer ext4, you are still running a 30-year old file format that is more than a little short on features.

There are those that believe ext4 is going to be the end of the line and a switch over to Btrfs is very likely. Btrfs (pronounced “butter-F-S”) is being developed by Chris Mason at Oracle. It is an open source project that has recently been added to the Linux kernel (as of 2.6.29) as experimental code.

Btrfs is based on several new ideas including b-trees (binary trees, which is where the btr comes from in Btrfs) and “copy-on-write” or COW. While, I won’t go into the details, b-trees and COW allow for some new features that would be difficult in the ext* line of file systems. (If you want to learn more about the technical details of Btrfs, see A short history of btrfs on lwn.net.) Some of the new features include file-system snapshots, check-summing, online defragmentation, compression, extents, resizing, and more. In particular, Btrfs allows one thing that has been difficult to achieve in the past — optimizing both access time and disk space.

The fact that Oracle sponsors Btrfs has lead to some concern. Recently, Oracle purchased Sun Microsystems which has been developing the ZFS file system for many years. ZFS is similar to Btrfs (it uses COW) and provides many of the same features, but it is very different in its internal implementation. ZFS will also “run” under Linux using Fuse. Mason and other have assured the community that Btrfs is important to Oracle and they will continue development. In addition, the open source nature of Btrfs ensures that it cannot be “taken away” now or in the future.

There is plenty more to consider and I suggest reading Linux Don’t Need No Stinkin’ ZFS: BTRFS Intro & Benchmarks by my friend Jeff Layton. The consensus seems to be that Btrfs is destined to become the default Linux file systems within two years. ZFS on the other hand must overcome some licensing issues before it can even make it into the Linux kernel for testing. Your next Linux install may offer a new and better (or “btr”) file system than in the past.