Disks On Cluster Nodes?

July 23rd, 2009 11:00 am
Posted by Douglas Eadline
Tags: , , , , ,

Do you need/want hard disks on your cluster nodes? Well, it all depends.

There is an ongoing discussion amongst cluster mavens concerning hard disk drives on cluster nodes. In one camp is the "no disk" or "disk-less" crew and in the other are the "must haves" or "disk-full" proponents. As with everything cluster it all depends and your needs will dictate the final decision.

Before we talk about specifics, let's clear up one important aspect about disk drives on cluster nodes. The first clusters used hard drives as a way to boot nodes as network booting (i.e. PXE booting) was not available or hard to implement. Indeed, a popular way to build nodes was to use a Red Hat kick-start floppy and provision each node with the same Red Hat install from a local server. The process worked quite well and small clusters could be built very quickly. When network booting became available, it was possible to boot and run the node entirely over the network and disk-less nodes began to appear. With disk-less nodes, all of the important files were either placed in a ram-disk or made available over NFS (Network File System), or a combination of both. Some open source examples of these types of provisioning include Perceus and oneSIS.

There is often a misconception about disk-less booting. That is, disk-less booting a node means you cannot use a local hard disk. Fortunately, there is absolutely no reason why you cannot use a hard disk on any node in a cluster. Many users who choose disk-less booting, choose not to place a hard disk on the nodes, but a good portion chose to include a hard disk for swap and temporary file space (scratch space). This type of local disk is not used to boot or hold any system flies, but can be used by applications that run on the node.

Of course you can always use a local disk to boot the node. Distributions that support this model, often manage disk images that are placed on a hard drive. These disk images are often copied to the node hard drive over the network from the head node (i.e. there is no need to use a kick-start floppy). Examples of open source image based distributions include Rocks Clusters and the Kusu Project.

Proponents of disk-less systems often cite, less failures, less cost, a single point of management (usually the head node), and quicker booting. Those that support disk-full systems contend that having swap is important for many applications and if the drive is on the node, why not use it for a full install and reduce NFS traffic. They also point out that node images are centrally managed as well.

How you choose to provision your nodes is often application dependent. In the past, there was a trend to keep the software on nodes to a minimum, but many newer cluster applications now expect to see a full distribution on the nodes. For instance, many applications now rely on scripting languages like Perl or Python. Thus, both disk-less and disk-full provisioning must guarantee a compete environment for each node. In the end however, the choice is yours. You may find it instructive to follow a recent thread on the Beowulf Mailing List about disks on nodes. Some topics never get old.


Comment from kalloyd
Time August 5, 2009 at 7:07 am


Excellent article for an increasingly complicated topic. Our Rocks/CUDA cluster uses NFS and of course PXE booting to compute nodes, but there are still times when persistence to a compute node disk is warranted for JIT compiled task routines on that node.

Comment from Douglas Eadline
Time August 10, 2009 at 5:20 pm

What kind of JIT applications are your using?


You must be a Registered Member in order to comment on Cluster Connection posts.

Members enjoy the ability to take an active role in the conversations that are shaping the HPC community. Members can participate in forum discussions and post comments to a wide range of HPC-related topics. Share your challenges, insights and ideas right now.

Login     Register Now

Author Info

Dr. Douglas Eadline has worked with parallel computers since 1988 (anyone remember the Inmos Transputer?). After co-authoring the original Beowulf How-To, he continued to write extensively about Linux HPC Clustering and parallel software issues. Much of Doug's early experience has been in software tools and and application performance. He has been building and using Linux clusters since 1995. Doug holds a Ph.D. in Chemistry from Lehigh University.