The use of virtualization and multi-core processors has made cloud computing an option for many users. The ability to buy cloud time as you need it and not purchase hardware is certainly attractive from a financial standpoint. The concept is not new and has its roots in time shared mainframes and grid computing. One might assume the the vast amount of computing resources in clouds may make them ideal candidates for HPC clustering. Unfortunately, it is not as simple as collecting cores.

One of the issues facing clouds is I/O. Basically, I/O is often not predictable or repeatable. From a storage standpoint read and write times can be fast, but not always fast. In terms of messages between servers, most clouds do not support high performance interconnects and similarly make no guarantees as to latency or bandwidth consistency. While grids paid attention to certain HPC performance guarantees in terms of I/O, clouds, in order to offer ease of use, have declined such guarantees. Unless a cloud has been specifically designed for HPC, the user cannot expect consistent and/or high performance. There are two papers which discuss this very idea. The first paper looks at Benchmarking Amazon EC2 for High-performance Scientific Computing and the second paper asks, Can Cloud Computing Reach The TOP500?. Both papers conclude that the cloud is not mature enough for HPC applications.

The limitations of the cloud become more apparent when one looks a little deeper at HPC applications. First, many applications rely on user space communication (i.e. high performance MPI programs transfer data directly from one node to another without using kernel services.) Such a close to the wire operation runs counter to the virtualization model. Secondly, as reported in the first paper (above), the performance of OpenMP applications was reduced by 7-21% when running in the EC2 cloud.

Recently Penguin Computing began offering POD (Penguin on Demand) for HPC cloud computing. The POD cloud offers both Ethernet and InfiniBand connections between nodes thus providing a dedicated high performance computing environment. This service can be considered a specialized HPC cloud.

There are some other other important issues to consider with cloud computing -- security and reliability. When data leaves your domain over the Internet it is virtually impossible to guarantee 100% security. If your organization can live with this situation, using the cloud may be an option. If on the other hand, you need to keep a tight reign on your data, then you may not want to be injecting it into the cloud. The other issue is reliability. If your day to day operations are based on using a cloud, then a contingency plan is a must. Interruptions in Internet traffic due to congestion or hardware failures can be common in some areas. In addition, the cloud provider may have issues (even go out of business) and thus not meet the service requirements.

I believe the cloud is an interesting model, but it is not a real solution for HPC (in its current form). My issue with clouds is that they are often categorized as "grid like" and then are somehow (incorrectly) considered "HPC like." Cloud offers utility computing like grid promised, but has pushed the application layer further away from the hardware. HPC practitioners spend a lot of time making sure the application is as close to the hardware as possible. At this point in time, HPC in the cloud is more of a curiosity than a solution. When examining HPC benchmarks it becomes clear that clouds are not the best means to provide HPC cycles. Whether efforts like POD can meet the HPC users needs in the cloud is still unknown.

To be fair, there are some HPC applications that lend themselves to clouds quite well. (i.e. those that do not require predictable I/O) Folding@home and Seti@home are two good examples. These applications could easily run in a cloud (in a sense they do run in the Internet cloud). Keep in mind they have been designed to work in a robust distributed fashion and are not virtualized. Clouds can be enticing and even enabling for some applications, but remember a collection of servers (in the cloud or in a rack) does not a cluster make.

Every six months the largest computers in the world are ranked as to how many floating point operations per second (FLOPS) they can perform. The results are tabulated on the Top500 list. The actual benchmark is called HPL, which stands for High Performance Linpack. The Linpack benchmark was designed to measure the floating point performance of various systems and the HPL version is designed to run on parallel computers like clusters.

The Linpack problem is something you may have seen in high school. Given a set of linear equations of the form,

Solve for x,y, and z. If you recall further, a system of linear equations can be generalized as the following:

Where A is a square matrix of the coefficients (the 3, 2 values etc. in the above equations), x is a vector list of the "unknowns" (x, y, z) and b is a vector list of the "answers" (1,-2,0). The Linpack benchmark solves for the unknowns. The size of the test is the the number of equations and is often referred to as N. If N were 10, then A would be a 10x10 matrix, x would be list of 10 unknowns, and b would be a list of 10 answers.

In terms of memory, if N were 10,000, then a desktop computer would need about 1GB of memory. If N were 1,000,000 then 7.5 TB of memory would be needed. Enter the cluster, where HPL is designed to distribute these really large problems over the nodes. The grunt work on each node is actually done by the BLAS library, which stands for Basic Linear Algebra Subprograms. These routines are used to solve the subprogram given to each node. At various points, the nodes must exchange information using MPI (Message Passing Interface).

There are various checks in the program to make sure the calculations are correct. HPL also creates random data for each size problem. To keep things fair, the random data is the same for each value of N. The BLAS library is available in many forms. There are several optimized versions that can make a huge difference in performance. For instance, on Intel processors, Intel offers the Math Kernel Library (MKL) that contains very fast hand optimized math routines (including the BLAS libraries).

In addition to problem size, there are many other tunable parameters for the HPL benchmark. The benchmark can take hours or days to run, thus getting a good "HPL number" can take a long time. It also requires the entire cluster, something that often disappoints regular users. There are many applications that use HPL type math and thus the benchmark is very relevant for some users. In other cases, where users codes are not solving dense linear equations, the benchmark offers a good historical measure of HPC progress, but offers little insight in terms of their application performance. Next time you hear HPL, BLAS, Linpack, MKL, and other funny sounding acronyms you will know it is just a bunch of software for solving big versions of those little high school math problems.

Ask anyone, "What are the two choices for HPC interconnects?"  and they will tell you "InfiniBand and 10 Gigabit Ethernet (10 GigE)." For the most part they are correct, but 10GigE is just entering the HPC market. It has not even landed in the Top500 arena, although users are still confident it will show up soon. On the other hand, InfiniBand use is climbing steadily.

The following table shows the June 2008 and 2009 interconnect families on the Top500 list.
I grouped any interconnect that had less than 1% share in 2009 into the "other" category.

Interconnect	6/2008	6/2009
Myrinet	2.4	2.0
GigE	56.6	56.4
IB	24.2	30.2
Proprietary	8.2	8.4
Other	8.6	3.0

The first thing to notice is that GigE still dominates the list with a 56% share as it did a year ago. Also of note, number 16 on the list, from University of Toronto, used Quad Xeon E55xx and GigE! The only other big change is the continued growth of InfiniBand (from 24% to 30%) and decrease of the "other" category.

So why the confidence in 10 GigE? Simple, at one point GigE was as expensive as 10 GigE is today, but due to the commodity uptake, the price came down to the "it is free on the motherboard" option. Plus, many users like the "plug and play" nature of Ethernet as it is well understood technology.

In closing, the Top500 is a single benchmark and not the only measure of HPC interconnects, but it does provide an interesting snapshot of what people are using. Right now it seem the biggest competitor to 10 GigE might be GigE.

The answer to "Who Uses HPC?" is undoubtedly the scientists and engineers that are developing the products and processes that touch our lives. Perhaps more importantly, is who benefits from HPC? The answer to that is of course, "you and I." As our ability of capture and manipulate "our world" in digital form continues to grow, so does our ability to make better decisions and create a better future.

Everyone in the high-performance computing industry watches the Top500 List. Twice a year the worlds fastest computers (mostly clusters) are ranked by how well they run a very large benchmark program.

The Top500 List is an interesting competition that measures a great deal of computing muscle but it also helps track the history HPC systems; the type and number of processors, operating systems, amounts of memory, and system architecture are all detailed for each system on the list.

The ranking extends back to 1993 when the list began. As a mater of fact x86 clusters are have only recently jointed the list. The fastest x86 cluster in November 2008 used 51,200 cores to run the benchmark.

While these levels of computing are heroic, they actually don't reflect how most HPC systems are constructed.

So, how many cores does an average HPC program use? It depends on who you ask, but there seem to be three distinct types of cluster systems:

• Small (64 processor cores or less)
• Large (over 64 cores but below 10,000 cores)
• Staggering (over 10,000 cores)

While 10,000 cores is an exciting number to visualize, clusters that use the smallest number of cores actually represent the largest segment of the HPC market.

In general, users that run smaller programs often share a cluster with other users. Conversely, the mammoth programs that use thousands of cores often consume every core in the cluster. To differentiate between these two types of usage, clusters are often classified as either a Capacity or Capability system.

Capacity clusters are the most common and are used to deliver a certain amount of "computing capacity" to the end users. For instance, a capacity cluster may support hundreds of users running any number of programs. These programs require a number of cores much less than the total number of cores in a cluster. In these types of clusters, all the compute resources are managed by a job scheduler which determines which programs (jobs) run on the cluster.

A Capability cluster is designed to handle (or be capable) of running large groundbreaking programs that were previously not possible to run. These systems usually push the limits of cluster technology because large numbers of systems must work together for long periods of time. In contrast, a capacity cluster has the ability to tolerate failure and continue running user programs.

Chances are if you are using a cluster it is a capacity system. If that is the case, the Top500 might interest you, but it probably has very little to do with your performance. If you are one of the the few high-end capability users, the Top500 is just for you.