Intel Cluster Checker can run a variety of tests depending on what the user is trying to accomplish. For general wellness of a cluster, Intel Cluster Checker offers five levels of thoroughness. For automation, we'll focus on levels one, three, and five.

Wellness Level One is a very short run of tests that check basic connectivity throughout the cluster as well as basic uniformity checks amongst the nodes.  This test focuses on BIOS settings, and processor, memory and system configurations. The level one tests are quick and show that the cluster is online and ready for use.

Wellness Level Three is the default run level for Intel Cluster Checker. It builds on level one and includes more rigorous modules that test parameters like disk and memory bandwidth, MFLOPS, and network performance. It also performs an in-depth hardware uniformity test along with an Intel MPI Collectives and message Integrity test. It takes a bit longer to run this check that assures the performance of the hardware is up to par.

Wellness Level Five adds a packages test, comparing currently installed packages with a generated list of expected packages at a given time. It also runs the HPCC module (a performance benchmark test). The level five tests can take a bit longer but assures the users that all the pieces of the system are working in harmony. This test is useful for the admin to make sure a user hasn’t installed or uninstalled anything that may affect the cluster.

For my clusters, I have set up the following schedule that runs these wellness levels at various times.

• Level One - set to run each weekday
• Level Three - set to run once a week, on Saturday
• Level Five - set to run monthly, on the first Sunday

For our lab with multiple smaller clusters this automation schedule is perfect, allowing users to login to one of the clusters, see when each of the wellness levels passed and know that their jobs will run as expected.

Every cluster has a different purpose and a different load. Running tests at the intervals or frequency I have set may not be feasible or needed for every cluster. However, automating the test runs on a schedule that meets your needs will help assure that your cluster performance remains optimal, and frees up the cluster(s) for jobs during normal business hours.

References:

>> Review Part 1

>> About Intel Cluster Checker

>> Intel Cluster Checker Knowledge Base

Whether you're a cluster user that expects optimal performance and functionality every time you run a job, or a system administrator that needs to keep the cluster in perfect working order for your users, running checks on a regular basis is important!

A great way to ensure that your HPC system, certified Intel Cluster Ready, remains in the same great shape as when it was first built, is to run the Intel® Cluster Checker tool regularly.

While it’s easy enough to run Intel Cluster Checker on your cluster(s) once a week, I've found there are times when I was too busy, or simply forgot to run a check. For this reason, and because I have multiple clusters to check, I developed a method to automate Intel Cluster Checker runs. This automated solution runs Intel Cluster Checker and reports any errors directly to me and/or the system administrator. Not only does automating this process save the administrator a substantial amount of time each week, it will eliminate the chance of missing a run, and ensures that the cluster remains in optimal health.

A Cluster That Checks Itself

In order to automate the process, it is important that your Intel Cluster Checker passes a manual run first. To do this, make sure the configuration file is optimized to ensure the cluster is operating at its best. With the perfect configuration on hand, a script is then needed to setup the system, run Intel Cluster Checker, and report the results.

For the first phase of the process, I wrote a script that sets up the environment and runs the most in-depth wellness check in Intel Cluster Checker. I used cron to schedule my script to execute once a week. After the results are complete, the script updates the message of the day to show when the last check ran and the results. If the check fails, the log file created by Intel Cluster Checker will be copied to the specified directory where it can be accessed and analyzed.

The cluster is now setup to check itself, in the middle of the night, and report its status upon login!

Read Part 2 for more about scheduling test runs, and the different levels of wellness tests in Intel Cluster Checker.

References:

>> About Intel Cluster Checker

>> Intel Cluster Checker Knowledge Base

How it started and evolved

Coming into HPC in 2005 was a great learning experience for me, as most of my former technical experience had been networking related. This included a position where I utilized WAP related technologies in the early "Wild West" days of using 802.11 wireless networking to span great distances, providing cheap broadband to rural commercial interests to save them the costs of ISDNs and even more cost-prohibitive T1s.

So, I started in 2005 by working for a year with one of the Intel HPC teams as a contractor, before officially joining Intel in 2006. Among my first tasks at Intel was to build a cluster. At that time this was accomplished by going through a how-to "recipe" document which contained a great many steps in order to manually set up the head (master) node, and then clone that node with “dd” (which was  slow) or kick-start (much better) and then continue following the steps to create a cluster "by hand." This method did not scale well beyond ~8 nodes as there was much room for operator error.

The next "evolution" I experienced was utilizing some home-grown setup scripts from a predecessor and writing some of my own. These were primitive setup scripts that left a lot to be desired, however, easier than the “by hand” method. Among these scripts was a relatively simple cluster health check called "cluster-checker.sh." If given a correct hostfile, this script would go through and check the compute nodes for simple problems such as ping, rsh/ssh connectivity, and a few others.

Another useful find was OSCAR - Open Source Cluster Application Resources. What was really neat about OSCAR was that the developers had engineered a setup framework that greatly simplified the setup of a cluster, in most cases. This seemed like absolute luxury at the time. After I joined Intel, I wanted to be a part of this group that had helped make my job easier. (My later experience with NPACI ROCKS was similar as it was also an easier way to set up and configure a cluster.)

Developing Intel Cluster Ready

The research and development work associated with the Intel® Cluster Checker tool was an exciting time! We had to ensure that:

• The tool enforced the Intel Cluster Ready Specification with the – compliance flag.
• The wellness checks covered a wide range of functions and could execute in acceptable times.

This required making some tests fairly generic to apply to a large range of hardware/OS/clusterware solutions. The process also included a campaign to encourage ISVs to register their applications as Intel Cluster Ready, to verify that their applications will run successfully on any certified Intel Cluster Ready system. These registrations added critical libraries and binaries to be placed into the software images that allowed the registered applications to “just run.”

Making it easier

With all of these changes, the process of building out and troubleshooting an HPC cluster became significantly easier. These processes encouraged the further enhancement of existing recipes we had developed over the years to become “Intel Cluster Ready Reference recipes.” The development of these recipes was key in helping to better understand what it meant for the software stack to truly be Intel Cluster Ready compliant, and search out the best ways to facilitate the automation of these recipes working with the commercial provisioning vendors.

We're proud to say that since we began this endeavor just 3 years ago, we have a number of provisioning vendors that offer ICR compliant clusterware solutions to OEM's to sell to their partners. Also, equally important, we are very proud of our OEM and system partners who have modified their own clusterware stacks to align with Intel Cluster Ready. They are the ones who primarily benefit from Intel Cluster Checker, for many, it has become a critical tool from the engineering of a cluster recipe (software stack), to checking that stack in manufacturing. This ensures that the customer gets a certified Intel Cluster Ready system, and the customer can use the Intel Cluster Checker tool to test the cluster wellness, or diagnose issues with their cluster system. Simplifying the entire process.

This goes full circle when the customer calls the OEM or provisioning vendor for support and poses the question, “Is your system Intel Cluster Ready?” If so, the process becomes much easier with a tool that can establish the functionality/wellness of a cluster, including diagnosing many common problems we see when dealing with clusters on the system side.

We have seen great success in working with the provisioning vendors and OEMs regarding compliance to Intel Cluster Ready. This is in no small part due to the features and functionality provided by Intel Cluster Checker. While its primary purpose lies in enforcing the Intel Cluster Ready specification, it has expanded into a tool with over 100 various tests that deal with consistency, performance, and correct configurations. With the auto-configuration tool, it can easily handle heterogeneous clusters, perform single node and network performance tests, and much more.

It has changed HPC for me, are you ready for it to change HPC for you?

Learn more about Intel Cluster Ready, and check back for future blogs on related HPC topics!

>> Read the Solution Guide

I think I do not have to state that setting up an HPC cluster sometimes is an adventure of compiling and installing different program versions and libraries. Nevertheless, when having all up and running, the Intel Cluster Checker is a really nice tool to verify that all services needed for your HPC cluster are set up correctly and you haven't forgotten one of those tiny details during the installation of your cluster.

My last experience with the Intel Cluster Checker and the newly installed HPC cluster were a bit different from the ones before. The first steps ran smoothly and everything seemed to work fine, with me having learned from my previous first Intel Cluster Checker experience. The tests were successful, I altered the benchmark thresholds and all tests ended with a "passed". But in the end, when checking the dmidecode output I got a "failed" on this test. First I thought this was due to some BIOS specific mismatches, which you can exclude. But when I had a closer look on the output file of the Intel Cluster Checker I saw that there seemed to be installed different RAM modules on the compute nodes.  Hmm...the sizes of the memories looked fine, all were 2GB and I double checked the product numbers of the DIMMs that the Intel Cluster Checker reported. Here we got two different types of memories installed. I was astonished on the one hand as I would not have checked this without the Intel Cluster Checker and on the other hand I hardly could believe that there were two different types of memory DIMMs installed. Well, the servers I had installed were Intel Nehalem based and I remembered the days when AMD started with the CPU built-in memory controller and the problems with memories that arose during those times...

I took a closer look at the compute nodes and opened the chassis of  those which were affected. And indeed, there were different types of memory modules built in, but they nearly looked the same and also had the same product number written down on them. After some investigation with our purchasing department I found out that those memory types have been mixed up. Unfortunately it was not possible to pin down the real cause for this confusion.

The DIMMs were swapped to the correct ones in the end and I ran the Intel Cluster Checker again. This time all tests were passed and the results were sent to Intel to verify the Cluster Ready Certificate, which now proudly resides beside the other ICRs we have scored.

First, always use the required and supplied "provisioning system" tools to update a cluster. It may be relatively easy to update all the nodes in a cluster using a couple RPMs and something like pdsh to script the installation of the update - but don't be tempted.  This manual or brute-force method bypasses the software that manages the image on each server. Provisioning systems may reimage nodes after a crash or maybe a node is replaced or added to the cluster. If software updates are applied manually (outside of the provisioning system) then the reimaged node will be inconsistent with the rest of the system. An admin would need to remember all the manual changes and apply them again. It's much better to let the provisioning software worry about that. That is one the reasons ICR required a provisioning system!

Once updates are applied, it's a good idea to verify the cluster is working as it did before the update was installed and it remains compliant with the Intel® Cluster Ready architecture. Many if not most software updates will behave well, but verification helps ensure an update didn't alter or remove a key system component that may lead to application failures. Intel® Cluster Checker provides an easy way to check the compliance after an update. By using the command-line --compliance option, the tool will verify the interface defined by the architecture still exists as before. It's an easy way to check that the update hasn't had any ill effect on the architecture interface used by ICR applications.

Finally, there may be needed updates to the Intel Cluster Checker configuration files to reflect the updated software. For example, if a newer version of the Intel C compiler is installed, the Intel Cluster Checker configuration file should be updated to utilize the newer version. Running the tool would then verify the new installation is functioning on all nodes. It's also valuable to update the list of packages that are expected on each node. The packages test verifies the RPMs installed on each node matches a predetermined list. Using the --packages command-line option will create new package lists based on the current installation (use it after all updates are complete). Save the original list file and set the configuration file to use the updated list. For more information on using the tool, see the Intel Cluster Checker Users Guide.

As we found out in the Web, the Intel MPI implementation has a known bug: the number of available cores is not detected correctly. But it took some time connecting this circumstance to one of our problems, because the original error did not immediately point to it:

Intel(R) MPI Library Runtime Environment (Single-node), (intel_mpi_rt).........................................................FAILED
subtest 'MPI Hello World! (I_MPI_DEVICE = sock)' failed
- failing All hosts returned: 'No one returned Hello World!'
subtest 'mpd shutdown' failed
- failing All hosts returned: 'mpdallexit: cannot connect to local mpd (/tmp/mpd2.console_transtec); possible causes:
1. no mpd is running on this host
2. an mpd is running but was started without a "console" (-n option)'

As it were, the real problem was the program "cpuinfo" of Intel MPI, reporting a wrong number of cores in the system. Setting an environment variable,

"export I_MPI_CPUINFO=proc"

fixed this issue, however.

Another issue with Intel MPI is that it seems to be incompatible with Python 2.6, which comes along with SLES 11:

Intel® MPI Library Runtime Environment (Single-node), (intel_mpi_rt).........................................................FAILED
subtest 'mpd startup' failed
- failing All hosts returned:
'/opt/intel/impi/3.2.0.011/bin64/mpdlib.py:27: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
import sys, os, signal, popen2, socket, select, inspect
/opt/intel/impi/3.2.0.011/bin64/mpdlib.py:37: DeprecationWarning: the md5 module is deprecated; use hashlib instead
from  md5       import  new as md5new'

Although this should not constitute not a real problem, cluster checker seems to be sensitive enough to end with an error code even for warning like this. We could fix it by installing Python 2.4, which, unfortunately, does not exist as a SLES 11 package, so we had to take a source tarball and recompile the package. Strange though, why SLES 11 does not include Python 2.4 as a fallback package for compatibility reasons as there are many Python programs out there based on that version.

In my opinion, this just demonstrates the power of Intel Cluster Ready, or cluster checker, respectively: Intel MPI causes a problem, but the cluster checker does its jobs and identifies a compatibility issue, impressively enough! Thus, the ICR program enables us to catch the problem before it reached the customer level.

A strange error that occurred was this one:

X11 runtime libraries are provided, (X11_libs).........................FAILED
subtest 'libGLw.so (x86-64) >= version 1' failed
- failing All hosts returned: 'missing'

Obviously, the library missing is part of the Mesa package, which is a necessary HPC cluster ingredient from Intel Cluster Ready's point of view. Doing a little bit of research, we found out that nearly all current Linux distributions explicitly remove this library from the standard Mesa package, for whatever reasons.

We could solve that in an elegant way by repackaging the SLES package Mesa with a recompiled one, having the SPEC file adapted in an appropriate way for including the libGLw libraries.

A funny thing that happened was that Korn shell missed the locales:

Korn Shell, (ksh)......................................................FAILED
subtest 'Hello World!' failed
- failing hosts node1 - node3 returned: 'en_US.UTF-8: unknown locale'

But the package glibc-locale seemed to be installed:

root@node1 # rpm -q glibc-locale
gcc-locale-4.3-62.198

What was going on?

It turned out to have nothing to do with ICR incompatibilities of SLES 11, but to be a problem with "xCAT 2" which we use for cluster deployment. As we have tested diskless installations, xCAT defined a number of files and directories to be deleted before creating the compressed root image. In the configuration file

/opt/xcat/share/xcat/netboot/sles/compute.exlist

The locale files are explicitly listed to be deleted before image generation. Having found that out, it was an easy thing to fix.

After solving all that issues, we finally have succeeded in developing a fully automatic deployment procedure for ICR certified HPC cluster based on SLES 11.

My favorite example is the damaged network cable that is dropping data intermittently but not to the point of outright failure. This might result in a support call reporting, "the cluster is broken," and an application that used to run in two hours now takes almost three. It might be intuitive to some to proceed right to checking the link speeds between each node pair, but it's likely that home grown methods are the primary approach to discovering the cause of the slowdown.

Enter Intel® Cluster Checker as the systematic approach finding these problems. The tool provides a diagnostic report by checking the usual suspects that can cause functional issues in the cluster. I have had a damaged InfiniBand cable continue to operate but at half the bandwidth. Intel Cluster Checker doesn't tell me I have a bad cable, but it does provide me data that the bandwidth over the fabric to one particular node is below par. It allows me to narrow in quickly on the failing component.

The inherent value is that there isn't the need to speculate on any particular issue. Step 1 is always to run Intel Cluster Checker and see what it reports. The tool does the investigation work and keeps you from chasing dead ends. It also helps expose those silent issues that are not obvious or routine to check.

Now, as the OS is anything but static, you will have to patch your system at some point.  Maybe you can afford a “real” test cluster, apply the patches, run the cluster-checker on these updated nodes, check that everything is fine and schedule a maintenance period, update the whole cluster and put it back in production. This sounds quite expensive in terms of time and hardware (depreciation and electricity) needed just for testing.

You can also take advantage of the Open Cluster Stack 5 framework that we developed at Platform Computing , take a snapshot of the repository used to provision the nodes, update this snapshot, decouple few nodes from the production cluster and let them use the new updated repository (all of this is done live). You can then validate the updated snapshot. Is everything fine? Perfect, you can then apply the patches on your production cluster (no interruption needed, except a reboot that can be scheduled via your scheduler in case of kernel update) and of course put back the few nodes you used to production. Sounds complicated? Not really when you can use a GUI to perform all those operations. Again, a nice amount of time (money) can be saved and more important, the certification process is not any more tied to a certain point in time (the installation) but is now part of the lifecycle of the cluster.

Let’s have a look at the problem from an IHV point of view. How do you get your hardware / software combination certified ? The answer is simple: you run the cluster-checker against a recipe.

So the first step is to build a recipe. This can be an iterative and time consuming process: install the stack, run the cluster-checker, update the recipe given the cluster-checker runs, and start over again. This is time consuming and can lead to highly complex recipes so you will probably end up at the end of the day with a nice set of scripts that you need to execute prior to successfully run the cluster-checker. Of course, your next step is to write a master script that will sequentially run all these small scripts: the last thing you want is a complex procedure on the assembly line.

Imagine now that most of the scripts needed are not only embedded in the software stack but that the certification process is also part of the development process and more precisely part of the QA process. This means for an IHV that the software that will be installed is already “certifiable” so the cost of writing a recipe becomes a lot more affordable. This is exactly what we’ve done at Platform Computing: using the ICR certification process as part of our QA process. Some steps needed for a successful cluster-checker run are part of the core of our clustering solution; no need to write scripts, just enable a software component in one click. Of course, this can be done through an intuitive GUI. Customization can then be added to the software stack in order to handle for example hardware deviations from an original recipe.

In conclusion, we’ve just demonstrated how easy it is to save time (and money) for both IHVs and users by integrating the ICR certification process not only in the cluster’s lifecycle (Platform Open Cluster Stack 5) but also in the development process (Platform Computing) in order to get a scalable certified solution.

[Excerpt] The Intel® Cluster Ready program provides a standardized,  replicable way to build and run high-performance   computing (HPC) clusters, helping simplify cluster deployment and management. By using Intel Cluster Ready-certified Dell™ HPC clusters, organizations can quickly install and configure clusters to begin running registered HPC applications.

Click here to download "Accelerated HPC Solutions with Intel® Cluster Ready"