View the Univa UD video

Intel Cluster Ready is enabling this innovation

The more that things change, the more they stay the same. The fact is that HPC systems will change over time. A newer compute node is likely not a copy-exact replacement of the original failed node, even when acquired from the same manufacturer. Differences can be magnified in larger systems where “MTBF laws” create incidents on a frequent and unpredictable basis (think of Murphy’s law applied to large high-use compute systems).

Often, there are multiple admins and even contractors that provide the care and feeding of the cluster. Many people, many users, many requests and many changes give rise to complexity and can create confusion. Vast amounts of time can be spent figuring out what happened, who did what and when it was done. This reduces efficiency of the system by prolonging downtime of valuable resources, delaying projects and interrupting expensive manpower needed to resolve situations.  Same problem as it ever was, even as individual component technologies improve and become more efficient themselves.

Sometimes you need more…

Package and configuration management unquestionably improve many aspects of maintaining a cluster; however, there continue to be gaps in the operational aspects of sustaining HPTC environments. This is an area of management that Univa has been innovating in partnership with Texas Advanced Computing Center (TACC). With five clusters and nearly 65,000 cores and limited manpower, TACC had to find better ways of managing large-scale clusters. The largest cluster, Ranger, has 3,936 nodes and is managed by both TACC and contract staff. Problem deduction, tracking and resolution have been greatly simplified with a set of operational tools that capture and codify TACC's collective (and impressive) man-years of experience.

Univa's HPC systems management and product development expertise are being leveraged to inventory, generalize and productize these systems. In short order Univa will release substantial improvements to UniCluster's systems management capabilities based on our work with TACC. These innovative systems, best practices and tools will revolutionize HPC systems management and redefine "cluster management."

What is critical to point out is that this innovation and the operational efficiency would not have been possible for a company the size of Univa without a program like Intel Cluster Ready. Since our engineers and product folk are free from solving the same, literally basic problems of provisioning and configuration, we were able to allocate some of our resources to advance the "science" of cluster operations.

Likewise, end-user (like TACC) researchers and admins' time will be freed up to make it possible to accomplish their objectives: that is, to enable science or grand problem solving.

Over the next few posts in my personal blog I will describe these operational tools and systems in more detail. Anyone interested in learning more or previewing the tools may contact Univa at any time. (shameless plug!)

Over the past six years I have discussed management aspects of clusters with a broad cross-section of cluster administrators (large and small), each with their own experiences and views. It was this feedback and perspective that inspired my discussions with Intel and Dell about ISV enablement that ultimately paved the way for Intel Cluster Ready in 2007.

The purpose of the Intel Cluster Ready program was essentially to “level the playing field” for three constituents:

•    ISVs who had to deal with too many incompatible stacks,
•    End-users who faced too many vendor-specific products,
•    The HPC ecosystem that wasted time solving the same problem again and again.

In this post I wanted to highlight what I consider “circular irony”: managing a cluster requires cluster ‘management’. That is, cluster management is more than provisioning, scheduling and packaging some pre-existing tools, and although it seems like middleware vendors don’t seem to get that, there is hope.

It has always struck me as ironic that the various cluster management packages available today rely on a somewhat narrow set of tools that are positioned to comprise all of cluster “management”. Missing, are tools for the operators to actually do something, or affect changes to the cluster and I don’t mean monitoring or staring at consoles. It is, after all, a fact that operating costs typically exceed the capital outlay over the life of the cluster. Sites are generally left to create their own tools or to muddle along forgoing the “network effect” of productized systems software.

Increasing Efficiency of Strategic Assets

High Performance Technical Computing (HPTC) has become linked to an organization’s value chain. Clusters are the dominant architecture in HPTC and these environments have become a strategic element of the product or service, often underpinning an organization’s competitive advantage. For such a strategic asset, one could make a very strong case for the development of a greater amount of efficiency and value!

And Intel Cluster Ready is helping turn that hope into reality.

A key benefit of the Intel Cluster Ready program is that it has allowed Univa engineers to focus on the countless yet-to-be-solved software aspects of the management and operations of a cluster by providing a replicable baseline from which to work.

Cluster users – engineers, scientists, “quants,” analysts and researchers – are interested in the outcome of the science or the software aspects of the cluster. An admin or operator of a cluster is primarily tasked with supporting that: maintaining uptime and handling hardware and software changes as required in support of system use. This involves adding, removing and updating software, installing security patches and replacing failed hardware.

It’s in this stratum that cluster systems software has the greatest room for improvement and thanks to ICR, Univa is able to.

After all, it’s about time.

Lately, I have been paying a lot of attention to the economy watching for signals of any effect on high performance computing spending. The article was based on a report from Gartner in 2008, which in principle flew in the face of the market reality we are seeing, so I pressed on and looked into the story behind the story.

So I pressed on…

“Gartner found spending even in times of economic uncertainty is supported by two factors: businesses are investing in improvements to internal processes aimed at reducing costs, along with their own innovations, and that globalization allows IT services providers to mitigate the risk of weakening demand by operating in more markets.”

And there it was. This reflects exactly what we are hearing from customers – it’s all about “improvements to internal processes aimed at reducing costs.”

To start, let’s frame the circumstance. First and second generation high performance compute infrastructures were built by IT as a service to engineering, research or science. The systems replaced manual processes and were thought of as a means to an end: more computing power. The construct was “throughput”– that is, simply a way to crunch more data in less time. The economic benefit of these generations, usually “get to market faster,” was fairly straightforward.

Over time, improved software modeling developments made possible the ability to simulate more business processes, and that in turn fueled the need for more  memory and compute power. The ability to use COTS components contributed to an explosion in use, size of problems solved and the number of computers in a cluster (also known as a ‘farm’).

After years of unprecedented growth, HPC has evolved from an engineering tool or asset to a far more lofty status including the fastest growing segment of the IT industry in server shipments, roughly one quarter of all CPU shipments; and an invaluable rank in the research, innovation and product development chain of most of the world’s leading companies in virtually every industry.

HPC  underpins product innovation

Having become linked to an organization’s value chain, like the personal computer many years before it, HPC environments are increasingly being viewed differently than in the past. No longer solely the pet projects of visionary CxOs, compute clusters have become a strategic element of the product or service and a competitive advantage. High performance computing has developed from an asset to a value contributor by increasing operating margin of product development through time or efficiency gains and by helping revenue growth through product and service innovation.

That would suggest that the infrastructure has become as visible as other key elements in the value chain, such as manufacturing. That visibility will incur regular lifecycle management inspection as part of business process improvement projects linked to aligning IT spending to corporate goals. Moreover, a greater amount of efficiency and value will be sought from such a strategic ‘asset’. (Similar to how Dell continuously seeks to improve its manufacturing process to improve margins)

Perhaps I should be blunt here. High performance computing is a strategic asset of most (if not all) of the organizations that have deployed it. As such, these organizations have recognized the direct link between investing in the infrastructure and the business rewards (spend more, get more). This is somewhat counter-intuitive to business systems IT where the goal is taking the cost out of the infrastructure.  Thus, processes will need to be developed and improved and a professional and commercial approach will ultimately need to prevail. And it is these developments that will drive requirements back into the ecosystem that seeks to sell solutions to this community.

Seeking Efficiency

So where will this efficiency come from? With the explosion of size and sheer numbers of computers used in a cluster, many organizations have been forced to adapt existing processes or create new processes, including the codifying of workflow, script wrappers and run books. This has only increased the complexity of the environment.  Costs can be driven out of clusters through the development of a sustainable growth model that considers size, complexity, pricing paradigm and inclusiveness. Additional reductions can be realized by employing a systems management software stack that considers the holistic environment and not simply a single aspect of its use. Often, the server:admin ratio can be increased significantly if the maintenance of the system can be offloaded to automation.

Conclusion

Professional management of IT budgets focuses and prioritizes projects on the creation of shareholder value. HPC projects, as we have described, are clearly in this category. However, budget management processes will always seek improvements and efficiencies. IT organizations are reacting and clearly have begun to dictate a new set of requirements to their vendors. The success of the Intel Cluster Ready program is a prime example of this.

I find it amusing if not ironic that organizations have come to expect innovation and more automation from the very software that manages the computing environment that enables their innovation.

An obvious recommendation would be to look for clear signs of innovation from prospective vendors. As vendors implement solutions at different sites their product should reflect the best-of-class ideas of how to manage a cluster. Each site will have very insightful methods or ideas that should be included and made available to a broader set of users. This will allow one to benefit from the ‘network effect’ of improvements in features and functionality across the entire systems management stack.

This ‘network effect’ will drive the most efficiency into the lifecycle management of the cluster. These improvements should allow IT staff to focus on more valuable and strategic projects instead of forcing them to individually learn the same ‘tricks’ and lessons as other organizations every time they push the physical or logical boundary of a system component.

Through this process HPC infrastructures will increase value, become simpler, and IT staff will be able to focus on big picture and interesting business problems. The days of tinkering and DIY may be on the decline.

Most of us credit Ford with the introduction of the assembly line – the “specialization” of the workforce. While true, this is more of an outcome of what Ford did. Ford knew that in order for him to fulfill his vision of a ‘car for everyone’, the way they were made had to change.

At the time every automobile was hand made and practically custom. Many companies were integrated and produced their “own” components. What Ford saw was the opportunity to adopt existing “good-enough” components and incorporate them into his design. Automobiles have a great deal of components: wheels, tires, brakes, engines, starters, seats (to name but a few).

Ford had the benefit of new perspective: he could connect the evolution in the typewriter and clock industry to the manufacture of automobiles. The choices before Ford came down to this: engineer a better automobile using proprietary components or embrace available components and drive towards volume.

The HPC industry, like the clock, typewriter and automobile market before it, is ready for adopting a standardized design and leveraging interchangeable parts. And this is where the inherent value of Intel Cluster Ready comes into play.

And from my point of view – IT’S ABOUT TIME.