Click here to download "Univa UD Product Brief"

Lately, I have been paying a lot of attention to the economy watching for signals of any effect on high performance computing spending. The article was based on a report from Gartner in 2008, which in principle flew in the face of the market reality we are seeing, so I pressed on and looked into the story behind the story.

So I pressed on…

“Gartner found spending even in times of economic uncertainty is supported by two factors: businesses are investing in improvements to internal processes aimed at reducing costs, along with their own innovations, and that globalization allows IT services providers to mitigate the risk of weakening demand by operating in more markets.”

And there it was. This reflects exactly what we are hearing from customers – it’s all about “improvements to internal processes aimed at reducing costs.”

To start, let’s frame the circumstance. First and second generation high performance compute infrastructures were built by IT as a service to engineering, research or science. The systems replaced manual processes and were thought of as a means to an end: more computing power. The construct was “throughput”– that is, simply a way to crunch more data in less time. The economic benefit of these generations, usually “get to market faster,” was fairly straightforward.

Over time, improved software modeling developments made possible the ability to simulate more business processes, and that in turn fueled the need for more  memory and compute power. The ability to use COTS components contributed to an explosion in use, size of problems solved and the number of computers in a cluster (also known as a ‘farm’).

After years of unprecedented growth, HPC has evolved from an engineering tool or asset to a far more lofty status including the fastest growing segment of the IT industry in server shipments, roughly one quarter of all CPU shipments; and an invaluable rank in the research, innovation and product development chain of most of the world’s leading companies in virtually every industry.

HPC  underpins product innovation

Having become linked to an organization’s value chain, like the personal computer many years before it, HPC environments are increasingly being viewed differently than in the past. No longer solely the pet projects of visionary CxOs, compute clusters have become a strategic element of the product or service and a competitive advantage. High performance computing has developed from an asset to a value contributor by increasing operating margin of product development through time or efficiency gains and by helping revenue growth through product and service innovation.

That would suggest that the infrastructure has become as visible as other key elements in the value chain, such as manufacturing. That visibility will incur regular lifecycle management inspection as part of business process improvement projects linked to aligning IT spending to corporate goals. Moreover, a greater amount of efficiency and value will be sought from such a strategic ‘asset’. (Similar to how Dell continuously seeks to improve its manufacturing process to improve margins)

Perhaps I should be blunt here. High performance computing is a strategic asset of most (if not all) of the organizations that have deployed it. As such, these organizations have recognized the direct link between investing in the infrastructure and the business rewards (spend more, get more). This is somewhat counter-intuitive to business systems IT where the goal is taking the cost out of the infrastructure.  Thus, processes will need to be developed and improved and a professional and commercial approach will ultimately need to prevail. And it is these developments that will drive requirements back into the ecosystem that seeks to sell solutions to this community.

Seeking Efficiency

So where will this efficiency come from? With the explosion of size and sheer numbers of computers used in a cluster, many organizations have been forced to adapt existing processes or create new processes, including the codifying of workflow, script wrappers and run books. This has only increased the complexity of the environment.  Costs can be driven out of clusters through the development of a sustainable growth model that considers size, complexity, pricing paradigm and inclusiveness. Additional reductions can be realized by employing a systems management software stack that considers the holistic environment and not simply a single aspect of its use. Often, the server:admin ratio can be increased significantly if the maintenance of the system can be offloaded to automation.

Conclusion

Professional management of IT budgets focuses and prioritizes projects on the creation of shareholder value. HPC projects, as we have described, are clearly in this category. However, budget management processes will always seek improvements and efficiencies. IT organizations are reacting and clearly have begun to dictate a new set of requirements to their vendors. The success of the Intel Cluster Ready program is a prime example of this.

I find it amusing if not ironic that organizations have come to expect innovation and more automation from the very software that manages the computing environment that enables their innovation.

An obvious recommendation would be to look for clear signs of innovation from prospective vendors. As vendors implement solutions at different sites their product should reflect the best-of-class ideas of how to manage a cluster. Each site will have very insightful methods or ideas that should be included and made available to a broader set of users. This will allow one to benefit from the ‘network effect’ of improvements in features and functionality across the entire systems management stack.

This ‘network effect’ will drive the most efficiency into the lifecycle management of the cluster. These improvements should allow IT staff to focus on more valuable and strategic projects instead of forcing them to individually learn the same ‘tricks’ and lessons as other organizations every time they push the physical or logical boundary of a system component.

Through this process HPC infrastructures will increase value, become simpler, and IT staff will be able to focus on big picture and interesting business problems. The days of tinkering and DIY may be on the decline.

HPC is complex. I haven’t forgot that – but frankly I dismiss that as a crutch. Personally I think there are harder things like learning when to keep your mouth closed in front of your wife so as a to not create more work for yourself (like my stone work, deck and gardening that keeps me busy when the Canadian winter melts into Spring until it returns in the Fall….).

HPC has seen its fair share of openness with many amazing open source projects like Globus, Grid Engine, OFED, Ganglia, Nagios, Lustre to name but only a few. Many of these projects are represented on www.grid.org or included as a component in UniCluster.

HPC has also been the epicenter of commoditization – from engineered performance, proprietary and expensive capability computing machines to off-the-shelf, bundled commodity computing systems available as close to turnkey as possible.

We’ve also had some success, albeit limited, in standardization through the work of many organizations like Open Grid Forum: DRMAA, HPC Basic Profile, BES are some recent specifications.

HPC has taken a BIG step forward through the introduction of Intel Cluster Ready, a specification and program to make it easier to run applications off the shelf. Intel® Cluster Ready is a great example of making complex HPC systems deterministic. In 2005 I set up a meeting with Intel and Dell and we talked about the challenges facing the broad adoption of HPC in savvy and non-savvy commercial organizations. Complexity, engineered performance and commoditization were on a crash course and it was slowing the market, particularly with entry-level customers. We agreed to work together to resolve ISV enablement. The outcome of this ‘French Riviera Summit’ (I was in the South of France during the conference call) was the Intel Cluster Ready program nearly two years later.

A key enabler of this program was open source software. That is, components and reference implementations were readily available because they were open and adoptable. The program would have been hard pressed (and less successful) if elements were closed and unavailable.