My top speculations:

1) Current cluster experts want total control over the solution. They feel like the only solution that will work is the solution that they build from scratch.  Gurus don’t trust a solution they didn’t design/build.

2) Cost is too high. Many solutions are designed in an integrator lab, but they’re still custom jobs for a specific purchaser.  Custom jobs cost more to deploy.  The solution is then to mitigate cost by buying the pieces separately and handling the design in-house.

3) Lack of a standard for clusters. I’d rather say this is the lack of awareness of a standard for clusters for obvious reasons.  If it isn’t well known that clusters need to have a common application interface and a common set of features, then buying clusters slants towards cafeteria style component selection and jamming everything into a final solution.

Some big barriers (but not all barriers) would come down with cluster designs sold in assembly line fashion.  I can select a model of a car then customize some specific options that meet my performance, comfort, and personal preference needs.  Why not a similar approach to clusters?

>> See Related Stories for more

In conjunction with Supercomputing 2010, Intel will release a new update to the Intel® Cluster Ready specification.  I previewed a couple of the biggest changes in the specification in this quarter’s Intel Cluster Ready Newsletter.

The big change is how certified solutions will now present the defined software interface to the application layer. The update will now require defined “sets” of specific versions of libraries rather than the minimum version or later approach that exists today. Backwards compatibility of application to solution is the key driver of this change. The new requirements provide flexibility for solutions to provide a range of defined interfaces to the application layer, and applications will better understand the exact interface present on a solution. Applications that require a specific set of libraries can quickly determine if the set is present on a given solution. Solutions can choose how many library sets, or how far back, to support.

There are other changes in the coming update as well aimed at simplifications, clarifications, and appropriate expansions to requirements aimed at making HPC clusters more attractive and simple to operate. Check out the newsletter for more information.

>> Read the full article

Not familiar with the Intel Cluster Ready specification?

>> Learn more about Intel Cluster Ready

>> Review the Intel Cluster Ready specification version 1.1

>> Join the forum discussion to find out more, or answer this question:

If you had the pen, how would you improve the Intel Cluster Ready specification?

See how Silicon Mechanics, SIMULIA, and a certified Intel Cluster Ready solution are helping a leading global manufacturer accelerate development of its structural building products.

Read more about their HPC solution, or download the case study.

Here’s some specific advantages in terms of the stages:

Stage 1 – Specification: Integrators can design solutions – hardware and systems software – applicable to any application the runs on the interface defined in the specification (found here). Hardware features transparent to the functionality but affecting how well applications execute are abstracted variants of the designs that customers are allowed to customize.

Stage 2 – Integration: Integrators can proceed with testing the design independent of (and ahead of) orders for the design. Design problems get found before solutions hit the floor. At the end of integration is Intel Cluster Ready solution certification demonstrating that the design specified presents the proper interface to the application layer and functions correctly. The design becomes a process for building a solution and the certification is final proof that the process builds the design. Intel provides the Intel Cluster Checker to solutions integrators to help with this step.

Drawing a virtual line around what the solutions integrator does in stage 1 and 2, we define an amount of work required to get a design ready to go to market. Hopefully, the advantage is already clear – this effort only needs to be done once per design! At the end of the day, a process exists to stamp out copies of a design, and that design has a certified and known interface to the application layer. Cluster purchasers choose configurations of the design based on their application needs, but they’re freed from having to know what’s under the hood.

Stage 3 – Manufacturing: How parts come together is going to be widely variant amongst integrators. However, now there’s an identified BOM – hardware and system software – with instructions on how to put the parts together into a known, proven solution.

Stage 4 – Deployment and testing: Deploying a copy of a certified design becomes a canned process. Stage 2 defined the process, and stage 3 delivered the parts. Stage 4 is following the instructions aligned with the method of delivery to the cluster purchaser (methods of delivery: full assembly in factory, full assembly at customer site, anywhere in between). Intel Cluster Checker helps here too. The same tool that was used to certify the design also verifies copies of that design. Now there’s a discrete point that defines the deployed cluster is ready to go. Best of all – the tool stays with the cluster and provides immediate and continual checking by the customer that the system is working. Nothing like being able to tell the customer, ‘as soon as I’m gone, you can run this tool again to see it work for yourself. In fact run it as often as you like to verify system health.’

Draw another virtual line around the integrator parts of stages 3 and 4, and you get the repeatable process done once per order. Tie in the customers’ needs for application configuration in stage 1 at the point of sale then tie in the proof of system check in stage 4, and you have a streamlined pipeline for cluster orders. Customers don’t need to get into the fine points of making and building clusters, and the streamlined approach allows production costs to decrease. Flexible solutions meeting a wide range of needs and at cheaper cost for production and support. Seems like that’s a good recipe for growth.

>> See Related Stories for previous articles and more information.

View Brock Taylor’s interview on Parallel Programming Talk on Intel Software Network TV.  Hear how Intel Cluster Ready started and details about the problems being solved for the HPC cluster ecosystem. Making clusters easier for the ecosystem and cluster users.

>> Learn more from Brock’s blog articles.

Once again, this is an extreme case and is meant to be the opposite end of the spectrum. Really looking at a turn-key solution here that either arrives ready to go or is built on-site by the solutions integrator and then turned over to the purchaser.

Stage 1 – Specifying the parts: all solutions integrator. We’re looking at the extreme, here, so the predefined solution has everything predefined down to the length of the power cord.  The only work the purchaser has to do is say, ‘I’ll take it.’ No prior experience needed.

Stage 2 – Integration: all solutions integrator. Obviously, can’t lock customers into a purchase without being able to build it and ensure it works. The purchaser, similar to stage 1, doesn’t need to know how to make the components work together.

Stage 3 – Manufacturing: all solutions integrator. Parts have to be assembled and the software procurement done according to the predefined design. Here, the purchaser is continually looking for the shipment tracking number to see when the system will arrive, but other than that, has nothing to do.

Stage 4 – Assembly and final testing: mostly solutions integrator. System is installed and deployed per the design. There’s always some level of confidence testing done by the cluster purchaser after the keys are turned over, but the more rigorous testing is likely done by the solutions integrator. Hopefully, the integrator left instructions on how to check the oil, tire pressure, wiper fluid level, air filter, etc. before leaving.

So, this approach is all on the solutions vendor to design and deliver. One problem lies in defining the right solution before there is an order for a cluster. This means that predefined solutions likely target a known application or set of applications. Throw in an application that isn’t part of the target list, and the solution may or may not support the new requirement. The solutions, therefore, aren’t as flexible to purchaser needs in the future.

Since the solutions integrator is doing all the work, you get some added cost as well. Having the solution predefined does allow a manufacturing line approach, however, that can keep cost in check. It would be expected that the first two stages are only done once per design, and the solutions integrator can set up a canned process for manufacturing and deploying the systems in stages 3 and 4.

To me, this model seems much more likely to attract more users towards using clusters to expand computing solution power. It seems to be the road less traveled, however, which implies that it isn’t as easy as it sounds. The flexibility of the solution is one of the keys, I think. Most customers want some level of customization, and with customizations come complexities in the processes of integration, manufacturing, and deployment.

There has to be some amount of commonality between HPC solutions that allows solutions integrators to produce and sell solutions that are flexible enough to meet the demands of a wide range of applications. Plus, there is still an expertise barrier that can keep potential cluster users away. Once the solution is in house, you still have to take care of it and know what to do to keep the system humming.

Intel Cluster Ready is an architecture with tools to help address these issues. Next post, I’ll try to explain (in a high level) how.

>> See Related Stories for more information.

I am somewhat hi-jacking the definition of a dark cluster.  I’m really looking at a cluster where the purchaser buys all the parts and assembles them – pure do-it-yourself approach. The stages break down with the purchaser doing pretty much all the work.

Stage 1 – Specifying the parts: All purchaser.  Need to have the expertise to determine what parts are needed – that includes networking fabric, storage needs, OS, provisioning system, and drivers.  The parts vendors are just selling their components, so the purchaser has to do all the research and effort to determine what the end solution needs to be.

Stage 2 – Integration: Pretty much all purchaser again.  Bad thing about the about this is that the integration of components likely occurs at deployment time.  If there’s an incompatibility of components, it probably isn’t discovered until late in the game.  Vendors involved may provide some assistance in debugging issues related to their individual component, but that may also require a service contract.  So, again the expertise level required by the purchaser is high and the effort is mostly theirs.

Stage 3 – Manufacturing: This is the one stage that the vendors are actively a part of.  Again, the parts have to come from somewhere.  Parts roll off the assembly line and are shipped off – beyond that, you’re on your own.  The purchaser still has to buy all the individual pieces and take care of the software procurements.

Stage 4 – Assembly and final testing: Similar to stage 2 in this case, it’s pretty much all purchaser.  In fact, in dark clusters stage 2 and stage 4 may blend into one.  As the integration of parts commences, so does the actual assembly of the cluster.  Once the integration is done, the cluster enters testing before being put into use.  There’s another problem – you also have to know how to test the cluster in addition to knowing how to put it together.  It actually takes a lot of knowledge to properly check if the system is running correctly.  Miss a “silent” error, and you wind up with costly downtime in the future.

So the expertise barrier for do-it-yourself clusters is pretty high.  You have to already be a cluster expert or factor in the ramp to becoming a cluster expert into the purchase.  Mistakes can be costly in time and money.  The cluster works fine, but the application for which the cluster was purchased for doesn’t run.  Oops.  The upside is that once you have become an expert, you can reduce the number of missteps with each new purchase.  Plus, many people in the community have tried to make it easier (and made great progress in doing so).  Still, this is a big hurdle to get over.

The big problem is really who is the cluster expert.  In a large company, this is likely dedicated staff in the IT department – a wise investment and required for large systems used by many people.  In a small company, however, this might be one or more of the primary users of the system.  That means engineers and scientists fill the role of cluster administrator, and that’s time taken away from producing results in order to build and maintain the system.  That type of investment is much harder for a small company to make.  It might also be hard for a small department within a large company to justify the cost of their own IT resources required for a much-needed workgroup cluster.

In academia, this barrier used to be solved by throwing graduate students at the issue.  I’d bet there is more than one thesis that morphed from ‘this is how to do science’ into ‘this is how to build a cluster to do science.’  It’s still time lost, though, that should be directed to doing work other than cluster administration.  Plus, there is turnover.  At some point, graduate students demand their degrees, leave school, and go off to small companies that ask them to administer their own cluster.  The point being that it’s a whole lot better for a researcher to be running a CFD simulation rather than figuring out why an RPM that was installed yesterday is now missing on one of the compute nodes of the cluster.

Solutions integrators exist for a reason – building a proper HPC cluster is more than just installing software on hardware.  The do-it-yourself approach can help keep financial cost to a minimum with respect to the parts, but that must be weighed against the cost in expertise and time to build and maintain it.  Because of the latter, broad adoption of HPC cluster use isn’t likely to go down this path.

>> See Related Stories for more information.

There are many different ways that someone can actually buy and deploy a cluster.  Complete do-it-yourself experts buy all the parts themselves and do all of the work to assemble the solution.  These systems are called dark clusters because only the buyer really knows that the end-game is an HPC cluster.  There’s the turn-key cluster where the buyer purchases a fully working and assembled solution from a vendor.  Then there’s all kinds of ways in between with varying levels of interaction between cluster buyer and vendors selling solutions and parts.

With all these different paths, though, I’ve looked for common steps that are always performed during the purchase process.  After many hallway conversations and lots of coffee, I see there are four stages that always occur when buying an HPC cluster.  The variables in the different paths are the the division of labor, expertise, and cost that is divided between buyer and seller of the solution.  I define the four stages as follows:

Stage 1: Specifying the cluster.  This is the determination of what hardware and what software is required for the cluster.

Stage 2: Component integration.  Making sure the components spec’ed out in stage 1 actually work together is crucial.

Stage 3: Component manufacturing and software procurement.  The parts have to come from somewhere.

Stage 4: Physical deployment and testing. Where all the parts come together, the solution is built, and there’s some method of declaring the system ready for use.

Now, the model isn’t without flaws.  There are paths where the above stages are combined or may overlap, but I think the steps are all still performed.  With this in mind, I am exploring how different paths present barriers to wider adoption of HPC clusters as problem solving resources.  Some paths put too much expertise demand on the cluster buyer while other paths may put too much cost on the systems integrator.  A volume cluster market has to find paths that both lower the expertise required to buy a system and provide a cost-effective approach for vendors to deliver and sell those solutions.

Read the Solution Brief to learn more.