COTS and who is the Integrator?

A follow-on to my last post, “<insert type of Server here> Server Testing, Part 7: Memtest86”. Some additional thoughts about why one goes to all this trouble to certify their COTS (Consumer / Off The Shelf) based hardware (and of course, Linux).

On my team we often debate how to best go about deploying on any given new application that we need to create and / or support. The points covered are usually these:

1. Is this a wheel being re-invented: can we just do this someplace where there already is a server providing that service. For example, we have a Production IT group different from our team: do they do this? If they do, is the service they provide sufficient for our needs?

2. What will the uptime requirement be once this is in production? Not to mention the ever sticky question of “what exactly is production when you are an R&D support group? What does that term mean? Is it enough that people rely on it? Does the fact that something came into being, had people start replying on it morph it from good idea phase into production phase, or throw it over the wall to the 'unfunded mandate' zone”. Never let it be said R&D support is easy.

3. If there is a “best practice” (buzzword bingo! I win!) here, does that practice apply directly? Is it current? Has that been a “Best Practice” since the Mercury Space Program? If it has, is it because it has reached the state of received knowledge, or has no one thought to question its basic assumptions lately.

4. Related to the previous: what is the current state of the art? Are there new paradigms to consider? Has (wait for it...) “outside of the box thinking” occurred (Bingo again! But seriously: just because that term is overused does not mean it is not relevant.)

The reasons of course to buy a vendor sourced solution are many. A few are:

1. It's not your core competency: you just want the service.

2. You have more money than you have time or people: you just want the service

3. It's complicated, hard to do, and you want to leverage the expertise of those who have it in depth. You might do it if you could, and / or you might just bring in some people and hardware (in house outsource it) for a while while you ramp up.

4. You have no ramp up time: You need the service and you need it now.

Those first two reasons obviously lead one towards Software as a Service / Managed Services as strong possibilities to fill your needs.

Invert all those for reasons why you might do something in-house on self certified gear. On my team I am stocked end to end with people with several decades *each* of experience, and who are comfortable on all sorts of platforms. Our job is to know how to operate is custom environments. The custom environment for us may actually be an environment just like our customers standard environment that we just built to chase a problem to ground.

If we have a problem in relation to this on my team, it is that we are maybe too comfortable with hand built applications and hardware. Here are two examples from our house:

• Our Build and Packaging process is not off the shelf. When it came time to build a new server that could support some of the new requirements of that process, we were on fresh ground. That Linux server and its custom application was deployed nearly 170 days ago, with no downtime. It is built from the ground up out of COTS gear, and with a specially stripped and customized version of Fedora Core.

• Our NAS environment is 2 tier (as discussed in this weblog several times in the past, like “Linux and NAS”): when you can build NAS for 1000 US dollars a terabyte, it changes the paradigm about how you do nearline data storage. Thus my musings on “out of the box” stuff above.

An inverse case from COTS for us: our primary file server is server class gear, on support contracts, and in addition we are fully trained on it: Belt and suspenders. That file server supports our manufacturing line. When it is down (which it is not very often: a couple of times in the last five years) it is an unpleasant time [Ed. Understatement]. Now that it is time to replace that server, we are not looking to take one of our COTS based NAS servers and scale it up. We are still looking at vendor sourced and supported gear. It just won't be Digital (err ....Compaq... uhm... HP) TruCluster this time around (which is a TruShame (tm): that was a great technology)

Here is another case form our place that could have arguments either way:

• Our internal web server is a Wiki: MediaWiki to be precise. We built the server it sits on, installed the application and the MySQL RDB that goes with it. One person on my team occasionally takes care of it. End to end COTS hardware. Another custom configured Linux based off all we have learned over the years from building NAS gear. It has 440 days of uptime as I type this, so clearly it is not a huge problem to maintain.

• The other option would have been to go with someones pre-packaged CMS, or maybe something like MS's Sharepoint.

The internal discussion about this always gets around to one major point: Who does the integration work? Who makes sure that this version of Linux works well on this hardware stack, running this application? If we do it, we are on the hook from now till that application spins off into the heat death of the universe as the support team. That may even be the right thing to do: maybe we'll support it better than anyone else ever could: We have a deep bench. But too many of those type things and it starts being a problem not of ability but of time and bandwidth. IT projects always expand to fill the available people to do them. I have over 350 computers per person on my team. We do have some other things to do.

In the case of the Wiki above, we went the way we did because we wanted something that would:

1. Work with any modern browser. We never know what computer we are going to be sitting in front of from day to day. We are willing to accept that Mosaic 1.0 or Netscape 3.0 might not work... we are realistic.

2. Work from anyplace in the WAN/LAN/VPN

3. Fit into our budget for this project of essentially zero dollars.

4. Allowed for document collaboration, with rollback for mistakes.

5. Search feature had to work.

Point 1 above eliminated all sorts of things, like MS Sharepoint. Point 2 indicated using standard, lightweight HTTP for transport. Point 3 meant we had to wait for some decent hardware with some HA features like RAID-able disks to become available. Point 4 meant it had to be RDB driven. So did point 5.

We tested a few free CMS's and WIKI's and settled on MediaWiki as being the one. Heck if it can drive the entire Wikipedia, it should work for us! It has.

The first line of this article on the Apple web site called “Seeing the Big Picture” about building this special wall of Apple Cinema displays I thought was appropriate to this general area of thought: “...we’re not interested in building something just to see if we can do it...”. Very few people have the luxury in their day jobs of bolting things together to see what they might make at the end of the day. Well... maybe people who work for Google. But other than them. Yes, many of us go home to our home labs at night, put on our propeller beanies and play with tech for the heck of it. We even learn and invent new things that way that we can sometime use at work. “Sharpen our saw” as it were. I'd claim a buzzword triple score there, but actually this is one I truly believe in.

Linux itself is both Vendor Supported and COTS. You need a vendors to support the OS and do all the testing and application integration work, you go with SUSE, Redhat, Xandros, Linspire, Mandriva, and the like. If you don't need all that, you have Debian, OpenSUSE, Fedora, etc. Solaris / OpenSolaris follows that bifurcated path too.

Linux itself came about as a a tool to solve a problem at school that Linus Torvolds was having, and has grown a wee little bit since then. And look at what happened to him: He has been stuck maintaining it ever since! He and a few hundred thousand of his close personal friends anyway.