Linux and NAS

We would hardly be the first shop in the world to use Linux as a file server. It’s one of it’s natural, most developed roles in the glass house. What may be slightly surprising is that we don’t yet use it for our most critical workloads, although that day may come soon. More on that in another entry.

We take a 2 tiered approach to NAS storage for R&D Support.

In our first tier is the 5 9’s type storage. The stuff that just can’t go down. The bits and pieces that are used on our “assembly line” to build and manufacturer our own products. The kind of storage that, if it were down would idle hundreds of people around the world in R&D and endanger our time to market. And we know with a great deal of pain just how critical this storage is, because we used to use a storage appliance there, and it could not survive our network. It crashed all the time, and we paid for it dearly.

Defining my terms here: A "storage appliance" meaning any hardware/software solution sold only as a NAS or SAN solution.

I should mention as an aside that our network is a hostile place to be for file servers of any type. With over 3600 computers on the R&D LAN alone, triple that for the WAN, running over 45 ‘variants’ of UNIX (counting AIX 3.2.5 as different from AIX 4.1, especially from a network client point of view, etc.), 6 or 8 variants of MS Windows (not counting differences in service packs), and of course Linux ranging back to Redhat 5.2 and up to the most recent versions from Redhat, SuSE, Mandriva, Debian, Feather, Ubantu, some custom, in-house, build from scratch ‘distros’, and I am sure others I am not thinking of at the moment... Well I am sure you get the idea.

Every possible version and combination of NFS versions 2, 3 and 4, and all the various SMB varieties as well. Plus the occasionally buggy client behavior where a published standard was not quite implemented the same way by one vendor as others did it. It is enough to make a NAS server run for cover, and some have. But we can’t have that on the manufacturing line. So five years ago we bought a Compaq TruCluster. At north of 130,000 US Dollars a Terabyte, in 5 years ago dollars, it was not cheap. But it didn’t take too many outages to justify the expense either, and our storage appliance was giving us outages with enough frequency that it took far longer to design the server than it did to sell it to management.

But that left us with a real need to also spend a great deal less on storage for things that do not need that super high level of availability. Things like:

• Archives that are accessed too often to be on tape
• Build trees for older versions of products that are only built ad-hoc to solve particular problems,
• Images of various levels of the OS captures with Ghost or Lab Expert
• VMware virtual disks that are ‘on the hook’, waiting for various deployments
• ISO Images of various Linux Distros, trailing back in time to when we first started downloading and publishing ISO images on the internel Web server for quick download.

We came up with a Linux solution shortly after we had the TruCluster up and running. Our operational theory for the first Linux file server was that commodity hardware should be catching up to the point that we could build “Pretty high availability” (PHA) for less than 5000 US Dollars a terabyte. The LCFS or FBCFS (Low Cost File Server, or Faster, Better Cheaper File Server. Hey, I used to work at NASA... what can I say? Faster Better Cheaper was all the rage till that Mars probe went in hard a few years ago. But I digress...)

The main thing that made this all work was Linux support from 3ware for their storage card. Able to hook up either 8 or 12 PATA disks, and set them up as RAID5 stripes with a hot spare, we were pretty sure that even if the disks weren’t as reliable as the SCSI units in the TruCluster, neither would a failure be customer facing: we’d shelve a few extras as cold spares, and replace them whenever they failed, and at PATA prices.

That first unit was 8 200 GB disks, Redhat 7.3, with a custom kernel from kernel.org, patched with reach-ahead NFS patches, and with IBM’s EVMS installed for volume management. This preceded Redhats acquisition of Sistina, and their new LVM that resulted, and we needed an enterprise class logical volume manager. EVMS filled that to a T.

That server was so successful (and less the $5000 too) that we have since built nine more, with higher density disks, SATA rather than PATA, and sometimes using the 12 port 3ware cards rather than the 8 port, so that we now have 15 terabytes of storage online with these servers. Far more storage online with these than with the TruCluster, which we rather jealously guard. The Linux Distro has been updated over time, and various experiments done with different Distros even.

We have never had an issue with any version of NFS or CIFS/SMB. Linux’s NAS stack of TCP/IP , NFS, and Samba have never had the issues we did with that earlier appliance: even our PHA file servers are more available than those were.

Three of them were not all we hoped though, with various stability issues that took us a while to chase to ground. Better than being “broken all the time” was not *not* PHA. Seven of them behaved exactly as we wanted and more.

Next time: “When good servers go bad --- or --- Even clusters get the blues.”