NAS Server testing, Intro

Over the next few articles I intend to fulfill a request made a little while back. In that email conversation the original idea was a companion article called "When NFS breaks", as a sequel to 'When Linux Breaks". But as I have thought this one over, and as we have worked through the certification of a new Linux based NFS and CIFS bilingual file server, I thought perhaps I would start a series and list how we go about testing a new file server first, and close with a diagnostic article.

Client Mix

It would be fair to say that testing a file server is a non-trivial event here. If there is a downside to heterogeny in the client mix, it is that is does make it a bit harder to make sure that everything works with a single server solution. Not every vendor implements a standard exactly the same way because they all read the standards a bit differently, and not every distro (in the case of Linux) is at the same place in it implementation: Take Fedora Core 4 for example: It shipped at 2.6.11, and is now at 2.6.15, and there were at a minimum all these kernels in between (from my /boot directory):

initrd-2.6.11-1.1286_FC4.img  vmlinuz-2.6.11-1.1286_FC4
initrd-2.6.11-1.1341_FC4.img  vmlinuz-2.6.11-1.1341_FC4
initrd-2.6.11-1.1353_FC4.img  vmlinuz-2.6.11-1.1353_FC4
initrd-2.6.11-1.1363_FC4.img  vmlinuz-2.6.11-1.1363_FC4
initrd-2.6.11-1.1366_FC4.img  vmlinuz-2.6.11-1.1366_FC4
initrd-2.6.11-1.1369_FC4.img  vmlinuz-2.6.11-1.1369_FC4
initrd-2.6.12-1.1387_FC4.img  vmlinuz-2.6.12-1.1387_FC4
initrd-2.6.12-1.1390_FC4.img  vmlinuz-2.6.12-1.1390_FC4
initrd-2.6.12-1.1398_FC4.img  vmlinuz-2.6.12-1.1398_FC4
initrd-2.6.12-1.1447_FC4.img  vmlinuz-2.6.12-1.1447_FC4
initrd-2.6.12-1.1456_FC4.img  vmlinuz-2.6.12-1.1456_FC4
initrd-2.6.13-1.1526_FC4.img  vmlinuz-2.6.13-1.1526_FC4
initrd-2.6.13-1.1532_FC4.img  vmlinuz-2.6.13-1.1532_FC4
initrd-2.6.14-1.1637_FC4.img  vmlinuz-2.6.14-1.1637_FC4
initrd-2.6.14-1.1644_FC4.img  vmlinuz-2.6.14-1.1644_FC4
initrd-2.6.14-1.1656_FC4.img  vmlinuz-2.6.14-1.1656_FC4
initrd-2.6.15-1.1830_FC4.img  vmlinuz-2.6.15-1.1830_FC4
initrd-2.6.15-1.1831_FC4.img  vmlinuz-2.6.15-1.1831_FC4

And that is assuming I didn't miss a day and upgraded all along the way. That is just FC4: There are over 100 distros listed at Distrowatch. Any of them can pull in NFS clients and servers from various places in the development trees depending on what they are trying to do. And while there are probably always minor variations with a major release as bugs are found, I know that NFS in the 2.6 kernel has had a fair amount of work done on it. It seemed to us in fact that NFS in the early 2.6 kernel was regressed over where the 2.4 kernel was in terms of speed and stability.

Then there are 2.2 (not too many of those anymore) and 2.4 kernels and all those variants and point releases and possible updates. It would seem impossible to test it all, and it is. Don't even try. it is not worth it: You only need to know about your environment anyway.

Now stir into the mix all the UNIX flavors, and all the sub-levels of each UNIX flavor. Top with corner cases like MS Windows running Hummingbird Maestro, SMB proxies and gateways, and serve while still warm.

That it works as well as it does is testament to everyone, vendors (both UNIX and Linux) and distros alike adhering to a standard, open or otherwise.

NFS protocol stack

After you figure out how "bumpy" your client mix is, you needs to consider the permutations of the standard itself. NFS comes currently in 3 versions, each with 2 flavors:

1. RFC 1094: NFS Version 2
1. TCP transport (The original V2 was UDP only, but many clients and servers backported support for TCP transport from V3.)
2. UDP transport
2. RFC 1813: NFS Version 3
1. TCP transport
2. UDP Transport
3. RFC 3530: NFS Version 4 (Brand new, and not widely implemented yet, at least for us)
1. TCP transport
2. UDP transport

The Wikipedia article on NFS is pretty good, and contains much about the long and strange history of the protocol worth knowing. That history impacts how the protocol came about, was variously implemented, and why testing it for your particular client mix is something you just have to do. I'll talk about ours here, and we are famous for breaking NAS servers here, but you would still have to take into account your situation above and beyond whatever we do.

SMB / CIFS protocol stack

In a 1998 Andrew Tridgell, one of the originators of the SaMBa project and not apparently a fan of Microsoft protocols, said of the SMB protocol "The protocol is really horrible. It sort of grew like a wart. You can tell by looking at it". Kinda funny in that the article about CIFS over at the Wikipedia notes that SMB was originally an IBM protcol. In another article I once read but that I can not find, Tridge (as he is often called) said that the conversations between SMB client and server were often about with bugs each of them had. I have looked at a few network traces when trying to chase issues to ground, and for me the hardest part was that the protocol is only well "documented" in the Samba source code: there are no handy control block maps or principals of operations that I can find that would allow me to easily take apart the conversations and debug them: but then again, if it was easy, anyone could do it!

The Wikipedia article about CIFS looks pretty good, despite having a note at the top that it might not meet "Wikipedia quality standards". Anne would probably have a much better idea than I about how well "evolved" documentation like this really works. Seems OK to me though.

The point of all this is merely that you have to test your client mix against your SMB/CIFS server to be sure that there are no issues: there is no other way to know if any given client and any given patch level is going to have a behavior change that will need to be dealt with.

Bilingual file server: where the rubber meets the NAS

Just when you thought it was over and that it could get no more complicated, you end up with a requirement: you need both your MS Windows and your Linux or UNIX clients to all be able to see the same files on the file server. In our case, we have common source code trees that all the build systems must be able to see at build time. You have now entered the NAS zone: it is now Network Attached Storage, not an NFS server or a CIFS server.

Now you have to deal with the fact that SMB/CIFS are stateful, and NFS is not. Well... versions 2 and 3 are not: a quick read through of the first 20 pages of the RFC for version 4 makes me think that the implementations of V4 might have more state: That is something we have not had to test yet in our client mix. In any case, with NFS if the server dies and reboots, a cluster fails over to another node, the NFS client only knows that "badness occurred" (TM) if they have active I/O at the time. If they are doing something else, they never know the server changed state at all as long as it is *back* by they time they feel chatty.

SMB/CIFS clients will know something happened: the protocol cares what happens to the server, even if it does not matter to the client.  In practice this is not hugely horrible: The clients are used to just "clicking on the drive" to force the re-connect. But it has led to occasional confusion here: someone waiting around for the server to re-appear when it had long since re-appeared, and they had just not tried it again.

Locking is another interesting issue: For one thing, SMB/CIFS clients have something called Oplocks, which is really more like "Client side caching" than it is a lock, depending on which Oplock the client was granted. NFS clients don't go for any locks till they need them, but in a high write environment you can have some queueing while clients wait on locks. The NAS server will have to deal correctly with the protocol differences. In practice on a Linux file server with Samba, I have never seen an issue here.

Security model is another biggie: you will have to choose how you will map authorizations between NIS/NIS+ on the one side, and AD or whatever your MS Windows userid / security mechanism is on the other. You have to map one set of userids and their permissions onto the others. We map Active Directory into NIS here so that NIS really controls access, but that is because most of our clients are UNIX or Linux. if we were predominately an MS Windows client set we would most likely do it the other way around.

It really is only us

Over the next few articles as I talk about what we do to test all of these things, this really will be only 100% good for us. Our client mix and server set will not be yours. We have tried to adopt a best practices approach where that made sense, mining the bits from the various tools that we needed. We also only regression test for problems that we have specifically had here. We may or may not have seen and tested for that problem you once had. Your mileage may vary, see your shop for details.

Why Bother?

I would like to address one last thing today: Why bother with all this? Wouldn't it be easier to put in a file server running MS Windows for the CIFS client set (Or Linux only running Samba not NFS), and a Linux / UNIX one for the other client set? Yes and no. If I could run utterly separate data on the two, and I had no overlap, and never had to copy data between the two servers then I would go that way more than likely. But if I need the same data, then keeping it in two places means at the end of the day I really have two different sets of data. Real time synchronization just does not work that well in practice. It is fine for DR and other cases where the data is read-only or hangs out on the fringe, being overwritten from the central server from time to time without regard to local change, but if my two sets of clients need to see the most recent files, and if I need to serialize access to any common file, then I pretty much have to do it this way.

As complicated as this all has been made to sound, the every day reality of it is that projects like Linux NFS [2] and Samba have really done most of the hard work here. Everything works pretty well most of the time with most of the clients, Linux, UNIX, or MS Windows. Testing is usually about finding your corner cases and evaluating their criticality.