NFS, GFS, nodirplus / readdirplus, and Tru64 updates

Last week I posted about some things we were up to in the Network Attached Storage area, and this post was new in that it was not about what we had already done, but what we were actively doing. I don't usually let it get out there that I don't know everything, but this seemed like a good time to get the truth out :)

Part of the idea of posting this way was to get some information out there as soon as possible, and to get feedback from any others that might be in the same boat as us. This week I have a batch of updates to various things in that post. I will not spend as much time doing background as I usually do: You are either going to care about this and already know about it, or have no idea what I am talking about and go see if I posted something more interesting over at http://on-being-open.blogspot.com. I haven't.... For one thing I have been traveling all week, and have not had time.

This post would have been up sooner, but I was waiting to hear back on permission to post some things that were mailed to me, and also to see how an experiment turned out. The experiment is probably of more interest to the general Linux folks, so I'll cover that first:

NFS and GFS

In the aforementioned post, I pointed at a Bugzilla link... actually Dan Goetzman, Chief NAS Officer (CNO: Like CFO, only different) pointed at the link. Dan looked to see what the patch was about, and saw that it was "only" a deletion of three lines: the code was checking for:

GFS(s) expects NFS fh_type and fh_len would have the same value

that would apparently never be true, and that was causing a problem for NFS when it was accessing GFS-provided file systems. Here is Dan's news:

That is terrific news, because if this works, that means the GFS layer will prevent us from having to do all sorts of customer failover code for the NFS and CIFS servers in the cluster! Well, that is the theory anyway. Warning: Law of Unintended Consequences in full effect here!

Tru64, TruCluster, and READDIRPLUS with some NFS Clients.

The other problem we have been having has to do with Tru64 and some NFS clients that do not deal well with each other. The CentOS cluster above is meant to replace the Tru64 TruCluster as soon as we can qualify it, but in the meantime various clients that are being 100% protocol complaint are failing because Tru64 does not do the right thing with READDIRPLUS calls. Graham Allan, IT Manager in the School of Physics and Astronomy at University of Minnesota, Dan, and I have been emailing about this at some length over the last week.

The good news is that the problem has been fixed in the Tru64 community: Here is the news from Graham:

One of our students here also found a reference to the issue in the
Redhat bugzilla database, which alludes to some possible fixes at the
Tru64 end:

https://bugzilla.redhat.com/show_bug.cgi?id=220451

In particular:

	Comment #11 From Ahmon Dancy  	 on 2007-03-08 15:35 EST  	 	 

	(In reply to comment #8)
	> > Just curious... is there a way to turn off READDIRPLUS
	> > support on Tru64 servers?

	Yes!  The following worked for me.  There is probably some better way to
	do this
	so that it remains permanent across boots, but I haven't figured it out yet:

	ladebug -k
	assign doreaddirplus=0
	quit

	-----------------------------------------------------

	Comment #12 From Ahmon Dancy 	on 2007-03-08 19:47 EST 		 

	If anyone is interested, I have hacked up a new nfs_server.mod kernel
	module for
	Tru64 which fixes the bug.  I have it deployed on a local system and it
	works
	fine.   I built it against a host that identifies itself as Tru64 V5.1B
	(Rev:
	2650).  It can probably be adjusted for other versions as well.

	-----------------------------------------------------

	Comment #14 From Ole Holm Nielsen  	 on 2007-04-17 16:48 EST  	 	 
	Regarding Comment #11 I have some further advice:  You may not have
	"ladebug" 
	installed on your Tru64 server, but probably you have /usr/bin/dbx.  
	You may then modify the "doreaddirplus" variable until the next reboot by:

	# dbx -k /vmunix
	(dbx) assign doreaddirplus=0
	(dbx) quit

	If you want the change to be persistent across reboots do:

	# dbx -k /vmunix
	(dbx) patch doreaddirplus=0
	(dbx) quit

	If you upgrade the kernel this will have to be repeated.

	We have verified that this workaround solves the NFS problem
	with a Tru64 NFS server and a RHEL5 Linux NFS client. 

	-----------------------------------------------------

	Comment #15 From Steve Dickson  	 on 2007-05-15 11:00 EST  	 	 
	Fixed in nfs-utils-1.0.10-10.fc6 by added the -o nordirplus mount option
	which will have the kernel support in the next kernel update.

I (Graham: Ed) had an interesting reply as well from Ahmon Dancy who had 
mentioned his "hacked" nfs server kernel module in the bugzilla discussion. 
I will paste his reply for you:

   There are two approaches available now, and mine is the more
   complicated of the two.

   Approach #1:

   See post 11 on that bug, followed by post 14.  This disables
   readdirplus support altogether in the NFS module.  Then, when linux
   makes a readdirplus call, it will return a not-supported code, in
   which case linux will revert to standard readdir, which works.

   Approach #2 (mine):

   My approach was to spend a (bunch) of time disassembling the nfs
   kernel module and patching it w/ a fix so that readdirplus works
   properly.

Impressive bit of reverse engineering! I use Ahmon's name here because it is in a public forum on this issue already. Graham said I could mention him...

Even better, HP *appears* to be working on a formal patch for this, so Tru64 is still being supported inside the company!

That is all this time. I hope this information helps others fighting one of these two problems!