Another Crossroad

The "2.4 or 2.6" choice coda to my last post is, or course, another crossroads. It is not as technologically ubiquitous as the 32/64 bit transition, unless you live in a 100 percent Linux environment, and there are not many of those just yet. Almost all shops these days have Linux or some other open source project at work doing something for them of course. I would imagine 100% Linux shop to still be rare.

This last week has been all about testing assumptions and various bits of "received wisdom" we have from over the years. We wanted to get down to some real testing of a few things, and figure out where we really needed to be 2.4-versus-2.6-wise, on the Linux based

Our first data point was purely observational: of the 10 or so Linux based file servers we have built and deployed over the years, which ones have we had the least problems with? The was easy: the 2.4 based ones. Which the most problems? Also easy: the 2.6.5 based ones.

The observation (noted in my last post) was also made that most commercial NAS products use Linux with a 2.4 kernel of some level.

Now, for some testing. We looked at the published 3ware benchmarks for their 8000 series card, noted we should be able to do 125 MB / second in a single stream to a RAID5 card. We build a 2.4.31 based test system, and used IOZONE to test that. No problem: we exceeded that easily. We went to 4 write streams. And the bottom fell out.

Wow. None of our 2.6 based servers were this slow, even if you added using them via NFS or CIFSs protocols, rather than local writes. Some quick tests proved the 3ware card was not the issue: the same card did fine in a 2.6 test box. Suddenly, 2.6 wasn't looking so bad.

One of the team had an idea. We booted 2.4 with a uni-processor genned kernel. Suddenly, performance was where it needed to be again. Some googling around on the Internet with these clues in hand, and the general observation that every benchmark we had read always used XFS rather than ext3, plus a memory of reading that one of the big wins with the 2.6 kernel was a great deal of work around the subject of making ext3 perform better (on of my guys found that later) led us to what was wrong.

Looking at the PDF linked above, Ext3 in 2.4 was stable but performance challenged. Apparently to correctly serialize things in SMP environments, it just locks all sorts of things rather than having a much more granular file system lock design (way over simplification: the PDF has the details). It was easy to test from there: we re-formatted the ext3 data area as XFS, and re-launched the same IOZONE tests. 20 times faster. Whew.

We are setting up the connectathon tests from the clients to re-certify the NAS side of the system, but we think we have it now. It was not a "finger-check" so much as a bad assumption. Ext3 worked fine on all the other servers, and going back and looking, the 2.6 servers are OK with ext3 because of the re-design, and it just so happened that all the 2.4 systems with high I/O loads (where this would have shown up) were uni-processor.

We think we have a stable, moderately well performing solution: this whole exercise was to help us get out of the woods on the Tru64 Trucluster file server problem. This new server will operate as the short term fix for that while we now go off and look at similar hardware running 2.6.14. We still have those 2.6.5 servers to deal with, and we need to know if getting current on our 2.6 level will fix them, or if we need to run them back to 2.4.

Our soon-to-be-tested theory is that 2.6.14 has all the stability patches in it now that we need, and that because of the 2.6 kernel re-designs in other areas (NFS Server fixes, IP stack improvements, better SMP support, etc.) that it will be much much faster than the 2.4 system even when running the XFS file system. If our tests prove XFS is to data-stable, we could always use that on 2.6.14, same as 2.4.31. But if Ext3 performs on 2.6, we do like the core stability / safety issues it offers. Time and testing will be the judge.

We are still standing at the kernel crossroad. We want to get to 2.6: clearly that is the future. For 64 bit processors like our new Opteron chipped servers, or for a laptop, it's the present. Laptops have different usage profiles of course.  2.6 is a big win there already.

For the server it is hard to give up the known stability of 2.4. 

Portable Weblogger

I wrote the other day about trying to get my old HP 620LX to sync with Linux. I wanted to use it or my NEC MobilePro 770 (Gadgets.... now you know why I work in R&D support...) to take notes for this weblog. Silly me. I was overlooking the obvious, easy way to do it. My wife ordered an inexpensive Compact Flash card from eBay for me, and now I just keep all my notes there, and upload them to my iBook or Linux with the CF adapter that came with my Sony Ericsson cell phone, and I'm done. No sync problems at all. Duh.

OpenOffice and AJAX

Jonathan Schwartz put up an interesting weblog entry about OpenOffice and AJAX. His major point there was the the value of an application has no relation to the language it is written in. He also acknowledges the irony of that, coming from Sun. That is true as far as it goes, and David Berlind pointed out that you would not implement a "heavy" client as a web app.

All true, but for me, both of them miss a big point. Everyone got excited about the idea that there would be a web version of OpenOffice for a reason. Unfulfilled, latent demand for such a thing clearly exists.

You would not, of course, re-implement all of OpenOffice as a web app. You would start with basic features that everyone uses all the time. Google has most of it already in gmail: web forms that can spell check. Add in more font stuff, the ability to save it in OpenDoc, download and upload it from any OpenDoc compatible "heavy" clients anywhere on the web, and you have "Google Office Beta".

Following the Google development model, they will stay in Beta for years, slowing adding features over time. Spreadsheets would probably be along pretty quickly (and no, they would not do their calculations on the server side, any more then a web browser uses Apache cycles to render web pages). Web based presentations would be right in there of course.

The OpenOffice source code would probably be a manual: a way to not have to re-do work *where it made sense not to*. The presentation code would be all new I am guessing: It would have to load as fast as any other web page does. The possibilities for collaboration are mind boggling. Marry in some Wiki tech from (pick a project, like MediaWiki) and some standards based calendaring from (pick a project/product, like Mozilla or Zimbra or Chandler, or ...) and now you have something.

All this will take time. But with Google involved, if they wanted to do it, not that long. 

Codeweavers Crossover Office 5.0 out!

I just loaded up Crossover Office 5.0 from CodeWeavers on my Fedora Core 4 system. The menus (which did not work under CO 4.2 and KDE 3.4) work now! And boy howdy is it ever faster. MS Outlook 2000 just popped up like it was meant to be there. It is almost sick in a way. IE 5 popped up quick too. I did not get a chance to do much with it yet though. it did take forever to download. The website was very busy.

I see one of the new things is support for MS Office 2003: I am almost tempted to try that to see how the little "new message" fader that MS Outlook 2003 has works. On the other hand, MS Office 2000 already does what very little I need these days. No point I guess.

Evolution 2.4.1

I was reading through an email archive to the an Evolution mailing list yesterday, and I noticed people talking about Fedora Core 4 and Evolution 2.4.1. I had 2.2.3, and the general commentary made it sound like I was really missing out. I was partly interested because I have 2.4.1 on my OpenSUSE 10 system, and I was curious if it would work better on FC4.

Google provided info about a packager (nrpms) for this new level of Evo, since clearly none of the sources I had enabled in Yumex had the updates. I laid it down, and sure enough: it was much faster. It looked very clean. For about an hour.

Then Connector started crashing, and nothing I immediately tried would get it going again. I fired off a bug-buddy on it. I am thinking that now the only way I can calendar is with MS Web Outlook, which is workable but sub-optimal. But desperation plus going off and working on other things made an idea pop into my head: I went through and cleaned out all references to Evolution in the .gnome2_private and .gconf directories, and suddenly, it was working again. It stayed up all day. So I cleaned out the same places on SUSE 10... and now it is working. One day and counting.  

I accepted meetings and worked as normal, although I do have to admit I did schedule one meeting using MS Outlook 2000 under Codeweavers, just to see how it would work. It worked great.

OpenOffice 2.0 is now GA

OpenOffice.org 2.0 has shipped. And had it's recent 5th birthday to boot. And already uses OpenDoc, but we knew that already. I have not had a chance to put up the GA version yet, but if the Beta program is any indication, this should be pretty nice. I am downloading the OS.X/X11 version now. It is not GA just yet, and NeoOffice works pretty well for me on the iBook, but I have to see how this looks.