Adventures in Linux has moved to the BMC Developer Network web site.

The feedburner link has already been swung over so if you are subscribed to that you should already be seeing the new site. I hope you'll join me in the new digs as the adventure continues....

It sometimes seems to me that the cleverest thing Gartner ever did was to create the concept of the hype curve or Hype Cycle . I like the term Hype Curve better because a hype cycle sounds like a song by Queen.[I want to ride my hype cycle, I want to ride my hype cycle!... great. I'm going to have that playing in my head the rest of the day]. For Cloud Computing you can argue the time frames for speed of adoption, because not everything moves at the same pace, but Gartner firmly places "Cloud Computing"in the initial "Technology Trigger" section, climbing the curve for all it is worth.

Yes: that is right. It has not even achieved the tip-top height of the Hype yet. You are going to hear about it even more! Not even counting this post(though I prefer to not believe that this post will be hype).

Convergence

In my last post I talked about how we have been here before, and that post was largely meant to step back a bit from the day to day execution of technology and look at the concepts behind it.When looked at conceptually, the technical side of the Cloud is not all that different from the old glass house.

Another concept the Cloud reminds me of being the same kind as the term as"Web 2.0" was. In and of itself it is not a discrete technology, but rather a critical mass of various other technologies assembled and used in a particular way. Even that is not 100% accurate though, because cloud computing is not just one thing. Maybe "set of ways"?.

I rather like the idea, voiced in the comments of whurley's InfoWorld Cloud Computing Column that the idea for Cloud Computing came from someone seeing a network diagram (for perhaps the first time) and suddenly having a vision of what the concept of the network was.

At its very core Cloud Computing a very old concept, originated in the saying by Sun that "The Network Is The Computer." For Cloud Computing this is more true than ever.

Another Day, Another Cycle

If you think about a computing cloud, and what it actually is under the covers, there is a data center someplace (or set of places) with computers in it running transactions and servicing requests from the network. Where the data center is is masked by the network: as long as it is close enough to you that there are no significant speed of light delays, then you have no idea where it exactly is, or what it exactly is made from. Your transactions could be running on Linux or BSD or some other OS, on real hardware or as a virtual machine, and that Linux-or-other-OS could be on X86 or a mainframe. You do not know or care as an end user. You just want it to work when you access it.

What connects one to this data center is the network, and under that is the protocol of the Internet, TCP/IP. Where did that come from? Work done years and years ago by the Defense Department (DARPA), with a goal that the network should be resilient: survive war taking out parts of the physical wires by routing dynamically around the damage, with protocols that knew they were inherently unreliable so that they dealt with re-queuing and retransmission.

Perfect for a Cloud.

Many point to the commodity, rent a cycle nature as being a differentiator between the Cloud and other concepts, but wasn't that what "Utility Computing was about, at least in part?.

I can not help but think of the very first time I wrote a computer program.On a Telex machine, connected to a modem, accessing a shared IBM S/360installed someplace where that modem went. I know the mainframe was shared,because one of the other people that used that computer had access to an account from a different school and could see and print out their programs and data.

Our school rented time on that computer, and so did other schools. The mysterious computer on the other end of the network was chopped up into bits and rented by the TIP and the Kilobyte.  At the end of every school year, the student accounts were "re-provisioned", so that the next wave of students started with a clean slate. Yes, it was incredibly dated by today's standards,but it was conceptually not that different. A question of degree, not of concept.

Pendulum

We have been here before. The pendulum swings back and forth between centralized and decentralized computing, but here is where I think one of the major differences of the Cloud might be: It is potentially both centralized and decentralized.

One reason that people wanted to run screaming from the glass house was issues around span of control. The folks in the data center forgot who the customer was, and started acting like they were the reason that the computer was there. This was enhanced by things like the BOFH series of comedy bits, wherein a computer operator torments the end users for their personal pleasure. I will freely admit that this is not exclusively a problem of IT people forgetting who their customer is, as I have been on the receiving end over my many years, or customers who think IT's primary function is fall guy and punching bag. Let us just say that it is a troubled relationship that requires constant work from everyone.

Cloud computing gives people the option to not tolerate being treated poorly by their internal IT staff. Say "No" too many times to the end users request,and if they have a corporate card, they'll just end run you by buying the computing resources they need elsewhere. This is not that different than when the folks started buying PC's and departmental servers in order to be in control of their own destiny, and it points to the same possible set of problems. If people are sticking things in the cloud because they can, then at some point no one knows what is where or who owns it.

My first thought about data in the cloud is not that data is not secure,although that is certainly possible, but that data put in the cloud is unique,and that all it would take is the corporate card funding it being canceled to have the unique and possibly critical data disappear back into the disk cloud from whence it came, to be provisioned over the top of. Yesterdays critical data location is re-provisioned and is now occupied by today's shopping list.

Don't forget: Milk. Eggs. Backup data... Opps.

A truism in decentralized computing is that yesterdays experiment or escape from central support is today's crisis because something critical got lost someplace. Looked at another way: Just because it is in the cloud someplace does not mean that BSM and ITIL principals do not apply. If anything, it is the reverse.

Saving Money in the Cloud

It is very easy to see how a centralized large data center can be more cost efficient that anything that a small group of people / small business could build themselves. With scale comes leverage for discounts from the entire supply chain. By being able to choose the location the data center it can be located near inexpensive power. This is probably a bigger cost factor than most think about, but super-critical to keeping costs down. A data center next to a hydroelectric dam is both cheaper and greener than one near a coal fired power plant.

You can also make sure that there are qualified people nearby to support the data center. By using large commodity servers or even the mainframe,virtualization, provisioning, and so forth computers can be run at 80% plus utilization's rather than the more common 2% of the end user PC. That is not just good for the company providing the Cloud, but less expensive for the end user, and less impact on the planet as a whole. Triple win.

None of that concept is new, or unique to cloud computing. It is just adapting an old paradigm to the current generation of computer gear.

IBM has had, for years, the capability to provide variable capacity on demand in many of their computers. Stick a Mainframe one the data center floor,and enable (for example) 10 processors for the daily workload. Now quarter close hits, and sudden you need 20 CPU's. No problem. They are actually really in the mainframe, and can be enabled on the fly for a price. At the end of the quarter, disable them and you are back to 10 CPU's till you need them again.

The Cloud can do this as well. As long as every single customer of a particular Cloud provider does not go computer crazy all at once, the Cloud provider should have reactive capacity if it is well run... Here is the usual balancing act scaled up. Try to run at 80% capacity to make efficient use of resources, but have enough in reserve to deal with the workload bubbles. How big that "enough" is is based on careful measurement, modeling, historical data, etc. Golly: Sounds like the mainframe and Best/1 (OK: Called BPA now...) all over again.

Not Saving Money in the Cloud

Fair warning though: Most of the numbers I have seen about saving money in the Cloud quote a number around 30%. That may seem like a lot, but it would be easy to lose that savings quickly without Cloud purchasing discipline.

Think for a moment about the many many spam ads for free things like laptops. I looked into that a bit last night, and it turns out that it is a matter of terminology: Even when there was a real laptop or some other nifty thing to be had, it required going through and doing things like accepting offers that were not free: that had registrations, shipping, handling, you gave away personal information, and you spent a great deal of your personal time sorting through all of it trying to get to the not-so-free laptop/device for the least amount of money.

Now consider the distributed systems swing of the pendulum: It was always less expensive than the mainframe. Except when it was more expensive. It had lower up-front costs, but it also had hidden costs all over the place. Support contracts, and time tracking hundreds of computers rather than just a few. Virus outbreaks and the time spent dealing with those. Lost data because it was not on backups cost what exactly?

The Cloud has the same issues: If it is not approached as just another computing resource: If all one can see if zero up front costs, then how long till everyone with a corporate card is charging up some tasty Cloud? How long till that 30% is long gone because the right hand department and the left hand department are doing the same thing in two different clouds from two different vendors?

The Cloud will save you money right up until it doesn't, and we have been there before.

Rapid Provisioning

It is pretty easy to see how one key to success in the Cloud would be the ability to rapidly provision operating systems and applications.If you look over many of the offerings from many of the Cloud vendors that is exactly what they have. Rapid provisioning is a success factor for a Cloud offering, but it is not a defining feature of the Cloud.

Rapid provisioning is one of the holy grails of the in-house data center as well, and, full disclosure here, something we offer via our Server Configuration Automation née BladeLogic tool. If rapid provisioning is the key factor one is after with the cloud then it is possible with provisioning tools to create in-house computing clouds.

Taking the example from the previous section, with rapid provisioning there is a solution to the 80%/20% problem. Assuming that there are computers that are set up to create a progression to production, then there are probably Alpha, Beta, pre-GA, and maybe other levels or flavors of test / QA / Customer support resources either in or near the data center. Don't want to take a chance on your critical data leaving your premises and being manipulated in the Cloud? Create your own cloud inside your own enterprise. Take the idea of Cloud Computing, and implement it internally. In fact, take the whole attitude of the Cloud: the the customer is the person paying, and that the computing resource is a tool: a means to an end, not the end itself. <Craig Ferguson> I knoowwww! </Craig Ferguson>... weird to think about it that way to a computer geek sometimes.

API or not to API

It does not really take any special API's to create a computing cloud. TCP/IP and all the things that run on it are enough: See Google Docs, or any other AJAX based application for details. When I wrote this entry, I mostly used Google Docs to create the HTML, and for me, it was off in the Cloud. I don't know where the actual HTML is being stored: Which of many of Google's servers have it. For all I know, I am being shuffled back and forth between multiple computers, statelessly on my end, and statefully on their end. This is not to say that one can not create API's to enable the Cloud, like Microsoft's Azure. When Cloud enabling legacy applications API's can be downright handy it seems. Be that as it may, having or not having an API is not a defining feature of the cloud, and in fact the API's run over .... TCP/IP.

Cloud Defined

It appears to me that the definition of Cloud Computing is a very fuzzy,nebulous, cloud-like thing. At its simplest all that is required is two network enabled devices, and a network between them. The network is the cloud, just like in the classic diagrams. The network still, after all these years, is the computer. Probably why Sun is big into Cloud Computing.

The postings in this blog are my own and don't necessarily represent BMC's opinion or position.

I have been doing several things lately that all intersect. One of them is designing a new data center for R&D labs. Another is looking at one of our VMware server farms that was an early proof of concept for the whole server virtualization thing, and trying to figure out where it needs to go *next*. Another is looking at Cloud Computing: where it is now, and where it is headed.

At the center of almost all of those things is virtualization technology, and that is where I started my career on the mainframe back in 1980. VM SP it was called back then. At the center of the current X86 version of virtualization is another old friend of mine, Linux.

Quick terminology note: The mainframe OS called VM created VM's. Yep. They called the OS what it did. Kind of like people being named "Smith". VM over the years has had many flavors: VM/370, VM/SP, VM/XA, VM/ESA, and the current z/VM. VM system programmers just called it VM mostly, and knew by context if the discussion was about the OS, or the Guest Operating Systems. A Guest OS is a virtual machine, running as a guest of the host OS.. named VM. I'll try an be contextually clear here. VM created virtual mainframe hardware that the Guest OS used without knowing it was not real. Mostly.

VMWare, Xen, and others are slightly less confusing here because the Virtual Machine tag is reserved for the virtualized, or guest OS only. When you talk about the host OS, it has a different name. Don't even get me started on bare metal virtualization, or hypervisors. Not going there.

Hardware Assist

VM on the mainframe predates me by a good bit. Depending on how you interpret a few things computer virtualization started in either the late 1950's or mid 1960's. The exact start date is not really that important to this post. What is important is that the early experiments turned up the fact the hardware virtualization worked better when hardware assisted in its own virtualization. To keep memory from being bashed, trashed and generally abused there were features added to allow indexing and translation assistance.

While the implementations are technically different, they are not that conceptually different from what AMD did with Pacifica (AKA AMD-V) or Intel did with Vanderpool (AKA VT-x). It was really a very old lesson.

As the mainframe evolved, the technology that it used changed and evolved until we reach the place where almost all the virtualization formerly being done by the VM Operating system started being done in the hardware via the SIE (Start Interpretive Execution) microcode.

We have not quite reached that place in X86 land yet, but AMD and Intel keep adding more and more hardware features to support OS virtualization to their processors. The Nehalem generation of processors from Intel adds Extended Page Tables (EPT) for example. Wow... where I have I seen that before? Oh yeah... I remember. AMD is adding I/O Memory Management to increase virtual machine isolation. Think I have seen that before too....

Don't take my tone wrong: these are great ideas, and my point here only is that we have been here before, and so if you want to know where we are going, just have a look at today's Z10 mainframe because the hardware features it has now to assist in virtualization will appear sooner or later, in some form, in X86 space.

Shared Memory

Back in the day, when we were running a large number of virtual machines under VM on the mainframe (oh.. wait. We still do that...), the thing we needed first, before anything else, was RAM. That is as true if not more so today.

One of the things that was discovered early on about virtualization was that if you do shared memory wrong, all you get is a computer that beats itself to death trying to manage memory. Thrashing. Doing nothing but paging. I was not there, but think that is more than likely why some of the very first hardware assists for virtualization back in the early 1960's came in the form of memory management. I was here when Intel and AMD added hardware assists for memory, so I think my theory has legs.

In our use of VMware internally, we find that more often than not the number of virtual machines that we can deploy on any given ESX server is more a function of RAM, not CPU, being the bottleneck. Even with features like memory over-commit, it is common for us to see in our BMC Performance Assurance data a two to one ratio of memory usage to CPU usage. It would be easy to assume from that data point that the speed of the CPU has outstripped the other components of the general computer.

When CPU's were not the fastest thing in the box, IBM spent a great deal of time and engineering trying make sure that they only they only did work that was high value. I/O was spun off to dedicated I/O processors. Memory was used for the I/O processors to communicate the actual data the CPU needed to work on next. In general, the mission was 'keep the CPUs busy'. All this customer engineering made for expensive computers, and so no mainframe data center worth their salt would buy new capacity before they needed it, and an underused mainframe was considered a failure on the part of the people that designed and recommended that configuration.

Aside: When I was at NASA as a subcontractor, I heard a quite believable story about a mainframe system programmer who wrote a program that was a CPU soaker. After a new upgrade, he would make it use more CPU, and keep end user response time more or less the same. When load increased, they would dial back the soak task, and things would return to normal. When there was no soak left in the soak task, it was time for a new mainframe.

Whether that story is true or not, that fact it was told says something about the way that the resources of the mainframe were viewed.

The constraint for us with VMware is RAM. With 256 GB of RAM I can virtualize twice as many virtual machines as with 128 GB of RAM, without adding any CPU resources. But the cost the 8GB SIMMS to do that more than double the cost of the system many times... It was often literally less expensive to buy two ESX servers with 128GB of RAM than one ESX server with 256GB of RAM. Boy will that set of numbers look stupid in a few years... but the point will still be the same.

Factor in the three year ROI, and that is not true of course. Power, air conditioning, rack space, network and KVM connections all more or less double with two ESX Servers instead of one. The problem in tough economic times is to take three year ROI into account. The good news is that being green: using less of this planets resources, is also a good thing and makes the discussion about buying fewer, more expensive computers feel a lot like the ones we used to have back in the heydays of the mainframe.

For a project I am working on I did a three year ROI on two 256GB VMware servers (Dell R900's, but the same held true for Del R905's) versus four 128Gb ESX servers (Also Dell R900's). Not even counting the VMware licenses for the CPU sockets, the costs came back with the 128Gb config costing 25% *more* over the three years of the server. It is even better than that, because I rounded down all the power costs in my model, did not include taxes on the purchase or the power, and did not count the VMware per-socket CPU license charges. The real number is probably closer to 50% in the real world, but I wanted this estimate to be utterly fiscally low-ball. Under promise, over-deliver, just like Mr. Scott on Star Trek.

[Geek Points for the Star Trek reference!!!!]

I/O and Channels

The mainframe is still the king of the I/O mountain, and at the core of that is the way the the I/O subsystem on the hardware is designed. There are lots of I/O channels, they can load balance, and more than one can connect to any given I/O device for not just load balance but redundancy. It also further allows virtual machines to start I/O on any given channel to any given device in a totally shared and transparent but still isolated way. Now add that the Z10 can have 1024 I/O channels to a single mainframe. X86 is not anywhere near this yet. Not even close.

Not being near and not being close is not the same thing as not knowing where they need to head over time though. Have a look at the Virtensys web site for example. Clearly they have in mind the same thing that the mainframe did: Decouple the I/O from the processor, and share it. The picture doesn't have 1024 I/O channels in it: There is a reason why the mainframe costs more for starters. Of course you could argue that makes the MF the perfect convergence platform at the core of server consolidation... and you would be right.

Virtual Networking

VM on the mainframe has virtual network switches interconnecting virtual machines. VMware has the same thing in ESX. TCP/IP allows all sorts of possibilities for tunneling other protocols inside it. Stepping back a second, it seems like whoever has the least expensive, fattest pipe can become the transport de jour for all the other I/O in the shop. iSCSI, FCoE... you name it.

In a reverse of the way things normally happen, distributed system brought networking to the mainframe. Well... not exactly. The mainframe has it's own way of networking before (SNA) but it did not survive "contact with the enemy". Mainframe people just had trouble getting their heads around the idea that the protocol did not guarantee the packet would get where it was sent. But I digress.

The mainframe has virtual network I/O long long before it was cool. We used to lash Virtual Machines together into virtual networks with Virtual Channel to Channel (VCTC) adapters. There was real hardware that let one mainframe talk to another over its high speed channels (high speed then...). It was a complicated sort of flipping transmit and receive, except that the mainframe channels were parallel, not serial. VM could do that same trick virtually, and then to guest OS's could converse not at hardware speed (4.5 Megabytes a second on a fully spiffed S/370 set of cables), but at *memory* speeds.

If you buy a large VMware server, it can have inside it a virtual network where a large number of its virtual machines converse with each other without ever touching a real wire. We have been here before, and it was good then too.

Virtual Disks

UNIX and other platforms have long had the concept of a virtual disk. The hardware design made this vary, but at the very base of this was the disk slice. Take a disk, and instead of using the whole thing as one disk address, write a partition table on it, and have it contain more than one disk image. PATA had, for example, four primary partitions available by design. In Linux terms, HDA became HDA1, HDA2, HDA3, and HDA4. When that was not enough, PATA layered on the "extended" partition, so that one of the four disk slices could be sub-divided into slices again.

VM (the MF OS) has always had the idea of a "mini-disk". Unlike a partition table, it was not written to the disk being sliced how the disk was divided up. The disk was blissfully ignorant and just stored the data written to it where it was told to by VM. The disk slicing was defined in the VM directory. MF disks mostly came in the Count Key Data flavor (with a few exceptions) and that meant that they were subdivided into "Cylinders". Different disk models have different numbers of cylinders. The smallest minidisk is one cylinder, meaning that a MF disk could contain literally thousands of mini-disks. The limit was how many cylinders the hardware presented to the OS.

VM would then take these minidisks and assign them to VM's or Guests. The Guest OS would have no idea that the minidisk was not a real, but smaller version of, a real disk.

In another echo of the past, I have seen various Linux recommendations being made to use disk labels rather than device addresses. In Linux, this is mostly in /etc/fstab. Mainframe has use disk labels forever. The Directory that defines the minidisks is keyed of the disk labels in fact. The reason is the same too: with labels, the underlying disk hardware address can change and it will not affect the operations of the system. Handy for system recovery and such.

Disk slice limitations were a real problem for UNIX and other OS's. Logical Volume managers sprang up to deal with that by abstracting the real underlying hardware from the applications on the OS. LUN's became VLUN's. A VLUN could be part of a disk, a disk, or many disks. In this the mainframe was exceeded for a while: to aggregate minidisks would be a long time coming. The first time I thought is was *easy* was in fact the convergence of Linux and VM, where Linux creates VLUN's over the top of VM supplied minidisks.

Driving Utilization Up and System Count Down

As noted before, when I started in the mainframe biz, it was just generally accepted that you did not run your MF at less than 80%. If you did, you had overbought your capacity. At the same time, there had to be headroom for peaks: things like quarter close, and billing runs and stuff like that which made your averages and your peaks both things your capacity planner / system programmer took into account when figuring out what kind of mainframe they needed to buy next time. Not counting the CPU soaker person. That story ends with them being fired.

One of the reasons that people loved distributed systems is that they could buy all these little computers with tons of spare capacity and just not worry about that anymore. Of course we now understand that the freedom came with a cost: OS license counts, applications license counts, and amber waving fields of systems that needed to be replaced every three years or so. This contrasted with the more structured world of the glass house where those things were planned for and dealt with by small staffs of people that understood all the issues around the troubles that come from things like hardware and OS upgrades.

There may not have been capacity planners in the distributed world, but now everyone was a desktop admin.

All those computers sitting about and doing nothing when someone was not sitting in front of them. Even when they are sitting in front of them, I am watching the CPU meter as I type this. The computer is not even noticing the keystrokes. I have two browsers open, email, and a couple other applications and the CPU is looking out the window and is frankly rather bored. Memory is at 60% though. When I need the CPU, I will spike it to 100% for a few seconds, but then it will return to its lackadaisical state. The average computer runs at about 2% CPU usage most of the time. With enough memory, this CPU should be able to handle 40 or 50 people doing the same things I am right now.

The mainframe stayed as busy as possible by buying fewer CPU's, offloading its I/O, and leveraging RAM and disk storage (VM's paging / swapping subsystem is a thing of beauty).

For the end user, the so-called "green screen" sat connected to a terminal controller, and the terminal controller buffered together the I/O of about 32 users, and batched it up to the mainframe when it needed to. These days you can do more or less the same thing with a web browser, AJAX, and a Linux server. That Linux server can be running with a bunch of other Linux servers at the same time on an ESX, Xen, or other virtualized server inside the data center. Work is offloaded onto my computer, and batched back to the server as needed. As I am writing this in Google Docs, the server is off someplace in a server cloud: I know not where.

Feet on the Ground

Cloud Computing. Seems like another name for "Central Data Center" in so many ways. What are the concepts that drive the cloud?

First off, while people sometime act like the idea that they are in one place, but their data is in another is new. They get jazzed about the idea that a modern computer screen looks so much better and is so much higher resolution and has all the nifty colors and nice icons and all. All that is true of course: We use tons of computer cycles and RAM to make all the pretty screens. We'll be using a lot more whenever we get to the place we can talk to the computers.

I was talking to a banker the other day, and he was talking about how they are supposed to open new accounts. There is this pretty GUI based thing and using it requires patience, especially when it crashes. All he wants to do is open an account, and the person is waiting right there! When the GUI widget crashes, he goes to a green screen hidden away somewhere in the back ... I am guessing an IBM terminal, but they did not know. May have been ASCII. There he flies through a series of screens, typing in codes and data, and in a minute or so, all is done.

That terminal accesses a computer someplace else. He does not know where. It is "out there". Sure, it is an internal to the bank cloud, but it is still mysterious and magical. Of course it helps he knows how to use the green screen. A new comer to the bank would more than likely not know the old system, and will have to be patient waiting for the new system to either catch up, or maybe even come back up.

Is the magic of the cloud the protocols? Does one really care where the computer is? Of course not. What they care about is whether or not someone can steal their data, their identity, or embarrass them.

I am not saying there is nothing new under the sun... or is that Sun here? The fact that there are over a billion wireless phones on the planet, and having a full browser in the wireless palm (or is that iPhone) of your hand is clearly new. It is nothing that science fiction did not think about for years, but now it is real. All those itty bitty computers in our hands would be useless without the cloud of computers behind them. On my iPhone, I have no idea where the computer is that makes Google Maps work, but I am fairly sure that it is not the same one as the computer that makes my Twitter app work. the protocols and the interfaces, and the speeds and feeds have all changed, but conceptually, how different is that then the green screen on the desk that somehow accesses the customer data and sets up the account? If the bank has web banking, that customer can now go and access the exact same information from their iPhone.

I'll have more to say about the clouds in the future, but as it related to this post I think I have beaten it enough. Probably not into submission. the cloud crowd are pretty loud and proud.

[More geek points for alliteration!]

Looking Forward

One of the areas I am looking into right now is storage virtualization. Making disk farms into blocks of storage, and abstracting the volumes at a layer above that. Spread I/O as far and wide as is required for the application at hand.

Even at that, I have been here before. The mainframe had a bit of disk storage back in the 90's called the Iceberg, from then-STK. The 'Berg was a real disruptive technology. It took Fixed Block, SCSI disks and virtualized them into Count Key Data volumes. The mainframe no longer knew where the actual blocks were anymore. IBM later came up with the RVA, then the Shark, and now the DS line, and all of this virtualization has continued and increased. There is, as far as I am aware, no such thing as a *real* Count Key Data disk anymore.

The IBM SVA , Xsigo Systems VP*, or HP SVSP do the same thing for disk arrays from a wide range of disk vendors, or, if you prefer, there are devices like the ones from Compellent (to name but one) where the disk back end is provided, but utterly virtualized.

Deja Vu. I feel like I have been here before. Not exactly. There are differences. Still, I wonder what is going to get mined from the mainframe next. It is not like the Mainframe is sitting still waiting to be overtaken either. Like TCP/IP, it has absorbed some ideas, concepts, and even operating systems like Linux. UTS would be proud

In my last post, I made a reference to queuing up and testing the new MAPI connector for Evolution under OpenSUSE 11.1. I have done that now.

First off, I added the repository to YAST using the information at the Evolution Wiki. The Repo file looks like this:

After adding the repo to YAST, updating the repositories, and then installing the MAPI bits, my system now had this on it:

The MAPI code was updated at the end of January, going from OpenChange .7 to .8, with all the related packages similarly revving.

MAPI-Clause is Coming to Town

The fact that MAPI has been added to Evolution so quickly is really very impressive. It is not an easy thing to do. Having the doc (as noted last time) certainly helps. That being said, this is called .8 for a reason. It is working against my MS Exchange 2007 server, but not without significant issues.

I want to stress right here and now that I am not testing this for prime time readiness, nor does the project make any claims that this is anything but Alpha level code. Use this at your own risk!! Really!!

I lost email with this. Really really. This is pre-GA.

I had tested .5, .6, and .7 of this MAPI code, and it never worked for me at all. I was testing it against MS Exchange 2003 though, so two factors changed: I moved to MS Exchange 2007, and Evolution MAPI .8 at the same time. Now it works. Sort of.

Here is what works:

• I can send* and receive email.

• I can see part of my calendar. I have not figured out the rhyme or reason to why Evolution can only see a small subset of my total calendar. Using today as an example, I can see one of seven meetings.

• Expunge really does Expunge. Everything. Entire Inbox, whether or not a note has actually been deleted. It does ask if you really want to do it first though.
• GAL: The Global Address List is still a work in progress. MAPI dropped the WebDAV / LDAP version of the code in favor of a protocol called NSPI (Name Service Provider Interface). It is not done yet, and not included in the current binaries.
• No setting any rules / filters
• No Exchange special stuff, like "Out of Office" autoreplies. Not even a screen to work with them yet.

Stability is not there either. I can walk away from the computer to go to lunch, and come back to find I have to restart all of the Evolution processes because it has hung. No messages. No tracebacks.

In related news, when I updated my Ubuntu 9.04 Alpha system this morning, it pulled down Evolution 2.25.90, so it looks like Evolution 2.26 is getting close. There is no MAPI available yet in the repository though, so it is still a race to see what goes first, 2.26, or MAPI support for 2.26.

I am encouraged. A version of this will ship with Gnome 2.26, and I assume that it will be revved very frequently after that for a while, but still: after all these years, MAPI / RFC support for MS Exchange is clearly headed to Linux. When it gets there, a *huge* wall to corporate Linux desktops will have fallen.

I have, for years, with varying degrees of stability, been able to access my Exchange based Calendar and email from Linux via Novell (really, Ximian's) Evolution product. I have written about all that at length here.

No more. Welcome MS Exchange 2007. Goodbye WebDAV. Microsoft's grand experiment in email open standards is over, and where Exchange 2000 and Exchange 2003 were accessible via the WebDAV protocol, Exchange 2007 drops this.

I do not know why. It was not because it did not work.

WebDAV was part of how MS created Web access to the MS Exchange inbox, Contacts, and Calendars. E2007 replaces that with a heavy and light client. The heavy client only works with IE, and is all ActiveX stuff as near as I can tell. The 'light' client appears to be mostly an HTML effort, and works with Safari and Firefox, among others. The light client is noticeably faster than the old light client was, and is cleaner and brighter to look at. It reminds me more than anything else of the Yahoo webmail interface.

It is serviceable, and will have to do for now, because with WebDAV removed, all I can access from Evolution is the Inbox via IMAP. That is not insignificant either: IMAP is faster than the old MS Exchange connector was: Clearly a lighter protocol. I also have Win7 and Outlook 2007 if I need it.

It Could be Worse: MAPI / RPC *is* coming to Linux. Slowly.

MS kept their access protocols carefully undocumented, non-Open-Standard, and in fact kind of catch as catch can. Need a new feature? MAPI protocol was not envisioned for that? No big deal: Add a Remote Procedure call. In addition to the MAPI protocol itself, when Outlook and MS Exchange talk, there are apparently 150 or so RPCs involved.

There is nothing about any of this that would keep any host platform that can talk TCP/IP from talking to MS Exchange. Neither MAPI nor RPC's are the exclusive realm of MS Windows. What has kept it exclusive has been lack of documentation. If you wanted to implement an email client with Calendaring, Contacts, and Tasks that talked to MS Exchange the same way Outlook does you had to grab the wire conversations and figure out how they worked. What they were doing.

This can be done. It is tedious and time consuming, but the Samba project figured out SMB this way. It can be done. What WebDAV did for projects like KDE's Kontact and Gnomes Evolution is make it far easier to figure out things. The wire protocol was WebDAV. They could see the mailstore, the Contacts, and other objects on the Exchange server via WebDAV. They still had to figure out the interactions, but by being readable, it was far easier than trying to start at zero like one would have with MAPI and undocumented RPCs (And we are talking about the undocumented MAPI here, not the documented SMAPI from years ago)

Even as relatively easy as it might have been, Evolution was never all that stable (At least when using the Exchange Connector, and some point releases were better than others and depended also on the Distro in odd ways that I have documented here in the past), and KDE never called their MS Exchange / WebDAV effort anything but experimental, and my experience of it was that while you could read your calendar, you could never add events to it with Kontact.

The EU has changed all this. MS has been told that if they want to do business there they have to document things like MAPI and the RPC's they have kept so under wraps for all this time. They have. In fact, MS also worked with Novell to get Silverlight going on Linux (the so-called 'Moonlight' project) so people could watch the Obama Inauguration on the Internet with Linux.

Now both KDE and Gnome are working with OpenChange to get support for MS Exchange into their projects. The first MAPI / RPC support is set for Evolution 2.26, due in March with the rest of Gnome 2.26. It will apparently implement a subset of the RPC's required to get started at a basic level with MS Exchange server access. Some 80 or so of the 150 RPC's MS has documented. In support of this, OpenChange just release a new library of fixes and new feature function on January 20th, 2009.

I have an OpenSUSE 11.1 / Gnome 2.24 based system set up and ready to test the new libraries as soon as I get a spare moment from my regular day job. That link also has repos for Fedora 9, 10, and OpenSUSE 11.0. I am also tracking Ubuntu 9.04 since it should ship with Gnome 2.26.

KDE is farther behind on this that Gnome, but they never really had WebDAV working as well either. This article documents the KDE's current status. In related news, after the setback that was the KDE 4.0 release, it looks like KDE is starting to get their Mojo back in general. KDE 4.2 is supposed to be much better, and by the time the MAPI / RPC support is added they should be well on their way to being a fully viable desktop again. Not that they stopped being one, as long as you stayed in the 3.x tree. But 4.x should be back to having all the feature/function of the 3.x tree, with the new underlying architectural improvements in place. It was painful, but it looks like the environment is nearly back. Just in time for Gnome to have a spasm of architecture changes no doubt.

Aside: I have no problem really with what KDE did when they moved to the new 4.x series. I get that they had to make some underlying changes to position themselves for the future. I just think that 4.0 and 4.1 were still Beta's. I have not yet tried 4.2 to see what it looks like: I will as soon as I have a chance. If nothing else, I will be tracking how KDE adds in the MAPI / RPC functionality. I like having options. It is probably telling that KDE centric distros like PCLinuxOS have chosen to stay with the 3.x tree so far. The exact quote, in reference to their upcoming PCLinux 2009 release:

"We decided to use kde3-5-10 as our default desktop as the we could not achieve a similar functionality from kde4. We will however offer KDE4 as an alternative desktop environment available from the repo once we stabilize it."

Waiting Is....

Geek points! I got in a "Stranger in a Strange Land" reference! In this case, it is not martian patience, it is just that there is not choice. MAPI support is coming soon, but it is not here yet, and it is getting here far faster than it might otherwise have, since the various projects have access to the actual protocols this time around. It still will take some time. I fully expect that Evolution 2.26.0 will be followed by a series of point releases while all the bugs get worked out on this brand new feature set.

The funny thing about all this is that it probably still is only a short term thing before all the angst about these protocols fades from relevance. Cloud Computing, Google Gears,, SaaS, Linux based Netbooks, and all the current technology has us heading away these paradigms can not help but have an impact here.

http://vmlmat.wiki.sourceforge.net

Background, in case this is the first post you have ever read about VMLMAT. VMLMAT is not a product of BMC's R&D organization. As we are organized internally, R&D Support is actually in the IT department, and VMLMAT was designed and built by Ron Michael, the man charged internally to support all of R&D Linux images on the mainframe. It expressly is written for the needs of our internal environment, and some of those parameters are:

• All Mainframe Linux images run under z/VM. While we have LPAR's, we do not use them for Linux.

• We made an arbitrary design decision to use Active Directory as our repository for users and passwords. VMLMAT stores the metadata about what each user can do, and which Linux images they own, but not the actual userids / passwords themselves

• Our R&D community was needing to treat MainFrame Linux as "Just Another Linux": the same as the Linux on the X86 server, the one in VMware, or the one on their desktop or laptop. As developers, they know Linux, but they may not know much or even anything about z/VM or the mainframe.

• Related to that: We had no idea what platform the R&D people would be using when they were managing their Linux images. Could be AIX, Solaris, Linux, HP-UX, OS.X, Tru64, VMS, MS Windows from 2000 thru 2008...

• With terrific z/VM system programmers in the support organization, we were not looking to do anything that either got in their way, or add any system z/VM management features.

• We have no MF attached SAN devices, therefore no MF Linux running on FCP devices.

Because of Whurley and the new direction he represented for BMC, and discussions with same, we knew from a fairly early point that VMLMAT might become one of BMC's first Open Source projects. However, that did not translate into making VMLMAT the be-all, end-all of Linux on the mainframe management tools. We needed VMLMAT to do certain things internally, and that was all we could justify from a time / cost point of view of Ron's efforts. All of VMLMAT's features are funded by IT, therefore they had to meet IT's needs, and in this case, that meant R&D's needs. The good news is that Ron knew this might go Open Source, and so he was very judicious about the codes documentation both inline and via the Wiki. This serves us well internally for anyone that might come along behind Ron, and it serves the Open Source community as well, since they were going to get clean, well documented code from the start.

Being well documented was key: We knew that there were missing features, and we knew we might not be the ones adding them. We made the leap of faith that any submissions back to the project from the community at large that were based off the codebase stood a better chance of being equally well documented and written. We will not accept anything into the main code tree that is not so (or that we can easily make so).

Taking my list above in order then, here are the implications about what VMLMAT could be made to do in future:

• VMLMAT can be thought of, at least in part, as a cold provisioning tool. It takes Virtual Machines and makes them into any version of Linux in the archive. It can also be used to store particular configurations so that they can be promoted from Alpha to Beta to pre-GA / release candidate, then to to production. But if Production is in an LPAR... not so much. Not yet anyway. We think adding LPAR support is very do-able though.

  • Related to that: VMLMAT is a stand alone tool right now, but what if you already have a provisioning tool in house? Something like BladeLogic? Could VMLMAT be made to interoperate? Again, we think the answer is yes.

• David Boyes, over on the VM Linux mailing list made a great suggestion, that VMLMAT use PAM rather than our Samba code, so that authentication could be made pluggable, and therefore it would not matter what the target shop has for userid / password authentication. LDAP, NIS, NIS+, AD... whatever.

• In a production environment, not everyone and their cat should have full access to the production versions of Linux. VMLMAT already has a pretty good security model for this, since it allows one to many, many to one, and many to many groupings of users to Linux VM's. But I can see how some shops might want to tie this is to something like RACF or VM:Secure. For our part, we would just like to store all this is something like PostGres or MySQL in the future, rather than in locked down flat files.

• Our choice about the user interface was Web 2.0ish (even though it is pure HTML right now). We assumed everyone would have a web browser, even if we did not know what the web browser might be. Other shops may prefer a Curses or an X interface. We have no internal demand for that, and I can not see us adding those other interfaces right now. What I can see us doing is getting far more sophisticated over time with the web interface. It is currently designed to be about as fast and simple as it can be: Real Web 2.0 technology Open Standards (such as AJAX) could make it much more active in nature. Whatever is done here, Open Standards will be at the core though. If we learned nothing from the web browser wars of the 1990's it is that using proprietary browser extensions is a "very bad idea" (tm).

• There is an API in z/VM called System Management API, or SMAPI, and not to be confused with Microsoft's Simple Messaging API, also called SMAPI. Other than name, and being an API, not even similar. Via VM's SMAPI, VMLMAT could be extended to "talk" to the host OS, and have it do things like create and destroy VM's, add mini-disks, change memory parameters, add Diag permissions... whatever is needed. While VMLMAT has steered pretty clear of most of the features of BMC's original cloning tool, such features would put it squarely back in that realm. We don't need such things, so they would more than likely not be added unless something extraordinary happened, or they were added by the community at large.

  • Such features could also be added by having VMLMAT make calls to whatever the shops systems management tool is. This is the way the the BMC Cloning tool went about it. VMLMAT shares no code with that tool though, so adding such a feature set would more than likely be a community endeavor.

• A key design point of VMLMAT was device and file system independence. The way that the systems are archived and restored does not care about what the underlying device is. It does not care if it is EXT2, EXT3, or (probably, but not tested) XFS, JFS, or Reiser. We take whatever device VM gives us, and we are glad to have it. Thus should it ever be. While we have never tested it, adding SAN devices should not be that hard: Mostly a matter of dealing with the device names themselves.

  • Related to that: We built the archive feature to match a resource that we had: that being our spiffy Linux NAS servers. Other shops might prefer to archive to FCP devices, CIFS, or even to MF DASD, leveraging at least the fact that the archives are gzipped and still saving space. All of this should be easy to add.

  • Further related to that: We internally use only VM presented Mini-disks as "raw" devices for Linux. No LVM or EVMS layer. Since VMLMAT is device independent, adding support for a Logical Volume Manager(s) should not be that difficult. In fact, installing Redhat without LVM has gotten more and more difficult, so this is something we may have to add for internal reasons.

Those are just some of the obvious things we have talked about (and we have talked about other, more esoteric things still), and to some degree they are based purely on our projections of what others needs might be. I freely admit that I have no idea all the possible things others might need or want from a tool such as VMLMAT. We tend to have a very BSM mindset internally for some reason, so that may color what we think about when we consider what we might do to VMLMAT.

In its current form, it has more than paid us back for the time Ron spent creating it. How it might be extended to be of similar help to others is an "Open" question.

I recently wrote up a post on my personal blog about installing Mint 6 RC1 on my Acer Aspire One. This is a followup to that one, with the focus shifted from personal to professional use.

Evolution

I noted in a previous post that I had very good success with Ubuntu 8.10 and Evolution 2.24. Since Mint 6 is based off Ubuntu 8.10, I would expect that the results would be similar. There is room for doubt though, because as I noted in my personal blog, Mint 6 does act differently about a few things than Ubuntu 8.10. For sanity, I did a comparison between the packages I have installed on the Ubuntu 8.10 system and the new Mint 6 system. Here is Ubuntu 8.10:

ii evolution                                   2.24.1-0ubuntu2
ii evolution-common                            2.24.1-0ubuntu2
ii evolution-data-server                       2.24.1-0ubuntu1
ii evolution-data-server-common                2.24.1-0ubuntu1
ii evolution-dbg                               2.24.1-0ubuntu2
ii evolution-exchange                          2.24.1-0ubuntu1
ii evolution-exchange-dbg                      2.24.1-0ubuntu1
ii evolution-plugins                           2.24.1-0ubuntu2
ii evolution-rss                               0.1.0-1ubuntu2
ii evolution-webcal                            2.24.0-0ubuntu1
rc libcamel1.2-13                              2.24.0-0ubuntu3
ii libcamel1.2-14                              2.24.1-0ubuntu1
ii libebackend1.2-0                            2.24.1-0ubuntu1
ii libebook1.2-9                               2.24.1-0ubuntu1
ii libecal1.2-7                                2.24.1-0ubuntu1
ii libedata-book1.2-2                          2.24.1-0ubuntu1
ii libedata-cal1.2-6                           2.24.1-0ubuntu1
ii libedataserver1.2-11                        2.24.1-0ubuntu1
ii libedataserverui1.2-8                       2.24.1-0ubuntu1
ii mail-notification-evolution                 5.4.dfsg.1-1build1
ii nautilus-sendto                             1.1.0-0ubuntu1

Here are the ones from Mint 6 RC1

ii  evolution                                 2.24.1-0ubuntu2
ii  evolution-common                          2.24.1-0ubuntu2
ii  evolution-data-server                     2.24.1-0ubuntu1
ii  evolution-data-server-common              2.24.1-0ubuntu1
ii  evolution-data-server-dbg                 2.24.1-0ubuntu1
ii  evolution-dbg                             2.24.1-0ubuntu2
ii  evolution-exchange                        2.24.1-0ubuntu1
ii  evolution-plugins                         2.24.1-0ubuntu2
ii  evolution-plugins-experimental            2.24.1-0ubuntu2
ii  evolution-webcal                          2.24.0-0ubuntu1
ii  libcamel1.2-14                            2.24.1-0ubuntu1
ii  libebackend1.2-0                          2.24.1-0ubuntu1
ii  libebook1.2-9                             2.24.1-0ubuntu1
ii  libecal1.2-7                              2.24.1-0ubuntu1
ii  libedata-book1.2-2                        2.24.1-0ubuntu1
ii  libedata-cal1.2-6                         2.24.1-0ubuntu1
ii  libedataserver1.2-11                      2.24.1-0ubuntu1
ii  libedataserverui1.2-8                     2.24.1-0ubuntu1
ii  libevolution3.0-cil                       0.17.5-0ubuntu1
ii  mail-notification-evolution               5.4.dfsg.1-1build1
ii  nautilus-sendto                           1.1.0-0ubuntu1
ii  openoffice.org-evolution                  1:2.4.1-11ubuntu2

Not that this makes a difference, but Ubuntu is installed on a Dell 745 desktop, and Mint 6 is on a Dell D620 laptop. Evolution is not an application that should care about such things though. The Mint and Ubuntu packages match in all their core parts: Mint does not change anything from Ubuntu so I expected that Mint will work just as well as Ubuntu in the office.

Mint does change one thing about Evolution, and that is that they do not install it by default. Thunderbird is the email client of choice for Mint. Hard to argue with, except I need Evolution and the exchange connector. Ubuntu 8.10 installs Evo, but not the "evolution-exchange" package. Either way, I have to tweak out the install with Synaptic or apt-get in order to have my MS Exchange 2003 resources available on my Linux desktop.

Evolution works exactly the same in both places. It has the same problems too, such as having trouble figuring out what the mail folder index should look like if I do a mass delete in one place. The other instance of Evolution often never sees the delete correctly, and loses track of what is in the INBOX folder. I wrote about this back in February, and nothing has changed. It is very annoying but not life threatening. I just delete the mail folder index, and everything re-syncs from MS Exchange. It would be nice if there was a resync button, or even better if it would detect that it lost sync and do it itself. Probably all of this is moot though, since focus appears to be on MAPI Exchange server access for 2.26 of Evolution.

I should note that in the comments section of my post about Ubuntu 8.10 there is a comment titled "Non-crashing evolution? I don't believe it"

OpenOffice.org

Both Ubuntu 8.10 and Mint 6 RC1 ship OOo 2.4.1 with the the addition that they have the ability to read and write to Office 2007 formatted documents. This is because they reached ahead and grabbed the Go-oo patch set, so 2.4.1 from Ubuntu 8.10 and Mint 6 has one of the big new features of 3.0 included. I have not seen many office 2007 documents yet, but I am glad I can already deal with them

I was disappointed enough about 3.0 on Ubuntu that I went ahead and added a repository and added it. I did not do this on Mint though. 2.4.1 is more stable nad I am thinking about backing 3.0 off Ubuntu. The whole reason why they did not put 3.0 on Ubuntu is here:

http://brainstorm.ubuntu.com/idea/14433/

Mint 6 appears to have followed the same path that Ubuntu chose, and stayed away from OOo 3 for now, even though they shipped enough after both the Ubuntu 8.10 and the OOo 3 releases that they could have included it if they had thought it wise.

Active Content

I have never really talked about things like flash and media player being things that an office desktop should or has to do. I'd be willing to bet that there are many IT departments that keep such things very locked down. On the other hand, in the Web 2.0, active content world we live in, being able to access active content or watch short movies (say, internal training programs or the like) is probably required. This was always one of the reasons I liked Mint so well. It made content a no-brainer. Flash was already installed. Many of the non-free non-Open Source stuff that so many Linux distros (like Fedora) steer clear of like the plague are installed and ready to go.

Turns out Ubuntu has made real strides there as well. As a test (and I hope the IT guys don't swoop in on me) I played the new Star Trek Trailer on both the Ubuntu and Mint machines. it worked on both, but it loaded faster on Mint. This is cool, because the ST trailer is in Quicktime format. I did not do anything special. It just worked.

Hardware Support

Ubuntu 8.10 works extremely well on the Dell 745 desktop, and Mint 6 works extremely well on the Dell D620 laptop. Each has their own challenges. The Dell desktop has an Nvidia graphics card and two monitors. the laptop is... well... a laptop. Wireless works out of the box and is the Intel Corporation PRO/Wireless 3945ABG. Intel and Atheros are my two favorite wireless vendors, because their stuff usually just works under Linux.

Both systems enabled Compiz by default and it works in both places without issue, even though the laptop has the relatively speaking graphics-challenged Intel Corporation Mobile 945GM/GMS. I say it is graphics challenged, but Compiz works without any issues at all, so I guess it is good enough!

Volume up/down buttons on the laptop are enabled by default, and that is always very nice to see. Those special laptop buttons are often orphaned.

Mint 6, even in its RC version, appears to just work at work.

http://vmlmat.wiki.sourceforge.net

I am going to go out on a limb and guess that if anyone in your shop is using Linux, they started using it on a PC of some kind. Probably an X86 box like a laptop or desktop. If Linux was studied in school for something like a compiler of OS architecture course, it was probably taught on small, desktop class gear. Point being, they are used to having control of their system.

In a small development group there is probably a re-purposed PC or small rack mounted server or a VMware Virtual Machine, and everyone accesses the Linux server via X or SSH from the Linux laptop or desktop in their office. Someone wants to reboot, they send out an email, IM, twitter, or just shout up and down the hall that the server is going down.

The traditional MF shop is the opposite of this. It only makes sense: the MF is too expensive to not have it be centrally managed. I often think that ITIL was developed by people that had MF background because a large number of the disciplines it discussed are things we had to do in the MF world long before they had formal names. Change Control for example: You make a change to the MF, you have the potential to blow up a whole bunch of people or a whole lot of batch work, all at one time. This tends to make people grumpy.

As one who has lived in both worlds, I have seen the culture shock of someone coming from the small Linux platform to the MF. Something like: "What do you mean I have to call the data center to get my VM recycled. Are you nuts? Just get me a 1U server and let me control it. I don't need these people in my way!" It is just not intuitive that they should not have control of their own Linux, unless it is the production web / app server or something. There is a sense of ownership: That is my Linux server.

This is compounded of course by the fact that with the general Hossness (TM) of the MF and the best OS on the planet, z/VM, there can be hundreds of Linux VM's. Thousands even. Now we get a real clash of cultures (the folks who want to own their compute resources vs. the central data center), and a real stress on time and resources (all the behind the scenes work it takes to keep the shop running and up to date vs. being responsive to the end user). If one runs this Linux shop the way that the MF has done it in the past, then a system programmer makes all the changes to the guest operating systems, and a system operator brings up and shuts down the same Guest OS's and the "owner" just uses it for whatever they need to do. And oddly, no one is happy.

• The Sysprog or Sysprog team is/are stress cadets because someone forgot to add more hours to the day to do all the work.

• The Operators are really really tired of all the reboot phone calls and emails and equally unhappy when they get a complaint that the reboot did not happen in five minutes, even though they were running backups over on MVS and mounting a tape, retrieving an archive from the vault, or something similar at the time.

• The end user is not happy because this is not nearly as good as if they just had their own computer. They could install what they wanted when they wanted and reboot all day long and no one would say anything about it.

Part of the problem here is that, for the most part and to the end user, Linux is Linux regardless of the platform it is running on. If it does not look different, why must their behavior towards it be different.

Why Indeed?

VMLMAT solves this culture clash by returning control of the individual Linux VM's to the end user that "owns" them. It is not the exact same thing as having a physical big red button, or the server tucked under the desk, but via a standards compliant, any browser should work, web page they can shutdown, restart, or rebuild their Linux to any version of Linux in the repository. This is something like Vmware's VI Client, except that it adds provisioning to the mix, and where Vmware has snapshots, VMLMAT has archives. Not the same thing, though they can function in the same role more or less. Now the Operators are focused on the things that need their physical presence, the sysprogs are working on more technical things, such as putting the latest version of Linux into the archive so that all can benefit from it.

Install once, run many (tm).

And of course, the reboots are happening right when they are requested. Everyone is much happier

VMLMAT takes the paradigm of real machine ownership a bit further: In the real world, one person may have several Linux machines, or a group or people working together might have just one... or a group of people have a group of machines. In database-speak, it is a one to many, many to one, or many to many relationship. VMLMAT implements this so that there can be individual machine ownership, or group machine ownership, except that the 'machine(s)' is/are really Linux Virtual Machine(s).

The time saving are real too: Ron noted in his "Beyond Linux Cloning..." post that VMLMAT returned 30 hours a week to his time. Time that he used to spend working on tickets for the users to make various changes to their Linux VM's to match their current requirements he got back, plus now the end user was not waiting on him to have time to get to their tickets either. Everyone was more productive. A huge win-win.

Our savings may be a bit overstated: We are an R&D environment, and we make changes all the time. We have test versions and test versions and test versions. We need something at base level, another with a particular set of patches installed, another with a different set of patches, another totally bleeding edge current, plus Alphas and Betas... Still, I think that the savings are there for everyone.

http://vmlmat.wiki.sourceforge.net/

My days ... years really... as a mainframer covered a period of time where IBM did something very technically interesting (for a V/geek anyway) where mainframe disks are concerned. Back in the 1960's IBM designed the disk data architecture called Count Key Data (CKD) and it is a very different beastie than the Fixed Block Architecture (FBA) we are all used to from UNIX, Linux, and MS Windows disk formats. Follow the links on the terms for a deeper dive. As the second link points out, FBA is a term that is hardly even used anymore since it is so common.

Modern mainframe disks from IBM, STK, and others are, as far as I have seen, all based off concepts introduced in STK's Iceburg project. This was the first place I came across them in any case. Among other things, what the 'berg did was to stuff a cabinet full of FBA SCSI disks, and then put a controller in front of them that emulated CKD to the mainframe. An early version of storage virtualization. Why go to all the trouble?

IBM had introduced real FBA devices for the mainframe much earlier: devices like the 3310 and the 3370. They were often hated and abused by the system programming community that was used to CKD. The reason for the dislike was mostly the granularity of addressing all the blocks on the disk. Despite this dislike (more on that in a bit) I was working in an IBM internal data center at the time and I was told by a hardware person that FBA was the future whether the mainframe folks liked it or not. They were partly right.

VM on the mainframe was into storage virtualization long before storage virtualization was cool. VM has what are called 'minidisks'. A real CKD device such as a 3350, 3380, or 3390 (in my time: there were other models) could be subdivided into many smaller virtual disks. This is not the same thing as disk slices either: nothing is written to a partition table on the disk, and the limit to the number of mini-disks was equal to the number of cylinders that the device had. A 3390 model 1 had 1113 cylinders, and each cylinder had 849,960 Bytes. So VM could create 1112 mini-disks. The 3390 model 54 (which is actually not real hardware, but was something that later disk products emulated on top of FBA SCSI disks) has 65520 cylinders.

To create a mini-disk meant going into the VM directory (the place that defines all the attributes for all the virtual machines that will be running on that computer) and entering the start and end cylinder of each mini-disk. You had to be sure that your new mini-disk did not overlay any other mini-disks. When one is working with numbers like 1113, the math is pretty easy. Minidisk 1 starts at cylinder 2 and ends at cylinder 10. Minidisk 2 starts at cylinder 11 and ends... where-ever. Point is, do not overlap them!

Enter FBA. The 3370-A2/B2 did not have cylinders, it had blocks. 712,752 blocks per disk. The data entry in the directory became very persnickity.

Here is a nifty table with all this kinds of data to continue that deeper dive.

The point of all that is this: IBM responded to the customer and did not put out any more FBA disk-stuffs for the mainframe. FBA was the future, but the customers (primarily, I was told, the MVS community) wanted CKD. This means in part that IBM was on the wrong side of history when it comes to commoditization of disk storage, even if they were doing the right thing by being responsive to the customer. There are a lot fewer CKD disks sold in the world than FBA, and in fact right now I am not aware of any real CKD. CKD looking disks are 'created' by smart storage controllers, essentially virtualized by microcode out of FBA disk arrays. The STK Iceburg led the way on that, and changed the way MF storage has been designed ever since. That is one of the reasons that there was never anything past the 3390 model in the IBM storage line of real CKD devices. Everything since then: the Ramacs, Sharks, and DS lines all use smart storage controllers that create "virtual" CKD devices out of real FBA ones. VM and MVS don't know they are not talking to 3390's for the most part.

The fact that the disk MF vendors have to emulate the old CKD architecture does have an effect on price. MF storage is far less expensive per Terabyte now than it was back then. It is more expensive than most of the straight FBA / SAN / NAS type solutions available today.

FCP

These days you can hook the mainframe to SAN disks, which are of course FBA. Last time I checked though, z/OS (the current name for MVS) could not use them at all, and while VM can see them, it can not fully utilize them. Linux guests of the MF can, even to the place of being able to boot off them, and that takes one to a funny place in mainframe-land. If one uses SAN disks of the MF, probably to save money, then they are also not getting full advantage of the mainframes monster I/O subsystem. In one of the oddities that is MF, even though it still uses arcane formats like ECKD (and I imagine I'll get some hate mail for saying ECKD is arcane) there is no getting around the fact that with 256 I/O channels,  multipathed devices, etc, that the MF is still the world champ for doing I/O. To the MF, doing I/O down one strand of fiber feels to it like drinking a real ice cream malt through a plastic coffee stirrer.

So: Sure, one can do FCP / SAN disks but we did not, and so there was no money to be saved there for us. We needed a different way to leverage the economies of FBA. This was something VMLMAT was created to do.

NAS and Archive

I have spend many many posts here talking about our various tiers of storage, and how we have built them out of commodity hardware. Most recently I have been talking about how we replaced the most mission critical NAS server in the shop that was a TruCluster with a CentOS cluster. This has had huge effect on our cost-per-Terabyte. Depending on how HA we need it to be, we are looking at anywhere from 1000 to 3000 USD per Terabyte now.

None of that is mainframe attached of course. No Escon or Ficon or even an external Fiber Channel connection (there is an internal-to-the-NAS-cluster FC SAN). To access this low cost storage requires a TCP/IP based protocol: NFS, ISCSI or CIFS. No problem for UNIX, Linux, or MS Windows based gear, and these days, no problem for the mainframe either... as long as you are talking about a MF with Ethernet connections, and most MF's these days have the "OSA" installed. The Open Systems Adapter sits where a channel adapter used to, and provides a number of industry standard Ethernet connections so that the MF is no longer isolated from the "Network is the Computer" world: Has not been for years actually, but the OSA has been a far better solution than what came before it, like the 3745 communications controller with an Ethernet adapter in it and a few other external, channel attached Ethernet (or FDDI or Token Ring...) communications adapters.

Now put Linux on the Mainframe. Native TCP/IP needs are met, and both Linux and VM understand the that protocol stack. NFS is there, NFS can be pretty fast, and it is inexpensive these days to use it.

Virtualization can lead pretty easily to server sprawl, and in some cases, like ours, there is a very strong business need to keep around all sorts of versions of Linux. It is not just the multiplicity of the Linux versions either, although there is that to consider. Here are some scenarios where there need to be a number of versions of Linux around on the Mainframe:

• Test copies: After Linux is installed, someone comes along and configures it to meet a special need. Installed more applications, configures it all to work, and then wants to save that off as a template so that others can do destructive testing against the same environment.

• Multi-Tier application roll out: One group writes the code, another Beta tests it, another QA's it, another stages it for production and another *is* production.

• Disaster Recovery and Business continuance: A running production environment needs to be replicated to another location.

It *easily* can get more complicated than that. It is no effort at all to end up with 100 running Linux Virtual Machines and have three and four times that many non-running copies staged out on the disks in case they are needed. The Mainframe disks. The very expensive Mainframe Disks.

This was where Ron's vision of VMLMAT started in part. It was not originally just a cloning tool, but an archiving tool. Once he had the ability to clone a VM to a new VM, he had what he needed to create archives. The only bad thing was, even tarred and gzipped (tgz), it would still be on the expensive mainframe disks.

Ron had been involved from the very early days with our Linux based NAS servers, building several of them. He knew they were inexpensive and very fast, able to run NFS at near Ethernet wire speed. He wanted to be able to leverage them to store his tgz's from the mainframe, so he added that feature to VMLMAT.

With the mainframe on one end, and the speedy NAS server on the other, it was a matter of less than 10 minutes to recall an archive stored out on NAS, unpack it onto a target VM, customize its identity, and boot it. It became cheap to keep multiple copies of various archives. It became easy to take the storage tree and mirror or otherwise rsync it to other, different NAS anywhere in the network. It also was fairly inexpensive to keep off site tape archives of the MF archive tgz's. Nothing more special than just being sure the archive directory is being picked up by your backup systems of choice: To the backup software the tgz's are just large files, and they do not care that they came from the MF at all.

It is such a simple idea that it almost seems obvious in retrospect, but oddly it was not. When one works on the MF, there is a reticence to leverage non-MF gear sometimes. It just doesn't feel as safe.

There is also nothing that says that this has to be NFS. That is what we used, but VMLMAT is Open Source. Adding a different protocol, say FTP, to the archiving function would be trivial.

The ROI was immediate: All the MF storage we had been using to keep all the various versions, flavors, and configurations was returned for other projects: We avoided a MF disk upgrade because of it. Only the running Linux VM's are stored on the MF disks. If there is a downside it is that by reducing the cost of storing the archive, there is a tendency to now keep *everything*. Kind of like server sprawl, the cost of storage makes the cost of maintaining and cleaning the archives kind of expensive.

This is not a huge issue though: If you are getting that big then maybe you are also looking at HSM and data de-duplication as part of your data center plan, and VMLMAT's way of storing things plays right into that. Archives not recently referenced can be moved out to tape and deleted, and only one or three or whatever you policy it copies of them stored.

Best of Both Worlds

With VMLMAT you now have the best of both worlds: Your running, important MF Linux images are getting all the benefits of being on the mainframe, such as the bodacious I/O subsystem, and your inactive archives are stored at extremely low cost and are quickly accessible.

I assume any reader of a technical blog has come across the "Law of Unintended Consequences". A decision, or path that leads to unexpected places. It usually has negative connotations, as in 

"We (the generic we) decided not to spend any money on a new firewall, and with our savings we fought off a series of virus and malware incursions that cost us 100 times what the new firewall would have" 

or 

"We (still the generic we) designed the car to be light weight so it would get good fuel economy, and with the money we saved by not using higher strength materials we paid for all the lawsuits we were served with when it turned out the cars fold up like lawn chairs when hit by a Vespa.".

There are lessor and greater examples of the law than this, and I totally made those two up (although I am willing to bet my first example has happened someplace to someone). The law is well known though: It has a Wikipedia entry even, and as noted there, when there is an upside to the law, it is called a serendipity or a windfall. VMLMAT provided us with a windfall recently.

Rote Work

Computers are supposed to help us out with repetitive tasks. That is what they are good at, and we... at least I.. am not.

We had a situation not that long ago where we were re-doing some internal TCP/IP addresses, and all the Linux guest on the mainframe needed to have new IP addresses and new subnets. We were going to have to go manually through over 100 VM guests and change their IP settings.

The chances for error were high, and boredom even higher. Ron and I talked about it, and a little light went on for us. VMLMAT could be made to do this in very short order. Not only was it Open Source, it was *our* Open Source. VMLMAT had been written to be easy to extend. Ron thought he could add the feature in a fraction of what it would take to re-IP all the VM's manually.

The secret sauce was in the module Ron had already written that preserves  the TCP/IP identity of each Virtual Machine, regardless of which actual version of Linux was running at the time. VMLMAT had allowed us to treat VM's Virtual machines more like they were group or personally owned assets. Once we had set up the VM in the directory, VMLMAT updated to know whose VM this was, then we were pretty much done. The end user, or group of people could go into the web page and make the virtual machine into any version of Linux in our archive. No matter what they did, the identity code would chain through all the right places and set up the new Linux VM as having the same IP address, Subnet, Default Route, and DNS settings as it did before it was changed.

Since VMLMAT could do that, why couldn't we slightly change it so that it could be handed a list of VM names, IP addresses, and Subnets, and have it chain through *all* the guests, and reset them to the new IP? This module already knows all the different places the information is kept for each distro, so all it had to do was detect the distro, change the IP to the *new* one, and then run on to the next VM. VMLMAT could do in minutes what would take us days to do by hand.

And so it came to pass that we could schedule one short outage, pull the trigger on the re-IP mod, and be done. No mistakes. No long outages. R&D went home Friday night and their VM's had one IP address, came back Monday and they had another, and never knew anything had changed.

VMLMAT already had the habit of returning over 80% of Ron's time versus pre-VMLMAT days.

This was a bonus.

UPDATE: Ron has now posted an in depth look at the re-IP module over in his blog: Open for Mainframe. Have a look!