Providing useful but effective advice on sizing and capacity planning has been a challenge to software vendors since they began.  If you consider the huge range in possible hardware platforms, and then the huge range in products and functionality for a given vendor, you realize its essentially intractable.  Various vendors have made interesting attempts at covering this space in the past.  At the now defunct ERP vendor Baan, they had a very sophisticated matrix of weighting factors for all their different applications. So you would break your user count into various functional groups, multiply each group a magic weighting factor, and Whallah! - you have a magic weight for the overall workload. Their competitor SAP had another approach, with one headline benchmark that would be run on every platform imaginable.  So you could flip through the tomb of SAP benchmark results, and eventually find a little box containing a few numbers about the specific box you were interested in - take it or leave it. 

I would argue that covering either of these two dimensions entirely means that you're stuck with so little information per data point that you really can't learn much.  So you then have the choice of taking those numbers as perfect and betting your deployment on it, or throwing them out and starting with nothing.  We've choosen a different approach for the sizing of the BMC IT Service Management Suite (ITSM 7.0).  We evaluate a very small set of platforms, but hope we can relay enough information about those platforms that the sizing of other configurations can follow logically from this basic knowledge. 

Three Sizes Fit All

Specifically, we provide sizing around a small, medium and large standard configuration, or reference architecture if you like that term better.  We then provide sizing of a few of the key applications in the suite for each of these architectures.  We don't contend to replace the thinking and work that usually takes place when designing a configuration for a ITSM deployment.  But we hope to give a jump start to those activities, and provide some quantitative data upon which to base such activities. 

How wrong could our sizing estimates be?  Unfortunately, plenty.  Consider the variances between really big databases vs very small databases, very busy users and very slow users, Wide-Area-Network response times versus local area network response times, etc.  But the estimates provide a baseline from real testing, and we hope we can help extrapolate to a specific customer case from that.  Why go out on a limb if we can't be sure?  I had various engineers asking me that question.  In the end, it came down to this.  In my team, we have many engineers who have done sizing and performance work for 10 to 20 years.  The average customer just wants to get this configuration built and move on to their day job.  Who should be making the leap of faith to do the best sizing they can?  Clearly my team is the right group to make a guestimate, and the best we can do is base those guesses on pretty solid data.  Now take those guestimates, add in a fair bit of concepts and learning that we got out of our testing, package up in a whitepaper and hopefully we've improved the world at least a teeny-weeny bit for our customers and partners. 

So what did you learn?

We learned this: once you settle on a solid three tier configuration, the fun is just beginning.  Now you'll want to tweak that configuration to do reporting and archiving - while not impacting your on-line users.  You'll want to tweak that configuration to have disaster recovery and failover, all for nearly free.  And you'll want to start planning for the acquisition that might swell your user count by 100%, as well as for the downsizing that might reduce your user count by 50%.  So these are the other fun topics we're trying to tackle next - those other things that customers care about. 

Here's something a bit more specific that we learned.  Grid architectures seem pretty much ready-for-prime-time with this product set.  So when you're thinking about hardware for the ITSM suite or the BMC Atrium CMDB, you don't have to go to page 7 or 8 of your hardware vendors catalog, where they keep the expensive SMP with 16 to 32 processors and 64 G of RAM.  Instead, you can stay on pages 1 and 2, where the cheap 2 CPU and 4 CPU boxes are.  Then just buy a couple boxes for the webtier, a couple  boxes for the apps tier, and perhaps even a couple boxes for the database tier (using something like Oracle Real Application Clusters).  Now if you need to add capacity, or even reduce capacity, you  can just roll another box in or out of the configuration.  Your price point per CPU and per gigabyte of RAM is much lower than an SMP (up to 10X lower!) - and you get some failover characteristics for free, as you put a set of machines behind a loadbalancer, instead of having on single point of failure.  So, despite all the over-the-top marketing that's flying out there on grid stuff - do believe the hype - at least enough so that you'll check it out, before buying an expensive SMP box.   

What does a customer look like?

Big customers have interesting challenges, but ones that are starting to sound typical in our world today.  First, they have a big data center in a place that sounds safe and clean, a place like London.  But of course, most of their users are in a place like Bangalore, because their stock holders thought they were not off-shore enough in the last business review.  They drive their load from one big data center across a WAN.  Except for their regional center in a place like Luxemborg, as they don't have legal rights to export customer data out from that location to their other instances.  And even though they do have instances spread across the world, their CIO lives in a place like Denver, and he likes nothing more than to wake up,  walk to the laptop while wearing his pajamas, and find out how many tickets they serviced in the world since he went to bed.   And the results of that query had better be right and quick, or we'll all hear about it pretty soon. 

So large global enterprises have interesting new problems, but are remarkably converging towards a standard set of requirements that are mostly common across them.  The real buzz around here is how to identify those requirements, and then provide effective solutions for them as fast as we can. 

thanks for reading,

Tim

PS.  Here is a link to the ITSM Sizing Guide I mentioned http://documents.bmc.com/supportu/documents/57/42/65742/65742.pdf

The Good Ol' Days

We all recall our heady days, before we knew that we needed a CMDB. Here's a little story to help remember that time. One day it happens that each time you start outlook on your laptop, you get an error and the application crashes. So you call the help desk:

[helpdesk] "Hello Mister Horrie - how can I help you?"

[you] "Hi, yeah, my laptop can't bring up outlook today. I kind of need it, so I'm hoping we can get this going quickly."

[helpdesk] "OK. Let me see, so this is the machine named tharris-lap-02 right?"

[you] "Ummmm - no, not really. tharris-lap-02 has been gone for a long time. This machine is tharris-lap-04."

[helpdesk] "Sorry - what do you mean by gone? I only have tharris-lap-02 listed for you."

You take a moment to reflect on the long and mostly happy life of tharris-lap-02. You remember the day that you presented it to your two year old son Bobby, and how happy Bobby was. At the time it was a 700 Mhz processor with 128M of RAM, and it blue-screened everytime you started microsoft word. So it was really only good for the scrap heap. You remember the happy pictures your wife took, of you working with the new tharris-lap-04 on the couch, while Bobby sat at your feet, banging on the keys of tharris-lap-02 with utter satisfaction. And then you remember a few months later, when Bobby had one of his first "big boy" cups of orange juice, and he carefully poored the whole cup into the keyboard of tharris-lap-02. He was feeding the machine breakfast evidently. The machine flickered through a series of interesting looking screenshots, as if its computational life was briefly passing before its monitor, and then it went dark one last time. Gone.

[you] "Well, its kind of decommissioned. Well, not officially I guess. But its gone. And I've had 04 for a long time now."

[helpdesk] "I see. Well here's what I need you to do for me. Go to the assets app off the corporate portal, and officially decommission tharris-lap-02, and then create the asset for tharris-lap-04 and assign it to yourself. Then call back and I can help you fix your machine. OK?"

Of course, anytime the helpdesk gives you a list of things you need to do for them, its a bad sign. But why didn't they know that tharris-lap-02 was long gone, and that tharris-lap-04 had been on their network for a few years? Because they lacked an accurate picture of their own IT environment. Therefore the users, and the IT staff, both had to eat the inefficiencies caused by that lack of knowledge.

The Good New Days

I'll now jump to some more detailed discussion of how to build a CMDB. If you're going to take away one aspect of CMDB design from this note, remember this - every bit of information in your CMDB is an investment, so invest wisely. I'll explain what I mean.

The CMDB is better than a simple database or spreadsheet, due to the added level of functionality that a mature CMDB provides. Specifically, it allows you to reconcile data from multiple data sources, it allows you to share that data with well integrated applications, and it allows you to pump data into the CMDB from external datasources with minimum fuss and hassle. Trust me, you want this level of functionality, and therefore it should be OK that you have to pay some processing for it. All these computational tasks are O(n) operations, meaning that their run times are roughly proportionally to the number of data elements being processed. Therefore more data takes more processing time, or it requires larger hardware to run, or in what ever form, it requires more IT investment.

Now back to the real world of an IT shop just starting to plan a CMDB deployment. Imagine this shop has a existing history of using automated discovery, likely called an IT Inventory application in this context. So this shop runs Inventory every day, and there are people who have staked their job on the fact that the results are detailed and accurate. Again, this data source does not have the extra CMDB functionality noted above, and so it is not a CMDB in itself, so it may be alot cheaper to generate a very big inventory dataset than to create a CMDB. But the folks who built this inventory datastore have come to love it like a child. They love every configuration item (CI), as a gleaming pearl that they pulled from an oyster themselves. No matter how seemingly insignificant a piece of data is, they wouldn't part with it.

The CMDB deployment planning starts, and the question comes up, just what CIs have real business value to us in the CMDB? The answer is obvious to the Inventory team - everything! Usually "everything" in this context means not only all hardware related CIs, but also all applications on every machine, and finally every patch of every software on every machine. Automated discovery (aka inventory) will routinely capture information like the fact that there are 23 device drivers on your server, and that some of these device drivers have up to 7 patches applied on them. Generally this information is useful for inventory as a way of tracking software compliance for certain profiles of users and their machines. But in the CMDB its questionable how valuable this information is, and if you really want to resource a system that can include all that info.

The dataset size for a given IT environment can vary up to 100X based on basic decisions on what you want to include in the CMDB. That's a huge range, and has nothing to do with what product you're using to implement such a CMDB. Imagine that you ask the IT manager in charge of the payroll application how many paychecks you're printing this week, and he says "Well boss, its a number between 1,000 and 100,000". This guy clearly doesn't know his job well. But that is the kind of range (100X) of dataset volume that a customer can effect by careful filtering (or lack of filtering) of CIs in his or her CMDB. Your CMDB might be between 1 million and 100 million CIs in size.

The Right Approach

Now I'll put a stake in the ground for best practices on such a deployement project. Start small, very small if need be, and consider growing later. This means the following: make sure every computer system (laptops, desktops, servers, loadbalancer and router) is represented with at least 10 or 20 CIs, and then add a minimum of critical software titles as needed. But ensure that the goal is to get something into production without big risks or large implementation overheads. Once its in production, you can start to get business value from the CMDB, and learn out to utilize this picture of your IT environment in your incident, problem, change and asset processes. When you need more detailed data you can access other datasources via federation. Then, overtime, you can have the occasional pow-wow where you say - here is another set of data, here is how much it would cost us to include it in the CMDB, and here is how much business value we think we'd get out of having it in there. But as a starting place - at least make sure that in the story above, tharris-lap-02 would disappear and tharris-lap-04 would appear - that's the obvious starting place. This is not just a BMC idea, but instead an industry standard best practice for ITIL and the CMDB. But of course, we like to believe that we thought of it first ;-)...

Happy holidays to all of you,

Tim

What's this Blog About?

I work with customers every day who not only want to talk about ITIL, but who want to build a production implementation of ITIL that helps their enterprise IT environment every day.  As many of us have learned, there is a big difference between sitting in an ITIL class for some days or weeks, and going live with a production service management solution that really improves the lives of your most important IT users and customers. 

So what are those differences between theory and practice?  In particular, how do we map those high level ITIL flow charts and processes down to the bits and bytes traveling around a network, sitting in your database server and eventually putting business information into the hands of your users?  I'm hoping to help with that mapping.  More specifically, I'll provide some ideas and direction on practical choices you'll need to make to deploy such software, and how those choices can improve your IT business.

Is it for Techies or Business people? 

My background is pretty techie, including a PhD in computer science, but my passion is in making this all work for your business.  Along those lines, I'll make a few basic assumptions:

With those ground rules in mind, these are some of the specific things that myself and my team are thinking about now.  We have a performance lab and a fair bit of smart folk and good hardware, all working on the following questions:

• What's the best tradeoff between building a really big CMDB that has every bit of data you could ever want in it, and buiding a smaller CMDB that can be built quickly and kept very up to date on modest sized hardware?
• If you have IT users spread across the globe, what are the costs and benefits of choosing a single global instance of IT service software, versus building regional centers around the world?
• How can a customer know that their hardware investment is sufficient to provide the IT services they require now, as well as scalable for the future as new users and workload are added into the mix?

So I'm interested in practical questions that I hope help you to improve your IT business, but with data from our own performance and scalability labs. 

How you can help improve the state of IT service Management products...

The most exciting thing about my role here at BMC is that I get to help set direction within R&D, but I'm also very much customer facing.  So I challenge the readers of this blog to help me identify what we could be doing better with future versions of the product, and I can then work with product development to make those changes happen.  I hope there is no one reading this blog thinking "yeah, but these products still can't solve this or that problem".  If there are things that keep you from getting the maximum gain out of enterprise implementations of these products, then its my role to understand those issues, characterize them in our internal labs if necessary, and then help drive new features or fixes to make them more useful.  So I hope the real deliverables of this blog are both on your plate and on mine.

Who is this Tim guy anyway?

If you're still reading, you're probably starting to wonder about how credible I am, and if I'm really qualified to give you any interesting tidbits.  My real passion in my 20+ years of IT involvement is to solve tough problems really really fast.  I got started in this area as a 19 year old tyke, when working summer jobs at the National Center for Atmospheric Research in Boulder, Colorado.  I wrote some code to see just how fast these fancy CRAY machines really were, and a few days later got an angry phone call from accounting to tell me that I'd racked up $15,000 worth of compute time.  Oops!  I was a bit embarassed (luckily they didn't charge my internal team), but I was also hooked on seeing how such big iron can really crank.

A bit later I joined a very cool MIT spinoff named Thinking Machines, where we had over 64,000 small processors inside one machine, and we spent alot of time scratching our heads on how to make them work well together to solve big problems.  12 years ago I got more interested in business problems, and joined Oracle corp.  In my 10 years at Oracle, I mostly worked on ERP applications and how to make them run like the dickens. 

At BMC we like to think that IT Service Management is the new ERP.  We also talk about how IT is kind of like the old story of the Cobbler and his kids.  The story goes this way: the cobbler was always so busy making everyone else's shoes, that he never could find the time for his own kids.  So the cobbler's kids were the only barefoot kids in the village.  For the last 10 years IT folks worked hard to deliver HR applications, Payroll applications, Order Management applications, you name it.  But who worked on delivering good IT applications?  Finally we're in an exciting time when we get to work hard to help ourselves, and our IT friends and colleagues.  In particular, I hope to make ITIL compliant solutions that run fast and are easy to deploy, so we can finally get some shoes onto the feet of our poor IT family.  I hope you'll all work with me in that quest...

Thanks for reading!

Tim