~kanarip

Today is the day we come together with a bunch of Linux experts, and start knocking down some of the items on our TODO list, as well as -hopefully- share more information and responsibility on the overall Linux infrastructure inside our company, and the Linux infrastructure at a lot of out customers both!

I love to be able to announce the first "sysadmin-main" meeting within my company, Operator Groep Delft. You read that right, that's the exact same name the Fedora Infrastructure team - an example to us all as far as I'm concerned, uses to indicate the group of people ultimately in control of all systems and services. Read the more detailed description. I'm going to try and apply this very concept to my company's internal system administration.

Instead of taking care of the Linux infrastructure with no dedicated Linux engineering resources, we decided that utilizing all of the available Linux competencies within the company would be more efficient, more effective, more flexible and would vastly improve redundancy, collaboration and expertise between the bunch of us.

Since we're a consultancy company, most of our Linux Engineers are hired out to customers, making billable hours, earning us all money, and are thus not in a position to really sit down and take care of OGDs Linux infrastructure as well. Basically our engineers can only spend that one minute they have left at the end of a day or the hour they would otherwise waste on one of those boring meetings -they merely require the facilities to do so.

So instead we decided to try and make it a community effort of some sort, in the sense that all sysadmin-main engineers require a minimal Linux certification level of RHCT, and will have access to Life, Everything and The Universe once we've sat together and introduced them to all procedures and such.

Now, nothing is set in stone yet, but at least we're going to have a bunch of interested people show up at our meeting soon, which will hopefully lead to a large group of engineers willing to do interesting stuff.

We'd plan and assign -amongst ourselves- the tasks in our queue. If either of us needs a little more time, we save up to 8 hours a week to spend at more intrusive changes like migrations, upgrades, planning, documentation, build & test, development, that sort of thing. We (as a group) would decide what we do, how we do it, and who actually gets to execute.

Ultimately, I'd be (held) responsible (or accountable?) for the group, and my manager would be responsible for the well-being of the infrastucture as a whole, just like he is now.

Just to give you an impression of what it takes to do what we do between the two of us at this moment, in the very little time that we have available; Zarafa, HA/LB Red Hat Directory Services, Puppet (with help of puppetmanaged.org modules), Cobbler, SELinux (enforcing), Nagios, Munin (looking at Zenoss to replace both), 15 mod_security enabled webservers, 4 database servers, 6 development environment staging boxes, and a couple of workstations.

PS. For those of you who read this, and are colleagues of mine, you can find more information on https://nix-noc.ogd.nl/trac/

Some organisations tend to use VMWare ESX(i) to virtualize their server environment and regardless of whether that is the best choice for any given environment, what you should most certainly not do, in my humble opinion, is run any type of database server on any kind of virtualization technology that;

1. does not support virtio,
2. does not natively support x86_64 guests,
3. does not permit you to tweak the hell out of it's virtualization stack,
4. does not permit you to fix their code (if at all the code is available)

To clarify statement #2 a little; You should know that if you choose the guest to be x86_64, you are going to see a relatively large performance hit if your virtualization technology only supports a x86 host node operating system and thus needs to apply binary translation (in series, increasing the number of levels to decend in the virtualization stack), to strip the (faster) x86_64 "shortcut" CPU instructions and have it all be supported by the x86 host operating system.

I'm not the absolute expert on the subject of virtualization myself, admittedly, regrettably, and whenever I talk about the benefits of later generations of Free Software virtualization in comparison to other virtualization technologies I tend to simplify and abstract my understanding of how it is more efficient and less error-prone, by saying that most of the throughput from the guest to the actual hardware needs to decend as few levels in the virtualization stack as possible, and some stuff can be done in parallel (memory page sharing, virtio, anything else?). It also helps to simply not support layer-on-top-of-another-layer foo such as binary translaction which results in building a giant stack to be descended (in series). I think in the past I've drawn a few very high-level diagrams illustrating this for the often not-so-technically-savvy audience, but I'm unable to find them right now.

So, I like to think that I have some understanding of how this stuff works even though in this blog post I'm obviously wrong when I say stuff like some of the aforementioned because it is technically not entirely accurate but regardless, it draws a nice picture for those that would not understand what I'd be saying if it were technically accurate.

Anyway, that's where I'm coming from, you be the judge of whether I understand the foo or not ;-)

A MySQL server on VMWare ESX(i) 3.X or 4.X virtualization

If you virtualize a database server, you are going to find that any given form of a performance hit by using that virtualization technology is greatly impacting the performance of your database server. It is commonly known that one should very, very carefully consider NOT to virtualize the database server on a virtualization technology that meets the aforementioned 4 symptoms. Taking one more step back, I'd say even virtualizing a database server on any given type of virtualization technology should be very, very carefully considered even if the virtualization technology of choice is the latest and greatest and offers 98% performance of bare metal (Guess which virt. tech. I'm talking about here and get a free beer at FUDCon -all you need to do is leave a comment and attend and pick up your beer).

Let me start by saying this database server is the backend store for one of those services that just needs to be snappy and fast and shiny all-over; end-user facing, it's the most signifcant and most important service a business can run internally; E-mail and calendering. It's what tends users to go nuts if unavailable or even not as responsive ans snappy as they would like. If you've ever done Helpdesk Support I bet you know what I mean.

Re: the performance hit; For one, the I/O performance on the hard disk for this one MySQL server that I manage, virtualized using VMWare ESX(i) is around 12 MB/s. My laptop hard disk does ~57 MB/s of buffered write I/O (if I've not used the wrong benchmark by mistake, I'm sure one of you readers can recommend me a util I do get correct numbers with if this seems a little off).

Since it's doing 33k of InnoDB reads per second (on average, so including weekends and outside office hours), you can imagine the peak is over 250k InnoDB reads -which I then imagine cannot be in memory at any given point the database server needs it, causing direct (yet buffered?) I/O to disk.

It's running on x86_64 guest operating system because of the memory footprint of MySQL being over 2GB so that it can cache a lot, but as it seems the VMWare ESX(i) node it runs on only natively supports the i686 CPU instruction set. Enter the binary translation from x86_64 to i686 (that is, if it is not i586), in series, accounting for yet another layer in the stack that increases the performance hit as well as makes the environment more error-prone. Note that the hardware itself is x86_64, and so the virtualization technology does not use all of the hardware capabilities.

Then of course there is the PAE CPU feature needed for x86 to be able to address over 4GB (which shows as 3290MB) of memory. No matter how small the performance hit actually is, it is a performance hit regardless.

I'm done throwing everything I know at the problem of performance in this situation and so here it goes;

Dear lazyweb,

What would you recommend I do (except raise my hands and stop supporting the environment which I've already seriously considered), given this awkward situation? What's left to tweak that you think I might not have tried yet? This MySQL box is going to production soon and I want it's dependent services to be snappier then ever before -something I don't think I'm going to get if the MySQL box remains to be virtualized using the current virtualization technology in use -but I secretly hope you can prove me wrong.

An article I've written has been published in the latest issue of the magazine "Linux on Blue". I've attached a scan of the pages -even though the magazine doesn't publish it's issues online it's a free magazine with about 7.000 copies being distributed with companies in the Netherlands so I don't think they'd mind me posting my own article.

Thema of this month's issue is Security on Linux, and so I've written a very high-level article on Security Enhanced Linux titled "Mandatory Access Control and Auditing", but the funny thing is the issue also has an article on AppArmor. Poor Novell...