~kanarip

Some organisations tend to use VMWare ESX(i) to virtualize their server environment and regardless of whether that is the best choice for any given environment, what you should most certainly not do, in my humble opinion, is run any type of database server on any kind of virtualization technology that;

1. does not support virtio,
2. does not natively support x86_64 guests,
3. does not permit you to tweak the hell out of it's virtualization stack,
4. does not permit you to fix their code (if at all the code is available)

To clarify statement #2 a little; You should know that if you choose the guest to be x86_64, you are going to see a relatively large performance hit if your virtualization technology only supports a x86 host node operating system and thus needs to apply binary translation (in series, increasing the number of levels to decend in the virtualization stack), to strip the (faster) x86_64 "shortcut" CPU instructions and have it all be supported by the x86 host operating system.

I'm not the absolute expert on the subject of virtualization myself, admittedly, regrettably, and whenever I talk about the benefits of later generations of Free Software virtualization in comparison to other virtualization technologies I tend to simplify and abstract my understanding of how it is more efficient and less error-prone, by saying that most of the throughput from the guest to the actual hardware needs to decend as few levels in the virtualization stack as possible, and some stuff can be done in parallel (memory page sharing, virtio, anything else?). It also helps to simply not support layer-on-top-of-another-layer foo such as binary translaction which results in building a giant stack to be descended (in series). I think in the past I've drawn a few very high-level diagrams illustrating this for the often not-so-technically-savvy audience, but I'm unable to find them right now.

So, I like to think that I have some understanding of how this stuff works even though in this blog post I'm obviously wrong when I say stuff like some of the aforementioned because it is technically not entirely accurate but regardless, it draws a nice picture for those that would not understand what I'd be saying if it were technically accurate.

Anyway, that's where I'm coming from, you be the judge of whether I understand the foo or not ;-)

A MySQL server on VMWare ESX(i) 3.X or 4.X virtualization

If you virtualize a database server, you are going to find that any given form of a performance hit by using that virtualization technology is greatly impacting the performance of your database server. It is commonly known that one should very, very carefully consider NOT to virtualize the database server on a virtualization technology that meets the aforementioned 4 symptoms. Taking one more step back, I'd say even virtualizing a database server on any given type of virtualization technology should be very, very carefully considered even if the virtualization technology of choice is the latest and greatest and offers 98% performance of bare metal (Guess which virt. tech. I'm talking about here and get a free beer at FUDCon -all you need to do is leave a comment and attend and pick up your beer).

Let me start by saying this database server is the backend store for one of those services that just needs to be snappy and fast and shiny all-over; end-user facing, it's the most signifcant and most important service a business can run internally; E-mail and calendering. It's what tends users to go nuts if unavailable or even not as responsive ans snappy as they would like. If you've ever done Helpdesk Support I bet you know what I mean.

Re: the performance hit; For one, the I/O performance on the hard disk for this one MySQL server that I manage, virtualized using VMWare ESX(i) is around 12 MB/s. My laptop hard disk does ~57 MB/s of buffered write I/O (if I've not used the wrong benchmark by mistake, I'm sure one of you readers can recommend me a util I do get correct numbers with if this seems a little off).

Since it's doing 33k of InnoDB reads per second (on average, so including weekends and outside office hours), you can imagine the peak is over 250k InnoDB reads -which I then imagine cannot be in memory at any given point the database server needs it, causing direct (yet buffered?) I/O to disk.

It's running on x86_64 guest operating system because of the memory footprint of MySQL being over 2GB so that it can cache a lot, but as it seems the VMWare ESX(i) node it runs on only natively supports the i686 CPU instruction set. Enter the binary translation from x86_64 to i686 (that is, if it is not i586), in series, accounting for yet another layer in the stack that increases the performance hit as well as makes the environment more error-prone. Note that the hardware itself is x86_64, and so the virtualization technology does not use all of the hardware capabilities.

Then of course there is the PAE CPU feature needed for x86 to be able to address over 4GB (which shows as 3290MB) of memory. No matter how small the performance hit actually is, it is a performance hit regardless.

I'm done throwing everything I know at the problem of performance in this situation and so here it goes;

Dear lazyweb,

What would you recommend I do (except raise my hands and stop supporting the environment which I've already seriously considered), given this awkward situation? What's left to tweak that you think I might not have tried yet? This MySQL box is going to production soon and I want it's dependent services to be snappier then ever before -something I don't think I'm going to get if the MySQL box remains to be virtualized using the current virtualization technology in use -but I secretly hope you can prove me wrong.

The Fedora Project has put installation media CDs with it's back to the wall, has loaded and aimed it's guns, and is ready to pull the trigger in 5 months. Here's the start of the thread on fedora-devel-list.

As you probably know, Fedora Unity releases Fedora once or twice or trice after the general availability for a given release, in the form of Re-Spins. We've included CD Installation media in our Re-Spins even if the official Fedora distribution didn't have them generally available (which was the case for Fedora 7 and Fedora 8).

The discussion on fedora-devel-list right now seems to evolve around simple statistics, but...

Regrettably, Fedora Unity lacks the statistics on how many times our Re-Spin is downloaded. If, at all, the numbers that we have are complete and trustworthy, then these are the amazing numbers.

Note that you should never trust statistics.

Our numbers at http://spinner.fedoraunity.org:6969/ show a grand total of over 5000 complete downloads (those of Fedora 8 not included), and over 22000 pending downloads... This I suspect is a number so wrong... If you believe this number let me give you another shock: Santa Claus does not exist and has never existed.

The other number says over 17.000 torrent clients have downloaded our products. Now this number I could argue could be true... but I simply don't know, so I'll let you make up your own mind.

The statistics we don't have include your Jigdo downloads for every release that we do (and have done with the help of Jigdo since Fedora 7 or so), but it's obvious that our Re-Spins are downloaded a lot. An awful lot, even. Even if the numbers I refer to are not at all true... given the amount of feedback that we get, the amount of requests for a new Re-Spin even just after a new Fedora release comes out ("Can you please, please include the 0-day updates???"), I suspect the number of users is in the thousands.

We, Fedora Unity, obviously provide Re-Spins as a service to the community. Those that like the pie, get the pie, then eat the pie. We're not as much concerned with the number of downloads... For us, it's a matter of quality, not quantity.

Quality in how you experience a Fedora installation a few months and a few hundred updates later. Quality in enabling fixes to be backported in to the installer images so that Fedora actually improves and the release matures or stabilizes. Not the quantity of hard numbers we can't trust -the number of downloads through one method or another.

Ergo, in the discussion on whether the Fedora Project should re-consider it's decision to include CD media in it's release once again, after re-including CD media in the release with Fedora 9 following some pressure from within the community, I can only argue that the numbers shown are not entirely representative, and that Fedora Unity will not bend over to lame argumentation, and Fedora Unity will continue providing this service to the community. The question is, how far will the Fedora Project let Fedora Unity do so through the Fedora Project proper? Now that this question has been asked, I'm wondering what the answer would look like. Mean time, if you have purpose for CD media, I suggest you join the discussion and have your voice be heard. Now.

After tonight's unknown twist of destiny, it seemed both my internet connections at home went down, and nothing, nothing at all was reachable. My home network really runs everything, from DNS for my domains to websites and email, and so downtime is... painful.

If not for a good friend of mine, Stefan Hartsuiker (LinkedIn, Fedora Project), this would have still been the case so I owe him a big thank you for fixing up the mess in my apartment and bringing all my stuff back online:

<big>Thanks Stefan!</big>

I owe you about a dozen or so beers of your choice!

After the installation is complete, you navigate to your website and voila, you have a rather complete Content Management System installed. Now, how do you personalize it?

Select, install and/or enable a few modules

Using the Administer -> Site Building -> Modules page, you can select the modules you want enabled on your website. Whenever you install a module, this is where you go to enable it. Whenever you remove a module, this is where you go to ensure all data is removed. There's plenty of modules available from Drupal's website, so I suggest you start your search for interesting stuff there.

Installing a module is often pretty straight-forward:

1. Download the tarball
2. Extract it in drupal-6.12/sites/all/modules/
3. Go to the Administer -> Site Building -> Modules page
4. Enable the module
5. Configure the module should the module require or allow additional configuration

In the setup I have on www.kanarip.com, I have the following modules installed - not necessarily enabled:

1. advanced_help
2. cck
3. feedapi
4. gallery
5. jobtrack
6. service_links
7. storm
8. ticketing
9. twitter
10. views
11. wysiwyg, including editors
    1. jwysiwyg
    2. openwysiwyg
    3. TinyMCE

We'll continue describing how that goes later on in this HOWTO.

Add Roles

Initially, for example, you only have two user roles:

1. Anonymous user
2. Authenticated user

Assigning permissions to different people using only those two roles can be a bit... brutal, so add roles. If you are going to have a blog on your website and you want someone else to be able to blog as well, but not all authenticated users, add a role Blogger and assign the appropriate permissions. Similarly, if you want multiple people to be able to administer the site, add a Site Admin role. As the word role indicates, add roles for anything you want other people to be able to do, then add those roles to the user accounts of those people.