~kanarip

I've always been a huge fan of Zarafa, one of merely two serious competitors in the Open Source groupware market.

The other competitor is Zimbra, but I have somewhat less of an incentive to sink my teeth into that Java mess, which installs in /opt/zimbra/, and uses it's own vendored libraries rather then those available in the system stack. This, in my opinion, is just wrong, raises cost and gives you less overall options and control. But enough about Zimbra, because obviously the suite *just works* (and a lot of people are very much happy with it).

I have (and still am) running Zarafa at my company for about a thousand users, and Zarafa's headquarters are pretty close to my company's. I get to speak to Zarafa's people regularly, and most of it is while I'm wearing my Fedora hat ;-)

Either way, now that Robert Scheck and myself are attempting to package Zarafa for our dear distribution, Robert and I run into the following when using Fedora 12:

[jmeeuwen@ghandalf SPECS.mine]$ sudo zarafa-spooler -F
zarafa-spooler: symbol lookup error: /usr/lib64/libmapi.so.1: undefined symbol: _ZN10ECMemTable6CreateEP14_SPropTagArrayjPPS_

Robert has created a topic on Zarafa's forum about this some time ago.

Let me first emphasize that Zarafa is upstream for three libraries:

1. inetmapi
2. perlmapi
3. libmapi

This symbol is undefined in libmapi only, as you can see in Robert's comment.

I once succeeded (I don't know how) to make the command result in a stack trace:

[jmeeuwen@ghandalf spooler]$ gdb /usr/bin/zarafa-spooler
GNU gdb (GDB) Fedora (7.0.1-30.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/zarafa-spooler...(no debugging symbols found)...done.
(gdb) run -F
Starting program: /usr/bin/zarafa-spooler -F
[Thread debugging using libthread_db enabled]
Unable to open configuration file /etc/zarafa/spooler.cfg
Continuing with defaults
Wed 03 Feb 2010 03:53:54 AM CET: Unable to open pidfile '/var/run/zarafa-spooler.pid'
Wed 03 Feb 2010 03:53:54 AM CET: Starting zarafa-spooler version 6,30,9,18385 (18385), pid 7285
Wed 03 Feb 2010 03:53:54 AM CET: using SMTP server: localhost

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7fcb820 (LWP 7285)):
#0  0x0000000000000000 in ?? ()
#1  0x0000003d0da20ca9 in M4LMsgServiceAdmin::GetMsgServiceTable(unsigned int, IMAPITable**) () from /usr/lib64/libmapi.so.0
#2  0x0000003d0e259782 in CreateProfileTemp(char*, char*, char*, char*, unsigned int, char*, char*) () from /usr/lib64/libinetmapi.so.1
#3  0x0000003d0e259a3b in HrOpenECSession(IMAPISession**, char*, char*, char*, unsigned int, char*, char*, char*) () from /usr/lib64/libinetmapi.so.1
#4  0x0000003d0e259b29 in HrOpenECAdminSession(IMAPISession**, char*, unsigned int, char*, char*) () from /usr/lib64/libinetmapi.so.1
#5  0x00000000004069a1 in InitThreadData(thread_data*) ()
#6  0x00000000004075bd in ProcessQueue(char*, char*) ()
#7  0x00000000004078ff in running_server(char*, char*) ()
#8  0x000000000040d04e in main ()
(gdb)

I'm not at all too familiar with C/C++ code, and/or libtool (a more recent version of libtool in Fedora is rumored to have caused this?), and so my first step is to Google. Googling for "undefined symbol" doesn't really give you anything else then forum topics with questions and often not even solutions to very particular problems though :/

So, I turn to you, dear Lazyweb, and I'm asking you to help me wrap my head around it and put the finger on the sour spot.

SPEC: http://www.kanarip.com/custom/SPECS/zarafa.spec

SRPM: http://www.kanarip.com/custom/f12/SRPMS/zarafa-6.30.10-1.fc12.src.rpm

Thanks in advance!

Some organisations tend to use VMWare ESX(i) to virtualize their server environment and regardless of whether that is the best choice for any given environment, what you should most certainly not do, in my humble opinion, is run any type of database server on any kind of virtualization technology that;

1. does not support virtio,
2. does not natively support x86_64 guests,
3. does not permit you to tweak the hell out of it's virtualization stack,
4. does not permit you to fix their code (if at all the code is available)

To clarify statement #2 a little; You should know that if you choose the guest to be x86_64, you are going to see a relatively large performance hit if your virtualization technology only supports a x86 host node operating system and thus needs to apply binary translation (in series, increasing the number of levels to decend in the virtualization stack), to strip the (faster) x86_64 "shortcut" CPU instructions and have it all be supported by the x86 host operating system.

I'm not the absolute expert on the subject of virtualization myself, admittedly, regrettably, and whenever I talk about the benefits of later generations of Free Software virtualization in comparison to other virtualization technologies I tend to simplify and abstract my understanding of how it is more efficient and less error-prone, by saying that most of the throughput from the guest to the actual hardware needs to decend as few levels in the virtualization stack as possible, and some stuff can be done in parallel (memory page sharing, virtio, anything else?). It also helps to simply not support layer-on-top-of-another-layer foo such as binary translaction which results in building a giant stack to be descended (in series). I think in the past I've drawn a few very high-level diagrams illustrating this for the often not-so-technically-savvy audience, but I'm unable to find them right now.

So, I like to think that I have some understanding of how this stuff works even though in this blog post I'm obviously wrong when I say stuff like some of the aforementioned because it is technically not entirely accurate but regardless, it draws a nice picture for those that would not understand what I'd be saying if it were technically accurate.

Anyway, that's where I'm coming from, you be the judge of whether I understand the foo or not ;-)

A MySQL server on VMWare ESX(i) 3.X or 4.X virtualization

If you virtualize a database server, you are going to find that any given form of a performance hit by using that virtualization technology is greatly impacting the performance of your database server. It is commonly known that one should very, very carefully consider NOT to virtualize the database server on a virtualization technology that meets the aforementioned 4 symptoms. Taking one more step back, I'd say even virtualizing a database server on any given type of virtualization technology should be very, very carefully considered even if the virtualization technology of choice is the latest and greatest and offers 98% performance of bare metal (Guess which virt. tech. I'm talking about here and get a free beer at FUDCon -all you need to do is leave a comment and attend and pick up your beer).

Let me start by saying this database server is the backend store for one of those services that just needs to be snappy and fast and shiny all-over; end-user facing, it's the most signifcant and most important service a business can run internally; E-mail and calendering. It's what tends users to go nuts if unavailable or even not as responsive ans snappy as they would like. If you've ever done Helpdesk Support I bet you know what I mean.

Re: the performance hit; For one, the I/O performance on the hard disk for this one MySQL server that I manage, virtualized using VMWare ESX(i) is around 12 MB/s. My laptop hard disk does ~57 MB/s of buffered write I/O (if I've not used the wrong benchmark by mistake, I'm sure one of you readers can recommend me a util I do get correct numbers with if this seems a little off).

Since it's doing 33k of InnoDB reads per second (on average, so including weekends and outside office hours), you can imagine the peak is over 250k InnoDB reads -which I then imagine cannot be in memory at any given point the database server needs it, causing direct (yet buffered?) I/O to disk.

It's running on x86_64 guest operating system because of the memory footprint of MySQL being over 2GB so that it can cache a lot, but as it seems the VMWare ESX(i) node it runs on only natively supports the i686 CPU instruction set. Enter the binary translation from x86_64 to i686 (that is, if it is not i586), in series, accounting for yet another layer in the stack that increases the performance hit as well as makes the environment more error-prone. Note that the hardware itself is x86_64, and so the virtualization technology does not use all of the hardware capabilities.

Then of course there is the PAE CPU feature needed for x86 to be able to address over 4GB (which shows as 3290MB) of memory. No matter how small the performance hit actually is, it is a performance hit regardless.

I'm done throwing everything I know at the problem of performance in this situation and so here it goes;

Dear lazyweb,

What would you recommend I do (except raise my hands and stop supporting the environment which I've already seriously considered), given this awkward situation? What's left to tweak that you think I might not have tried yet? This MySQL box is going to production soon and I want it's dependent services to be snappier then ever before -something I don't think I'm going to get if the MySQL box remains to be virtualized using the current virtualization technology in use -but I secretly hope you can prove me wrong.