~kanarip

I'm very much pleased to be able to announce that Zarafa, one of the best Linux groupware suites, is now available through the standard Fedora and EPEL repositories.

After Zarafa itself had already announced inclusion to the repositories, news that was dented through major news sites such as heise.de, Robert Scheck and myself sat together at FOSDEM and worked on the packaging until we were both sufficiently satisfied.

It's currently set up to allow the maximum amount of flexibility one could ever wish for. Normally, the packages provided by Zarafa consist of the backend server, gateway (IMAP/POP), the PHP libraries needed for the Webmail interface, and too many other things to really build up a scalable infrastructure without installing all capabilities on all servers in such infrastructure -which introduces it's own world of pain.

Now, apparently, we still need to figure out some things. For one, I get a SIGPIPE / Broken pipe when I run zarafa-server with UNIX passwd authentication. The availability of a platform like Fedora (fast-pace moving forward) allows us however to solve this kind of issues way before Enterprise Linux 6 hits public beta.

You gotta love it!

I've always been a huge fan of Zarafa, one of merely two serious competitors in the Open Source groupware market.

The other competitor is Zimbra, but I have somewhat less of an incentive to sink my teeth into that Java mess, which installs in /opt/zimbra/, and uses it's own vendored libraries rather then those available in the system stack. This, in my opinion, is just wrong, raises cost and gives you less overall options and control. But enough about Zimbra, because obviously the suite *just works* (and a lot of people are very much happy with it).

I have (and still am) running Zarafa at my company for about a thousand users, and Zarafa's headquarters are pretty close to my company's. I get to speak to Zarafa's people regularly, and most of it is while I'm wearing my Fedora hat ;-)

Either way, now that Robert Scheck and myself are attempting to package Zarafa for our dear distribution, Robert and I run into the following when using Fedora 12:

[jmeeuwen@ghandalf SPECS.mine]$ sudo zarafa-spooler -F
zarafa-spooler: symbol lookup error: /usr/lib64/libmapi.so.1: undefined symbol: _ZN10ECMemTable6CreateEP14_SPropTagArrayjPPS_

Robert has created a topic on Zarafa's forum about this some time ago.

Let me first emphasize that Zarafa is upstream for three libraries:

1. inetmapi
2. perlmapi
3. libmapi

This symbol is undefined in libmapi only, as you can see in Robert's comment.

I once succeeded (I don't know how) to make the command result in a stack trace:

[jmeeuwen@ghandalf spooler]$ gdb /usr/bin/zarafa-spooler
GNU gdb (GDB) Fedora (7.0.1-30.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/zarafa-spooler...(no debugging symbols found)...done.
(gdb) run -F
Starting program: /usr/bin/zarafa-spooler -F
[Thread debugging using libthread_db enabled]
Unable to open configuration file /etc/zarafa/spooler.cfg
Continuing with defaults
Wed 03 Feb 2010 03:53:54 AM CET: Unable to open pidfile '/var/run/zarafa-spooler.pid'
Wed 03 Feb 2010 03:53:54 AM CET: Starting zarafa-spooler version 6,30,9,18385 (18385), pid 7285
Wed 03 Feb 2010 03:53:54 AM CET: using SMTP server: localhost

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7fcb820 (LWP 7285)):
#0  0x0000000000000000 in ?? ()
#1  0x0000003d0da20ca9 in M4LMsgServiceAdmin::GetMsgServiceTable(unsigned int, IMAPITable**) () from /usr/lib64/libmapi.so.0
#2  0x0000003d0e259782 in CreateProfileTemp(char*, char*, char*, char*, unsigned int, char*, char*) () from /usr/lib64/libinetmapi.so.1
#3  0x0000003d0e259a3b in HrOpenECSession(IMAPISession**, char*, char*, char*, unsigned int, char*, char*, char*) () from /usr/lib64/libinetmapi.so.1
#4  0x0000003d0e259b29 in HrOpenECAdminSession(IMAPISession**, char*, unsigned int, char*, char*) () from /usr/lib64/libinetmapi.so.1
#5  0x00000000004069a1 in InitThreadData(thread_data*) ()
#6  0x00000000004075bd in ProcessQueue(char*, char*) ()
#7  0x00000000004078ff in running_server(char*, char*) ()
#8  0x000000000040d04e in main ()
(gdb)

I'm not at all too familiar with C/C++ code, and/or libtool (a more recent version of libtool in Fedora is rumored to have caused this?), and so my first step is to Google. Googling for "undefined symbol" doesn't really give you anything else then forum topics with questions and often not even solutions to very particular problems though :/

So, I turn to you, dear Lazyweb, and I'm asking you to help me wrap my head around it and put the finger on the sour spot.

SPEC: http://www.kanarip.com/custom/SPECS/zarafa.spec

SRPM: http://www.kanarip.com/custom/f12/SRPMS/zarafa-6.30.10-1.fc12.src.rpm

Thanks in advance!

I wanted to share with you the way Operator Groep Delft monitors and profiles Zarafa, the best Open Source Groupware available today. We have about a thousand users in LDAP, of which approximately 200 are internal employees (the ones in the branch offices) compared to approximately 800 external users (consultants, part-timers, etc.). We use Nagios for monitoring, and Munin for profiling. Munin integrates with Nagios in that given a set of thresholds, when a profiled resourse is out-of-bounds, it can let Nagios trigger the alerting.

Anyway, let's see some Munin graphs (who doesn't like colored graphs?):

The number of Connections to the Zarafa server

The number of connections to the Zarafa server is a simple `netstat | grep <process-name> | wc -l`. Some users will have more then one connection (Outlook users for example), and so the number of users (see below) is a different number. Also mind that, contrary to the Outlook and/or IMAP (persistent) connections, the webmail connections are polled once per 5 minutes, and thus the number of webmail (non-persistent) connections is probably off by a factor X. We chose to not choose X and instead derive statistics based on the actual numbers in the graph.

You can find our version of the zarafa_connections Munin plugin here.

The number of Unique Source IP addresses

Like I said, the number of actual users is something different then the amount of connections. Some users may have more then one connection so how do we derive the number of users from the number of connections? While Zarafa comes with a neat utility called zarafa-stats, it is not what we were looking for and so we chose to pick unique IP addresses for Outlook, IMAP and webmail users, and pick unique users for ActiveSync connections (also through the webserver, like webmail access, so we parse the webserver logs here). There's one more cheat in the number of webmail and ActiveSync connections I need to tell you about:

Instead of examining the webserver access_log files per hour, we take into account the hour before the current hour too. In a way, we examine the access_log files (of combined format by the way) in a range of 60 to 120 minutes. The reasoning is as follows; If we don't, at the start of every hour the number of users is zero. Then, as the hour passes, the number of users would increase and increase a little more until the hour passes and the next hour starts. This would have caused the graph to look like the teeth of a saw, which would misrepresent the number of users entirely.

You can find our version of the zarafa_users Munin plugin here.

The number of Queries per Second on the dedicated MySQL Database server

These users cause Zarafa to put some load on your database server, of course, and since we have a dedicated MySQL server just for Zarafa, the number of queries per second is interesting as well. This is a standard Munin plugin -or at least it's shipped in the version of Munin for Extra Packages for Enterprise Linux 5 as well as Fedora.

Note how the number of queries per second turns out to have > 50% cache hits at average (for the week). Over a longer period of time, I can tell you, the number of cache hits averages about 50% overall, and anywhere between about 30-40% during office hours.