~kanarip

In terms of an expirement, I've been rebuilding Enterprise Linux packages, including updates, including Extra Packages for Enterprise Linux, in order to learn from it and take away a couple of notes on the subject; Here's a brief overview of what I've found so far;

• Package build requirements may be off
  Maybe some package required foo-1.0 in the buildroot, but foo has been updated to foo-1.1, obsoleting the foo-1.0 update as it is pushed to the updates channel.

  The most obvious case in a name-change of python-setuptools-devel, for which the -devel part became part of the python-setuptools package itself, without providing "python-setuptools-devel". All of the dependent packages will now fail to build because of this change. In terms of Enterprise Linux, this type of change causes everything in it's realm to no longer be reproducible, which is a key aspect of Enterprise Linux to begin with, as no reproducibility means no certainty about what's going to end up on the consumer's system.

• Koji does not allow all builds of all packages in its static repositories by default
  By default, Koji will only take the latest successful build in a tag in its static repositories. While perfectly fine in the case of FTBFS exercises, it bubbles down the inheritance all the way to the point where you don't need the latest package, but the former version. FWIW, "latest" isn't defined as "highest package NEVRA" either, it's "last build completion time". I would rather make it a configuration setting at some point, but I'll share the patch I made anyway.
• BuildNotifications are at the end of the queue
  When building a lot of packages, all of the build tasks have a higher priority then the notification tasks that let people know what the result of a build was. Initially, I thought something was wrong with my environment, but with the help of Dennis Gilmore over in #koji on FreeNode, I learned that it is in fact the priority in Koji, not my environment.

  Again I would love to make this a configurable thing some day, but here's the patch I made to let buildNotification tasks have a higher priority then anything else.

• "%patch without Patch"
  This is the most amazing sort of problem. RPM spec files will have a conditional %if statement that in a simple `rpmbuild -bs` will cause the Patch file to not be included. A very simple fix would be to never ever use the conditional inclusion of a Patch file when building the source RPM package, but instead use the conditional during the actual build.
• (No) Naming Convention
  The naming convention, or lack thereof, makes it so one needs a `yum whatprovides` or `repoquery whatprovides` to resolve build requirements. A naming convention could be to say, that if a package includes libuuid.so.*, a (meta-)package libuuid is (automatically) created, or to make yum so intelligent as to resolve a "libuuid" type of build requirement to the actual package, not unlike it does for binary packages when you use `yum install` and the package in question doesn't explicitely require libuuid.so through the original RPM spec file, but RPM inserted the library dependency automatically.
• The (lack of) use of dist-tags
  Some packages do, while others do not, include a dist-tag. For downstreams like myself, this causes the downstream to only be able to build the packages in question only once, since the N-V-R needs to be unique all across the build system. In certain circumstances, it is entirely justifiable to rebuild a large subset of packages included in the upstream distribution as well as the add-on repository; for one, being able to run Ruby 1.9.1 with a large part of its Gems causes one to re-negotiate the use of dist-tags:

1. On the end-user system, rubygem-foo for Ruby 1.9.1 needs to be distinguishible from rubygem-foo for Ruby 1.8.6. And that's just the user side of things. Failure to make it distinguishible in some or the other fashion causes a significant load on the support side of things.
2. Throughout the build- and distribution-channels, whether they be repositories or not, rubygem-foo-1.0 for Ruby 1.8.6 *must* be smaller in terms of package NEVRA then, say, rubygem-foo-1.0 for Ruby 1.9.1, in order for the Ruby 1.9.1 fast-track to be viable on any end-user system.
3. Since not using dist-tags would allow rubygem-foo-1.0-1.src.rpm (or whathaveyou) to be built only once, this cannot be a package that has any kind of binary/library linking going on with Ruby itself. In order for the package to be allowed to be rebuilt more then once, against different stacks, a dist-tag must be introduced.

• buildsys-macros is not available through the distribution
  It appears as if the buildsys-macros package is not available through the EL or EPEL repositories. Of course, this is a build system thing, and would not be used on the end-user system. For a downstream buildsystem though, having this package be part of the repository might save someone some headache.
• There's no build group
  When setting up a downstream build system environment, you use groups (or a buildroot initialization command actually), which will initially attempt to use "build" as the group name. However, EPEL only includes a buildsys-build group (of which buildsys-macros isn't part of course). It might be a little easier for downstream build environments to (re-)use an existing build group as to not modify their local koji foo all too much.
• It takes a lot of CPU cycles
  Plan for an enormous amount of CPU cycles to be spent at whatever you're building. In my particular case, a great deal of the cycles are I/O wait ;-) Maybe I should plan for better performance storage across my builders, but for now I'll have to work with what I have available. As a not-so-minor side-note, if you plan on just building a few packages every once in a while, as opposed to continuously rebuilding a lot of packages, this is a less significant parameter.
• When using external repositories...
  If you use external repositories, plan to have some memory available. Every once in a while, the `mergerepos` command executed by Koji consumes more memory then Thunderbird does at that moment, which should tell you enough.

So far so good, while you may be wondering while I'm doing this type of stuff. Well, it's a very interesting and challenging area, which makes it fun to do. Besides, it allows me to play around with different tasks I was going to try and execute, like providing some of the packages that cannot be in Fedora (mod_passenger), or cannot go into EPEL (rails3, ruby-1.9.1, you name it). I would like to return a certain amount of option value into the hands of Linux consumers, but in an efficient manner as opposed to everyone who needs certain foo doing it themselves in a million different ways (== unsustainable). That's what I'm working on, or at least some of the details concerning such.

I'm heavily experimenting with Drupal. I have 95 or so modules I want to experiment with, which is just too cool! The more modules, the more obvious it is that this thing might actually turn out to do exactly what you wish ;-)

For now, I'm looking for a good case tracker, issue tracker or you-name-it. Preferably one that integrates with organic groups, and/or user roles. It seems "Project" and "Project Issues" module are unsuitable after some initial testing -I'm not quite done with them yet, though. "Project" seems to be very specifically aimed at software projects with releases, whereas I just need a ticketing system for tasks, basically. I've done this before with "Support", so maybe that's what I'll end up using.

Just for fun, I started building ruby-1.8.6 and ruby-1.9.1 packages for Enterprise Linux 5. These would be opt-in repositories, "channels" if you will, fast-tracking an Enterprise Linux 5 box to newer versions of Ruby, and many of the packages that depend on Ruby one way or the other (such as Ruby on Rails packages, and many Ruby Gems).

I started out with the Fedora packages, obviously, and after getting a bunch of packages in said repositories, and testing and patching and rebuilding a bunch of times, it became obvious;

For the type of situation where you want two of these "fast-track" repositories, or even just multiple versions of the same packages built under different conditions, it turns out to be mandatory that a Koji build of a certain package contains a globally unique package NEVRA (Name, Epoch, Version, Release, Architecture), so they can be distinguished between the two (very much different) versions of the package. That is to say, one package NEVRA can only be built once, and can only be duplicated to other destination tags.

Example: The ruby-shadow package (a requirement for Puppet, which makes it very important to me) is binary compiled, either against ruby-1.8.6 or ruby-1.9.1. For one version (e.g. upstream's 0.9.7) to be available through both repositories, one builds ruby-shadow-0.9.7-1.el5.src.rpm.

I build the package against two Koji build targets, each one using a build tag that causes a buildroot to be created with either ruby-1.8.6 or ruby-1.9.1; in my case these build targets are feature-el5-ruby-1.8.6 and feature-el5-ruby-1.9.1.

Koji however will only allow one specific package NEVRA to be built just once. But, ruby-shadow-0.9.7-1.el5.src.rpm has to be built twice; once for ruby-1.8.6, and once for ruby-1.9.1. Since one build with the same NEVRA already exists, another cannot be built.

Ergo, you need some kind of indication of the build-root/build-tag/destionation-tag in the package NEVRA...

And we go off rebuilding everything again... ;-)

Note to self: when using external repositories and building on those, please remember that priorities in tag inheritance does matter.

Thank you.

If you have level 1 Linux Professional Institute certification, you can get Certified Linux Administrator from Novell for free:

1. Go to the LPI login-page and login with your LPI ID or email address.
2. Click on "Verify Certification" and then "certifications" near the bottom of the page.
3. Copy your LPI ID and the Verification Code for LPIC-1
4. Go to Novell Certified Linux Administrator Certification Registration Form
5. Fill out the form and press "Continue".
6. Congratulations!
7. Ask your employer for that raise he promised you in case you would obtain (yet another) certification.

Have fun!

Today is the day we come together with a bunch of Linux experts, and start knocking down some of the items on our TODO list, as well as -hopefully- share more information and responsibility on the overall Linux infrastructure inside our company, and the Linux infrastructure at a lot of out customers both!

On February 9th, Mike McGrath wrote:

Toshio Kuratomi is awesome.

That is all.

+1 Mike, and so are you!

I'm very much pleased to be able to announce that Zarafa, one of the best Linux groupware suites, is now available through the standard Fedora and EPEL repositories.

After Zarafa itself had already announced inclusion to the repositories, news that was dented through major news sites such as heise.de, Robert Scheck and myself sat together at FOSDEM and worked on the packaging until we were both sufficiently satisfied.

It's currently set up to allow the maximum amount of flexibility one could ever wish for. Normally, the packages provided by Zarafa consist of the backend server, gateway (IMAP/POP), the PHP libraries needed for the Webmail interface, and too many other things to really build up a scalable infrastructure without installing all capabilities on all servers in such infrastructure -which introduces it's own world of pain.

Now, apparently, we still need to figure out some things. For one, I get a SIGPIPE / Broken pipe when I run zarafa-server with UNIX passwd authentication. The availability of a platform like Fedora (fast-pace moving forward) allows us however to solve this kind of issues way before Enterprise Linux 6 hits public beta.

You gotta love it!

I've always been a huge fan of Zarafa, one of merely two serious competitors in the Open Source groupware market.

The other competitor is Zimbra, but I have somewhat less of an incentive to sink my teeth into that Java mess, which installs in /opt/zimbra/, and uses it's own vendored libraries rather then those available in the system stack. This, in my opinion, is just wrong, raises cost and gives you less overall options and control. But enough about Zimbra, because obviously the suite *just works* (and a lot of people are very much happy with it).

I have (and still am) running Zarafa at my company for about a thousand users, and Zarafa's headquarters are pretty close to my company's. I get to speak to Zarafa's people regularly, and most of it is while I'm wearing my Fedora hat ;-)

Either way, now that Robert Scheck and myself are attempting to package Zarafa for our dear distribution, Robert and I run into the following when using Fedora 12:

[jmeeuwen@ghandalf SPECS.mine]$ sudo zarafa-spooler -F
zarafa-spooler: symbol lookup error: /usr/lib64/libmapi.so.1: undefined symbol: _ZN10ECMemTable6CreateEP14_SPropTagArrayjPPS_

Robert has created a topic on Zarafa's forum about this some time ago.

Let me first emphasize that Zarafa is upstream for three libraries:

1. inetmapi
2. perlmapi
3. libmapi

This symbol is undefined in libmapi only, as you can see in Robert's comment.

I once succeeded (I don't know how) to make the command result in a stack trace:

[jmeeuwen@ghandalf spooler]$ gdb /usr/bin/zarafa-spooler
GNU gdb (GDB) Fedora (7.0.1-30.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/zarafa-spooler...(no debugging symbols found)...done.
(gdb) run -F
Starting program: /usr/bin/zarafa-spooler -F
[Thread debugging using libthread_db enabled]
Unable to open configuration file /etc/zarafa/spooler.cfg
Continuing with defaults
Wed 03 Feb 2010 03:53:54 AM CET: Unable to open pidfile '/var/run/zarafa-spooler.pid'
Wed 03 Feb 2010 03:53:54 AM CET: Starting zarafa-spooler version 6,30,9,18385 (18385), pid 7285
Wed 03 Feb 2010 03:53:54 AM CET: using SMTP server: localhost

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7fcb820 (LWP 7285)):
#0  0x0000000000000000 in ?? ()
#1  0x0000003d0da20ca9 in M4LMsgServiceAdmin::GetMsgServiceTable(unsigned int, IMAPITable**) () from /usr/lib64/libmapi.so.0
#2  0x0000003d0e259782 in CreateProfileTemp(char*, char*, char*, char*, unsigned int, char*, char*) () from /usr/lib64/libinetmapi.so.1
#3  0x0000003d0e259a3b in HrOpenECSession(IMAPISession**, char*, char*, char*, unsigned int, char*, char*, char*) () from /usr/lib64/libinetmapi.so.1
#4  0x0000003d0e259b29 in HrOpenECAdminSession(IMAPISession**, char*, unsigned int, char*, char*) () from /usr/lib64/libinetmapi.so.1
#5  0x00000000004069a1 in InitThreadData(thread_data*) ()
#6  0x00000000004075bd in ProcessQueue(char*, char*) ()
#7  0x00000000004078ff in running_server(char*, char*) ()
#8  0x000000000040d04e in main ()
(gdb)

I'm not at all too familiar with C/C++ code, and/or libtool (a more recent version of libtool in Fedora is rumored to have caused this?), and so my first step is to Google. Googling for "undefined symbol" doesn't really give you anything else then forum topics with questions and often not even solutions to very particular problems though :/

So, I turn to you, dear Lazyweb, and I'm asking you to help me wrap my head around it and put the finger on the sour spot.

SPEC: http://www.kanarip.com/custom/SPECS/zarafa.spec

SRPM: http://www.kanarip.com/custom/f12/SRPMS/zarafa-6.30.10-1.fc12.src.rpm

Thanks in advance!

I love to be able to announce the first "sysadmin-main" meeting within my company, Operator Groep Delft. You read that right, that's the exact same name the Fedora Infrastructure team - an example to us all as far as I'm concerned, uses to indicate the group of people ultimately in control of all systems and services. Read the more detailed description. I'm going to try and apply this very concept to my company's internal system administration.

Instead of taking care of the Linux infrastructure with no dedicated Linux engineering resources, we decided that utilizing all of the available Linux competencies within the company would be more efficient, more effective, more flexible and would vastly improve redundancy, collaboration and expertise between the bunch of us.

Since we're a consultancy company, most of our Linux Engineers are hired out to customers, making billable hours, earning us all money, and are thus not in a position to really sit down and take care of OGDs Linux infrastructure as well. Basically our engineers can only spend that one minute they have left at the end of a day or the hour they would otherwise waste on one of those boring meetings -they merely require the facilities to do so.

So instead we decided to try and make it a community effort of some sort, in the sense that all sysadmin-main engineers require a minimal Linux certification level of RHCT, and will have access to Life, Everything and The Universe once we've sat together and introduced them to all procedures and such.

Now, nothing is set in stone yet, but at least we're going to have a bunch of interested people show up at our meeting soon, which will hopefully lead to a large group of engineers willing to do interesting stuff.

We'd plan and assign -amongst ourselves- the tasks in our queue. If either of us needs a little more time, we save up to 8 hours a week to spend at more intrusive changes like migrations, upgrades, planning, documentation, build & test, development, that sort of thing. We (as a group) would decide what we do, how we do it, and who actually gets to execute.

Ultimately, I'd be (held) responsible (or accountable?) for the group, and my manager would be responsible for the well-being of the infrastucture as a whole, just like he is now.

Just to give you an impression of what it takes to do what we do between the two of us at this moment, in the very little time that we have available; Zarafa, HA/LB Red Hat Directory Services, Puppet (with help of puppetmanaged.org modules), Cobbler, SELinux (enforcing), Nagios, Munin (looking at Zenoss to replace both), 15 mod_security enabled webservers, 4 database servers, 6 development environment staging boxes, and a couple of workstations.

PS. For those of you who read this, and are colleagues of mine, you can find more information on https://nix-noc.ogd.nl/trac/

If I'm not mistaken, today is the birthday of not just one, but two great friends!

Congratulations both Max Spevack and Jan Wildeboer!

Overheard:

Question: "You apply a Server, Domain Controller and Client security policy requiring all network traffic to be encrypted. Some of your users report that they cannot log in or access network resources. What is the easiest way to resolve the problem?"

Answer: "Tell the user to reboot the computer."

This means, essentially, implicitly, that the policy that *all* network traffic should be encrypted, doesn't apply to *all* traffic ('cause clients can get their updated configuration without being compliant to the new configuration). *sigh* :/

Over the last couple of days it seems more and more people are stepping up to get Ruby 1.9.1 in Fedora, along with the packaging changes and all that. From a new list just two months ago, we now have 17 members working on the same problems collaboratively.

Worth noting is that I'm receiving patches from people that understand Ruby way better then I do, so it's one big happy learning experience for me too ;-) (Noted I do not use Ruby myself, nor do I program using Ruby)

Thanks to Ben Shakal, we're now over the issue I posted about earlier on the Ruby SIG mailing list, where gem install would work, but executing or requiring the gem wouldn't. Great work Ben, thanks!

From here on out, I'm going to extend the Ruby repositories for Fedora 12 and Fedora Rawhide to include some of the ruby gems with the new packaging guidelines. I'm still going to need new packaging guidelines to go along with the new packages so I'm probably going to build those packaging guidelines as I go along.

More news on Ruby later this year, hope to have some working stack early next year.

Hope you all have a merry Christmas!

I've finally recovered some of what broke down earlier this week. Long story short:

• The CPU of my web-/file-/mail-/dns- server is fubar, and the only other Pentium 4 CPU that I could find would not work
• I used the x86_64 CPU and mainboard from slevin (my girlfriend's desktop PC)
• Pulling out and stuffing back in the hardware wasn't difficult, but took me some time anyway (I did not have the right tools, go figure)
• Upgrading -slash- migrating Fedora 10 i386 to Fedora 12 x86_64 as you can imagine took me even longer, only to find out that a lot of configuration and other stuff relevant to various services were incompatible.
• The first thing I wanted to restore was email, and so I did (including external DNS), and then I left it all behind to do some other stuff.
• Now I've gotten around to rebuilding php-5.2 for Fedora 12 so that Drupal would actually work. Various kinds of functions Drupal-6.x uses are deprecated in php-5.3, and
• I had to restore name services internal to my network as well (the Pentium 4 CPU I got from this other machine made this other machine not boot up anymore)

The fun! ;-)

Dear Hyves,

Would you please stop sending me email that does not make sense?

I quote:

Hyves notification

Unfortunately you can't read this mail properly, due to the fact that our outgoing mail is
html-formatted. Your mail program probably only support plain texts.

There's a couple of things wrong with this email;

1. I can read it properly, it's just not the same contents you expect me to read when I open this email.
   You expect me to use a mail client that parses HTML right-away, without me choosing to do so when I want to. I'm sure your engineers will be familiar with the "why HTML mail is bad" meme that has been going on since the '90s.
2. It's not up to you to say anything about my ability to read anything.
3. You say my mail program probably only supports plain text, but that surely does not take into account the fact that I may have chosen to not read HTML emails.

You could, of course, include the actual message you're trying to send in plain text as well, rather then this weird looking (static) message.

I was at a round table session a few weeks ago, where consultants and advisors to local governments in the Netherlands (there's around 400 of them, and then some) discussed "Nederland Open in Verbinding", an initiative by state-secretary Frank Heemskerk with the Ministry of Economic Affairs to get government agencies of all kinds to realize the value of using Open Standards, and inherently (but not primarily) appreciate the value of Open Source software (ergo Free Software, because the rest is Crippleware).

It was the first time for me to talk on such a level, very non-technical, hypothetical and on the policy side of things. Normally, I work on design, development, implementation, administration, innovation, what-have-you, but all from the technical perspective.

Let's finally burn down the list of notes I've taken from that session.

Like within any other organisation, the main factor that influenced the atmosphere during our discussion was money. Not value. No sir! Cold hard cash, of course.

Apparently, or so I was told, government agencies wonder what the TCO on a thing such as OpenOffice.org would look like.

While I'm sure there's plenty of people out there that can whip up a comparison sheet from the top of their heads (and so I told them), that's not the issue I take offense with.

The actual issue I do take offense with is that while asking for a TCO on OpenOffice.org, it seemed that none asking for such a TCO analysis had ever done the exact same thing for either alternative product; most prominently Microsoft Office '97/2000/2003, or even a pending migration to either Microsoft Office 2007 or 2010. If you don't know what the costs were, why are you asking for comparison to what the costs might be should you use the product at zero licensing price? The next big money burning factor was considered to be migration costs.

Going down the list of examples that crossed the table; The next issue was migration off of, say, Microsoft Office (regardless what version), to OpenOffice.org. This one made me laught out loud, since such migrations are part of every single Microsoft Office release you upgrade to, with -in the case of Microsoft Office at least- the added danger of actually 1) being forced to upgrade, and 2) losing data in ways that cause you to never be able to read it again.

I emphasized that regardless of the piece of software used (you write your own if you will), the most important thing was the use of Open Standards, and every single governments agency's choice to either "use or explain". "Use Open Standards or explain why you can't or won't", is what the federal government initiative is trying to tell the other government bodies.

Either way, the choice to use Open Standards or Open Source Software is up to each individual government body. While there is no actual enforcement of the "use or explain" guidelines, the motivation to make a well-founded decision based on facts is moot. The way this is going down now merely creates awareness about the existence of an alternative technology, whether a piece of software or an open standard, you could use. The well-foundedness of the decision is quickly annihilated using false rumors and assumptions, though, and there's noone out there to tell them about the facts.

Moreover, and this is where I was truly shocked, the overall consensus seemed to be that Open Standards amongst themselves are difficult to implement and be compatible. For one, someone said, there's more then one version of the Open Document Format. This poses a problem, since Microsoft Office 2007 SP2 only implements ODF 1.1 (and so will Microsoft Office 2010 from what I can see in different preview builds), and so maybe could not deal with ODF 1.2. Not realizing that this is actually a Microsoft Office problem, I wonder if this is how it is going down;

Customer: Yeah, so, give us your offering.

SupplierA: Sure. Here's costs, they're easy. Remember that when you pay less, you get less, huh? There's also benefits, but only for Microsoft Office, primarily because we make it so extremely easy for you to get back at us and give us more of your money. In return, we'll only implement the one true version of this Open Standard, because you know multiple versions of an Open Standard are just going to confuse you, right? So, well, here's the contract, sign at the bottom please.

SupplierB: Our version of the product works wonderfully well with Open Standards, it is free of charge and you only start paying us when our phone rings. You can sign this contract, or look around for a better value proposition. On top of that, we are compatible with all versions of Open Standards and you'll never notice a thing. No transition, no migration, no hassle. If you're dissatisfied, there's no clause stopping you from getting what you get from us somewhere else.

Customer: So, SupplierB, are you saying I can take your product and use it at a zero price license? That all my problems go away? How much trouble am I going to be in and what solving those problems going to cost me? That I can just upgrade the software without a defined project involving migrations and consumer education? That I can just leave and go to your competitor? You must be freaking kidding me. SupplierA, you have a pen?

Long story short, the consultants and advisors to government bodies and other government agencies and organisations are in a position to make sure a conscious choice is being made, by matter of policy. They are in a position to emphasize the true facts and false rumors, and create some incentive to at least consider a particular implementation over another, despite the initial cost -if any more significant, in favour of the long term value. Because that's what this is about finally. Money. The tax payer's money, no less.

You may have seen Lydia's blogpost on FUDCon Toronto as she experienced it. It's the first time she's been at an event like this, while it is my trazillionth time. So, she was interested to see what kind of work we do at such events, what kind of unmess the unconference is like (I've told her many stories about Greg DeKoenigsberg throwing the markers on the floor in Brno, for example), and of course she misses Max Spevack ;-)

All and all, from her story, it may have appeared to you that I'm more of an Ambassador out of the house then I am at home ;-) Hope you all appreciate her $0.02

So now that I have this new netbook thingy, kinda like a key-chain accessory, I want the packages I have installed on my huge-ass need-a-truck laptop to be on the netbook as well. I could, of course, do a rpm -qa on one end and install the packages listed on the other end, but that's not very sustainable. Here's what I did:

1. Let Puppet collect the packages on node ghandalf.kanarip.com
2. Let Puppet install all of those packages on weee.kanarip.com

Like the snail said, riding on the back of the turtle: Weeee!

With a little help from my friends (Andreas Thiennemann in particular), I managed to get my hands on a piece of equipment I was planning on buying anyway; a netbook.

Here in Canada though, these things are particularly cheap.

Next on the list of things to do is:

1. Never boot anything but a superior version of the greatest Operating System on a piece of equipment as cool as a netbook. The first thing that Acer will attempt to boot is a Windows Vista based eRecovery Suite in order for you to install Windows XP Home. I rest my case.
2. Download Fedora 12.
3. Check the list of things you need to know before trying all kinds of things to make the foo work. In my case, I found this page very useful. For one, it helped me boot Fedora ;-)

I finally managed to come up with a bunch of proper patches that give us Ruby 1.9.1 packages again, the way we want them to be after our little HackFest at FUDCon in Toronto. At first, they would succeed in rpmbuild, but not in mock or koji, but after all I got them to build in mock. Assuming they will then build in koji too, I'm submitting a couple of scratch builds now in the background as the Internet in the Hotel isn't all that fast, and after the builds have finished we may have some packages to play around with ;-)

Only a handful people seemed interested in a FUDCon session on Ruby, Ruby-1.9.1, the Enterprise Edition, packaging foo and so forth, so it's not part of the BarCamp ;(

I'm planning to have the session tomorrow though, just after lunch, in one or the other room, so if you are interested please follow around the guy with the bad haircut wairing a Von Dutch hooded sweater ;-)

FUDCon starts! ;-)

Weeeee! Release party is ongoing!

It's more of a socializing thing then it is about Fedora 12, really, but what the heck ;-)

Also, we're speaking English rather then Dutch ;-)

I'm having trouble creating a Re-Spin for Fedora 11, as you might have suspected given the number of Re-Spins released by Fedora Unity for this number 11 release.

This time, it is networking in the installer image that isn't at all functional. I suspect changes to NetworkManager, released as an update for Fedora 11, have caused the dependencies to shift, and those dependencies might not be in the installer image (install.img) nor the initrd.img.

To troubleshoot such things is rather difficult, and takes up a lot of time. Maybe I'm just being inefficient at it (any hints can go to kanarip@fedoraunity.org please!).

Anyway, I just wanted to let you know that that's what causing the hold-up.

I wanted to share with you the way Operator Groep Delft monitors and profiles Zarafa, the best Open Source Groupware available today. We have about a thousand users in LDAP, of which approximately 200 are internal employees (the ones in the branch offices) compared to approximately 800 external users (consultants, part-timers, etc.). We use Nagios for monitoring, and Munin for profiling. Munin integrates with Nagios in that given a set of thresholds, when a profiled resourse is out-of-bounds, it can let Nagios trigger the alerting.

Anyway, let's see some Munin graphs (who doesn't like colored graphs?):

The number of Connections to the Zarafa server

The number of connections to the Zarafa server is a simple `netstat | grep <process-name> | wc -l`. Some users will have more then one connection (Outlook users for example), and so the number of users (see below) is a different number. Also mind that, contrary to the Outlook and/or IMAP (persistent) connections, the webmail connections are polled once per 5 minutes, and thus the number of webmail (non-persistent) connections is probably off by a factor X. We chose to not choose X and instead derive statistics based on the actual numbers in the graph.

You can find our version of the zarafa_connections Munin plugin here.

The number of Unique Source IP addresses

Like I said, the number of actual users is something different then the amount of connections. Some users may have more then one connection so how do we derive the number of users from the number of connections? While Zarafa comes with a neat utility called zarafa-stats, it is not what we were looking for and so we chose to pick unique IP addresses for Outlook, IMAP and webmail users, and pick unique users for ActiveSync connections (also through the webserver, like webmail access, so we parse the webserver logs here). There's one more cheat in the number of webmail and ActiveSync connections I need to tell you about:

Instead of examining the webserver access_log files per hour, we take into account the hour before the current hour too. In a way, we examine the access_log files (of combined format by the way) in a range of 60 to 120 minutes. The reasoning is as follows; If we don't, at the start of every hour the number of users is zero. Then, as the hour passes, the number of users would increase and increase a little more until the hour passes and the next hour starts. This would have caused the graph to look like the teeth of a saw, which would misrepresent the number of users entirely.

You can find our version of the zarafa_users Munin plugin here.

The number of Queries per Second on the dedicated MySQL Database server

These users cause Zarafa to put some load on your database server, of course, and since we have a dedicated MySQL server just for Zarafa, the number of queries per second is interesting as well. This is a standard Munin plugin -or at least it's shipped in the version of Munin for Extra Packages for Enterprise Linux 5 as well as Fedora.

Note how the number of queries per second turns out to have > 50% cache hits at average (for the week). Over a longer period of time, I can tell you, the number of cache hits averages about 50% overall, and anywhere between about 30-40% during office hours.

I can't seem to find a nice, simple, stupid web administration interface for LDAP. I want the users of said web interface to just be able to click "Add User" and fill out givenName, sn. I'll derive from that a uidNumber, uid, homeDirectory, mail, generate a password, send out an email. Or something like that.  Same goes for groups "Add User to Group", or "Remove User from Group". All based on authorization levels or ACLs in LDAP, of course. Reliably enforcing a naming convention and all kinds of standard operating procedures.

So, I started my first Ruby on Rails project, and I called it SLAP (Simple LDAP Administration Panel). Since it's my first Ruby on Rails project, I'm trying to get used to the semantics that is Rails... Enabling the brand new Rails project to simply authenticate and authorize through LDAP for example is being a PITA. I'm looking at ActiveLdap, but I can't seem to figure it out -it is very scarcely documented.

Apparently people follow this tutorial to install PHP 5.2.x on their Red Hat Enterprise Linux systems. It uses the repositories from our dear Remi Collet, and at first I didn't understand why anyone would do that, install an unsupported version of PHP on a platform you specifically chose to get support for. I was under the impression Red Hat themselves would have a decent offer in the Application Stack channel, but what I found out is that a subscription to the Red Hat Application Stack v2 would cost around $1.999 a year (source). Go figure ;-)

It feels like this is part 7 or 8 in the ruby series, but I hope you're still hanging in there ;-)

Last time I blogged about Ruby 1.9.1 search paths, but after some converstation with Stephan Kasal in private, and some more on the Ruby SIG mailing list, I'm leaning towards the simplest ever search path;

/usr/local/lib{,64}/ruby
/usr/local/share/ruby
/usr/lib{,64}/ruby
/usr/share/ruby

As I also mentioned in this message, this looks very clear and straightforward to me! Regrettably, I have not yet had much response on the mailing list. I hope to be doing some rebuilds of packages depending on ruby later on, see how that goes, and maybe release a 1.9.2 release candidate version somewhere.

Our services (primarily our web services and puppetmaster services) have recovered from unexpected interruption.