Job

I’m working full time as an Open Source/Linux consultant these days – and after work I do not really have the time nor the energy to invest even more time into Open Source (besides the Fedora packaging).

Company’s Blog

I was successful in convincing enough people in our company to start a blog – and I blog there since then, so when I get home I usually already have blogged about whatever comes to my mind.

That means effectively that you will not receive any more new posts here. It hurts my heart and kills kittens, but you can remove the blogfeed. @planets where I might still be listed at: please remove this blog feed as well.

However, if you *do* want to keep up with my thoughts: credativ’s company blog is working quite nice these days. Btw., in case you don’t know, credativ is an Open Source/Linux company and the one behind the Open Source Support Center (OSSC) and the Open Source Support Card (yeah, “catchy” names, I know). They are focussed on Open Source support (Linux-Support, PostgreSQL-Support, etc.) and have offices in DE, UK, US, etc. So the general topics are pretty close to this blog. If you look close you will recognize my style: short italic introduction, eye catcher on the upper right side, special headline markings for Howtos and Short Tips, and so on. Also, the categories are quite the same, and it is actually available in German and English. Also, I am not the only person writing there – one very active PostgreSQL developer keeps blogging there, if I want it or not.

However – it is a company blog, so you will (!) find information regarding the company itself, or newest marketing things. You are warned!

So this is it: the last post. Thanks everyone for wonderful years full of blogging, discussions, news, Howtos and good tips. So long, and thanks for the fish!

It was, however, also a big reason why many developers didn’t choose it: some companies wanted to develop closed source without paying this licence free, and other Free Software projects argued that GPL is not free enough for their taste. These reasons are not valid any more, and with Nokia supporting Qt the question arises how this will influence the software world.

Of course the world won’t change it’s face over night, but Qt had always a strong support, and every company which decided to go cross platform at least had a closer look at Qt. Now, with the new license and also combined with the new IDE Qt there are even more reasons to choose Qt for cross platform software development. And since it is a powerful toolkit anyway it might also happen that more and more Windows developers choose Qt – which would make it easier to ask them t develop for Linux as well – and/or for KDE.

$ ls *.ics | xargs -n 1 konsolekalendar --file ~/.kde/share/apps/korganizer/std.ics --import

Another way is to copy all ics files to a directory. A new calendar can be created with this new directory which afterwards contains all the data from the ics files. This calendar can again be exported as one single ics files which can be easily imported everywhere.

Source #1: Fedora

One of the best statistical sources regarding Linux usage are the Fedora statistics. There the number of downloaded images as well as the number of unique IPs getting software updates is counted.
The data are difficult to interpret: no one knows if a downloaded image was only used to test the new system or to bun it onto a CD and distribute it to thousands of magazine readers or thousands of company computers. The second number is problematic because one new IP can mean a big NAT network or just a dial-in user who re-connected. So flaws everywhere, but it is a interesting coincidence that the IP numbers and the downloads are rather close.

There are also the smolt data. It tracks the users who opted in to a tracking system. Currently the smolt web server seems to be lacking behind. But there are current data available for older Fedora releases: these informations say that every month Fedora still gets more than 10k new Fedora 7 users – although there are already Fedora 8 and Fedora 9 available.

So the question is how to read all the data. One way was recently suggested by Paul Frields, Fedora’s project leader: he sums up the data to be around 11.5 million. Together with currently 2.5 million Red Hat subscriptions this would result in 13 million users. Focussing on Fedora alone and leaving the Fedora 6 users Fedora would still have a user base of 9.5 million users.

Source #2: relative statistics

There are hardly any other trustfully data from other Linux distributions available. Therefore, the Fedora number does say a lot about Fedora, but not that much about Linux in general. However, there are other statistics which measure the relative acceptance of Linux distributions.

One such source is the 2007 Linux Desktop Survey done by DesktopLinux.com. There the relative importance of Fedora/Red Hat is measured with 9%. Unfortunately there is no more recent survey available. I wonder why no one has picked up such a survey in 2008. Maybe I should start one on my own?

The result

Given that Fedora/Red Hat has roughly 10% and also roughly 10 million users together (which in fact seems like a at least slightly realistic data base, given the facts), the total number of Linux users world wide would sum up to 100 million Linux users. Nice.

That would leave Mac OS far behind, which is however not that surprising: Mac OS is hardly used in Offices or the government outside the US, and it is far easier to give Linux a try and keep it as a dual boot option besides a Windows installation. Also, the EU governments are pushing Linux quite a lot, and many companies and governments indeed switch to Linux right now or already switched over in the client space for some of the day-to-day workstations.

Still, last year I played the “numbers game” already (unfortunately with the same relativity source, btw.) and the result said something about 20 to 30 million users. I doubt that the number of Linux users spiked that much in the last year, but think that we can safely say the number of Linux users world wide is somewhere in the middle two digit million area, somewhere around 50 million installations worldwide.

Keep in mind that this counts mainly workstations – not traffic lights, shop information terminals or any other specialized hardware. Including all these devices would result in much, much larger numbers.

The problem: discrepancy

Most numbers available guessing the number of Linux users world wide say that there are not that many Linux installations out there, not at all. Most often it is said that, in percentage, Apple has a low one-digit number, while Linux has a dot before its first non-zero number.
The statistical backup for such numbers is most often created by browser strings aggregated from Web pages. This procedure has the flaw that these strings are often faked to make it easier to access specific pages. Also, the monitored pages are only a subset of the entire web, and surprisingly often focus on the US only which is not representative for the world at any rate.

Still, I do often wonder why such numbers and my estimates are different in the order of magnitudes. I welcome any comment on that issue!

Btrfs

With ext4 a new file just left the developer corner. However, ext4 is an old-style file system and does not offer “hot” features like on-line snap shots, versioning and so on. ZFS does, but it is not an option for Linux due to licence reasons. Here comes Btrfs into play: it is in development for quite some time now and many Kernel developers already asked to include Btrfs in the Kernel to speed up the development process. Additionally, several Kernel developers already mentioned that they expect Btrfs to be the next generation default file system for Linux in the mid-term.

In 2009 Btrfs will most likely stabilize its file system format and publish a beta version for testing purposes.

oVirt

oVirt is a small host image that provides libvirt services and hosts virtual machines. Additionally it also has a well designed web based management system. The aim is to provide an enterprise ready VM management console capable of managing large sever clusters hosting large numbers of virtual machines, but is also supposed for single users.

In 2009 oVirt will hopefully see its first beta release ready for first real-world-like tests. Additionally, with some luck, it might be bundled with openfiler to ease the storage management. Last but not least it could include support for Xen in a future version.

OpenGL 3.0

The release of OpenGL 3.0 this year was rather surprising: it was delayed for almost a year without any notice at all, which is usually a clear sign that a project is dead. However, left aside the question if the OpenGL 3.0 release is the beginning of a new era or or just a last breath of the project, OpenGL 3.0 is now out in the wild and the Free Software community will adopt it sooner or later.

While Nvidia has already released a first version of an OpenGL 3.0 capable driver the FLOSS OpenGL implementation Mesa hasn’t released anything yet. But Mesa is alive and vibrant again since 2007 and a new release can be expected in the near future. Also likely is that AMD/ATI will release a new version of their OpenGL stack featuring the newest OpenGL spec. I would like to see AMD/TI team up with Mesa on that one but that’s just a wish, I’m afraid.

So in 2009 we will see OpenGL 3.0 coming to the masses – in proprietary as well as in Free drivers. This way newest graphics card technology will come to Linux and application developers can built upon that.

Gallium

Simply said, Gallium3D is an attempt to make graphics card driver development on Linux much easier: it abstracts the driver development from the underlying graphics standard implementation (for example OpenGL). Due to that abstraction, switching to another graphics standard should also be fairly easy. That way it should be easier to write one single graphics card driver for different devices (which do often need something else than OpenGL). And in case OpenGL is really dead, it could be a way to more or less painlessly replace it with something new.
Right now Gallium3D is in heavy development and we yet have to see it in the wild. There are only few drivers ported to it and I haven’t seen any distribution shipping it yet.

In 2009 this could switch: a first testing release for the broader masses is likely, and it could speed up the development of drivers for Gallium3D.

Gem and KMS

Speaking about graphics, there are other things which are in development and which are already surfacing here and there: the new graphics memory manager GEM. Using GEM the graphics cards does not have to be re-initialized as soon as you switch to another application. Also, everything will be written to the memory and the composition manager can simply access it there, avoiding some problems current drivers have when for example running videos on AIGLX.
Besides, Kernel Mode Setting (KMS) will move other tasks of the graphic subsystem away from X towards the kernel. As a result it will be much easier and flicker-free to switch from X to a tty console, and the graphical system will be able to show kernel oops. Linux will get its own blue screen capability, finally!

These features have partially found their way into newer Fedora releases, but only for specific hardware and under certain conditions. In 2009 it can be expected that the current FLOSS/Nvidia/AMD drivers will switch over to GEM and KMS to provide a much saner graphics experience to the user.

KDE 4.3: Pimp your PIM

Curently KDE’s PIM is in a difficult situation: Kontact is one of the best free groupware clients out there, but it was never designed to be one, and using it as such today can be an adventure. To fix that Akonadi was created. It was shipped with KDE 4.1 to back up Mailody, KDE 4.2 will see it the first time together with Kontact. This will give the developers quite some time to sanitize and improve the Akonadi service as well as to add new plugins to provide something revolutionary right in time for KDE 4.3.

In 2009 we will finally see a FLOSS groupware client which is working with a broad range of groupware servers, has a maintainable code base – and is perfectly integrated on all major platforms.

Qt on the mobile mass market

This year almost started with the news that Nokia acquired Trolltech. Recently it was announced that Qt now runs on Symbian S60. Also, with the iPhone, Google’s G1 and even a new Blackberry Nokia seriously needs a cool new device with fancy graphics and an appealing software platform.

Now put two and two together. With a bit of luck we will see the first Qt-Nokia devices with multi touch screen in 2009. With even a bit more of luck, it will be shipped in a way that Qt developers can use the tools they are used to to develop software for the new platform. Think of running KDE on these devices.

Gnome 3.0 development

In summer this year the Gnome developers started planning their next big release – Gnome 3.0. Currently not too many information have surfaced, but such breaks need their time. A state tracker for the Gtk+ changes is online and shows that indeed some work is underway already.

In 2009 first Alpha release could surface to show in which direction Gtk+ and Gnome are heading, and how the transition progress works out. That will definitely be an interesting time – the transition was a major task for KDE, and the Gnome team better takes a close look at that to learn from KDE’s experience.

Conclusion

While I already called 2007 the Year Of Open Source Graphics, 2009 can become a good candidate for it as well. In this post it got three paragraphs, and if everything comes true, 2009 will revolutionize the world of Linux graphics. This will, however, happen mostly under the hood. The users will not notice several fixes, but not the large underlying changes, which is different to 2007.

But in general 2009 will be exciting in almost all FLOSS areas. Keep in mind that this list is not and cannot be complete! So I ask every reader to drop a comment here containing his or her tip for revolutionary changes or news in the FLOSS world in 2009!

The background

3G is the so called third generation mobile technology, best known for it’s well known standard UMTS. Using that technology users are able to get quite fast Internet connections via mobile phone technology. In fact, in Germany the UMTS upload is currently faster than most of the ADSL connections you can get.

However, recently my company (credativ GmbH) was contacted by a customer who reported that the UMTS upload speed with Linux is rather slow compared with the upload speed on Windows. We did some tests and the result was disturbing: the factor is roughly two. While we had upload speeds of 1200 kbit/s on Windows XP, Linux only did less than 600 kbit/s (both using HSUPA). Hardware problems where out of question because we tested with a rather large set of UMTS hardware. We also tried different Linux distributions, different computers and so on. Additionally, the problem was at least verified by three (!) labs around the world: one lab of a major German mobile pone company and one lab each of two international hardware vendors. (And yes, it was me who tried to keep track of all the tests and people who are spread on several continents in different time zones ).

The fix

In the end, however, we were able to produce a rather simple patch to fix this problem. The upload speed in our test is now almost as large in Linux as in Windows, and this result was verified by the mentioned labs. The patch is:

--- drivers/usb/serial/option.c.old        2008-11-27 12:45:50.173275119 +0100
+++ drivers/usb/serial/option.c    2008-11-27 12:46:06.089274050 +0100
@@ -487,9 +487,9 @@
 /* per port private data */

 #define N_IN_URB 4
-#define N_OUT_URB 1
+#define N_OUT_URB 4
 #define IN_BUFLEN 4096
-#define OUT_BUFLEN 128
+#define OUT_BUFLEN 4096

 struct option_port_private {
        /* Input endpoints and buffer for this port */

At this point I’d like to plastinka who pointed me to that part of the kernel to fix the problem. I owe you something!

The problem: getting the fix upstream

The problem is now: how to get this fix upstream? I’m certainly not a kernel developer, and I could not even explain why this patch fixes the problem – I “just” have the test results. Joining the kernel list might also not be the best idea when I am not even a programmer.

So I decided to send the patch to the Red Hat guys – they have lots of developers, good connections to upstream, and I know how to talk to them. It’s bug #473252, let’s see what happens. In case you experience similar problems, send this patch upstream via your distribution, maybe that helps as well.

64bit in General

Switching from 32bit to 64bit on an operating system is a huge and complicated task involving effectively all larger applications. This can e a real pain – unless you have an operating system where all software usually supports 64bit anyway. This is the case with most open source operating systems and therefore also with Linux. So grabbing the 64bit image and installing it was just like grabbing the 32bit image. In case of Fedora the download link offered by default was 32bit, but 64bit was just a click away. I wonder when that will change.

There are numerous advantages and disadvantages regarding 32bit and 64bit, for a first introduction start with the Wikipedia article.

Flash

The problems regarding 64bit arise when you deal with non-Open Source software: this might only be provided as 32bit. In case it depends on any other library, the system must provide these libraries in 32bit and 64bit. While on RPM systems this is not a problem at all, this can be rather problematic when browser plugins are 32bit only, because then the browser needs to be 32bit only as well, the same is true then for all other plugins, and so on. There are wrappers to deal with that, but these are sub-optimal.

Luckily, Adobe has now released a 64bit Alpha version of their Flash player. While it is still missing several features and is not even provided as a rpm or deb file, in my first tests it worked without problems. As a side note, the 64bit versions for Windows and Mac OS are still not out there – Linux is a clear technology and development pusher here!

For the sake of completion (and since someone would point it out in a comment anyway), there are also free (as in FLOSS) alternatives to the Flash player – which are available in 64bit for quite some time now, of course.

Skype

Another issue is Skype – this is not provided as a 64bit version at all (bug report). For Ubuntu users there is at least a 32bit version modified for easy installation on 64bit systems. Btw., hardly anyone seems to know that, even the German Ubuntu wiki doesn’t mention that at all.

Anyway, that doesn’t help the Fedora community anyway – but since Fedora runs on RPM installing all the compatibility libraries is just a question of hard disk space:

yum --nogpgcheck localinstall skype*rpm
yum install alsa-plugins-pulseaudio-1.0.18-1.rc3.fc10.i386

PulseAudio

Well, PulseAudio is a difficult thing. It has a rather strong community and people are making sure it comes up everywhere and works everywhere like it should. However, while I read all the rather long papers and documents why I should need PA, nothing of these papers really stuck, and I always wonder why it is really needed – apart from the more esoteric reasons that Alsa is not suitable for the future. Besides, I do wonder if the Alsa guys would say the same.
Additionally, in my first tests PA worked just fine – under Gnome, but not in KDE. So my first step after installing Fedora since my first contact with PA was always to remove PA.

But I always tend to give things another try, and this time I didn’t remove it. And indeed, almost everything works, and I haven’t even met a delay yet. It all just works, even on KDE with its Phonon!

So it looks like PA finally fits in well. Now what I only need some ideas what to do with it Seriously, what I would appreciate to see is a simple one sheet drawing with all the usual suspects of the Linux audio blob (from Phonon over xine down to Alsa) to see where PA fits in and what it does there.

KDE, Nvidia and performance

Fedora 10 now includes KDE 4.1.3 (included in the updates), and together with RPM Fusion Nvidia drivers are just a

yum install kmod-nvidia

I am slightly surprised and wonder what was wrong with my Fedora 10 Beta setup. But on the other hand, my work machine is running Kubuntu 8.10 and there the performance is similar fast. So to me it looks like the days of slow KDE 4.x on Nvidia hardware are finally over, given that the drivers are the newest stable ones and KDE is of version 4.1.3.

Fedora’s encryption

This time I decided to not go with a full hard disk encryption, but rather with a home disk encryption. And while I still dislike Fedora’s disk druid for not letting me chose the disk setup in detail I appreciate that clicking a checkbox was all I had to do to activate the home partition encryption. It is even nicely integrated with the boot process.

Overall impression

The overall impression of Fedora 10 is very good. Most bugs I encountered running the Beta version are fixed – except for a strange coding problem, but I will survive that one.

Also, my first move into the lands of 64bit are also far less complicated than expected. Your mileage may vary, depending on the used proprietary software, but then again kvm might be a solution to work around that problem.

Background

Recently my company, the credativ GmbH, was faced with the task to sync the usual groupware data (calender, tasks, addresses) between a mobile phone and a PIM client – using FLOSS and via Internet. The natural solution of course is Funambol to get the data of the phone. The next step is to connect that one to a groupware server and access the server via a well known KDE 3 groupware client.

However, having a closer look at the currently available FLOSS groupware solutions was quite disillusioning. While there are many nice servers, there are hardly any solutions where a server works correctly with a client. The best working example is maybe Kolab-Kontact, but Kolab doesn’t work easily together with Funambol. Actually, the current groupware-Funambol connector only supports three groupware servers: Citadel, OpenGroupware and eGroupware. For various reasons we settled on Citadel, and now the task was to check how well this works together with Kontact.

Citadel

Citadel is rather unique in the groupware world. It is one of the oldest projects, and started off as something different than it is today. For that reason there are many historically grown things – like the room concept. However, once you get used to that the server is quite usable, and the developer community was very helpful.

Regarding the described setup there were only smaller things to fix, and the Citadel guys did that for us, so thanks to them. Citadel still desperately needs a theme which is pleasing the eyes – the current one looks horrible. But that is a task for a web designer and can come after the code basically works.

Kontact – solved problems

Kontact has quite some problems on many levels. The main reason behind all these problems is the protocol: there is no single open standard for doing groupware stuff, there are many different. There is though Groupdav, an attempt to create such a common standard, but until now Groupdav is still a proposal, and besides some missing main functions (no notifications when the content was changed, for example) it wasn’t accepted by everyone. As a result, the current Groupdav support in Kontact is mixed, and we tried to check what worked, what didn’t, and what was fixable for us.

The first problem was that Kontact of KDE 3 has problems with special characters like German umlauts. That is described in Bug #159795, and the patches necessary to fix that behavior are added to the bug report. This is fixed in KDE 4 anyway, so the patch is only interesting for people who are working with Kontact of KDE 3.x for some more time.

The second problem was that tasks flagged as complete were not synced properly – at least, the complete flag part wasn’t synced. When in Kontact a task was flagged as completed it didn’t show up as such on Citadel and vice versa. The problem seems to be that Kontact looks at the percentage value of the completion while Citadel looks at the parameter ‘COMPLETED’. See Bug #171905 for more details – and a patch.

The base for the third problem lies more in the groupdav protocol than in Kontact: there is no mechanism defined to inform the client that the content on the server has been updated. That is sad – and requires the client to pull for changes. This is not properly implemented in Kontact. Therefore we settled on a hack: whenever a resource in Kontact (addressbook or organizer) is deactivated and again activated, the content is reloaded. We decided to trigger exact that action with a refresh button.
Immediately the question comes up if that should not be triggered by time, but the usual work flow in an office is: “Hey, Joe, it’s me, John, have you seen the new appointment I just added to your calendar?” – a time base pulling would confuse the user, a button to pull would not. The bug to this problem is Bug #175409 – including the patches.

Kontact – problems still valid

While we were able to fix some of the problems (hurray Open Source) there are still some things left: we were not able to properly implement a working Free/Busy system with Kontact and Citadel. This is definitely due to some rather strange bus in Kontact, and will require quite some rewrite of Kontact’s resource plugins. But since Akonadi will be out soon the question is if that should be fixed now or if we should wait until Akonadi is out and then have another look at it.

Final words

Kontact and Citadel do work together somehow. And the bits which are missing can be fixed. However, it is clear that Kontact developed into something which was not planned originally, and that a revolutionary change is needed.
Akonadi will hopefully bring this change, but currently it is not there yet.

But even if Akonadi arrives right in time – and with plugins for all the needs we have – the underlying problem of the broken groupdav protocol remains. It is sad somehow that there are so many fine FLOSS Groupware servers out there but that there is hardly any protocol which is supported by all of them and delivers all the features the users want. No wonder that Outlook/Exchange has such a strong position right now.

x11vnc -display :0

In case you need to access a running login manager (where no user is logged in yet) first get the authentication file of the running X and hand it over to x11vnc (as root):

host:~# ps x|grep X
 1425 tty8     Ss+    0:01 /usr/bin/X -br :0 vt8 -auth /var/run/xauth/A:0-1EkR5c
host:~# x11vnc -display :0 -auth /var/run/xauth/A:0-1EkR5c

The Problem

The KHelpCenter is the central place to access the KDE program handbooks. It also offers an option to index all these data to quickly search through them. However, currently this is done on a per-user base and is offered as a dialog once the KHelpCenter is started the first time.

In larger deployments with several users per machine/server or where you have central login servers you might not want your central server to do the same task again and again, for each new user. Therefore it makes sense to a) pre-compile the data and b) configure all your KDE users to access a central directory.

The Solution

The first task can be accomplished by starting the index process manually and let it write to a system-directory:

khc_docbookdig.pl --indexdir=$SYSTEMDIR --docpath=khelpcenter:kde_application_manuals --identifier=kde_application_manuals

Now the user-KDE configuration need to know about the new index files. Since it might even be that some users have already created an index and some not, so the best is to overwrite their configuration without actually modifying anything in their configuration files – which is not a problem with KDE’s Kiosk. Depending on your distribution, your KDE system configuration is somewhere in /etc/kde/ or similar. There the file khelpcenterrc must be created/modified with the following content:

[Search][$i]
IndexDirectory=$SYSTEMDIR
IndexExists=true
ScopeSelection=0

Closing Words

The KHelpCenter is a ncie program – but needs quite some love. Recently apacheLog identified several problems and was only able to fix some of them. Some slightly larger tasks are an integration with Nepomuk/Strigi and the inclusion of the online documentation: Userbase and Techbase. Last but not least it might be worth a look if KHelpCenter could take advantage of QtAssistant.

Finally a last word about the command mentioned above: it only include the KDE handbooks, the man pages might not be included.