Last call for ….

I haven’t posted anything in a while on this blog, and now I made the decision that this will not change: it is unlikely that this blog will be updated anytime soon. The reason is actually twofold:

Job

I’m working full time as an Open Source/Linux consultant these days – and after work I do not really have the time nor the energy to invest even more time into Open Source (besides the Fedora packaging).

Company’s Blog

I was successful in convincing enough people in our company to start a blog – and I blog there since then, so when I get home I usually already have blogged about whatever comes to my mind.

That means effectively that you will not receive any more new posts here. It hurts my heart and kills kittens, but you can remove the blogfeed. @planets where I might still be listed at: please remove this blog feed as well.

However, if you *do* want to keep up with my thoughts: credativ’s company blog is working quite nice these days. Btw., in case you don’t know, credativ is an Open Source/Linux company and the one behind the Open Source Support Center (OSSC) and the Open Source Support Card (yeah, “catchy” names, I know). They are focussed on Open Source support (Linux-Support, PostgreSQL-Support, etc.) and have offices in DE, UK, US, etc. So the general topics are pretty close to this blog. If you look close you will recognize my style: short italic introduction, eye catcher on the upper right side, special headline markings for Howtos and Short Tips, and so on. Also, the categories are quite the same, and it is actually available in German and English. Also, I am not the only person writing there – one very active PostgreSQL developer keeps blogging there, if I want it or not.

However – it is a company blog, so you will (!) find information regarding the company itself, or newest marketing things. You are warned!

So this is it: the last post. Thanks everyone for wonderful years full of blogging, discussions, news, Howtos and good tips. So long, and thanks for the fish!

Playing the numbers game 2008: number of Linux installations world wide

The number of Linux users and installations is impossible to determine. But there are several different statistical information available which can be used to at least get a rough idea of the number of Linux installations world wide.

Merging different statistical data into one number is a tricky exercise and the result is questionable at best. Keep that in mind when you read the following information. The idea is not to get exact numbers but to get a rough idea of the dimension, nothing else.

Source #1: Fedora

One of the best statistical sources regarding Linux usage are the Fedora statistics. There the number of downloaded images as well as the number of unique IPs getting software updates is counted.
The data are difficult to interpret: no one knows if a downloaded image was only used to test the new system or to bun it onto a CD and distribute it to thousands of magazine readers or thousands of company computers. The second number is problematic because one new IP can mean a big NAT network or just a dial-in user who re-connected. So flaws everywhere, but it is a interesting coincidence that the IP numbers and the downloads are rather close.

There are also the smolt data. It tracks the users who opted in to a tracking system. Currently the smolt web server seems to be lacking behind. But there are current data available for older Fedora releases: these informations say that every month Fedora still gets more than 10k new Fedora 7 users – although there are already Fedora 8 and Fedora 9 available.

So the question is how to read all the data. One way was recently suggested by Paul Frields, Fedora’s project leader: he sums up the data to be around 11.5 million. Together with currently 2.5 million Red Hat subscriptions this would result in 13 million users. Focussing on Fedora alone and leaving the Fedora 6 users Fedora would still have a user base of 9.5 million users.

Source #2: relative statistics

There are hardly any other trustfully data from other Linux distributions available. Therefore, the Fedora number does say a lot about Fedora, but not that much about Linux in general. However, there are other statistics which measure the relative acceptance of Linux distributions.

One such source is the 2007 Linux Desktop Survey done by DesktopLinux.com. There the relative importance of Fedora/Red Hat is measured with 9%. Unfortunately there is no more recent survey available. I wonder why no one has picked up such a survey in 2008. Maybe I should start one on my own?

The result

Given that Fedora/Red Hat has roughly 10% and also roughly 10 million users together (which in fact seems like a at least slightly realistic data base, given the facts), the total number of Linux users world wide would sum up to 100 million Linux users. Nice.

That would leave Mac OS far behind, which is however not that surprising: Mac OS is hardly used in Offices or the government outside the US, and it is far easier to give Linux a try and keep it as a dual boot option besides a Windows installation. Also, the EU governments are pushing Linux quite a lot, and many companies and governments indeed switch to Linux right now or already switched over in the client space for some of the day-to-day workstations.

Still, last year I played the “numbers game” already (unfortunately with the same relativity source, btw.) and the result said something about 20 to 30 million users. I doubt that the number of Linux users spiked that much in the last year, but think that we can safely say the number of Linux users world wide is somewhere in the middle two digit million area, somewhere around 50 million installations worldwide.

Keep in mind that this counts mainly workstations – not traffic lights, shop information terminals or any other specialized hardware. Including all these devices would result in much, much larger numbers.

The problem: discrepancy

Most numbers available guessing the number of Linux users world wide say that there are not that many Linux installations out there, not at all. Most often it is said that, in percentage, Apple has a low one-digit number, while Linux has a dot before its first non-zero number.
The statistical backup for such numbers is most often created by browser strings aggregated from Web pages. This procedure has the flaw that these strings are often faked to make it easier to access specific pages. Also, the monitored pages are only a subset of the entire web, and surprisingly often focus on the US only which is not representative for the world at any rate.

Still, I do often wonder why such numbers and my estimates are different in the order of magnitudes. I welcome any comment on that issue!

The Open Source Year 2009

Every year Open Source technology is improved and extended. This post sheds some light on new technologies which might arrive in 2009.

Btrfs

With ext4 a new file just left the developer corner. However, ext4 is an old-style file system and does not offer “hot” features like on-line snap shots, versioning and so on. ZFS does, but it is not an option for Linux due to licence reasons. Here comes Btrfs into play: it is in development for quite some time now and many Kernel developers already asked to include Btrfs in the Kernel to speed up the development process. Additionally, several Kernel developers already mentioned that they expect Btrfs to be the next generation default file system for Linux in the mid-term.

In 2009 Btrfs will most likely stabilize its file system format and publish a beta version for testing purposes.

oVirt

oVirt is a small host image that provides libvirt services and hosts virtual machines. Additionally it also has a well designed web based management system. The aim is to provide an enterprise ready VM management console capable of managing large sever clusters hosting large numbers of virtual machines, but is also supposed for single users.

In 2009 oVirt will hopefully see its first beta release ready for first real-world-like tests. Additionally, with some luck, it might be bundled with openfiler to ease the storage management. Last but not least it could include support for Xen in a future version.

OpenGL 3.0

The release of OpenGL 3.0 this year was rather surprising: it was delayed for almost a year without any notice at all, which is usually a clear sign that a project is dead. However, left aside the question if the OpenGL 3.0 release is the beginning of a new era or or just a last breath of the project, OpenGL 3.0 is now out in the wild and the Free Software community will adopt it sooner or later.

While Nvidia has already released a first version of an OpenGL 3.0 capable driver the FLOSS OpenGL implementation Mesa hasn’t released anything yet. But Mesa is alive and vibrant again since 2007 and a new release can be expected in the near future. Also likely is that AMD/ATI will release a new version of their OpenGL stack featuring the newest OpenGL spec. I would like to see AMD/TI team up with Mesa on that one but that’s just a wish, I’m afraid.

So in 2009 we will see OpenGL 3.0 coming to the masses – in proprietary as well as in Free drivers. This way newest graphics card technology will come to Linux and application developers can built upon that.

Gallium

Simply said, Gallium3D is an attempt to make graphics card driver development on Linux much easier: it abstracts the driver development from the underlying graphics standard implementation (for example OpenGL). Due to that abstraction, switching to another graphics standard should also be fairly easy. That way it should be easier to write one single graphics card driver for different devices (which do often need something else than OpenGL). And in case OpenGL is really dead, it could be a way to more or less painlessly replace it with something new.
Right now Gallium3D is in heavy development and we yet have to see it in the wild. There are only few drivers ported to it and I haven’t seen any distribution shipping it yet.

In 2009 this could switch: a first testing release for the broader masses is likely, and it could speed up the development of drivers for Gallium3D.

Gem and KMS

Speaking about graphics, there are other things which are in development and which are already surfacing here and there: the new graphics memory manager GEM. Using GEM the graphics cards does not have to be re-initialized as soon as you switch to another application. Also, everything will be written to the memory and the composition manager can simply access it there, avoiding some problems current drivers have when for example running videos on AIGLX.
Besides, Kernel Mode Setting (KMS) will move other tasks of the graphic subsystem away from X towards the kernel. As a result it will be much easier and flicker-free to switch from X to a tty console, and the graphical system will be able to show kernel oops. Linux will get its own blue screen capability, finally!

These features have partially found their way into newer Fedora releases, but only for specific hardware and under certain conditions. In 2009 it can be expected that the current FLOSS/Nvidia/AMD drivers will switch over to GEM and KMS to provide a much saner graphics experience to the user.

KDE 4.3: Pimp your PIM

Curently KDE’s PIM is in a difficult situation: Kontact is one of the best free groupware clients out there, but it was never designed to be one, and using it as such today can be an adventure. To fix that Akonadi was created. It was shipped with KDE 4.1 to back up Mailody, KDE 4.2 will see it the first time together with Kontact. This will give the developers quite some time to sanitize and improve the Akonadi service as well as to add new plugins to provide something revolutionary right in time for KDE 4.3.

In 2009 we will finally see a FLOSS groupware client which is working with a broad range of groupware servers, has a maintainable code base – and is perfectly integrated on all major platforms.

Qt on the mobile mass market

This year almost started with the news that Nokia acquired Trolltech. Recently it was announced that Qt now runs on Symbian S60. Also, with the iPhone, Google’s G1 and even a new Blackberry Nokia seriously needs a cool new device with fancy graphics and an appealing software platform.

Now put two and two together. With a bit of luck we will see the first Qt-Nokia devices with multi touch screen in 2009. With even a bit more of luck, it will be shipped in a way that Qt developers can use the tools they are used to to develop software for the new platform. Think of running KDE on these devices.

Gnome 3.0 development

In summer this year the Gnome developers started planning their next big release – Gnome 3.0. Currently not too many information have surfaced, but such breaks need their time. A state tracker for the Gtk+ changes is online and shows that indeed some work is underway already.

In 2009 first Alpha release could surface to show in which direction Gtk+ and Gnome are heading, and how the transition progress works out. That will definitely be an interesting time – the transition was a major task for KDE, and the Gnome team better takes a close look at that to learn from KDE’s experience.

Conclusion

While I already called 2007 the Year Of Open Source Graphics, 2009 can become a good candidate for it as well. In this post it got three paragraphs, and if everything comes true, 2009 will revolutionize the world of Linux graphics. This will, however, happen mostly under the hood. The users will not notice several fixes, but not the large underlying changes, which is different to 2007.

But in general 2009 will be exciting in almost all FLOSS areas. Keep in mind that this list is not and cannot be complete! So I ask every reader to drop a comment here containing his or her tip for revolutionary changes or news in the FLOSS world in 2009!

Fixing UMTS upload performance on Linux

UMTS, or better said, 3G, is a convenient way to connect to the Internet while you are travelling. However, the upload speed in Linux is only half as large as in Windows – but this can be fixed by a Kernel patch.

The background

3G is the so called third generation mobile technology, best known for it’s well known standard UMTS. Using that technology users are able to get quite fast Internet connections via mobile phone technology. In fact, in Germany the UMTS upload is currently faster than most of the ADSL connections you can get.

However, recently my company (credativ GmbH) was contacted by a customer who reported that the UMTS upload speed with Linux is rather slow compared with the upload speed on Windows. We did some tests and the result was disturbing: the factor is roughly two. While we had upload speeds of 1200 kbit/s on Windows XP, Linux only did less than 600 kbit/s (both using HSUPA). Hardware problems where out of question because we tested with a rather large set of UMTS hardware. We also tried different Linux distributions, different computers and so on. Additionally, the problem was at least verified by three (!) labs around the world: one lab of a major German mobile pone company and one lab each of two international hardware vendors. (And yes, it was me who tried to keep track of all the tests and people who are spread on several continents in different time zones ).

The fix

In the end, however, we were able to produce a rather simple patch to fix this problem. The upload speed in our test is now almost as large in Linux as in Windows, and this result was verified by the mentioned labs. The patch is:

--- drivers/usb/serial/option.c.old        2008-11-27 12:45:50.173275119 +0100
+++ drivers/usb/serial/option.c    2008-11-27 12:46:06.089274050 +0100
@@ -487,9 +487,9 @@
 /* per port private data */

 #define N_IN_URB 4
-#define N_OUT_URB 1
+#define N_OUT_URB 4
 #define IN_BUFLEN 4096
-#define OUT_BUFLEN 128
+#define OUT_BUFLEN 4096

 struct option_port_private {
        /* Input endpoints and buffer for this port */

This is around line 490 in drivers/usb/serial/option.c in the current linux tree.

At this point I’d like to plastinka who pointed me to that part of the kernel to fix the problem. I owe you something!

The problem: getting the fix upstream

The problem is now: how to get this fix upstream? I’m certainly not a kernel developer, and I could not even explain why this patch fixes the problem – I “just” have the test results. Joining the kernel list might also not be the best idea when I am not even a programmer.

So I decided to send the patch to the Red Hat guys – they have lots of developers, good connections to upstream, and I know how to talk to them. It’s bug #473252, let’s see what happens. In case you experience similar problems, send this patch upstream via your distribution, maybe that helps as well.

Moving on: 64bit Linux, PulseAudio, Fedora 10 and so on

With the release of Fedora 10 I took the opportunity to finally switch over to 64bit Linux – including the proprietary stuff like Flash, Skype, and so on. Also, Fedora 10 itself had several rather pleasing surprises for me.

I already used Fedora 10 since it’s Beta release. However, recently I decided to re-install it, this time in 64bit, and check how that would go. Also, since I had some rather strange problems and performance issues I wondered if a re-install would fix them.

64bit in General

Switching from 32bit to 64bit on an operating system is a huge and complicated task involving effectively all larger applications. This can e a real pain – unless you have an operating system where all software usually supports 64bit anyway. This is the case with most open source operating systems and therefore also with Linux. So grabbing the 64bit image and installing it was just like grabbing the 32bit image. In case of Fedora the download link offered by default was 32bit, but 64bit was just a click away. I wonder when that will change.

There are numerous advantages and disadvantages regarding 32bit and 64bit, for a first introduction start with the Wikipedia article.

Flash

The problems regarding 64bit arise when you deal with non-Open Source software: this might only be provided as 32bit. In case it depends on any other library, the system must provide these libraries in 32bit and 64bit. While on RPM systems this is not a problem at all, this can be rather problematic when browser plugins are 32bit only, because then the browser needs to be 32bit only as well, the same is true then for all other plugins, and so on. There are wrappers to deal with that, but these are sub-optimal.

Luckily, Adobe has now released a 64bit Alpha version of their Flash player. While it is still missing several features and is not even provided as a rpm or deb file, in my first tests it worked without problems. As a side note, the 64bit versions for Windows and Mac OS are still not out there – Linux is a clear technology and development pusher here!

For the sake of completion (and since someone would point it out in a comment anyway), there are also free (as in FLOSS) alternatives to the Flash player – which are available in 64bit for quite some time now, of course.

Skype

Another issue is Skype – this is not provided as a 64bit version at all (bug report). For Ubuntu users there is at least a 32bit version modified for easy installation on 64bit systems. Btw., hardly anyone seems to know that, even the German Ubuntu wiki doesn’t mention that at all.

Anyway, that doesn’t help the Fedora community anyway – but since Fedora runs on RPM installing all the compatibility libraries is just a question of hard disk space:

yum --nogpgcheck localinstall skype*rpm
yum install alsa-plugins-pulseaudio-1.0.18-1.rc3.fc10.i386

That’s it. In my tests Skpe indeed worked, even a video test image was shown although I haven’t actually made a real video call. Also, I had problems with the microphone, but that might be due to problems with PulseAudio. I appreciate any tips on that issue.

PulseAudio

Well, PulseAudio is a difficult thing. It has a rather strong community and people are making sure it comes up everywhere and works everywhere like it should. However, while I read all the rather long papers and documents why I should need PA, nothing of these papers really stuck, and I always wonder why it is really needed – apart from the more esoteric reasons that Alsa is not suitable for the future. Besides, I do wonder if the Alsa guys would say the same.
Additionally, in my first tests PA worked just fine – under Gnome, but not in KDE. So my first step after installing Fedora since my first contact with PA was always to remove PA.

But I always tend to give things another try, and this time I didn’t remove it. And indeed, almost everything works, and I haven’t even met a delay yet. It all just works, even on KDE with its Phonon!

So it looks like PA finally fits in well. Now what I only need some ideas what to do with it Seriously, what I would appreciate to see is a simple one sheet drawing with all the usual suspects of the Linux audio blob (from Phonon over xine down to Alsa) to see where PA fits in and what it does there.

KDE, Nvidia and performance

Fedora 10 now includes KDE 4.1.3 (included in the updates), and together with RPM Fusion Nvidia drivers are just a

yum install kmod-nvidia

away. While I had trouble with this way with the beta version, and general serious performance problems with the drivers installed manually, it turned out that with Fedora 10 final everything works like a charm – fast and snappy!

I am slightly surprised and wonder what was wrong with my Fedora 10 Beta setup. But on the other hand, my work machine is running Kubuntu 8.10 and there the performance is similar fast. So to me it looks like the days of slow KDE 4.x on Nvidia hardware are finally over, given that the drivers are the newest stable ones and KDE is of version 4.1.3.

Fedora’s encryption

This time I decided to not go with a full hard disk encryption, but rather with a home disk encryption. And while I still dislike Fedora’s disk druid for not letting me chose the disk setup in detail I appreciate that clicking a checkbox was all I had to do to activate the home partition encryption. It is even nicely integrated with the boot process.

Overall impression

The overall impression of Fedora 10 is very good. Most bugs I encountered running the Beta version are fixed – except for a strange coding problem, but I will survive that one.

Also, my first move into the lands of 64bit are also far less complicated than expected. Your mileage may vary, depending on the used proprietary software, but then again kvm might be a solution to work around that problem.