Job

I’m working full time as an Open Source/Linux consultant these days – and after work I do not really have the time nor the energy to invest even more time into Open Source (besides the Fedora packaging).

Company’s Blog

I was successful in convincing enough people in our company to start a blog – and I blog there since then, so when I get home I usually already have blogged about whatever comes to my mind.

That means effectively that you will not receive any more new posts here. It hurts my heart and kills kittens, but you can remove the blogfeed. @planets where I might still be listed at: please remove this blog feed as well.

However, if you *do* want to keep up with my thoughts: credativ’s company blog is working quite nice these days. Btw., in case you don’t know, credativ is an Open Source/Linux company and the one behind the Open Source Support Center (OSSC) and the Open Source Support Card (yeah, “catchy” names, I know). They are focussed on Open Source support (Linux-Support, PostgreSQL-Support, etc.) and have offices in DE, UK, US, etc. So the general topics are pretty close to this blog. If you look close you will recognize my style: short italic introduction, eye catcher on the upper right side, special headline markings for Howtos and Short Tips, and so on. Also, the categories are quite the same, and it is actually available in German and English. Also, I am not the only person writing there – one very active PostgreSQL developer keeps blogging there, if I want it or not.

However – it is a company blog, so you will (!) find information regarding the company itself, or newest marketing things. You are warned!

So this is it: the last post. Thanks everyone for wonderful years full of blogging, discussions, news, Howtos and good tips. So long, and thanks for the fish!

Source #1: Fedora

One of the best statistical sources regarding Linux usage are the Fedora statistics. There the number of downloaded images as well as the number of unique IPs getting software updates is counted.
The data are difficult to interpret: no one knows if a downloaded image was only used to test the new system or to bun it onto a CD and distribute it to thousands of magazine readers or thousands of company computers. The second number is problematic because one new IP can mean a big NAT network or just a dial-in user who re-connected. So flaws everywhere, but it is a interesting coincidence that the IP numbers and the downloads are rather close.

There are also the smolt data. It tracks the users who opted in to a tracking system. Currently the smolt web server seems to be lacking behind. But there are current data available for older Fedora releases: these informations say that every month Fedora still gets more than 10k new Fedora 7 users – although there are already Fedora 8 and Fedora 9 available.

So the question is how to read all the data. One way was recently suggested by Paul Frields, Fedora’s project leader: he sums up the data to be around 11.5 million. Together with currently 2.5 million Red Hat subscriptions this would result in 13 million users. Focussing on Fedora alone and leaving the Fedora 6 users Fedora would still have a user base of 9.5 million users.

Source #2: relative statistics

There are hardly any other trustfully data from other Linux distributions available. Therefore, the Fedora number does say a lot about Fedora, but not that much about Linux in general. However, there are other statistics which measure the relative acceptance of Linux distributions.

One such source is the 2007 Linux Desktop Survey done by DesktopLinux.com. There the relative importance of Fedora/Red Hat is measured with 9%. Unfortunately there is no more recent survey available. I wonder why no one has picked up such a survey in 2008. Maybe I should start one on my own?

The result

Given that Fedora/Red Hat has roughly 10% and also roughly 10 million users together (which in fact seems like a at least slightly realistic data base, given the facts), the total number of Linux users world wide would sum up to 100 million Linux users. Nice.

That would leave Mac OS far behind, which is however not that surprising: Mac OS is hardly used in Offices or the government outside the US, and it is far easier to give Linux a try and keep it as a dual boot option besides a Windows installation. Also, the EU governments are pushing Linux quite a lot, and many companies and governments indeed switch to Linux right now or already switched over in the client space for some of the day-to-day workstations.

Still, last year I played the “numbers game” already (unfortunately with the same relativity source, btw.) and the result said something about 20 to 30 million users. I doubt that the number of Linux users spiked that much in the last year, but think that we can safely say the number of Linux users world wide is somewhere in the middle two digit million area, somewhere around 50 million installations worldwide.

Keep in mind that this counts mainly workstations – not traffic lights, shop information terminals or any other specialized hardware. Including all these devices would result in much, much larger numbers.

The problem: discrepancy

Most numbers available guessing the number of Linux users world wide say that there are not that many Linux installations out there, not at all. Most often it is said that, in percentage, Apple has a low one-digit number, while Linux has a dot before its first non-zero number.
The statistical backup for such numbers is most often created by browser strings aggregated from Web pages. This procedure has the flaw that these strings are often faked to make it easier to access specific pages. Also, the monitored pages are only a subset of the entire web, and surprisingly often focus on the US only which is not representative for the world at any rate.

Still, I do often wonder why such numbers and my estimates are different in the order of magnitudes. I welcome any comment on that issue!

Btrfs

With ext4 a new file just left the developer corner. However, ext4 is an old-style file system and does not offer “hot” features like on-line snap shots, versioning and so on. ZFS does, but it is not an option for Linux due to licence reasons. Here comes Btrfs into play: it is in development for quite some time now and many Kernel developers already asked to include Btrfs in the Kernel to speed up the development process. Additionally, several Kernel developers already mentioned that they expect Btrfs to be the next generation default file system for Linux in the mid-term.

In 2009 Btrfs will most likely stabilize its file system format and publish a beta version for testing purposes.

oVirt

oVirt is a small host image that provides libvirt services and hosts virtual machines. Additionally it also has a well designed web based management system. The aim is to provide an enterprise ready VM management console capable of managing large sever clusters hosting large numbers of virtual machines, but is also supposed for single users.

In 2009 oVirt will hopefully see its first beta release ready for first real-world-like tests. Additionally, with some luck, it might be bundled with openfiler to ease the storage management. Last but not least it could include support for Xen in a future version.

OpenGL 3.0

The release of OpenGL 3.0 this year was rather surprising: it was delayed for almost a year without any notice at all, which is usually a clear sign that a project is dead. However, left aside the question if the OpenGL 3.0 release is the beginning of a new era or or just a last breath of the project, OpenGL 3.0 is now out in the wild and the Free Software community will adopt it sooner or later.

While Nvidia has already released a first version of an OpenGL 3.0 capable driver the FLOSS OpenGL implementation Mesa hasn’t released anything yet. But Mesa is alive and vibrant again since 2007 and a new release can be expected in the near future. Also likely is that AMD/ATI will release a new version of their OpenGL stack featuring the newest OpenGL spec. I would like to see AMD/TI team up with Mesa on that one but that’s just a wish, I’m afraid.

So in 2009 we will see OpenGL 3.0 coming to the masses – in proprietary as well as in Free drivers. This way newest graphics card technology will come to Linux and application developers can built upon that.

Gallium

Simply said, Gallium3D is an attempt to make graphics card driver development on Linux much easier: it abstracts the driver development from the underlying graphics standard implementation (for example OpenGL). Due to that abstraction, switching to another graphics standard should also be fairly easy. That way it should be easier to write one single graphics card driver for different devices (which do often need something else than OpenGL). And in case OpenGL is really dead, it could be a way to more or less painlessly replace it with something new.
Right now Gallium3D is in heavy development and we yet have to see it in the wild. There are only few drivers ported to it and I haven’t seen any distribution shipping it yet.

In 2009 this could switch: a first testing release for the broader masses is likely, and it could speed up the development of drivers for Gallium3D.

Gem and KMS

Speaking about graphics, there are other things which are in development and which are already surfacing here and there: the new graphics memory manager GEM. Using GEM the graphics cards does not have to be re-initialized as soon as you switch to another application. Also, everything will be written to the memory and the composition manager can simply access it there, avoiding some problems current drivers have when for example running videos on AIGLX.
Besides, Kernel Mode Setting (KMS) will move other tasks of the graphic subsystem away from X towards the kernel. As a result it will be much easier and flicker-free to switch from X to a tty console, and the graphical system will be able to show kernel oops. Linux will get its own blue screen capability, finally!

These features have partially found their way into newer Fedora releases, but only for specific hardware and under certain conditions. In 2009 it can be expected that the current FLOSS/Nvidia/AMD drivers will switch over to GEM and KMS to provide a much saner graphics experience to the user.

KDE 4.3: Pimp your PIM

Curently KDE’s PIM is in a difficult situation: Kontact is one of the best free groupware clients out there, but it was never designed to be one, and using it as such today can be an adventure. To fix that Akonadi was created. It was shipped with KDE 4.1 to back up Mailody, KDE 4.2 will see it the first time together with Kontact. This will give the developers quite some time to sanitize and improve the Akonadi service as well as to add new plugins to provide something revolutionary right in time for KDE 4.3.

In 2009 we will finally see a FLOSS groupware client which is working with a broad range of groupware servers, has a maintainable code base – and is perfectly integrated on all major platforms.

Qt on the mobile mass market

This year almost started with the news that Nokia acquired Trolltech. Recently it was announced that Qt now runs on Symbian S60. Also, with the iPhone, Google’s G1 and even a new Blackberry Nokia seriously needs a cool new device with fancy graphics and an appealing software platform.

Now put two and two together. With a bit of luck we will see the first Qt-Nokia devices with multi touch screen in 2009. With even a bit more of luck, it will be shipped in a way that Qt developers can use the tools they are used to to develop software for the new platform. Think of running KDE on these devices.

Gnome 3.0 development

In summer this year the Gnome developers started planning their next big release – Gnome 3.0. Currently not too many information have surfaced, but such breaks need their time. A state tracker for the Gtk+ changes is online and shows that indeed some work is underway already.

In 2009 first Alpha release could surface to show in which direction Gtk+ and Gnome are heading, and how the transition progress works out. That will definitely be an interesting time – the transition was a major task for KDE, and the Gnome team better takes a close look at that to learn from KDE’s experience.

Conclusion

While I already called 2007 the Year Of Open Source Graphics, 2009 can become a good candidate for it as well. In this post it got three paragraphs, and if everything comes true, 2009 will revolutionize the world of Linux graphics. This will, however, happen mostly under the hood. The users will not notice several fixes, but not the large underlying changes, which is different to 2007.

But in general 2009 will be exciting in almost all FLOSS areas. Keep in mind that this list is not and cannot be complete! So I ask every reader to drop a comment here containing his or her tip for revolutionary changes or news in the FLOSS world in 2009!

The background

3G is the so called third generation mobile technology, best known for it’s well known standard UMTS. Using that technology users are able to get quite fast Internet connections via mobile phone technology. In fact, in Germany the UMTS upload is currently faster than most of the ADSL connections you can get.

However, recently my company (credativ GmbH) was contacted by a customer who reported that the UMTS upload speed with Linux is rather slow compared with the upload speed on Windows. We did some tests and the result was disturbing: the factor is roughly two. While we had upload speeds of 1200 kbit/s on Windows XP, Linux only did less than 600 kbit/s (both using HSUPA). Hardware problems where out of question because we tested with a rather large set of UMTS hardware. We also tried different Linux distributions, different computers and so on. Additionally, the problem was at least verified by three (!) labs around the world: one lab of a major German mobile pone company and one lab each of two international hardware vendors. (And yes, it was me who tried to keep track of all the tests and people who are spread on several continents in different time zones ).

The fix

In the end, however, we were able to produce a rather simple patch to fix this problem. The upload speed in our test is now almost as large in Linux as in Windows, and this result was verified by the mentioned labs. The patch is:

--- drivers/usb/serial/option.c.old        2008-11-27 12:45:50.173275119 +0100
+++ drivers/usb/serial/option.c    2008-11-27 12:46:06.089274050 +0100
@@ -487,9 +487,9 @@
 /* per port private data */

 #define N_IN_URB 4
-#define N_OUT_URB 1
+#define N_OUT_URB 4
 #define IN_BUFLEN 4096
-#define OUT_BUFLEN 128
+#define OUT_BUFLEN 4096

 struct option_port_private {
        /* Input endpoints and buffer for this port */

At this point I’d like to plastinka who pointed me to that part of the kernel to fix the problem. I owe you something!

The problem: getting the fix upstream

The problem is now: how to get this fix upstream? I’m certainly not a kernel developer, and I could not even explain why this patch fixes the problem – I “just” have the test results. Joining the kernel list might also not be the best idea when I am not even a programmer.

So I decided to send the patch to the Red Hat guys – they have lots of developers, good connections to upstream, and I know how to talk to them. It’s bug #473252, let’s see what happens. In case you experience similar problems, send this patch upstream via your distribution, maybe that helps as well.

64bit in General

Switching from 32bit to 64bit on an operating system is a huge and complicated task involving effectively all larger applications. This can e a real pain – unless you have an operating system where all software usually supports 64bit anyway. This is the case with most open source operating systems and therefore also with Linux. So grabbing the 64bit image and installing it was just like grabbing the 32bit image. In case of Fedora the download link offered by default was 32bit, but 64bit was just a click away. I wonder when that will change.

There are numerous advantages and disadvantages regarding 32bit and 64bit, for a first introduction start with the Wikipedia article.

Flash

The problems regarding 64bit arise when you deal with non-Open Source software: this might only be provided as 32bit. In case it depends on any other library, the system must provide these libraries in 32bit and 64bit. While on RPM systems this is not a problem at all, this can be rather problematic when browser plugins are 32bit only, because then the browser needs to be 32bit only as well, the same is true then for all other plugins, and so on. There are wrappers to deal with that, but these are sub-optimal.

Luckily, Adobe has now released a 64bit Alpha version of their Flash player. While it is still missing several features and is not even provided as a rpm or deb file, in my first tests it worked without problems. As a side note, the 64bit versions for Windows and Mac OS are still not out there – Linux is a clear technology and development pusher here!

For the sake of completion (and since someone would point it out in a comment anyway), there are also free (as in FLOSS) alternatives to the Flash player – which are available in 64bit for quite some time now, of course.

Skype

Another issue is Skype – this is not provided as a 64bit version at all (bug report). For Ubuntu users there is at least a 32bit version modified for easy installation on 64bit systems. Btw., hardly anyone seems to know that, even the German Ubuntu wiki doesn’t mention that at all.

Anyway, that doesn’t help the Fedora community anyway – but since Fedora runs on RPM installing all the compatibility libraries is just a question of hard disk space:

yum --nogpgcheck localinstall skype*rpm
yum install alsa-plugins-pulseaudio-1.0.18-1.rc3.fc10.i386

PulseAudio

Well, PulseAudio is a difficult thing. It has a rather strong community and people are making sure it comes up everywhere and works everywhere like it should. However, while I read all the rather long papers and documents why I should need PA, nothing of these papers really stuck, and I always wonder why it is really needed – apart from the more esoteric reasons that Alsa is not suitable for the future. Besides, I do wonder if the Alsa guys would say the same.
Additionally, in my first tests PA worked just fine – under Gnome, but not in KDE. So my first step after installing Fedora since my first contact with PA was always to remove PA.

But I always tend to give things another try, and this time I didn’t remove it. And indeed, almost everything works, and I haven’t even met a delay yet. It all just works, even on KDE with its Phonon!

So it looks like PA finally fits in well. Now what I only need some ideas what to do with it Seriously, what I would appreciate to see is a simple one sheet drawing with all the usual suspects of the Linux audio blob (from Phonon over xine down to Alsa) to see where PA fits in and what it does there.

KDE, Nvidia and performance

Fedora 10 now includes KDE 4.1.3 (included in the updates), and together with RPM Fusion Nvidia drivers are just a

yum install kmod-nvidia

I am slightly surprised and wonder what was wrong with my Fedora 10 Beta setup. But on the other hand, my work machine is running Kubuntu 8.10 and there the performance is similar fast. So to me it looks like the days of slow KDE 4.x on Nvidia hardware are finally over, given that the drivers are the newest stable ones and KDE is of version 4.1.3.

Fedora’s encryption

This time I decided to not go with a full hard disk encryption, but rather with a home disk encryption. And while I still dislike Fedora’s disk druid for not letting me chose the disk setup in detail I appreciate that clicking a checkbox was all I had to do to activate the home partition encryption. It is even nicely integrated with the boot process.

Overall impression

The overall impression of Fedora 10 is very good. Most bugs I encountered running the Beta version are fixed – except for a strange coding problem, but I will survive that one.

Also, my first move into the lands of 64bit are also far less complicated than expected. Your mileage may vary, depending on the used proprietary software, but then again kvm might be a solution to work around that problem.

RPM Fusion was announced more than a year ago and is the attempt to merge several of the many existing 3rd party repositories for Fedora (Dribble, Freshrpms,. livna). The idea of the common repository is to enhance the compatibility as well as avoid duplication – in the former time many packages were provided by more than one repository. The aim of the new RPM Fusion is however just the same as it was for the original repos: provide high quality packages of software which cannot be part of Fedora due to licence/patent problems.

There was a lot of development going on in the background, but RPM Fusion wasn’t ready for Fedora 9 unfortunately – there was just too much to do: importing the packages, sorting out incompatibilities, creating a common build- and mirror-system, etc. Now most of the work is done and the new repository is ready for testing – more details can be found in Thorsten Leemhuis’ blog post. There you will also find ready-to-install repository packages.

With the new repository the situation for users regarding mixed repositories and the possible incompatibilities will hopefully improve a lot. Also, new users don’t have to make a choice between the repositories. And last but not least, with a common and clear infrastructure RPM Fusion will hopefully attract more users to submit their packages to the project so that RPM Fusion grows even more.

• New NetworkManager with connection sharing
• Improved printer handling
• Remote virtualization and easier virt storage
• Sectool, an auditing and security testing framework
• RPM 4.6, the first big RPM change in several years

Especially the first and the third part are pretty important to me since prefer to have networking “made easy” and also plan to get more experience regarding virtual machines. However, for me all the other goodies from Fedora 9 are also new since I never came around using Fedora 9 after some very bad experiences. So I decided to take the beta and give it a try with all goodies together: KDE 4.1, encrypted main partitions, kvm, new NetworkManager, PackageKit, etc.

And so far I must say I’m very pleased. The system pretty much works and KDE 4.1.x is really amazing. Of course, there are still quite some glitches, but after all, Fedora 10 is still in development, and many glitches seem to be X related and and my Nvidia card with the binary drivers is certainly a good candidate to be the source of some problems.

Some glitches

The main glitch in the system is the same reason why I haven’t yet filled bug reports against the glitches: I have no realiable browser. Firefox is crashing all the time with segmentation faults (unusable for more than maybe two minutes, but very hard to exactly reproduce), Konqueror sometimes seems to kill the DNS/internet connection somehow (!), and Arora often takes seconds to start actually loading a page – and additionally cannot log me in to my wordpress.com account. The fact that the NetworkManager applet (Gnome one) dies occasionally isn’t helping.
Another problem right at the start was that the GDM login manager somehow didn’t realize that I switched the keyboard layout to German. It took me quite some time to figure that out. Also, if I deactivate quiet boot and rhgb (and I always do that) the password dialog for my encrypted file system is lost in the kernel output.

The good things

Besides these annoying glitches (well, Beta is Beta) the system is stable – and promising: I’m looking forward to test the network connection sharing, which seems to be nicely integrated with the rest of the system. Also, the encryption is of course very important – and since Luks is used it would make sense to provide a GUI to easily add and remove other keys for the decryption. Last but not least KDE 4.1 is important for me – maybe the most important reaon of all. Even with Nvidia drivers it is working surprisingly well, but more about that later.

Conclusion

To summarize, Fedora 10 is shaping up quite nicely, and the Beta already runs much better for me than the Fedora 9 release. However, I wouldn’t advise normal users to start using Fedora 10 because it is still in heavy development – too much is still changing at the moment. I for example got an update right before the weekend which made it impossible to run the binary Nvidia driver on the system and had strange side effects when I connected an external monitor – sad if you want to watch DVD in the evening and show off KDE 4′s bling the next morning.

# cat /var/log/Xorg.0.log|grep -i nvidia
(--) PCI:*(0@1:0:0) nVidia Corporation Quadro NVS 135M rev 161, Mem @ 0xfd000000/0, 0xe0000000/0, 0xfa000000/0, I/O @ 0x0000ef00/0, BIOS @ 0x????????/131072
(II) NV: driver for NVIDIA chipsets: RIVA 128, RIVA TNT, RIVA TNT2,
(--) NV: Found NVIDIA Quadro NVS 135M at 01@00:00:0

# cat /var/log/Xorg.0.log|grep -i -e TNT -e nvidia
(--) PCI:*(0@1:0:0) nVidia Corporation Quadro NVS 135M rev 161, Mem @ 0xfd000000/0, 0xe0000000/0, 0xfa000000/0, I/O @ 0x0000ef00/0, BIOS @ 0x????????/131072
(II) NV: driver for NVIDIA chipsets: RIVA 128, RIVA TNT, RIVA TNT2,
        Unknown TNT2, Vanta, RIVA TNT2 Ultra, RIVA TNT2 Model 64,
        Aladdin TNT2, GeForce 256, GeForce DDR, Quadro, GeForce2 MX/MX 400,
(--) NV: Found NVIDIA Quadro NVS 135M at 01@00:00:0

This is a typical problem I often thought about but never got around to look it up. And there are surprisingly few people out there who now that feature! And yes, I know that cat should not be used that way, but it’s hard to forget once you’ve learned it wrong.

# chkconfig --list|grep httpd
httpd           0:Aus   1:Aus   2:Aus   3:Aus   4:Aus   5:Aus   6:Aus
# chkconfig --level 345 httpd on
# chkconfig --list|grep httpd
httpd           0:Aus   1:Aus   2:Aus   3:Ein   4:Ein   5:Ein   6:Aus
# chkconfig --level 345 httpd off
# chkconfig --list|grep httpd
httpd           0:Aus   1:Aus   2:Aus   3:Aus   4:Aus   5:Aus   6:Aus

# sysv-rc-conf --list|grep cups
cupsys           1:off   2:off   3:off   4:off   5:off
# sysv-rc-conf --level 345 cupsys on
# sysv-rc-conf --list|grep cups
cupsys           1:off   2:off   3:on   4:on   5:on

A NFS/hostname example

In this example most servers just get the same setup, but only few servers are monitoring servers which require very specific additions. Additionally, a NFS server is already available on the net (maybe to deliver the kickstart files), and the machines are distinguished by their hostname.

First, the specific additions need to be placed in a shell script. The shell script(s) are named after the hostname of the corresponding host and are copied to the NFS share. Second, the kickstart file gets a section to mount the NFS share right after the installation. To avoid problems with NFS support in the newly installed machines, the share is mounted right before the default %post section in a special section which has not changed the root yet:

%post --nochroot
mkdir -p /mnt/sysimage/media/nfsshare
mount myserver.com:/nfsexport /mnt/sysimage/media/nfsshare

Afterwards, the normal %post section follows as usual, but has access to the NFS share in /media/nfsshare.

Next, the kickstart file needs a function in %post to first check if the host name is set, if a file of that name exists on the server, and if yes, copy and launch it. This requires of course the preparations explained in part one to filter for the boot parameters.

if test "${myop_hostname+set}" = set ; then
  if [ -e /media/nfsshare/$myop_hostname ]; then
    cp /media/nfsshare/$myop_hostname /root/.
    chmod +x /root/$myop_hostname
    /bin/sh /root/$myop_hostname
  fi
fi

The kickstart installation can can now be called with the hostname as a boot parameter: linux ks=nfs:/myserver.com/ks.cfg myop_hostname=blueserver. If no hostname is given, host-specific configuration is ignored.

Other ways

Of course there is no need to use a NFS server, or a server it all: the different shell scripts can be provided directly on the installation medium, or on a web server. Also, using host based configuration makes it more difficult again to maintain all the different scripts for all the different servers. If the number of servers grow above a certain number the best way would be to manage all the options, scripts and parameters in a database and export host-based configuration files when needed or provide it via ldap.

Extra: Tiny installation

Quite often it happens that people only need small Fedora/RHEL setups. However, even with no arguments in the %packages section a default installation takes quite some space. This is due to the fact that in the default configuration kickstat installs all packages from the base group. A special flag must be set to avoid that. Together with that flag, a base installation requiring “only” roughly 160 packages looks like:

%packages --nobase
openssh-clients
openssh-server
ntp
yum
dhclient
sendmail
man
-system-config-securitylevel-tui