Flatpak – a solution to the Linux desktop packaging problem [Update]

Linux packaging was a nightmare for years. But recently serious contenders came up claiming to solve the challenge: first containers changed how code is deployed on servers for good. And now a solution for the desktops is within reach. Meet Flatpak!

Preface

In the beginning I probably should admit that over the years I identified packaging within the Linux ecosystem as a fundamental problem. It prevented wider adoption of Linux in general, but especially on the desktop. I was kind of obsessed with the topic.

The general arguments were/are:

  • Due to a missing standard it was not easy enough for developers to package software. If they used one of the formats out there they could only target a sub-set of distributions. This lead to lower adoption of the software on Linux, making it a less attractive platform.
  • Since Linux was less attractive for developers, less applications were created on/ported to Linux. This lead to a smaller ecosystem. Thus it was less attractive to users since they could not find appealing or helpful applications.
  • Due to missing packages in an easy-accessible format, installing software was a challenge as soon as it was not packaged for the distribution in use. So Linux was a lot less attractive for users because few software was available.

History, server side

In hindsight I must say the situation was not as bad as I thought on the server level: Linux in the data center grew and grew. Packaging simply did not matter that much because admins were used to problems deploying applications on servers anyway and they had the proper knowledge (and time) to tackle challenges.

Additionally, the recent rise of container technologies like Docker had a massive impact: it made deploying of apps much easier and added other benefits like sandboxing, detailed access permissions, clearer responsibilities especially with dev and ops teams involved, and less dependency hell problems. Together with Kubernetes it seems as there is an actual standard evolving of how software is deployed on Linux servers.

To summarize, in the server ecosystem things never were as bad, and are quite good these days. Given that Azure serves more Linux servers than Windows servers there are reasons to believe that Linux is these days the dominant server platform and that Windows is more and more becoming a niche platform.

History, client side

On the desktop side things were bad right from the start. Distribution specific packaging made compatibility a serious problem, incompatible packaging formats with RPMs and DEBs made it worse. One reason why no package format ever won was probably that no solution offered real benefits above the other. Given today’s solutions for packaging software out there RPM and DEB are missing major advantages like sandboxing and permission systems. They are helplessly outdated, I question if they are suited for software packaging at all today.

There were attempts to solve the problem. There were attempts at standardization – for example via the LSB – but that did not gather enough attraction. There were platform agnostic packaging solutions. Most notably is Klik which started already 15 years ago and got later renamed to AppImage. But despite the good intentions and the ease of use it never gained serious attention over the years.

But with the approach of Docker things changed: people saw the benefits of container formats and the technology technology for such approaches was widely available. So people gave the idea another try: Flatpak.

Flatpak

Flatpak is a “technology for building and distributing desktop applications on Linux”. It is an attempt to establish an application container format for Linux based desktops and make them easy consumable.

According to the history of Flatpak the initial idea goes way back. Real work started in 2014, and the first release was in 2015. It was developed initially in the ecosystem of Fedora and Red Hat, but soon got attention from other distributions as well.

Many features look somewhat similar to the typical features associated with container tools like Docker:

  • Build for every distro
  • Consistent environments
  • Full control over dependencies
  • Easy to use  build tools
  • Future-proof builds
  • Distribution of packages made easy

Additionally it features a sandboxing environment and a permissions system.

The most appealing feature for end users is that it makes it simple to install packages and that there are many packages available because developers only have to built them once to support a huge range of distributions.

By using Flatpak the software version is also not tied to the distribution update cycle. Flatpak can update all installed packages centrally as well.

Flathub

One thing I like about Flatpak is that it was built with repositories (“shops”) baked right in. There is a large repository called flathub.org where developers can submit their applications to be found and consumed by users:

The interface is simple but has a somewhat proper design. Each application features screenshots and a summary. The apps themselves are grouped by categories. The ever changing list of new & updated apps shows that the list of apps is ever growing. A list of the two dozen most popular apps is available as well.

I am a total fan of Open Source but I do like the fact that there are multiple closed source apps listed in the store. It shows that the format can be used for such use cases. That is a sign of a healthy ecosystem. Also, there are quite a few games which is always good 😉

Of course there is lots of room for improvement: at the time of writing there is no way to change or filter the sorting order of the lists. There is no popularity rating visible and no way to rate applications or leave comments.

Last but not least, there is currently little support from external vendors. While you find many closed source applications in Flathub, hardly any of them were provided by the software vendor. They were created by the community but are not affiliated with the vendors. To have a broader acceptance of Flatpak the support of software vendors is crucial, and this needs to be highlighted in the web page as well (“verified vendor” or similar).

Hosting your own hub

As mentioned Flatpak has repositories baked in, and it is well documented. It is easy to generate your own repository for your own flatpaks. This is especially appealing to projects or vendors who do not want to host their applications themselves.

While today it is more or less common to use a central market (Android, iOS, etc.) some still prefer to keep their code in there hands. It sometimes makes it easier to provide testing and development versions. Other use cases are software which is just developed and used in-house, or the mirroring of existing repositories for security or offline reasons: such use cases require local hubs, and it is no problem at all to bring them up with Flatpak.

Flatpak, distribution support

Flatpak is currently supported on most distributions. Many of them have the support built in right from the start, others, most notably Ubuntu, need to install some software first. But in general it is quite easy to get started – and once you did, there are hundreds of applications you can use.

What about the other solutions?

Of course Flatpak is not the only solution out there. After all, this is the open source world we are talking about, so there must be other solutions 😉

Snap & Snapcraft

Snapcraft is a way to “deliver and update your app on any Linux distribution – for desktop, cloud, and Internet of Things.” The concept and idea behind it is somewhat similar to Flatpak, with a few notable differences:

  • Snapcraft also target servers, while Flatpak only targets desktops
  • Among the servers, Snapcraft additionally has iot devices as a specific target group
  • Snapcraft does not support additional repositories; there is only one central market place everyone needs to use, and there is no real way to change that

Some more technical differences are in the way packages are built, how the sandbox work and so on, but we will noch focus on those in this post.

The Snapcraft market place called snapcraft.io provides lists of applications, but is much more mature than Flathub: it has vendor testimonials, features verified accounts, multiple versions like beta or development can be picked from within the market, there are case stories, for each app additional blog posts are listed, there is integration with social accounts, you can even see the distribution by countries and Linux flavors.

And as you can see, Snapcraft is endorsed and supported by multiple companies today which are listed on the web page and which maintain their applications in the market.

Flathub has a lot to learn until it reaches the same level of maturity. However, while I’d say that snapcraft.io is much more mature than Flathub it also misses the possibility to rate packages, or just list them by popularity. Am I the only one who wants that?

The main disadvantage I see is the monopoly. snapcraft.io is tightly controlled by a single company (not a foundation or similar). It is of course Canonical’s full right to do so, and the company and many others argue that this is not different from what Apple does with iOS. However, the Linux ecosystem is not the Apple ecosystem, and in the Linux ecosystem there are often strong opinions about monopolies, closed source solutions and related topics which might lead to acceptance problems in the long term.

Also, technically it is not possible to launch your own central server for example for in-house development, or for hosting a local mirror, or to support offline environments or for other reasons. To me this is particularly surprising given that Snapcraft targets specifically iot devices, and I would run iot devices in an closed network wherever I can – thus being unable to connect to snapcraft.io. The only solution I was able to identify was running a http proxy, which is far from the optimal solution.

Another a little bit unusual feature of Snapcraft is that updates are installed automatically, thanks to theo.9dor for the hint:

The good news is that snaps are updated automatically in the background every day! 

https://tutorials.ubuntu.com/tutorial/basic-snap-usage#2

While in the end a development model with auto deployments, even dozens per day, is a worthwhile goal I am not sure if everyone is there yet.

So while Snapcraft has a mature market place, targets much more use cases and provides more packages to this date, I do wonder how it will turn out in the long run given that we are talking about the Linux ecosystem here. And while Canonical has quite some experience to develop their own solutions outside the “rest” of the community, those attempts seldom worked out.

AppImage

I’ve already mentioned AppImage above and I’ve written about it in the past when it was still called Klik. AppImage is “way for upstream developers to provide native binaries for Linux”. The result is basically a file that contains your entire application and which you can copy everywhere. It exists for more than a dozen years now.

The thing that is probably most worth mentioning about it is that it never caught on. After all, already long time ago it provided many impressive features, and made it possible to install software cross distribution. Many applications where also available as AppImage – and yet I never saw wider adoption. It seems to me that it only got traction recently because Snapcraft and Flatpak entered the market and kind of dragged it with them.

I’d love to understand why that is the case, or have an answer to the “why”. I only have few ideas but those are just ideas, and not explanations why AppImage, in all the years, never managed to become the Docker of the Linux desktop.

Maybe one problem was that it never featured a proper store: today we know from multiple examples on multiple platforms that a store can mean the difference. A central place for the users to browse, get a first idea of the app, leave comments and rate the application. Docker has a central “store”, Android and iOS have one, Flatpak and Snapcraft have one. However, AppImage never put a focus on that, and I do wonder if this was a missed opportunity. And no, appimage.github.io/apps is not a store.

Another difference to the other tools is that AppImage always focused on the Open Source tools. Don’t get me wrong, I appreciate it – but open source tools like Digikam were available on every distribution anyway. If AppImage would have focused to reach out to closed source software vendors as well, together with marketing this aggressively, maybe things would have turned out differently. You do not only need to make software easily available to users, you also need to make software available the people want.

Last but not least, AppImage always tried to provide as many features as possible, while it might have benefited from focusing on some and marketing them stronger. As an example, AppImage advertises that it can run with and without sandboxing. However, sandboxing is a large benefit of using such a solution to begin with. Another thing is integrated updates: there is a way to automatically update all appimages on a system, but it is not built in. If both would have been default and not optional, things maybe would have been different.

But again, these are just ideas, attempts to find explanations. I’d be happy if someone has better ideas.

Disadvantages of the Flatpak approach

There are some disadvantages with the Flatpak approach – or the Snapcraft one, or in general with any container approach. Most notably: libraries and dependencies.

The basic argument here is: all dependencies are kept in each package. This means:

  • Multiple copies of the same libraries on the system, leading to larger disk consumption
  • Multiple copies of the same library in the RAM during execution, leading to larger memory consumption
  • And probably most important: if a library has a security problem, each and every package has to be updated

Especially the last part is crucial: in case of a serious library security problem the user has to rely on each and every package vendor that they update the library in the package and release an updated version. With a dependency based system this is usually not the case.

People often compare this problem to the Windows or Java world were a similar situation exists. However, while the underlying problem is existent and serious, with Flatpak at least there is a sandbox and a permission system something which was not the case in former Windows versions.

There needs to be made a trade off between the advanced security through permissions and sandboxing vs the risk of having not-updated libraries in those packages. That trade off is not easily done.

But do we even need something like Flatpak?

This question might be strange, given the needs I identified in the past and my obvious enthusiasm for it. However, these days more and more apps are created as web applications – the importance of the desktop is shrinking. The dominant platform for users these days are mobile phones and tablets anyway. I would even go so far to say that in the future desktops will still be there but mainly to launch a web browser

But we are not there yet and today there is still the need for easy consumption of software on Linux desktops. I would have hoped though to see this technology and this much traction and distribution and vendor support 10 years ago.

Conclusion

Well – as I mentioned early on, I can get somewhat obsessed with the topic. And this much too long blog post shows this for sure 😉

But as a conclusion I say that the days of difficult-to-install-software on Linux desktops are gone. I am not sure if Snapcraft or Flatpak will “win” the race, we have to see that.

At the same time we have to face that desktops in general are just not that important anymore.  But until then, I am very happy that it became so much easier for me to install certain pieces of software in up2date versions on my machine.

Advertisements

Current distribution of WhatsApp alternatives [Update]

Android_robotMany people are discussing alternatives to WhatsApp right now. Here I just track how many installations the currently discussed, crypto-enabled alternatives have according to the app store.

WhatsApp was already bad before Facebook acquired it. But at least now people woke up and are considering secure alternatives. Yes, this move could have come earlier, but I do welcome the new opportunity: its the first time wide spread encryption actually has a chance in the consumer market. So for most of the people out there the question is more “which alternative should I use” instead of “should I use one”. Right now I do not have the faintest idea which alternative with crypto support will make the break through – but you could say I am well prepare.

Screenshot installed instant messengers
Screenshot installed instant messengers

Well – that’s obviously not a long term solution. Thus, to shed some light on the various alternatives and how they stand right now, here is a quick statistical overview:

Secure Instant Messengers, state updated 2014-03-11
Name WebPage/GooglePlay installed devices Ratings Google +1
ChatSecure Website / Google Play 100 000 – 500 000 1 626 2 620
Kontalk Website / Google Play 10 000 – 50 000 237 265
surespot Website / Google Play 50 000 – 100 000 531 632
Telegram Website / Google Play 10 000 000 – 50 000 000 273 089 97 641
Threema Website / Google Play 500 000 – 1 000 000 9 368 12 594
TextSecure Website / Google Play 100 000 – 500 000 2 478 2 589

The statistics are taken from Google’s Android Play Store. I would love to include iTunes statistics, but it seems they are not provided via the web page. If you know how to gather them please drop me a note and I’ll include them here.

These numbers just help to show how fat an application is spread – it does not say anything about the quality. For example Threema is not Open Source and thus not a real alternative. So, if you want to know more details about the various options, please read appropriate reviews like the one from MissingM.

Android 4.4 now *can* sync multiple calendars via ActiveSync

Android_robotWith the release of Android 4.4 called KitKat Google made some interesting changes to their ActiveSync implementation: the code is now set up to sync more than one calender, and the first KitKat user already confirmed that new feature.

In February I described in a blogpost why Android cannot sync multiple calendars via ActiveSync. The problem was that Google did not implement the necessary parts of the ActiveSync specification in Android.

However, that seems to have changed: if you look at the current ActiveSync implementation of Android 4.4 KitKat, the source code (tag 4.4rc1) does list support for multiple calendars – and also for multiple address books:

        MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_CALENDAR, Mailbox.TYPE_CALENDAR);
        MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_CONTACTS, Mailbox.TYPE_CONTACTS);

I had no chance yet to test that on my own, but there are reports that it is indeed working:

Today i flashed a Android 4.4 Rom on my smartphone. After adding the Exchange Profile all my Calendars are there […]
I’ve uploaded a screenshot here:
http://postimg.org/image/5d4u364ub/

Looks like Google actually listened to…erm, corporate users? At least to someone, though 😉

But: Since I have no first-hand-experience in this regard I would like to ask all of my nine readers out there if anyone has a stock KitKat running and if the could check this feature. Please test this and leave a report about your experiences in the comments. I will include it in the article.

By the way, the above mentioned source code snippet also tells quite exactly which other ActiveSync functions are not yet supported in Android:

        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_TASKS,  Mailbox.TYPE_TASKS);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_NOTES, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_JOURNAL, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_TASKS, Mailbox.TYPE_TASKS);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_JOURNAL, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_NOTES, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_UNKNOWN, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(MAILBOX_TYPE_RECIPIENT_INFORMATION_CACHE, Mailbox.TYPE_NONE);

I guess syncing tasks could come in handy in corporate environments. Combined with support for multiple task folders you could even design your own Kanban “board” that way.

Nevertheless I’d like to add that ActiveSync is no big deal for me anymore because I am very happy with a – albeit 3rd party and not yet Open Source – CalDav implementation, which can even sync multiple task folders.

Google continues CalDav support for everyone, now also adds CardDav

Android_robotYesterday Google announced that it will not restrict the CalDav access to their calendars to registered partners only, but that they will continue to provide it for everyone. Additionally, Google now offers CardDav support.

A couple of weeks ago Google announced that they would restrict CalDav access to their calendars to registered developers only. That resulted in a huge uproar among developers, users and open standards advocates and made many people wondering if Google will become a closed standards/software company in the future.

However, the pressure (and most likely the bad press and reputation) Google got worked, and they announced that the CalDav API will be continued as an API open for everyone:

In response to those requests, we are keeping the CalDAV API public.

And it becomes even better: CardDav support is added as well, meaning the address data can be accessed via open protocols as well:

And in the spirit of openness, today we’re also making CardDAV – an open standard for accessing contact information across the web – available to everyone for the first time.

This way CalDav and CardDav have an even better chance to become THE royalty free and open alternative to Microsoft’s ActiveSync protocol. Additionally, application developers don’t have to worry to add special code to support Google calendars and address books: they just add CalDav and CardDav support and they automatically support almost all groupware servers and services available.

This is good news and gives me back some trust in Google’s policies and priorities. There is still no CalDav or CardDav support in Android, yes – but at least the server side is better now.

[Howto] Installing Owncloud News, a self hosted RSS reader

two_glossy_cloudsOwncloud News, a RSS news reader for the self hosting cloud service Owncloud, is available in an Alpha version. That comes right at the time Google Reader is bound to see its end soon.

I must admit that I do not understand why Google decided to shut down the Google Reader service. Social media with their unstructured news areas are nice, but no match to a well structured news feed full of read and unread news. But, there are replacements, and one pretty wise choice would be to not depend on yet another web service, but to host it yourself.

In comes Owncloud: it can already host your addresses, calendars, files and musik and can be integrated with your desktop as well. Now a RSS reader app, Owncloud News was released as an Alpha version, and indeed already looks promising:

Owncloud-Reader-General

The installation is pretty smooth as well. The requirements are a running Owncloud 5 version, so 4.5 won’t do it. The installation itself basically consists of two steps: installing and activating the so called App Framework, which is supposed to be the foundation for other Owncloud apps in the future, and afterwards installing the news app itself:

# cd /var/www
# git clone https://github.com/owncloud/appframework.git
Cloning into 'appframework'...
[...]
# git clone https://github.com/owncloud/news.git
Cloning into 'news'...
[...]

I choose /var/www here because it is recommended in the manual and because there the appropriate user has the necessary access rights. But it could be any dir, since you only link the plugins anyway:

# ln -s /var/www/appframework /var/www/owncloud/apps
# ln -s /var/www/news /var/www/owncloud/apps

Speaking about rights, make sure the web server can write cache files:

# sudo chown -R www-data:www-data /var/www/news/cache

Afterwards, login to your owncloud, and active the plugins: first the framework, followed by the actual application. Add feeds, play around, as you will see it works pretty nice.

What is still missing right now is an Android news reader which could sync with the server. When that is available as well, Owncloud News might become *the* Google Reader descendant.

Pass – A perfect shell based password manager

920839987_135ba34fffPass is a tool to store and manage passwords and other data securely and on command line – even with built in support for Git and remote Git repositories. Thus it is a welcomed alternative for existing password managers which often require a GUI, or do not provide repository support.

What it is

Pass is a shell based password manager to store passwords and login data – or anything you want, actually. The name “the standard unix password manager” however is pretty misleading: the author wanted to stress that it only uses standard Unix tools, but failed to highlight that with a catchy name and instead just created confusion.

But the author is right with his main point: pass is in fact just gluing together already well known and tested Unix tools: the encryption of all information is ensured by GPG, passwords are queried using gpg-agent, the version control and remote repository support is done by Git, and the tool itself is written in shell code. Thus you have features you can rely on – in fact, if you want you can directly access the Git repository and the Gnupg files, you do not have to use Pass at all.

Pass stores information in simple files, which can be grouped in folders. While the main idea of Pass is to store one password in one file you can actually access each file with editors to store as many information in it as you want. Each file is encrypted with the gpg key which was defined during the initial setup of Pass. As a result the Pass database is nothing else but a folder full of other folders and gpg encrypted files:

$ ls -1 $HOME/.password-store
business
commerce
financial
$ ls -1 $HOME/.password-store/business/
linkedin.com.gpg
example.com.gpg
important.com.gpg

Pass is included in all major distributions like Fedora, Ubuntu, Debian, and so on, and thus can be installed with the usual package management tools.

How it works

If you call Pass without any further options, it just outputs the content of its password store:

$ pass
Password Store
|-- business
|   |-- linkedin.com
|   |-- example.com
|   `-- important.com
|-- commerce
|   `-- amazon
|-- financial
|   |-- badbank.com
|   `-- mybank.com

The file type ending “gpg” is not shown here to not confuse users (I guess).

Showing the content of a file is straight forward:

$ pass business/example.com
login:  example
pass:   password

Adding new entries can be done with the command pass insert $FOLDER/$FILENAME. But it might be more convenient to just use the default editor to edit a new file: pass edit $FOLDER/$FILENAME. That way multi line information can be added more easily.

However, the real strength of Pass is that after each change – like adding a new password – git-add and git-commit are called: the new file is automatically committed to a local git repository:

$ pass edit business/example.com
[master 4c09c76] Added password for business/example.com using /usr/bin/vim.
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 business/example.com.gpg

As a result all changes are automatically under version control and can be reverted. But it gets better: Pass forwards arbitrary options and commands to Git itself. Thus it is possible to access the full functionality of Git – and to push the files to an online repository:

$ pass git push
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 823 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
To git@example.net:pass
   aa2aff7..2011296  master -> master

That way the password store can be shared with any remote Git repository – and thus can be re-used by other clients, given that they have the proper GPG key.

Missing pieces

As shown above Pass is almost perfect if you need a way to manage passwords (or any other data at all) on command line level, including repository and encryption support.

But while Pass replaced all my other password managers literally in a few minutes there is still one big feature I miss: the support for GUI tools! It would be nice if Pass support could be included in the major Desktop Environments and major GUI programs used in the Linux desktop world:

  • KDE’s Kwallet
  • Gnome’s Keyring
  • Android
  • Firefox
  • Chrome/Chromium

To summarize it: Pass is great, but would be even better if it could server as a backend for the usual GUI tools and desktop environments. There is already an experimental iOS client, so there is at least hope for an Android client…

Google & ActiveSync, Microsoft & CalDav: Pure irony

Android_robotToday Microsoft announced plans to implement CalDav and CardDav support in Windows Phone. That will enable users to still sync with Google services once these shut down their ActiveSync support in Summer. That is highly ironic and almost ridiculous, since Google itself does not support CalDav and CardDav in Android.

It all started with Google’s Winter cleaning: Google announced a couple of weeks ago that their services will soon be no longer offer an ActiveSync interface. That means: all client devices accessing Google’s services via ActiveSync need to switch to some other way of synching. Btw., read carefully: this has nothing to do with Android. Not at all! Also, iPhones don’t have to bother because they can simply switch to CalDav and CardDav which is natively supported in iOS. However, id does affect users of Microsoft’s Windows Phone. They only had ActiveSync as an option.

Now Microsoft announced they are going to implement CardDav and CalDav support in their Windows Phone. So that users can happily sync their Windows Phones with Google services.

And here comes the irony: Google itself does not support CalDav nor CardDav on client side. Google’s Android operating system does not offer it, not at all! Google only supports its own, proprietary sync way used in the Google apps, and has support for ActiveSync, albeit pretty limited support.

So, to summarize: Google forces others to use open standards which they do not support themselves.

While it is good that Microsoft is forced to implement open standards, Google’s acting nevertheless looks ridiculous, that is just sad. I wish Google would have the guts to just add CardDav and CalDav support and have a party with the people fighting for open standards. I mean, how bad would it look like if a Microsoft operating system would support open standards better than a Google operating system?