Impressions from #AnsibleFest London 2016 [Update]

Ansible LogoThe #AnsibleFest was taking place in London, and I was luckily able to attend. This post shares some impressions from the event, together with interesting announcements and stories.

Update: The slides of the various presentations are now available.

Preface

The #AnsibleFest London 2016 took place near the O2 Arena and lasted the entire day. The main highlight of the conference was the network automation coming along with Ansible now. Other very interesting talks covered very helpful tips about managing Windows Servers, the 101 on modules, how to implement continuous deployment, the journey of a french bank towards DevOps, how Cisco devices can be managed and how to handle immutable infrastructure. All focused on Ansible, of course.

But while the conference took place during Thursday, the #AnsibleFest started already the evening before: at the social event Ansible Social.
Ansible Social
And it was a wonderful evening: many people from Ansible, partners, coworkers from Red Hat and others were there to enjoy drinks, food and chatting through the evening. Getting to know many of the people there went pretty well, it was a friendly bunch meeting at a pretty nice place.
Ansible Social

Keynote

Upon arrival at the conference area one of the sponsor desks immediately caught the eye: Cisco!
20160218_084833-01
For everyone following Ansible news closely it was obvious that networking would be a big topic, especially since it was about to be featured twice during the day, once by Peter Sprygada from Ansible and later on by Fabrizio Maccioni from Cisco.

And this impression was confirmed when Todd Barr came to the stage and talked about the current state of Ansible and what to expect in the near future: networking is a big topic for Ansible right now, they are pushing resources into the topic and already hinted that there would be a larger announcement during the #AnsibleFest. During the presentation the strengths of Ansible were of course emphasized again: that it is simple to setup, to understand and to deploy. And that it does not require agents. While I do have my past with Puppet and still like it as a tool in certain circumstances, I must admit that I had to smile at the slide about agents.
Todd Barr at AnsibleFest
I have to admit, for many customers and many setups this is in fact true: they do not want agents for various reasons. And Ansible can deliver actual results without any need for a client.

The future of Ansible

Next up was Bill Nottingham talking setting the road for the future of Ansible. A focus is certainly better integration of Windows (no beta tag anymore!), better testing – and Python 3 support! It was acknowledged that there are more and more distributions out there not providing any Python 2 anymore and that they need to be catered.
Future of Ansible by Bill Nottingham
Ansible Tower was also covered, of course, and has very promising improvements coming up as well: the interface will be streamlined, the credentials and rights system will be improved, and there will be (virtual) appliances to get Ansible Tower out of the box in an instant. But the really exciting part is more large-scale, enterprise focused: Ansible Tower will be able to cater federated setups, meaning distributed replication of Ansible Tower commands via proxy Towers.
Federated Ansible Tower
Don’t expect this all in the next weeks, but we might see many of these features already in Ansible Tower 3.0. And it was mentioned that there might be a release in early fall.

Scaling abilities are indeed needed – many data centers these days have more than one location, or are spread over several departments and thus need partially independent setups to manage the infrastructure. At the same time, there are Ansible customers who are using Ansible with 50k nodes and more out there, and they have a demand for fine grained, federated infrastructure setups as well.

Networking with Ansible

While the upcoming Ansible Tower had some exciting news, the talk about networking support by Peter Sprygada really blew everyone away. Right at the moment of presentation Red Hat issued a press release that they bring DevOps to the network via Ansible:

[Red Hat] is bringing DevOps to networking by extending Ansible – its powerful IT automation and DevOps platform – to include native agentless support for automating heterogeneous network infrastructure devices using the same simple human and machine readable automation language that Ansible provides to IT teams.

Peter picked that up and presented a whole lot of technical details. The most important one was that there are now several networking core modules for commands, configuration and templates.
Ansible networking automation support
They cover a huge load of devices:

  • Arista EOS
  • Cisco NXOS
  • Cisco IOS
  • Cisco IOSXR
  • Cumulus Linux
  • Juniper Junos
  • OpenSwitch

While some of these devices were already supported by the raw module or some libraries out there, but fully integrated modules supported by Ansible and the network device manufacturers themselves takes networking automation to another new level. If you are interested, get the latest Ansible networking right away.

Ansible in a visual effects studio

The next talk was by the customer “Industrial Light and Magic”, a visual effects studio using Ansible to handle there massive setup via Ansible. It showed in particular how many obstacles you face in your daily routine running data centers and deploying software all the time – and how to tackle them using Ansible and Ansible’s features. I forgot to take a photo, though…

Ansible & Windows

John Hawkesworth from M*Modal came up to the stage next and delivered a brilliant speech about all the things needed to know when managing Windows with Ansible. Talking about the differences of Ansible 1.9 vs 2.0 briefly, he went over lessons learned like why the backslash should be escaped every time just to be sure (\t …) but also gave his favourite development and modules quite some attention. Turns out the registration module can come in very handy!
Ansible and Windows

Writing modules, 101

Next up James Cammarata introduced how to write modules for Ansible. And impressively, this was live demonstrated by a module he had written the days before to control his Philips Hue lights. They could be controlled via Ansible live on stage.
Ansible Modules 101
Besides the great live demo the major points of the presentation were:

  • It is quite easy to develop modules.
  • Start of simply, get more complex the further you go down the road.
  • Write a module when your Playbook for a single task exceeds ten lines of code.
  • Write in Python/Powershell when you want it to be integrated with Ansible Core.
  • Write in any language you want if you won’t share it anyway.

While I am sure that other module developers might see some of these points different, it gives a rather good idea what to keep in mind when the topic is approached.

Of course, the the code for the Philips Hue Ansible module is available on Github.

Continuous deployment

Continuous integration is a huge topic in DevOps, and thus especially with Ansible. Steve Smith of Atlassian picked up the topic and discussed what needs to be taken into account when Ansible is used to enable continuous integration.
Continous Integration with Ansible
And there were many memorable quotes during the talk which made it simply fun to watch. I particularly like this one:

Release features, not dumps.

It means: do release when you have something worth releasing – not at arbitrary dates. It is a strong statement against release or maintenance windows and does make sense: after all, why should you release when its not worth? And you certainly will not wait if it is important!

Also, since many maintenance windows are implemented because doing maintenance is hard:

Everything which is hard should be done more often, not less.

Combined with the fact that very complex, but successful enterprises do 300 releases an hour it is clear that continuous deployment is possible – but what often is needed is the right culture and probably at some point a great, simple to use tool able to cater the needs of complex infrastructure.

Ansible accelerates deployment

The next talk focused on a vertical which might usually say that they are too regulated and “special” to integrate DevOps: financial. Fabrice Bernhard presented the journey of the Bank Société Générale introducing DevOps principles with the help of Ansible to become more agile, more flexible and to be able to respond quicker to changes. The reason for that was summarized in a very good quote:

It’s not the big that eat the small. It’s the fast that eat the slow.

This is true for all the enterprises out there: software enabled companies have attacked almost any given business out there (Amazon vs Walmart, Uber vs cabs, Airbnb vs hotels and hostels, etc.). And there are enough analysts right now who see the banking market as the next big thing which might be seriously disrupted due to mobile payment, blockchain technology and other IT based developments.
Ansible and the challenges for businesses

But that also shows what the actual change must be about: the new companies do not take over because they have the better technology. They take over because they have a different culture, and approach problems totally different. And thus, to keep up with the development, change your culture. Or, as said on stage:

Automation is about cultural change. Move fast and break things!

DevOps discussion

After these two powerful talks the audience had a chance to catch some breath during the interactive DevOps discussion. It mainly picked up the topics from the previous talks, and it showed that everyone in the room is pretty sure that DevOps as such is more or less a name on the underlying situation that enterprises need to adopt – or they will fail in the long term, no matter how big they are.

Managing your Cisco data center – with Ansible

As already mentioned, Fabrizio Maccioni from Cisco had the second talk about managing networks with Ansible.
Ansible and Cisco
Interestingly enough, he mentioned that the interest to support Ansible was brought to them by customers who were already managing part of their infrastructure with Ansible. A key point is that Ansible does not require an agent. While Cisco does support some configuration management agents on their hardware, it seems that most of the customers would not do that.
Ansible is good becaue agentless

Immutable infrastructure

The last presentation was held by Vik Bhatti from Beamly. Their problem is that sometimes they have to massively scale in seconds. Literally, in seconds. That requires them to have images of machines up and running in no time. They do this with Ansible, having the playbooks right on the images on one hand, and using Ansible to control their image build process on the other. Actually, the image builder is Packer and it uses Ansible to partially build the image.

As a result, down the line they have images ready to deploy and can extend their environment very, very, very quickly. Since they are able to respond that fast, they were able to cut down hardware costs massively.

Final discussions, happy hour

The final panel dealt mainly with questions about Open Source Tower (it will be there eventually, but no fixed date) and similar questions. After that, everyone went to enjoy drinks and a beautiful skyline.
AnsibleFest skyline and happy hour

Conclusion

In conclusion the #AnsibleFest was a great success, in terms of the people I met as well as in terms of the technical discussions. I can’t wait to get my hand on the networking modules. I’d like to thank the people from Ansible making this event possible, and of course my employer Red Hat for making it possible to visit this event.

[Howto] ownCloud auto setup including LDAP

ownCloud LogoThe self hosting file sharing solution ownCloud is becoming increasingly popular, even in companies you regularly come across installations. To make auto setup of ownCloud easier the following howto shows the steps to automatically connect it to a LDAP server.

File exchange services like Dropbox or Google Drive offer a neat and quick way to exchange even large amounts of data. However, they only work because the data are uploaded to the servers of such corporations in the first hand, which is in times a bit questionable when you deal with sensitive data.

Here ownCloud comes into play: it offers the possibility to self host a file sharing service on infrastructure you trust. Additionally it is Open Source, thus providing at least a minimum amount of trust. And it is not anymore a solution only used by few people for their private servers: these days ownCloud is used in the public sector, universities and companies of all sizes. For example the sciebo project offers ownCloud based file exchange services for 300k students with 5 PB of storage.

It is thus no wonder that the interest in hosting ownCloud services is unbroken. Here at credativ we often see corresponding requests from customers who want support in setting up such installations.

Among the challenges to setup ownCloud in a business environment, two of the biggest ones are the connection to the central authentication service like LDAP and unattended installation. The first task is important to fully integrate ownCloud into the existing user space and make it a first class citizen in the existing infrastrucutre. The second task is especially relevant if you want to easily deploy the service reproducible: Think of containers, docker, VMs, etc. here.

especially the combination of both tasks is challenging: usually ownCloud expects the admin to follow through several steps manually which involve a lot of clicking and entering data until it is up, running and connected to the LDAP. But it is possible to avoid these point-and-click-adventures: Configuration templates can help pre-configuring the ownCloud service, and the setup of the LDAP connection can be automated using ownCloud’s configuration command line tool occ.

So let’s go through the process step by step: At first, ownCloud has to be installed – that can usually be done by the usual package management tools like yum, apt, etc. After the installation, the ownCloud URL is usually opened via browser to start the first run wizard. This can be automated by providing the configuration template $owncloud/config/autoconfig.php which contains all necessary information usually queried in the first run wizard: admin user, pwd, db type, db user, db password, etc. ownCloud checks at start if the file is present and if, omits the first run wizard. Here is an example of such a autoconfig template:

<?php
$AUTOCONFIG = array (
  'directory' => '/var/www/html/owncloud/data',
  'adminlogin'    => 'mmu',
  'adminpass'     => '123456',
  'dbtype'        => 'pgsql',
  'dbname'        => 'owncloud',
  'dbuser'        => 'postgres',
  'dbpass'        => '123456',
  'dbhost'        => '192.168.123.45',
  'dbtableprefix' => 'oc_',
);

Note that further configuration of your ownCloud can also be placed int the usual config.php file: the values of the autoconfig file will be merged into the existing configuration file. This way you can pre-configure most parts of your entire server. More details can be found in the admin documentation.

To actually start the processing of the autoconfig file the ownCloud URL must be called at least once. This can be done from the server itself via the help of curl: curl -s -k 127.0.0.1/owncloud/ > /dev/null.

When the basic configuration is done, the next step is to connect the server to LDAP. This would usually be done by opening the ownCloud URL, activating the LDAP app and configuring it. Instead of clicking through the web page, these tasks can be accomplished with the help of the occ tool. It can be used to activate the app, write and an empty configuration (thanks mark0n for this) and also to set the basic LDAP data. Make sure to call all commands as the user the webserver is called at – otherwise you might get all kinds of problems. The individual steps are:

php -f $ocpath/occ app:enable user_ldap
php -f $ocpath/occ ldap:create-empty-config
php -f $ocpath/occ ldap:set-config "" ldapHost 192.168.123.45
php -f $ocpath/occ ldap:set-config "" ldapPort 389
php -f $ocpath/occ ldap:set-config "" ldapBase \"dc=example,dc=net\"
php -f $ocpath/occ ldap:set-config "" ldapConfigurationActive 1

In case you are debugging problems, check the configuration of the ownCloud server via php -f $ocpath/occ ldap:show-config.

And that’s it already – your ownCloud should be connected to your LDAP server now. If you script all commands for example in Ansible or write a Puppet module it is even easily reproducible.

In case you are interested, I also wrote a German blog article about the problem on credativ’s blog: Owncloud Auto-Setup mit LDAP-Anbindung.

Current distribution of WhatsApp alternatives [Update]

Android_robotMany people are discussing alternatives to WhatsApp right now. Here I just track how many installations the currently discussed, crypto-enabled alternatives have according to the app store.

WhatsApp was already bad before Facebook acquired it. But at least now people woke up and are considering secure alternatives. Yes, this move could have come earlier, but I do welcome the new opportunity: its the first time wide spread encryption actually has a chance in the consumer market. So for most of the people out there the question is more “which alternative should I use” instead of “should I use one”. Right now I do not have the faintest idea which alternative with crypto support will make the break through – but you could say I am well prepare.

Screenshot installed instant messengers
Screenshot installed instant messengers

Well – that’s obviously not a long term solution. Thus, to shed some light on the various alternatives and how they stand right now, here is a quick statistical overview:

Secure Instant Messengers, state updated 2014-03-11
Name WebPage/GooglePlay installed devices Ratings Google +1
ChatSecure Website / Google Play 100 000 – 500 000 1 626 2 620
Kontalk Website / Google Play 10 000 – 50 000 237 265
surespot Website / Google Play 50 000 – 100 000 531 632
Telegram Website / Google Play 10 000 000 – 50 000 000 273 089 97 641
Threema Website / Google Play 500 000 – 1 000 000 9 368 12 594
TextSecure Website / Google Play 100 000 – 500 000 2 478 2 589

The statistics are taken from Google’s Android Play Store. I would love to include iTunes statistics, but it seems they are not provided via the web page. If you know how to gather them please drop me a note and I’ll include them here.

These numbers just help to show how fat an application is spread – it does not say anything about the quality. For example Threema is not Open Source and thus not a real alternative. So, if you want to know more details about the various options, please read appropriate reviews like the one from MissingM.

Android 4.4 now *can* sync multiple calendars via ActiveSync

Android_robotWith the release of Android 4.4 called KitKat Google made some interesting changes to their ActiveSync implementation: the code is now set up to sync more than one calender, and the first KitKat user already confirmed that new feature.

In February I described in a blogpost why Android cannot sync multiple calendars via ActiveSync. The problem was that Google did not implement the necessary parts of the ActiveSync specification in Android.

However, that seems to have changed: if you look at the current ActiveSync implementation of Android 4.4 KitKat, the source code (tag 4.4rc1) does list support for multiple calendars – and also for multiple address books:

        MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_CALENDAR, Mailbox.TYPE_CALENDAR);
        MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_CONTACTS, Mailbox.TYPE_CONTACTS);

I had no chance yet to test that on my own, but there are reports that it is indeed working:

Today i flashed a Android 4.4 Rom on my smartphone. After adding the Exchange Profile all my Calendars are there […]
I’ve uploaded a screenshot here:
http://postimg.org/image/5d4u364ub/

Looks like Google actually listened to…erm, corporate users? At least to someone, though😉

But: Since I have no first-hand-experience in this regard I would like to ask all of my nine readers out there if anyone has a stock KitKat running and if the could check this feature. Please test this and leave a report about your experiences in the comments. I will include it in the article.

By the way, the above mentioned source code snippet also tells quite exactly which other ActiveSync functions are not yet supported in Android:

        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_TASKS,  Mailbox.TYPE_TASKS);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_NOTES, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_JOURNAL, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_TASKS, Mailbox.TYPE_TASKS);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_JOURNAL, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_USER_NOTES, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(Eas.MAILBOX_TYPE_UNKNOWN, Mailbox.TYPE_NONE);
        //MAILBOX_TYPE_MAP.put(MAILBOX_TYPE_RECIPIENT_INFORMATION_CACHE, Mailbox.TYPE_NONE);

I guess syncing tasks could come in handy in corporate environments. Combined with support for multiple task folders you could even design your own Kanban “board” that way.

Nevertheless I’d like to add that ActiveSync is no big deal for me anymore because I am very happy with a – albeit 3rd party and not yet Open Source – CalDav implementation, which can even sync multiple task folders.

Thoughts on crypted communication

network-63770_150Due to the recently published information about mass surveillance on a yet not known level right now the question remains how to encrypt communication. I had some thoughts regarding that topic involving a GPG like web of trust combined with user friendliness which I’d like to share here.

Given everything which was published so far, un-encrypted communication is not save at all. The same is unfortunately true for encryption methods which rely on encryption provided by the servers of some organization. If there is a centralized organization storing the keys for you, or just providing you with the encryption technology, you are screwed, because the intelligence agencies will force them to cooperate. For that reason, the encryption must take place on the end users system already (and the software must be Open Source).

However, if you have end-user encryption, you have the problem of the key exchange – if two people want to communicate securely, they need to exchange the keys or at least securely verify that a public key indeed belongs to their private key. That only works if they meet in person – or if there is a web of trust.

A common example of such a web of trust is the GPG web of trust: people who have properly verified that person A belongs to key M sign this key. If person B trusts person A, it can just use key M since it is already verified by person A. However, in case of GPG the web of trust never reached mainstream. It is mainly used by technical minded people. Most users never got used to it.

So, from what I can tell the only chance to establish a web of trust is to hide the technical details as much as possible from the users. The same is true for the actual key exchange – it needs to be as simple as possible so that each normal user can use it.

Given this background I would suggest the following solution at least for mobile phones. You download the app, and it asks the user for a password. In the background, a key pair is generated and secured with the password, and all data stored on the device are encrypted using the public key. If user A meets user B all they need to do is pressing a button in the app, and a QR code is shown. The other user scans that QR code, and its done. The user shows up in the contact list, and they can chat. In the background, the app extracts the key ID and fingerprint from the QR code, downloads the public key, signs it and uploads the signature automatically.

The biggest problem comes up when user C comes into play, wants to communicate with user A, but they both have no common connection in their web of trust. They would have to meet – or use some other way of exchanging the data securely. A simple way would still be to talk on the phone, but that never worked for GPG. So some kind of web service to host their QR code for a short time only would probably a solution, although it would be pretty risky.

To lower the danger of a man in the middle attack in the above given web example the key servers must only accept one key pair for each identity, which is different to the way GPG works. That would in fact mean that you can have each login only once – if you loose your key, your are screwed.

One question though remains: how many steps in the web of trust are still trustworthy? I guess that could be left as a configuration option if, and only if, a user wants to modify that.

To summarize: I guess that the current cryptography technologies we have could really help to establish secure communication. But to really bring that communication to the masses we need easy-to-use (read: your grandma!) applications doing everything in the background.

Google continues CalDav support for everyone, now also adds CardDav

Android_robotYesterday Google announced that it will not restrict the CalDav access to their calendars to registered partners only, but that they will continue to provide it for everyone. Additionally, Google now offers CardDav support.

A couple of weeks ago Google announced that they would restrict CalDav access to their calendars to registered developers only. That resulted in a huge uproar among developers, users and open standards advocates and made many people wondering if Google will become a closed standards/software company in the future.

However, the pressure (and most likely the bad press and reputation) Google got worked, and they announced that the CalDav API will be continued as an API open for everyone:

In response to those requests, we are keeping the CalDAV API public.

And it becomes even better: CardDav support is added as well, meaning the address data can be accessed via open protocols as well:

And in the spirit of openness, today we’re also making CardDAV – an open standard for accessing contact information across the web – available to everyone for the first time.

This way CalDav and CardDav have an even better chance to become THE royalty free and open alternative to Microsoft’s ActiveSync protocol. Additionally, application developers don’t have to worry to add special code to support Google calendars and address books: they just add CalDav and CardDav support and they automatically support almost all groupware servers and services available.

This is good news and gives me back some trust in Google’s policies and priorities. There is still no CalDav or CardDav support in Android, yes – but at least the server side is better now.

Skype is following your links – that’s proprietary for you

network-63770_150
Yesterday it was reported that Skype, owned by Microsoft these days, seems to automatically follow each exchanged https link. Besides the fact that this is a huge security and personal rights problem in its own it again shows how important it is to not trust a proprietary system.

The problem, skin deep

Heise reported yesterday that Skype follows https links which have been exchanged in chats on a regular basis. First and foremost, this is a privacy issue: it looks like Skype, and thus Microsoft, scans your chat history and acts based on these findings on a regular base. That cannot be explained by “security measures” or anything like it and is not acceptable. My personal data are mine, and Microsoft should not have anything to do with as long as there is no need!

Second, there is the security problem: imagine you are exchanging private links, or even links containing passwords and usernames for direct access (you shouldn’t, but sometimes you have to). Microsoft does follows these links -and therefore gains full access to all data hidden there. Imagine these are sensitive data (private or business), you have no idea what Microsoft is going to do with them.

Third, there is the disturbing part: Microsoft only follows the https links, only the encrypted URLs. If this action would be a security thing, they would surely follow the http links as well. So there must be another explanation – but which one? It is disturbing to know that Microsoft has a motivation to regularly follow links to specifically secured content.

The problem, profound

While these news are shocking, the root problem is not Skype or the behavior of Microsoft – I am pretty sure that their Licence Agreement will cover such actions. And it is most likely that others like WhatsApp, Facebook Chat or whatnot do behave in similar ways. So the actual problem is handing over all your data to a company which you have no inside to. You have no idea what they are doing, you have no control about it, and you cannot even be sure that nothing bad is done with it. Also, most vendors try to lock you in with your service, so that switching away from them is painfully due to used workflows, tools and social networks.

The solution

From my point of view, my personal perfect solution is hosting such sensitive services on my own. However, that cannot be a solution for everyone, and I for myself cannot provide for example the SLAs others need.

Thus I guess the best solution is to be conscious about what you do – and what the consequences are. Try to avoid proprietary solutions where possible. For example for chats, try to use open protocols like XMPP. Google Talk is a good example here: company based, but still using open protocols, they even push the development forward (Jingle, …). Or, if you upload files to web services, make sure you have local backup. Also, try not to upload sensitive data – if you have to, encrypt it beforehand. And if you use social networks, try to not depend on one of them too much, use cross posts for various services at the same time if possible.

And, last but not least: ask your service providers to establish transparency and rules for a responsible and acceptable usage of your data. After all, they depend on the users trust, and if enough users are requesting such changes, they will have to follow.