[Howto] Acting on Inotify events in a shell with inotifywait

TuxInotify is a subsystem of the Linux kernl monitoring file systems and reporting changes to user space appliactions. In a shell like Bash, the tool inotifywait can be used to call scripts or other actions based on Inotify events.

The Kernel subsystem Inotfiy monitors file systems for events like access to files, changes, creations, deletions. User space programs like a desktop search engine can connect to Inotify, and ask it to send reports about the event X on file Y. If the event takes place, Inotify reports to the user space program, which for example re-indexes the file again.

For shell environments tools are already available to do talk to Inotify, settings watchers for certain files and directories, and receiving the reports: inotify-tools.

The inotify-tools contain two main programs: inotifywait and inotifywatch. The second one, inotifywatch, simply gathers statistics about file events. Much more interesting though is the first tool, inotfiywait. It asks Inotify to watch files and directories (even recursively) and it can call arbitrary actions on various Inotify events. The tool also offers the usual command line flags to make live easier: formatting strings for the output format, possibility to run as a daemon, read in file list from a file, write events to a file, exclude patterns, and so on.

For example: imagine you need to review a huge LaTeX script, thus occasionally have to change some smaller things but do not want to fire up “make” each time you changed something. If you are lazy, you can simply use inotifywait to monitor the directory containing your LaTeX files and calling make each time something changes:

$ while true; do inotifywait -r -e modify --exclude=".swp" . && make; done
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
./mainmatter/ MODIFY File15.tex
latexmk -pdf -r ./pdflatexmkrc
Latexmk: This is Latexmk, John Collins, 11 Nov. 2012, version: 4.35.
**** Report bugs etc to John Collins <collins at phys.psu.edu>. ****
Latexmk: applying rule 'pdflatex'...
Rule 'pdflatex': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'mainmatter/File15.tex'
------------
Run number 1 of rule 'pdflatex'
------------

In the above example inotifywait is called with the flag -e modify which restricts it to only monitor modifying events. Also temporary vim files are excluded and thus ignored. The watches are set up, the program is waiting – until Inotify reports the event “MODIFY” on “File15.tex” in the sub-directory “mainmatter”. Afterwards, “make” was called, which again launched “latexmk”. Also the while loop ensures that after a change inotifywait is called again and thus monitors the files and directories.

Summarizing I can say inotifywait is an easy and quite usable approach to access the Inotify subsystem in simple and daily shell scripts and tasks. It makes life easier without making things complicated.

If you know people speaking German who are interested in this topic, I also wrote a German version of this howto for my employer’s blog.

[Howto] Connecting a USB GSM modem to a KVM guest – USB pass through

TuxWith current virtualization technologies it is possible to pass through devices from the host to the guest, calld USB pass through. KVM is no exception here, it even works with a USB GSM modem.

Many of customers I work with are migrating old IT systems and existing servers over to a newer and virtualized infrastructure. That often works without any problems. However, some services do depend on extra hardware like additional PCI cards – or, as in my case, on an external USB GSM modem.

To pass through such a device to the VM guest first the vendor and the product ID must be identified on the VM host:

$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 002: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
Bus 002 Device 003: ID 12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem / E230/E270/E870 HSDPA/HSUPA Modem

The last entry shows the mentioned GSM modem, built by Huawei. The interesting numbers are the vendor ID 12d1 and the product ID 1003. THe VM guest is oblivious of the device right now:

$ lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub

Next, the device must be defined in the VM guest XML. This can be done by directly editing the XML file within virsh: $ sudo virsh edit example-server. The command brings up an editor with the content of the XML definition file of the host. The USB device must be added in the device section:

  <devices>
    [...]
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x12d1'/>
        <product id='0x1003'/>
      </source>
    </hostdev>
  </devices>

Please note that a leading 0x must be added to the IDs! Save the file, reboot the VM guest, and check if the guest now shows the new device:

$ lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
Bus 001 Device 004: ID 12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem / E230/E270/E870 HSDPA/HSUPA Modem

There it is. And the syslog shows that it was properly detected and can now be used in the usual ways, it’s done.

USB Serial support registered for GSM modem (1-port)
option 1-2.1:1.0: GSM modem (1-port) converter detected
usb 1-2.1: GSM modem (1-port) converter now attached to ttyUSB0
option 1-2.1:1.1: GSM modem (1-port) converter detected
usb 1-2.1: GSM modem (1-port) converter now attached to ttyUSB1
usbcore: registered new interface driver option
option: v0.7.2:USB Driver for GSM modems

Short Tip: Fix input/output error while creating LVM backed VMs with libvirt

920839987_135ba34fff
Today I run into a strange error where I was not able anymore to create new VMs with virt-manager: I always got an input/output error when I tried to start the machine after installation.

A look into /var/log/syslog showed quite some errors on the dm-device – note that my VMs disks usually are on logical volumes.

Sep 12 15:27:55 example kernel: [19298.163712] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Sep 12 15:27:55 example kernel: [19298.243980] Buffer I/O error on device dm-5, logical block 1081985
Sep 12 15:27:55 example kernel: [19298.243983] lost page write due to I/O error on dm-5
Sep 12 15:27:55 example kernel: [19298.243994] Buffer I/O error on device dm-5, logical block 1081986
Sep 12 15:27:55 example kernel: [19298.243994] lost page write due to I/O error on dm-5

The fix is pretty easy: when you create the disk and thus the LV for the virtual machine, make sure you tell virt-manager that it should allocate the entire disk right from the start. It looks like sparse LV images are not supported right now.

[Howto] Monitoring Puppet agent run with NRPE-Plugin [Update]

920839987_135ba34fffCentralized configuration management like Puppet is a bless. If it runs properly. So it makes sense to monitor the run of the Puppet agent, and I wrote NRPE plugin to do just that.

Puppet is great, and many of our customers use it, often in combination with a Icinga monitoring setup. However, it might happen that the Puppet agent, for some reason, does not run, or does not run properly. If the infrastructure is large enough, that might slip through your fingers. Thus it makes sense to monitor the Puppet client.

There are already several solutions out there to do just that. Since the Puppet agent does write plenty of status information to /var/lib/puppet/state/last_run_summary.yaml current solutions check the last run time stamp of the file, or try to verify the validity of the Yaml structure. However, a correct Yaml structure does not tell anything about when the Puppet agent actually run last time. Also, the time stamp is also written even if the Puppet agent run fails in the end. There is even a Bash script which does both – but it is a difficult-to maintain piece of code and cannot really speak Yaml, it just greps for certain elements.

Thus I wrote my own script inspired by the solutions mentioned above, but checking the last run state as well as verifying that the Yaml file has proper content – and written in Python, by the way. The script can be tested on command line:

$ sudo /usr/local/lib/nagios/plugins/check_puppetagent -w 3600 -c 9000
OK: Puppet was last run 13 minutes and 21 seconds ago

If the Yaml file is not properly formatted, the script throws an error:

$ python check_puppetagent -w 3600 -c 9000
CRIT: Yaml file not properly formatted, last puppet run failed.

The script does not support any further options or functions. Since the Yaml file does contain much more information it might make sense to give more information back to the monitoring server, or for example also give back the number of failures given in the status file if there are any. But for now, that is not implemented.

The script was also uploaded to Monitoringexchange. Since my employer strongly supports the ideas behind Open Source, I was able to publish the script under the MIT licence. I also wrote a blog post about the script on my German company’s blog.

Update
What I totally forgot: there is also a ruby check script which does mainly the same as the Python script I wrote and was a good inspiration for my code.

[Howto] Monitoring OpenVPN ports with Nagios/Icinga

920839987_135ba34fffA OpenVPN server is usually a crucial part of the IT infrastructure, and thus should be monitored properly. But monitoring UDP is sometimes not that easy, so I wrote a script which can be used in Nagios/Icinga.

OpenVPN is usually accessed via UDP. Since UDP is not as easy to monitor as TCP ports are, many administrators restrain themselves to just monitor if an OpenVPN process is running on the OpenVPN server. However, that does not unveil network problems, and can only be used on machines where you have proper access to: 3rd party machines or appliances are out of your reach with this attempt. Another attempt is to monitor the management port. However, that requires that the port is reachable by the monitoring server which might not be the best idea in case of distributed monitoring. And this is still no option in case of 3rd party machines or other black boxes.

A customer of my employer credativ GmbH had exactly that kind of problem, so I wrote a script in Python. It checks the UDP port of a given server. If the server does respond, the script gives back the state “OK” together with the hex form of the response. The script can be tested on command line:

$ python check_openvpn openvpn.example.com
OK: OpenVPN server response (hex): 4018062d97f85c21d50000000000

The port can be changed by the flag “-p”:

$ python check_openvpn -h
usage: check_openvpn [-h] [-p PORT] [-t] host

positional arguments:
  host                  the OpenVPN host name or ip

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  set port number
  -t, --tcp             use tcp instead of udp

As you see, it also supports testing TCP ports. However, in that case we do not have a return code, we effectively just test if the given tcp port can be reached. Here we switch on TCP support and also modify the port to 443:

$ python check_openvpn -t -p 443 openvpn-tcp.example.com
OK: OpenVPN tcp port reachable.

If the server does not respond within a given time period – 5 seconds – the server throws an error:

$ python check_openvpn slowserver.example.com
CRIT: Request timed out

The script was also uploaded to Monitoringexchange. Since my employer strongly supports the ideas behind Open Source, I was able to publish the script under the MIT licence. I also wrote a blog post about the script on my German company’s blog.

[Howto] Remapping buttons with xbindkeys and xte

TuxSometimes you buy devices like a mouse or a keyboard which provide additional buttons for special functions. Or which have buttons which do not behave as expected. In such cases the button actions can be mapped to other functions, or even to other buttons.

I recently bought a Logitech M705 mouse which works almost perfectly. However, there is one nagging bug: the middle mouse button does not trigger the usual event “mouseclick two” which is interpreted as pressing the middle button, the scroll wheel which is crucial for example browsing the web. Nothing happens.

In such cases the first step is to see if the operating system receives any input at all. Fire up a shell and start the program xev. It opens a small, white window where you can move your cursor to. As soon as the cursor enters the window, you will see a lot of log data on the shell: xev shows you all X events, thus all data you enter via keyboard or mouse.

In my case I pressed the middle button, and saw:

ButtonPress event, serial 40, synthetic NO, window 0x5800001,
    root 0xc7, subw 0x5800002, time 19610234, (46,38), root:(1784,61),
    state 0x10, button 6, same_screen YES
[...]
ButtonRelease event, serial 40, synthetic NO, window 0x5800001,
    root 0xc7, subw 0x5800002, time 19610234, (46,38), root:(1784,61),
    state 0x210, button 6, same_screen YES

The interesting part is that the button was not interpreted as “button 2” as I would have expected, but as “button 6”. That’s not what I expected, and thus the button must be remapped to the event “button 2”.

Mapping of keys and buttons can be done via xbindkeys: in case of KDE I created a symlink to start the program at each startup:

ls -la ~/.kde/env/xbindkeys 
lrwxrwxrwx 1 liquidat users 18 Aug  1 10:56 .kde/env/xbindkeys -> /usr/bin/xbindkeys

xbindkeys reads its configuration from ~/.xbindkeysrc, so that’s the place where we need to configure the actual mapping. The syntax is:

#    "command to start"
#       associated key

The most interesting part of the mapping is: how to trigger the action “button 2”? That is done by the program xte which generates fake input. Thus the final configuration is:

"xte 'mouseclick 2'"                                                                                                                                                                                           
  b:6  

And you are done. Pressing the mouse button 6 on the Logitech M705 now launches the mouseclick 2.

However, as stated correctly by the comments below, this is just an intermediate solution! A long time solution is to fix the mapping in evdev, the Linux input handling.