[Short Tip] Quickly copy files to Samsung Galaxy S20+ on Fedora 32

Transferring files from a Fedora 32 machine to an Android phone is usually not a problem: plugin in it (via USB), unlock screen, make sure that USB connection is set to file transfer, and open the phone in Nautilus.

However, I recently had to get a new phone, decided to opt for the Samsung Galaxy S20+ – and there I was not able to write files to the device:

rsync: open "/run/user/1000/gvfs/mtp:host=SAMSUNG_SAMSUNG_Android_R58N23ZDCVN/Phone/appdata/2020-04-12_23-38 - foldersync.db" failed: Operation not supported (95)

Note that other phones made no problems so far. And things got even more weird when I realized that I was able to create folders – but not files?!

After the usual tricks (enabling USB debugging, switching cables, etc.) I realized that this might be a special problem with the implementation of mtp by this phone. So instead of using libmtp, wich is used by default by for example gvfs, I tested other mtp implementations – and found simple-mtpfs, which worked like a charm:

$ sudo simple-mtpfs -l
1: SamsungGalaxy models (MTP)
$ sudo simple-mtpfs --device 1 /mnt
# you have to acknowledge access to the phone on the phone screen
# then you have to mount it again
$ sudo simple-mtpfs --device 1 /mnt
$ sudo rsync --verbose --progress --size-only --omit-dir-times --no-perms --recursive --inplace /home/liquidat/backup/ /mnt/Phone/

The performance is good – way better than trying to copy files via gphoto2, btw 😉

Image by Martin Pyško from Pixabay

[Short Tip] Flatten nested dict/list structures in Ansible with json_query

A few days ago I was asked how to best deal with structures in Ansible which are mixing dictionaries and lists. Basically, the following example was provided and the questioned remained how to deal with this – for example how to flatten it:

    myhash:
      cloud1:
        region1:
          - name: "city1"
          - size: "large"
          - param: "alpha"
        region2:
          - name: "city2"
          - size: "small"
          - param: "beta"
      cloud2:
        region1:
          - name: "city1"
          - size: "large"
          - param: "gamma"

I was wondering a lot how to deal with this – after all dict2items only deals with dicts and fails when it reaches the lists in there. I also fooled around with the map filter, but most of my results also required some previous knowledge about the data structure, were only acting by providing “cloud1.region1” or similar.

The solution was the json_query filter: it is based on jmespath and can deal with the above mentioned structure by list and object projections:

  tasks:
  - name: Projections using json_query
    debug:
      msg: "Item value is: {{ item }}"
    loop: "{{ myhash|json_query(projection_query)|list }}"
    vars:
      projection_query: "*.*[]"

And indeed, the loop does create a simplified output of all the elements in this nested structure:

TASK [Projections using json_query] **********************************************************
ok: [localhost] => (item=[{'name': 'city1'}, {'size': 'large'}, {'param': 'alpha'}]) => {
    "msg": "Item value is: [{'name': 'city1'}, {'size': 'large'}, {'param': 'alpha'}]"
}
ok: [localhost] => (item=[{'name': 'city2'}, {'size': 'small'}, {'param': 'beta'}]) => {
    "msg": "Item value is: [{'name': 'city2'}, {'size': 'small'}, {'param': 'beta'}]"
}
ok: [localhost] => (item=[{'name': 'city1'}, {'size': 'large'}, {'param': 'gamma'}]) => {
    "msg": "Item value is: [{'name': 'city1'}, {'size': 'large'}, {'param': 'gamma'}]"
}

Of course, some knowledge is still needed to make this work: you need to know if you are projecting on a list or on a dictionary. So if your data structure changes on that level between executions, you might need something else.

Image by Andrew Martin from Pixabay

[Short Tip] Accessing Nautilus mounted locations via shell/terminal

When using Gnome, it is quite convenient to mount remote locations directly in Nautilus. As an example, I often mount my Google Drive work folder, also my personal NextCloud instance.

While this makes it easy to work with remote accessible files from within Gnome tools, it is less obvious how to access for example files from within a shell like gnome-terminal.

Nautilus uses FUSE to mount remote locations. You can find more in the Gnome documentation GVfs.

With this knowledge, the solution is this directory:

$XDG_RUNTIME_DIR/gvfs

In that directory you will find the actual fuse mounts of the remote locations linked from within Nautilus:

$ ls -1 $XDG_RUNTIME_DIR/gvfs
'dav:host=nc.bayz.de,ssl=true,prefix=%2Fremote.php%2Fwebdav'
'google-drive:host=redhat.com,user=ABCDEF'

Those are directories, so you can just work with them as with usual directories and change into them, edit files, etc. Of course, depending on the internet connection and the endpoint and the protocol the interactions will not be comparable to working on files on your local SSD. But it might be just enough for your use cases.

Thanks to strugee for this detailed comment on StackExchange.

[Short Tip] Handling “can’t concat str to bytes” error in Ansible’s uri module

When working with web services, especially REST APIs, Ansible can be of surprising help when you need to automate those, our want to integrate them into your automation.

However, today I run into a strange Python bug while I tried to use the uri module:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: can't concat str to bytes
fatal: [localhost]: FAILED! => {"changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [200]: An unknown error occurred: can't concat str to bytes", "redirected": false, "status": -1, "url": "https://www.ansible.com"}

This drove me almost nuts because it happened on all kinds of machines I tested, even with Ansible’s devel upstream version. It was even independent of the service I targeted – the error happened way earlier. And a playbook to showcase this was suspiciously short and simple:

---                                 
- name: Show concat str byte error  
  hosts: localhost                  
  connection: local                 
  gather_facts: no                  
                                    
  tasks:                            
    - name: call problematic URL call
      uri:                          
        url: "https://www.ansible.com"
        method: POST                
        body_format: json
        body:                       
          name: "myngfw"

When I was about to fill an issue at Ansible’s Github page I thought again and wondered that this is too simple: I couldn’t imagine that I was the only one hitting this problem. I realized that the error had to be on my side. And thus meant that something was missing.

And indeed: the body_format option was not explicitly stated, so Ansible assumed “raw”, while my body data were provided in json format. A simple

        body_format: json

solved my problems.

Never dare to ask me how long it took me to figure this one out. And that from the person who write and entire how to about how to provide payload with the Ansible URI module….

[Short Tip] Exclude files in Git diff

Did you ever saw a git diff with file changes you wanted to omit? I recently had to go through a very large git diff – and I realized that some modified files were not needed and I had to somehow remove them from the diff.

To go through the huge diff and remove all patches to certain files manually would be way too much work. Luckily since Git 1.9 you can use pathspec patterns to limit the output of git – and thus the output of git-diff:

git diff devel..feature ':!path/to/file'
git diff devel..feature ':!path/to/*manyfiles'
git diff devel..feature ':!just/a/path'

The exclamation mark is just a short form for (exclude). If you use the bracket style you can even add additional “magic words” – for example to make the exclusion case insensitive:

git diff devel..feature ':(exclude,icase)PATH/TO/FILE

Other magic words are literal, top and glob.

Git pathspecs also work with git-grep, git-log and others.