[Short Tip] Plot live-data in Linux terminal

Recently I realized that one of the disks in my server had died. After the replacement, the RAID sync started – and I quickly had to learn that this was going to take days (!). But I also learned that the time it might take massively jumped up and down.

Thus I thought it would be fun to monitor the progress of this. First, I just crated a command to watch the minutes (calculated into days) every few seconds with watch:

watch 'cat /proc/mdstat |grep recovery|cut -d " " -f 13|cut -d "=" -f 2|cut -d "." -f 1|xargs -n 1 -I {} echo "{}/60/24"|bc'

But since it was jumping so much I was wondering if I could live-plot the data in the terminal (remote server, after all). There are many ways to do that, even gnuplot seems to have an options for that, but I wanted something more simple. Enter: pipeplot

First I tried to use watch together with pipeplot, but it was easier to just write a short for loop around it:

while true;
do
  cat /proc/mdstat |grep recovery|cut -d " " -f 13|cut -d "=" -f 2|cut -d "." -f 1|xargs -n 1 -I {} echo "{}/60/24"|bc;
  sleep 5;
done \
| pipeplot

And the result is rather nice (also shown in the header image):

[Short Tip] Accessing tabular nushell output for non-nushell commands

After I learned how subshells can be executed within nushell I was confident that I could handle that part. But few minutes ago I run into an error I didn’t really understand:

❯ rpm -qf (which dwebp)
error: Type Error
   ┌─ shell:24:16
   │
24 │ rpm -qf (which dwebp)
   │                ^^^^^ Expected string, found row

I thought the parameter was provided somehow in the wrong way, and put it into quotes: "dwebp". But it didn’t help. I tested around more with sub-shells, some of them worked while others didn’t. The error message was misleading for me, letting me think that there is a difference in how the argument is interpreted:

❯ rpm -qi (echo rpm)
Name        : rpm
Version     : 4.16.1.3
Release     : 1.fc34
[...]

❯ echo (which dwebp)
───┬───────┬────────────────┬─────────
 # │  arg  │      path      │ builtin 
───┼───────┼────────────────┼─────────
 0 │ dwebp │ /usr/bin/dwebp │ false   
───┴───────┴────────────────┴─────────

It took me a while until I understood what I was looking at – and to make the error message make sense: the builtin nushell command which can give back multiple results, thus returning a table. The builting nushell command echo returns a string!

Thus the right way to execute my query is to get the content of the cell of the table I am looking at via get:

❯ rpm -qf (which dwebp|get path|nth 0)
libwebp-tools-1.2.1-1.fc34.x86_64

Note that nth 0 is not strictly necessary here since there is only one item in the table anyway. But it might help as a reference for future examples.

You don’t have to use pipe, btw., there is an even shorter way available:

❯ rpm -qf (which dwebp|nth 0).path
libwebp-tools-1.2.1-1.fc34.x86_64

Figuring out the container runtime you are in

Containers are meant to keep processes contained. But there are ways to gather information about the host – like the actual execution environment you are running in.

Containers are pretty good at keeping everyone and everything inside their boundaries – thanks to SELinux, namespaces and so on. But they are not perfect. Thanks to a recent Azure security flaw I was made aware of a nice trick via the /proc/ file system to figure out what container runtime the container is running in.

The idea is that the started container inherits some /proc/ entries – among the entry for /proc/self/. If we are able to load a malicious container, we can use this knowledge to execute the container host binary and by that get information about the runtime.

As an example, let’s take a Fedora container image with podman (and all it’s library dependencies) installed. We can run it and check the version of crun inside:

❯ podman run --rm podman:v3.2.0 crun --version
crun version 0.20.1
commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

Note that I take an older version here to better highlight the difference between host and container binary!

If we now change the execution to /proc/self/exe, which points to the host binary, the result shows a different version:

❯ podman run --rm podman:v3.2.0 /proc/self/exe --version
crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

This is actually the version string of the binary on the host system:

❯ /usr/bin/crun --version
crun version 1.0
commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

With this nice little feature we now have insight into the version used – and as it was shown in the detailed write-up of the Azure security problem mentioned above, this insight can be crucial to identify and use the right CVEs to start lateral movement.

Of course this requires us to be able to load containers of our choice, and to have the right libraries inside the container. This example is simplified because I knew about the target system and there was a container available with everything I needed. For a more realistic attack container image, check out whoc.

[Short Tip] Output/redirect content to a file in Nushell

And another short tip about Nushell – I promise that those will be less frequent the more I get used to it.

My current problem was: how do I redirect content to a file like echo hello > foo.txt and echo world >> foo.txt? The typical approach didn’t work:

❯ echo "hello" > foo.txt
───┬─────────
 0 │ hello   
 1 │ >       
 2 │ foo.txt 
───┴─────────

Yeah, certainly not what I had in mind. Instead, I had to rethink the approach here. What is what we want to do here? First we output content and need to save it to a file. The connection is a pipe, as usual in Nushell:

❯ echo hello | save foo.txt
❯ open foo.txt
hello

That worked! Second, we want to append something. So we need to open the file, append something, and save it again. In between all steps we need pipes – Nushell, after all:

❯ open foo.txt | append "world"| save --raw foo.txt
❯ open foo.txt
hello
world

Note that save needs the --raw flag here: it tries to be smart to guess in what format we want to save it, and for some reason in my case it didn’t save the new lines without the flag.

And that’s it. It is not as short as I would like it to be compared to Bash and others. On the other hand it is way more flexible (it can also handle structured data this way like json) and it is not like I use redirection all the time.

[Short Tip] Executing a subshell in Nushell

I just run through a howto where I was asked to execute a command which used the command output from a subshell as an argument for another command. Copy&paste of such typical command examples don’t work with nushell:

❯ sudo usermod --append --groups libvirt $(whoami)
error: Variable not in scope
  ┌─ shell:9:40
  │
9 │ sudo usermod --append --groups libvirt $(whoami)
  │                                        ^^^^^^^^^ unknown variable: $(whoami)

The right way to do that in nushell is only slightly different – using subexpressions:

❯ sudo usermod --append --groups libvirt (whoami)