DevOps Notes

Clever Disk Encryption with dm-crypt Nuke

Thu, 16 Jan 2025 00:00:00 +0000

Introduction ¶

Disk encryption using dm-crypt is a reliable way to protect data. However, in extreme situations, it may be necessary to instantly and irreversibly destroy encryption keys to render the data inaccessible. The nuke mechanism can be used for this purpose, activated by entering a special password.

In this article, we will explore how to implement nuke in dm-crypt using initramfs, cryptsetup, and mkpasswd.

The developed hook is capable of:

Decrypting the disk using data from a file.
Prompting for a password if the file is not found.
Deleting all keys if the entered password is a nuke password.

Important! This setup works on Ubuntu 22.04.5. Other distributions might require different configurations. In any case, here is an article on recovering Grub.

Environment Setup ¶

Before starting, ensure that the necessary utilities are installed:

1
2


sudo apt update
sudo apt install cryptsetup initramfs-tools whois

Adding `mkpasswd` Support in initramfs ¶

The nuke mechanism requires mkpasswd, which is used to verify the entered password. We need to add it to initramfs by creating a hook:

File /etc/initramfs-tools/hooks/mkpasswd:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


#!/bin/sh
set -e

if [ "$1" = "prereqs" ]; then
 exit 0
fi

. /usr/share/initramfs-tools/hook-functions

if ! command -v mkpasswd 1>/dev/null; then
 echo "error: mkpasswd executable is not found, install it." 1>&2
 exit 1
fi

copy_exec "$(command -v mkpasswd)"
copy_exec "$(command -v dd)" /sbin

Make it executable:

1

sudo chmod +x /etc/initramfs-tools/hooks/mkpasswd

Creating the `decrypt_smart_nuke` Script ¶

File /usr/lib/cryptsetup/scripts/decrypt_smart_nuke:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155


#!/bin/sh
set -e

KEYFILE_ENABLED='false'
NUKE_ENABLED='false'

KEYFILE_PATH=''
KEYFILE_OFFSET=0
KEYFILE_SIZE=''

NUKE_PASSWORD=''

CRYPTTAB_TRIED="${CRYPTTAB_TRIED?:CRYPTTAB_TRIED is not defined}"
CRYPTTAB_NAME="${CRYPTTAB_NAME?:CRYPTTAB_NAME is not defined}"
CRYPTTAB_SOURCE="${CRYPTTAB_SOURCE?:CRYPTTAB_SOURCE is not defined}"
ASKPASS='/lib/cryptsetup/askpass'

# ┬─┐┬ ┐┌┐┐┌─┐┌┐┐o┌─┐┌┐┐┐─┐
# ├─ │ │││││ │ ││ ││││└─┐
# ┘ ┘─┘┘└┘└─┘ ┘ ┘┘─┘┘└┘──┘

# Logging helpers. Send the argument list to plymouth(1), or fold it
# and print it to the standard error.
message() {
 local IFS=' '
 if [ "${0#/scripts/}" != "$0" ] && [ -x /bin/plymouth ] && plymouth --ping; then
 plymouth message --text="cryptsetup: $*"
 elif [ ${#*} -lt 70 ]; then
 echo "cryptsetup: $*" 1>&2
 else
 # use busybox's fold(1) and sed(1) at initramfs stage
 echo "cryptsetup: $*" | fold -s | sed '1! s/^/ /' 1>&2
 fi
 return 0
}

parse_options() {
 local IFS="," key value regex
 for option in $1; do
 key="${option%%=*}"
 value="${option#*=}"
 case "$key" in
 keyfile-path)
 if [ ! -e "$value" ]; then
 message "warning: keyfile at $value does not exist"
 else
 export KEYFILE_ENABLED="true"
 export KEYFILE_PATH="$value"
 fi
 ;;
 keyfile-offset)
 export KEYFILE_OFFSET="$value"
 ;;
 keyfile-size)
 export KEYFILE_SIZE="$value"
 ;;
 nuke)
 regex='(\$[^$]+){3,}'
 if ! echo "$value" | grep -qE "$regex"; then
 message "$(printf "error: password hash does not match the regex '%s': %s\n" "$regex" "$value")"
 exit 1
 fi
 export NUKE_ENABLED="true"
 export NUKE_PASSWORD="$value"
 ;;
 none) true;;
 *)
 message "error: unknown option $key: $value" 1>&2
 exit 1
 ;;
 esac
 done
}

read_keyfile() {
 if [ "$KEYFILE_ENABLED" != 'true' ] || [ "$CRYPTTAB_TRIED" -ne 0 ]; then
 return
 fi
 if [ -n "$KEYFILE_SIZE" ]; then
 dd if="$KEYFILE_PATH" iflag=skip_bytes skip="$KEYFILE_OFFSET" bs="$KEYFILE_SIZE" count=1 2>/dev/null
 else
 dd if="$KEYFILE_PATH" iflag=skip_bytes skip="$KEYFILE_OFFSET" 2>/dev/null
 fi
 exit 0
}

nuke() {
 echo YES | cryptsetup erase "$CRYPTTAB_SOURCE"
 echo 1>&2 '''
 ________________
 ____/ ( ( ) ) \___
 /( ( ( ) _ )) ) )\
 (( ( )( ) ) ( ) )
 ((/ ( _( ) ( _) ) ( () ) )
 ( ( ( (_) (( ( ) .((_ ) . )_
 ( ( ) ( ( ) ) ) . ) ( )
 ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
 ( ( ( ) ( ) ( )) ) _)( ) ) )
 ( ( ( \ ) ( (_ ( ) ( ) ) ) ) )) ( )
 ( ( ( ( (_ ( ) ( _ ) ) ( ) ) )
 ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
 (( ( )( ( _ ) _) _(_ ( (_ )
 (_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
 ((__) \\||lll|l||/// \_))
 ( /(/ ( ) ) )\ )
 ( ( ( ( | | ) ) )\ )
 ( /(| / ( )) ) ) )) )
 ( ( ((((_(|)_))))) )
 ( ||\(|(|)|/|| )
 ( |(||(||)|||| )
 ( //|/l|||)|\\ \ )
 (/ / // /|//||||\\ \ \ \ _)
-----------------------------!!! YOU ARE NUKED !!!-----------------------------
'''
 message "Data destroyed! They may try to extract information from you, but there's nothing more you can do. Good luck!"
 exit 1
}

read_interactive() {
 local password salt
 password="$($ASKPASS "Enter passphrase for $CRYPTTAB_NAME ($CRYPTTAB_SOURCE): ")"
 if [ "$NUKE_ENABLED" = 'true' ]; then
 salt="$(echo "$NUKE_PASSWORD" | cut -d$ -f3)"
 for method in bcrypt bcrypt-a sha256crypt sha512crypt md5crypt; do
 if mkpasswd -m "$method" -S "$salt" "$password" 2>/dev/null | grep -qF -- "$NUKE_PASSWORD"; then
 nuke
 fi
 done
 fi
 printf "%s" "$password"
 exit 0
}

# ┌─┐┬ ┬┬─┐┌─┐┬┌ ┐─┐
# │ │─┤├─ │ ├┴┐└─┐
# └─┘┘ ┴┴─┘└─┘┘ ┘──┘

if ! command -v mkpasswd 1>/dev/null 2>&1; then
 message "error: mkpasswd binary is not found"
 exit 1
elif [ ! -f "$ASKPASS" ]; then
 message "error: $ASKPASS does not exist"
 exit 1
elif [ ! -e "$CRYPTTAB_SOURCE" ]; then
 message "error: crypttab source device does not exist"
 exit 1
fi

# ┌┌┐┬─┐o┌┐┐
# ││││─┤││││
# ┘ ┘┘ ┘┘┘└┘

parse_options "$1"
read_keyfile
read_interactive

Make it executable:

1

sudo chmod +x /usr/lib/cryptsetup/scripts/decrypt_smart_nuke

Boot Configuration ¶

Edit /etc/default/grub and add the cryptdevice argument:

1

GRUB_CMDLINE_LINUX_DEFAULT="... cryptdevice=UUID=bb85fe0a-4ef7-49c4-b15c-fb613d51e9ef:cryptroot"

Then add the following line to /etc/crypttab:

1

cryptroot UUID=bb85fe0a-4ef7-49c4-b15c-fb613d51e9ef keyfile-path=/dev/mmcblk0,keyfile-offset=8932578598,keyfile-size=11045,nuke=HASH luks,discard,keyscript=decrypt_smart_nuke

To generate a hash for the nuke password:

1

mkpasswd -m sha512crypt --rounds=5000

Updating initramfs ¶

After making all changes, update initramfs and grub:

1
2


sudo update-initramfs -u
sudo update-grub2

Conclusion ¶

Now, if the nuke password is entered at boot, the script will destroy the encryption keys, making the data inaccessible. This method is suitable for critical situations requiring rapid and irreversible data protection.

Additionally, the system will decrypt itself automatically if the key file is present. If needed, the USB drive can be removed, prompting the system for a password.

Backup your LUKS header before enabling this!

1

sudo cryptsetup luksHeaderBackup /dev/sda1 --header-backup-file ./backup.luks

Kubernetes Node role labeler

Tue, 08 Oct 2024 00:00:00 +0000

Description ¶

Node Role Labeler has long been sought after, a simple operator, but the task is fulfilled. The problem is that Kubernetes, specifically kubelet, cannot assign itself a label of node-role.kubernetes.io/ROLE_NAME when creating a node, because of the kubernetes.io protected prefix. Not very convenient.

Here I have Karpenter, or Cluster Autoscaler, no matter what, I want nodes to have nice roles immediately, and it does not come out of the box only by hand. Now we can attach node-role.cluster.local/NAME and the operator will create an automatically copy with a domain kubernetes.io. In reverse also works. More information - read on GitHub.

Demo ¶

Karpenter configuration ¶

Here is how we can immediately attach labels to Karpenter nodes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92


apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
 name: private
spec:
 amiFamily: AL2
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 2
 httpTokens: required
 role: Karpenter-development-role
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: development
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/development/subnet: private
 tags:
 karpenter.sh/discovery: development
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
 name: on-demand-amd64
spec:
 disruption:
 budgets:
 - nodes: 20%
 - nodes: "5"
 - duration: 20h
 nodes: "0"
 schedule: 0 5 * * *
 consolidationPolicy: WhenUnderutilized
 expireAfter: 720h
 limits:
 cpu: "200"
 memory: 400Gi
 template:
 metadata:
 labels:
 node-role.cluster.local/on-demand-amd64: ""
 spec:
 nodeClassRef:
 name: private
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: node.kubernetes.io/instance-type
 operator: In
 values:
 - r5a.4xlarge
 - r5a.2xlarge
 - r5a.xlarge
 - r5a.large
 - r5a.medium
 - r5.4xlarge
 - r5.2xlarge
 - r5.xlarge
 - r5.large
 - r5.medium
 - m6i.4xlarge
 - m6i.2xlarge
 - m6i.xlarge
 - m6i.large
 - m6i.medium
 - m5a.4xlarge
 - m5a.2xlarge
 - m5a.xlarge
 - m5a.large
 - m5a.medium
 - c6a.4xlarge
 - c6a.2xlarge
 - c6a.xlarge
 - c6a.large
 - c6a.medium
 - c6.4xlarge
 - c6.2xlarge
 - c6.xlarge
 - c6.large
 - c6.medium
 - t3a.4xlarge
 - t3a.2xlarge
 - t3a.xlarge
 - t3a.large
 - t3a.medium
 - key: karpenter.sh/capacity-type
 operator: In
 values:
 - on-demand

BTRFS checksum benchmark

Sun, 30 Jul 2023 00:00:00 +0000

Introduction ¶

I’m switching to a new hardware and I’m using a clean SSD. Now, I’m facing the choice of the algorithm for the BTRFS file system. Here is a script that I used to determine which one was the fastest on my specific processor.

Body ¶

I started by searching for an utility to measure the performance of different hash algorithms. Why? Because, no matter how fast an algorithm seems on paper, if it’s not supported by my processor - it will be slow. And how can I know what’s supported by my processor? In the specification online, I didn’t find any information.

I stumbled upon the rhash utility, which has a --speed key. That’s where I tried to measure performance, but the results of blake2b and crc32c were almost identical, which seemed strange to me. In the end, I came up with this script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


#!/usr/bin/env bash
set -eo pipefail

TMPFS="$(mktemp -d)"
SIZE="5G"

mount -t tmpfs -o size="${SIZE}" tmpfs "${TMPFS}"
cd "${TMPFS}"
fallocate -l 4GiB btrfs

for CSUM in crc32c xxhash sha256 blake2; do
 mkfs.btrfs -f -d single -m single --csum "${CSUM}" btrfs
 mount btrfs /mnt
 echo "
 cd /mnt
 echo "info: ${CSUM}"
 dd if=/dev/urandom of=test bs=4M count=1024 2>/dev/null
 rm -f test
 for i in {1..1000}; do
 dd if=/dev/urandom of="\$i" bs=4M count=1 2>/dev/null
 done
 rm -f /mnt/*
 " | /usr/bin/time bash
 umount /mnt
done

umount "${TMPFS}"
rm -rf "${TMPFS}"

What’s going on here?

A tmpfs temporary mount point is created on 5 GB (right in the RAM, so it doesn’t depend on SSD/HDD)
Inside a file of 4 GB is created and then formatted as BTRFS with different hash algorithms
For each algorithm, two file writing modes are performed:
1. One large file of 4 GB
2. 1000 files of 4 MB
At the end, the time command outputs the script execution time

On my 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz processor, the best result was shown by the old and good crc32c (~35 seconds), while the others were around ~52 seconds.

Redis Migration Script

Sun, 25 Jun 2023 00:00:00 +0000

Old Method ¶

Sometimes you need to migrate data from one Redis server to another, or between other databases. Previously, I could transfer data from a server only through a full backup, but this does not allow for migrating data between bases. Here is the algorithm.

Enter into the original Redis and make Save

1
2
3


REDISCLI_AUTH="password"
redis-cli CONFIG GET DIR
redis-cli SAVE

The first command will print the path to the directory where the dump will be stored.

Add arguments to the target Redis to disable AppendOnly and restart it

1
2


--dbfilename dump.rdb
--appendonly no

Copy the dump to the target Redis and execute its loading

1
2


REDISCLI_AUTH="password"
redis-cli BGREWRITEAOF

Move the target Redis to the AppendOnly mode by removing previously added keys

Why migrate data between bases?

In my old cluster, I had two services, each with their own Redis server. In any Redis there are 16 default databases, why not use #0 for service A and #2 for service B? Here you hit into the migration problem from base to base.

New Method ¶

Based on MIGRATE. The script is presented below. Its advantages are also.

can migrate between bases
no need to restart Redis
no need to copy backups
There is a check after transferring by calculating the hash sum considering both key contents

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


#!/usr/bin/env sh
set -e

SRC_HOSTNAME='source-redis-hostname'
SRC_PORT='6379'
SRC_PASSWORD='password'
SRC_DATABASE='0'
DEST_HOSTNAME='destination-redis-hostname'
DEST_PORT='6379'
DEST_PASSWORD='password'
DEST_DATABASE='1'

main() {
 migrate | xargs -0 printf 'migration status: %s'
 CHECKSUM="$(mktemp)"
 hash --no-auth-warning -h "${SRC_HOSTNAME}" -p "${SRC_PORT}" \
 -a "${SRC_PASSWORD}" -n "${SRC_DATABASE}" | tee "${CHECKSUM}" | xargs -0 printf 'source checksum: %s'
 CHECKSUM_VERIFY="${CHECKSUM}" hash --no-auth-warning -h "${DEST_HOSTNAME}" -p "${DEST_PORT}" \
 -a "${DEST_PASSWORD}" -n "${DEST_DATABASE}" | xargs -0 printf 'target checksum validation: %s'
 rm -f "${CHECKSUM}"
}

hash() {
 EXTRA_ARGS="${@}"
 redis-cli ${EXTRA_ARGS} --raw --scan | \
 sed -r 's/"/\\"/g; s/^(.+)$/GET "\1"/g;' | \
 redis-cli ${EXTRA_ARGS} | sort | \
 if [ -f "${CHECKSUM_VERIFY}" ]; then
 sha256sum -c "${CHECKSUM_VERIFY}"
 else
 sha256sum -
 fi
}

migrate() {
 AUTH="--no-auth-warning -h ${SRC_HOSTNAME} -p ${SRC_PORT} -a ${SRC_PASSWORD} -n ${SRC_DATABASE}"
 redis-cli ${AUTH} --raw --scan | \
 tr '\n' '\0' | \
 xargs -0 redis-cli ${AUTH} \
 MIGRATE "${DEST_HOSTNAME}" "${DEST_PORT}" "" "${DEST_DATABASE}" 5000 \
 COPY REPLACE AUTH "${DEST_PASSWORD}" KEYS
}

main

My Experience with Git

Sun, 18 Jun 2023 00:00:00 +0000

Introduction ¶

Frequently used commands and explanations for them. I hope this will be as useful to someone else as it has been to me.

Main ¶

Git has a built-in graphical interface ¶

Gitk - a simple built-in graphical interface. I really like it for its precision and clarity. I try not to use any Git UI and do everything with commands, and this one works only in view mode.

Commit tree with tags and branches
Hash of the selected commit from the tree
List of changed files in this commit
Diff mode toggle
The diff of the commit with its description at the top

On Manjaro, in addition to Git itself, you need to install tk for it to work. I don’t know how it works on other distributions.

1

pacman -Sy git tk

How to update environment settings based on the diff of the new environment ¶

Here’s the thing. In my infrastructure repository, I have, for example, values for Helmfile environments Development and Production.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


.
├── development
│ ├── main.yaml
│ └── values
│ ├── cert-manager.yaml.gotmpl
│ ├── external-secrets.yaml.gotmpl
│ ├── fluentbit.yaml.gotmpl
│ ├── kube-prometheus-stack.yaml.gotmpl
│ ├── loki.yaml.gotmpl
│ └── promtail.yaml.gotmpl
└── production
 ├── main.yaml
 └── values
 ├── cert-manager.yaml.gotmpl
 ├── external-secrets.yaml.gotmpl
 ├── fluentbit.yaml.gotmpl
 ├── kube-prometheus-stack.yaml.gotmpl
 ├── loki.yaml.gotmpl
 └── promtail.yaml.gotmpl

5 directories, 14 files

So I was developing new applications, changing settings in development, and now I need to transfer all the same to production. To avoid recalling all the little things that need to be changed, I do the following.

Switch to the branch with the current development

1

git checkout development

Check when production was last changed

1

git log -n 1 --oneline -- ./production

Here, ./production is the path to the folder we are interested in for file changes. You can specify a specific file or multiple sources. In this case, we will get the hash of the last commit that made any changes in production.

Check the changes from that moment to the present

Now we need to track all changes from the last version of production to the current moment, which is the current development.

1

git diff 493d3aa..HEAD -- ./development

The hash 493d3aa is the one I obtained in the previous point for production.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


diff --git a/development/values/cert-manager.yaml.gotmpl b/development/values/cert-manager.yaml.gotmpl
index 493d3aa..cb34bd2 100644
--- a/development/values/cert-manager.yaml.gotmpl
+++ b/development/values/cert-manager.yaml.gotmpl
@@ -34,7 +34,8 @@ asdf:
 alpha: cert-manager-a
 contextKey:
 beta: something-else
- key: json
+ someSetting:
+ subkey: cert-manager-example
 urls:
 api: {{ quote .Values.apiURL }}
 web: {{ quote .Values.webURL }}

Thus, we can now manually transfer all the changes we are interested in to production without losing anything.

How to search for passwords and sensitive data using commit search ¶

Git has a built-in search for files in commits. You can use regular expressions to search for secret data. This is applicable in individual manual operations, but it would be much better to set up and constantly use Gitleaks or Trivy.

1
2


git rev-list --all | \
xargs git grep -irnE '"type":\s*"service_account"'

This way, you can find all files in history that contain secrets and perform rotation. It doesn’t make sense to cut them out of history; it’s better just to change the secrets.

Rebase strategies ¶

When you need to perform a rebase with a lot of conflicts and you know that you simply need to keep everything as it is in the current/target branch, you can do the following.

Make sure that our local branch (which we will be rebasing) is pushed and is the same as in the cloud.

Why is this necessary? For self-control. For example, I have a task to rebase the development branch onto main without changing it, just moving it. If at the start of the transfer I checked that there was no diff for origin/development, then after the transfer, I can compare git diff origin/development..development, and it should also be empty.

1
2


git pull
git push

Perform one of the operations below

1
2


git rebase -i --strategy=ort -Xtheirs main --empty=keep
git rebase -i --strategy=ort -Xours main --empty=keep

Git will resolve all conflicts in favor of one of the branches, depending on ours or theirs.

How to reset a local branch to the state of the cloud if someone pushed with force ¶

This is quite simple. For example, we are working on development.

Switch to the branch

1

git checkout development

Perform git fetch

1

git fetch --all

Perform a reset

1

git reset --hard origin/development

That’s it; our branch is now exactly as it is in the cloud.

Kubernetes in Docker

Mon, 12 Jun 2023 00:00:00 +0000

Introduction ¶

Sometimes you need Kubernetes locally for quick testing.

Checking out a custom Helm chart
Asking kubectl explain something
Checking settings

Starting to look for some cluster on first available project, which is not good because it can break something accidentally. It’s not convenient to do something. Turns out there are many different lightweight and fast installers of Kubernetes. This may not be all of them, but those I either tried or passed by. They work a bit differently.

Minikube - one of the oldest, so to say. We met that standard installation kube-prometheus-stack on it simply didn’t start, and on the next one - with it.
k3s - one of three engines from Rancher. Great thing, used and using them for small projects at work. It’s also two personal clusters exactly on k3s, but it’s not vanilla, everything is put together in a single binary. It’s both good and bad. Very lightweight and easy to install, but I ended up with Wireguard VPN rule issues with IPTables myself. I had to change Wireguard FwMark to 0x8000 and everything worked.
rke - another one of three engines. First one from Rancher. It’s very good, especially because there is a Terraform Module - great. I’ve already been more similar to vanilla installation.
rke2 - rke + k3s. Everything is put together in one binary, but apiserver, scheduler etc. are still seen as separate pods. I tried it, and it was a shame that there was no support from Terraform like with rke. It didn’t impress me, practically speaking, because it was less stable than k3s or maybe my hands were just crooked.
Microk8s - from Canonical, haven’t tried.
k0s - same as k3s, but community is smaller and documentation is worse, and that’s… good.
kind - the highlight of today’s program.

kind ¶

Why I want to tell you about it? It’s the most vanilla way to get Kubernetes locally. There are many advantages.

You can throw in a config
Creates Kubernetes in Docker
You can create more than one node!
Uses vanilla kubeadm!
All configurations, paths, and formats - standard kubeadm!
It doesn’t mess with iptables on the host

There are some downsides.

Control plane eats up 900 MiB RAM at a time
You need to throw out ports from nodes on the host

Here’s an example configuration of kind:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
 extraMounts:
 - hostPath: /home/bzm/projects/kind/
 containerPath: /host
 kubeadmConfigPatches:
 - |
 kind: InitConfiguration
 nodeRegistration:
 kubeletExtraArgs:
 seccomp-default: "true"
 apiServer:
 extraArgs:
 enable-admission-plugins: NodeRestriction
- role: worker
 extraPortMappings:
 - containerPort: 80
 hostPort: 80
 listenAddress: "127.0.0.1"
 - containerPort: 443
 hostPort: 443
 listenAddress: "127.0.0.1"

So, you can put a bunch of nodes on one computer and everything is in containers. Configurations and settings are all like the official documentation. You need to try out some keys for api-server - easy. You need to refine something about security - easy. I really liked it.

Redis Sentinel Watchdog

Fri, 09 Jun 2023 00:00:00 +0000

Introduction ¶

I really love the charts from Bitnami; they are well-written and quite stable. Among other things, I always deploy Redis from their chart. Everything would be fine if you just needed Redis without any High Availability, but that’s not always the case.

What’s the difficulty in setting up Redis HA?

The issue is that the connection drivers to Redis in client services must support Redis Sentinel. When we set up a cluster with Redis, a container with Sentinel is also launched—it’s a kind of controller for the cluster that will replace the master if the current one goes down. It’s very similar to a Mongo Arbiter. Therefore, the connection mechanism changes because of this Sentinel.

Connect to Sentinel
Request the master’s address
Connect to the master
If the master is unavailable, go back to Sentinel

Very often, applications do not support this. So what should you do in such a case? I decided that, as always, workarounds would help.

Below is a fragment of values.yaml that I use to launch bitnami/redis using Helmfile.

What’s going on there?

Among other things, I create an additional sidecar container that requests the master’s address from Sentinel every N seconds and updates the service if it differs from the one created as an ExternalName type. Thus, we can simply point applications that do not know how to communicate with Sentinel to this service, and everything will work since the service will always point to the correct pod.

This sidecar container consumes about 65 mCPU and 1 MiB. Of course, you could build an image and include redis-cli to connect directly, but I don’t want to maintain and store this image or update it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132


# ┬─┐┬─┐┬─┐o┐─┐
# │┬┘├─ │ ││└─┐
# ┘└┘┴─┘┘─┘┘──┘

{{- $name := "redis" }}
{{- $masterServiceName := printf "%s-master" $name }}
nameOverride: {{ $name }}
fullnameOverride: {{ $name }}
architecture: replication
auth:
 enabled: true
 password: {{ quote .Values.secrets.redisPassword }}

replica:
 replicaCount: 3
 resources:
 limits:
 memory: 75Mi
 requests:
 cpu: 50m
 memory: 25Mi
 persistence:
 enabled: true
 storageClass: standard
 size: 1Gi
 podAntiAffinityPreset: hard
 sidecars:
 - name: master-service-watchdog
 image: '{{`{{ printf "bitnami/kubectl:%s.%s" .Capabilities.KubeVersion.Major .Capabilities.KubeVersion.Minor }}`}}'
 imagePullPolicy: IfNotPresent
 command: ["/bin/sh", "-ec"]
 args:
 - |
 while sleep 5; do
 MASTER_ADDRESS="$(kubectl exec -c sentinel "${POD_NAME}" -- sh -c "REDISCLI_AUTH='$REDISCLI_AUTH' redis-cli -p '${REDIS_SENTINEL_PORT}' SENTINEL get-master-addr-by-name this" | head -1)"
 if ! kubectl get svc "${MASTER_SERVICE}" -o jsonpath='{.spec.externalName}' | grep -qF "${MASTER_ADDRESS}"; then
 echo "info: master address has changed to ${MASTER_ADDRESS}"
 jq -Mnr --arg address "${MASTER_ADDRESS}" '[{op:"replace",path:"/spec/externalName",value:$address}]' | \
 xargs -0 kubectl patch svc "${MASTER_SERVICE}" --type=json --patch
 fi
 done
 env:
 - name: MASTER_SERVICE
 value: {{ quote $masterServiceName }}
 - name: POD_NAME
 valueFrom:
 fieldRef:
 fieldPath: metadata.name
 - name: REDISCLI_AUTH
 valueFrom:
 secretKeyRef:
 name: '{{`{{ template "common.names.fullname" . }}`}}'
 key: redis-password
 - name: REDIS_SENTINEL_PORT
 value: '{{`{{ .Values.sentinel.containerPorts.sentinel }}`}}'
 resources:
 limits:
 cpu: 100m
 memory: 50Mi
 requests:
 cpu: 100m
 memory: 50Mi
 securityContext:
 readOnlyRootFilesystem: true
 runAsNonRoot: true
 runAsUser: 1000
 privileged: false
 allowPrivilegeEscalation: false
 capabilities:
 drop: [ALL]
 seccompProfile:
 type: RuntimeDefault

serviceAccount:
 create: true

sentinel:
 enabled: true
 masterSet: this
 automateClusterRecovery: true
 downAfterMilliseconds: 2000
 resources:
 limits:
 memory: 50Mi
 requests:
 cpu: 50m
 memory: 50Mi

networkPolicy:
 enabled: true
 extraIngress:
 - ports:
 - port: 6379
 from:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: {{ .Release.Namespace }}

rbac:
 create: true
 rules:
 # Allow watchdog to update our service
 - apiGroups: [""]
 resources: ["services"]
 verbs: ["get", "update", "patch"]
 resourceNames: [{{ quote $masterServiceName }}]
 # Allow exec into Redis pods
 - apiGroups: [""]
 resources: ["pods/exec"]
 verbs: ["create"]
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list"]
pdb:
 create: true

raw:
 Service:
 apiVersion: v1
 kind: Service
 metadata:
 name: {{ $masterServiceName }}
 spec:
 type: ExternalName
 # 1.1.1.1 is just a stub value that is used for initial deployment
 # It will be overridden as soon as the first watchdog is ready
 externalName: 1.1.1.1
 ports:
 - port: 6379
 protocol: TCP
 targetPort: 6379
 name: redis

Timeshift BTRFS Backups on Manjaro

Sun, 01 Jan 2023 00:00:00 +0000

Introduction ¶

Timeshift is a great tool! It helps making automated system backups. You can do it manually if you want to, but BTRFS snapshots are where it’s at!

Copying somewhere with rsync, with clever linking to save space as much as possible
Using BTRFS snapshot

The first option is pointless. This takes forever and isn’t exciting, but BTRFS is what you need! It’s done instantly and where duplicates are, the file system saves it itself. Plus, you can load them right when booting up.

System Requirements ¶

Part of the page on the main repository, but I also found some more:

The default subvolume for the BTRFS filesystem should be /

When mounting a BTRFS partition either without specifying the mount option subvol= it will mount at the root. If not, the program crashes. Here’s a script to check and correct this:

1
2
3
4


MP="$(mktemp -d)"
mount | awk '/on \/ type btrfs/{print $1}' | sudo xargs -I{} mount {} "$MP" && \
sudo btrfs subvolume set-default 5 "$MP"; \
sudo umount "$MP"

The root filesystem partition in /etc/fstab should be correctly configured

The same subvol= option should be set to either @* or /@, it doesn’t matter. Here’s a script to tell if everything is correct:

1
2
3
4
5
6


grep -E '^[^#].+/\s+btrfs' /etc/fstab | \
grep -oE 'subvol=[^,]+' | \
cut -d= -f2 | \
grep -qE '^/?@$' && \
echo 'OK' || \
echo 'Not OK'

How it works ¶

When installing any package through pamac or yay, timeshift automatically kicks in if etc/timeshift-autosnap.conf doesn’t include its skip.

Convenient and Colorful ZSH with p10k

Sun, 25 Dec 2022 00:00:00 +0000

Introduction ¶

Previously, I described my experience with ZSH here. Now, things are a bit different; I have fully switched to Powerlevel10k.

It’s a very beautiful and convenient shell prompt that is easy to install and configure on the fly.

Configuring p10k ¶

You can configure p10k very easily, right on the go, even ten times a day.

1

p10k configure

I have embedded the installation of p10k directly in my .zshrc, so you don’t even have to follow the official instructions on the website. Well, almost. There are recommended fonts to ensure everything looks good, which are described here. Below, I recorded and showed what the configuration of p10k looks like. Unfortunately, the asciinema recording shows the beautiful elements as squares since these symbols are not supported in the recording. So, here’s a screenshot of the terminal window.

While the recording didn’t turn out very well, at least I can show you the menus.

Explanation of my .zshrc ¶

Everything written below consists of snippets from my .zshrc, which you can download at the end of the article. For your convenience, I will duplicate the link here.

Installing p10k ¶

This is done with a simple condition and direct cloning from GitHub. If we have Git and p10k is not yet installed, we can clone it.

1
2
3
4


if command -v git >/dev/null && [ ! -d ~/powerlevel10k ]; then
 echo "info: installing p10k"
 git clone --depth 1 --single-branch https://github.com/romkatv/powerlevel10k.git ~/powerlevel10k
fi

Plugins ¶

I install plugins simply by cloning them from GitHub. I added comments directly in the code for clarification.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


# if we have git installed
if command -v git >/dev/null; then
 # and folder with plugins is absent
 if [ ! -d /usr/share/zsh/plugins ]; then
 # let's create it with sudo
 sudo mkdir -p /usr/share/zsh/plugins
 fi
 # and jump in
 cd /usr/share/zsh/plugins
 # now, let's read every line between <<EOF and EOF lines below
 while IFS= read -r REPO; do
 # and for every line, which will be available in variable named REPO
 # let's cut only last part of the url (ex: zsh-autosuggestions.git) and save to BASENAME var
 BASENAME="$(basename "$REPO")"
 # now, we will remove .git from the end of BASENAME and check either such folder exists
 if [ ! -d "${BASENAME%.git}" ]; then
 # if not, let's print an info message
 echo "${REPO}: Cloning to $(pwd)/${BASENAME%.git}"
 # and finally clone that plugin
 sudo git clone --depth 1 --single-branch "$REPO"
 fi
 done <<EOF
https://github.com/zsh-users/zsh-history-substring-search.git
https://github.com/zsh-users/zsh-syntax-highlighting.git
https://github.com/zsh-users/zsh-autosuggestions.git
EOF
 # after all plugins cloned, we cd back to the location we were
 cd - >/dev/null
fi
# okay, now we assume that all plugins are cloned and we have to include them
# once again, for every hardcoded path to .zsh file
while IFS= read -r SOURCE; do
 # we skip the plugin if it starts with '#' (commented out)
 if echo "$SOURCE" | grep -qE '^#'; then
 continue
 elif [ -s "$SOURCE" ]; then
 # if it is enabled, we just include it in current shell session with source command
 source "$(realpath "$SOURCE")"
 fi
done <<EOF
/usr/share/zsh/plugins/zsh-history-substring-search/zsh-history-substring-search.zsh
/usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
/usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh
EOF

Adding Paths to PATH ¶

Often, we install additional binaries in non-standard folders. For example, I use ~/.local/bin for various tools like kubectl, helm, k9s, and others. Here, we check each folder, if it exists, and add it to the PATH variable.

1
2
3
4
5
6
7
8
9


while IFS= read -r P; do
 if [ -d "$P" ]; then
 export PATH="${P}:${PATH}"
 fi
done <<EOF
${HOME}/.local/share/gem/ruby/3.0.0/bin
${HOME}/projects/stm32/gcc-arm-none-eabi-10.3-2021.10/bin
${HOME}/.local/bin
EOF

Aliases ¶

Here everything is straightforward, with lots of different aliases.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


alias ..="cd .. && ls -lhA --color=auto"
alias d='sudo -E docker'
alias dc="docker-compose"
alias e="exit"
alias j="sudo journalctl"
alias l="ls -lh --color=always"
alias la="ls -lhA --color=always"
alias p="ping 8.8.8.8"
alias please='sudo'
alias reborn="sudo shutdown -r now"
alias s='sudo'
alias sctl="sudo systemctl"
alias svi="sudo vim"
alias svim="sudo vim"
alias t='terraform'
alias k="kubectl"
alias kd="kubectl describe"
alias kg="kubectl get"
alias ky="kubectl get -o yaml"
alias ke='kubectl exec -it'
alias vi="vim"
alias g="git"
alias b="sudo btrfs"
alias c="code ."
alias ansible-time="time ansible-playbook"

Git Config ¶

By the way, I also have a Git config. The link is here or the same at the end of the page. Two interesting commands here:

git ps - pushes the branch, and if it doesn’t exist yet, it catches the message from the Git error log and pushes with –set-upstream.
git dpush (dummy push) - adds all files and folders to the commit, writes a standard commit message, and pushes.
git cc - simply makes a commit with a standard message.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[alias]
co = checkout
a = add .
ci = commit
cc = commit -m 'Nothing special. Just regular commit.'
ps = ! sh -c 'git push >/tmp/git-ps 2>&1 || grep -- "git push --set-upstream" /tmp/git-ps | sh'
pt = push --tags
pts = ! sh -c 'git push && git push --tags'
pl = pull
st = status
br = branch
hist = log --pretty=format:'%h %ad | %s%d [%an]' --graph --date=short
type = cat-file -t
dump = cat-file -p
dpush = ! sh -c 'git add . && git commit -m "Nothing special. Just regular commit."' && git push

KUBECONFIG ¶

Here, we look for the standard kubeconfig for k3s or rke2 and set it in KUBECONFIG.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# if there is no file at the path that is already being set in KUBECONFIG var
if [ -z "$KUBECONFIG" ]; then
 # for every filepath from the list below
 while IFS= read -r K; do
 # check if file exists and it is not empty
 if [ -s "$K" ]; then
 # change owner and group to current user
 if ! stat -c '%u:%g' "$K" | grep -qF "$(id -u):$(id -g)"; then
 sudo chown "${USER}:${USER}" "$K"
 echo "info

: chown $K"
 fi
 # assign it to the KUBECONFIG variable
 export KUBECONFIG="${K}:${KUBECONFIG}"
 fi
 done <<EOF
/etc/rancher/rke2/rke2.yaml
~/.kube/config
~/.kube/k3s.yaml
EOF
fi

Here’s the translated content of your Hugo page section into English, formatted in valid Markdown:

Automatic Utility Downloads ¶

Here’s what’s happening: .zshrc automatically checks to ensure that we have the binaries of the utilities we use installed. If the utility is missing, it downloads it; if not, it does nothing. However, I can manually update k9s by executing _update_k9s. This section installs the following utilities:

k9s - more details in my article
kubepug - a tool for checking deprecated APIs in the cluster before upgrading the Kubernetes version
kubecolor - a wrapper around kubectl for colorful output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


_update_kubepug() {
 mkdir -p ~/.local/bin
 LOCATION="$(curl --connect-timeout 1 -sS https://github.com/rikatz/kubepug/releases/latest -D- | awk '/^location:/{print $2}')"
 LATEST_VERSION="$(echo "$LOCATION" | awk -F/ '{print $NF}' | grep -oE '[0-9a-zA-Z_.-]+')"
 if ! k9s version --short | grep -qF "$LATEST_VERSION"; then
 printf "Updating kubepug to %s\n" "$LATEST_VERSION"
 printf 'https://github.com/rikatz/kubepug/releases/download/%s/kubepug_linux_%s.tar.gz' \
 "$LATEST_VERSION" "$(uname -p | sed 's/unknown/amd64/g' | tr -d '[[:space:]]')" | \
 xargs curl -sSL | \
 tar -xpzf - kubepug
 install kubepug ~/.local/bin/kubepug -m 0755
 rm -f kubepug
 fi
}
if ! command -v kubepug &>/dev/null; then
 _update_kubepug
fi

_update_k9s() {
 mkdir -p ~/.local/bin
 LOCATION="$(curl --connect-timeout 1 -sS https://github.com/derailed/k9s/releases/latest -D- | awk '/^location:/{print $2}')"
 LATEST_VERSION="$(echo "$LOCATION" | awk -F/ '{print $NF}' | grep -oE '[0-9a-zA-Z_.-]+')"
 if ! k9s version --short | grep -qF "$LATEST_VERSION"; then
 printf "Updating k9s to %s\n" "$LATEST_VERSION"
 printf 'https://github.com/derailed/k9s/releases/download/%s/k9s_Linux_%s.tar.gz' \
 "$LATEST_VERSION" "$(uname -p | sed 's/unknown/x86_64/g' | tr -d '[[:space:]]')" | \
 xargs curl -sSL | \
 tar -xpzf - k9s
 install k9s ~/.local/bin/k9s -m 0755
 rm -f k9s
 fi
}
if ! command -v k9s &>/dev/null; then
 _update_k9s
fi

_update_kubecolor() {
 mkdir -p ~/.local/bin
 LOCATION="$(curl --connect-timeout 1 -sS https://github.com/hidetatz/kubecolor/releases/latest -D- | awk '/^location:/{print $2}')"
 LATEST_VERSION="$(echo "$LOCATION" | awk -F/ '{print $NF}' | grep -oE '[0-9a-zA-Z_.-]+' | sed 's/^v//g')"
 if ! kubecolor --kubecolor-version | grep -qF "$LATEST_VERSION"; then
 printf "Updating kubecolor to %s\n" "$LATEST_VERSION"
 printf 'https://github.com/hidetatz/kubecolor/releases/download/%s/kubecolor_%s_Linux_%s.tar.gz' \
 "v$LATEST_VERSION" "$LATEST_VERSION" "$(uname -p | sed 's/unknown/x86_64/g' | tr -d '[[:space:]]')" | \
 xargs curl -sSL | \
 tar -xpzf - kubecolor
 install kubecolor ~/.local/bin/kubecolor -m 0755
 rm -f kubecolor
 fi
}
if ! command -v kubecolor &>/dev/null; then
 _update_kubecolor
 alias kubectl="kubecolor"
 typeset -g POWERLEVEL9K_KUBECONTEXT_SHOW_ON_COMMAND="${POWERLEVEL9K_KUBECONTEXT_SHOW_ON_COMMAND}|kubecolor"
fi

Additionally, for several commands, if they exist, automatic loading of completion scripts is performed so that zsh knows what to suggest.

1
2
3
4
5
6


## Completions for 3rd-party binaries
for COMMAND in kubectl helm k3d flux kubepug; do
 if command -v "$COMMAND" >/dev/null; then
 source <("$COMMAND" completion zsh)
 fi
done

Separate Environments ¶

This is the smartest and at the same time simplest trick in my arsenal. The most valuable thing in work is that when you work on different projects/environments, the scariest thing you can do is to break something. The easiest way to do that is forgetting to switch to the desired environment and apply something where you didn’t intend.

This snippet of code prevents exactly that.

1
2
3


if [ -f env.sh ]; then
 source ./env.sh
fi

It’s that simple. What actually happens? When zsh starts, it checks the directory it was launched from, and if there’s a file named exactly env.sh, it includes it in the context. Let me explain.

I use Ranger for navigation through folders from the terminal. Let’s say I’m working on a project and also have a blog. I created the following directories for my projects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


.
├── MyBlog
│   ├── env.sh
│   └── kubeconfig.yaml
└── SuperWorkProject
 ├── development
 │   ├── env.sh
 │   └── kubeconfig.yaml
 └── production
 ├── env.sh
 └── kubeconfig.yaml

4 directories, 6 files

In my production project, I have two environments, all running Kubernetes, each with its separate kubeconfig.yaml. Let’s say I need to work on production. In its env.sh, I will write the following.

1
2
3
4
5
6
7
8


#!/usr/bin/env sh

export KUBECONFIG='~/projects/SuperWorkProject/production/kubeconfig.yaml'
export AWS_PROFILE='production'

pf() {
 nohup sh -c 'kubectl get po -n monitoring -l app.kubernetes.io/name=grafana -o name | xargs -I{} kubectl port-forward {} 3000' 1>/dev/null &
}

Now, I just open zsh in this folder or run source ./env.sh. That’s it; now I’m sure I’m working with production. Additionally, I declared a function for port-forwarding.

Kubernetes Authorization via OIDC

Thu, 01 Dec 2022 00:00:00 +0000

Introduction ¶

To authenticate in Kubernetes through kubeconfig for access via kubectl, ServiceAccounts are generally used. They are simply easier to create, but this is not the right approach. Tokens from ServiceAccounts are accessible to anyone who can read secrets in the namespace where the ServiceAccount was created, which means that actions can be taken on your behalf. Not secure…

You can create a certificate, sign it in the cluster, and operate as a User. This is much better; no one can do anything on your behalf because we generated the private part locally and did not share it. I’ll explain how to do this since I’ve figured it out.

Recently, I finally tried Authentik and also attempted to link it to the cluster via API server configuration. It turned out to be even simpler in some aspects than dealing with certificates for users. This is what I will talk about.

Access via ServiceAccount ¶

Let’s create a ServiceAccount.

1

kubectl create serviceaccount test

Previously, before Kubernetes 1.22, a token was automatically generated; now it needs to be created separately. You can generate a temporary one.

1

kubectl create token test --duration=10m

Alternatively, you can obtain a permanent token by creating a secret for it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: test-sa-secret
 annotations:
 kubernetes.io/service-account.name: test
type: kubernetes.io/service-account-token
EOF
SA_TOKEN="$(kubectl get -o yaml secret test-sa-secret -o jsonpath='{.data.token}' | base64 -d)"

Now we can write it in kubeconfig and use it.

1
2
3
4


kubectl config current-context | \
xargs kubectl config get-contexts --no-headers | \
awk '{print $4}' | \
xargs kubectl config set-credentials "--token=${SA_TOKEN}"

Creating a User ¶

This part is a bit more complicated. You need to create a certificate, a signing request, sign it, retrieve the public part, and only then are we ready to proceed. I’m attaching my script, but I don’t want to describe all of this. I’m focusing on OIDC, so consider this as a nice bonus 😄

How to use it:

Write your name in the variable NAME
Write your company name in the variable GROUP
Change CLUSTERROLE if you really want to
Run the script and read the instructions on the screen

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


NAME='john-johnson'
GROUP='asdfqwer'
CLUSTERROLE='cluster-admin'

openssl ecparam -genkey -name prime256v1 | openssl ec -out "$NAME.key"
KEY_BASE64="$(base64 -w0 "$NAME.key")"
openssl req -new -key "$NAME.key" -out "$NAME.csr" -subj "/CN=$NAME/O=$GROUP"
CSR_BASE64="$(base64 -w0 "$NAME.csr")"
rm -f "$NAME.csr" "$NAME.key"

cat <<EOF
# ┬ ┐┐─┐┬─┐┬─┐ ┌─┐┬─┐┬─┐┬─┐┌┐┐o┌─┐┌┐┐
# │ │└─┐├─ │┬┘ │ │┬┘├─ │─┤ │ ││ ││││
# ┘─┘──┘┴─┘┘└┘ └─┘┘└┘┴─┘┘ ┘ ┘ ┘┘─┘┘└┘

1. Please, execute the following code using kubectl that has access to the cluster (using default admin credentials, for example)

kubectl apply -f - <<EOT
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
 name: $GROUP:$NAME
spec:
 expirationSeconds: 31536000 # 1 year
 signerName: kubernetes.io/kube-apiserver-client
 usages: ["client auth"]
 username: $NAME
 request: $CSR_BASE64
EOT
kubectl certificate approve '$GROUP:$NAME'
kubectl apply -f - <<EOT
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: user:${NAME}:${CLUSTERROLE}
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: $CLUSTERROLE
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: User
 name: $NAME
EOT
kubectl get -o jsonpath='{.status.certificate}' csr '$GROUP:$NAME'
echo

2. Please, execute the following code locally, on your PC, where you want to have those credentials inserted

kubectl config set "users.$GROUP:$NAME.client-key-data" "$KEY_BASE64"
kubectl config set "users.$GROUP:$NAME.client-certificate-data" "-" --set-raw-bytes=true

3. Now place the certificate that has been printed during step #1 in your kubeconfig client-certificate-data for user $GROUP:$NAME marker
4. Specify $GROUP:$NAME user in the context you want to use
EOF

Google OIDC ¶

Configure the OAuth consent screen

Create OAuth2 credentials of type Web application, Authorized redirect URIs - in the list below.

1
2
3
4


http://127.0.0.1:18000
http://127.0.0.1:8000
http://localhost:18000
http://localhost:8000

As a result, you will get the Client ID and Client Secret.

Great, now you need to install the kubectl oidc-login plugin.

After installation, try to ensure everything works.

1
2
3
4


kubectl oidc-login setup \
 --oidc-issuer-url=https://accounts.google.com \
 --oidc-client-id='YOUR CLIENT ID' \
 --oidc-client-secret='YOUR CLIENT SECRET'

If everything is fine, your browser should open, you log in as someone, and a big beautiful instruction will appear in the console. Something like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


authentication in progress...
Opening in existing browser session.

## 2. Verify authentication

You got a token with the following claims:

{
 "iss": "https://accounts.google.com",
 "azp": "longalphanum",
 "aud": "longalphanum",
 "sub": "longalphanum",
 "at_hash": "longalphanum",
 "nonce": "longalphanum",
 "iat": longalphanum,
 "exp": longalphanum
}

## 3. Bind a cluster role

Run the following command:

 kubectl create clusterrolebinding oidc-cluster-admin --clusterrole=cluster-admin --user='https://accounts.google.com#12345678901234567890'

## 4. Set up the Kubernetes API server

Add the following options to the kube-apiserver:

 --oidc-issuer-url=https://accounts.google.com
 --oidc-client-id=longalphanum

## 5. Set up the kubeconfig

Run the following command:

 kubectl config set-credentials oidc \
 --exec-api-version=client.authentication.k8s.io/v1beta1 \
 --exec-command=kubectl \
 --exec-arg=oidc-login \
 --exec-arg=get-token \
 --exec-arg=--oidc-issuer-url=https://accounts.google.com \
 --exec-arg=--oidc-client-id=longalphanum \
 --exec-arg=--oidc-client-secret=longalphanum

## 6. Verify cluster access

Make sure you can access the Kubernetes cluster.

 kubectl --user=oidc get nodes

You can switch the default context to oidc.

 kubectl config set-context --current

 --user=oidc

Authentik ¶

This tool is great! Among its direct analogs, it’s worth mentioning Okta and Keycloak. All three products are excellent; Okta is often considered the benchmark. Other options may be better or worse, but they are always compared to Okta. 😄

I installed Authentik from the official chart. It’s running locally, and I needed a certificate. I obtained one using Certbot with DNS verification. After that, we proceed as follows:

Applications > Providers

Create a Kubernetes provider of type OAuth2/OIDC
- Client type: Confidential
- Client ID: remember this
- Client Secret: remember this
- Redirect URIs: http://(127.0.0.1|localhost):1?8000(/.*)?
Application > Applications

Create an application Kubernetes and select the provider you created in step 1.
System

Create users (surprise, right?) and also create groups. I made kubernetes:admin and kubernetes:restricted. The first group will be assigned the cluster-admin role in Kubernetes, while the second will have read-only access. Don’t forget to add users to the corresponding groups.
System > Certificates

I’m not sure about this step; I just haven’t checked it without a certificate. I uploaded the certificate obtained via Certbot here, and then selected this certificate in Application > Providers > Kubernetes > Edit > Signing Key. Kubernetes didn’t complain, so it might be possible to use a self-signed certificate, but I’m not sure; you’ll have to try it yourself.

Customization > Property Mappings

Create a mapping with the following parameters:

Name: Kubernetes
Scope name: kubernetes

Expression

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


all_groups = [
 "kubernetes:admin",
 "kubernetes:restricted"
]
return dict(
 groups=[
 group for group in all_groups
 if ak_is_group_member(request.user, name=group)
 ]
)

This way, for those who request this scope, Authentik will append the groups the user belongs to, iterating only through the Kubernetes groups.

Verify kube-oidc-login

1
2
3
4
5


kubectl oidc-login setup \
 --oidc-issuer-url=https://authentik.example.com/application/o/kubernetes/ \
 --oidc-client-id='YOUR CLIENT ID' \
 --oidc-client-secret='YOUR CLIENT SECRET' \
 --oidc-extra-scope=kubernetes

Everything should work; just follow the instructions.

Grant permissions to the group in Kubernetes

Authentik will pass the group list, and now we need Kubernetes to pay attention to this. Add the following arguments to the API server:

1
2


--oidc-groups-claim=groups
--oidc-groups-prefix=oidc:

Now you can create resources like these:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: oidc:kubernetes:admin
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: oidc:kubernetes:admin
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: this-name-is-custom
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: view
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: oidc:kubernetes:restricted

wg-quick network namespace

Fri, 25 Nov 2022 00:00:00 +0000

TL;DR ¶

1
2
3


PreUp = ip netns list | grep -qF -- %i && ip netns delete %i || true
PostUp = (ip netns list | grep -qF -- %i || ip netns add %i) && ip -n %i link set up lo && IPS="$(ip address save dev %i | base64)" && ip link set %i netns %i && echo "$IPS" | base64 -d | ip -n %i address restore dev %i && ip -n %i link set up %i && (ip -n %i route show default | grep -qF default || ip -n %i route add default dev %i) && echo 'IyEvdXNyL2Jpbi9lbnYgc2gKc2V0IC1lCgpuZXRucz0nJXMnCgpjYXQgPDxFT0YgfCB4YXJncyAtMCBzdWRvIC1FIG5zZW50ZXIgIi0tbmV0PS92YXIvcnVuL25ldG5zLyRuZXRucyIgdW5zaGFyZSAtLW1vdW50IHNoIC1jCmZvciBjb25mIGluICIkbmV0bnMiICJ0dW4uJG5ldG5zIjsgZG8KICBpZiBbIC1mICIvdmFyL3J1bi9yZXNvbHZjb25mL2ludGVyZmFjZXMvXCRjb25mIiBdOyB0aGVuCiAgICBtb3VudCAtLWJpbmQgIi92YXIvcnVuL3Jlc29sdmNvbmYvaW50ZXJmYWNlcy9cJGNvbmYiIC9ldGMvcmVzb2x2LmNvbmYKICBmaQpkb25lCmV4cG9ydCBEQlVTX1NFU1NJT05fQlVTX0FERFJFU1M9dW5peDpwYXRoPS9kZXYvbnVsbApzdWRvIC1FIC11ICcjJHtTVURPX1VJRDotJChpZCAtdSl9JyAtZyAnIyR7U1VET19HSUQ6LSQoaWQgLWcpfScgLS0gJEAKRU9GCg==' | base64 -d | xargs -0 -I{} printf {} %i > /usr/local/bin/wg-%i-exec && echo 'IyEvdXNyL2Jpbi9lbnYgc2gKc2V0IC1lCgppZiBbICIkKGlkIC11KSIgLW5lIDAgXTsgdGhlbgogIGVjaG8gImVycm9yOiBydW4gbWUgYXMgcm9vdCIgMT4mMgogIGV4aXQgMQpmaQoKbmV0bnM9JyVzJwppcCBuZXRucyBleGVjICIkbmV0bnMiIGlwIGxpbmsgc2V0ICIkbmV0bnMiIG5ldG5zIDEKaXAgbmV0bnMgZGVsZXRlICIkbmV0bnMiCnN5c3RlbWN0bCBzdG9wICJ3Zy1xdWlja0AkbmV0bnMuc2VydmljZSIK' | base64 -d | xargs -0 -I{} printf {} %i > /usr/local/bin/wg-%i-down && chmod 0755 /usr/local/bin/wg-%i-* && chown root:root /usr/local/bin/wg-%i-* && echo 'bmV0bnM9JyVzJwpjYXQgPDxFT0YKIyAg4pSQIOKUrG/ilKzilIDilJDilKzilIDilJDilIzilIDilJDilKwg4pSQ4pSs4pSA4pSQ4pSs4pSA4pSQ4pSs4pSA4pSQICDilIzilJDilJDilKzilIDilJDilIzilJDilJDilIzilJDilJDilJDilIDilJAKIyAg4pSC4pSC4pSC4pSC4pSC4pSs4pSY4pSc4pSAIOKUgiDilKzilIIg4pSC4pSC4pSA4pSk4pSC4pSs4pSY4pSCIOKUgiAg4pSC4pSC4pSC4pSc4pSAICDilIIg4pSC4pSC4pSC4pSU4pSA4pSQCiMgIOKUlOKUtOKUmOKUmOKUmOKUlOKUmOKUtOKUgOKUmOKUmOKUgOKUmOKUmOKUgOKUmOKUmCDilJjilJjilJTilJjilJjilIDilJggIOKUmOKUlOKUmOKUtOKUgOKUmCDilJgg4pSY4pSU4pSY4pSA4pSA4pSYClRoYW5rcyBmb3IgdXNpbmcgbmV0bnMgd2ctcXVpY2sgd3JhcHBlciEKCi91c3IvbG9jYWwvYmluL3dnLSRuZXRucy1leGVjIENPTU1BTkQgICAtIGV4ZWN1dGUgc2hlbGwgY29tbWFuZCBpbiAkbmV0bnMgbmV0d29yayBuYW1lc3BhY2UKL3Vzci9sb2NhbC9iaW4vd2ctJG5ldG5zLWRvd24gICAgICAgICAgIC0gdXNlIGl0IGluc3RlYWQgb2Ygd2ctcXVpY2sgZG93biAkbmV0bnMKRU9G' | base64 -d | xargs -0 -I{} printf {} %i | sh - 
PostDown = rm -f /usr/local/bin/wg-%i-* || true

Introduction ¶

Wireguard is an awesome, simple VPN that is already included in the kernel. A very powerful and simple tool. It’s much easier to set up than OpenVPN. For its simple configuration, the wg-quick utility is used, which is part of wireguard tools. For better understanding, I’ll additionally clarify that from my experience, most solutions like WGDashboard and wireguard-ui generate client configs specifically for wg-quick.

Problem Statement ¶

Move the wireguard interface to a separate network namespace so that applications can be run in it. Why do this? This eliminates the need to proxy traffic to blocked sites through Wireguard with some complex solutions. Need to access some restricted place - just launch another browser in the Wireguard namespace and that’s it! For example, I rewrote the Exec line in the Thunderbird shortcut, which has no equally functional alternative yet, and now it always starts through VPN and can peacefully collect mail from any SMTP. We need to use vanilla wg-quick without modifications so it works out of the box.

Solution Analysis ¶

In the documentation, there are 4 ways for additional wg-quick interface configuration: PreUp, PostUp, PreDown, PostDown. So we need to write scripts that do everything needed. The ready solution is presented at the beginning of the article, and now let’s analyze it. I’ll transform this compressed code into human-readable form - it looks different but is identical in content. Note: the code contains %i, this is a wg-quick macro itself, wherever it encounters this macro, it substitutes the name of the interface being created.

PreUp ¶

Everything is simple here.

1
2
3
4
5


# search for our namespace in the list of network namespaces
if ip netns list | grep -qF -- %i; then
 # if found - delete it
 ip netns delete %i
fi

PostUp ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78


if ! ip netns list | grep -qF -- %i; then
 # create network namespace if it doesn't exist yet (and it shouldn't)
 ip netns add %i
fi
# start the loopback interface there (the 127.0.0.1 one)
ip -n %i link set up lo
# save the list of network addresses that wg-quick assigned to the interface during creation
# and encode it in base64, because this is a binary stream in the ip utility format
IPS="$(ip address save dev %i | base64)"
# move the interface to our special namespace
ip link set %i netns %i
# restore the addresses back, decoding them from the variable created earlier
echo "$IPS" | base64 -d | ip -n %i address restore dev %i
# start the network interface in that namespace
ip -n %i link set up %i
# look for a route to 0.0.0.0/0 in our namespace's route list
if ! ip -n %i route show default | grep -qF default; then
 # add it if not found
 ip -n %i route add default dev %i
fi
# here was a script in base64, I decoded it directly for you in the text to make it clear what's there
# this script is decoded and the interface name is inserted into it using printf, which will always
# work with the script
cat <<EOT | xargs -0 -I{} printf {} %i > /usr/local/bin/wg-%i-exec
#!/usr/bin/env sh
set -e

# the interface name will land here in the variable
netns='%s'

# and I'll explain this mess separately
cat <<EOF | xargs -0 sudo -E nsenter "--net=/var/run/netns/$netns" unshare --mount sh -c
for conf in "$netns" "tun.$netns"; do
 if [ -f "/var/run/resolvconf/interfaces/\$conf" ]; then
 mount --bind "/var/run/resolvconf/interfaces/\$conf" /etc/resolv.conf
 fi
done
export DBUS_SESSION_BUS_ADDRESS=unix:path=/dev/null
sudo -E -u '#${SUDO_UID:-$(id -u)}' -g '#${SUDO_GID:-$(id -g)}' -- $@
EOF
EOT
# same kind of script, but we execute not a user command there, but wg-quick down
cat <<EOT | xargs -0 -I{} printf {} %i > /usr/local/bin/wg-%i-down
#!/usr/bin/env sh
set -e

# must be run as root
if [ "$(id -u)" -ne 0 ]; then
 echo "error: run me as root" 1>&2
 exit 1
fi

netns='%s'
# return the interface back to the root netns
ip netns exec "$netns" ip link set "$netns" netns 1
# delete our custom netns
ip netns delete "$netns"
# and stop via wg-quick, which will remove the interface itself
systemctl stop "wg-quick@$netns.service"
EOT
# change owner and permissions on files
chmod 0755 /usr/local/bin/wg-%i-*
chown root:root /usr/local/bin/wg-%i-*
# just output text to screen when bringing up the interface - user help,
# which tells about the two scripts created earlier. This text is decoded and directly
# thrown to sh, i.e., executed on the fly, and there's only one cat there
cat <<EOT | xargs -0 -I{} printf {} %i | sh -
netns='%s'
cat <<EOF
# ┐ ┬o┬─┐┬─┐┌─┐┬ ┐┬─┐┬─┐┬─┐ ┌┐┐┬─┐┌┐┐┌┐┐┐─┐
# │││││┬┘├─ │ ┬│ ││─┤│┬┘│ │ │││├─ │ │││└─┐
# └┴┘┘┘└┘┴─┘┘─┘┘─┘┘ ┘┘└┘┘─┘ ┘└┘┴─┘ ┘ ┘└┘──┘
Thanks for using netns wg-quick wrapper!

/usr/local/bin/wg-$netns-exec COMMAND - execute shell command in $netns network namespace
/usr/local/bin/wg-$netns-down - use it instead of wg-quick down $netns
EOF
EOT

And here’s the real madness. Let me explain the main points.

When you move a network interface to another namespace, unfortunately, all IP addresses are lost from it. Therefore, we save them at the beginning and then restore them after the transfer.
The loopback interface in the new namespace is disabled by default, so we enable it ourselves
A route is added only to 0.0.0.0/0 because in principle our traffic has nowhere to go except to wireguard, so we send everything there regardless of what’s written in AllowedIPs on the interface. This makes the script simpler, works the same way, and creates no vulnerabilities.
Two scripts are created: wg-%i-exec, wg-%i-down. The first is needed to run programs in our network namespace, and the second is needed to simply do wg-quick down, because just executing wg-quick down won’t stop the interface - you need to execute wg-quick inside our namespace

How wg-%i-exec works ¶

There are several problems that the script solves:

Inside the namespace there’s no resolv.conf, or there might not be. wg-quick always creates it for itself through resolvconf, even without all our tricks. That is, it’s created for the interface. This is needed for DNS name resolution.
We need to make it so that even when running exec as a user, they don’t get root privileges as a bonus, but remain with their own privileges.

I’ll break down the script line by line into possibly even non-working shell code, but it will be more convenient to comment this way.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


#!/usr/bin/env sh
set -e

# the interface name will land here in the variable
netns='%s'

# and I'll explain this mess separately
cat <<EOF | xargs -0 sudo -E nsenter "--net=/var/run/netns/$netns" unshare --mount sh -c
for conf in "$netns" "tun.$netns"; do
 if [ -f "/var/run/resolvconf/interfaces/\$conf" ]; then
 mount --bind "/var/run/resolvconf/interfaces/\$conf" /etc/resolv.conf
 fi
done
export DBUS_SESSION_BUS_ADDRESS=unix:path=/dev/null
sudo -E -u '#${SUDO_UID:-$(id -u)}' -g '#${SUDO_GID:-$(id -g)}' -- $@
EOF


# execute on behalf of sudo, preserving the current user's env, this will still be with root privileges
sudo -E
# nsenter will run us in this namespace
nsenter "--net=/var/run/netns/$NETNS"
# and then unmount everything that was mounted
unshare --mount
# shell command that will execute the script
sh -c
'
# where we mount resolv.conf created by wg-quick, if such exists at all
# on Ubuntu and Fedora this is needed, on Manjaro - /etc/resolv.conf is already in place
for conf in "$netns" "tun.$netns"; do
 if [ -f "/var/run/resolvconf/interfaces/\$conf" ]; then
 mount --bind "/var/run/resolvconf/interfaces/\$conf" /etc/resolv.conf
 fi
done
# browsers won't start without this
export DBUS_SESSION_BUS_ADDRESS=unix:path=/dev/null
# and using sudo preserving the same current user's env
sudo -E
# not on behalf of root, but already on behalf of the user who ran the exec script
-u '#${SUDO_UID:-$(id -u)}'
-g '#${SUDO_GID:-$(id -g)}'
# execute what they requested
-- $@
'

Conclusion ¶

Just insert these 3 lines into the config and that’s it! No Ansible, dancing with tambourines, complex setup and maintenance.

GPT (UEFI) for Btrfs (/ and /boot) over dm-crypt

Sun, 20 Mar 2022 00:00:00 +0000

Introduction ¶

The program in the article’s photo is GParted. I love and practice it. It’s convenient and simple to do whatever you want with partitions. I couldn’t decide what to use as the main image for the article. Let it be my current partitioning scheme 😄
I reinstall systems fairly often, a couple of times a year. The reasons vary: I built a small server, swapped my work PC for a laptop, or upgraded my parents’ computer with a larger drive… Just little things, really. I believe I have extensive experience with both Windows and various Linux distributions. This article is mostly just chit-chat, so please don’t take offense. If someone happens to discover that comments have been allowed on the site for half a year, I would be sincerely glad 😄

Partition Table Type ¶

I used to think: “Well, there’s the old good MBR, and GPT was invented with their cursed UEFIs to run on laptops and prevent people from installing other operating systems! The 4-partition limitation - so what? If you want more, use logical partitions.” I thought this way until January 2022, when I received a laptop for work, and of course, I started thinking about how to set it up. “I’ll try GPT; I’ve never used it before. Let’s see if it works, what’s the difference?” The result: GPT UEFI forever! Why…

There’s no limit on the number of partitions.
There’s no bootloader area at the beginning of the disk, meaning that the BIOS doesn’t load the disk by following a link at the start, but rather a specific program found on one of however many FAT32 partitions.
GRUB sees all the neighbors on the disk, and so does the BIOS, as it turns out that the entry point is exactly the same UEFI program. You can access the BIOS from the bootloader menu, then back to the bootloader, then again to the BIOS, you get the idea…
When you install Windows and it wipes the MBR without asking you (I won’t express my eloquent, uncensored opinion about Microsoft’s respect for products not developed by them), and then you need to restore GRUB from a LiveUSB, or maybe some system update failed. It doesn’t matter what happened. You just boot another OS straight from the BIOS! I really liked that.
A downside, but really just a minor one, is that UEFI programs can only be on FAT32 partitions. Well, that’s okay. I created it and forgot about it.
Windows wasn’t initially planned, but when it appeared, it turned out that it could be installed on any partitions in order, not just the first one as in the case of MBR, which is a huge plus.

File Systems ¶

I switched to BTRFS everywhere, on my work machine, server, and even the archive partition (auto, moto, photo …) is also on BTRFS. I really love and respect it; I tried it recently as well. I looked for various performance benchmarks and realized that on average, BTRFS is about the same as EXT4 but slower than XFS. “To take it or not to take it?” I wondered. So I took it for one simple and clear reason: my machines are not high-load servers, and I wouldn’t feel the performance difference 😄 I’m satisfied with my choice.

I have BTRFS RAID1 on two 1TB Barracuda HDDs on my home server - a software RAID, with no problems and at the first attempt.
Built-in features: compression, defragmentation, error checking (which can indicate when drives are dying), snapshots. It’s a dream!
It turns out GRUB loads BTRFS and boots from BTRFS without any issues!
It’s a pity there’s no built-in encryption. I had to layer dm-crypt on top - and no worries, everything is excellent.
BTRFS is like a logical partition with rubber partitions inside, which aren’t arranged sequentially as we are used to, but simply exist, all at once 😄 No need to add space if it runs out or think about what will consume how much.

Interestingly, I had to migrate the OS from one hardware to another. Here’s how it looked with BTRFS.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# mount the root btrfs on the old machine
mount /dev/sda1 -o subvol=/ /mnt
# do the same on the new machine
ssh user@new-pc mount /dev/sda1 -o subvol=/ /mnt
# create a readonly snapshot of the system partition
btrfs subvolume snapshot -r /mnt/@manjaro /mnt/@manjaro-ro
# send it via SSH! simply amazing...
btrfs send /mnt/@manjaro-ro | ssh user@new-pc btrfs receive /mnt/@manjaro-ro
# on the new machine, create a readwrite snapshot from the received readonly one
ssh user@new-pc btrfs subvolume snapshot /mnt/@manjaro-ro /mnt/@manjaro
# delete the readonly snapshot
ssh user@new-pc btrfs subvolume del /mnt/@manjaro-ro
# unmount everything
umount /mnt
ssh user@new-pc umount /mnt

I have never migrated so easily and simply. Just remember to do the following manually later:

Fix UUIDs in /etc/fstab
Update /etc/hostname
grub-install ... && grub-mkconfig ...

GPT Partitions ¶

If Windows needs more space, I’ll shrink Manjaro from the end.
If I need another Linux, it will also need an EFI partition, but the boot partition isn’t necessary! Because multiple Linux systems can boot from one BTRFS boot - just different (subvolume) partitions.

Operating Systems ¶

It’s hard to imagine anything better than Manjaro. I really like everything - documentation, utilities, ease of kernel update control with mhwd-kernel, compatibility with all Vanilla Arch documentation. Fresh software, but not as bleeding-edge as Fedora, which is also nice! There’s mari0
Windows 10, just for games and Steam.

Important Notes ¶

Windows does not ask anyone and formats partitions with the boot,esp flags in FAT32 and puts its EFI there. This was resolved by manually creating the EFI-Windows partition and setting these flags. Linux doesn’t care where they are - it doesn’t affect booting, but the Windows installation fit in the space I needed.
Initially, I didn’t separate boot, and it was also encrypted! It works perfectly! One downside: GRUB does not pull drivers and uses software-implemented algorithms for decryption, without leveraging CPU hardware capabilities. The initial decryption took me 5-7 seconds. People online complained about tens of seconds… For me, those 5-7 seconds were not critical, especially since it doesn’t affect work later, just at the beginning of the boot. However, I decided to remove that delay; I also don’t need paranoid security.
To restore the bootloader, you also need to mount efivars before chrooting.
Here’s excellent documentation on bootloader recovery.
It’s hard to find better documentation on system installation than this for an encrypted partition.
UUIDs in /etc/fstab must match those you need from the output of blkid.

Unimportant Notes ¶

This is what my boot screen looks like thanks to plymouth. Theme Pack 2 - Flame. (the image flickering only occurs in the browser; during boot, it’s smooth)
I chose a theme for Grub here. This is the one I took.
dm-crypt can be patched to create a nuke password. If entered, it destroys all keys, and the partition can no longer be decrypted. No matter who steals it, they will never be able to use it.

Grub recovery script ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# расшивровуем раздел с системой
cryptsetup luksOpen /dev/nvme0n1p1 cryptroot
# монтируем систему
mount /dev/mapper/cryptroot -o subvol=@root /mnt
# монтируем boot
mount /dev/nvme0n1p2 -o subvol=@boot /mnt/boot
# создаём папку efi если нет
mkdir -p /mnt/boot/efi
# монтируем efi в boot
mount /dev/nvme0n1p3 /mnt/boot/efi
# монтируем кучу всего для chroot
for i in dev proc sys ; do mount --bind /$i /mnt/$i; done
# монтируем efivars а то grub не встанет
mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
# запрыгиваем в нашу систему
chroot /mnt
# ставим grub. тут нужны пояснения
# 1. в случае с mbr мы привыкли указывать диск, куда прописать загрузчик
# в случае с efi он ничего никуда не прописывает, а просто наполняет /boot
# 2. bootloader-id это просто для красоты - именно так BIOS будет подписывать нашу ось в списке
# можно и не писать, там по умолчанию будет manjaro
grub-install --bootloader-id="Manjaro"
# если профукал весь раздел /boot вместе с образами ядра, то вот так они перегенерятся
pacman -S linux
# собираем initramfs образы
mkinitcpio -P
# собираем меню grub
grub-mkconfig -o /boot/grub/grub.cfg
# выходим из chroot
exit
# размонтируем и закрываем всё в обратном порядке
umount /mnt/sys/firmware/efi/efivars
for i in dev proc sys; do umount /mnt/$i; done
umount /mnt/boot/efi
umount /mnt/boot
umount /mnt
cryptsetup luksClose cryptroot

Simple Explanation of OS Booting (EFI) ¶

The BIOS starts up and finds all FAT32 partitions that contain an EFI folder and the familiar files. It loads one of the programs it finds, as requested.
The EFI program is not as limited as MBR; it can do pretty much anything. In our case, it accesses the /boot partition with GRUB, discovers BTRFS, and loads the GRUB menu. This is where the theme we chose for GRUB is displayed.
Now, we select what to boot from the GRUB menu.
GRUB loads the entire initramfs of the selected operating system into RAM. This is what we generated with the mkinitcpio command. Initcpio is a program similar to EFI, but when it was created, EFI didn’t exist yet; only the limited MBR was around. I have a feeling that you could boot the system directly from EFI without the extra layer, but that’s another story… Also, along with initramfs, the kernel is loaded here since it’s also stored on /boot.
Initcpio is a layered cake of hooks that are executed sequentially. Each one does something important and necessary. If we installed Plymouth, then right here, in one of the very first hooks, some beautiful graphics are drawn on the screen to hide a bunch of loading logs.
One of the hooks is encrypt or plymouth-encrypt. This particular piece of the initramfs cake asks us for the password for the encrypted root. Up to this point, it hasn’t been accessed at all 😄
Decryption, creating the first process, starting services, drivers, the graphical shell, and the whole process of booting up.

Conclusion ¶

I hope the article helped someone or provided some information. If you have suggestions or ideas, feel free to contact me!

Overview of k9s

Mon, 17 Jan 2022 00:00:00 +0000

Introduction ¶

k9s - my favorite tool for working with Kubernetes clusters. I really like this tool, there are quite good analogues and even much prettier ones, but exactly here, hotkeys work well and it’s very convenient to use from the terminal.

Everything you can read in the repository page is also indicated what’s fresh in memory, only that which I constantly use.

Overview ¶

Viewing logs ¶

Just by pressing L you can view the logs of a separate pod, there too you can quickly choose a range of time for which these logs should be shown.

Copying to clipboard buffer ¶

By pressing C in the mode of viewing logs or manifest object, the screen content (the entire file or log, not only what is visible) is copied into the clipboard buffer. You need to have xsel installed as well.

Port forwarding ¶

Pressing Shift + F allows you to choose which ports to forward to your local machine so that you can access any service.

Viewing problematic pods ¶

By pressing Ctrl + Z in the list of pods, only those remain that are not Running.

Searching by substring ¶

Just press the backslash and write a substring that needs to be searched in the name or tags of object objects from the list. It’s convenient when there are many objects.

Demonstration ¶

Notes on Terraform

Sun, 16 Jan 2022 00:00:00 +0000

Introduction ¶

Terraform is a fantastic tool for automating resource creation in clouds like AWS, Azure, GCP, Hetzner, and many others. It’s really cool. With tfenv, you can easily and conveniently update Terraform locally and choose the desired version. You can create a file in the project’s root to specify the required version, allowing tfenv to automatically use the correct Terraform version for your project.
There are several practical recommendations I’ve developed from reading the official documentation, which is excellent, as well as the modules and code written by AWS.

Recommendations ¶

Study the official recommendations. Below are my own notes.

1. Do not duplicate resource type in the name ¶

Incorrect

1
2
3
4
5
6
7


resource "aws_vpc" "production_vpc" {
 cidr_block = "172.16.0.0/16"

 tags = {
 Name = "Production VPC"
 }
}

Correct

1
2
3
4
5
6
7


resource "aws_vpc" "production" {
 cidr_block = "172.16.0.0/16"

 tags = {
 Name = "Production"
 }
}

Explanation

In the code, you’ll still reference the resource as ${aws_vpc.production.id}—it’s already clear that this is a VPC, so there’s no need to waste characters on what’s already written.
The same goes for the resource name; you’ll always see the VPC in the list on the VPC page in the AWS Console, and there won’t be anything else. Again, why duplicate?

2. Use `this` as a name, but only in modules ¶

Suppose you have a module that creates a virtual machine and minimal attachments. Within the module, the virtual machine is unique.

1
2
3
4
5


resource "aws_instance" "this" {
 ami = data.aws_ami.this.id
 instance_type = var.instance_type
...
}

Why come up with names for resources if it’s the only one within the module? There’s no need. It’s also important to understand that this is the object name within the Terraform code. It’s convenient when any resource type is simply this, but you should only do this within a single module that creates a logical unit. A virtual machine without its Security Group, which opens the ports, doesn’t make sense; thus, logically, it’s one piece.

3. Create standard files for modules ¶

File	Content
vars.tf	All variables
outputs.tf	Output parameters
versions.tf	Section with provider requirements
main.tf	File with the root content

4. Do not declare providers in modules ¶

Specify version constraints, but leave initialization to the discretion of the user of your module.

5. Tags variable ¶

Add a variable that will apply tags to all resources in the module.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


variable "tags" {
 type = map(string)
 description = "Extra tags."
 default = {}
}

locals {
 default_tags = {
 Custom = "Some default tags"
 Terraform = "Path to module folder in your Git repo - helps to find code in the repo fast"
 }
 tags = merge(
 local.default_tags,
 var.tags
 )
}

resource "aws_vpc" "production" {
 cidr_block = "172.16.0.0/16"

 tags = merge(
 local.tags,
 {
 Name = "Production"
 }
 )
}

In the local.tags variable, we first collect the final dictionary and then use it where needed within the module.

6. Resource naming with `name_prefix` ¶

Always, without exception, use a name prefix for creating resources. By changing just this variable, the code should create exactly the same infrastructure in the same account without any conflicts—this is a good module that can be reused.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


variable "name_prefix" {
 type = string
 description = "Name prefix for all resources (required)."
 validation {
 condition = can(regex("^[a-z][a-z0-9-]+[a-z]$", var.name_prefix))
 }
}

resource "tls_private_key" "this" {
 algorithm = "RSA"
 rsa_bits = 4096
}

resource "aws_key_pair" "this" {
 key_name = var.name_prefix
 public_key = tls_private_key.this.public_key_openssh
}

And there’s no point in adding anything to the name if there’s only one resource. Suppose this code is part of the same module for creating a virtual machine; it will be used like this.

1
2
3
4
5
6


module "ec2_backend" {
 source = "./ec2"

 name_prefix = "production-backend"
 instance_type = "t3a.small"
}

This way, it will be clear from the prefix to which virtual machine the EC2 Key Pair belongs in the list. If there were two, then the name in the code would be "${var.name_prefix}-some-suffix".

7. No hyphens in resource names ¶

Forget about using hyphens in resource names. It breaks the code analyzer in the IDE, and it causes issues in various situations.

Incorrect

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


module "ec2-backend" {
 source = "./ec2"

 name_prefix = "production-backend"
 instance_type = "t3a.small"
}

output "backend-ip" {
 value module.ec2-backend.private-ipv4
}

Correct

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


module "ec2_backend" {
 source = "./ec2"

 name_prefix = "production-backend"
 instance_type = "t3a.small"
}

output "backend_ip" {
 value module.ec2_backend.private_ipv4
}

I would like to emphasize that for the name of the resource in the cloud, I recommend using hyphens as much as possible—it looks nicer—but in Terraform, never.

Figlet ASCII Graphics

Fri, 07 Jan 2022 00:00:00 +0000

Bomberman from here.

Introduction ¶

1
2
3
4
5
6
7


# ┬ ┬┬─┐┬ ┬ ┌─┐ # ┳━┓┳━┓┏┓┓┳━┓┳━┓┳━┓┓━┓┳━┓┓━┓
# │─┤├─ │ │ │ │ # ┃ ┃┃━┫ ┃ ┃━┫┃━┃┃━┫┗━┓┣━ ┗━┓
# ┘ ┴┴─┘┘─┘┘─┘┘─┘ # ┇━┛┛ ┇ ┇ ┛ ┇┇━┛┛ ┇━━┛┻━┛━━┛

# ┳━┓┓ ┳┓━┓ ┳━┓┏━┓┏━┓ # ┌┐┐┬─┐┌┐┐┐ ┬┌─┐┬─┐┬┌
# ┃━┫┃┃┃┗━┓ ┣━ ┃ ┏━┛ # │││├─ │ ││││ ││┬┘├┴┐
# ┛ ┇┗┻┇━━┛ ┻━┛┗━┛┗━━ # ┘└┘┴─┘ ┘ └┴┘┘─┘┘└┘┘ ┘

I insert such captions in the source codes to visually separate chunks. In Terraform, for example.

Installation ¶

Install figlet in any way suitable for your distribution.
Download additional fonts.

1
2


curl -Lo- 'https://github.com/vlasov-y/figlet-fonts/archive/refs/tags/v1.1.tar.gz' | \
sudo tar -C "$(figlet -I2)" --strip-components 1 -xpzvf -

Viewing All Fonts ¶

1
2
3
4


find "$(figlet -I2)" -type f -name '*.*f' | while read -r i; do
 echo "> $i"
 figlet -f "$i" "$(basename "$i")"
done | less

Function for .bashrc ¶

Be careful here. In the overwhelming majority of fonts, there are characters that won’t display at all or will be shown incorrectly (as rectangles) for your colleagues who pulled the same code from Git. Through trial and error, I found a universal and beautiful font for myself - Rusto. There’s also RustoFat, which I actually like more, but it has its problems. 😢

1
2
3
4
5
6
7


# ┬─┐┬ ┐┐─┐┌┐┐┌─┐
# │┬┘│ │└─┐ │ │ │
# ┘└┘┘─┘──┘ ┘ ┘─┘

# ┳━┓┳ ┓┓━┓┏┓┓┏━┓┳━┓┳━┓┏┓┓
# ┃┳┛┃ ┃┗━┓ ┃ ┃ ┃┣━ ┃━┫ ┃ 
# ┇┗┛┇━┛━━┛ ┇ ┛━┛┇ ┛ ┇ ┇ 

The captions above were generated using my function, so your Rusto will differ slightly. Add this function to your .bashrc or .zshrc, and you can use it. Also, you’ll need to install xsel.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


rusto () {
 figlet -f rusto "$*" | sed "s@^@${P:-# }@g; s@┆@┘@g" | tee /tmp/tt | xsel -i -b
 cat /tmp/tt
 rm -f /tmp/tt
}

rustofat () {
 figlet -f rustofat "$*" | sed "s@^@${P:-# }@g" | tee /tmp/tt | xsel -i -b
 cat /tmp/tt
 rm -f /tmp/tt
}

What happens here? When you execute the command in the console rusto or rustofat, the text passed as arguments to the commands will be transformed, displayed on the screen, and immediately copied to the clipboard. Also, the symbol ┆ is replaced with ┘ to make Rusto completely problem-free. Additionally, # is prepended to the start of each line, allowing you to paste it directly into the source code. You can replace the hash by setting the variable P.

1
2
3
4
5
6
7
8
9


❯ rusto hello world
# ┬ ┬┬─┐┬ ┬ ┌─┐ ┐ ┬┌─┐┬─┐┬ ┬─┐
# │─┤├─ │ │ │ │ ││││ ││┬┘│ │ │
# ┘ ┴┴─┘┘─┘┘─┘┘─┘ └┴┘┘─┘┘└┘┘─┘┘─┘

❯ P='// ' rustofat changed comment char
// ┏━┓┳ ┳┳━┓┏┓┓┏━┓┳━┓┳━┓ ┏━┓┏━┓┏┏┓┏┏┓┳━┓┏┓┓┏┓┓ ┏━┓┳ ┳┳━┓┳━┓
// ┃ ┃━┫┃━┫┃┃┃┃ ┳┣━ ┃ ┃ ┃ ┃ ┃┃┃┃┃┃┃┣━ ┃┃┃ ┃ ┃ ┃━┫┃━┫┃┳┛
// ┗━┛┇ ┻┛ ┇┇┗┛┇━┛┻━┛┇━┛ ┗━┛┛━┛┛ ┇┛ ┇┻━┛┇┗┛ ┇ ┗━┛┇ ┻┛ ┇┇┗┛

Bonus ¶

I generate the daily report form like this.

1
2
3
4


LC_ALL=C.POSIX date '+%d %B' | \
figlet | \
xargs -0 printf '```\n%s```\nNo blocks\nNo questions\nDone\\Doing:\n' | \
xsel -ib

It produces something like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


```
 ___ __ _ 
 / _ \ / /_ | | __ _ _ __ _ _ __ _ _ __ _ _ 
| | | | '_ \ _ | |/ _` | '_ \| | | |/ _` | '__| | | |
| |_| | (_) | | |_| | (_| | | | | |_| | (_| | | | |_| |
 \___/ \___/ \___/ \__,_|_| |_|\__,_|\__,_|_| \__, |
 |___/ 
```
No blocks
No questions
Done\Doing:
- Drinking tea
- Sleeping
- Writing evening report
- Emulating work

Then I add what I did and send it to Slack.

Safe Update of Manjaro

Thu, 06 Jan 2022 00:00:00 +0000

History ¶

It somehow happened that I updated my system fully, and both of my machines - the work machine and the personal server - broke. The Python 3.10 library crashed with a segfault, which caused the Kodi media server to malfunction. The Python Pip package also failed. It was quite frustrating in general. Fortunately, I had a backup. I really love Manjaro - it’s the best operating system for professional use, and I never had any problems with updating except maybe New Year’s Eve 😄 This is my second time, and both times were on New Year’s Eve. December 31, 2020, I made a full update of the system, but when I woke up to the new year, I found that X11 would not start because the drivers for Nvidia had expired. And then I thought: “Well, I fixed it in 5 minutes, how about others? Are they DevOps or ordinary users?” It seems like there was a transfer of drivers to the distribution’s repository, a simple renaming, but that was enough. And again New Year’s Eve! I install an update and Python3 crashes at the level of .so libraries 😄 The system works overall, but it needs something in many places and some software can fail. Right away the ranger stopped working, so I had to reinstall it again. This is what happened, and here’s the script below.

Script ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


#!/usr/bin/env sh
## Making root BTRFS subvolume snapshot before making full Manjaro update

set -e

PARTITION="$(mount | awk '$3~/^\/$/{print $1}')"
SUBVOL="$(mount | awk '$3~/^\/$/{print $6}' | grep -oE 'subvol=[^,)]+' | cut -d= -f2- | sed -r 's/^\///g')"
MOUNTPOINT="/mnt"
BACKUP="$(date +%Y-%m-%d | xargs printf "backup/%s-%s" "$SUBVOL")"

main() {
 backup
 update
}

backup() {
 echo "Mounting $PARTITION to $MOUNTPOINT"
 mount "$PARTITION" "$MOUNTPOINT" -o subvol=/
 cd "$MOUNTPOINT"
 dirname "$BACKUP" | xargs mkdir -p
 if [ ! -d "$BACKUP" ]; then
 echo "Backing up $SUBVOL to $BACKUP"
 btrfs sub snap -r "$SUBVOL" "$BACKUP"
 else
 echo "$BACKUP exists already"
 fi
 cd /
 echo "Unmounting $MOUNTPOINT"
 umount "$MOUNTPOINT"
}

update() {
 echo "Running update"
 yay -Syu
}

main "$@"

Example Usage ¶

And here’s what happens if the backup is made today already.

PulseAudio over TCP

Wed, 17 Nov 2021 00:00:00 +0000

Introduction ¶

PulseAudio is a sound server, in my understanding. It runs on any Linux PC. Microphones, speakers, virtual devices - everything goes through PulseAudio and is controlled by it.

Task ¶

There’s a small server connected to a wall-mounted TV. I’m working on a PC with Manjaro and, to be honest, the headphones aren’t great. They’ll do for work, but I want to listen to music through the TV.

Sources ¶

Solution ¶

The traffic will be sent over TCP from the PC to the server (for simplicity, I’ll just refer to it as TV in the text, but it’s actually a server on Manjaro).

On both devices, we install the Zeroconf module so that the computers can find each other without issues.

1

yay -S pulseaudio-zeroconf

Now, let’s take the cookie file from one and copy it to the other so they are the same. I found it in my home directory on the PC.

1
2


scp ~/.config/pulse/cookie server:/etc/pulse/cookie
sudo cp -f ~/.config/pulse/cookie /etc/pulse/cookie

Now we configure:

The TV will be the PulseAudio server, as it receives the audio stream.
The PC will be the PulseAudio client, as it sends the stream.

On the TV, we run it as a regular user because there’s no need to start pulseaudio --kill/--start as root.

1
2
3
4
5
6
7
8
9


SUBNET='192.168.0.0/24'
sudo mkdir -p /etc/pulse/default.pa.d/
cat <<EOF | sudo tee /etc/pulse/default.pa.d/zeroconf.pa
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;${SUBNET}
load-module module-zeroconf-publish
EOF
sudo systemctl enable --now avahi-daemon.service
pulseaudio --kill
pulseaudio --start

On the PC, it’s almost the same.

1
2
3
4
5
6
7
8
9


SUBNET='192.168.0.0/24'
sudo mkdir -p /etc/pulse/default.pa.d/
cat <<EOF | sudo tee /etc/pulse/default.pa.d/zeroconf.pa
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;${SUBNET}
load-module module-zeroconf-discover
EOF
sudo systemctl enable --now avahi-daemon.service
pulseaudio --kill
pulseaudio --start

That’s it. Now in pavucontrol, you can direct some application through the tunnel to the PC!

Bonus that doesn’t work ¶

What would be nice next? Right, something that won’t be useful but is interesting. I want both the PC and the TV to play the same stream.

On the PC, let’s check what outputs we have and get their names.

1
2
3
4
5


# pacmd list-sinks | grep name:
name: <alsa_output.pci-0000_00_1b.0.analog-stereo>
name: <tunnel.aurelius.local.auto_null>
name: <tunnel.aurelius.local.auto_null.2>
name: <tunnel.aurelius.local.alsa_output.pci-0000_00_0e.0.hdmi-stereo>

Here we can see that the first one seems to be the headphones, while the others have come through the network. The 2nd and 3rd are likely the rear and front outputs on the case, while the last one is definitely the HDMI from the computer - which is what we need. So we need to combine the 1st and 4th.

Still on the PC:

1
2
3
4
5
6
7
8
9


SLAVE_1='alsa_output.pci-0000_00_1b.0.analog-stereo'
SLAVE_2='tunnel.aurelius.local.alsa_output.pci-0000_00_0e.0.hdmi-stereo'
DEFAULT_SLAVE="$SLAVE_1"
cat <<EOF | sudo tee -a /etc/pulse/default.pa.d/zeroconf.pa
load-module module-combine-sink sink_name=combined sink_properties="device.description='Combined TV and PC' slaves=${SLAVE_1},${SLAVE_2}"
set-default-sink ${DEFAULT_SLAVE}
EOF
pulseaudio --kill
pulseaudio --start

That’s it 😄 … Except it doesn’t work. I don’t know why. The device has been added, it can be selected, and there are no errors in the logs, but the sound only goes to the headphones and not to the TV. At the same time, when I simply select the TV, it works. So it seems this module cannot combine tunnel and simple devices. I decided to write this anyway; maybe it will be useful to someone.

Wireguard Ansible

Tue, 16 Nov 2021 00:00:00 +0000

GitHub

Update from November 25, 2022 ¶

I switched to classic wg-quick from this Ansible setup. The solution turned out to be technically much simpler, and it meets my functional needs.

Update from March 21, 2022 ¶

I removed automatic key generation due to excessive implementation complexity and the inability to configure servers individually; it could only be done collectively. Now it’s different. Configuration changes are incompatible with the old version; please check the examples in the repository, as I have added documentation there.

Introduction ¶

Wireguard is an amazing, simple VPN that is already included in the kernel. It’s a very powerful and straightforward tool. It installs much more easily than OpenVPN.

Problem Statement ¶

I live in Ukraine, and unfortunately, some internet resources are blocked, and access is sometimes necessary.
I want to be able to remotely connect from my phone or someone else’s computer to my home server.
I want traffic to be proxied on the PC only for domains on the list, not everything.

With these requests, I started working. For myself, all for myself. No one else needs it for now, although I posted it on GitHub. Posted is a strong word. No documentation, nothing! Yes! Just lazy to write… Oh well.

Solution Overview ¶

I used Ansible, which can configure the network immediately. What it can do:

Use a constant preshared key for all connections.
Ability to specify your AllowedIPs.
Ability to add peers with statically defined keys (since Ansible can’t be run on the phone).
Automatic selection of free rt_table IDs and fwmarks.
Ability to set up more than one network on the same machines with a single run!

What it can’t do:

Separate runs on one or a subgroup of hosts. Only all at once.

Understanding Ansible Variables ¶

all.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


# CIDR networks
wireguard_networks:
 home: 192.168.99.0/24
 vpn: 10.0.0.0/24
# marks for routing - each network has its own
wireguard_fwmarks:
 home: 1000
 vpn: 2000
# preshared key for the network
wireguard_preshared_keys:
 home: longalphanum
 vpn: longalphanum
# keys for each server in each network where it exists
wireguard_keys:
 home:
 public_host:
 public: longalphanum
 private: longalphanum
 home_server:
 public: longalphanum
 private: longalphanum
 laptop:
 public: longalphanum
 private: longalphanum
 vpn:
 public_host:
 public: longalphanum
 private: longalphanum
 laptop:
 public: longalphanum
 private: longalphanum
 smartphone:
 # only public key for the phone, as the private is not needed here
 public: longalphanum
wireguard_addresses:
 # addresses for each server in each network
 home:
 public_host: 192.168.99.1
 laptop: 192.168.99.2
 home_server: 192.168.99.3
 vpn:
 public_host: 10.0.0.1
 laptop: 10.0.0.2
 smartphone: 10.0.0.3

We have a server with a public IP, and it will serve as the connection point for everyone. The topology is a star.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# these are variables for a specific host - the server
wireguard_enabled: yes
# wireguard-tools have already been installed, so we turn this off
wireguard_install_tools: no
# list of peers
wireguard_peers:
 # for the home network
 home:
 # the first peer is our laptop, where we run Ansible
 - name: laptop
 # the second has inventory_hostname home_server
 - name: home_server
 # and keep the network so that it can always be reached,
 # because the server is behind NAT
 keepalive_seconds: 25
 # the second network will only be used from the phone and PC
 vpn:
 - name: laptop
 # and here is the phone
 - name: smartphone
# now we'll choose the ports it should listen on
# for the others, it doesn't matter, but
# for the server, it's better to specify everything statically
# so nothing is regenerated if anything happens
# and all clients don't need to be reconfigured
# with new server credentials
wireguard_ports:
 home: 2000
 vpn: 3000
# enable masquerading so that the server can
# release peer traffic to the Internet as a gateway
wireguard_masquarade:
 vpn: auto

Now, let’s specify variables for the laptop we are working on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# and again explicitly enable the role
wireguard_enabled: yes
wireguard_peers:
 vpn:
 # in the VPN network peers, only our public server
 - hostname: public_host
 # we have a domain pointing to the server so we don’t need to remember the address
 endpoint: vpn.example.com
 # routing all traffic through the VPN, and it's important to understand
 # that we are not sending all system traffic through Wireguard
 # we are telling Wireguard on the host to allow
 # everything that specifically comes to you on the output and does not match
 # in other networks by more narrow rules
 allowed_ips:
 - 0.0.0.0/0
 - ::0
 keepalive_seconds: 25
 home:
 # the contact point for the home network is the same
 - hostname: public_host
 endpoint: vpn.example.com
 keepalive_seconds: 25
 allowed_ips:
 # and in the paths, we specify only the subnet of the home network
 - "{{ wireguard_networks.home }}"
# and enable proxying specific domains through the VPN
wireguard_proxy_domains:
 vpn:
 list:
 - site.we.want.access
 - another.one

And the home server just needs to be accessible. It doesn’t need to be in the VPN.

1
2
3
4
5
6
7
8
9


# here I hope everything is already clear
wireguard_enabled: yes
wireguard_peers:
 home:
 - hostname: public_host
 endpoint: vpn.example.com
 allowed_ips:
 - "{{ wireguard_networks.home }}"
 keepalive_seconds: 25

Conclusion ¶

From any peer, there is access to any within the network. Traffic finds its way through the public server. Everything works. I measured with iperf; out of 80 Mbps, Wireguard delivers ~60 - to me, that’s an excellent result!

Age Encryption

Sun, 12 Sep 2021 00:00:00 +0000

Introduction ¶

Age – a new utility, written in Go, that supposedly performs excellent asymmetric encryption and more. Indeed, it looks simple and somewhat reminds me of Wireguard (utility wg). It encrypts the input stream to the output stream. Simple and understandable. Among its distinguishing features is the ability to encrypt for a group of recipients, so each can decrypt. Today’s task was to create backups btrfs disk partitions with home photos/video somewhere in the cloud, e.g., Backblaze, to save money. This is just not good enough (no trust in clouds). Data needs to be encrypted.

A 300 GiB partition and loading it in one piece, then storing it as a whole is very uninteresting. Split will help divide the file into chunks.

Generating Key ¶

It’s claimed that you can use a simple SSH public key instead of Age keys. Let’s try both ways.

1
2
3
4


ssh-keygen -t ed25519 -o -a 150 -qN '' -f id_ed25519
age-keygen -o age.key
age-keygen -y age.key > recipients.txt
cat id_ed25519.pub >> recipients.txt

We get such keys.

1
2


age1qrcn5glms66thfv9cj5pzdwl3z87t9evzaqlxdj9ksravmw3mypswawuag
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIObk2OuhYlrr3ezvf5gAWXgUdcLn4XQFa8B8GCRpBr9q user@host

Encryption ¶

For simplicity, let’s generate 10 MiB of random data and immediately compare the hash sums.

1
2


dd if=/dev/urandom of=data bs=1M count=10
cp data data.orig

Now we can encrypt. Using both archiving and splitting.

1

tar -cz data | age -R recipients.txt | split -b 1MiB -d - chunk-

We get 10 files from chunk-00 to chunk-10.

Decryption ¶

Decrypting using SSH key.

1
2


cat chunk-* | age -d -i id_ed25519 | tar xz
mv data data.ssh

Decrypting using Age key.

1
2


cat chunk-* | age -d -i age.key | tar xz
mv data data.age

Verification ¶

Checking the hash sums.

1
2
3
4


# sha256sum data*
5ebc271a22cfb9af0430253c2d8f3b83d1c3f5fdd062dd0ecc6c1f2d7925b828 data.age
5ebc271a22cfb9af0430253c2d8f3b83d1c3f5fdd062dd0ecc6c1f2d7925b828 data.orig
5ebc271a22cfb9af0430253c2d8f3b83d1c3f5fdd062dd0ecc6c1f2d7925b828 data.ssh

It works. As always, later we’ll find vulnerabilities, but you can use it.

Installing Python Packages Offline

Sun, 29 Aug 2021 00:00:00 +0000

Suppose we want to install Ansible offline.

Creating a Virtualenv ¶

We will create a virtual environment to download all dependencies without accidentally including unnecessary ones. First, let’s install virtualenv.

1
2


python3 -m pip install --upgrade --user virtualenv
python3 -m virtualenv venv

Now we can activate it.

1

. ./venv/bin/activate

Now, for the duration of the terminal session, we are in an isolated Python virtual environment.

Preparing Required Packages ¶

By installing only what is needed in a clean environment, we will also pull in all dependencies. Additionally, make sure to include setuptools and pip.

1

python3 -m pip install --upgrade pip setuptools ansible

Now we can create a list of what we have.

1

python3 -m pip freeze > requirements.txt

You can simply add pip and setuptools to requirements.txt without specifying versions (if they aren’t already in the list).

Example requirements.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


ansible==3.4.0
ansible-base==2.10.11
cryptography==2.8
jinja2==2.11.3
netaddr==0.7.19
pbr==5.4.4
jmespath==0.9.5
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.2
MarkupSafe==1.1.1
pip
setuptools

Having the list of packages with dependencies, we can download them even without venv.

1

python3 -m pip download --dest my-pip-packages -r requirements.txt

For convenience, let’s pack everything into an archive.

1

tar -cpvzf pips.tgz my-pip-packages requirements.txt

Installing in an Offline Environment ¶

Python should already be installed in advance.

Unpack your archive and install everything you brought. Pip and setuptools should be installed separately at the beginning, followed by everything else. If the command is not run as a superuser, you need to either:

Add sudo at the beginning
Add the --user argument to pip to install packages only for the user running the command

1
2
3


tar -xpzvf pips.tgz
python3 -m pip install --upgrade --no-index --find-links python-packages pip setuptools
python3 -m pip install --upgrade --no-index --find-links my-pip-packages -r requirements.txt

Now you can run Ansible!

1
2
3
4
5
6
7


root@289f34a67cbd:/host# ansible --version
ansible 2.10.11
 config file = None
 configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
 ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
 executable location = /usr/local/bin/ansible
 python version = 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]

Locales ¶

There have been cases where in a severely stripped-down environment, like Docker Ubuntu 18.04, the package installation fails with the following error.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


ERROR: Exception:
Traceback (most recent call last):
 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 173, in _main
 status = self.run(options, args)
 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
 return func(self, options, args)
 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 316, in run
 reqs, check_supported_wheels=not options.target_dir
...
 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 249, in unpack_url
 unpack_file(file.path, location, file.content_type)
 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/unpacking.py", line 256, in unpack_file
 untar_file(filename, location)
 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/unpacking.py", line 226, in untar_file
 with open(path, "wb") as destfp:
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 117: ordinal not in range(128)

This can be resolved by setting environment variables before installing packages.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


export LANGUAGE=""
export LANG="C.UTF-8"
export LC_CTYPE="C.UTF-8"
export LC_NUMERIC="C.UTF-8"
export LC_TIME="C.UTF-8"
export LC_COLLATE="C.UTF-8"
export LC_MONETARY="C.UTF-8"
export LC_MESSAGES="C.UTF-8"
export LC_PAPER="C.UTF-8"
export LC_NAME="C.UTF-8"
export LC_ADDRESS="C.UTF-8"
export LC_TELEPHONE="C.UTF-8"
export LC_MEASUREMENT="C.UTF-8"
export LC_IDENTIFICATION="C.UTF-8"
export LC_ALL="C.UTF-8"

Creating a Virtual Camera for Conferences

Thu, 22 Jul 2021 00:00:00 +0000

We don’t always want to include a camera during the conference, so we found a way to display video or static images instead. The means of OS or browser doesn’t provide this out of the box. I found this article and, indeed, created a virtual camera module that is recognized by browsers as an ordinary camera 😄

Below is a script with detailed comments. The only argument to it is the path to the file or image that should be transmitted in the stream.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


#!/usr/bin/env sh
set -e

# label for the new webcam as it will be seen in browser
LABEL="VirtualCam #0"

# module insertion in case if it was not inserted during this boot
if ! lsmod | grep -qE '^v4l2loopback'; then
 # camera is being created during the module loading
 sudo modprobe v4l2loopback devices=1 exclusive_caps=1 max_buffers=2 card_label="$LABEL"
 # wait for a while to finish module initialization
 sleep 1
fi

# path to media file
F="${1:?File is not passed}"
# getting path to newly created camera device
DEV="$(v4l2-ctl --list-devices 2>/dev/null | grep -A1 "$LABEL" | grep -oE '/dev/video[0-9]+')"

if [ ! -f "$F" ]; then
 # check in case if media file path does not exists
 echo "error: file $F does not exists" >&2
 exit 1
fi

# get file type and use proper streaming command
case "$(file --mime-type --brief "$F" | cut -d/ -f1)" in
 video)
 # in case if file is video
 ffmpeg -re -i "$F" -f v4l2 -vcodec rawvideo -pix_fmt yuv420p "$DEV"
 ;;
 image)
 # in case if file is picture
 ffmpeg -loop 1 -re -i "$F" -f v4l2 -vcodec rawvideo -pix_fmt yuv420p "$DEV"
 ;;
 *)
 # in case if file is something else
 echo 'error: unknown mime type' >&2
 exit 2
 ;;
esac

So, in the end, we were able to stream Rick Ashley - Never Gonna Give You Up.

One notable drawback is that FFmpeg consumes almost an entire core. Additionally, we are, of course, only transmitting the image, without any sound.

CEPH Upgrade

Mon, 16 Nov 2020 00:00:00 +0000

Introduction ¶

We have the following cluster:

Ceph 15.2.1 installed with cephadm
14 OSD
3 MON
3 RGW
3 MDS
3 CRUSH

Version 15.2.4 has been released, and we want to upgrade.

Getting Started ¶

Cephadm provides a mechanism for upgrading, but it didn’t work for me right away, so I had to assist it. Let’s see what exactly needs to be upgraded.

1

ceph orch upgrade check ceph/ceph 15.2.4

A JSON file appears with a list of daemons that have already been upgraded and those that differ from the requested version, i.e., all of them at the moment. Time to get started.
So, the procedure has been launched, and now there are some commands worth running to get a summary.

ceph status - for general information about the cluster’s state. You can run it in a separate terminal like this: watch -n 10 ceph status
ceph status -W cephadm - definitely run this command in the terminal to monitor the orchestrator’s log.

Temporary Difficulties ¶

We call ceph orch ps and see that all monitors and managers are upgraded, good, but now we’re in HEALTH_ERROR with the error shown below.

1

module 'cephadm' has failed: auth get failed: failed to find client.crash.01 in keyring retval: -2

It turns out that some keys are missing. You need to create one for each crash and restart the orchestration.

1
2
3
4
5
6
7


for i in $(ceph orch ps | grep -oE 'crash.[0-9]+'); do
 ceph auth get-or-create "client.$i" mgr "profile crash" mon "profile crash"
done
ceph orch upgrade stop
ceph mgr module disable cephadm
ceph mgr module enable cephadm
ceph orch upgrade start ceph/ceph 15.2.4

Now we see that the crashes are ready too, good, but then everything seems to be stuck. Yes, HEALTH_OK, but ceph orch ps shows that all OSDs are still on the old version. The logs show the following.

1

Upgrade: It is NOT safe to stop osd.0

Aha, the orchestrator is afraid to stop OSD daemons with data. Let’s help. Here is my situation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 14.00000 root default
-3 5.00000 host 01
 0 hdd 1.00000 osd.0 up 1.00000 1.00000
 1 hdd 1.00000 osd.1 up 1.00000 1.00000
 2 hdd 1.00000 osd.2 up 1.00000 1.00000
 3 hdd 1.00000 osd.3 up 1.00000 1.00000
 4 hdd 1.00000 osd.4 up 1.00000 1.00000
-5 5.00000 host 02
 5 hdd 1.00000 osd.5 up 1.00000 1.00000
 6 hdd 1.00000 osd.6 up 1.00000 1.00000
 7 hdd 1.00000 osd.7 up 1.00000 1.00000
 8 hdd 1.00000 osd.8 up 1.00000 1.00000
 9 hdd 1.00000 osd.9 up 1.00000 1.00000
-7 4.00000 host 03
10 hdd 1.00000 osd.10 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000

At first, I stopped OSDs one by one.

1

ceph osd out osd.0

Then I relaxed and began to decommission a whole server sequentially. When we decommission osd.0, after some time, when rebalancing finishes, ceph status -W cephadm complains about osd.1, and ceph orch ps shows that osd.0 is upgraded. The idea is clear; just don’t forget to reintegrate it.

1

ceph osd in osd.0

In short, we decommission the OSDs of the first server, wait until cephadm complains about the next ones, and then reintegrate them. Further, wait for HEALTH_OK before decommissioning the next server’s OSD, or you could lose data…
Based on your OSD tree, the plan, if done from scratch, is as follows:

out 0, 1, 2, 3, 4
wait for complaints about 5
in 0, 1, 2, 3, 4
wait for HEALTH_OK
out 5, 6, 7, 8, 9
wait for complaints about 10
in 5, 6, 7, 8, 9
wait for HEALTH_OK
out 10, 11, 12, 13
check that everything is upgraded with ceph orch ps
in 10, 11, 12, 13
wait for HEALTH_OK

```plain ╔═╗╔═╗╔═╗╦ ╦ ╦ ╦╔═╗╔═╗╦═╗╔═╗╔╦╗╔═╗╔╦╗┬ ║ ║╣ ╠═╝╠═╣ ║ ║╠═╝║ ╦╠╦╝╠═╣ ║║║╣ ║║│ ╚═╝╚═╝╩ ╩ ╩ ╚═╝╩ ╚═╝╩╚═╩ ╩═╩╝╚═╝═╩╝o ```

Notes ¶

The cluster is half-empty, so it was quick. If it were otherwise, it would have taken days…

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


% sudo ceph osd df [22]
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
 0 hdd 1.00000 1.00000 5.5 TiB 1.4 GiB 438 MiB 39 KiB 1024 MiB 5.5 TiB 0.03 1.11 37 up
 1 hdd 1.00000 1.00000 5.5 TiB 1.4 GiB 454 MiB 2.2 MiB 1022 MiB 5.5 TiB 0.03 1.12 46 up
 2 hdd 1.00000 1.00000 5.5 TiB 1.3 GiB 314 MiB 1.0 MiB 1023 MiB 5.5 TiB 0.02 1.01 24 up
 3 hdd 1.00000 1.00000 5.5 TiB 1.2 GiB 161 MiB 2.5 MiB 1022 MiB 5.5 TiB 0.02 0.90 34 up
 4 hdd 1.00000 1.00000 5.5 TiB 1.2 GiB 242 MiB 21 KiB 1024 MiB 5.5 TiB 0.02 0.96 34 up
 5 hdd 1.00000 1.00000 5.5 TiB 1.2 GiB 212 MiB 3.0 MiB 1021 MiB 5.5 TiB 0.02 0.93 34 up
 6 hdd 1.00000 1.00000 5.5 TiB 1.5 GiB 485 MiB 533 KiB 1023 MiB 5.5 TiB 0.03 1.14 35 up
 7 hdd 1.00000 1.00000 5.5 TiB 1.3 GiB 280 MiB 1.8 MiB 1022 MiB 5.5 TiB 0.02 0.99 41 up
 8 hdd 1.00000 1.00000 5.5 TiB 1.3 GiB 265 MiB 19 MiB 1005 MiB 5.5 TiB 0.02 0.97 38 up
 9 hdd 1.00000 1.00000 5.5 TiB 1.4 GiB 359 MiB 949 KiB 1023 MiB 5.5 TiB 0.02 1.05 28 up
10 hdd 1.00000 1.00000 5.5 TiB 1.2 GiB 179 MiB 467 KiB 1024 MiB 5.5 TiB 0.02 0.91 33 up
11 hdd 1.00000 1.00000 5.5 TiB 1.1 GiB 132 MiB 2.8 MiB 1021 MiB 5.5 TiB 0.02 0.87 23 up
12 hdd 1.00000 1.00000 5.5 TiB 1.3 GiB 358 MiB 19 MiB 1005 MiB 5.5 TiB 0.02 1.04 36 up
13 hdd 1.00000 1.00000 5.5 TiB 1.3 GiB 301 MiB 734 KiB 1023 MiB 5.5 TiB 0.02 1.00 39 up
 TOTAL 76 TiB 18 GiB 4.1 GiB 54 MiB 14 GiB 76 TiB 0.02
MIN/MAX VAR: 0.87/1.14 STDDEV: 0.00

Introduction to Docker

Fri, 21 Aug 2020 00:00:00 +0000

Written in collaboration with ReBrainMe

Docker is the de facto standard containerization engine, widely used for running applications in both development environments and high-load production environments. You can learn more about terminology here.

The purpose of this article is to install and run Docker locally, start getting acquainted with this fascinating system, and grasp the basics.

Installing Docker ¶

Docker has official instructions for installation. In practice, I used the installation script (for Linux):

1

curl -LO get.docker.com | sudo sh -

If the curl utility is not installed, download the script and run it manually:

1

sh /path/to/downloaded/script.sh

If you want to install a specific version of Docker, you can use the installation scripts from Rancher, which you can find and download here. The script is run in the same way, you can change the URL in the curl command, and it will work.

Configuring Docker ¶

It works out of the box, but there are a couple of things I would recommend adjusting right away based on personal experience.

Configure the logging driver ¶

By default, the json-file driver is used, which does not limit the size of the JSON file where logs for the corresponding container are stored. You can adjust this on Linux by editing the /etc/docker/daemon.json file. If the file does not exist, create it. The content to add:

1
2
3
4
5
6
7


{
 "log-driver": "json-file",
 "log-opts": {
 "max-size": "100m",
 "max-file": "1"
 }
}

This configures the json-file driver to not create more than one log file per container, with the file size not exceeding 100 MB. This can be overridden when starting the container, but having these defaults saves you from wondering, “Where did 6 GB go in one day?”

Start the Docker service ¶

Sometimes it does not start automatically after installation. You can fix this on Linux with:

1

sudo systemctl enable --now docker.service

Here, we used systemctl to start the Docker Engine and enable it to start on boot.

Add the user to the Docker group ¶

To be able to interact with the Docker Engine as a non-privileged user (without sudo), you need to add the user to the Docker group.

1

sudo usermod -aG docker "$USER"

For the changes to take effect, you need to reboot or use ACL.

1

sudo setfacl -m "u:${USER}:rwx" /var/run/docker.sock

Getting acquainted with DockerHub ¶

Containers are created from images. An image is an ordered set of filesystem layers that were created during the build process. People share images by uploading them to a Docker Registry. The most popular one is DockerHub. Many software products are published here and supported officially, and there are many tools created by individuals or small communities that are as good as the official ones. It is important to check what you are downloading and running – whether it’s an unknown source with no documentation or an official build that is regularly updated and comes with clear documentation. There have been rumors of miners in unknown images… but that’s more of a rarity than a rule (in my experience).

Here are some links to official images: HaProxy, Nginx, PostgreSQL, MySQL, Grafana, Prometheus.

Running the first container ¶

1

sudo docker run --rm -it -p 8080:80 nginx:stable-alpine

Opening http://localhost:8080 shows:

What did we do? Let’s break down the command step by step.

sudo – run the following command as another user, defaulting to the superuser root.
docker – run the Docker executable, found in one of the paths specified in the PATH environment variable.
run – create and start a new container.
--rm – remove it after it finishes running, so it doesn’t take up space.
-i – attach stdin/out/err from the shell to the container; run in interactive mode.
-t – create a TTY for the container.
-p 8080:80 – forward port 8080 from the host to port 80 inside the container.
nginx:stable-alpine – use the nginx image with the tag stable-alpine to create the container. Other tags can be found on the Nginx DockerHub page.

Mounting ¶

It’s often necessary to attach some data to the container or take it out so that it’s not lost when the container is recreated. More details can be found here.

1
2


echo "Hello world! Date: $(date)" > /tmp/my_file.txt
sudo docker run --rm -it -p 8080:80 -v /tmp/my_file.txt:/usr/share/nginx/html/index.html:ro nginx:stable-alpine

Here we created the file /tmp/my_file.txt and mounted it to /usr/share/nginx/html/index.html. According to the container description, and by empirical means, we can determine that the web server’s file path in the default configuration file is /usr/share/nginx/html, and the default file to open as the main page is index.html. Thus, we mount from the host to the container at the found path.

When mounting, be careful. If mounting a file, it must already exist on the host at startup, otherwise, a directory will be created. When mounting, the path on the host must be absolute. When mounting a directory, the contents of the target path inside the container will disappear and be replaced by the contents of the host folder.

Running in background mode ¶

1

sudo docker run -d --name my1 -p 3000:3000 grafana/grafana

Here, a couple of new flags appeared. -d – run in the background. --name my1 – set the container name to “my1”, as a random one is generated by default, but we want to assign our own. Check the container status.

1
2


sudo docker ps
sudo docker stats

The first command displays information about all running containers, and the second provides a brief summary of resource consumption. Open Grafana.

We can log in, explore (login: admin, password: admin), and then stop the container.

1
2


sudo docker stop my1
sudo docker ps -a

As you can see, the container is stopped but not deleted. We can start it again or remove it.

1
2


sudo docker start my1
sudo docker rm -f my1

Conclusion ¶

We installed Docker, got acquainted with DockerHub, launched a couple of containers, and tried basic commands for running applications with Docker Community Edition.

Plans

Wed, 22 Jul 2020 00:00:00 +0000

Articles to Write ¶

Reverse SSH Tunneling. There was an article that got lost. How to forward a port from a remote host using SSH.
Migrating from Ghost to Hugo. Why I decided to switch to Hugo and how it went.
The k9s Utility. A great tool that I use daily for working with Kubernetes.
Terraform Tips and Tricks. Examples, tricks, and useful links for working with Terraform.
Boltology: The 20\80 Rule in IT. Who knows the 80/20 rule? It’s true that 80% of efforts give 80% results, but you need to put 100% effort into the remaining 20%.
Wireguard Ansible. An article about my development - Ansible playbooks for setting up and configuring Wireguard.
Healthcheck.io (https://healthcheck.io). A great service that can be sure of successful backups.
How to search and work with Helm charts.
My experience with ArgoCD. (Stable, automated, flexible, but not without flaws. I use it on a large production with clean hands. Recommend)

Resume

Wed, 22 Jul 2020 00:00:00 +0000

xFreeRDP

Tue, 07 Jul 2020 00:00:00 +0000

xFreeRDP – a tool for connecting to RDP from Linux. After moving to Manjaro I was unable to install Remmina, so I had to look for an alternative. This option pleased me. You can create a one-line script for hosts that you often connect to and use like a catalog.

1
2


#!/usr/bin/env sh
xfreerdp /v:1.2.3.4 /u:username /p:password /w:1360 /h:730 /drive:tmp,/tmp

Here I explicitly set the display size for connection and mount /tmp from local machine to remote one, so you can get some files.

About Me

Wed, 01 Jul 2020 00:00:00 +0000

About Me ¶

I work as a system engineer (DevOps), and I am studying and trying to do my best. In the process of working, I understood how much everything is lacking in terms of understanding systems at various stages of the life cycle of software development.
I studied at KhAI as a computer engineer and received a master’s degree in May 2020. You should take what is written on this site critically - do not believe everything you hear. I am sometimes uneducated in many questions, but in any case, I try to write about what I am sure of. Please, keep in mind, that english version of the site is autotranslated with AI’s, so errors are possible 😄

My Goals ¶

The list contains the goals that I pursue while keeping this blog afloat. They are ordered by priority according to importance, in descending order:

Do not write nonsense.
Write and maintain a set of materials that cover at least the minimum knowledge required for every IT professional working in the industry.
Publish articles based on interesting and unusual solutions. These solutions should be interesting but should not standardize them as necessary to study.
Support feedback and adjust the course of this blog.
Use this resource as a “CV” when transitioning to any other company.

Search

Wed, 01 Jul 2020 00:00:00 +0000

Rsnapshot

Tue, 12 May 2020 00:00:00 +0000

Introduction ¶

Rsnapshot is a great utility that can:

Backup folders in the local system and store them in a backup storage (a designated folder somewhere nearby).
Run scripts that create backups, store them locally, and rsnapshot takes these files and puts them in the same backup folder.
Send backups via sftp somewhere.
Store only diffs between backups, saving space.

Retention ¶

Rsnapshot has Retention Policies (I might not be calling it exactly right), which in the configuration start with retain. These are descriptions of abstract plans about how many recent backup copies to keep. I struggled for a while until I realized that their names don’t affect anything. So if a plan is called daily, rsnapshot won’t necessarily run it every day—you can run it monthly in cron if you want. I find it convenient to use names like daily, weekly, and monthly for configuration (for easier reading and maintenance).

Retention policies written lower in the config are considered to be executed less frequently than those higher up, so the number for those should be smaller. Rsnapshot uses the content from the higher retention i+1 to create the next lower retention i. Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


+ date
Tue May 12 05:58:33 UTC 2020
+ rsnapshot daily
+ ls -la /data/backups
total 48
drwxrwxr-x. 11 y.vlasov y.vlasov 4096 May 12 05:57 .
drwxrwxr-x+ 6 root root 4096 Apr 27 09:43 ..
drwxr-xr-x. 6 root root 4096 May 12 05:57 daily.0
drwxr-xr-x. 6 root root 4096 May 12 00:00 daily.1
drwxr-xr-x. 6 root root 4096 May 11 00:00 daily.2
drwxr-xr-x. 6 root root 4096 May 10 00:00 daily.3
drwxr-xr-x. 6 root root 4096 May 9 00:00 daily.4
drwxr-xr-x. 6 root root 4096 May 8 00:00 daily.5
drwxr-xr-x. 6 root root 4096 May 7 00:00 daily.6
drwxr-xr-x. 6 root root 4096 May 4 00:00 weekly.0
drwxr-xr-x. 6 root root 4096 May 2 00:00 weekly.1
+ rsnapshot weekly
+ ls -la /data/backups
total 48
drwxrwxr-x. 11 y.vlasov y.vlasov 4096 May 12 05:57 .
drwxrwxr-x+ 6 root root 4096 Apr 27 09:43 ..
drwxr-xr-x. 6 root root 4096 May 12 05:57 daily.0
drwxr-xr-x. 6 root root 4096 May 12 00:00 daily.1
drwxr-xr-x. 6 root root 4096 May 11 00:00 daily.2
drwxr-xr-x. 6 root root 4096 May 10 00:00 daily.3
drwxr-xr-x. 6 root root 4096 May 9 00:00 daily.4
drwxr-xr-x. 6 root root 4096 May 8 00:00 daily.5
drwxr-xr-x. 6 root root 4096 May 7 00:00 weekly.0
drwxr-xr-x. 6 root root 4096 May 4 00:00 weekly.1
drwxr-xr-x. 6 root root 4096 May 2 00:00 weekly.2

Backup, backup_script ¶

So how does it perform a backup? Rsnapshot backs up all the steps listed in the configuration each time it is called.

backup - you specify a folder path to be backed up, and it copies it completely to the storage.
backup_script - you specify the path to a shell script that performs the backup itself, and places the resulting files/folders nearby, then rsnapshot moves them to the storage.
other options - man rsnapshot.

Plan ¶

Create a folder at the snapshot_root path.
Write the rsnapshot.conf file (by default located at /etc/rsnapshot.conf).
Add rsnapshot calls to the scheduler - cron.

Example ¶

Let’s create the snapshot_root folder: mkdir -p /data/backups.

Contents of /etc/rsnapshot.conf ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


## Note 1: You must use tabs to separate arguments on a line
config_version 1.2
snapshot_root /data/backups
no_create_root 1
## Note 2: Paths may vary between OSes
cmd_rm /usr/bin/rm
cmd_rsync /usr/bin/rsync
cmd_logger /usr/bin/logger
## Retention configuration
retain daily 7
retain weekly 4
## Logging settings
verbose 2
loglevel 3
lockfile /var/run/rsnapshot.pid

## Describing the actions
backup /data/ghost ghost/
backup_script /srv/my_soft/backup.sh my_soft/

Contents of crontab ¶

1
2


0 0 * * * /usr/bin/rsnapshot daily 2>&1 | systemd-cat -t rsnapshot-daily
10 0 * * 0 /usr/bin/rsnapshot weekly 2>&1 | systemd-cat -t rsnapshot-weekly

Contents of /srv/my_soft/backup.sh ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


#!/usr/bin/env sh
set -e

CMD='''
rm -rf backup; \
pg_basebackup -U my_user -X stream -F plain -D backup 1>/dev/null 2>&1 && \
tar -C backup -czf - $(ls backup)
'''

docker exec \
 -w /tmp \
 my_soft.postgres \
 /bin/sh -c "$CMD 1>pg_basebackup.tgz

Contents of daily.0 ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


/data/backups/daily.0
├── ghost
│   └── data
├── my_soft
│   └── pg_basebackup.tgz
├── proxy
│   ├── certificates.tgz
│   └── haproxy.tgz
└── another_one
 └── another_backup_from_backup_script.tgz

5 directories, 4 files

RPM2PKG

Mon, 27 Apr 2020 00:00:00 +0000

It turned out that Skype and Slack can be installed from AUR with one command
1
yay -S slack-desktop skype
I left the article just for history

I switched to Manjaro and encountered the fact that Skype and Viber are packaged in RPM and DEB. What to do with Arch? On GitLab I placed a simple script that converts rpm to pkg.

Using the Script ¶

Install dependencies

1
2


sudo pacman -S rpm rpm2cpio cpio git
python3 -m pip install --upgrade --user pip pyyaml jinja2

Clone repository

1
2


git clone --depth 1 --single-branch git@gitlab.com:vlasov-y/rpm2pkg.git
cd rpm2pkg

Convert

1
2
3
4


curl -fLO https://repo.skype.com/latest/skypeforlinux-64.rpm
python3 rpm2pkg skypeforlinux-64.rpm
# or add --install to install package automatically (interactive mode, will prompt you for confirmation)
# python3 rpm2pkg --install skypeforlinux-64.rpm

Install

1

sudo pacman -U skype*.pkg.tar.gz

Remove

1

sudo pacman -R skype

Nextcloud

Sat, 25 Apr 2020 00:00:00 +0000

Launching the Stack ¶

Nextcloud - personal cloud with gallery, email checking, online document editing, audio and radio listening, etc. A demo can be seen here. A clone of OwnCloud, I tried initially and didn’t find any issues, but there are slightly more applications than what I saw.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


version: '3.7'

services:
 nextcloud:
 image: nextcloud:18.0.3-apache
 container_name: nextcloud
 restart: unless-stopped
 depends_on:
 - postgres
 ports:
 - '8080:80'
 volumes:
 - ./nextcloud:/var/www/html:rw
 logging:
 driver: json-file
 options:
 max-size: 100m

 postgres:
 image: postgres:11.7-alpine
 container_name: postgres
 restart: unless-stopped
 networks:
 - nextcloud
 volumes:
 - ./postgres:/var/lib/postgresql/data:rw
 environment:
 - POSTGRES_DB=nextcloud
 - POSTGRES_USER=nextcloud

 - POSTGRES_PASSWORD=94dbd6b2-bd9f-4867-99af-37f8e4444640 # gitleaks:allow
 logging:
 driver: json-file
 options:
 max-size: 100m

Notes ¶

Tried updating from 16.0.4.1 to 18.0.3 through 17.0.5 - everything failed. Reinstalled and restored files beforehand by downloading them myself from ./nextcloud/data/username/files. Went back using file synchronization via client application.
The Admin account cannot be deleted through the UI. I created my own account, then tried to delete the standard one as I believe all accounts should always be unique, no fuss. You can delete it like this:

1

sudo docker-compose exec -u www-data nextcloud php occ user:delete admin

For WebDAV and CalDAV to work properly, add the following configuration to your haproxy configuration file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61


global
 log 127.0.0.1 local0
 stats socket /var/run/haproxy.sock mode 660 level admin expose-fd listeners
 stats timeout 30s
 user root
 group root
 daemon
 maxconn 4096
 nbproc 1
 # Default SSL material locations
 ca-base /etc/ssl/certs
 crt-base /etc/ssl/private
 # general SSL config
 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets
 tune.ssl.default-dh-param 4096
 ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

defaults
 log global
 mode http
 option http-buffer-request
 option httplog
 option dontlognull
 option forwardfor
 timeout connect 5s
 timeout client 25s
 timeout server 25s
 timeout tunnel 3600s
 timeout http-keep-alive 1s
 timeout http-request 15s
 timeout queue 30s
 timeout tarpit 60s
 compression algo gzip
 compression type text/plain text/css text/xml text/javascript application/javascript application/x-javascript application/xml application/json
 errorfile 400 /usr/local/etc/haproxy/errors/400.http
 errorfile 403 /usr/local/etc/haproxy/errors/403.http
 errorfile 408 /usr/local/etc/haproxy/errors/408.http
 errorfile 500 /usr/local/etc/haproxy/errors/500.http
 errorfile 502 /usr/local/etc/haproxy/errors/502.http
 errorfile 503 /usr/local/etc/haproxy/errors/503.http
 errorfile 504 /usr/local/etc/haproxy/errors/504.http

frontend http_https
 bind :80
 bind :443 ssl crt /certificates/cloudflare.haproxy alpn h2,http/1.1
 # acls
 acl dav path_reg -i ^/.well-known/(carddav|caldav).*$
 acl webfinger path_reg -i ^/.well-known/(carddav|caldav).*$
 # proxy headers
 http-request set-header X-Forwarded-Port %[dst_port] if forwarded_port
 # http-response set-header Access-Control-Allow-Credentials "true"
 http-request add-header X-Forwarded-Proto https if https forwarded_proto
 http-request replace-path (.*) /remote.php/dav if dav
 http-request replace-path (.*) /public.php?service=webfinger if webfinger
 default_backend cloud

backend nextcloud
 http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload;"
 server default nextcloud:80 check

Authorization was rotating endlessly and only F5 allowed access. On the PC, the problem disappeared, but how about with a client for Android? Here, it turned out that you can simply add a magic word to your config and you’ll be able to log in forever.

Command Line Setup

Wed, 22 Apr 2020 00:00:00 +0000

Written together with ReBrainMe

The command-line shell is used for any server manipulation. There are many ways to configure servers. It’s important that it’s comfortable for the administrator, but the main thing is that it’s efficient in terms of time, effort, and cost. The command line can be colorful, with arrows of different colors showing the path to the current directory, time, CPU load, and other stuff. Here’s an example from here:

It’s beautiful, yes, but too many colors distract from the essence. Also, the Git repository branch is displayed, taking up vital space on the left, though it could be moved to the right.

Command Line Shells ¶

Shells come in different types, though they solve the same problems in very similar ways, they are configured differently and have different functionality. Try different ones and find your own! … or write one yourself.

bash – the de facto standard shell, with basic autocompletion and path completion, included in most distributions by default, often used as a script shell.
fish and zsh – two different shells that greatly surpass bash in functionality, often heard of.
ksh, tsh, csh … other shells, which may be just as good or even better, but are slightly less common.

The goal of this article is to subjectively figure out what really needs to be in the terminal window and suggest a solution based on zsh.

What is needed for the command line to be effective?

Shell Requirements ¶

Path autocompletion. This is undeniably the main feature needed for navigating directory trees. It may seem that Bash has path autocompletion by default by pressing Tab twice. Try Zsh – you only need to press Tab once, a small thing, right? Now go back to Bash :) Additionally, Zsh can be configured to display a list of files matching the given prefix, which eliminates the need to interrupt input and run ls -la, you can even select the option with arrow keys. This is not used often, but it’s convenient.
Command history search. If you’ve already switched to ranger, this might become the most important requirement for you. At first, we learn about Ctrl+R to search for a command in the history of typed commands. Convenient, but not perfect. In Zsh, you can just type a piece of the command and use the arrow keys to select the one you need from the autosuggestions. Subjectively more convenient, and here’s why: to show the next command from history, you need to press Ctrl+R again, which is not very comfortable, especially when you skip over the one you need.
Highlighting the command prompt. When scrolling through a large number of logs in the terminal from different commands, you start to lose track and spend extra time distinguishing which block of logs belongs to which command, where the command itself is, etc. Highlighting the command prompt is really necessary, but not in a way that overshadows everything. Often, the first thing a beginner changes in a non-standard shell is the contents of the PS1 variable. Here’s a good description of PS1-4 variables.
Lightweight. Briefly: when you’re on a server where the loadavg is tens of times exceeded, you’ll regret setting up resource-intensive code in motd or the beginning of .zshrc (.bashrc) that outputs extensive CPU, memory, network, disk usage statistics, etc.
Uncluttered. Even if zsh is chosen for daily use, I write all scripts with a shebang for classic sh because it’s practically everywhere, and the syntax is standardized. In other words, it’s not recommended to write scripts using the non-standard syntaxes of advanced shells, as it will be a hassle to install them on every server.
Comfort. Efficiency and comfort are closely related. Setting a semi-transparent image as the background or picking fonts to your liking is also very important when you find yourself thinking that ugly fonts and a clunky interface kill your motivation to work completely. Just don’t overdo it.

Aliases ¶

It’s very convenient to declare aliases to speed up command typing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


alias ..="cd .. && ls -lhA --color=auto"
alias a="ls -lhA --color=auto"
alias d="sudo docker"
alias dc="sudo docker-compose"
alias die="sudo shutdown now"
alias e="exit"
alias g="git"
alias j="sudo journalctl"
alias k="kubectl"
alias l="ls -lh --color=auto"
alias la="ls -lhA --color=auto"
alias p="ping 8.8.8.8"
alias r="ranger"
alias reborn="sudo shutdown -r now"
alias s="sudo"
alias sctl="sudo systemctl"
alias sr="screen -R"
alias svi="sudo vim"
alias svim="sudo vim"
alias t="terraform"

And a little humor: alias please=‘sudo’ will make your interaction with the computer much smoother.

Zsh Configuration ¶

As you can see, each command is highlighted between separators like ; && |, and the command is entered on the second line, after the prompt indicating the current directory and username with host. This makes long commands more readable and manageable. On the right, honestly borrowed from the standard settings shipped with zsh in Manjaro, is the current Git repository branch. If there are changes, deletions, or additions, yellow, green, and red circles appear. Since it didn’t take up much space, it was decided to keep it. Why not. When typing a command, the last one that matches the prefix is suggested. You can confirm the selection by pressing the right arrow key. The terminal emulator window is Terminator.

To set up and configure the terminal window as in the example, follow these steps:

Install ohmyzsh
Copy .zshrc to your home directory

1

install -o "$(id -u)" -g "$(id -g)" -m 0644 .zshrc ~/.zshrc

Unpack the plugins

1
2


sudo tar -xpzvf plugins.tgz /
sudo usermod -s "$(command -v zsh)" "$USER"

Set zsh as the default shell

1

sudo usermod -s "$(command -v zsh)" "$USER"

Introduction to CEPH

Wed, 22 Apr 2020 00:00:00 +0000

Installation ¶

CEPH is a distributed network storage system that supports S3, iSCSI, RBD, etc. It’s a great solution, with plenty of documentation available, but it can be tricky to figure out the correct installation method at first.

Rook ¶

The officially recommended way to install CEPH on Kubernetes. You can read about the experience of the team at Flant here.

Cephadm ¶

Based on this, I wrote some Ansible playbooks and successfully managed a cluster deployed on a host. It accepted 40 GiB over the local network in 2 minutes and remained stable :)

Deprecated / Not Recommended ¶

The methods I tried that turned out to be either non-working or deprecated:

ceph-deploy: somewhere in the documentation, I found that this method is not recommended. Unfortunately, I had already spent a couple of days on it. I can’t provide a link since their SSL certificate expired, and I’m blocked from accessing the site due to HSTS.
ceph ansible: comprehensive and feature-rich, but CentOS 7 is no longer supported on the master branch, and the monitors didn’t synchronize. I couldn’t find a way to install the dashboard without monitoring, so I implemented it myself. I gained experience with Ansible, but then new issues arose, prompting me to switch to cephadm, which I’m glad I did.

Useful Commands ¶

Here’s a cheatsheet I found helpful.
To view the list of disks in use on the servers:

1
2
3


ceph orch device ls
ceph device ls
ceph device ls-by-host ceph-server-0

To check the space usage summary:

1
2
3


ceph osd df
ceph df
ceph df | grep -oE '[0-9.]+\s*[a-zA-Z]+$' | uniq | sort -n

To view the cluster status:

1
2


ceph status
ceph -w

To see the infrastructure, including which services/containers are running and where:

1
2


ceph orch ls
ceph orch ps

Replacing an OSD ¶

Here’s a good article from the same Flant team, along with a useful comment that complements it well. I recreated the OSD when I expanded the block device on the server (virtual machine) to add space to the cluster. I followed this process:

The cluster must be in HEALTH_OK state.
Remove the OSD from the cluster.
Delete the daemon (container) holding this OSD.
Purge the OSD.
Clear the partition.
Reconnect it.
Wait for HEALTH_OK.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# The OSD we want to recreate
OSD='osd.0'

# Find the host
HOST="$(ceph device ls | awk "/${OSD}/{print \$2}" | cut -d: -f1)"

# Get the name of the block device allocated to the OSD
DEV="/dev/$(ceph device ls | awk "/${OSD}/{print \$2}" | cut -d: -f2)"

# Step 1
ceph health | grep -q 'HEALTH_OK'

# Step 2
ceph osd out "$OSD"

# Step 3
ceph orch daemon rm "$OSD" --force

# Step 4
ceph osd purge "$OSD" --force

# Step 5
ceph orch device zap "${HOST}:${DEV}" --force

# Step 6
NEW_ID="osd.$(ceph orch daemon add osd "${HOST}:${DEV}' | awk '{print $3}')"
sleep 5
ceph osd crush reweight "$NEW_ID" 1

# Step 7
while ! ceph health | grep -q 'HEALTH_OK'; do
 sleep 5
done

In my experience, replacing an OSD with 15 GiB of data took about 30 minutes. This was tested on three CX21 Hetzner instances.

RBD and Docker ¶

Setting up RBD and integrating it with Docker:

Create a pool.
Initialize the pool.
Create a user for the setup.
Disable unnecessary features. Some RBD features aren’t supported in Linux kernels before 4.17, so they had to be disabled. You can check the options with:

1

ceph config help rbd_default_features

Install the Docker volume plugin.
Test on the container.
Mount the volume and verify the file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Step 1
ceph osd pool create rbd

# Step 2
rbd pool init rbd

# Step 3
ceph auth get-or-create client.rbd mon 'profile rbd' osd 'profile rbd pool=rbd' mgr 'profile rbd pool=rbd' > /etc/ceph/ceph.client.rbd.keyring

# Step 4
ceph config set global rbd_default_features 7

I used this volume driver. Other options I found hadn’t been updated in 3-4 years and seemed abandoned. This driver doesn’t require anything to run on the side and integrates directly into Docker. Install it like this:

1
2
3
4
5
6
7
8
9


# Step 5
docker plugin install wetopi/rbd \
 --alias=rbd \
 --grant-all-permissions \
 LOG_LEVEL=1 \
 RBD_CONF_POOL=rbd \
 RBD_CONF_CLUSTER=ceph \
 RBD_CONF_KEYRING_USER=client.rbd \
 MOUNT_OPTIONS='--options=noatime,discard'

Run a simple Docker Compose stack and mount the image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


version: "3.8"

services:
 test:
 image: alpine
 command: sh -c 'cat /vol/test; date | tee /vol/test'
 volumes:
 - data-volume:/vol

volumes:
 data-volume:
 name: vol1
 driver: rbd
 driver_opts:
 size: 123 # MiB

Run and verify:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Step 6
sudo docker-compose up

# Step 7
rbd map vol1
mkdir /tmp/vol1
mount /dev/rbd0 /tmp/vol1
cd /tmp/vol1
ls -la
cat test
cd /
umount /tmp/vol1
rbd unmap /dev/rbd0

Remmina

Wed, 22 Apr 2020 00:00:00 +0000

Great client application for connecting by RDP/VNC/etc. I installed it on Ubuntu 18.04 like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


sudo add-apt-repository ppa:remmina-ppa-team/remmina-next
sudo apt update
sudo apt install remmina \
 remmina-plugin-exec \
 remmina-plugin-kwallet \
 remmina-plugin-nx \
 remmina-plugin-rdp \
 remmina-plugin-secret \
 remmina-plugin-spice \
 remmina-plugin-vnc \
 remmina-plugin-www \
 remmina-plugin-xdmcp \
 libfreerdp-plugins-standard
sudo ln -s /usr/lib/x86_64-linux-gnu/libssh.so.4 /usr/lib/x86_64-linux-gnu/libssh_threads.so.4

More information is available on their page.

Basics of SSH

Mon, 09 Mar 2020 00:00:00 +0000

Introduction ¶

Secure Shell - de facto standard way to get access to a remote terminal on an virtual server. On the computer where you need to get access, SSH server is installed, and on the computer where the operator works, SSH client is installed. Through SSH we can:

Transfer files
Execute commands on the target server
Get information about the target system and perform simple automation

On Linux and MacOS, we have a terminal - SSH, which is usually set up by default. For Windows, I used the following options:

MobaXTerm - convenient, but paid. Suitable for non-commercial use, but you’ll need to buy a license if you want to use it in a company. Advantages:
- Immediately provides a file manager with drag-n-drop files to the target system
- Normal keyboard support and RSA key support
Git - when setting up Git, it includes Git Bash and an SSH client. Very close to the usual Bash.

Connecting ¶

1

ssh [-p <ssh порт>] <пользователь>@<ip целевого хоста>

The port is optional, defaulting to 22, but it’s highly recommended not to run the SSH server on the standard port, as the world is full of automated bots that scan networks and find virtual machines with SSH on the standard port and simple passwords, trying to add them to some botnet. I once fell for this, but it’s a different story…

Copying Files ¶

To copy files to the target host, you can use scp.

1

scp [-P <ssh порт>] /path/to/local/file <пользователь>@<ip целевого хоста>:<путь на сервере назначения>

Note:

The port for scp is specified using -P, whereas for ordinary SSH - -p.
The folder on the target machine should belong to the user you’re connecting with, or you’ll need permission to write to that folder\file.

First Post on Ghost

Sun, 08 Mar 2020 00:00:00 +0000

Now the blog is written on Hugo. This article is the first post from the old blog, which, just like that, was on Ghost.

Notes

First article. Reminded me of Isaac Asimov’s “The Last Question”, listened to on the way to work one of those ordinary days. A good thing - recommend it. Writing a good article, which will be a worthwhile read, is not as easy as it seemed at first. There are two reasons for this:

Thoughts must be useful.
The text should be systematized.

The system is launched on Ghost - the server is good, fast, and started easily, there’s enough tooling for such a pseudo-blogger like me. There’s a built-in plugin for choosing images from Unsplash - images that can be published without fear of being sued for copyright infringement. docker-compose.yml file, which makes this whole thing easy to set up, is presented below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


version: '3.7'

services:
 ghost:
 image: ghost:3.9.0-alpine
 container_name: ghost
 restart: unless-stopped
 port:
 - '80:2368'
 volumes:
 - /data/ghost:/var/lib/ghost/content:rw
 environment:
 - url=http://example.com
 logging:
 driver: json-file
 options:
 max-size: 100m

Notes ¶

Always keep the version of the image, from which the container will be started. This helps avoid problems with the stack restarting later on. If you use latest, it’s possible that when you recreate the stack, up -> down might start again a newer version, which won’t be compatible with the current data and settings etc. Moreover, it’s impossible to quickly and easily remember which version was exactly launched at the moment of startup?
Always control the logging process. If using the standard Docker logging driver - json-file, then by default, the Docker daemon will receive logs and write them to a file for this container until the disk runs out. By default, the file is always one, but its location can be limited by setting the max-size: 100m option (I just took 100 MB, it’s unlikely that anyone would read more than that). More information about the json-file Docker logging driver.
I recommend always specifying container_name - it’s easier to read docker ps.

Reverse SSH Tunnels

Sun, 20 Oct 2019 00:00:00 +0000

Introduction ¶

When needing to access a server located behind NAT, direct connection cannot be made. Instead, we need to come up with alternative methods. We’ll explore how to use both direct and reverse SSH tunnels for these purposes.

Tunnels ¶

Reverse Tunnel ¶

Created from the target server (CS) to a intermediate (PS).
Socket created on PS.
Direct tunnel - created from the initial server (NS) to PS.
Socket located on NS.

Using two tunnels, we can access CS. Ports for any website or even just SSH server in CS can be cast and gain access into the system. SOCKS5 proxy can also be used to exit the network through CS.

Intermediate Server must be accessible from both CS and NS

Creating User on PS ¶

Using a user that can only create tunnels is recommended for security purposes. Below code creates a new user with no home directory or interpreter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


sudo adduser -s /bin/false -M -N tunnel
sudo usermod -d / tunnel
echo 'tunnel:tunnel' | sudo chpasswd
cat <<EOF | sudo tee -a /etc/ssh/sshd_config

Match User tunnel
 PasswordAuthentication yes
 AuthenticationMethods "password"
EOF
sudo systemctl restart sshd

Reverse Tunnel ¶

Now let’s move on to creating tunnels. We’ll need a reverse tunnel from PS to CS.

Below is sshpass command, which is necessary for automatically entering passwords when connecting via SSH:

1
2


mkdir -p ~/.ssh
ssh-keyscan -H jump-host >> ~/.ssh/known_hosts

Creating the reverse tunnel on CS:

1

sshpass -p 'password' ssh -fNR 2222:localhost:22 tunnel@jump-host

ssh-keyscan -H jump-host adds SSH Server’s key to PS’s known hosts.
-fN executes command and doesn’t create a tty session.
-R creates the reverse tunnel.
2222:localhost:22 translates into: proxying 127.0.0.1:2222 (default localhost) to PS at 127.0.0.1:22 on CS. So, it does exactly this: proxies 127.0.0.1:2222 from NS’s default address localhost to PS’s address 127.0.0.1. Now we can access CS via PS.
Checking the tunnel can be done with:

1

ssh -p 2222 localhost hostname

Direct Tunnel ¶

To get into reverse tunnel on PS, a direct tunnel from NS needs to be created:

1
2
3


mkdir -p ~/.ssh
ssh-keyscan -H jump-host >> ~/.ssh/known_hosts
sshpass -p 'password' ssh -fNL 1234:localhost:2222 tunnel@jump-host

1234:localhost:2222 translates into: proxying 127.0.0.1:1234 from NS to PS at 127.0.0.1:2222 on CS, which means proxying from NS’s address localhost at port 1234 to PS’s address 127.0.0.1 at port 2222. So, now we can access reverse tunnel on PS using direct tunnel.
To access the tunnel:

1

ssh -fND 1329 -p 1234 user-on-th@localhost

SOCKS5 Proxy ¶

We can set up any browser to use a SOCKS5 proxy for directing all traffic. This is convenient when trying to access resources in local network CS from NS. After creating tunnels, the following command needs to be executed on NS:

1

ssh -fND 1329 -p 1234 user-on-th@localhost

And to configure a browser’s SOCKS5 proxy settings, look at this screenshot.

OpenSSL Snippet

Thu, 25 Apr 2019 00:00:00 +0000

Here is the translated text:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91


#!/usr/bin/env sh
set -e

# path to private part of CA (PEM encoded)
CA_KEY='ca.key'
# path to public part of CA (PEM encoded)
CA_CRT='ca.crt'
# password to use CA
CA_PASSWORD='longalphanum'

# Country
C='UA'
# State
ST='Kiev'
# Location
L='Kiev'
# Organization
O='No organization'
# Organization unit
OU='No unit'
# Domain to issue final cert for
DOMAIN='vlasov.pro'
# Days while the certificate will be valid
DAYS='365'

subj() {
 echo "/C=${C}/ST=${ST}/L=${L}/O=${O}/OU=${OU}/CN=$1"
}

if [ ! -f "$CA_KEY" ] || [ "$1" = '-ca' ]; then
 echo 'basicConstraints=CA:true' > extfile.txt
 # CA - used for signing of both server and client certs
 # generate private key
 openssl ecparam -genkey -name prime256v1 | openssl ec -out "$CA_KEY"
 # generate signing request
 openssl req -new -key "$CA_KEY" -out ca.csr -subj "$(subj "$O CA")"
 # csr + key = crt
 openssl x509 -req -days 3650 -in ca.csr -signkey "$CA_KEY" -extfile extfile.txt -out "$CA_CRT"
 rm -f ca.csr
 # convert crt to DER format
 openssl x509 -inform PEM -outform DER -in "$CA_CRT" -out "$CA_CRT.der"
 echo 'Generated CA'
fi

# generate and sign certificate
cat <<EOF >extfile.txt
[ req ]
req_extensions = req_ext
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ req_ext ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = $DOMAIN
DNS.2 = *.$DOMAIN
EOF
# issue
openssl ecparam -genkey -name prime256v1 | openssl ec -out "${DOMAIN}.key" # key
openssl req -new -key "${DOMAIN}.key" -out "${DOMAIN}.csr" -config extfile.txt -subj "$(subj $DOMAIN)" # csr
openssl x509 -req \
 -days "$DAYS" \
 -in "${DOMAIN}.csr" \
 -CA "$CA_CRT" \
 -out "${DOMAIN}.crt" \
 -CAkey "$CA_KEY" \
 -CAcreateserial \
 -passin "pass:$CA_PASSWORD" \
 -extensions req_ext -extfile extfile.txt # csr + rootCA.crt = crt 
rm -f "${DOMAIN}.csr"
# convert DER -> PEM
openssl x509 -inform PEM -outform DER -in "${DOMAIN}.crt" -out "${DOMAIN}.der.crt" # convert
# convert PEM -> pkcs12
openssl pkcs12 -export \
 -inkey "${DOMAIN}.key" \
 -in "${DOMAIN}.crt" \
 -out "${DOMAIN}.pfx" \
 -password "pass:$JKS_PASSWORD" \
 -name "$DOMAIN"
# create fullchain
cat "${DOMAIN}.crt" "$CA_CRT" > "${DOMAIN}.full"
# verify
openssl verify -CAfile "$CA_CRT" "${DOMAIN}.crt"
# info
log_info "Generated $DOMAIN"
done
rm -f extfile.txt

Mon, 01 Jan 0001 00:00:00 +0000

How to Set Up Commit and Tag Signing in Git Using GPG and SSH ¶

Signing commits and tags ensures that the changes are genuinely made by you. This is an essential step for project security and transparency. In this article, I will guide you through configuring automatic commit and tag signing in Git and show you how to separate configurations for personal and work projects.

Main Git Configuration File ¶

The main configuration file, ~/.gitconfig, defines general settings for all repositories. Here, you can specify your username, enable mandatory signing for commits and tags, and include additional configuration files for specific project groups.

Here’s an example of a basic configuration file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# ~/.gitconfig
[user]
 name = John Smith

[commit]
 gpgsign = true

[tag]
 gpgSign = true

[includeIf "gitdir:~/Personal/"]
 path = ~/Personal/.gitconfig

[includeIf "gitdir:~/Work/"]
 path = ~/Work/.gitconfig

With the [includeIf] directives, Git will automatically apply the appropriate configuration based on the repository’s location.

Configuration for Personal Projects ¶

If you use SSH for signing in personal projects, the setup looks like this. In the configuration file ~/Personal/.gitconfig, you specify your personal email, set the signing format to ssh, and provide a command that allows Git to find the appropriate key in the ssh-agent.

1
2
3
4
5
6
7
8
9


# ~/Personal/.gitconfig
[user]
 email = personal@a.com

[gpg]
 format = ssh

[gpg "ssh"]
 defaultKeyCommand = sh -c "ssh-add -L | awk '$3~/^personal@a\\.com$/{print \"key::\"$0}'"

This configuration ensures that Git uses the key already added to the ssh-agent, simplifying the signing process.

Configuration for Work Projects ¶

For work projects, GPG is often used. In the configuration file ~/Work/.gitconfig, you define your work email and the GPG key ID to be used for signing.

If you don’t have a GPG key yet, you can create one using the following command:

1

gpg --full-gen-key

To get the key ID, run:

1

gpg --list-keys

Here’s an example configuration file for work projects:

1
2
3
4
5
6
7


# ~/Work/.gitconfig
[user]
 email = company@b.com
 signingKey = 35C1A64CD7FC0AB6EB66756B2445463C3234ECE1

[gpg]
 format = openpgp

This configuration ensures that commits and tags in work repositories are signed using the specified GPG key.

With these configurations, Git will automatically use the appropriate signing key depending on the repository. This helps maintain order and transparency when working with commits and tags.

DevOps Notes

Clever Disk Encryption with dm-crypt Nuke

Introduction ¶

Environment Setup ¶

Adding mkpasswd Support in initramfs ¶

Creating the decrypt_smart_nuke Script ¶

Boot Configuration ¶

Updating initramfs ¶

Conclusion ¶

Kubernetes Node role labeler

Description ¶

Demo ¶

Karpenter configuration ¶

BTRFS checksum benchmark

Introduction ¶

Body ¶

Redis Migration Script

Old Method ¶

New Method ¶

My Experience with Git

Introduction ¶

Main ¶

Git has a built-in graphical interface ¶

How to update environment settings based on the diff of the new environment ¶

How to search for passwords and sensitive data using commit search ¶

Rebase strategies ¶

How to reset a local branch to the state of the cloud if someone pushed with force ¶

Kubernetes in Docker

Introduction ¶

kind ¶

Redis Sentinel Watchdog

Introduction ¶

Timeshift BTRFS Backups on Manjaro

Introduction ¶

System Requirements ¶

How it works ¶

Convenient and Colorful ZSH with p10k

Introduction ¶

Configuring p10k ¶

Explanation of my .zshrc ¶

Installing p10k ¶

Plugins ¶

Adding Paths to PATH ¶

Aliases ¶

Git Config ¶

KUBECONFIG ¶

Automatic Utility Downloads ¶

Separate Environments ¶

Kubernetes Authorization via OIDC

Introduction ¶

Access via ServiceAccount ¶

Creating a User ¶

Google OIDC ¶

Authentik ¶

wg-quick network namespace

TL;DR ¶

Introduction ¶

Problem Statement ¶

Solution Analysis ¶

PreUp ¶

PostUp ¶

How wg-%i-exec works ¶

Conclusion ¶

GPT (UEFI) for Btrfs (/ and /boot) over dm-crypt

Introduction ¶

Partition Table Type ¶

File Systems ¶

GPT Partitions ¶

Operating Systems ¶

Important Notes ¶

Unimportant Notes ¶

Grub recovery script ¶

Simple Explanation of OS Booting (EFI) ¶

Conclusion ¶

Overview of k9s

Introduction ¶

Overview ¶

Viewing logs ¶

Copying to clipboard buffer ¶

Port forwarding ¶

Adding `mkpasswd` Support in initramfs ¶

Creating the `decrypt_smart_nuke` Script ¶

2. Use `this` as a name, but only in modules ¶

6. Resource naming with `name_prefix` ¶