Linux for blockchain engineers

A starting question

You rent a small server from a host. A few minutes after it goes online, the logs show dozens of login attempts from all over the world. Nobody is "targeting" you: programs continuously try passwords on every address that answers on port 22. A password, even a long one, is a permanent target. A key solves the problem, but only if you use it correctly.

What SSH does

SSH (Secure Shell) opens an encrypted terminal on a remote machine. It uses TCP port 22 by default (workshop 101). It does two distinct things:

1. Authenticate the server to you (so you are not talking to an impostor).
2. Authenticate the user to the server.

Key authentication: the idea

A key pair has a secret private key that produces signatures, and a public key anyone can use to verify those signatures (principle detailed in workshop 105). For SSH:

• On your computer you create a pair: ssh-keygen -t ed25519 -C "my-laptop". The file ~/.ssh/id_ed25519 is the private key; ~/.ssh/id_ed25519.pub is the public key.
• You copy the public key into ~/.ssh/authorized_keys on the server (the command ssh-copy-id user@server does it).
• At login, your client signs data tied to this session with the private key (a simplification: there is no separate random challenge); the server checks the signature with the public key. The private key never leaves your computer and is never sent to the server.

Protect the private file with a passphrase: if it is stolen, it remains encrypted.

First connection: the server's fingerprint

The first time, the client displays:

The authenticity of host 'srv.example.test (203.0.113.10)' can't be established.
ED25519 key fingerprint is SHA256:AbCdEfGhIjKlMnOpQrStUvWxYz0123456789FICTIF.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

(The address 203.0.113.10 and the fingerprint are fictitious examples; the range 203.0.113.0/24 is reserved for documentation.) Compare the fingerprint with the one your host shows in its console. If you answer "yes" without comparing, you may be accepting an impostor. Once accepted, the fingerprint is stored in ~/.ssh/known_hosts. If it changes later, SSH prints a big warning ("REMOTE HOST IDENTIFICATION HAS CHANGED"): do not ignore it, investigate (legitimate reinstall or interception).

Permissions: without them, SSH refuses

SSH refuses a private key that others can read. Usual rules:

The message "Permissions 0644 for 'id_ed25519' are too open. UNPROTECTED PRIVATE KEY FILE!" tells you exactly this. Fix: chmod 600 ~/.ssh/id_ed25519.

Closing password access — without locking yourself out

The file /etc/ssh/sshd_config configures the SSH server. The useful settings:

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
PubkeyAuthentication yes

(KbdInteractiveAuthentication no closes another route by which a password may still be requested on some server images; depending on the OpenSSH version, this keyword may be called ChallengeResponseAuthentication.)

Safe procedure, in order:

1. Keep a second session open (never close your only connection).
2. Check that a key works: open a new connection with the key.
3. Edit the configuration, then test its syntax: sudo sshd -t (no output = syntax is correct).
4. Reload the service (sudo systemctl reload ssh on Ubuntu; the service is named sshd on other distributions). A port change may need a restart rather than a reload, especially if the service is started by a systemd socket: check with ss -tlnp.
5. From a third terminal, check that key login still works and that a password attempt is refused.

What this does NOT guarantee

• A key protects the connection, not the machine: if your laptop is compromised, so is the key. Hence the passphrase and one key per device.
• Disabling the password does not stop attempts, only their success. Changing the port may reduce log noise, it is not a protection.
• An account with no password can still be attacked through other open services: closing SSH is not closing the server.

Recap

Create a key pair, deposit the public key, compare the fingerprint at first connection, set the right permissions, then disable password and remote root — keeping a session open and checking the effective value.

The problem

A developer starts their node with sudo "to make it work". The node therefore runs as root. A flaw in the program then gives the attacker the whole machine. The least privilege principle: each program runs with an account that can only do its job.

Accounts: one human, one service

• An administration account for you (non-root): sudo adduser deploy then sudo usermod -aG sudo deploy (on Ubuntu the sudo group allows sudo). You log in with this account and raise your rights only when needed.
• A service account for the node: sudo useradd --system --no-create-home --shell /usr/sbin/nologin nodeuser. Nobody logs in as it; it only owns the node's data directory.

Reading permissions

$ ls -l /var/lib/mynode
drwxr-x--- 4 nodeuser nodeuser 4096 Mar  4 10:12 chain
-rw------- 1 nodeuser nodeuser   64 Mar  4 10:12 node.key
-rw-r--r-- 1 root     root      210 Mar  4 09:58 README.txt

Each line starts with ten characters: the type (d directory, - file) then three groups of three rights for the owner, the group, the others. Each group is worth: read r = 4, write w = 2, execute x = 1. You add them.

• rwx = 4 + 2 + 1 = 7; r-x = 4 + 0 + 1 = 5; r-- = 4; rw- = 6; --- = 0.
• drwxr-x--- → directory, 7 5 0 → 750.
• -rw------- → 6 0 0 → 600: only the owner reads and writes. This is the expected mode for a key.
• -rw-r--r-- → 6 4 4 → 644: readable by everyone.

For a directory, x means "being able to enter it": without it you cannot traverse the directory, even with r.

Commands: chmod 600 file, chmod 750 directory, sudo chown -R nodeuser:nodeuser /var/lib/mynode.

Firewall with ufw

A firewall filters incoming connections. On Ubuntu, ufw (Uncomplicated Firewall) is a simple front end to the kernel's firewall. Starting rule: deny everything incoming, open only what is needed.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH          # or: sudo ufw allow 22/tcp
sudo ufw allow 30333/tcp        # node p2p port (example; see the client docs)
sudo ufw enable
sudo ufw status verbose

Order matters: if you enable ufw before allowing SSH, you cut yourself off from the server. Port 30333 is an example; each node software documents its ports.

RPC: bind to localhost

A chain node often exposes an RPC interface (an API that applications call remotely). If it listens on 0.0.0.0, it answers on all interfaces, hence from the Internet. If it listens on 127.0.0.1, only the server itself can call it. Check with ss -tlnp:

State  Recv-Q Send-Q  Local Address:Port  Peer Address:Port Process
LISTEN 0      4096        127.0.0.1:8545       0.0.0.0:*    users:(("mynode",pid=812,fd=20))
LISTEN 0      128           0.0.0.0:22         0.0.0.0:*    users:(("sshd",pid=701,fd=3))

Here the RPC (example port 8545) is reachable locally only; SSH is open to everyone. Beware: an IPv6 address such as [::] ("all interfaces" in IPv6) exposes the service just like 0.0.0.0. To expose an RPC to the public, put a reverse proxy with rate limiting in front, never the raw node.

Worked example: a permissions diagnosis

The node (account nodeuser) fails with "Permission denied" on /var/lib/mynode/node.key, whose mode is -rw------- and owner root. Reasoning: the owner (root) has 6, the group 0, others 0; nodeuser is neither the owner nor in the group: it has no rights. Fix: sudo chown nodeuser:nodeuser /var/lib/mynode/node.key (keep 600). No chmod 777.

What this does NOT guarantee

• A local firewall does not protect what is already allowed: a vulnerable open service stays vulnerable.
• Unix rights limit ordinary accounts, but not root: an attacker who obtains root bypasses everything.
• The system firewall and the host's firewall are two distinct levels: a rule in one does not open what the other blocks.
• A port published by Docker can bypass ufw rules (you will see this in workshop 117): do not trust ufw status alone, check with ss -tlnp and from outside.

Recap

One account per role, minimal rights (600 for a secret, 750 for a service directory), a firewall that denies by default, and RPC interfaces bound to localhost.

The problem

You start your node in a terminal. You close the terminal: the node stops. The server reboots after a power cut: the node does not come back. A serious program must be a service: started at boot, restarted on a crash, with its messages kept.

systemd in a nutshell

systemd is the service manager of most modern distributions, including Ubuntu. A service is described in a unit file (.service), placed in /etc/systemd/system/.

[Unit]
Description=Example chain node (training)
After=network-online.target
Wants=network-online.target

[Service]
User=nodeuser
EnvironmentFile=/etc/mynode/env
ExecStart=/usr/local/bin/mynode --data-dir /var/lib/mynode --rpc-bind 127.0.0.1
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

• User: the service account (least privilege).
• ExecStart: the command, preferably with an absolute path. systemd rejects a relative path such as ./mynode; a plain name with no / is looked up in a few fixed directories, which depends on the installation: the absolute path removes any ambiguity.
• Restart=on-failure + RestartSec=5: restart 5 s after a crash.
• WantedBy=multi-user.target: started at normal boot once enabled.
• EnvironmentFile: a file of variables, set to mode 600 owned by root if it contains secrets. Avoid writing a secret directly into the unit file, which every account can read.

The commands

sudo systemctl daemon-reload          # re-read unit files after editing
sudo systemctl enable --now mynode    # start at boot AND now
sudo systemctl status mynode          # state, latest messages
sudo systemctl restart mynode
sudo systemctl stop mynode

After each edit of a unit file, daemon-reload is needed: without it, systemd keeps the old version in memory.

A status looks like this:

● mynode.service - Example chain node (training)
     Loaded: loaded (/etc/systemd/system/mynode.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2026-03-04 10:31:02 UTC; 3s ago
    Process: 2345 ExecStart=/usr/local/bin/mynode ... (code=exited, status=1/FAILURE)
   Main PID: 2345 (code=exited, status=1/FAILURE)

"activating (auto-restart)" with "exited, status=1/FAILURE" means: the program crashes in a loop, systemd relaunches it every five seconds. The automatic restart hides the problem: you must read the logs.

Logs with journalctl

systemd gathers every service's messages in the journal.

journalctl -u mynode -n 50            # last 50 lines of this service
journalctl -u mynode -f               # follow live
journalctl -u mynode --since "1 hour ago"
journalctl -u mynode -p err           # errors only

Example output:

Mar 04 10:31:02 srv mynode[2345]: ERROR: cannot open /var/lib/mynode/node.key: Permission denied
Mar 04 10:31:02 srv systemd[1]: mynode.service: Main process exited, code=exited, status=1/FAILURE

It explains everything: it is the end of the previous lesson's diagnosis (the file's owner).

Logs and disk: rotation

Logs that grow without limit end up filling the disk (see lesson 4). Two tools:

• journald (systemd's journal) already caps its own size by default (a fraction of the file system, limited to a few GB according to its documentation, journald.conf). journalctl --disk-usage shows the space used; sudo journalctl --vacuum-size=500M deletes the oldest down to 500 MB; and for a permanent, predictable limit, SystemMaxUse=500M in /etc/systemd/journald.conf (then restart systemd-journald). This limit bounds the whole journal, all units together, not a single service.
• logrotate: for log files a program writes itself (for example /var/log/mynode/*.log), where unlimited growth is the real risk. A configuration in /etc/logrotate.d/ says how many files to keep, whether to compress and how often to rotate:

# /etc/logrotate.d/mynode (example)
/var/log/mynode/*.log {
    daily
    rotate 7
    compress
    missingok
    copytruncate
}

(copytruncate copies then empties the file: handy if the program keeps the file open, at the price of possibly losing a few lines written during the copy.)

Worked example

A node writes 40 MB of messages per day to a file /var/log/mynode/node.log that it manages itself. With no rotation, a year amounts to 40 × 365 = 14,600 MB, about 14.6 GB: it does not break in the first month, but eventually shows. With the logrotate configuration above (rotate 7: seven files kept, roughly a week of history), the volume stays in the order of a few hundred MB, compression included. On the journald side, SystemMaxUse=500M guarantees that the whole system's journal never exceeds 500 MB: you lose the oldest messages, you keep the disk. If you need a longer history, export the logs to another system rather than piling them on the node's disk.

What this does NOT guarantee

• Restart=on-failure repairs nothing: a node that crashes at every start loops forever. Monitor it; limit it with systemd's start-limit settings if needed.
• Restart=on-failure does not restart after a stop counted as normal: according to systemd's documentation, a stop by SIGTERM (what a plain kill sends) counts as a clean exit. To simulate a crash, use sudo kill -9 <PID> (SIGKILL).
• enable does not start immediately (without --now); start does not enable at boot. They are two different things.
• A log is not a backup: if the machine dies, it dies with it.

Recap

A unit file with a service user and absolute path, daemon-reload after every edit, journalctl -u to understand, and a size limit on logs.

The problem

At 3 a.m. your node stops. Three ordinary causes: full disk, exhausted memory, port already taken. You must recognise them within a minute from a command's output. The method: observe, form a hypothesis, confirm with a second command, and only then act.

1. Disk

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       200G  188G  2.2G  99% /
tmpfs           3.9G     0  3.9G   0% /dev/shm

df -h shows file-system usage. Here the root is at 99%. Notice that Size − Used (12 GB) is not Avail (2.2 GB): on the ext4 file system, a share (5% by default, about 10 GB here) is reserved for root. That is normal.

To find what takes the space:

$ sudo du -xh --max-depth=1 /var | sort -h | tail -4
1.4G    /var/cache
12G     /var/log
171G    /var/lib
185G    /var

(-x stays on a single file system.) Chain data (/var/lib) weighs 171 GB, logs 12 GB. Two levers: limit the logs (lesson 3), and above all plan a bigger data volume (a chain node's data keeps growing).

Two variants to know:

• Exhausted inodes: df -i. A disk can be "full" in number of files while bytes remain (millions of small files).
• Deleted but still open file: df shows a lot of space used, but du cannot find the files. A process still holds a deleted file; sudo lsof +L1 lists it. The space returns when the process stops.

2. Memory

$ free -m
               total        used        free      shared  buff/cache   available
Mem:            3913        3611         101          12         189         120
Swap:              0           0           0

Read the available column (memory the system can give to a program without resorting to swap), not free. Linux uses free memory for its cache: a low free is normal. Here available = 120 MB out of 3,913 MB and no swap: the system is in danger.

When memory runs out, the kernel may trigger the OOM killer, which kills a process. Its trace is in the kernel log:

$ journalctl -k | grep -i "out of memory"
Mar 04 03:12:44 srv kernel: Out of memory: Killed process 812 (mynode) total-vm:6120344kB, anon-rss:3402112kB

That is the proof: the node did not "crash by itself", it was killed for lack of memory. Remedies: a machine with more memory, a less greedy cache configuration (depending on the software), or another service that uses less. Swap can cushion it, but a node on a slow disk that swaps becomes very slow.

3. Ports

A program cannot listen on a port already in use:

mynode: ERROR: failed to bind 0.0.0.0:30333: Address already in use (os error 98)

Who uses it?

$ sudo ss -tlnp | grep :30333
LISTEN 0 128 0.0.0.0:30333 0.0.0.0:* users:(("mynode",pid=2210,fd=18))

Here it is another instance of mynode (PID 2210), often a forgotten old process or two services started by mistake. Remedy: stop the right service (systemctl stop), not kill at random. Do not confuse with a port closed by the firewall: then the program does listen, but connections from elsewhere do not arrive (with ufw's default policy, which drops packets, the symptom is a timeout on the client side, not an error on the server side; a rule that actively rejects would give "connection refused").

Worked example: ranking three alarms

Three alarms arrive: disk at 99%, available memory at 120 MB, port occupied. Here is a reasonable ranking by risk (a judgement, not a law: a node looping on OOM kills may come first): (1) a full disk can corrupt a database; (2) low memory leads to an OOM kill; (3) an occupied port only prevents starting. Act first on (1), after confirming with du what takes the space, and never delete the chain's data directory to "make room".

What this does NOT guarantee

• A command gives a snapshot: a one-second memory spike may not show. Watch over time (monitoring is a topic of its own).
• These symptoms have other possible causes: a faulty disk, a file-descriptor limit, a firewall. The hypothesis must be confirmed.
• Freeing space does not fix the cause: if data grows, a bigger volume is needed.

Recap

Disk: df -h then du; memory: free -m available column and the OOM trace in the kernel log; ports: ss -tlnp to see who listens. Confirm before acting.