One Server, Done Properly: A Debian Build You Can Audit

Ask a small business how their servers are secured and you’ll hear about the firewall. Ask who ran the command that broke the database last Thursday and you’ll get silence — because three people share root, the shell history was cleared, and the logs lived on the same disk that’s now in question.

That gap — between “we have security” and “we can prove what happened” — is the one this article closes. It’s the build I use for a plain Debian server in a small-business environment: one VM, one runbook, and three properties at the end that most small estates don’t have:

1. Nobody logs in as root, and nobody has a personal root password
2. Every privileged action is attributed to a named human, with a replayable transcript
3. Logs leave the box within seconds, and survive the box being lost

Nothing here needs an enterprise tool. It’s the packages already on the mirror, a couple of hundred lines of idempotent shell, and a handful of decisions made deliberately instead of by default.

Identity first: one place users exist

The provisioning rule is strict: the OS install creates root and nothing else. No local admin user, no “temporary” account that becomes permanent. Root gets an SSH key for exactly as long as it takes to join the machine to your directory — added via a drop-in file under sshd_config.d/, never by editing the main config — and then the account is locked in a single atomic step:

usermod --password '!' root

From that point, humans authenticate through the directory. Many small businesses already run Active Directory because the office is Windows; SSSD joins a Debian box to it cleanly, with a group naming convention mapping straight to Linux privileges — a linux_admins group that gets sudo, a linux_users group that gets a login and nothing more. Offboarding becomes “disable the AD account,” and it’s instantly true on every server. If you don’t run a directory, the pattern still holds: individual named accounts, no shared ones, and everything below applies unchanged.

One honest detail from the runbook: root-locked plus directory-authenticated means SSSD is the single service that can lock you out if a restart goes wrong. The runbook’s retrofit scripts therefore refuse to touch SSSD when they don’t need to, and every operation that edits sudoers opens with the same demand — keep a second SSH session open before you run this. Automation with validation and rollback is good; a second session is what saves you when the validation was wrong.

Accountability: you cannot block sudo -i, so attribute it

The tempting move once several admins have sudo is to forbid interactive root shells — blacklist sudo -i, sudo su -, sudo bash in the sudoers file. It doesn’t work, and it’s worth saying plainly why: any admin with broad sudo has dozens of documented ways to get a shell out of a permitted binary (the GTFOBins catalogue exists for exactly this). A blacklist gives you the feeling of control and a false line in the audit report.

The honest alternative is attribution, and Linux gives you two layers of it.

Layer 1: auditd with login-UID tracking. The kernel tags every process with the auid of the human who originally logged in, and — this is the whole point — that tag survives sudo, su, and any setuid dance. The admin who became root is still, to the kernel, the admin who logged in. Rules keyed on auid>=1000 capture who actually did what:

# Who changed the rules that govern privilege
-w /etc/sudoers       -p wa -k sudo_config
-w /etc/sudoers.d/    -p wa -k sudo_config
-w /etc/sssd/sssd.conf -p wa -k sssd_config

Layer 2: sudo I/O session recording. Enable it in a sudoers drop-in and every sudo session — including the interactive shell you couldn’t forbid — is transcribed keystroke by keystroke:

Defaults log_input, log_output, use_pty
Defaults iolog_dir=/var/log/sudo-io
Defaults iolog_file=%{seq}

sudoreplay then plays it back like a video: which human, which terminal, what they typed, what came back. That transcript is the evidence. Not “we have logging” — here is what happened at 14:31 on Thursday.

The GDPR decision hiding in that directory

Keystroke transcripts of your administrators are personal data about employees. This isn’t a reason not to do it — legitimate interest in the security of systems is well-trodden ground — but it’s a decision, and it needs the paperwork: a documented balancing test, staff informed that privileged sessions are recorded, and a finite retention period that you actually enforce. The runbook sets 90 days and a daily purge job, and the number is written down next to the reason. Indefinite retention of who-typed-what is how a security control turns into a liability; you’ll recognize that sentence from the backup series, because it’s the same principle.

Do the transcripts contain passwords?

The question everyone asks the moment they understand keystroke recording, and the honest answer has three parts.

The sudo prompt itself: never. Authentication happens before the I/O session begins, so what you type at sudo’s own prompt is not in the transcript.

Passwords typed inside a session: masked by default, heuristically. Since sudo 1.9.10, unless you set log_passwords, sudo watches the terminal output for something that looks like a password prompt — matched against passprompt_regex — and replaces the input that follows with * until the next line break. Debian 12’s sudo is well past that version (confirm with sudo -V). Two limits: a tool whose prompt doesn’t match the regex list gets no masking, and if a program echoes feedback while you type (pwfeedback, which some distributions now ship enabled), only the first character is masked. Keep pwfeedback off on any host with session recording.

Everything on screen: recorded, no exceptions. log_output captures what is printed. A cat of a .env file, an env dump, a config with a token in it — all of it is in the transcript, and nothing masks it. Secrets passed as command arguments (mysql -pHunter2) additionally land in sudo.log and the process list.

Which is why the real protection is the build, not the masking feature. Root is locked and identity lives in the directory, so there is no local password to type. SSH is key-based, so there is no password prompt to another host. Databases are reached through option files or socket authentication, never -p. Design the sessions so no secret is ever typed or printed in them; the masking is the seatbelt, not the driving. And treat the transcript directory as the sensitive material it is: owned by root, mode 0700, finite retention, and forwarded to the collector over an encrypted channel rather than left sitting on the box.

The mistake I made, and why it matters

Here’s the part I’d normally leave out of a polished article. The first version of my audit rules included this, to catch anyone tampering with the transcripts:

-w /var/log/sudo-io/ -p wa

Reasonable intent. Terrible rule. A -w watch on a directory is recursive — and sudo is writing transcript data into that directory continuously, potentially per keystroke. Every write generates an audit event, which auditd writes to disk, which the watch sees. A self-feeding flood that would have filled the disk, on a box whose entire purpose was to keep logs safely. If you’ve ever inherited “the audit logs are enormous and nobody knows why,” this class of rule is a likely culprit.

The fix keeps the intent and drops the noise — watch attribute changes and deletions, not writes:

-a always,exit -F arch=b64 -F dir=/var/log/sudo-io -F perm=a -F auid>=1000 -F auid!=unset -k sudo_io_tamper
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat,rmdir -F dir=/var/log/sudo-io -F auid>=1000 -F auid!=unset -k sudo_io_tamper

Same protection, a tiny fraction of the events. The broader lesson: audit rules are code, and they need load testing like code. Look at the event rate for a day before you call the build done.

Logs that survive the box

Attribution is worthless if the evidence lives only on the machine you’re investigating — an attacker with root, or a plain disk failure, takes the logs down with it. Three controls, in increasing importance:

Cap local volume so logging can’t take down the host. Don’t trust auditd’s shipped defaults — check them, because a KEEP_LOGS rotation policy never deletes anything and is exactly how audit fills a disk. Set max_log_file, num_logs, and max_log_file_action = ROTATE explicitly; set the disk-full actions to SUSPEND rather than halting the box. Bound journald the same way, and give sudo.log a logrotate stanza.

Forward everything off-host, immediately. rsyslog to your central log target — a self-hosted SIEM or a plain syslog collector; either is fine, as long as it’s a different machine with different credentials. The audit stream rides along via the audisp syslog plugin, so auid-tagged events reach the collector in near real time.

Buffer for the outage you will have. A disk-assisted queue in rsyslog — a spool sized to hold roughly 72 hours of your normal volume — means a collector outage over a long weekend costs nothing: events queue locally and drain on reconnect. Configure saveOnShutdown so a reboot mid-outage doesn’t lose the queue either.

And once logs are on a separate host, the immutability question from the backup articles applies to them too: an attacker who reaches your collector can rewrite history. Same answer as before — a locked, append-only copy — and it’s a subject I’ll come back to.

Build it so you can build it again

Two operational habits turn all of this from a heroic one-off into infrastructure:

Scripts are idempotent, and they validate before they commit. Every sudoers change in the runbook is written to a temp file, checked with visudo -cf, installed, checked again in place, and rolled back automatically if either check fails. Running the script twice is safe; running it on a machine built by an older edition is safe. That’s what lets you retrofit a control onto a server from last quarter without a rescue-boot afternoon.

The server knows which runbook built it. A one-line marker file — /etc/build-runbook-version — written at build time, plus a changelog in the runbook itself. When you fix a bad audit rule (see above), you can answer “which servers still have the old one?” in a one-line ssh loop instead of a guess.

The habit, restated

None of this is exotic. Root is locked and identity lives in one place; privilege is attributed rather than forbidden; sessions are recorded with a documented retention; logs are capped locally, forwarded immediately, and buffered for the outage. Every one of those is a decision you could make this week on the servers you already have — and the next time someone asks who ran the command that broke the database last Thursday, you’ll press play.