← Back

2026

SSH and Key Management on Linux

SSH is the protocol that makes remote Linux administration possible. Here is how key-based authentication works, how to manage keys across sessions with an agent, how to configure the client for jump hosts and tunnels and how to harden the server.

SSH is the protocol you use to connect to remote machines. It encrypts the connection, authenticates both ends and gives you a shell session over a network that you cannot trust. Almost everything in Linux system administration that happens remotely happens over SSH.

Most people start with password authentication, run a few commands and move on. The deeper value of SSH shows up later: key-based authentication that removes passwords from the equation entirely, the config file that saves you from typing the same options on every connection, agent forwarding that lets your credentials travel through a bastion without landing on it and tunnels that route traffic through encrypted pipes to reach services that are not exposed to the internet.

This article covers all of that in a practical order. Key generation comes first because nothing else works well without keys in place.

How it works

SSH uses asymmetric cryptography to establish identity. You generate a key pair: a private key that never leaves your machine and a public key that you place on any server you want to reach. When you connect, the server generates a challenge encrypted with your public key. Only the holder of the matching private key can decrypt it. The server knows it is talking to you without either side sending a password.

The first time you connect to a host, SSH shows you its host key fingerprint and asks whether to trust it. If you say yes, the fingerprint is written to ~/.ssh/known_hosts. On subsequent connections, SSH checks the fingerprint against that record. A mismatch is a hard stop — it means either the host key changed legitimately after a reinstall, or something is intercepting the connection. Both require investigation before proceeding.

Never skip the fingerprint check

Setting StrictHostKeyChecking no in your SSH config suppresses the fingerprint verification prompt and automatically accepts any host key. This is common in automation scripts where prompts are not possible. On interactive sessions it is a habit worth avoiding. Accepting a changed fingerprint without verifying the reason is exactly the scenario that SSH host key checking exists to catch.

Generating keys

Key generation happens once per identity, not once per server. You generate a key pair locally, then copy the public key to every server you need to reach. The private key stays in ~/.ssh/ on your machine. Protect it with a passphrase.

ssh-keygen -t ed25519 -C "comment"

Generate an Ed25519 key pair. Ed25519 is the recommended algorithm. Small keys, fast operations and strong security. The comment field is written into the public key for identification.

ssh-keygen -t rsa -b 4096 -C "comment"

Generate a 4096-bit RSA key pair. RSA is still widely supported but Ed25519 is preferable for new keys on systems that support it.

ssh-keygen -p -f ~/.ssh/id_ed25519

Change the passphrase on an existing private key without regenerating it. You will be prompted for the current passphrase before setting the new one.

ssh-keygen -l -f ~/.ssh/id_ed25519.pub

Print the fingerprint of a public key. Fingerprints are how you verify a key's identity without comparing the full base64 blob.

ssh-keygen -R hostname

Remove all entries for a given hostname from the known_hosts file. Necessary when a host's key changes and SSH refuses to connect.

A passphrase encrypts the private key at rest. Without one, anyone who gets a copy of the file has full access to every server that trusts it. The tradeoff is that you have to enter the passphrase every time you use the key — unless you use an SSH agent, which handles that for you.

Deploying public keys

Authentication works by placing your public key in the ~/.ssh/authorized_keys file on the remote server. The file permissions matter: if it is group-writable SSH will refuse to use it. ssh-copy-id handles all of this correctly.

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host

Append your public key to the remote user's authorized_keys file. The cleanest way to set up key-based authentication. Handles permissions correctly.

cat ~/.ssh/id_ed25519.pub | ssh user@host 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys'

Manual equivalent of ssh-copy-id when that tool is not available. Creates the .ssh directory if it does not exist.

SSH agent

The SSH agent is a background process that holds your decrypted private keys in memory. You unlock a key once with your passphrase. Every subsequent SSH connection in the same session uses the key without asking again. On macOS the system keychain integrates with ssh-agent automatically. On Linux you typically start the agent yourself in your shell profile.

eval $(ssh-agent -s)

Start an SSH agent in the current shell. The eval sets the environment variables so ssh can find the agent socket.

ssh-add ~/.ssh/id_ed25519

Load a private key into the running agent. You will be prompted for the passphrase once. The agent holds the decrypted key in memory for the rest of the session.

ssh-add -l

List all keys currently loaded into the agent by their fingerprint.

ssh-add -D

Remove all keys from the agent. The agent process keeps running but holds nothing.

ssh-add -t 3600 ~/.ssh/id_ed25519

Add a key that expires from the agent after 3600 seconds. After that, the agent forgets it and you will need to re-authenticate.

Agent forwarding is a separate feature that allows a server you are connected to use your local agent for onward connections. When you SSH into a bastion and then SSH from the bastion to a production host, the bastion can use your local key without having a copy of it. Enable it with ForwardAgent yes in your SSH config or ssh -A on the command line. Be careful with which hosts you forward to — an administrator on the intermediate host can use your agent socket while your session is active.

Client commands

Beyond opening shells, the SSH client handles tunnels, jumps and proxied connections. Most of the complexity lives in the flags passed to the ssh command, though the SSH config file is a much cleaner place to put anything you use regularly.

ssh user@host

Open an interactive shell session on the remote host as the specified user. SSH will attempt password authentication if no key is configured.

ssh -i ~/.ssh/id_ed25519 user@host

Connect using a specific private key file rather than whatever ssh-agent or the default key paths offer.

ssh -p 2222 user@host

Connect to a non-standard port. The default port is 22. Some servers move SSH to a different port to reduce automated scanning noise.

ssh -L 8080:localhost:80 user@host

Forward local port 8080 to port 80 on the remote host. Traffic sent to localhost:8080 travels through the SSH tunnel and arrives at port 80 on the remote side.

ssh -R 9090:localhost:3000 user@host

Reverse tunnel. Port 9090 on the remote server forwards back to port 3000 on your local machine. Useful for exposing a local service to a remote system.

ssh -D 1080 user@host

Open a SOCKS proxy on local port 1080. All traffic sent through the proxy is routed via the SSH connection. Useful for tunnelling browser traffic through a trusted host.

ssh -N -f user@host -L 5432:db:5432

Start a tunnel in the background without opening a shell. -N means no remote command is executed. -f sends the process to the background after authentication.

ssh -J bastion user@target

Connect to a target host by jumping through a bastion. The connection to the target is proxied through the bastion server without storing credentials there.

The client config file

The SSH client config file lives at ~/.ssh/config. It lets you set options per host so that short names replace long connection strings. A jump host configured here means you type ssh prod-db rather than specifying the proxy, the user, the key and the port every time.

Per-host configuration
Host bastion
  HostName 203.0.113.10
  User deploy
  IdentityFile ~/.ssh/id_ed25519
  Port 2222

Host prod-db
  HostName 10.0.1.50
  User postgres
  ProxyJump bastion
  IdentityFile ~/.ssh/id_ed25519

Host blocks in ~/.ssh/config let you define aliases and settings per destination. ssh prod-db connects through the bastion automatically using the settings defined here.

Global defaults
Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519

The wildcard Host * block applies to all connections. ServerAliveInterval sends keepalive packets every 60 seconds to prevent idle sessions from timing out. AddKeysToAgent loads keys into ssh-agent automatically on first use.

Server configuration

The SSH server configuration lives at /etc/ssh/sshd_config. After changing it you need to reload the daemon with systemctl reload sshd for the changes to take effect. Always keep an active session open when making changes to this file so you can recover if you accidentally lock yourself out.

PermitRootLogin no

Disable direct root login over SSH. Administrators should authenticate as a regular user and escalate with sudo.

PasswordAuthentication no

Reject password-based authentication entirely. Only key-based logins are accepted. This is the single most effective change you can make to an SSH server.

PubkeyAuthentication yes

Explicitly enable public key authentication. This is the default but making it explicit avoids confusion when troubleshooting.

AllowUsers deploy admin

Restrict SSH access to a specific list of users. Any user not named here is denied before any other check runs.

Port 2222

Move SSH off port 22. This does not improve actual security against targeted attacks but it dramatically reduces noise from automated scanners.

MaxAuthTries 3

Limit the number of authentication attempts per connection. After three failures the connection is dropped.

LoginGraceTime 20

The server closes the connection if the user does not authenticate within 20 seconds. Reduces the window for slow brute-force attempts.

X11Forwarding no

Disable X11 forwarding unless your environment genuinely requires it. Leaving it enabled adds unnecessary attack surface.

The order of changes matters. Set up key-based authentication and verify that it works before you disable password authentication. Disabling passwords first and then discovering that your key is not trusted leaves you locked out with no clean recovery path other than console access.

Where it fits

SSH is the layer that everything else in remote Linux administration sits on top of. Ansible runs its tasks over SSH. Git pushes to remote repositories over SSH. Kubernetes operators SSH into nodes for maintenance. The jump host pattern that protects production networks from direct internet access is SSH configuration.

Key management becomes an operational concern as teams grow. Individual developers each have their own keys. Service accounts used by CI pipelines need their own dedicated keys that are rotated on a schedule. Bastion hosts need to trust the right set of keys without accumulating old ones that belong to people who no longer work there. Tools like HashiCorp Vault can issue short-lived SSH certificates instead of relying on static authorized_keys files, which removes the problem of stale access entirely.

For most people starting out, a well-managed key pair, a clean SSH config and a hardened sshd_config are enough to work safely and efficiently. The more advanced patterns — certificate authorities, short-lived credentials, bastion fleets — exist to solve the same problem at a scale where the simpler approach starts to break down.