Getting More Out Of SSH
04 Jun 2015
If you’ve been working with Linux servers for any length of time then you’ve used SSH to connect to a server. Though
the way you use SSH and the features you use may vary, SSH is incredibly powerful and has a lot of features the average
user may not be aware of, or use. I don’t want to say this is a post about power user features, or a list of hidden
tips and tricks; the more you use something, the more you understand it, and more efficiently use it. This post is just
a collection of features I use that others I’ve seen don’t, and may prove useful to others.
Getting Started Fundamentals
The first few sections below outline how to generate a key pair and give a high level overview of what public key
cryptography means in this context.
I’d imagine you probably already use keys rather than plain password authentication. If you don’t, you should. And
First, you need to generate a new key pair. A key pair consists of a public key and a private key. The private key stays
on your client machine. The public key is added to any server that you wish to access. A passphrase is used to secure
the key pair, but successful authentication also depends upon using that password together with your local private key
and the presence of your public key on the target server.
To generate your key pair, you can run
This will create two files, by default, in your
` to the `ssh-keygen` command above. This will create `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub` as your private
and public keys respectively.
Now, you need to append the contents of `~/.ssh/id_rsa.pub` to `~/.ssh/authorized_keys` on the remote server.
cat /.ssh/id_rsa.pub | ssh user@remotehost 'cat >> /.ssh/authorized_keys'
Now, when you try to connect to the remote host, you won’t be prompted for your password. You may be prompted for the
passphrase to unlock the key, but you should only need to do that once (or e.g. whenever you restart your computer).
### SSH Agent
Most Operating Systems allow you to use an *agent* program with SSH. On Linux systems, this is typically `ssh-agent`. On
windows, you can use Pageant with Putty. An SSH agent securely stored unlocked private keys in memory for the duration
of a session, to avoid the need to always specify commonly used keys when using SSH. Note that SSH will attempt each key
stored in the agent. If all of the keys are exhausted unsuccessfully the SSH server may return an authentication failure
to avoid brute force attacks with many different keys.
### Private Keys Are Private
By way of an analogy, imagine you are Alice, and Bob wants to allow you access to his server. Key based authentication
is like opening a door that’s padlocked. Bob could send Alice a key to unlock the padlock and gain access (a private
key), but that means the private key is out in the open while it’s being sent, for any nefarious party to grab. Instead,
imagine the door can be unlocked by Alice giving Bob her padlock (the public key); and that anybody with a key to their
own padlock can unlock the door. In this way, you can give padlocks (public keys) to anybody who wants to give you
access to their server, but you keep the key (private key) to yourself.
For this reason, the private key should be kept very secure. For all intents and purposes, think of it as your password.
You don’t just hand that out over email, right!
### Diagnosing Connection Issues
If you’re having trouble connecting to a remote machine, and you’re not sure why, then you can increase the verbosity of
the SSH command line. By default, adding `-v` to your SSH command will make things *a bit more verbose*. You can get
ever more verbose output with `-vv` or `-vvv`. The verbose output can often quickly point you in the right direction.
### Every Day Usage
Connect to `remotehost` as the same user as your local user, optionally using non-standard port `port`.
ssh remotehost [-p port]
Connect to `remotehost` as user `user`, optionally using non-standard port `port`.
ssh user@remotehost [-p port]
Connect to `remotehost` as the same user as your local user, using the private key `/path/to/private-key`, optionally
using non-standard port `port`.
ssh -i /path/to/private-key remotehost [-p port]
Connect to ‘remotest’ as the user `user`, using the private key `/path/to/private-key`, optionally using the
non-standard port `port`.
ssh -i /path/to/private-key user@remotehost [-p port]
Note that if you don’t specify a private key, prior to attempting password authentication, an attempt will be made to
use key-based authentication using any keys in your `ssh-agent`. This will typically contain any default keys such as
`id_rsa` or `id_dsa`. You may need to unlock the key the first time you use it in a given session.
SSH can be used for so much more than just connecting to a shell session on a remote server. If you have access to a remote server, you can do all manner of useful things involving port forwarding and tunnelling.
### Local Port Forwarding
If you need access to a port on a remote server that isn’t publicly exposed, for example a remote MySQL server that’s
only listening to localhost, you can forward a local port to a remote port.
ssh -f -N -L 4406:127.0.0.1:3306 user@remotehost
This command will forward the local port 4406 to the remote port 3306. The `-f` flag puts the SSH command into the
background; the `-N` option makes sure that no remote commands are executed; the `-L` flag is used to forward a local
port to a port on a remote host, rather than forwarding traffic from a remote port to the local host.
Note that you can also use other options, as described earlier, such as specifying a private key with `-i` or a
non-standard port with `-p`.
You could use the same local and remote port, provided you don’t have a local MySQL server that’s already listening on
Note that you don’t have to use localhost. In fact, you can forward any service that the remote machine you have access
to, can see. If the MySQL database was hosted on a completely private machine that `remotehost` can route to and access,
called `db-remotehost`, you could do the following.
ssh -f -N -L 4406:db-remotehost:3306 user@remotehost
### Remote Port Forwarding
Remote forwarding is useful in situations where you and another party both have access to a common machine, and you
would like to use that machine to offer access to the other party, to a service running on your local machine.
In this example, you might have a local web server listening on port 8080, and you’d like to allow a third part to
access that local web server. You’d like the third party to be able to browse to the IP address of the common shared
server, on port 9090, and have them access your local web server on port 8080. To do so, you’d use a command like the
ssh -f -N -R 9090:localhost:8080 user@remotehost
Note that in this example, we use the `-R` flag for remote forwarding, rather than the `-L` flag we used earlier. Also
note that the order of the ports is reversed when compared to the earlier example: the remote port comes first, and the
local port comes second.
In order for this to work, you also need to enable the `GatewayPorts` option on the SSH daemon on the remote host.
You’ll also need to ensure that any firewall running on the remote host permits access to port 9090.
## SSH Config
All of the examples given thus far should prove useful, but may become a pain to use regularly, especially if you are
administering a large number of Linux servers. To ease this pain, you can use an SSH config file to define common host
aliases and sets of options. On a per-user basis, the config file is typically located at `~/.ssh/config`.
The file is arranged in terms of a series of `Host` blocks. Each definition continues until the next `Host` block.
Wildcards can also be used to broaden the scope of a match. The configuration file is parsed from top to bottom, looking
for hosts that match the host used in the SSH command you run. The matching doesn’t immediately stop at the first match.
Any later matches in the file can add to options previously defined, but only for options that have not yet been defined
If we want to connect as user `bob` to `remotehost1.example.com` on port 2222 with a private key named `bob-key`, we
could use a command line such as the following:
ssh -i /path/to/bob-key -p 2222 firstname.lastname@example.org
We could also use the SSH config option names rather than the SSH command line options:
ssh -o "IdentityFile /path/to/bob-key" -o "Port 2222" -o "User bob" -o "HostName remotehost1.example.com"
But this would get pretty tiresome to type all the time. It would be nice if we could instead just type `ssh
remotehost1`. We can achieve this by adding the following block to `~/.ssh/config`.
Because of the ordering described earlier, you can construct your file with less specific, shared options, toward the
end of the file. You should think of such entries as “catch all” rather than “default” and always ensure they come last.
For example, I use the following sensible options that will apply to all connections, unless override with more specific
options earlier in the file.
### Proxy Commands
You may require access to a large number of private servers, all of which can be accessed via an intermediary bastion
host. However, you might not want to send your agent everywhere. In this case, you can use a `ProxyCommand` option in
your SSH config file. In the example below, we use the netcat in a `ProxyCommand` to effectively tunnel onward to the
target private host.
With this in place, we can connect to e.g. `db.private.example.com` or `www2.private.example.com` and SSH will
transparently connect firstly to the bastion host, and then set up a netcat tunnel to pass the SSH connection through to
the destination host.
ProxyCommand ssh bastion.example.com nc -q0 %h %p
### Removing A Bad Host
If you try to connect to a host, and the server signature differs to that recorded in the known hosts file, you may need
to remove the offending signature from the known hosts file. You should do this with caution, and only when you know the
server you are connecting to is legitimate: for example if you’ve relaunched an AWS EC2 instance with the same hostname.
You can either use `ssh-keygen`:
ssh-keygen -R hostname
Or you can use `sed`, since the error message will give the line number of the offending known hosts entry (line 25 in
sed -i '25d' ~/.ssh/known_hosts
### Forcing Password Login
If you are working with a host that you know only accepts password authentication, and you want to quickly avoid the
overhead of trying keys, you can use a command line such as the following.
ssh -o "PreferredAuthentications password" -o "PubkeyAuthentication no" user@remotehost
### Skipping Known Hosts Checking
If you work with e.g. Amazon Web Services, or otherwise interact with a lot of short-lived and disposable or ephemeral
servers, you may want to skip checking the known hosts file, since you expect server signatures to change for a given
host quite frequently. This should be used with caution, and you wouldn’t want to globally take this step for security
ssh -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" user@remotehost
This command will automatically add the host to the known hosts file, but also use /dev/null as that file.
### Check If Host Key Exists
You can check for the presence of a given host key in your known hosts file with the following:
ssh-keygen -F hostname-or-ip
### Diff Local And Remote
You can quickly diff a remote file with your local copy with either of the following two examples.
ssh user@remotehost "cat remote_file.txt" | diff - remote_file.txt
diff local_file.txt <$(ssh user@remotehost 'cat remote_file.txt')
### Pseudo TTY Allocation
You’ll find some commands, such as `sudo`, will give you an error when you try to run the command on a remote server via
SSH. You may see an error about being unable to allocate a TTY. To overcome this issue, you use the `-t` flag to force
allocation of a pseudo-tty.
ssh -t user@remotehost sudo cat /etc/passwd
### Skipping Lastlog
Similarly to the above example, you can disable allocation of a pseudo-tty with the `-T` flag. This will mean your
session doesn’t appear in the output of `who` or `lastlog`.
ssh -T user@remotehost
### SSH Escape Sequences
You’ve no doubt used telnet before, to diagnose network connectivity issues; to do some basic interaction with a web
serve or an MTA etc. If you have, you’ll probably be aware of the `^]` escape sequence to hang up the session and drop
you to the telnet shell. From there you can `^d` to get back to your regular shell. SSH comes with similar functionality
via the `~` escape sequence. If you’re connected to a machine with SSH, you can type `~?` to see a list of
supported options. You’ll most often use `~.` to kill a hung SSH session.
Supported escape sequences:
~. - terminate connection (and any multiplexed sessions)
~B - send a BREAK to the remote system
~C - open a command line
~R - request rekey
~V/v - decrease/increase verbosity (LogLevel)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice
~/.ssh/ directory. You can change the location by adding a `-f