Getting More Out Of SSH

If you’ve been working with Linux servers for any length of time then you’ve used SSH to connect to a server. Though the way you use SSH and the features you use may vary, SSH is incredibly powerful and has a lot of features the average user may not be aware of, or use. I don’t want to say this is a post about power user features, or a list of hidden tips and tricks; the more you use something, the more you understand it, and more efficiently use it. This post is just a collection of features I use that others I’ve seen don’t, and may prove useful to others.

Getting Started Fundamentals

The first few sections below outline how to generate a key pair and give a high level overview of what public key cryptography means in this context.

Using Keys

I’d imagine you probably already use keys rather than plain password authentication. If you don’t, you should. And here’s how.

First, you need to generate a new key pair. A key pair consists of a public key and a private key. The private key stays on your client machine. The public key is added to any server that you wish to access. A passphrase is used to secure the key pair, but successful authentication also depends upon using that password together with your local private key and the presence of your public key on the target server.

To generate your key pair, you can run

ssh-keygen -t rsa

This will create two files, by default, in your ~/.ssh/ directory. You can change the location by adding a -f <pathname> to the ssh-keygen command above. This will create ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub as your private and public keys respectively.

Now, you need to append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys on the remote server.

cat /.ssh/id_rsa.pub | ssh user@remotehost 'cat >> /.ssh/authorized_keys'

Now, when you try to connect to the remote host, you won’t be prompted for your password. You may be prompted for the passphrase to unlock the key, but you should only need to do that once (or e.g. whenever you restart your computer).

SSH Agent

Most Operating Systems allow you to use an agent program with SSH. On Linux systems, this is typically ssh-agent. On windows, you can use Pageant with Putty. An SSH agent securely stored unlocked private keys in memory for the duration of a session, to avoid the need to always specify commonly used keys when using SSH. Note that SSH will attempt each key stored in the agent. If all of the keys are exhausted unsuccessfully the SSH server may return an authentication failure to avoid brute force attacks with many different keys.

Private Keys Are Private

By way of an analogy, imagine you are Alice, and Bob wants to allow you access to his server. Key based authentication is like opening a door that’s padlocked. Bob could send Alice a key to unlock the padlock and gain access (a private key), but that means the private key is out in the open while it’s being sent, for any nefarious party to grab. Instead, imagine the door can be unlocked by Alice giving Bob her padlock (the public key); and that anybody with a key to their own padlock can unlock the door. In this way, you can give padlocks (public keys) to anybody who wants to give you access to their server, but you keep the key (private key) to yourself.

For this reason, the private key should be kept very secure. For all intents and purposes, think of it as your password. You don’t just hand that out over email, right!

Diagnosing Connection Issues

If you’re having trouble connecting to a remote machine, and you’re not sure why, then you can increase the verbosity of the SSH command line. By default, adding -v to your SSH command will make things a bit more verbose. You can get ever more verbose output with -vv or -vvv. The verbose output can often quickly point you in the right direction.

Every Day Usage

Connect to remotehost as the same user as your local user, optionally using non-standard port port.

ssh remotehost [-p port]

Connect to remotehost as user user, optionally using non-standard port port.

ssh user@remotehost [-p port]

Connect to remotehost as the same user as your local user, using the private key /path/to/private-key, optionally using non-standard port port.

ssh -i /path/to/private-key remotehost [-p port]

Connect to ‘remotest’ as the user user, using the private key /path/to/private-key, optionally using the non-standard port port.

ssh -i /path/to/private-key user@remotehost [-p port]

Note that if you don’t specify a private key, prior to attempting password authentication, an attempt will be made to use key-based authentication using any keys in your ssh-agent. This will typically contain any default keys such as id_rsa or id_dsa. You may need to unlock the key the first time you use it in a given session.

Tunnelling

SSH can be used for so much more than just connecting to a shell session on a remote server. If you have access to a remote server, you can do all manner of useful things involving port forwarding and tunnelling.

Local Port Forwarding

If you need access to a port on a remote server that isn’t publicly exposed, for example a remote MySQL server that’s only listening to localhost, you can forward a local port to a remote port.

ssh -f -N -L 4406:127.0.0.1:3306 user@remotehost

This command will forward the local port 4406 to the remote port 3306. The -f flag puts the SSH command into the background; the -N option makes sure that no remote commands are executed; the -L flag is used to forward a local port to a port on a remote host, rather than forwarding traffic from a remote port to the local host.

Note that you can also use other options, as described earlier, such as specifying a private key with -i or a non-standard port with -p.

You could use the same local and remote port, provided you don’t have a local MySQL server that’s already listening on that port.

Note that you don’t have to use localhost. In fact, you can forward any service that the remote machine you have access to, can see. If the MySQL database was hosted on a completely private machine that remotehost can route to and access, called db-remotehost, you could do the following.

ssh -f -N -L 4406:db-remotehost:3306 user@remotehost

Remote Port Forwarding

Remote forwarding is useful in situations where you and another party both have access to a common machine, and you would like to use that machine to offer access to the other party, to a service running on your local machine.

In this example, you might have a local web server listening on port 8080, and you’d like to allow a third part to access that local web server. You’d like the third party to be able to browse to the IP address of the common shared server, on port 9090, and have them access your local web server on port 8080. To do so, you’d use a command like the following.

ssh -f -N -R 9090:localhost:8080 user@remotehost

Note that in this example, we use the -R flag for remote forwarding, rather than the -L flag we used earlier. Also note that the order of the ports is reversed when compared to the earlier example: the remote port comes first, and the local port comes second.

In order for this to work, you also need to enable the GatewayPorts option on the SSH daemon on the remote host. You’ll also need to ensure that any firewall running on the remote host permits access to port 9090.

SSH Config

All of the examples given thus far should prove useful, but may become a pain to use regularly, especially if you are administering a large number of Linux servers. To ease this pain, you can use an SSH config file to define common host aliases and sets of options. On a per-user basis, the config file is typically located at ~/.ssh/config.

The file is arranged in terms of a series of Host blocks. Each definition continues until the next Host block. Wildcards can also be used to broaden the scope of a match. The configuration file is parsed from top to bottom, looking for hosts that match the host used in the SSH command you run. The matching doesn’t immediately stop at the first match. Any later matches in the file can add to options previously defined, but only for options that have not yet been defined earlier.

If we want to connect as user bob to remotehost1.example.com on port 2222 with a private key named bob-key, we could use a command line such as the following:

ssh -i /path/to/bob-key -p 2222 bob@remotehost1.example.com

We could also use the SSH config option names rather than the SSH command line options:

ssh -o "IdentityFile /path/to/bob-key" -o "Port 2222" -o "User bob" -o "HostName remotehost1.example.com"

But this would get pretty tiresome to type all the time. It would be nice if we could instead just type ssh remotehost1. We can achieve this by adding the following block to ~/.ssh/config.

    Host remotehost1
      IdentityFile /path/to/bob-key
      Port 2222
      User bob
      HostName remotehost1.example.com

Because of the ordering described earlier, you can construct your file with less specific, shared options, toward the end of the file. You should think of such entries as “catch all” rather than “default” and always ensure they come last. For example, I use the following sensible options that will apply to all connections, unless override with more specific options earlier in the file.

    Host *
      identitiesonly yes
      cipher arcfour
      compression yes
      serveraliveinterval 60
      gssapiauthentication no

Proxy Commands

You may require access to a large number of private servers, all of which can be accessed via an intermediary bastion host. However, you might not want to send your agent everywhere. In this case, you can use a ProxyCommand option in your SSH config file. In the example below, we use the netcat in a ProxyCommand to effectively tunnel onward to the target private host.

With this in place, we can connect to e.g. db.private.example.com or www2.private.example.com and SSH will transparently connect firstly to the bastion host, and then set up a netcat tunnel to pass the SSH connection through to the destination host.

    Host bastion.example.com
      ProxyCommand none
    Host *.private.example.com
      ProxyCommand ssh bastion.example.com nc -q0 %h %p

Miscellaneous

Removing A Bad Host

If you try to connect to a host, and the server signature differs to that recorded in the known hosts file, you may need to remove the offending signature from the known hosts file. You should do this with caution, and only when you know the server you are connecting to is legitimate: for example if you’ve relaunched an AWS EC2 instance with the same hostname.

You can either use ssh-keygen:

ssh-keygen -R hostname

Or you can use sed, since the error message will give the line number of the offending known hosts entry (line 25 in this example):

sed -i '25d' ~/.ssh/known_hosts

Forcing Password Login

If you are working with a host that you know only accepts password authentication, and you want to quickly avoid the overhead of trying keys, you can use a command line such as the following.

ssh -o "PreferredAuthentications password" -o "PubkeyAuthentication no" user@remotehost

Skipping Known Hosts Checking

If you work with e.g. Amazon Web Services, or otherwise interact with a lot of short-lived and disposable or ephemeral servers, you may want to skip checking the known hosts file, since you expect server signatures to change for a given host quite frequently. This should be used with caution, and you wouldn’t want to globally take this step for security reasons.

ssh -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" user@remotehost

This command will automatically add the host to the known hosts file, but also use /dev/null as that file.

Check If Host Key Exists

You can check for the presence of a given host key in your known hosts file with the following:

ssh-keygen -F hostname-or-ip

Diff Local And Remote

You can quickly diff a remote file with your local copy with either of the following two examples.

ssh user@remotehost "cat remote_file.txt" | diff - remote_file.txt

diff local_file.txt <$(ssh user@remotehost 'cat remote_file.txt')

Pseudo TTY Allocation

You’ll find some commands, such as sudo, will give you an error when you try to run the command on a remote server via SSH. You may see an error about being unable to allocate a TTY. To overcome this issue, you use the -t flag to force allocation of a pseudo-tty.

ssh -t user@remotehost sudo cat /etc/passwd

Skipping Lastlog

Similarly to the above example, you can disable allocation of a pseudo-tty with the -T flag. This will mean your session doesn’t appear in the output of who or lastlog.

ssh -T user@remotehost

SSH Escape Sequences

You’ve no doubt used telnet before, to diagnose network connectivity issues; to do some basic interaction with a web serve or an MTA etc. If you have, you’ll probably be aware of the ^] escape sequence to hang up the session and drop you to the telnet shell. From there you can ^d to get back to your regular shell. SSH comes with similar functionality via the ~ escape sequence. If you’re connected to a machine with SSH, you can type ~?<enter> to see a list of supported options. You’ll most often use ~. to kill a hung SSH session.

    Supported escape sequences:
    ~.  - terminate connection (and any multiplexed sessions)
    ~B   - send a BREAK to the remote system
    ~C   - open a command line
    ~R   - request rekey
    ~V/v - decrease/increase verbosity (LogLevel)
    ~^Z  - suspend ssh
    ~#   - list forwarded connections
    ~&   - background ssh (when waiting for connections to terminate)
    ~?   - this message
    ~~  - send the escape character by typing it twice

Comments

comments powered by Disqus