atp

Like most people I've got an amazon ec2 cloud instance for all sorts of things. Mine is used as a DNS secondary.

I'm currently fed up with the gyrations that running a proper dns secondary makes me go through. It was not complex until I introduced views. At that point you find that the replication of DNS zone data starts becoming mixed up with the acls on the zone views themselves, forcing you to use multiple IP addresses to replicate both internal and external views of the DNS. Thats painful, and hard to maintain.

When I added the EC2 instance, I also had the problem of dealing with replicating across the internet, for which the usual remedy is to use a VPN. However I found it far simpler to use rsync and make each one believe it is the master for the zones. Its easiest to rsync as the named user, or root itself. However, you don't really want to let privileged users accept stuff from the internet directly, and turning PermitRootLogins on is anathema.

So this post is about how to use ssh forced-command-only and the from options to lock down rsync backups,  as I had to assemble this from bits of blog posts and man pages, and I'll probably need it again in the future.

Firstly, we're going to set up the EC2 instance (B, IP: 4.3.2.1) as the server. The goal is that it will only accept connections, as root, from the hosted system (A, IP 1.2.3.4) which is the client. It will only run rsync, to an existing directory under /var/named.

SSHD Setup

Firstly for sshd on the EC2 server B we need to ensure that PermitRootLogin is set correctly. If you're using the amazon version of linux, you'll get this turned on by default;

[root@ip-10-56-59-200 bin]# grep PermitRoot /etc/ssh/sshd_config
PermitRootLogin forced-commands-only
# "PermitRootLogin without-password". If you just want the PAM account and

Now we set up the authorized keys file for EC2 Server B to accept only incoming connections from A and to run a wrapper script.

from="1.2.3.4",
  command="/root/bin/backup_wrapper",
  no-port-forwarding,no-X11-forwarding,no-pty 
  ssh-rsa AAAA...verylongsshpublickey...LqyAsId8HpKUIAaw== root@hostedsystem-a

 Things to note;

• That should all be on one line. Its been line broken for clarity. But then if you're reading this, hopefully you already know that.
•  "from=" This is the IP address (or name) for the client. See the PATTERN section in man 5 ssh_config
• "command=" This is the wrapper we're going to run, instead of any command supplied by the client.
• "no-port-forwarding,..." These are options to disable unwanted functionality see [1]
• "ssh-rsa" - the normal ssh keytype
• "AAAA... " - the public key of the client  A.
• "root@hostedsystem-a" a comment that describes the key.

The next step is the wrapper.

Rsync Wrapper Script

This may be slightly over the top, but I believe in layered defence.

#!/bin/bash 
# rsync wrapper
#
command=`echo $SSH_ORIGINAL_COMMAND | cut -d' ' -f 1`
# check command is rsync
if [ "$command" != "rsync" ]
then
 exit 1;
fi

# extract the target directory
target=`echo $SSH_ORIGINAL_COMMAND | cut -d' ' -f 5`

# check its not empty
if [ "$target" == "" ]
then
 exit 2;
fi

#canonicalise - gets rid of symlinks and /var/named/../../etc/ssh
realpath=`readlink -f "$target"`;

# extract the start.
prefix=`expr substr $realpath 1 10`;

# we allow a free for all under /var/named
if [ "$prefix" != "/var/named" ]
then
 exit 3;
fi

# check its a directory that exists under /var/named
if [ ! -d "$target" ]
then
  exit 4;
fi

# run rsync
exec rsync --server -vlogDtprz . $target

Points of interest;

• Different return codes for different errors make debugging easier
• The equivalent of php's realpath() in bash was taken from [2]. Thanks for saving me some time. The main thing it's there for is directory traversal attacks.
• Even though sshd ignores the original command, it still stashes it in SSH_ORIGINAL_COMMAND which we can use to pull out the target directory. See [3] for other ideas as to how to use this.
• We allow a free for all under /var/named If you prefer its possible to modify the script to check the dirname of the realpath against a set of allowed directories. 

The alternative to this approach is to have a different public private key for each directory under /var/named  we want to rsync to. For each key, we'll also need a forced command in the servers authorized_keys for that directory, giving lots of keys and lines in the authorized_keys file. Which is messy.

Test with a command like this;

To fail;

ssh -v 4.3.2.1 rsync --server -vlogDtprz . /var/named/../../etc/
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 4.3.2.1 [4.3.2.1] port 22.
debug1: Connection established.
:
debug1: Exit status 3

To succeed;

ssh -v 4.3.2.1 rsync --server -vlogDtprz . /var/named/chroot/var/named/chroot/foo
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 46.137.27.241 [46.137.27.241] port 22.
debug1: Connection established.
:
debug1: Sending command: rsync --server -vlogDtprz . /var/named/chroot/foo
0x1b

Assuming that /var/named/chroot/foo exists on the target. The 0x1b is the character returned by rsync in server mode.

Summary

I'm sure there are vulnerabilities in this script/approach. At least one of the guys in the team will probably have a go at breaking it. There's a mars bar in it for you radek.

The inspiration for this approach was [3] - thank you.

--

[1] http://www.eng.cam.ac.uk/help/jpmg/ssh/authorized_keys_howto.html or "man sshd"

[2] http://andy.wordpress.com/2008/05/09/bash-equivalent-for-php-realpath/

[3] http://binblog.info/2008/10/20/openssh-going-flexible-with-forced-commands/