unix


  1. Get an EC2 account.
  2. Choose an international server zone from the drop down in upper right.
  3. Create a tiny instance, keys, etc.
  4. Edit security zone to allow connections from your network to the instance via some non-priv port, say, 1999.
  5. SSH to the new instance.
  6. Use ssh-keygen to generate a local keypair on the server & add the public key to ~/.ssh/authorized_keys
  7. Run: ssh -gND 1999 localhost
  8. Use your server instance’s IP and the port from steps 4 & 7 in your proxy settings.

If you’re trying to use the proxy from a Mac or iOS devices, create a .pac file that looks like this, replacing the IP_ADDRESS and PORT to match the server:

function FindProxyForURL(url, host) {
 return "SOCKS <IP_ADDRESS>:<PORT>";
}

Obviously, if you’d like to keep the international spooks out of your traffic, you could do step seven between your local machine and the EC2 instance and configure your client to proxy locally. The traffic would then be encrypted by SSH while traveling over FLAG Atlantic 1, but if that’s the problem you’re trying to solve TOR is probably a better solution.

BEWARE: this is a completely open proxy. Don’t leave it running. If you do, someone will find it and proxy their child porn through it and the NSA will find you.

Normally git uses your default keys for authentication – ~/.ssh/id_rsa or whatever. Sometimes though, you want to use some other key pair. The fix is simple, but not obviously documented in the few places I looked, so here you go:

Let’s say you would typically use something like this:

# git remote add origin git@git.foo.com:bar/baz.git

But you want to use ~/.ssh/my_other_key to authenticate.

In ~/.ssh/config add a block like this:

Host git-foo-com-other-key
  HostName git.foo.com
  IdentityFile ~/.ssh/my_other_key

Then, instead of the above git-remote, use this:

# git remote add origin git@git-foo-com-other-key:bar/baz.git

Now, git push origin will use the appropriate key (the corresponding public key is known to git.foo.com, right?)

So you have a running EC2 instance. It works great, except it’s one of the ephemeral, kill-it-and-you-lose-everything kind. An EBS-backed instance is the logical choice, so how do you convert it? Easy:

Step 1: Create the EBS volume

Just do it in the web interface. You could use the command-line tools, but why? While you’re there, attach it to your running EC2 instance, making note of the volume-id and device it’s connected to, eg: vol-abcd1234 and /dev/sdf

While you’re in the web interface, make a note of the ramdisk and kernel your running instance is using. This will be important later. They’ll be something like “ari-12345678” and “aki-abcdef12”, respectively.

Step 2: Sync your running instance

If you have things like mysql running, shut them down. It’ll save you hassles later. Then create a FS on your EBS volume:

# mkfs.ext3 /dev/sdf

Next, mount it:

# mkdir /mnt/ebs

# mount /dev/sdf /mnt/ebs

Now, use rsync to copy everything over to the EBS volume:

# rsync -a –delete –progress -x / /mnt/ebs

You won’t have /dev/sda2 for your /mnt partition on EBS, so you need to remove it from the copied fstab in /mnt/ebs/etc/fstab. Comment it out, remove it, whatever.

(Added 11/2010 thanks to Mark Smithson in the comments)

You may need to create some device files on the new EBS volume. If console, zero or null don’t exist in /mnt/ebs/dev, create them using some or all of these:

# MAKEDEV -d /mnt/ebs/dev -x console
# MAKEDEV -d /mnt/ebs/dev -x zero
# MAKEDEV -d /mnt/ebs/dev -x null

Unmount the EBS volume:

# umount /mnt/ebs

Step 3: Get your keys in order

You’ll need an EC2 X.509 cert and private key. You get these through the web interface’s “Security Credentials” area. This is NOT the private key you use to SSH into an instance. You can have as many as you want, just keep track of the private key because Amazon doesn’t keep it for you. If you lose it, it’s gone for good. Once you have the files, set some environment variables to make it easy:

# export EC2_CERT=`pwd`/cert-*.pem

# export EC2_PRIVATE_KEY=`pwd`/pk-*.pem

Step 4: Make your AMI

Now you can make a snapshot of your EBS volume. This is the basis of the AMI you’ll be creating. Whatever you copied to the EBS volume in step 2 will be there — user accounts, database data, etc. First, the snapshot (using the volume-id from step 1):

# ec2-create-snapshot vol-abcd1234

That’ll give you a snapshot-id back. You then need to wait for the snapshot to finish. Keep running this until it says it’s “completed”:

# ec2-describe-snapshots snap-1234abcd

Finally, you can register the snapshot as an AMI:

# ec2-register –snapshot snap-1234abcd –description “your description here” –name “something-significant-here” –ramdisk ari-12345678 –kernel aki-abcdef12

(The arguments to ec2-register should be normal Unix-style long options: “-“, “-“, “snapshot”; “-“, “-“, “kernel”. WordPress seems to be displaying those as an mdash instead. It needs to be a double-dash.)

Step 5: Launch!

At this point, you should see your EBS volume, the snapshot, and your AMI in their respective areas of the web interface. Launch an instance from the AMI and you’ll find it pretty much exactly where you left your original instance.

So you have a directory structure that contains all of the applications needed to run a set of servers, but no server needs the entire tree. This rsync configuration is the easiest way I can come up with to rsync only what is needed for a single server:

If /export/apps contains bin, lib, var, etc, share and sbin but you only want bin, lib, and one subdirectory of share rsynced, `hostname`-exclude-list.txt then looks like this:

+ /apps/bin/
+ /apps/lib/
+ /apps/share/vim/
- /apps/share/*
- /apps/*

Using this command, you can then sync the directories:

# rsync -avz --delete-excluded \
        --exclude-from=`hostname`-exclude-list.txt \
        /export/apps /local

With the --delete-excluded, if you change the exclude file, newly excluded entries will be removed from the destination. The copy will end up in /local/apps. More usefully, /export/apps could be on a remote server and the destination would be /export.