internet


  1. Get an EC2 account.
  2. Choose an international server zone from the drop down in upper right.
  3. Create a tiny instance, keys, etc.
  4. Edit security zone to allow connections from your network to the instance via some non-priv port, say, 1999.
  5. SSH to the new instance.
  6. Use ssh-keygen to generate a local keypair on the server & add the public key to ~/.ssh/authorized_keys
  7. Run: ssh -gND 1999 localhost
  8. Use your server instance’s IP and the port from steps 4 & 7 in your proxy settings.

If you’re trying to use the proxy from a Mac or iOS devices, create a .pac file that looks like this, replacing the IP_ADDRESS and PORT to match the server:

function FindProxyForURL(url, host) {
 return "SOCKS <IP_ADDRESS>:<PORT>";
}

Obviously, if you’d like to keep the international spooks out of your traffic, you could do step seven between your local machine and the EC2 instance and configure your client to proxy locally. The traffic would then be encrypted by SSH while traveling over FLAG Atlantic 1, but if that’s the problem you’re trying to solve TOR is probably a better solution.

BEWARE: this is a completely open proxy. Don’t leave it running. If you do, someone will find it and proxy their child porn through it and the NSA will find you.

Normally git uses your default keys for authentication – ~/.ssh/id_rsa or whatever. Sometimes though, you want to use some other key pair. The fix is simple, but not obviously documented in the few places I looked, so here you go:

Let’s say you would typically use something like this:

# git remote add origin git@git.foo.com:bar/baz.git

But you want to use ~/.ssh/my_other_key to authenticate.

In ~/.ssh/config add a block like this:

Host git-foo-com-other-key
  HostName git.foo.com
  IdentityFile ~/.ssh/my_other_key

Then, instead of the above git-remote, use this:

# git remote add origin git@git-foo-com-other-key:bar/baz.git

Now, git push origin will use the appropriate key (the corresponding public key is known to git.foo.com, right?)

So you have a running EC2 instance. It works great, except it’s one of the ephemeral, kill-it-and-you-lose-everything kind. An EBS-backed instance is the logical choice, so how do you convert it? Easy:

Step 1: Create the EBS volume

Just do it in the web interface. You could use the command-line tools, but why? While you’re there, attach it to your running EC2 instance, making note of the volume-id and device it’s connected to, eg: vol-abcd1234 and /dev/sdf

While you’re in the web interface, make a note of the ramdisk and kernel your running instance is using. This will be important later. They’ll be something like “ari-12345678” and “aki-abcdef12”, respectively.

Step 2: Sync your running instance

If you have things like mysql running, shut them down. It’ll save you hassles later. Then create a FS on your EBS volume:

# mkfs.ext3 /dev/sdf

Next, mount it:

# mkdir /mnt/ebs

# mount /dev/sdf /mnt/ebs

Now, use rsync to copy everything over to the EBS volume:

# rsync -a –delete –progress -x / /mnt/ebs

You won’t have /dev/sda2 for your /mnt partition on EBS, so you need to remove it from the copied fstab in /mnt/ebs/etc/fstab. Comment it out, remove it, whatever.

(Added 11/2010 thanks to Mark Smithson in the comments)

You may need to create some device files on the new EBS volume. If console, zero or null don’t exist in /mnt/ebs/dev, create them using some or all of these:

# MAKEDEV -d /mnt/ebs/dev -x console
# MAKEDEV -d /mnt/ebs/dev -x zero
# MAKEDEV -d /mnt/ebs/dev -x null

Unmount the EBS volume:

# umount /mnt/ebs

Step 3: Get your keys in order

You’ll need an EC2 X.509 cert and private key. You get these through the web interface’s “Security Credentials” area. This is NOT the private key you use to SSH into an instance. You can have as many as you want, just keep track of the private key because Amazon doesn’t keep it for you. If you lose it, it’s gone for good. Once you have the files, set some environment variables to make it easy:

# export EC2_CERT=`pwd`/cert-*.pem

# export EC2_PRIVATE_KEY=`pwd`/pk-*.pem

Step 4: Make your AMI

Now you can make a snapshot of your EBS volume. This is the basis of the AMI you’ll be creating. Whatever you copied to the EBS volume in step 2 will be there — user accounts, database data, etc. First, the snapshot (using the volume-id from step 1):

# ec2-create-snapshot vol-abcd1234

That’ll give you a snapshot-id back. You then need to wait for the snapshot to finish. Keep running this until it says it’s “completed”:

# ec2-describe-snapshots snap-1234abcd

Finally, you can register the snapshot as an AMI:

# ec2-register –snapshot snap-1234abcd –description “your description here” –name “something-significant-here” –ramdisk ari-12345678 –kernel aki-abcdef12

(The arguments to ec2-register should be normal Unix-style long options: “-“, “-“, “snapshot”; “-“, “-“, “kernel”. WordPress seems to be displaying those as an mdash instead. It needs to be a double-dash.)

Step 5: Launch!

At this point, you should see your EBS volume, the snapshot, and your AMI in their respective areas of the web interface. Launch an instance from the AMI and you’ll find it pretty much exactly where you left your original instance.

Edward Tufte’s sparklines are a nice way to present small minimal graphs in contexts where larger, full charts are not needed or warranted. The growing support for the HTML <canvas> tag gives an opportunity to create these graphs dynamically in Javascript.

(Download yahoo-widget-sparkline-1.0b.zip.)

This widget was built against YUI version 2.7.0 but may work in other versions. It subclasses the “charts” module and is easily loaded via YUILoader. The test.html creates a auto-updating sparkline based on a simple FunctionDataSource within a paragraph of text.

Features:

  • If passed a height of “font-size” or “line-height” will cause the module to examine the CSS of the element to which it will be rendered to determine its height. This allows you to easily insert a sparkline into blocks of text without worrying about scaling the graph by hand.
  • By default it will remove all content within the element that will contain the <canvas>. This allows “alt=” type data to be displayed when <canvas> is not available.
  • Arbitrary highlighting of points with custom colors.
  • mean, median, standard deviation and arbitrary “normal” range shading.
  • Uses the Chart object’s polling setup to auto-update charts by re-polling the DataSource.

Usage:

Pull in the module:

var loader = new YAHOO.util.YUILoader();
loader.addModule({ name: 'sparklinecss', type: 'css',
                   fullpath: 'Sparkline.css'});
loader.addModule({ name: 'widget.Sparkline', type: 'js',
                   requires: ['charts', 'sparklinecss'],
                   fullpath: 'Sparkline.js'});
loader.require('widget.Sparkline', 'datasource', 'element');

Then create a sparkline:

var sl = new YAHOO.widget.Sparkline("sparkline-container",
                                    myDataSource,
                                    { height: 20, yField: "value" });
sl.subscribe("sparklineUpdate", updateHandler);

Configuration Options:

max
Force a maximum y-value. Default is the largest value in the data.
min
Force a minimum y-value. Default is the smallest value in the data.
width
Width of the graph. Default is one pixel per data point.
height
A pixel value or “font-size” or “line-height”. Default is 20 pixels.
logscale
Change the y-axis to a logarithmic scale.
clearParent
Remove any children of the node passed to render() before adding the canvas. Default true
zero, mean, median
Add horizontal lines at the appropriate places. Each can have a boolean true value or a string specifying the color for the line.
stddev
Calculate the standard deviation of the data and shade the background between +/- one standard deviation from the mean. Can be true or a color.
normal
An object with two or three fields: min, max, and (optionally) color. A background rectangle covering the range will be drawn in the specified (or default) color.
color
Graph color
highlightPoints
An array of two-field objects: x and color. x is the index of the point in data to highlight and color is the color to use. x-values of max and min highlight the maximum and minimum values. An x-value of -1 highlights the last point in the data.

To Do:

  1. Graceful failure when <canvas> is unavailable.
  2. Allow graphing of data extracted from HTML markup.

(updated 7/7/2009)

Yesterday at work someone was trying to pass traditional Apache SSI directives through an XSL transformation on a Google search appliance. Long story short, they vanished: HTML comments don’t make it out of that device.

Anyway…I had a simple solution. Since we were pumping the Google results through a Perl CGI anyway, there was no reason we couldn’t just output a fake HTML tag which the CGI would then turn into an SSI comment for Apache. This was born <ssi virtual="/foo/bar.html" />.

Then a simple s/<ssi (virtual=\"[^\"]+\")><\/ssi>/<!--#include $1 -->/; in Perl will give something Apache can understand.

That solved the immediate problem, but got me thinking about emulating the full Apache mod_include set of SSI directives using the <ssi> tag. I’m thinking of something like this:

<ssi element="include" virtual="/foo/bar.html" />

<ssi element="set" var="FOO" value="BAR" />

<ssi element="if">
  <ssi_if expr="test_condition">YES!</ssi_if>
  <ssi_elif expr="test_condition">MAYBE!</ssi_elif>
  <ssi_else>NO!</ssi_else>
</ssi>

And with that format, the original version still works if you assume a missing “element” attribute implies element="include". The <!--#if --> block isn’t quite satisfying here — any text nodes inside the <ssi> block but outside the <ssi_(if|elif|else)> blocks would be ignored, but that’s no different than odd content in, say, a <table> that doesn’t actually fall into a cell.

I don’t actually have the Perl that would do the transformation, but it wouldn’t be hard. I’ll wait until someone actually needs it.

Until about six months ago, I ran my own mail and web servers. It all ran on Linux with mirror hard drives and was fine except for the fan and the occasional problems with software versions and upgrades and shared libraries and spam — which is to say, it was a total pain.

When my last Linux box finally died, I bought a nice Dual Core iMac and decided, although I *could* run all that stuff on MacOS 10, I didn’t really feel like it anymore. I paid Pobox.com $62 to host my email and put a simple, static page on the Apache server that runs automatically on my iMac. But I still had a problem. I wanted some sort of site beyond what I was willing to install on my new desktop.

Web 2.0 has brought all sorts of community-enabled, RSS- and XML-accessible services. I upload photos to Flickr. I bookmark stuff using del.icio.us. I sync iTunes with Last.fm. My daily life is already web-enabled and feed-friendly. This got me thinking about why I didn’t want to put the effort into running a website for myself.

When it comes down to it, I don’t want to find a good way to upload pictures to my site because I already have a way I like: Flickr. I don’t want to create a special “link blog” sidebar on my site showing things I’ve read that other people might like, because I already have that: del.icio.us. My friends can already track all those things — if they know where to look.

Which leads me to the new Webkist.com. I still have my own domain. I want it to be a central place for finding my stuff: photos, music, whatever. But I’m not willing to manage all the tedious aspects of each datatype. Instead, I’m aggregating the data I create on *other* sites. Sites that take care of uploading and linking and categorizing and counting. I take the RSS feeds from each service, run it through a bit of XSL to create a simple block of HTML. I then combine each of those HTML files into a single page on webkist.com. It’s totally custom — I reformat the feeds however I like, and can style them simply using CSS. The result is pretty much what I would have created if I did it all from scratch, but without having to do ANY of it from scratch. I even used someone else’s XSL to start.

The last piece of the puzzle is this blog, right here. A simple, free wordpress blog, to handle whatever won’t fit in a del.icio.us bookmark description field. We’ll see if I use it.