atp

I promised matt carter I'd start writing up aspects of how we run agile operations (aka devops) at LMAX. Apologies this is so overdue.

One of the major tensions in any small techical operations team is the tension between project work and "interrupt driven" work. Interrupt work is hard to define, other than the slightly circular "being anything that is not project work". In practise this ranges from problems in production through to getting users working mice.

Small IT teams are usually pretty bad at dealing with interruptions, so we've developed the concept of an "Interruption Monkey" to keep things manageable.

Like most people I've got an amazon ec2 cloud instance for all sorts of things. Mine is used as a DNS secondary.

I'm currently fed up with the gyrations that running a proper dns secondary makes me go through. It was not complex until I introduced views. At that point you find that the replication of DNS zone data starts becoming mixed up with the acls on the zone views themselves, forcing you to use multiple IP addresses to replicate both internal and external views of the DNS. Thats painful, and hard to maintain.

When I added the EC2 instance, I also had the problem of dealing with replicating across the internet, for which the usual remedy is to use a VPN. However I found it far simpler to use rsync and make each one believe it is the master for the zones. Its easiest to rsync as the named user, or root itself. However, you don't really want to let privileged users accept stuff from the internet directly, and turning PermitRootLogins on is anathema.

So this post is about how to use ssh forced-command-only and the from options to lock down rsync backups,  as I had to assemble this from bits of blog posts and man pages, and I'll probably need it again in the future.

In this article I run through how to create a simple ticker app off the LMAX API using PHP.

LMAX is the London Multi Asset Exchange. This is a low latency financial exchange which can be accessed directly by retail customers over the web using JSON or XML over a REST based interface.

The raw protocol not that pretty, so there are client libraries written in C# and Java. For fun and monitoring, I have written an unofficial PHP client library which I'm using here.

Its long been our plan to move to a better network filesystem than NFS - ideally a distributed one so that we can get rid of our current NFS+DRBD+Pacemaker storage servers, and move to something a bit more integrated and active/active.

Looking at our requirements the options are; Lustre, Gluster or MooseFS.  There are lots of other good candidates out there; Pomegranate, Mogile, Sector/Sphere, Ceph etc... See the wikipedia page on distributed filesystems for more detail.

Here are our first impressions, and some basic performance data comparisons.

In a previous post I looked at the lowest practical resolution using the rdtsc instruction call.

What about more common or garden ways like clock_gettime() or even the humble gettimeofday()?

Breakfast madeleines - what could be sweeter? Ok, maybe getting tgtadm to actually remove an iscsi target on the fly successfully. The suggestion was that they're best eaten within the hour. They didn't last 30 minutes.

I needed to have a cell have a yellow or red background colour based on its contents. I also needed a second cell to go yellow or red, depending on whether the first was yellow or red.

This is an enriched dough. The rolls are light and fluffy and slightly sweet. They are best eaten warm.