Farm Update

So, I’ve had some ups and downs in the 2 weeks since school ended. I’ll be focusing on one particular thing, which is the distributed computing cluster me and a friend set up to crunch through solutions to the optimization problem I’ve been looking at.

Well, you may be thinking, you have a computing cluster? In that case, man, you’ve got it made! The sky’s the limit when you’ve got a computing cluster! Well, maybe I’m exaggerating, but having a whole lot of computing power behind you makes solving certain problems easier or just plain out possible. In this case, having a bunch of cpu time crunching on this optimization problem makes it tangible study in depth.

So, a certain very good friend of mine heard that I was looking for computing time, and since he worked for a branch of his university’s IT department, he knew that a certain computer lab would go unused for most of the summer. So I whipped up some customized live cd’s that would be devoted to solving these problems, along with a distributed computing system to tie everything together (part of why I got this domain in short notice) and we filled up the lab with systems running off the live cds. For a few short glorious days, I had over 100 cores crunching numbers for me: it’s a great feeling to wake up and have a job that would’ve taken a month with one dual core computer finish overnight.

Then, some of the other IT guys stumbled into the lab looking to play some games, and failing to closely read the friendly note I had left on every computer in the lab, proceeded to freak out and raise a pretty good shit storm over it. Looking back, we could’ve prepared better for this, by at least telling the IT people with access to the closed down lab that we were up to something in there. In our defense, though, we didn’t think that people would be dumb enough to freak out about it (which was a mistake on our part: sometimes, you just have to assume that people are insane. To be honest, I’m glad I’m learning it now rather than later). At one point we were afraid that my friend would get fired, but all they did was close down the majority of the cluster, and now I’m back to having some 20 odd cores at my disposal. It’ll get done eventually, but it’ll take a bit longer.

So what’s the point of this little story, other than you should prepare for unforeseen consequences? There isn’t one, but it got me thinking about something that I’ve been wondering about. Your usual university might have a dedicated research cluster, but most of the smaller ones don’t have a one. However, all universities (the ones I’m familiar with, at any rate) have at least a few computer labs, all with non-ancient hardware. Most people use those computers at sane hours, or if at insane hours, then they’re usually there by the time insane hours start.

I’ve never gotten why no one has tried to tie together all the labs as a giant cluster, but one that only functions at night. If there aren’t any jobs, then the cluster shuts down like it’s supposed to: if there are jobs, then they crunch through them until it’s time for the lab to open, and then the cluster shuts down and becomes a normal terminal-ish thing. That way, smaller universities have an appreciable amount of computing power on hand, in house. That would be cool.

There’s plenty of  objections to this sort of scheme, with security, cost, and maintenance topping the list of concerns that I can make off the top of my head. However, so far, it’s just an idea I’m tossing out there, mind candy to kick around and think about when you can’t sleep. Like all the other ideas bouncing around in my head, we’ll see where it goes.