Toljari Cycle #1: Pypi Packaging Party

Despite the date on the post, the Toljari Cycle started last week, not this weekend, but I didn’t know it at the time.

Storytime:

I thought I had enough time to get my ducks in order when I decided that to hell with it, I would kick off this regular release schedule one and a half weeks ago. However, it turns out that what I was originally planning to do for my first coding release was pretty difficult, but I only realized this after I pushed the deadline to halfway into the week after when the release was due. So much for starting off strong. So now I’m just taking something else I’ve done codewise and plopping it in the release box, and hoping no one notices.

So, what’s in the release? Just two of my python libraries, base92 and encryptedfile, that I’ve finally packaged for pypi, so that now one can run pip install base92 or pip install encryptedfile and run code I’ve written. Squee. Going through this process did open my eyes to the fact that pip just plain does nothing to validate packages, though, which is a problem. Why? Well, when the “moar encryption” side of my mind meets the python-lovin’ side, there might be some energy release. Hopefully enough energy to fix pip package signing, or even more hopefully enough energy to get someone else to do it.

But yeah, getting a toehold in the python cheeseshop! It’ll have to be good enough for the next 3 weeks, so you take this release and you like it!

Check your Assumptions

NOTE: yes, I’m getting around to this 2 weeks after the fact. I have long-lived drafts, okay?

If you haven’t been keeping up with tech news in a particularly frantic way, you might not have noticed that Google is changing their privacy policy. On one hand, this change will result in a somewhat longer policy, but without the fragmentation across all of Google’s services, so that one need not read 5 different policies to use 5 different Google services. On the other hand, this means that data is share-able across application boundaries, such that the bountiful web searches can be used to customize which youtube videos are shown to a person, and vice versa.

The EFF being who they are, issued an article alerting us to this state of events and how to ameliorate it, by deleting our web history. However, looking over it, I noticed that this is a ton of data. These records go all the way back to 2005; in other words, it’s a treasure trove of personal data. And you ought to know that I’m somewhat obsessed about personal metrics, and I’m not going to just straight-up delete perfectly good data if I can’t get my hands on it.

The problem is that Google didn’t offer a nice way to download the data: there’s an RSS feed, but that means stepping through an RSS feed manually, since I couldn’t just generate a feed with 29k searches in it: instead, I had to step through by 1000 items, and who would download 30 xml files? Obviously, this was a job for lazy-in-some-ways programmer.

There was some prior work, but I threw it out for dumb reasons (CSV?? Flash?? It’s like we’re back in the stone ages!) (Also, don’t tell curl I didn’t choose it) Plus, I wanted to learn how to work Chrome extensions, and by just thinking about it, it seemed like I could inject javascript into the history page, use AJAX calls to get the subsequent RSS feed files (since the injected javascript would side-step the same-origin policy), and save it using the HTML5 FileSystem API. But would this all work? Just because you work it out in your head doesn’t mean it should work, especially with technologies one is not familiar with.

After reading up on it, though, it still seemed feasible, and I spun up a git repo and started writing a Chrome extension. Then, I ran into a small snag: how should I make sure I don’t download and save more files than needed? So I tried to see if asking for an RSS feed too far into the past would just pass back a 404. Instead, I found that Google had somehow squashed my 29k searches into 5 XML files, instead of 30.

Oh.

Now, downloading 5 files does not sound like much work: neither does 30, in the grand scheme of things, but 30 somehow manages to be more fearsome with the mere addition of a brotherly digit. Cut down to size, I just downloaded and saved the XML files containing essentially my entire search history, packed into a little over a megabyte of data. Also, it was 30 minutes until midnight, and tarrying wouldn’t do me good (of course, the privacy policy might have already shifted, say, when midnight appeared over the International Date Line, but nevermind…) (also, I couldn’t find where Chromium actually, you know, saved the File data).

However, it might have balanced out to the best: I learned how to work basic Chromium extensions, and I also got my data down, safe and sound. I suppose there’s a moral in here somewhere… oh! In the title! Check your assumptions, fail fast, etc etc.

Methinks I have to work on my fable telling.

Why Hello

I just whipped up a quick flyer generator for the ADI. It kind of works-ish, definitely needs more work/documentation/a github repo, but it’s good enough for a minimal viable product.

Revision 2, the latest version, is over here.

Why’s it named lolhawk? Well, it makes flyers; the first flyer is arguably the Wright flyer, which was launched at Kitty Hawk; everyone knows kitties as lolcats, so if you spin everything around fast enough, you get lolhawk. Also, hawks are fliers, too ;P

EDIT: I still don’t have a computer to call my own, so that was especially painful to write. Ah well, put off homework for a bit, which lessens the net loss (or increases??! STAY TUNED FOR NEXT TIME WHE

EDIT 2: Fantastic, and now there’s a new HPMofR! I like to think it’s a reward for opening myself up to universal ridicule and scorn.

Farm Update

So, I’ve had some ups and downs in the 2 weeks since school ended. I’ll be focusing on one particular thing, which is the distributed computing cluster me and a friend set up to crunch through solutions to the optimization problem I’ve been looking at.

Well, you may be thinking, you have a computing cluster? In that case, man, you’ve got it made! The sky’s the limit when you’ve got a computing cluster! Well, maybe I’m exaggerating, but having a whole lot of computing power behind you makes solving certain problems easier or just plain out possible. In this case, having a bunch of cpu time crunching on this optimization problem makes it tangible study in depth.

So, a certain very good friend of mine heard that I was looking for computing time, and since he worked for a branch of his university’s IT department, he knew that a certain computer lab would go unused for most of the summer. So I whipped up some customized live cd’s that would be devoted to solving these problems, along with a distributed computing system to tie everything together (part of why I got this domain in short notice) and we filled up the lab with systems running off the live cds. For a few short glorious days, I had over 100 cores crunching numbers for me: it’s a great feeling to wake up and have a job that would’ve taken a month with one dual core computer finish overnight.

Then, some of the other IT guys stumbled into the lab looking to play some games, and failing to closely read the friendly note I had left on every computer in the lab, proceeded to freak out and raise a pretty good shit storm over it. Looking back, we could’ve prepared better for this, by at least telling the IT people with access to the closed down lab that we were up to something in there. In our defense, though, we didn’t think that people would be dumb enough to freak out about it (which was a mistake on our part: sometimes, you just have to assume that people are insane. To be honest, I’m glad I’m learning it now rather than later). At one point we were afraid that my friend would get fired, but all they did was close down the majority of the cluster, and now I’m back to having some 20 odd cores at my disposal. It’ll get done eventually, but it’ll take a bit longer.

So what’s the point of this little story, other than you should prepare for unforeseen consequences? There isn’t one, but it got me thinking about something that I’ve been wondering about. Your usual university might have a dedicated research cluster, but most of the smaller ones don’t have a one. However, all universities (the ones I’m familiar with, at any rate) have at least a few computer labs, all with non-ancient hardware. Most people use those computers at sane hours, or if at insane hours, then they’re usually there by the time insane hours start.

I’ve never gotten why no one has tried to tie together all the labs as a giant cluster, but one that only functions at night. If there aren’t any jobs, then the cluster shuts down like it’s supposed to: if there are jobs, then they crunch through them until it’s time for the lab to open, and then the cluster shuts down and becomes a normal terminal-ish thing. That way, smaller universities have an appreciable amount of computing power on hand, in house. That would be cool.

There’s plenty of  objections to this sort of scheme, with security, cost, and maintenance topping the list of concerns that I can make off the top of my head. However, so far, it’s just an idea I’m tossing out there, mind candy to kick around and think about when you can’t sleep. Like all the other ideas bouncing around in my head, we’ll see where it goes.

Optimization of a Color Cycler

One of my projects in progress depends on the generation of different colored objects (Trust me, once I actually get a release pushed out, it will make sense (actually, I also need to push out a new iteration of my LaTeX editor. Note that I’m putting the name up for reconsideration)). First, I’ll define HSV space for y’all: assuming that you are familiar with RGB space… or, RGB space is the definition of colors by red, green, and blue components (hence the acronym RGB). Usually, computers will use 8 bits to define each channel, for a total of 24 bits to define a pixel. But I digress. HSV uses hue, saturation, and value for it’s definition, hue being the color (say, orange versus blue), saturation the presence of that color (ex. 0 saturation is white, total saturation a bloody red), and value the brightness (ex. 0 value is black, high value something not-black). In this particular case, I’m using HSV since I can set the saturation low, value high, and change the hue for a rainbow of easy-on-the-eye colors. However, it is problematic if one can’t distinguish between two objects that are supposed to be different.

One strategy that comes to mind is using some sort of binary division (for illustrative purposes, I’ll take hue 360 = hue 0): Take the first value to be 0, then 180, then 270, then 90, etc. However, it isn’t dead easy to code, and another programming scheme that could work is merely adding a constant to each iteration and modulating it by 360 (ex. 0+105->105+105->210+105->(315+105)%360->60 (where % is the modulus, otherwise known as the remainder)), which is dead easy to code. However, adding a constant could result in trouble: if you choose 90, you’ll start getting duplicate colors after 4 steps. 91 isn’t much better, because while the colors are not exactly the same, they are pretty close, perhaps nigh indistinguishable. How do you choose the constant that will put off disaster for the longest time?

Guess and check is a time-honored method, and it yields okay reults: I made up a constant and used to to almost no ill-effect. However, one could do better. Checking each constant by hand is tedious, but only by hand: if you brew up a script, then its turns out to be a bit simpler.

For a naive script, we do iterations until we get a duplicate of a previous iteration: the number of iterations before termination is a measure of quality, where higher numbers are better. Note, as a curiosity, that if the bound were some prime number, rather than 360, then many constants will turn out to be the maximum number of iterations with this method. However, we still need to take into account how close the subsequent hues are (for instance, 1,2,…365 works, but it really doesn’t).

To work around this, I used a sort of logistic function to decrease the amount of influence later numbers have, since the application doesn’t expect someone to be using 50+ objects at once. To do this, I weighted the hue-distance between the nearest colors (for each color added: I didn’t go back and recalculate all the distances for each iterations, although that might give us a different answer) by the reverse logistic function translated so that the first 20 colors had a pretty big say relative to everything else. Ah, well, here, have some code.

[python] #colorlen.py import math

n = 360

def modlen(a,b,limit): nominal = abs(b-a) h = (limit-b)+a L = (limit-a)+b return min([nominal,h,L])

def fitness(i,prev): return sum([1.0/(1.0*modlen(x,i,n)**1.0) for x in prev])

def total_fitness(m): i=m f=0.0 steps=0 prev = [0] while not(i in prev) and steps<50: f += fitness(i,prev)1.0/(1+math.exp(0.01(steps-10))) prev.append(i) i = (i+m)%n steps +=1 return f-steps

def main(): lst = [total_fitness(m+1) for m in range(n-2)] best = lst.index(min(lst)) return best+1

print main() [/python]

While writing this up, I’ve realized that I could do some more work on this and have some pretty graphics to show to everyone. Hmm, I’ll have to fit this in somewhere…

Oh, and hope this was helpful to someone. Dunno, go and write code fruitfully?

Midweek update - before the storm

Hi. The MCM starts in 2 days.

I have to finish a paper and a bunch of lab questions before then, and I want to do at least one run through with my talk on maglev trains (as well as hacking S5 to auto-advance ala ignite), run at least one reprap part into existence, find if we can do etching cheaply, start the E/M test, and talk to a prof about what sort of equation I should use in a feedback loop.

Whew.

So with all of this to do, I took 3.5 hours off and coded up a framework for a problem posed by my prof that could have been a potential summer research topic: looking at the bottleneck assignment problem, what’s the relationship of n and the probability distribution used to assign weights to the assignments? If you are terribly lost, and I manage to do some work on this after the MCM, then I’ll post an update. Point is, I’m code deprived or something, and I just finished something I thought would take me at least a week, which shows me just how good my estimation abilities are (off).

And updating wordpress isn’t such a lark: I gotta do backups? Hell, gotta wrap this stuff up into a script

Printing Envelopes with Cups and the P1006 [Redacted]

Although it’s not technically a hack, I put some good minutes and iterations into making this work, so here’s a system I cobbled together one late night, when the rent was due and my handwriting bad.

Doing envelopes is not especially straightforward under Linux (as of yet). Open office does have an envelope wizard, but one has to play with the page/printer settings until it prints in the right direction, and then it prints down the middle of the page. Oh, and the HP P1006 does not take a single envelope: it won’t see a full page, and will refuse to print.

To get the printer to take an envelope, you have to combo it with another, full sheet of paper. Merely laying the envelope on the sheet has mixed results: it definitely works, but the envelope tends to move and the addresses come out strangely. Tape may work, but I’m not sure how much abuse the printer will take (I’m guessing a whole lot, but don’t fix what ain’t broke? (Which is actually a bad sentiment: I’m guessing real scientists would look to break things)).

To work around the fact that it seemed nothing could print an envelope correctly, I finally went back to the CLI, and found a new command: lp. This guy prints stuff from the command line, with arguments that control where it prints and how it positions the printing document. Read the man pages for more: the final command I ended up with this

#!/bin/sh
lp -d HP_LaserJet_P1006 -o position=right -o landscape -o com10 $1

Which reminds me, I really need to grab a code formatter for wordpress. Tally ho!

Anyways, save that into a file, chmod it so you can execute it, and pass it the name of the PDF. You can make a envelope PDF with open office just fine.

Hope this helped any poor souls out there.

EDIT: I stand corrected: Openoffice will print in the right orientation, you just have to browse around in the options tab next that is discretely hidden in the envelope wizard. I also found that you do not need a sheet of paper to act as a sabot for the printer. facepalm