ttool

Boy, is this project old: I’m writing up the documentation on this project in 2011, and I last worked on it in… 2008. My drive to document everything, though, is all-powerful, and I decided that even though I have probably forgotten many important details about this project, that this piece of code was too important in my development to ignore since it’s one of the very few pieces of code that is even still around from my high school years.

This was partly because I wasn’t coding that much, as many of the projects I wanted to do were out of my reach at the time, and I had that troublesome chicken-and-egg problem that plagues newbs everywhere: projects are necessary to become better, but you don’t know which ideas are potential projects, and which are just crock. Or, you don’t know how to convert ideas to projects. Or you can’t recognize project ideas. At any rate, working on projects is a difficult prospect, and only more difficult when lacking experience, so actually shipping code was a big deal.

Thinkquest

The motivation for this particular project grew out of my drive to compete in a programmatic arena: at the time, I hadn’t figured out I could find groups of geeks through the internet (or didn’t want to; I was still somewhat in denial about being a geek) but I wanted to put my programming chops to work. The one competition I felt competent enough to compete in was Thinkquest. The current incarnation of Thinquest has multiple competitions (websites, mobile, etc) at many levels of education (technically, I could still compete in the webapp arena), but back in 2007 there was only a website competition among three different grade tiers covering the elementary grades to high school graduates. Basically, the competition entailed making educational websites that were graded by style and content, and the winners got some prize and a chance to present their work.

Looking at the previous winning websites, it was clear to me that no one knew what they were doing, as I saw scant content being made subservient to pretty design, or globs of content without any structure to it, or a modicum of content that sprawled across amorphous pages (think old-school geocities: if you need an example from the past, you should totally browse through reocities. Chances are if you load up any random reocities page, it’ll be… interesting). I thought that I could do a whole lot better by combining lots of useful content with smashing design, and rounded up a couple of friends to help out (I know I didn’t want to admit it to myself, but I had terrible design. I think I knew this deep in my heart, though, and hence had to get help). Sam did design, and Zach and Lance and I did content. The product site itself focused on computer competency, trying to serve as a central resource that could educate computer novices that knew nothing about computers and bring them up to doing simple programming, while also retaining tutorials on a wide breadth of topics.

Since we were trying to deliver a website, the obvious thing to write these tutorials in was HTML. Writing out raw HTML wasn’t our idea of a good time, though, and the visual design was still in flux while we were writing the content such that changes in the overall design would mean having to go through each written tutorial and changing the necessary design elements by hand. As I had already read about how programmers are lazy, I decided that I would also be lazy, and head off hours of reformatting HTML by hand by building a friggin compiler. At the time, I didn’t realize that what I had been calling a compiler was actually a sort of reverse-templating system, but nevermind…

Now, I wrote the this ‘compiler’ (I’ll continue to refer to it as a compiler) in Common Lisp: I had also read about the 5 languages that hackers should learn, and rightly decided that in order to really familiarize myself with a language, I had to build a non-trivial project with it. Additionally, I bought into the claims that lisp was orders more powerful than any other language on that list, and lisp become an obvious choice.

The origins of the name sprung up from my infatuation with the demoscene: I had recently found the rockin’ 96kb game .kkrieger, where krieger was German for warrior, and built by farbrausch, one of the best demoscene groups at the time and my favorite to this day (a close second is Conspiracy, and they’ve actually put out things recently). Now, this compiler wasn’t anywhere near a game, but gosh darn it if it wasn’t a useful tool, and wanting to pay homage to my idols, I named it ttool.

In the end, we didn’t even get these tutorials into the competition for two years in a row: it turns out that writing a bunch of tutorials in a short amount of time is really hard, so much so I remember pulling all nighters both years to try and finish tutorials and push them to Sam to make sure the styles all fit and then make sure oh crap what do you mean we’re not registered why is the Thinkquest site lagging oh god oh god oh… well. It was fun while it lasted.

Internals

As all lispers know, there is no one standard lisp: like any other standard, there are all sorts of implementations and deviations. At the time, I was seriously considering both CLISP and SBCL with which to build ttool, and eventually ended up choosing SBCL (I’ll be ignoring our first attempt, in which I tried to use CLISP). I have no clue how I ended up deciding that, so I’m afraid I need to leave that probably amusing digression to the era of time travel. At least one feature of SBCL that was important was its ability to create a standalone executable from a lisp file, bundling the runtime and lisp-bytecode to allow random people to run SBCL programs without installing SBCL: my team members probably wouldn’t have minded installing SBCL, but I decided to be vicariously lazy and remove that particular trouble for them.

Basically, the compiler worked by reading in files that were formatted in a wiki-like syntax, and then emitted a file of HTML. That way, if Sam changed the design requirements that affected the content, then we could just re-run the compiler on the tutorial files, and a bunch of brand-new HTML would pop out.

More specifically, the compiler read in a character at a time and checked if some special character is at the beginning of the line, which denoted special constructs that mapped to different HTML structures. Some constructs like =header were similar to wiki-syntax, as well as the *list notation and the convention that an empty line separated paragraphs. Some stranger constructs included the |code block| notation, and :! to denote folding notes (folding done with javascript), and the convention that the first two lines of the file were the title and subtitle. There’s a full-er tutorial in the 2nd year files provided in the download section, in the lisp tutorial where I throw the source of ttool at unsuspecting readers and run away.

Looking through the code itself, it seems that I had something of an aversion to putting comments on their own line, which I now think is a very good idea (baring specific circumstances). I also used a macro in a single place, as an attempt to also support CLISP as well as SBCL. However, since I didn’t support CLISP anywhere else, the point was kind of moot.

My linked library of functions (nlib.lisp, for those of you keeping track) included more macros, but nothing ground breaking. And of course, it includes my attempt at an infix-math macro, which implemented a reader-macro that allowed code like #!(x+2+f(5)) (I only mention this because it’s said that every lisper eventually implements some form of infix notation in lisp, as a sort of rite of passage). Another thing I had picked up from lisp and seemed to subsequently forget was the use of closures: here, I used closures to implement counters for generating unique ids to implement folding divs.

Something I’m not so proud of was my tendency to hard code everything: this wasn’t such a big deal because most of the design changes were solely based on CSS which allowed just changing the external CSS file, but every change affecting the HTML output meant ‘recompiling’ the executable and redistributing it. And let me tell you, distributing ~25mb executables over a small pipe (I forget exactly how small, but this was in a non-cutting edge household in 2008) is non-trivial.

To reiterate, the concepts I used in this code base were not at all new or impressive, but the fact that it was essentially the only piece of code of mine that survived from high school has some personal significance.

Download

Oh, right, you actually want to see the code, not just hear me pontificate on my past. Excuse me while I go fetch it…

Ok, have a tarball of the source (7kB)! I touched only one line of code, changing (load “nlib.lisp”) from an absolute to a relative path, and otherwise left everything the same as it was back in 2008. You’ll need an SBCL: I tested this with version 1.0.45.0 under Ubuntu 10.10. To pack it into an executable, open up an SBCL instance, just execute:


(load "ttool.lisp")
(in-package :ttool)
(make-ttool)


Which will give you a ttool executable in the current directory, ready to use! If you want some examples, then you can download the working files from our first attempt at the competition here (677kB). Now, try running ttool on any of the .tt files from the command line, like:


./ttool beg/tut0.tt


This will output beg/tut0.html, which should be the same as the original beg/tut0.html that came with the tarball.

If you’re more interested in the content in the tutorials themselves, then you can also download the HTML files that comprised our 2nd attempt (17.2MB) before Sam got her hands on it to impose some pretty design onto it. These don’t have any associated .tt files, as they’ve been lost to posterity. However, the content in these tutorials should be much more complete than those from 2007.

Note: there aren’t many comments in the code, as this is before I learned that comments are gold in the programming world. Here be dragons: be safe and godspeed.