Why I Love Python

Plenty of people, including die-hard lisp-heads and big names in the open source field, have already written about why they love Python. Google does most of their new development in Python, and it's cut their development cycles radically. And Python is the language behind BitTorrent, without which I wouldn't have been able to download replacements for all the CDs that have been stolen or broken over the years (not that I would ever do such an illegal thing). Any company that uses Java or C++ or .NET without thinking about Python first is giving a sure sign that they don't know what they're doing. (Of course I still end up working for them. Well, if they're going to throw money away, I'll take some of it.)

But forget about the technical merits of high-level languages, the beauty of indentation as structure, the interactive environment, etc. You can find all of that in other languages, but none of them are Python. Here are some of the less-commonly given reasons why Python rocks.

First, there's the name. It started off as a Monty Python reference, which is a great way to declare uber-geekiness while (to distinguish it from C++) at the same time not taking yourself too seriously. The only way to top that would be to name the language after a Hitchhikers or Discworld reference, like Slartibartfast or Rincewind. And using spam and eggs instead of foo and bar makes it impossible for any Python user to deny being a geek.

The fact that Python is also the name of a snake (not to mention the Israeli update to the Sidewinder missile) made it way too easy for O'Reilly to come up with the cover for their "Programming Python" book (unlike, say, caml, which is named for animal already in use by Programming Perl). However, now that Python is big news, the community has to worry about offending ophidophobes and Christians--so all of the old logos involving snakes and/or 16-ton weights (especially the MacPython logo, with a realistic snake wrapped around an apple) had to go.

This led to one of the most entertaining discussions I've ever seen, as hardcore geeks tried to figure out how to be politically correct. And the resulting logo is pretty cool, too.

In fact, the Python development lists are often entertaining to read, which I can't say about any other language.

Then there's the fact that Python's Beneficial Dictator For Life has the same name as Risky Business's Killer Pimp (played by Joe Pantoliano in what I think was his first major role). Yes, his name is Guido--and he's not even Italian; he's Dutch. How cool is that?

Python is also the only programming language with its own font (designed by Guido's brother Just).

Pythonistas (and isn't that much cooler than "C++ experts" in itself?) talk about the Zen of Python, and really mean it. Try typing "import this" into a Python interactive session.

Meanwhile, while C++'s big future plans are for a "200x" version, and Perl is looking ahead to a version 6 that may come out any decade now, Python is looking ahead to the year 3000. When we're all disembodied software agents, we will reprogram ourselves in Python.

Then again, what do I know; I used to code in Forth.

Python Macros

OK, I should post this on comp.lang.python, but I wanted to think it through first by posting it somewhere nobody's going to read. Also, I wanted something techy on my blog.

For the last 300 years or so, Python fans with lisp envy have been arguing that Python needs macros, while Python fans with lispophobia have been arguing that Python should avoid macros like the plague.

The problem is simple: How do we make macros usable (as opposed to the C preprocessor) and implementable, while still being Pythonic?

The only answer anyone ever comes up with is "sort of lisp-like" or "sort of scheme-like" or "anything but the C preprocessor," or at best, "sort of lisp-like but with Python doc strings." Without a good proposal, Guido tells everyone to shut up, and Python stays macro-free.

I think the answer has to come from Dylan. Dylan has a macro system is both hygenic (like Scheme) and powerful (unlike Scheme). But Dylan isn't a syntaxless Lisp derivative; it's a language with a quasi-Algol syntax (like Python). And the macros preserve the syntax. Pretty cool. The classic article is D-Expressions: Lisp Power, Dylan Style (Jonathan Bacharach and Keith Playford, 1999).* (Watch out, that's a PDF.) You can also read The Dylan Reference Manual.**

I'm not going to explain Dylan macros in detail. There are three secrets.

(1) Pattern-matching rewrite rules are sufficiently powerful for almost all purposes. Full-scale code-generating procedures break syntax, but fortunately, they're unnecessary. While there are a few annoying limitations to Dylan macros, they have to do with the implementation, not the concept.

(2) Algol-like languages have useful Skeleton Syntax Trees (SSTs), just like Lisp. Macros based on raw or tokenized text either break syntax, or are too restricted (like cpp). Macros based on ASTs are useless unless you can modify the AST parser on the fly. With macros based on SSTs, a single one-time change to the parser is needed to allow macros, but then you're done.

(3) Hygenic macros need an escape clause. One reason that Lisp and C macros are dangerous is that they accidentally inject names into the scope of the calling code, possibly overriding the caller's names. One reason that Scheme macros suck is that there is no way to inject names into the calling scope. Dylan solves this by making new names hygenic by default, but providing the "?=" form to circumvent this.

Here's an example of a Dylan macro:

 define macro with-open-file
   { with-open-file (?stream:name, ?options:*) ?:body end }
   =>
   { let ?stream = #f;
     block()
       ?stream := make(, ?options);
       ?body
     cleanup
       ?stream & close(?stream)
     end block }
 end macro

 with-open-file(stream, locator: "phonenumbers")
   process-phone-numbers(stream);
 end with-open-file;

Here's what the same thing could look like in Python:

 defmacro withfile:
   """withfile(stream, °options, **kw) -> Open a file for use without the enclosing scope"""
   withfile(?stream:name, ?options:*, ?kw:**):
     ?:body
   =>
   ?stream = file(*?options, **?kw)
   if (?stream):
     try:
       ?body
     finally:
       ?stream.close()

 withfile(stream, name="phonenumbers"):
   processPhoneNumbers(stream)

Of course this is a pretty bad example for Python--but it's short.

Anyway, there are a few obvious differences. In Python, indentation is part of the SST. There are docstrings. The list of matchable forms is somewhat different--in particular, "*" matches an argument list, rather than arbitrary syntax. (We might need a form to match arbitrary syntax, the "*" and "**" forms should obviously match the usual Python meaning. Maybe "***"?)

Why do macros need names? Not just for debugging, but so you can undefine them with "del withfile" the same way you can with functions.

I haven't written an implementation. The obvious first step is to build a preprocessor that converts Python source with macros into macro-expanded standard Python source. But I'm not sure that's sufficient to show off the idea, because it doesn't show how macros will integrate with the parser, or how macros can be used in interactive code.

* Look, an actual blog-style link in my blog!
** And another one!

Why I'm Not an Engineer

My official job title at most of my jobs has been "Senior Software Engineer." And I hate it.

I won't belabor the fact that "Senior" makes it sound like I've been at the company for 15 years, when actually I've been there 6 months and the company was only around for 3 months before that.

The real problem is with the word "Engineer."

Engineering is about physics and mathematics. Engineering is about designing to exacting specifications. Engineering is about being able to prove that a solution will work before implementing it. Engineering is about discovering all relevant factors and taking them into account. Engineering is about repeatable successes. Most of all, engineering is about using proven processes to guarantee reliability.

Anyone can design a bridge, or a skyscraper, or a fuel pump. But we don't let just anyone do it. And there's a reason. If you design a bridge, a "crash" is not acceptable. People die, there's a major scandal, your company gets sued for negligence, and you get fired, if you're lucky.

But when you're building a mail server, or a web browser, or a real-time strategy game, it's perfectly normal for a sizeable percentage of your customers to have regular crashes. What happens? Your company charges them for support, and gives you a bonus for releasing on time.

Worse, when something goes wrong in a software project, it's usually so difficult to pinpoint the problem that nobody even tries. Remember the Challenger O-ring hearings? Imagine something like that applied to Microsoft Windows. In fact, you don't have to. The Navy tried to hold similar hearings to determine why a Windows crash left one of their ships dead in the water. Ask them how that went.

From the other side, what we do--when we do it well--feel something like magic, even to us. That's simply not true for engineers. That's not to denigrate engineers. They get the satisfaction of solving a difficult problem (just like us), or doing a job and knowing they've done it right (unlike us), but if they pull something off and can't believe that it works, they consider that a red flag, not a reason to celebrate.

So, if what we do isn't engineering, what is it?

Computer Science. Adobe calls all of their code-writing employees Computer Scientists. After all, that's what most of us studied in college, right?

This is completely bogus. Someone who studies algorithmic complexity, or programming language semantics, or type theory: that's a computer scientist. Calling someone who writes software a computer scientist is like calling an auto mechanic a physicist.

Software Craftsmanship. Some people (most famously Pete McBreen) argue that what we do is more like craftsmanship than engineering. There's something to that. Despite all the college degrees, the knowledge that we pass down is more like guild wisdom and rules of thumb than rigorous training in known methodologies.

The problem is that craftsmanship, even more than engineering, is about repeatability. A craftsman builds similar products over and over, and gets very good at doing so. But software is supposed to be not just better, but radically different, than its predecessors. A marble table that's every bit as good as one from the Victorian era is a triumph. A program that's just as good as last year's is a failure. Forget our products; even our methodologies are supposed to be radically different than those from a decade ago.

Software Development. "Developer" doesn't sound bad at first glance. But that's only because it doesn't seem to mean anything.

As a consequence of that vagueness, most fields use it for those people who are neither actual workers nor creatives. Within the software industry, we're the workers and the creatives--or at least what passes for them. Look at development executives in television, or the career development team at a performing artists' agency, or the business development team at your own company. That's about as far from what we do as anything I can imagine.

Tinkering. Others have likened us to tinkerers. There are no rules to what we do, but there are lots of tricks that we figure out, and these tricks look like magic to the outside world even when we try to explain them, even though they're really pretty simple.

But that doesn't really work, for much the same reason. Tinkerers make minor improvements to existing products, Whether or not we succeed, we're always trying to build new things.

Mechanics. Sometimes people in "traditional engineering" fields like to disparage us by calling us mechanics or technicians. But mechanics only exist in fields where there are already engineers. Without automotive engineers to build cars, there'd be no auto mechanics to fix them. We're the top guys in our field, sad as that may sound.

Besides, this suffers from the same problem as the last two categories, only more so. A mechanic who ripped out your working engine and replaced it with an untested new model because he had a cool idea would not be in business long.

Software Writing. Why not run with the "creative" thing. We write programs, so we're writers. Maybe it's not exactly writing novels, but then neither is punching up dialogue in screenplays, producing tech manuals, or doing biographical research--and yet people who do those jobs are called writers. And if our hours and work conditions aren't as weird as writers', they're not exactly normal.

That all sounds appealing, but I think it misses something crucial: We're creating products. Actual things that do work for people--at least when they work. Writers don't do that.

Plus, nobody wants to read what we write. I know, the linux kernel is available online, and thousands of people download it and read it. But every one of them is doing so in order to improve it, or get ideas for a different kernel, or write linux drivers, or debug linux applications. Not a single person reads the linux kernel just to read it.

Programming. What about programmer? We make programs, so we're programmers.

If it sounds a bit dull and old-fashioned and unglamorous, so what?

That's really the crux of the matter. Most people in our field want an impressive title. "Engineer" has some kind of cachet that many of us covet.

But why? Who are we trying to impress?

Girls aren't impressed because you call yourself an engineer. If they're impressed by your job, it's because you're so overpaid that you can afford to buy ridiculously stupid toys. Or maybe because you can get away with dressing like a teenager and showing up to work at 11:00.

Banks and lenders aren't impressed because you call yourself an engineer. They're impressed because you make enough to pay interest. And because even if your company tanks or you get fired, there's so much turnover in your industry that you'll have another job in a month anyway.

Your grandmother may be impressed, but she doesn't know the difference between you and a computer repair technician or an AOL tech support drone. Besides, your grandmother would be a lot more impressed if you just called more often, or married someone nice.

Do you really want to be a guy who wears a buttoned-up white shirt, black socks even with sneakers, and a slide rule in a pocket protector? Not me.

The best title I ever had, I was in charge of a team that, as far as our founder could tell, just randomly banged away at keyboards until eventually an approximation of working code came out. Like Hamlet, but he didn't need an infinite number of us.

So I was Big Chief Codemonkey.

Unfortunately, we soon had to start dealing with H1B visas and partnership deals and next thing you know, I was Lead Software Engineer.

Sigh....