Joe Ganley

3 good books

2010-01-14T12:30:00.005-05:00

I don't generally post anything about the books I read, but the last three were a hat trick of semi-obscure books that I really enjoyed, so I thought I'd share.

Jennifer Government by Max Barry. This is somewhat absurd (in a good way; think Catch-22) near-future satire, not unlike one of my all-time favorites, Sewer, Gas, and Electric. In an ultracapitalist America, the government is almost nonexistent and big corporations are the seats of power. An ambitious executive decides that his company should take military action against its competitors, which arouses unwanted attention from one of the few remaining government agents.

The Ridiculous Race by Steve Hely and Vali Chandrasekaran. A true, double-first-person narrative by two friends who compete to race each other around the world without using airplanes. The goal is not only to win, but to outdo one another in the experiences had along the way. Absolutely hilarious.

Don't Sleep, There Are Snakes by Daniel Everett. A memoir by a missionary/linguist who has spent much of the last 30 years with a tribe of native Amazonians called the Piraha. They're really enchanting: happy and lighthearted, almost never angry or violent, nonmaterialistic and mostly at peace with each other and with the world. It makes you think about what it really means to be 'civilized.'

Graphical résumés

2010-01-11T12:06:00.004-05:00

I've posted a couple of times about using design and data visualization techniques to create a résumé that is graphical, rather than just a big pile of text.

Well, the Cool Infographics blog has posted a bunch of examples of the sort of thing I was talking about.

An API is a user interface

2010-01-08T08:27:00.004-05:00

I just ran across a pretty good CACM article on API design. This is a topic I feel very strongly about. In numerous interviews, as I was touting my user-interface design skills, my interviewer has said something along the lines of, "But ours is a batch-mode product; it doesn't have a user interface." My usual response to this is that all programming, especially API design, is user-interface design. The exact same principles that make a good user interface apply to a programming API, and indeed to programming in general. A program is (among other things, obviously) a user interface between programmers, including between you and your future self. A poorly-designed API (or more generally, poorly designed code) is a long-term source of friction, and as with most design work, extra time spent at the beginning to create a usable interface can have a tremendous return on investment.

As that article mentions, a really good API almost disappears. Methods exist to do what you want, they are named what you expect them to be, and they do what you need them to without requiring any gymnastics of the sort described in the article. The only two APIs I've ever seen that come close to what I would consider the gold standard are Qt and OpenAccess (disclaimer: I worked on the latter, so I may be biased there).

A search for API usability guidelines turns up a number of good resources on the topic, such as this one and this one.

Ghosts in the machine

2010-01-01T19:12:00.004-05:00

My friend Lawrence pointed me to a web page (static local copy) that appears to be some kind of Markov-chain-generated linkfarm nonsense, that contains both of our names. This shocked us at first, but there are a number of logical explanations; for example, Lawrence has commented on my blog, so the two names do coexist elsewhere on the web. Still, I'm not completely sure that's what is going on here, and regardless it's a little bit spooky.

This reminded me a little of Lawrence's outstanding short story Coding Machines, so I'll take this opportunity to plug that.

Separated at birth?

2009-12-31T11:23:00.004-05:00

On the left, a $25 Timex Camper. On the right, a $100 Bertucci A-2T. I wanted the Bertucci, but that black-on-black design was out of stock everywhere, so I got the Timex instead. It amazes me how similar these are; I wonder which was first? I also wonder, as I always do with watches, about the relative quality of these two. There's no doubt that the Bertucci is a better watch, but four times better? I've been dubious about expensive watches ever since my $250 Casio bit the dust after snorkeling in 12 feet of water. I'd love to see someone do a real field stress-test of a bunch of watches in a range of prices.

Aside: The closest thing I could find to an official G.I. standard issue watch looks very similar to these.

Project release: Dots and boxes

2009-12-05T20:59:00.004-05:00

More clearing the decks of old projects: This one is a JavaScript implementation of dots and boxes with a very simple computer opponent. It works perfectly in Firefox, but incorrectly in Chrome and not at all in Internet Explorer. This was an evening's work, and is under-tested and a bit brittle.

Project release: Simulated annealing library

2009-12-03T20:46:00.005-05:00

I have a lot of old projects laying around in varying degrees of completeness, and I am finally going to put them all to bed so as to clear the decks for some new projects.

The first is a simulated annealing library (tarball or at GitHub). I wrote most of this on a plane several years ago. It is complete, except that there is an issue with the stop criterion that makes it often pass better solutions and end up on solutions not quite as good. This could be mitigated by writing the move manager to save the best solution seen, but the right answer is to just fix the stop criterion. Both it and the cooling schedule are very simple; my problem in writing an open simulated annealing library is that I know more sophisticated techniques for virtually every aspect of the annealing machinery, but frankly I don't feel like doing to research to figure out whether they're known prior art or whether we invented them at my past jobs (I've written several annealing engines before this one, and this one is extremely unsophisticated compared to those).

It's in C++. There is a Visual Studio solution and project, but no makefile; however, it's straightforward to compile.

I'm releasing this under what I like to call the "do the right thing" license. Which means, do whatever you want with it. No restrictions whatsoever. But please do the right thing. That means credit me. Let me know if it's useful to you, or if you make improvements to it that you choose to make open. If you use it in something that makes you money, maybe throw an Amazon.com gift certificate or something my way.

My desk calendar

2009-11-09T08:51:00.008-05:00

Due to the nature of my work, I keep a calendar on my desk at work that is separate from my various personal calendar and note-taking machinery.

Initially I used a standard month-at-a-glance calendar, but I found it a nuisance writing things inside of little boxes. Then I switched to a linear perpetual calendar (similar to this) - basically just a lined, numbered list - but found that in locating today on the page, I often don't know the date exactly, but rely on the day of the week as an indexing cue.

So what I came up with was a list format, but with the day numbers offset according to the day of the week. Here is this month's in PDF form: weekdays only, which is what I use, or with the full week. One of these days I may automate the generation of these, but in the meantime I'll try to post each month's as I generate it for myself.

No sooner had I posted this than I ran across a similar (but prettier) design from Amy Marcella (via Oh So Beautiful Paper).

Update: Here's December, weekdays only and full week. If anyone uses these, please make a comment to that effect, as otherwise I won't bother posting subsequent months.

Low tech 'laser' pointer

2009-10-28T12:28:00.002-04:00

A couple of summers ago we were in Costa Rica, and took a tour down the Tempisque River. To point out animals and such on the banks, rather than a flashlight or laser pointer, our guide used a small compact mirror to reflect the sun onto the spot where he wanted us to look. Cheap, easily obtained, no batteries, and no worries about getting wet - just the sort of low-tech solution that I love. I've since learned that this is a common trick used by soldiers as well.

BTW, our tour guides, Jose and Amy, were incredible. Highly recommended.

Highways

2009-10-20T17:13:00.001-04:00

I was playing around with the OpenStreetMap data, and I noticed how interesting and organic the highway systems of major cities look. So, I extracted a few to monochrome images, which can be found here.

Where is Ashley?

2009-10-13T09:16:00.005-04:00

This picture, taken by my daughter,

followed by a conversation with my neighbor, inspired a quick round of other simulated Ashley destinations:

Touch

2009-10-01T07:14:00.004-04:00

I remember when Star Trek: The Next Generation first aired (yikes, 22 years ago), I looked at their consoles - big flat glass panels covered with virtual buttons and sliders - and thought that had to be the dumbest user interface ever. Now look: With the iPhone and its various clones, and the soon-to-be-legion (if you believe the breathless press) tablet PCs, this would seem to be the user interface of the future. (TechCrunch declares the imminent end of button keyboards.) My verdict is still out on this; while the interface is fraught with exactly the kinds of problems I envisioned when I saw that Star Trek console, I must admit that I've gotten fairly proficient even on the miniscule keyboard on the iPhone screen, and in general the interface works better than I would've expected it would. I have to imagine that it will be even more usable with a larger screen. There's no denying that it's incredibly flexible as an interface medium, that it provides a lot of really useful new affordances (e.g. multitouch), and that - perhaps most important of all - it's really cool and fun to use.

Ideas

2009-09-30T08:23:00.003-04:00

Entrepreneurs (and wannabes) often talk about the value of ideas. I'm firmly in the camp that good business ideas are a dime a dozen, and that execution is the hard part. Here are a few ideas for businesses (or just little projects) that I don't expect to ever get around to executing.

A web site for managing emergency contact information. Every time my kids go on a field trip, I have to fill out a form with all of our emergency contact information, the kids' allergies and medications, etc. I'd like to see a web site where I can enter this information once. Then, I can point the field trip leaders to that site. They would request access, and I would get an email letting me allow that access (or not). There are obviously some minor security concerns here, but it's not as if my phone numbers and my kids' allergies require Fort-Knox-level security.
Similarly, a web site for managing immunization records. Doctors would be able to add to the records when the kids get their shots, and schools would be able to access the records; again, with a similar access mechanism that would allow me to grant one-at-a-time access. Same security concerns, but again, this isn't terribly sensitive data.
Matt Mullenweg (founder of WordPress) used to have a business card that just said "(1) Go to Google, (2) Enter 'Matt', (3) Hit 'I'm Feeling Lucky'." It would be cool for those of us who aren't the first return (or even in the first thousand) to have a widget that, when you enter a term, tells you where a given term would rank in Google's search results. I'm not sure if this could be done by scraping, or if you'd need direct access to Google's indices (i.e. you would have to be Google).
I just love the vibe of little paper-goods stores like Dandelion Patch, but such places always seem very (forgive me) girl-oriented. I'd love to run a store that is sort of a brick-and-mortar version of LifeHacker. It would stock high-end and niche office supplies, planners and other organizational tools, and the like. I'd want sort of a coffeehouse vibe - one of my local Barnes and Nobles has an attached Starbucks, right next to the section with all of the planners, pens, and the like. Like that, except smaller and with a much larger inventory of supplies and only a very small section of books relevant to organization, life management, Getting Things Done, and so forth. I'd love for it to provide an inventory of what are currently mostly "print it yourself" or "have Kinko's make this for you" supplies, such as those from David Seah. They might also offer custom printing services, and classes on GTD and similar organizational skills. Sort of a hybrid of coffeehouse, Barnes and Noble, Dandelion Patch, Office Depot/Staples, Levenger, the Franklin-Covey store, and The Container Store. My best idea for the store's name: Index.
I'd love to have a news reader that learns from what articles you find interesting, and over time shows you more like them. It seems hard to believe that this doesn't already exist. It would also have functionality to allow you to explicitly tag specific subjects for which you'd like to see subsequent articles.
I'd love to see a web-2.0 project management solution. Sort of MS Project, except web-based and dead simple in a 37Signals sort of way.
I'd love to have a single-serving site for "what should I do this weekend?" It would obviously be localized, probably fed by users, and would just dump a few ideas for fun things to do in your area - both specific things that are happening this weekend and interesting destinations in your area that aren't tied to a specific date.

These, like many business ideas, are all things that I wish existed. If anyone is motivated to make them real, please do.

Response links in email

2009-09-22T14:51:00.002-04:00

Open letter to web services companies: When you send me emails that need me to follow a link to respond, don't make me jump through a bunch of hoops to do so. For example, take Twitter. About twice a week, I would get an email telling me that someone wants to follow me. All of these are spammers. In order to block them, I have to click a link in the email, log into Twitter, click a button to block them and then another to confirm. Instead, there should be a link in the email to accept and one to block, and neither should require me to log in. This is not a high-security situation; I wouldn't want emails from my bank to work this way, but for Twitter there really aren't any consequences if someone gets into my email and accepts a follow, or if I do so accidentally. In the end, the easiest way for me to solve this problem was to just terminate my Twitter account, which is what I did.

Similarly, Meetup: I get an email asking me to RSVP. They conveniently provide Yes, Maybe, and No buttons in the email. However, in order to follow them, I have to log into my Meetup account. It's a pain. So, when my answer is "No" (as it usually is), I just don't bother. This can only end up hurting the meeting organizers, and thus ultimately hurting Meetup itself.

For an example of doing this right, take Netflix. Periodically I get an email from them asking when a DVD was sent, or when it arrived. There are appropriate links in the email. I click the right one, and I'm done. No login, no confirmation. One click. The companies who are doing this wrong should take a lesson here.

Working from home

2009-09-17T07:08:00.004-04:00

Especially in the software industry, a topic that gets revisited again and again is that of working from home vs. working in an office. There are those who seem to believe that working from home is a thoroughly bad idea, and at the other end of the spectrum there are companies that are entirely locationless, with all of their employees working from their own homes.

I worked for 12 years, for four different companies, from home. All four companies were headquartered in Silicon Valley, and I live in the suburbs of Washington, D.C. I worked from my home, except that I visited headquarters for a week at a time, at intervals ranging (depending on the company and on where we were in the release cycle) from once per month to once per quarter. At the first company, I had worked in the office in California for 18 months before I moved to D.C. At the second, a startup, I moved to California for the first 6 months, then returned to D.C. The remaining companies, I entered cold, working from home from the beginning.

Now, for almost two years, I've switched to the other extreme. I work for a local company, and due to the nature of the work it must be done 100% in the office, and typically only 40 hours per week.

The first thing that people always seem to talk about is productivity. From my own experience, I can report that I was at least as productive at home as I am in the office. However, that claim can be a little misleading; I was more personally productive, but I made fewer contributions to other developers' work, and in general I now do a lot more of that and more generally facilitating the work of others. All in all, I would say that the net sum is about even; however, as my career advances, my job becomes more about collaborating, overseeing, and facilitating others and less about my own productivity, so it becomes more important to have the kind of face-to-face, high-bandwidth contact that you get in an office. Indeed, this is a big part of the reason why I finally stopped working at home in favor of working on-site. Proponents of telecommuting are quick to point out all of the ways that modern technology facilitates communication: Not just phone, but instant messaging, wikis, chat clients, screen sharing and collaboration software, video conferencing, and on and on. These things definitely help - I was certainly able to be more effective in the later days of high-speed broadband than I was at the beginning with nothing but phone and 128KB/sec ISDN - but they're still a pale shadow of being together in person in front of a whiteboard.

In some ways, a good telecommuter turns this handicap into a win. You do more of your design up front, and more carefully and thoroughly, because that reduces the need for high-bandwidth communication throughout the development process. You do this design work during those times when you are together face to face. As a side effect, careful design at the beginning leads to better software. On the other hand, it is impossible to nail down all of the design up front; inevitably you figure things out during development that require changes, sometimes large ones, in the architecture of the system. Here, being geographically distant becomes a real hindrance; what happens is generally that the lead architect comes up with a design, and then the other members of the team critique it, and in my experience the result is often inferior to what would have come of true collaboration through the entire design.

As far as your work schedule, being at home is obviously very flexible. This is both good and bad. It is good because you can work when you are most productive. I am a lark, and I am at my sharpest first thing in the morning. At home, I could go straight to work when I woke up; now, I spend 90 minutes of what would be my most effective time of day showering, getting dressed, and commuting. Similarly, you can put that flexibility to work on a larger scale: If you are having one of those days where you just can't seem to focus, just step away; clean the house, or take a 3-hour lunch, or get a white chocolate mocha. And on those days when you are really in the zone, you can work 12 or 14 hours while still having a few hours with your family. I also found it easier to stay focused on work in general, because my hours were all mine; not only was there none of the kind of non-productive water-cooler BS'ing that you do in the office, but I also spent very little time surfing the web, reading XKCD, and the like, because it was easy to see that the time I spent doing so was mine. You can also move your schedule around to enable more time with your family; I typically got up very early and did an hour or two of work before anyone else woke up, and did another hour or two of work after the kids went to bed, thus freeing those hours during the day for family time.

The downside is that it's hard to maintain any kind of work-home separation. You feel like you are always at work, or at least like when you're not at work, you should be. Exacerbating this is the fact that the bulk of my teams were in a timezone where it was three hours earlier, so I often fielded calls and IMs through the dinner and kids' bedtime hours. Similarly, despite my constantly telling them not to worry about it, my coworkers were loathe to call me when they knew it was my evening, and so whatever issue they had got delayed until mid-day (i.e. their morning) the next day.

Some telecommuters report having trouble staying focused on work, with the constant potential distractions around: Housework to be done, kids to play with, etc. I never had much of a problem with this; I'm a fairly disciplined person, and I love what I do, so except for those occasional times I mentioned where focus is elusive, the call of the laundry was rarely louder than that of the work. Any anyway, a lot of household tasks, like laundry, don't require require a lot of attention. But if you're the sort who has trouble focusing on work at home, then working at home simply might not be for you. It's also notable that my wife was home during my core work hours, and she ran interference with the kids and such.

Another factor to be considered is the social interaction of being around other people. If you are the sort who needs this on a regular basis, then working at home is probably not for you. However, many software people, even in the office, spend most of their day hunkered down in their cube, talking to others only when work requires it. For such people, being at home may not make much of a difference in their social schedule.

Another important factor is that employees who want to work from home are happy doing so. The effect of this is often underestimated; a happy employee is a productive employee. For many of us, giving us this kind of flexibility and trusting us to do the work in whatever way we find most effective has an incredible ROI in terms of employee productivity.

Most of this applies to the line engineer, a leaf node in the org chart. Things get much more difficult when you need to manage others, or to lead efforts with multiple developers. Then, it becomes not just about you; those other developers need to interact with you, both for technical guidance and for the more touchy-feely managerial stuff. It can be done, especially if you're really proactive about communicating with them and giving feedback, but it's hard and arguably inherently less effective than being together in person. Again, this is why I stopped working at home; I reached a point where I felt that the level of leadership I needed to do could not be effectively accomplished remotely. At least, not by me.

Finally, note that for really great developers, you may not have a choice - they either work from home or they don't work for you. Folklore says that an outstanding developer is many times more productive than an average one, so even if such a person isn't quite as effective from home as they would be from an office, hiring them can often still be a net win. Certainly the companies I worked for were happy with my performance working from home.

The bottom line: A leaf-node developer, if they have the right discipline and temperament, can be every bit as productive as a telecommuter as they could in the office (if not moreso). As their responsibilities of leadership and mentorship grow, this will become increasingly difficult, until a point where they may not be able to effectively do their job remotely. Even for such people, though, the need to be in the office isn't constant, and while I haven't ever been in a situation where this is possible, I believe that even a manager could effectively spend a day or two a week working at home.

Update: I found this study interesting, and it squares well with all of my experience.

C++ copy constructors

2009-08-28T08:24:00.006-04:00

I ran into a bug the other day that I've run into enough times in my career that I thought it was worth sharing.

One of the perils of C++ is that it does certain things behind your back, and if you don't understand what those things are, they can bite you. One way this manifests is that apparently meaning-preserving changes can have surprising consequences.

In this recent case, I inherited some code that I was cleaning up. One of the changes was to change this:

SomeClass foo; foo = factory.getSomeClass();

into this:

SomeClass foo = factory.getSomeClass();

Simple enough, right? But after making this change, a later access to foo segfaulted.

Can you guess why? Here's a hint: SomeClass uses a reference-counted shared-data implementation, similar to std::string.

Here's what happened: SomeClass had defined an assignment operator, but not a copy constructor. The old code used the former, and the new code used the latter. In the absence of a user-defined copy constructor, the compiler generates one that just does a ~~bitwise~~memberwise copy. As a result, the reference count is copied and not incremented, and so later it becomes 0 before it should, and the data is deleted. A subsequent access to that data produces the segfault.

The solution is simple: Define a correct copy constructor. My own policy is never to rely on the compiler-generated copy constructor and assignment operator. When I write a new class, if I think I don't need these, I immediately add method prototypes for them that are private and have no implementations. Then, if you ever invoke them (deliberately or not), the compile will fail, and you'll know that you have to implement them for real (or change the calling code).

Personal finance software fail

2009-07-26T10:13:00.002-04:00

Since Microsoft is terminating Money, I've been looking around for a replacement, and so far it's just a whole lot of fail. I tried both GnuCash and MoneyDance, and not to put too fine a point on it, but they both suck. Quicken doesn't offer a free trial, as far as I can find. They do offer a free online version, but I just can't get comfortable with giving all of my credit-card logins, much less all of my financial data itself, to an online service. (This is the same thing that has kept me from trying out Mint.) I guess I'll keep looking.

New things

2009-07-11T12:30:00.002-04:00

Now that we know we're not moving, we got a fun new car (I had sold mine) and a new pet.

Sydney

2009-07-09T08:22:00.004-04:00

Inquiring minds want to know what happened to abort our move to Sydney. The answer is quite simple: It's really amazingly expensive there. Think Manhattan expensive. You'd think this would've been something I looked into before it got as far as it did, and to some extent it was, but it's actually not an easy thing to find a quantitative answer to. I did push back against the offer I was given by Google, and was told repeatedly that the cost of living there was lower than here in the US, and that relative to it the offer was quite generous. I can only assume that those who told me this were badly mistaken about the cost of living here, or about the state of the family housing market there, or some combination of those and related factors.

The result is that when we got there and began to house-hunt, we discovered that we were totally unable to find anything we would live in, within an hour of work downtown, in a good public high school's catchment, within our budget. Note that said budget was about 30% higher than what we pay for our large 5-bedroom home here in the DC area. Our choices were either a 3-bedroom apartment about the size of an extended-stay hotel (note that I have three kids, so this would've meant the youngest two sharing the third, tiny bedroom), or a slightly larger 4-bedroom house or townhouse that is very old and generally unrenovated. The latter places reminded me of places I lived when I was in college.

After the fact, someone at Google (who wasn't involved in the original conversations) told me this was pretty much the status quo: That families have to live an hour or more from work. I was definitely unwilling to do that, especially in addition to paying so much more for housing and pretty much everything else as well.

I will say, though, that Sydney is a really great city, and the people there are the nicest I've ever encountered anywhere.

So we're staying where we are. I've happily returned to WOTI, though at a different site and in a different job. No hard feelings; though we were excited about the would-be adventure, we love it here, and are quite delighted to be staying.

Undo

2009-07-01T16:05:00.001-04:00

After our house-hunting trip was unsuccessful to say the least, we are not moving to Australia. Details later.

Going walkabout

2009-06-17T21:39:00.003-04:00

After a great deal of back and forth, false starts, and difficult hurdles, we are moving to Australia. I am going to work for Google in Sydney. Wish me luck!

Creeps

2009-06-07T07:22:00.001-04:00

People may have noticed that the photos of my children are no longer public on Flickr. After the third time someone with no photos of their own on Flickr favorited a picture of my 12-year-old, I decided it was time to draw the curtains. If you're a friend of ours, email me and I'll give you access.

More iTunes suckage

2009-03-10T20:36:00.003-04:00

It's a minor thing, but this sort of thing is endemic to the suckfest that is iTunes on Windows. For security reasons, I disabled autorun on my machine. The next time I fired up iTunes, I got this alarming dialog:

Oh, crap. I don't want to turn autorun back on, but this seems to be saying that I won't be able to use CDs with iTunes if I don't. Oh well, I'll click No and deal with this later. The very next thing, I got this dialog:

Just kidding! When I said iTunes wouldn't recognize CDs any more, I meant unless you hit F5. I just wanted to see if I could make you turn autorun back on.

Disgraceful.

Coda

2009-02-28T09:54:00.000-05:00

Mission accomplished: I wrote a new blog post every day in the month of February.

I love taking photos, but I hate traveling with a camera. You end up looking at everything as a photo opportunity, and in effect watching your vacation through the viewfinder of the camera instead of fully participating in it.

So it is, I've found, with trying to blog every day. You end up considering every thought and every experience through the filter of, "Can I blog about this?" As a result, you aren't fully participating in those experiences; to some degree, you become a spectator. It's an interesting way to feel, but largely not a positive one; while I'm happy to have done this, I don't expect to continue or repeat the effort. I can only assume that many journalists and writers feel like this all the time, and for that I feel bad for them.

Furthermore, the pressure to post frequently conflicts with the desire to write longer and more deeply considered work. I find that I've generated a small backlog of things that I'd like to write about, but that will require more than a day's investment to produce. Hopefully I'll get to these in the coming months.

Big mistake

2009-02-27T17:52:00.007-05:00

This is the story of the biggest mistake I've made in my professional career. (I say I; there were other people involved, but it was as much my fault as anyone's.) It was early in my career, and I was tasked with building a new algorithmic optimization system from the ground up. This meant we needed a database; this is a database of geometric objects and connectivity, not the kind people normally think of when they hear the word database. This is exactly what OpenAccess, which I helped develop some years later and which had it existed back then would've allowed me to avoid this whole fiasco, is.

When I came on board, we needed a prototype done, like, yesterday. So I slapped together the quickest, dirtiest database that could possibly work, with the intention that eventually it would be replaced by something more production-worthy. That may have been a mistake too, but it's not the big one that the title refers to.

Fast-forward about a year. The prototype is done, and is producing fantastic results. We've written a real database, and it is time to move the system onto it. Here's the big mistake: I did that as open-heart surgery, completely ripping the system apart and replacing the old database calls with new ones. The data model had changed substantially, so this was a major effort, not just a syntactic change, and calls to the database were woven through the entire codebase; to make another surgical analogy, imagine trying to replace all of a person's nerves. There was a period of a couple of months in which the system did not work at all. Eventually we got it running, and then there were several weeks of debugging - just plain old bug fixes, many of them fixing bugs that had probably already been fixed in the old version but the fixes were lost with the old database calls.

Meanwhile, we weren't able to just freeze the old system in carbonite; we needed to continue improving it, competing in benchmarks, and the like. So it continued to evolve forward from the code base that had been used to begin the conversion. Had the new version ever worked properly, we would have had to make all of the same improvements to it when we were done that we had made to the original system during those months.

Because here is the worst part: The system running on top of this database was mostly Monte Carlo optimization algorithms. Such machinery is highly dependent, in unpredictable and hard-to-debug ways, on such harmless-seeming transformations as changing the units in which a size is expressed, or changing the order in which a graph vertex's edges are enumerated. There were many such differences between the old and new databases, and the new system never did produce results as good as the old one.

After it was all over, it was clear to me that this way of making this conversion was totally wrong-headed. The right way would have been to first write a bunch of regression tests. Then write a facade over the new database that had the old database's API. Move the system onto it (which is nothing more than recompiling against the new library). Then slowly migrate the code, and the facade API, to look more like the new database's API. Run the regression tests frequently, so that if you make a change that breaks things, you know what change is to blame. Eventually the facade API looks just like the new database's API, and at this point the facade is vestigial and can be removed.

This approach has two key features: There is just one version of the system, and it is always working throughout the process. It probably takes substantially more time than the open-heart surgery approach would if everything went smoothly, but how often does that happen?

So imagine how I felt listening to this week's Stack Overflow podcast, in which Jeff talks about facing the same problem with Stack Overflow. Evidently the schemas of his core tables turn out to be really wrong, and force a lot of baroque and complicated over-joining and such in the site's code. Joel suggested something almost identical to what I decided so long ago I should have done: Create a view that looks like what he wants the new table to look like but has the original tables underneath. Then, migrate the code a piece at a time from the underlying tables to the new view. Again, there remains just one, working system the whole time. To my horror, Jeff disagreed quite vehemently and said he planned to go the open-heart surgery route. He went on a bit about the romance and adventure of that sort of swashbuckling. Surprisingly, Joel acquiesced a little and said that might be the right approach for Jeff. I seriously doubt it, and I was disappointed that Joel didn't let Jeff have it with both barrels; after all, this is just a smaller-scale version of the exact same mistake Joel wrote about in Things You Should Never Do, Part I, for pretty much the same reasons. (By the way, that hadn't been written yet when I had my little misadventure.)

Just as Joel says quite unequivocally that you should never do a full rewrite of an application, I'll say just as unequivocally that you shouldn't perform this kind of massive surgery on a working application unless it is simply impossible to do it incrementally. Indeed, in the decade since then I've formed a habit of never having my software be broken for more than a few hours at a time.