Software Rants & Other Miscellany

Not All Early Optimization is Premature

2011-07-10T15:53:00.003-04:00

Andrew Parker and Jeff Atwood have both had great posts recently about performance as a feature, but I think they've each actually stopped short of a powerful point - improving a product experience by 10 or 20 percent through optimization is great, but there's incredible power when you unlock fundamentally different feature sets through radical optimization. This is an area that the code-first-then-optimize process misses entirely, because incremental improvements on the same basic design will never lead to order-of-magnitude performance improvements.

The power of such performance improvements is one of the most important lessons I learned at Google. A couple examples demonstrate the kind of optimization I'm thinking of:

In 2004, when the standard storage for webmail was 2MB, Google was able to launch Gmail with 1GB of storage, because GFS provided a means for managing disk that was orders of magnitude cheaper than what the other providers were using. Rumor has it that Yahoo went out and gave NetApp millions of dollars to buy storage devices in order to come anywhere close to what Google was offering. Underlying this all is an optimization of storage and disk that was deeply more efficient that what others in industry were capable of at the time.
One of the coolest features of Google maps is the ability to see a route, then grab it with your mouse and drag it to change the route. That feature is possible because Google developed a radically more efficient route-finding algorithm, years ahead of what anyone else in the market can offer. The difference between computing a route in 1 second and computing it in 10 milliseconds means you can suddenly offer users the ability to compute hundreds of times more routes.

It's part of the modern software engineering zeitgeist that "premature optimization is the root of all evil," but as I researched this post, I found out that the full Knuth quote is a lot more illuminating than just that snippet; the full statement attributed to Knuth is actually, "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" (emphasis mine).

So yeah, we can all agree that "performance is a feature," but that fails to convey the power of high-performance systems. We can talk about caching database results or using a CDN for static content, and everyone should be doing those things, but let's not be afraid to go much, much deeper. Consider the core operations of your service, then imagine that you could speed them up by 100x - what radically new features would be enabled? With those radical new features in mind, start working backwards to figure out actually make those 100x improvements.

The big challenge is that these order-of-magnitude optimizations are design optimizations, not the kind of changes you can make after the fact. In design discussions, the engineer arguing for keeping the entire datastore in memory is immediately shouted down with the "premature optimization" line, but I think it's time we start fighting back on behalf of design-time optimization.

My hardest bug

2011-05-23T16:27:00.003-04:00

Lately I've been thinking a lot about the question "What is the hardest bug you've ever tackled?" It's an interesting index into the interests and style of different software engineers, and usually provides a jumping off point for a really great conversation.

With that in mind, I figured it was time to write up my hardest bug. This is a pretty old one, but I still enjoy it for just how head-slappingly bad the eventual resolution was.

Early in the year 2000, I was working on a cool cross-platform data warehousing product. One of the coolest things about it was that we had a vendor-independent SQL macro layer, so normal developers could write in one SQL dialect, and it would automatically translate to each of the underlying platforms (eg, SQLServer, Oracle).

My first project was to port the reporting engine to DB2; it was relatively straightforward, it just required figuring out some of the idiosyncracies of DB2, auditing the code to match those, and getting the regression tests to pass. It was a fun project and shipped without much trouble.

A couple months later, though, our largest DB2 customer called in with major complaints about reporting performance. This was my responsibility, so I took a copy of their logs and started digging in. The weird thing was, the logs showed that all the database queries were running very quickly, and yet, periodically, there were these weird 30 second gaps in the logs, as though nothing were happening at all.

I instrumented the code a bit more, shipped a patch to the customer, and asked them to send back new versions of the logs. Everywhere I had added log statements continued to show things running smoothly, and yet, there were still those odd gaps where nothing was happening.

In desperation, I asked the customer to send me a full copy of their datamart, which they did, and after taking a day to get the system up locally, I tried out the problem operations, and they ran perfectly fine - no performance issues, no weird gaps in the logs, nothing. Now I was really annoyed.

I returned to instrumenting the log files, adding a log statement after nearly every line of code. When the customer sent me the output of this very detailed logging, I could identify the exact line of source code that corresponded to the gaps, but I looked at it and said “there’s no way this could be taking 30 seconds, all it’s doing is macro replacement from our SQL meta-language to DB2 - it’s a straight string substitution.”

Still, I dug into it a little more, and that’s when my mind was blown. It turned out that the developer who had ported the macro language to DB2 had changed the string-replacement function to do a database query as part of the replacement step. The query had a performance profile that would grow as the square of the size of a certain internal log table, so the longer a system had been in production, the longer it would take to do string substitution.

It hadn’t shown up in the local test I did because I had only replicated the warehouse data, not the operational data, and it didn’t show up in the logs in production because the developer had intentionally covered his tracks to route around our built-in query logging.

Once we hit on this, it was a matter of minutes to refactor the n^2 query and ship a patch to the customer.

Making Effective Use of Code Reviews

2011-04-10T18:07:00.002-04:00

I’ve been reading chunks of Coders at Work this weekend, and the topic of code reviews has come up a few times. Code reviews are clearly a very useful development technique, but it can be tricky to apply them in a way that improves code quality without slowing productivity.

Poking around the web, I don’t see a lot of great writing about code reviews, so I wanted to share the guidelines we use at Yext for effective code reviews. The guidelines below are captured from an email I sent to the the engineering team almost a year ago, and I’m proud to say that our code reviews do a lot to improve code quality while keeping the team operating at peak efficiency.

These guidelines are heavily biased by my experience at Google. There, I saw how code reviews could identify and eliminate many preventable bugs, including many that the original developer never would have found. I also saw innumerable cases of reviewers who lost perspective on the larger goals of the team and the company, and thus acted to prevent progress on important projects as a result of matters of personal preference or sheer obstinance. My goal for Yext is that we capture the best aspects of code reviews while eliminating the worst.

With that in mind, my recommendation for code reviews is that they address the following points:

Correctness: Does the code do what it claims to do? Is the code correct in both the nominal case and the boundary cases? As a reviewer, this is your opportunity to point out edge conditions of which the original developer may not have been aware. An important special circumstance is when you may be aware of legacy systems or features that interact with the modified code in some non-obvious way.
Complexity: Does the code accomplish its task in a reasonably straightforward way? If you can point out simpler approaches that do not compromise the correctness or performance of the code, you should.
Consistency: Does the code achieve its basic goals in a way that is consistent with how similar code in the codebase achieves those goals? Is it re-using the available libraries and utility classes? Where possible, has code been refactored for re-use instead of just copying and pasting?
Maintainability: Could the code be extended by another developer on the team with a reasonable amount of effort? More than any item on the list, this is the karma investment you make by doing code reviews – the code you review today may be the code you have to update tomorrow, so taking the time to make sure it’s maintainable by others pays itself back to you.
Scalability: Will the code be performant at the expected volumes? It is important that this question always be asked in the context of expected volumes. When building a new product in an untested market, it is fine to write code that works for 100 users but not 10,000; if the product should be that successful, you can profile, optimize, and, when necessary, re-write the critical bits. The corollary is that you should not spend time optimizing code when the market demand is unproven.
Style: Does the code match the team style guide? This should rarely be controversial. The obvious assumption here is that your team should have a style guide.

There are some items I believe should only rarely be addressed during a code review:

Scope or mission feedback: “I don’t think you should be doing this project” is almost never a useful comment for a code review. If you think the team is embarking on projects that are not worthwhile, that is great feedback to share, but not in the context of a code review. The exception here is if someone is introducing a new way of doing something that is already well-handled in some other way.
Design review: A code review is not the time to evaluate the overall design of a project. For example, "I don't think you should be using the DB to store this data" is not useful. It is incumbent upon the developer to have their designs reviewed before implementation, and there will be scenarios in which the fundamental design is questioned during the implementation, but for a project that has been through a design review, let the results of that design stand.
Personal preference: “I would rather you do it my way” is an invitation to an unproductive debate. If you have a way that is demonstrably better, you should always argue for it. The hardest part about this point is identifying when a review has deteriorated to matters of personal preference; the hallmark I spot most often is when people are trading hypothetical scenarios in which alternative solutions might be advantageous, with no way of determining the likelihood of said scenarios. In these cases, the default is to use what the developer has already written.

How can you, as the developer, write your code in such a way as to make a code review? A few simple practices help.

Correctness: Comprehensive unit tests are the best demonstration that code functions as intended.
Complexity: Favoring small methods and cleanly-separated functional units makes it easy for your reviewer to see how everything fits together.
Consistency: When building new functionality, you can maximize the consistency of your code with existing work by taking the time to research how similar code solves similar problems. If you suspect someone else has solved the same problem before, ask!
Maintainability: Thorough commenting and the use of meaningful names throughout your code help ensure that others will be able to easily understand your code.
Scalability: My #1 recommendation in demonstrating the performance of new code is to just take 30 minutes and write a little driver to run your code through its paces. This can be total throwaway code, but simply being able to tell your reviewer that you’ve done a performance test makes this topic less debatable.
Style: The most important thing you can do to maintain style consistency is to configure your editor to implement your style guide. (As an aside, this also means that your team should adopt a style guide that is simple to automate in the editors used by the team.)

Even when everyone on the team follows these guidelines, there will frequently be strong debate during code reviews, and that’s a great thing - the point of these guidelines is to focus the debate on what matters.

All of this ignores some very tactical questions about code reviews like what code gets reviewed and what tools we use to aid that process. If you’re interested in hearing more about that, leave a comment and I will follow up with another post.

The Dark Side of Passion

2011-03-13T20:49:00.005-04:00

There was a lot of great response to my post about the Passion Gap, but some people misunderstood it to mean that I fall in with the camp of career counselors who tell you to “do what you love and the money will follow”. While I believe that doing something you love is important, I also strongly believe that following your passion must be grounded in reality.

This first became clear to me when was in high school. Sometime during freshman year, they marched us all into the career center and made us look up the key facts on our ideal jobs. We had to find out good college majors to prepare us for those careers, the demand for people with those jobs, and the average pay we could expect. I looked up the job I’d be dreaming about since age 9, and I was incredibly disappointed at the average salaries. When my father was pushed into early retirement while I was a senior in high school, the cold hard reality of making a living was a key input in my eventual decision to study computer science.

To be completely clear, I’m incredibly happy with what I do - I love going to work every day. But I also love providing for my family.

Moderating your passion with realism is not at all inconsistent with having a deep enduring passion for your startup. Even Dennis Crowley, protagonist of the Passion Gap post, is on the record saying that the one thing he most wants to be good at is … karaoke. But belting out Whitesnake just doesn't pay the bills.

The dark side of passion is blindly following a passion that won’t support you. This post at Study Hacks goes so far as to make the argument that the “follow your passion” culture is responsible for the fact that self-reported job satisfaction rates have fallen every year since measurement started in 1987 - the argument is that because people are so focused on the perfect job, the one they can be most passionate about, they are ultimately disappointed by having simply a great job.

The unrealistic view that it’s sufficient to simply be passionate about something and not match that passion to reality is captured in this op-ed from the Times in 2009, in which the author describes a difficult search for work as an art instructor, despite qualifications including an MFA. The author admits, “In my master’s program, we … tried not to dwell on earthy, unpleasant topics like money, or how to make it.”

As I said in my earlier post, I deeply believe that passion can be a competitive advantage for startups, and I also think it’s crucial to success in most everything else, but that passion must be channeled in a way that is connected to reality.

Foursquare, Facebook, Founders, and Passion

2011-03-09T08:27:00.003-05:00

Late in the summer of 2009, I was talking to a very successful entrepreneur at a tech industry meet-and-greet when Dennis Crowley, co-founder of Foursquare, came into the room, at which point the person I was talking to commented, “I first met Dennis at 3am on a street corner on the lower east side. I have never met another founder who is a more direct physical embodiment of their startup.”

That comment always comes back to me when I hear someone saying that a particular startup “is just a feature,” or, “has no defensible technology,” or, “could be replicated in a weekend.” There are a lot of startups out there that are subject to this critique, and yet, they continue to do extremely well in the face of competition. And I think, in many cases, it derives from the fact that these founders are the physical embodiment of their startups.

In the case of Foursquare, we should expect by now that Facebook Places would have obliterated the use of Foursquare. And yet, as SAI points out this morning, Foursquare’s user base has doubled since the launch of Facebook Places. That’s pretty stellar growth in the light of competition that should be winning on every dimension. And that’s before the release of the awesome goodness that is Foursquare 3.

Why does Foursquare just keep winning? If I could tell you the specific features or user interactions that are lacking in Facebook Places, I’d be a rich man. I’m in the Facebook mobile app 3 times a day, minimum, let alone being on Facebook over the web at least twice a day - I should be using Places all the time, and yet I just keep coming back to Foursquare. Somehow Foursquare just hums.

This, to me, is the difference between a product built by someone who is deeply invested in the in the underlying product idea, as compared to a product built by someone who is just trying to check off a set of feature boxes. This is what I think of as the Passion Gap.

If you’ve ever heard Dennis talk about Foursquare, or mobile devices, or cities, you can’t help but see that everything you think of, he’s already thought of, turned over in his head eight times, and reached the conclusion that you would eventually come to if you spent eighteen months in deep thought about the topic. Does Facebook have a Dennis Crowley? Or do they have a product manager who just started thinking about location-based services eight months ago? That PM may have 600 million users and every engineering resource they desire, but they haven’t spent the last ten years thinking about how to get people more engaged with their cities via their mobile devices. The difference between Dennis Crowley and that Facebook PM is the Passion Gap.

The Passion Gap is evident when you see a founder or product manager so deeply engaged in their product that they can’t help but think about it all the time, and, as a result, they see all the fine details that are required to make a product that exactly matches what the market needs. This is true even when the market hasn’t yet realized the need.

Another great demonstration of the impact of the Passion Gap is the difference between StackOverflow and Experts Exchange. StackOverflow is amazing, and we should all be thankful to Jeff Atwood for creating it. But Experts Exchange is basically the same product idea, created years before, with an entrenched user base. And yet, somehow the user experience at Experts Exchange is all friction, whereas StackExchange just hums. If you’ve read any of Coding Horror, it makes perfect sense - Jeff Atwood has a deep passion for making every software engineer out there better at what they do. Of course someone with that passion was going to be the one to make a tremendously useful knowledge-sharing site for software engineers!

So I think of the Passion Gap whenever someone claims that a successful startup has “no defensible technology”. For some of the most interesting companies out there right now, the key bits of technology are not in some single large algorithmic piece, but rather in dozens of fine-grained product choices that make a total experience. The people who accuse those products of lacking defensible technology are taking a one-dimensional view of the product. It’s like we’re all color-blind to the features that make the product hum, and yet these highly passionate founders see the colors we’re completely unaware of.

Just to enumerate some other examples of where I see the Passion Gap at work:

Andrew Mason (Groupon) - Andrew’s startup prior to Groupon was The Point; it was a startup for social causes where one person acting alone could not have an impact. I would have to guess that Andrew Mason's passion isn’t about marketing local businesses, as much as it is about leveraging the power of groups of people acting together. Certainly, we have to assume that his time spent thinking about the dynamics of group action while working on The Point provides a nice competitive advantage to Groupon.
Mark Zuckerberg - the Time Man of the Year profile of Zuckerberg does an excellent job of showing just how passionate Zuckerberg is for thinking about personal relationships and how to extend them with technology. Remember for a moment that when Facebook first started to get big, many people thought it was just another wave in the Friendster/Myspace ebb and flow of social networking sites. I see Zuckerberg's passion as key to how Facebook achieved and maintained dominance.
Steve Jobs - Steve is an interesting case, because his passion, by observation, seems to be more about beauty in technology rather than any particular product application. And yet, Apple consistently puts out products where the beauty of the thing itself is a huge part of the sales appeal. The Passion Gap is particularly well-demonstrated when competitors set out to copy and improve upon Apple’s category killers, for example, the Zune.

The most common way that people talk about the Passion Gap is when they advise you to “start a company that scratches your own itch”. I posit that the underlying logic in that advice is that the best startup you can create is one where you will be constantly engaged in thinking about improving the product, maximizing the user experience, and planning for the future -where you have real passion for making it work.

Anyway, the next time you see a startup with “no defensible technology”, take a look at the founder, analyze their passions, and consider whether their defense lies in the Passion Gap.

Addendum: Please check out the follow-up post to this one, The Dark Side of Passion.

I'm calling it now: The engineering talent bubble

2010-11-12T08:59:00.002-05:00

My 6th grade composition teacher would dock me 10 points for throwing my thesis right up into the title, but I just want to go out there and call it now: we are in the middle of a bubble of valuations for individual software engineers.

What do I mean by bubble? For my purposes, a bubble exists when prices of some commodity are increasing at a blistering pace that becomes disconnected from reality. Most of the time we fail to recognize a bubble because we think "this time it's different." This time is NOT different.

What are the signs of the bubble? Two key things I see include:

Google paying outrageous retention bonuses to keep people from leaving for Facebook (see here)
Startups, less than two years old, with very little user traction and no revenue, getting acquired simply for talent, at valuations that give even the front-line engineers a seven-figure outcome. (I won't provide any links here because I don't want to point too many fingers)

Maybe even more importantly, though, the key anecdotal indicator of a bubble is when it becomes "common sense" that prices will always go up or sustain the current market. That is what I am seeing more and more every day now - software engineers saying they're sure they can start a "company", sit in their apartment for a year, and then get acquired by Facebook for $10 million.

The unfortunate thing about a bubble is that it always bursts. You can argue that this bubble won't burst, because the number of software engineers is relatively fixed, and the demand is going up pretty drastically these days. However, a few years ago, people argued for the real estate bubble by saying "they aren't making any more land". That bubble didn't burst because supply expanded, it burst because demand collapsed as the ability to pay fell through the floor.

This is going to end, and badly. There are a lot of engineers who are going to make serious life choices based on current trends, and in a year or two, when this blows up, they will find themselves unemployed and without savings.

Please Drive Your Kids to School

2009-09-15T08:08:00.005-04:00

The article in Sunday's Times about driving your kids to school has generated a lot of discussion. I just posted a mini-rant in Reader about this, and figured it was worth re-posting here.

As a parent and a computer scientist, to me this comes down to basic probability theory. The idea of the "only 115 abductions a year" stat is to make it sound as though the chances are so low that it's not worth protecting against.

As we all remember from discrete math, though, you don't care about the probability, you care about the expected value. As a parent, the cost of losing a child is, for all practical purposes, infinite. So any "lose my child" event with non-zero probability becomes worth preventing against. The cost of driving a kid to school each day is astonishingly low; there's no point in NOT doing it.

The author is basically appealling to the sentiment of "why aren't things the same as when I was a I kid?" To them I say: the world changes, get over yourself.

This is not meant to be an argument for protecting kids against all possible dangers, but driving your kids to school is pretty much a no brainer. It's right up there with "should I buy my kid a car seat or not?"

By the way, most states these days have much stricter car seat and seat belt laws for children than when we were kids. My kids will never know the joy of bouncing around, unrestrained, in the way back of an '85 Oldsmobile wagon. That's life.

Using Puzzles in Interviews

2009-07-24T10:44:00.005-04:00

Albert Wenger has a great post up on Continuations about Using Puzzles in Interviews. He makes a number of good points, most importantly:

1. Puzzles should be just one of many different parts of the interview.

2. Try to find a set of puzzles that works for you and use the same puzzles with many candidates to get better comparability.

3. Focus primarily on approach and behavior and only secondarily on outcome.

4. Unless the position specifically calls for ability to think under stress (e.g. certain ops positions), try to put the candidate at ease. Being in an interview is stressful for many folks.

5. Ask for a vocalization or visualization of the thought process.

I agree with most of this approach, but I still feel that everyone needs to be very cautious about using puzzles that are basically brain-teasers. In the first comment on Albert's post, one of the readers made part of my point for me - they posted a potential puzzle, and it was immediately pointed out that is well known amongst the brain-teaser set. I posted a fairly lengthy response on Albert's blog, but I figured I might as well cross-post it here.

At the core, the most important distinction is between coding/algorithm puzzles and brain-teaser puzzles.

E.g., one of the best interview questions I've ever been asked basically boiled down to performing bipartite graph matching - it required deep understanding of algorithms and how to match them to real-world problems, and then being able to implement the code behind them - pure awesome.

On the other hand, some of the worst interview questions I've encountered are the form of "N pirates have to divide X gold among themselves" or "You're blindfolded in a room wearing a hat..."

Brain-teaser questions tend to have the drawback that they are very uneven - for people who love brain-teasers, many brain-teasers are well-known or simple variations on existing problems, but for people who don't, they're starting without the benefit of any of the background.

In addition, in my time at Google I saw heavy anecdotal evidence that brain-teaser aptitude correlates negatively with overall effectiveness as a developer. Sometimes the developer who searches too hard for clever solutions winds up missing the simple solutions.

ImportError: No module named django.core (fun with Windows)

2009-07-09T16:48:00.004-04:00

I just spent 45 minutes tracking this down, hopefully if I post a trace here, others will be able to resolve the issue more quickly in the future.

So I am building a little django app at work, but my latest installed version of django is pre-1.0, so why not upgrade, right?

After getting everything downloaded and running setup.py, I did the standard thing to start a new project:

C:\Users\jtb\Documents\Clickable\src>django-admin.py startproject myproject

Traceback (most recent call last):

File "C:\Python25\Scripts\django-admin.py", line 2, in

from django.core import management

ImportError: No module named django.core

All the search results on that error pointed to issues with my PYTHONPATH, but no amount of tinkering with it would make django actually run.

The real problem was that, due to a recent installation of Python 2.6, my PATH and my PYTHONPATH had fallen out of sync. My PATH was pointing to the Python 2.5 directories, while my PYTHONPATH was pointing to the 2.6 directories. It seems the Python installer I used knows how to update PYTHONPATH but not PATH. Which makes sense.

If you found this post through a web search and it helped you debug a similar issue, please post a comment.

Xoogler Tip: What happens to your email after you leave?

2008-09-19T09:57:00.004-04:00

For all you Xooglers out there, here's a little tip to smooth the transition outside of the 'plex.

When you leave Google, your existing mail account becomes a black hole - all mail sent to the address silently goes away. There's no bouncing or auto-responder. No big deal, right? Anyone at Google can probably get your new address off go/epitaph.

BUT: If, like me, you worked with anyone outside of Google (for example, patent attorneys), they might not know about your departure, and they can haplessly send email into the black hole for months without discovering that you're not there any more. I found this out recently when a lawyer who had been trying to reach me all summer finally found out that I had left the company.

Simple fix: before you leave, you can file a ticket to have an auto-responder set up. If you've already left, your HR Business Partner can help you out.

Happy Anniversary to Me!

2008-05-30T10:22:00.000-04:00

I've been awfully silent for the past little while, and for no good reason, really. But today is my 1-month anniversary at my new company, Clickable, and I thought I'd take this chance to share a little about this transition.

Clickable is an exciting place; we are building a software-as-a-service offering for people who want to manage their online advertising in a single, simple interface. So if you are advertising with Google, Yahoo, and MSN, you can learn all of those products and spend time in all three of them managing and optimizing your accounts, or you can just log in to Clickable and do all your work in one spot. The team at Clickable is immensely talented, and the product has real customers who find our product to be a big advantage over the services offered from the ad networks.

I'm proud to say I have definitely found what I was looking for by leaving Google for a startup: tons of responsibility, the opportunity to have a major impact on a company, and the chance to learn a lot of new skills. It's given me an enthusiasm for my work that was sorely lacking towards the end of my time at Google.

I'm working as a semi-specialized engineering jack-of-all-trades. My main focus is our optimization engine, which tells customers how to tweak their accounts for the best advertising results. But I'm also working on a broad swath of other tasks, such as setting product direction, production firefighting, integration testing, customer analytics, and even a dash of business development. Most importantly of all, I am hiring for talented engineers. If you think the startup life is worth a try, leave a comment or shoot me an email.

What am I doing wrong with Reader off-line mode?

2008-05-02T09:14:00.008-04:00

I'm temporarily commuting with a laptop that has no WLAN device, so I've been using Google Reader's off-line mode for the last couple days, and I've noticed the following oddness.

I start out with a giant backlog of unread feeds, a la:

(Sorry for the quality of the screenshots in this post; if you click through to the original image on any of these, you should be able to see the folder counts clearly).

I synchronize to go off-line:

Once that's complete, all my folders show fewer unread items than before, such that my total unread drops from "1000+" to 580:

Such rapid progress! Until I go back on-line again:

And now I'm back to "1000+".

Anyone on the Reader team care to comment on this one?

Today is my last day at Google

2008-04-11T07:09:00.002-04:00

After nearly five years at Google, I am leaving the company, effective at the end of the day today. My time at Google afforded a tremendous number of opportunities; I was able to take part in growing the company's first remote engineering office from 5 engineers in 2003 to nearly 450 today, I was able to conduct blue-sky exploration of applying Google's data and computing infrastructure to web information extraction, and, more recently, I was able to help build an all-star team that just shipped Google Ad Manager.

Contrary to what glowing articles in Fortune or Fast Company might tell you, the best part of working at Google has nothing to do with the free meals, subsidized massages, or even the stock options. The best part of working at Google has been the chance to collaborate, to innovate, and to explore with the most talented collection of people I could ever imagine. Missing those people every day is the hardest part of leaving this company.

I don't know yet what my next challenge will be, but I hope to go somewhere very small and take part in building a company from the ground up. Wish me luck!

Followup: which skills to list on your first resume

2008-03-17T20:06:00.002-04:00

Over at techinterviews.in, they've picked up on my recent series about writing your first resume and posted some good comments, including:

There’s nothing like picking a language or operating system or database management system that a candidate listed towards the end of his resume, and having an interviewer, who’s the expert in the field (”Ah! So you did Nintendo DS assembly programming, too! Let’s discuss this in detail!”) then discover that the knowledge of this listed tech is, to say the least, limited. It makes the whole resume look like a creative writing projects, the aim of which is to specify as many buzzwords as possible.

I do advise undergrads to be a little liberal in listing technical skills, and I want to take the time to explain that a more clearly.

Here's an example from my own experience. One of my undergraduate research jobs was with a research group studying human cognition, primarily via a cognitive modeling system built in LISP. I joined the group in my sophomore year and stayed on through graduation, during which time I wrote several hundred lines of LISP glue code to manipulate my models and experiments. By the time I was interviewing for jobs, I was very comfortable writing complex LISP code and tackling problems with a functional approach. However, I knew very little about the overall design of the language or some of the important big concepts like CLOS. But I still listed LISP on my resume.

I recently had cause to write some LISP code for the first time in 10 years, and I was still able to work effectively in the language. But I wouldn't even consider listing LISP on my resume these days, because it's far outside the core of my expertise.

On the other hand, my first research job in college involved writing a mix of Fortran and C code for a parallel supercomputing group. In the year or so I worked with that team, I probably wrote a few hundred lines of Fortran code, and it was ugly. In my senior year, I left Fortran off my resume - if someone had asked me to write even a line of Fortran code, I would have been helpless. In any event, I knew I didn't want to work for a company that would be excited by my experience with Fortran.

In brief, when someone fresh out of school lists a particular technical skill, my expectation is that it conveys less expertise than when someone 10 years out lists the same skill.

If you really want to be precise in listing technical skills, the best approach is to quantify your level of experience. Some people take the straightforward approach of listing the number of years they've worked with a technology. I find it more useful to communicate a self-assessment of your skill level, using phrases such as "expertise in ..." and "familiar with ..."

But if you're just coming out of your undergraduate program, list anything where you think you can speak intelligently about the technology. If an interviewer asks a question that goes beyond your expertise, admit it, and offer a quick description of how you've used that technology and what areas you're best suited to answer questions about.

The CS undergrads' guide to writing your first resume, part 3

2008-03-12T08:50:00.002-04:00

Hopefully my first two posts in this series have given you a good idea about how to start your resume, and what to emphasize and downplay. By now you've got a good draft, and you've had it reviewed by a number of friends. What's next? Take a pass over this list and make sure you're not committing any cardinal sins.

Some resume mistakes are so bad that you must never commit them. Please. The ones that always set me off are:

Multiple pages. I've have reviewed hundreds, if not thousands, of resumes, and in all those examples I have seen only one multi-page resume come with a strong candidate behind it. All the other ones, the ones where people list every class project, or every club they've been in, or the names of all their teachers in high school? Worthless. Take the time to distill your education and experience into the points that can be expressed in a single page of text. If you can't get it down that far, hand a draft to one of your editors and ask them what's least interesting.
Spelling mistakes. See my earlier comments about having several other people read your resume before you share it with any companies you're interested in. When I see a really egregious spelling mistake, I will discard the resume completely.
Using the Microsoft Word resume template. First off, the template just doesn't look that great, and second, it sends the message "I just put this together in five minutes because that's about how much I care about your company. Please pretend not to notice." I accept that page layout can be hard, but if you can't create something that looks decent, ask a friend for help. The layout for my first resume came from a good friend who actually has a talent for these things, and I used it successfully for the next several years.
A non-CS undergraduate degree followed by a CS Master's degree from either (a) a no-name computer science program, of (b) a top-tier school that is known to be a diploma mill for anyone with the money to buy a degree. I'm not naming names here, but you know who you are.

Lastly, I just want to touch on the long-standing debate about whether or not to include an "Objective" at the top of your resume. In the modern era, most resumes are submitted directly for one or a small number of job openings, so I'm going to assume that your objective is to get hired for one of those positions. Including a comment in your resume to tell me you want a career like the one I have to offer you only wastes space that you could be using to tell me more about your background and expertise. I say skip the objective, but you can decide what works best for you.

Thanks for reading this far. If you found this guide useful, please leave a comment.

Please keep your laptop open in meetings

2008-03-11T10:06:00.002-04:00

Jeremy Zawodny has mixed but generally positive feelings about not having laptops open in meetings, but I think he's actually going after the wrong point. What you really want is a "no distractions in meetings" rule - working on email, thinking about your vacation plans, or what you'd rather be doing at the time - are all detrimental to the effectiveness of a meeting.

If you can avoid distractions like email or feed readers, a laptop is an invaluable tool in making a meeting more effective. Almost every useful meeting I sit in these days involves constant references to documents stored in the cloud, real-time scheduling for follow-on conversations, and back-channel talk among participants. All of these, obviously, require a network-connected laptop available throughout the meeting.

Allowing people to keep their laptops open requires a certain amount of trust; namely, you're trusting that people will use their laptops to further the goals of the meeting, not to be virtually elsewhere while others are counting on your input and feedback. But a laptop is a great tool; let's not throw out the good with the bad.

The CS undergrads' guide to writing your first resume, part 2

2008-03-11T06:36:00.002-04:00

Ideally yesterday's post gave you the basic structure for your first resume. Today I'll give some more details on what to emphasize, and what to downplay if you want me to read your resume and call you for an interview.

The Top Things I Look For In Your Resume

A degree in Computer Science. I see lots of resumes from people with degree names like "Information and Decision Systems" or "Technology Management". I won't discount the value of those degrees, but for the kind of software development I do, my experience has been that a rock-solid understanding of computer science is the cost of admission. I'll always award bonus points for a minor or a second major that reflects to your interests and could be applied in your work.
A good GPA. For many schools, especially the top-tier schools, this means roughly something over a 3.5. I know you think college is hard and I don't understand your courseload and blah blah blah, but the fact of the matter is, a large part of your success in college is determined by your ability to manage and schedule your time and your work. If you had a B average, my first hypothesis is that you can't manage your own time, and transitioning to the professional world isn't going to make that problem magically go away.
Did you do more than just go to class? This is where you really set yourself apart from every other graduating senior with a degree in Computer Science. If you think you went to college for the classes, you really wasted the other 158 hours of every week.
As a follow-on to the last point: did you demonstrate leadership talent outside of class? Most people shouldn't be President of Student Government, but you can be Vice President of Students for the Exploration and Development of Space. Heck, I was. I'm not saying that I only want to hire leaders, but I am interested in people who are willing to step up and volunteer their time and energy to do something larger than themselves. These are the people who see a problem and attack it, not just complain about it. Even better than taking on a volunteer leadership role is taking on an elected leadership role - it demonstrates that other people look to you as a leader.
Interesting projects. For me, if I read a project description on your resume of something I don't know how to do myself, I'll want to talk to you just so I can learn more about that project. Translation: while you're in school, look for opportunities to do hard projects. The homework assignments in your intro CS class aren't going to cut it, and the final project in Operating Systems probably won't, either. Get involved in a research group, or start something with your friends, but set out to tackle something difficult. Great big warning: if you list a project on your resume, be ready to answer questions about any aspect of it. Any hint of bullshit on this point will taint the outcome of your interview.

I appreciate that this list is completely useless if you're about to graduate with a C average in Database Systems, having done nothing outside of the classroom except get high. Sorry.

Also, and this is not critical, but please, pretty please, use a format that will work on my platform. Since you have no idea what platform I'm using, go with something nice and portable. PDF is still the gold standard, though I'd be very impressed by anyone who sent me a resume by sharing a Google Document. Plain text works as well, though it severely limits your formatting choices. Whatever you do, just stay away from MS Word - almost any fancy formatting you do in word will not translate correctly into, for example, a Linux-based word processor like OpenOffice. I have a fixed amount of time to review your resume, and if I spend most of that time piecing together bits of text that wound up splayed across several pages, I'm probably not going to notice that hidden gem of a project you did sophomore year.

I've found a common problem on resumes is to really drum up things that turn out not to be that important. This is based entirely on my experience on the observed correlation between background and impact as an engineer. To wit, here are ...

The Top Things I Don't Look For In Your Resume

An extremely high GPA. Once you go beyond the 3.5 - 3.75 range, there's a leveling effect; in my experience the kid who got a 4.0 is just as likely to be good at writing software as the kid with a 3.75.
A top-tier school. There are a huge number of benefits to going to one of the three or four "best" CS schools in the country, but they seem to have little predictive power in the quality of the work someone will do professionally. I've known too many screwups from MIT and CMU, and too many rock stars from lesser-known schools, to care too much about where you got your degree.

Caveat: Sure, if you went to CMU, that's an added bonus for me, because I know we immediately have some common background and we can swap stories about Mark Stehlik or late nights spent in Wean. But that's still not going to get you a job, and if the rest of your resume is weak, it might not even get you an interview.

Some minor contribution to some minor open-source project. Open source is great, and as an undergrad, contributing to an open source project is a great way to get some exposure to real software development. However, your two-line contribution to libsecondlife, correcting the spelling of "Linden" in the comments, is really not going to get me excited, so please let's just skip it.

Obviously none of the items on this list should be considered a negative; I'm just saying that don't waste a few dozen words extolling the virtues of MIT, or trying to convince me of the quality of your non-MIT school. Tomorrow we'll cover the real pitfalls that destroy a lot of otherwise strong resumes.

The CS undergrads' guide to writing your first resume

2008-03-09T21:04:00.004-04:00

Continuing on the theme of bridging the gap between an undergraduate CS degree and a career in software development, I want to talk today a little about resumes.

What a Resume is Good For

The most common mistake people make with their resume, young and old, is to write a resume that makes the case for hiring you. No smart company is ever going to hire you on the basis of your resume, so don't even try. Your goal is to have me read your resume and think, "Hey, this person sounds really interesting, I should set up a phone call with them." And that's it. If you want to, you could really stop reading here - once you understand the critical purpose of a resume, everything else is just details. So let's really drive the point home: a resume's only purpose is to get someone, anyone, to read it and be interested in you.

If you're still with me, let's start things off with...

How to Write Your First Resume

Put your name and contact information at the top of the page, in a font larger than anything else you'll put on the page.
Start a section called "Education", and include:

the name of your college or university,
the name of your degree program,
and your GPA. Please consider your GPA to be mandatory. Your career counselor will likely advise you not to list your GPA if it's below some threshold (usually a 3.0), but please go ahead and list it anyway. It's easier for all of us if a weak GPA can be weighed against all the other information on your resume at the same time. To put it another way: if you have a really strong resume with no GPA listed, you leave a big gnawing unknown out in the ether. If you at least list your 2.8 GPA and then go on to tell me about a whole bunch of awesome stuff you've done, I can make an informed decision about whether to proceed; if your GPA isn't on your resume, I'm likely to pass it over without ever talking to you.

Next, list the most interesting and challenging CS courses you've taken. You're shooting for 3-5 here, and if they don't have names I'd intuitively understand (ex: "How to Think Like a Computer Scientist") please do me the favor of translating them ("Discrete Math and Algorithms").*
Create a section called "Experience" and write down everything you've done that resembles a non-academic programming position. Personal projects, summer internships, research projects, everything can go in here. This is the meat of your resume, and where you're really likely to hook me into wanting to learn more.
Create a section called "Technical skills" and list every language, operating system, database, and development platform you're competent in. This means leaving off the stuff that you experimented with that one weekend freshman year. I want you to be able to tell me some of the ins and outs of any of the items in this section. I don't expect seniors in college to be experts in anything, so don't be too conservative, but you want to avoid setting off my bullshit meter [addendum].
Finally, in a section called "Activities and Awards", list the most important activities you've participated in and awards you've received. This should be much shorter than your "Experience" section, but don't leave it off completely.
Find someone to read your resume and give you fair and honest feedback. Start with a classmate. If you can, find a recent alum from your program and ask them to review it. Show it to your advisor. You can't have too many people read and comment on your resume. Listen to what they say, and incorporate their feedback.

Still with me so far? Great! Keep reading for basic tips on what to emphasize and the cardinal sins to avoid that will take this recipe and turn it into a winning resume.

* Real example taken from the Carnegie Mellon undergraduate curriculum circa 1996.

The Do-it-Yourself Degree in Software Engineering: Bonus Round

2008-02-01T11:58:00.000-05:00

So it’s six months later, you’re a coding rockstar, you know how to run a team, your organization follows the principles laid out in Peopleware, and now you’re wondering, “Hey, what’s next?” Here are some great reads that are illuminating but not part of the required curriculum. I consider everything I’ve posted earlier as the basic cost of doing business as a professional software developer. These are just for fun, but still very useful in making you an excellent engineer.

Dreaming in Code presents a sweeping view of the history of software engineering, told through the lens of Mitch Kapor and the Open Source Application Foundation’s efforts at developing Chandler, the Personal-Information-Manager-That-Will-Totally Change-Your-Life-If-We-Could-Ever-Ship-Running-Code. It brings together many of the lessons of the Mythical Man Month and Peopleware in the context of a real effort at building a new product.

Go To is another great history of software, focusing more on the sequence of technical developments between World War II and where we find ourselves today. The exciting running thread throughout the book is the number of great advances that have been made by individuals building the exact tool to suit their own needs. If there was every a compelling argument against software design by committee, this is it.

The Innovator’s Dilemma is an excellent read because it shows over and over how entrenched players in a market are unseated by small teams working with great focus on an emerging market or technology. Every time someone tries to convince me that Google can never be unseated from a position of dominance in search, or that Microsoft and Yahoo are the companies we should be concerned about, I tell them to read this book. For a big company, the competition isn’t the hundred-billion-dollar software shop down the road, it’s a few folks in an apartment somewhere using sheets of plywood for desks and borrowed hardware.

There’s a great flipside to the Innovator’s Dilemma, too: if you’re thinking of building something, and you think it’s going to go up against some established megacorp, take inspiration from the number of times that some unproven David has taken down a Goliath.

Lastly, In Search of Stupidity presents a unique perspective on how so many software companies managed to compete with Microsoft and fail completely. The underlying thesis of the book is that Microsoft isn’t necessarily that great, but everyone else is so bad; Microsoft happens to be the one software company that didn’t make any horrendously stupid mistakes in the eighties or early nineties. You can debate that thesis, but it’s still entertaining to read some of the brain-dead things that Microsoft competitors did.

This is now the final final conclusion of my list. Please let me know if you end up reading any of these on my recommendation, and let me know what you think of the books. Enjoy.

The Do-it-Yourself Degree in Software Engineering: Software Engineering Management

2008-02-01T11:03:00.000-05:00

Congratulations on becoming a coding rock star! You can do anything you set your mind to. You might not be ten times more productive than the average engineer, but you’re probably six or seven times more productive.

This now puts you in an unstable equilibrium. Sooner or later someone is going to notice how good you are at building software, and they’ll ask you to take on some form of leadership or management role, in which you’ll be primarily responsible for getting other people to build software. Welcome to a life spent in meetings, doing “coordination.”

In all seriousness, even if you never get asked to manage other developers, understanding the principles of how to manage software teams will make you a better engineer. These two books are each a quick read and will equip you with an entire toolbox of ideas about how to make teams work well.

The first book to read is The Mythical Man Month, the classic software engineering text by Fred Brooks. There are any number of things that you can disagree with in Brooks’ treatise, but the logic of Brooks’ Law is fundamental in understanding how to deal with late projects. It’s been 40 years since he made the excellent argument that adding more people to a late project makes it later, and yet, even at enlightened software companies like Google, the first reaction to a late project is to add more engineers.

Once you’ve read Mythical Man Month, and you can argue successfully with your manager about how to make the right tradeoffs between schedule, quality, and features, it’s time to learn a little about how successful software organizations treat their people. For this, you need to pick up the classic Peopleware by DeMarco and Lister. A lot of people espouse theories about how to operate software organizations; these guys did the research and the experimentation to demonstrate what makes a difference between the mediocre software shops and the stellar operations.

I am proud to say that I no longer own a copy of Peopleware. Back in early 2003, when Sergey would still come out to NYC and hang out with the nascent engineering team out here, I managed to get him into a debate about running engineering organizations, and I learned that he had never read Peopleware. The next day I pressed my copy into his hands, and told him to just take it, as long as he promised to read it. We still don’t have private offices at Google, so I’m not sure if he ever actually read it, but at least I tried.

Congratulations on making it this far. Upon finishing these books, you will now be qualified for a degree in Software Engineering in the Real World. Tomorrow, just for kicks, I'll share a few more interesting reads that wrap up many of these ideas in a fun package.

The Do-it-Yourself Degree in Software Engineering: Complementary Skills

2008-01-30T21:58:00.000-05:00

The first three books on my list help you build great code, use and understand patterns, and even refactor the legacy code you got saddled with. What are the skills you need to be able to package that up and have a real impact?

You need to be able to write. You need to be able to write clearly and succinctly about technical issues. No interesting software is ever built by a team without real debate about the design, and those debates usually happen in written text, via email or design documents. If you want to see your ideas advance, you need to be able to write in a manner that is clear and persuasive.

Unfortunately I can’t point you to a text on how to write about technical matters. The best thing I can do is to point you to the archetype of clear, succinct technical writing, The C Programming Language, by Kernighan and Ritchie. This book provides the model every technical writer should use when trying to explain a new technology or concept to technical readers.

You need to be able to work on user interfaces. This is the hardest one for me, because I have no idea how to arrange pixels on a screen to convey meaning. It drives me nuts every time I try. Given a choice, I will hand off UI design to a trained professional every chance I get. But the fact is, skilled UI designers are exceptionally rare, and a service with a working UI is 100 times more meaningful than some script outputting text on a console. Most engineers will never be able to build beautiful interfaces, but User Interface Design for Programmers by Joel Spolsky is a great beginner’s guide.

You need to know how to think about hard problems in Computer Science. This is a fuzzy concept to describe, but what I really mean to say is, you must read Programming Pearls by Jon Bentley. And when you read it, do the freaking exercises. That’s what really matters. I always pose the same challenge about this book when I speak to undergrads: in the chapter on proving correctness of programs, find the bug in the program that is proven correct.

Coming up tomorrow: how to learn everything you need to know about software engineering management.

The Do-it-Yourself Degree in Software Engineering: Code Construction and Design

2008-01-30T21:42:00.001-05:00

Picking up where I left off yesterday, what are the code construction and design books you need to read in order to become a practicing software engineer?

There's one absolutely critical task in building software, and that's creating the actual sequence of operations that comprise your software. If you can't get this right, you're nowhere.

Short tangent: think back with me for a moment to the first computer science class you ever took – that one in high school, with the hand-me-down PCs from the typing lab, and the part-time Math/part-time CS teacher. Ah, the heady days when making “hello world” appear on the screen seemed like magic, and HTML felt like a “programming language”.

I’m willing to bet that one of the first lessons you learned was about “routines” – nice, isolated bits of code that did one thing and did it well, bits that could be incorporated into other bits with a simple call. Boom, magic.

There’s always some smart student in the room who asks, “when do I use a routine and when should I just write the code in-line with everything else I’m doing?”

This question has one standard answer given in CS classes all over the world: “write a routine when you’re creating a piece of code that is going to get used frequently.” And for introductory CS classes, for the kind of code written by 16-year-olds, this is a fine enough answer. But for real software, with real users, and more than one developer, there are a lot more reasons to put a piece of code in its own routine. If you can’t name at least three, you need to read the beginning of chapter 7 of Code Complete. And that’s just three pages out of this 800+ page tome! (Aside: by a wide margin, Code Complete is the longest book on my list).

If you read nothing else on this list, please, for the sake of software, please read Code Complete. This book is jam-packed with hundreds of little tips that will make your code better, one line at a time

Code Complete will equip you with the skills to start thinking at a high-level about the organization and construction of code. There are two books I recommend here.

First, Design Patterns. A lot of developers like to write off Design Patterns as some pointy-haired fluff, but the fact is, in the 14 years since this book was first published, Design Patterns have become established as a useful shorthand for discussing high-order organization of code. If nothing else, you are likely to encounter other people using Design Patterns when they describe their code, and you need to understand the language in order to communicate with them.

There is a risk associated with reading Design Patterns - many people read this book and decide that patterns are the meaning of life itself. I am not advocating that you become a Design Patterns weenie, searching for ways to needlessly incorporate a Strategy or a Factory in your code, or, pure Design Pattern nerd-vana, a StrategyFactory. Nonetheless, there are a lot of fantastic ideas in this book, and your code will be easier to maintain, by yourself and others, if you apply the ideas espoused in Design Patterns.

Next, and finishing out the Code Construction and Design category, is Refactoring, by Martin Fowler. This book is fantastic for the genuine focus on a codebase as dynamic and evolving. For most computer science graduates, working on an existing codebase is completely foreign in academic settings, and one of the first experiences in the professional setting. The ability to think clearly about how to improve the design of existing code, while making strong assurances about the quality of the refactored product, makes an immense difference in your ability to work effectively with software.

Once you're through these three books, the raw quality of your code will be immensely better. Code Complete is the longest, but it's a fairly quick read. The other two should be easy to get through in free time.

Coming up tomorrow: now that you're writing great code, we'll talk about the skills you need to go along with that in order to round out your abilities.

The Do-it-Yourself Degree in Software Engineering: Intro

2008-01-30T21:40:00.000-05:00

Depending on where you go to school, your degree in computer science can be useful for a lot of goals – becoming a professional developer, going into finance or biotech, getting a Ph.D. in computer science, becoming an IT project manager, or even something barely related to computer science. The one thing a CS degree is almost useless for is practical training in how to build production-quality software. The curriculum at most universities, especially the “top tier” universities, is (rightly) focused on an academic grounding in the field, not a practical guide to building great software.

I’ve been trying to teach undergrads about this gap between a Computer Science degree and professional software development in a talk that I’ve given twice now, once at Princeton and once at Carnegie Mellon. The kids seem to like it, and the number one question I get is for the reading list I mention.

My list is intentionally brief. Anyone with a degree and some spare time should be able to make it through the entire set in under six months. If you haven't read these books, and I mean all of these books, you are probably building bad software. When people talk about the 10:1 productivity difference between great developers and average developers, well, you're on the "1" side of that ratio. Take the time to read through these and everyone impacted by your software will benefit.

Just for the sake of your ease in reading, I’ve taken this list and broken it up over several posts. I’ll post the first portion of the list, code construction and design, tomorrow, followed by a few books on complementary skills, then books on software engineering management, and finish up with a short bonus of fun and engaging books that help fill the gap between a degree in computer science and professional software development.

All JTB, All The Time

2007-12-04T22:11:00.000-05:00

Hey, if you are one of the 5 people on the planet who subscribe to this blog, you might be interested in my tumblr stream. Tumblr is an interesting not-quite-blogging, not-quite-aggregation product that's also better than either style of product might give you.

By watching my stream you'll get my:

posts to this blog
tweets
del.icio.us bookmarks
Google Reader shared items
posts actually native to tumblr, including this great one

You could also friend me or follow me or whatever the tumblr term is, and see me on your tumblr dashboard.

Experimenting with Custom Search Engines

2007-07-01T16:13:00.001-04:00

I've just been playing around with Google Co-op's "Custom Search Engine" feature. I'm interested to see if it's possible to build a decent topic-specific search engine given the controls exposed in that product (mainly, url whitelisting).

The test topic area I'm starting with is, not surprisingly, software engineering. I've added a half dozen sites, mostly blogs, related to software and software engineering. You can try it out right here, using the box there on the right.

One of the more interesting mechanisms is the ability to invite collaborators to contribute sites. If you read this blog and you have sites you recommend for software-related information, drop me a line or post a comment, and I will add you to the collaborators.

(Program note: while adding the custom search box, I also re-enabled AdSense on this blog, because it seems to have disappeared.)

Somebody noticed GoogleLookup!

2007-06-22T08:55:00.001-04:00

Over at Google Blogoscoped, they just noticed that the Google Q&A data is exposed in Google Spreadsheets. This has actually been out for quite a while, but it never got much press.

In my mind, this kind of application was always far more interesting than the Q&A onebox. It's nice to give people quick responses to simple fact-finding queries, but it's far more interesting to aggregate, mashup, and expose data that they can re-use themselves. The spreadsheet integration is just one example of that sort of effort.

In addition to the data, it's an exciting demonstration of what you can do with a spreadsheet product that is native to the web. There are a lot of people with big ideas about where this can go from here.

Regarding the question about which queries trigger the Q&A onebox: it's based on a corpus that gets updated over time; as the scoring functions and data coverage change, you can expect to see certain queries start or stop triggering over time. I assure you it's still under development.

In case you missed it: timeline and maps views on Google Labs

2007-05-17T08:33:00.000-04:00

All of today's press about Universal Search is overshadowing a very important launch -- timeline and map visualizations of search results. This product really demonstrates the power of digging into the structured information that lives on the web. The best thing about it is, the information that goes into these visualizations was not designed for this use. Whereas the Semantic Web folks have always envisioned getting the entire world to author RDF repositories, the Google Views product shows just how much you can do by reverse-engineering from the information that's already published.

If I sound a little enthusiastic about this, it's because I worked on similar projects for a long time at Google. The entire team over there has worked through an immense number of technical challenges to get to this launch. The technology stack they've built for understanding information published on the web is really very impressive.

My favorite use of Views is to explore historical topics. Want to understand the history of Iraq, or al Qaeda? It's too damn easy.

Congratulations to everyone involved in this launch. What's next? :)

Google culture

2007-05-03T18:03:00.000-04:00

One of the great things about Google culture is that anyone can fix any problem.

Sure, it's a big place, and some people get set in their ways, and you might have to convince a lot of people that your solution is the right one. Still, no one ever makes the argument, "that's not your area" or "I don't need to listen to a peon like you." Every time I've mailed one of our senior managers with a suggestion or request, they've responded more promptly than I would have, and with detailed feedback or comments.

In short, many of the best things at Google happen because someone in the trenches had a great idea, they made it work, and they convinced a lot of people to change how things are done.

And of course, that is great.

What's bugging me these days is that a lot of people are trying to describe Google culture as "bottoms up," when they actually mean "bottom up." Bottoms up means a whole different thing.

So, to all you Googlers throwing around the phrase "bottoms up" like it's a good thing: please stop. It hurts my ears when you say it.

Random dork humor

2007-03-15T01:04:00.000-04:00

Spotted on CA 101, the following license plate:

NACLHHO

Think about it for a minute, it'll come to you.

Congrats to the team behind "Places Mentioned in this Book"

2007-01-25T09:40:00.000-05:00

One of the great things about working at Google is the interesting kinds of data we have available, and the sheer quantity of it. It enables a lot of cutting-edge experiments, especially when you take the cross product of different areas of expertise. For example, Google has developed a lot of good technology for recognizing mentions of place names in text, as you might expect would be necessary for Local Search. Additionally, we've scanned and OCR'd a lot of books, as you might expect would be necessary for Book Search.

Today, a few friends of mine launched an exciting new feature of Book Search that exposes the result of applying our location-identification technology to the corpus of books. For a lot of books, the "about this book" page now includes a Google Map annotated with places mentioned in the book, where each annotation actually gives you the snippet of text that mentions that place. I've had a lot of fun exploring the results on some of my favorite books.

This is one of those interesting software products you really can only build at few companies, and I'm proud that Google is one of them.

Check out the new stuff over on the right-hand side

2007-01-18T09:28:00.000-05:00

I'm playing some with Google Reader's sharing feature, and one of the cool things it gives you is a widget for clippings of shared items, which I have now added to this blog over there on the right-hand side.

Of course, since no one reads this blog on blogspot.com, you can also try the feed.

Do you have a feed of shared items in Reader? Let me know, I'm interested to learn more about this.

Yahoo Finance is still a better product

2007-01-04T08:57:00.000-05:00

Obviously, as a Googler, I want all Google products to so drastically outshine the competition that people around the world just send us checks out of a sheer sense of gratitude. If I had to pick one product that I'd like to see succeed like this, I'd choose Google Finance -- no one has really been innovating in the world of basic consumer-targeted financial web sites for a long time, and the Google Finance team is trying a lot of things to change the game.

But the product still misses some of the basics.

Case in point: Yesterday afternoon, I checked the Google stock price via the [goog]
query from the Google Toolbar.

Hey, we're up seven bucks, that's great! But that weird cliff in the middle of the day is odd. My gut instinct was that something market-wide had happened, so I went to the Google Finance home page to check for interesting headlines.

Nope, nothing interesting there that looks as though it should be bringing down GOOG, but the broad market summary on the left shows the same curve that GOOG had for the course of the day, so it does seem like *something* happened. Let's see what they have to say over at Yahoo Finance:

Boom! Right there -- "Stocks Turn Mixed After Fed Minutes".

The Google Finance team is innovating in a lot of great ways, and they're getting a lot of things right, but Yahoo Finance is still getting the most important things right.

WikidPad is the Awesomest

2006-12-17T21:57:00.000-05:00

This is a post about an unbelievably awesome piece of software. As in, so awesome I'm kicking myself for not having learned about it a year ago. But first, some background.

One of the best developers I've ever met was Andrew Bunner. We worked together for about six months almost four years ago, and I'm still amazed at how good he was. It was flat-out unfair.

One of the things that most stood out to me about Andrew's working style was a massive MS Word document he kept open at all times with the complete state of everything he was working on. It was amazing. Any time you asked him a question, he could page through his doc and pull up some notes from an experiment three weeks earlier. He had perfect recall for all facets of all topics.

The technique worked so well that, in the years since we worked together, I've tried repeatedly to copy it. I've tried using Word, Notepad, Wordpad, Gmail, and custom-coded HTML as the mechanism for storing notes, but none have worked - the cost of doing a retrieval or starting a new topic is always just too high. I always wind up with notes scattered across several different media, including physical notebooks, post-its, and scraps of paper. I finally realized that what I actually need is a wiki that runs entirely on my laptop.

This week I finally bothered to start looking, and I found a small client application that is already profoundly impacting the way I work. The app is called WikidPad. You should go download it and try it right right now.

My favorite features:

Formatting codes that are exactly what I'd expect. Like a lot of engineers at Google, I store a lot of information in twiki, so this may just be happy coincidence, but almost all the twiki formatting codes work in WikidPad.
Special text markers for items like TODOs, questions, and issues, along with an easy way to navigate to all such items in your wiki.
Killer search
Auto-completion for Wiki words. This is way cooler than it sounds.
Client-side speediness. It's been so long since I've spent a lot of time in a complex client app that runs quickly (my browser and my IDE certainly don't) that I forgot it could be done.
A pretty clear api for extending the app with simple python hooks

The only problem I've found so far is that it seems to have some problems with the pen input on my tablet PC. I can't isolate that problem though, yet.

In the three days since I installed this app, I have dumped tons of information and ideas into it, including the first draft of this post. This is without a doubt the coolest piece of software I've seen in a few years.

Fun quotes

2006-12-07T18:06:00.000-05:00

About 3 and a half years ago, when I started interviewing with Google, my first phone screen was with a fellow named Vijay. I remember doing well for a good chunk of it, and then blowing some parts at the end. Lucky for me, Vijay focused on the good bits and recommended that Google continue interviewing me.

I learned today that in Vijay's write-up of that interview, he had the following to say:

"I am concerned that based on his previous experience we may not be interviewing him for the correct position. Although he would make an excellent software engineer, I think that perhaps a manager position may be more in order."

Vijay had cause to share this with me today because I took the plunge and finally became a manager. In an odd coincidence, I actually became Vijay's manager, along with a handful of other great folks working on AdWords in New York.

The other notable quote of the day came from Chris, who I've been working with for a few months now:

"Jonathan's been a manager for five minutes and already he's being compared to Idi Amin"

It's taken a long time and a very particular set of circumstances to convince me to take on a management role at Google, but I'm very excited about this new challenge. The Ads team at Google is phenomonal for the pace at which they push out an ever-expanding set of mission-critical features to hundreds of thousands of advertisers around the world. This is an awesome time to take on this role.

PS. To the Google engineers in NY: If you're not already in Ads, come talk to me. I have a long list of great projects with a lot of opportunity to make a big impact.

Google Culture Can Not Be Diagnosed from the TechCrunch Ranch

2006-11-15T23:05:00.000-05:00

Mike Arrington believes that Google reporting apps don't support download to Google Spreadsheets because of "an obvious lack of conversations internally".

I'm in a unique position to comment on this, since I work on advertiser-facing reporting for AdWords (and have been for about six months). And Mike's right, we don't let you open your report data directly in Google Spreadsheets yet. But it's not because the AdWords reports team is unaware of Google Spreadsheets -- I've known that entire team from the first week they joined Google, and Fuzzy (the Tech Lead) and I are friends. We talk about what's going on with Spreadsheets all the time, and we've talked about incorporating it into the AdWords Report Center. Our offices are about 100 feet away from each other. We see each other at the coffee machine all the time.

The only problem is, Spreadsheets integration is in line behind about 30 other revenue-impacting features that hundreds of thousands of advertisers want first. And it's behind those other features because thoughtful people working hard to meet the needs of our advertisers have decided that such an integration is not a top priority.

Mike: please don't pretend that you can diagnose the root cause of a missing feature from the TechCrunch Ranch.

Engineers at Google have a raw technical curiosity that is expressed in frequent conversations about other teams and other projects. No one would say they know about everything going on, but engineers here, especially the best ones, tend to know A LOT about other projects throughout a variety of areas of the company.

Definition of a bubble

2006-10-12T07:51:00.000-04:00

Short note: everyone keeps wondering in print if Google's 1.65 billion dollar acquisition of YouTube constitutes a bubble or, even better, "irrational exuberance".

I don't want to comment directly on moves like this by Google, but I can't help but draw a comparison between this acquisition and an acquisition that took place at my last company.

Google spent about 1% of its market cap to buy the most successful independent new site on the web, an amount equivalent to approximately 15% of the company's total cash on hand.

At the absolute peak of the bubble, March 15, 2000 (prior day's NASDAQ close: 4706.63), E.piphany, an unprofitable public company, diluted its share value by 25%, spending an amount equal to 7 times total cash on hand to buy Octane, a company with no real product or customers to speak of.

Yes, it really happened.

That's a bubble.

Startups and Small Sample Sizes

2006-08-15T19:23:00.001-04:00

Here's a great way to commit mail fraud: pick 1024 names out of the phonebook, and send each of those people a free letter describing your analysis of the stock market, chock full of details like flattening yield curves, oil futures, and federal reserve monetary policy. In half the letters, state that all of these factors, taken together, will make the stock market go up in the coming month. In the other half, conclude that the market will go down. In each letter, recommend that the reader go long or short the S&P 500 according to your prediction.

512 people will receive a letter that turns out to be wrong; forget about them. Repeat the process the following month, splitting the remaining group in half again, so that 256 people get the "go long" recommendation, and 256 get the "short the market" recommendation. Continue every month, discarding the people who got the wrong recommendation in the prior month.

At the end of ten months, there will be one person who has received an accurate prediction every month. Send them a letter saying, "you've benefited for my market-beating advice for almost a year now. To keep getting my newsletter, just send me a check for $5000."

Your final person ("the mark") has every reason to believe that you are infallible at predicting the direction of the stock market. Why, they'd have to be an idiot NOT to send you $5000 to continue getting such valuable market insight! You've never been wrong!

As outside observers, we can, of course, see that you've made 1024 distinct predictions, and you've been wrong in 1023 of them. But your target doesn't know that. They have the wrong sample.

So far, this has nothing to do with software. But reconsider the above scenario, replacing each of the targets with employees at each of 1024 different software startups, each with a 50% chance of surviving any given year. At the end of 10 years, we should expect that exactly one of our startups will still be around.

Is that one startup filled with people whose talent that is so far above the mean that it was destined to be the one to survive? Or were there a variety of random and semi-random factors at play, completely outside the control of the company, that helped them survive while the other guys failed?

I'm a very strong believer in the importance of a strong team, and I do believe that strong teams build successful companies. But all too often, people incorrectly make the reverse inference -- "If I'm at a successful startup, it's because I'm on a strong team. If the team is strong, it must because I am strong. In fact, my company has never made a mistake. I am personally, and in all ways, infallible! Wow, it is so great to be me!"

This phenomenom can easily get so far out of hand as to hurt your company. Entrenched gatekeepers from the early days of your company believe that their own judgement is solely responsible for the success of the company, and as such, why would they ever need to consider other opinions?

The best way to detect whether someone is truly a rainmaker is to watch how they do in their second startup. Just to pick on an old favorite: everybody remembers the whiz kid from UIUC who was going to destroy Microsoft with a web browser... but how many people remember LoudCloud? Broadcast.com was fantastic! But what ever happened to THAT guy? Oh, apparently he owns a basketball team...

Of course, if you're the late-joiner at a successful startup, you can't just wait around for everyone to leave and see how they do in their next venture (tempting though it may be to encourage them to head out on their own).

But you can at least come in with an understanding that people might be unwilling to entertain dissenting opinions for completely artificial reasons. When you see the early employees of your company making decisions without examing the arguments, that's your sign to push back, hard.

[Many people will read this post and ask the natural question, "well, does Google have this problem?" A year or two ago, we definitely had a large number of people walking around who were convinced of their own perfection. Fortunately, enough late-joiners have made enough critical contributions to the company that the problem has dissipated. There are certainly still pockets of this attitude, but it doesn't take long to figure out where they are and how to avoid them.]

Building Complete Teams

2006-08-14T07:26:00.000-04:00

Mike Arrington has a problem with Google's custom version of the IT'S IT. At first glance, Mike's complaint is completely valid, but a little deeper examination shows he's way off the mark.

First off, just to be clear, I don't work at Google for the free food, and I don't know anyone here who does. That being said, let me take a little tangent before returning to free ice cream sandwiches.

A textbook mistake made by software companies, especially fast-growing software companies, is to focus immensely on hiring the most talented software engineers available, and then to fill any non-engineering position with whoever happens to come along - your recruiters, your admins, your sales department - everyone except for the engineers. The reasoning goes, briefly, "Gosh, compared to engineering, all of those jobs look so easy, how could it matter that much who we hire for them? Let's just fill the seats and get back to finding the best software engineers out there."

If you were to do this in professional basketball, it would be like signing Kobe Bryant and LeBron James, then hiring all your coaches and trainers by walking into the nearest bus depot and shouting, "Would anyone here like to coach basketball?"

This kind of hiring is a big part of what went wrong in the bubble. A lot of companies hiring for our hypothetical basketball coach position would look at a candidate and say, "Well, he's never coached professional basketball before, but he did coach little league baseball once, and it's really the coaching skills we're hiring for - he's a perfect match."

As a result, you wind up with an amazing engineering team surrounded by an absolutely craptacular company. And engineers figure that out really quickly. When good engineers try to get other good engineers hired, but your recruiters can't manage to contact a candidate for four months, engineers notice, and they get unhappy. When your sales team goes out and sells a product that doesn't exist and never will, engineers get unhappy. And when an admin sits at her desk IM'ing her friends instead of setting up the multi-continent conference call that needs to happen yesterday, engineers get unhappy.

The great thing about Google is, we don't have this problem. Plenty of bad things have been said about Google's hiring process being slow, or elitist, or biased toward good-looking women, but we don't hire anyone mediocre to fill any role in the company.

So imagine you're in charge of Google's food operations, and one of your goals is to attract the best and brightest corporate chefs that money can buy. How would you go about that? You might give them the luxury to work with only the finest, freshest, organic ingredients. You might let them work in a state-of-the-art kitchen. And every once in a while, you might give them a chance to take a classic ice cream treat and redo it in a way that aligns with their approach to cooking.

That's my take on it -- a free custom version of the IT'S IT isn't about the people eating it, it's about the people making it. From the description in the blog post, I'd imagine it was a pretty tough task, and it sounds like Nate and his guys did a stupendous job.

Kudos to them.

PS. Remember, when I post on my own blog, I'm not speaking for Google in any way.

Fun Bugs: Spamming Jeff Bezos

2006-04-13T16:54:00.000-04:00

From late 1999 through mid-2002, Amazon loved to send email that said "As someone who's purchased books by Joe Blow in the past, we'd like to offer you a 10% discount on his new book, 'My Life as a Joe'." The mail was powered by my company, E.piphany, and throughout most of 2000, I was at Amazon helping implement new features of that software. It was during that stint that I saw one of my Favorite Bugs of All Time.

One random afternoon, we were all happily hacking away, optimizing SQL queries, configuring new reports, when the VP of Customer Relations marched into the hallway and shouted, literally, shouted, "I NEED TO STOP ALL OUTGOING EMAIL RIGHT NOW. WHO CAN MAKE THAT HAPPEN?"

I was working with one other guy, and we immediately looked at each other and froze. We both knew that Amazon was very serious about protecting customers from spam -- by their rules, they could only mail unsolicited offers to each customer once a month. Two weeks earlier, our software had, by virtue of a bug, mailed a small set of customers more than once in the same month.

Now, a Vice President was standing outside our office, shouting at the top of his lungs, that he wanted to stop outgoing email. We were the most likely culprit, and we knew it.

We were in trouble.

It quickly came out that Jeff Bezos had received a few dozen email campaigns in the past half hour. Seeing these, Jeff concluded that Amazon's outgoing email had gone completely haywire, and we were spamming every Amazon customer around the world.

The rest of the afternoon was simultaneously gruesome and amusing. It slowly became public that someone at Amazon was working on a email marketing system to replace our software, and he'd been debugging it that day. Part of his code involved looking up a customer's email address, and the developer confessed that he had a couple means for looking up the address, and the fallback choice was "jeff@amazon.com". I imagine it looked something like this:

string GetEmailAddress(userid) {
  string result = GetEmailAddressMethod1(userid);
  if (result == "") result = GetEmailAddressMethod2(userid);
  if (result == "") // this can never happen
    result = "jeff@amazon.com";
  return result;
}

He was testing that code with his own ID, but he wasn't getting mail, so he kept testing it.

And testing it.

And testing it.

And every time he did, Jeff Bezos got mail.

For me, the bottom line was that we weren't responsible for the bug. Phew. The culprit and his boss had to explain themselves to Jeff, and it probably didn't help his career any, but nobody lost their job.

I'd love to be able to say there's some big lesson about software development to be had here, but the best lesson I can offer is "don't be dumb." I share this only for the entertainment value.

Google perks: interns

2006-03-31T22:11:00.000-05:00

One of the great perks of working at Google is our intern program. In three years, I've had three interns: Shubin Zhao (summer 2004), Brian Kernighan (summer 2005), and Aron Culotta (fall 2005).

First off, for those of you that don't believe me, yes, really, that Brian Kernighan. Brian is old friends with Peter, who I was working with in the summer of 2005. Brian likes to visit our New York office from Princeton, especially during the summer when class is out. Peter got Brian very excited about our project, then promptly disappeared for several weeks, so I got to be Brian's host.

But working with the co-author of K&R is just the tip of the iceberg! The opportunity to work at Google attracts some of the best and brightest students around, and we get to bring them in and see what they can do. My favorite strategy for a successful internship is to say, "Here's what we've built, and here are the problems on the horizon that we haven't time to address yet. Pick something from that general area and apply your big massive grey matter toward making my life easier."

And it works! Shubin built an amazing architecture for information extraction based on some heuristic-oriented work I had done. Aron worked on combining signals in structured and unstructured data to learn new relations among entities in the universe. He even got a paper out of that work. And when things go really well, your interns come back full time. Shubin joined our group full time in January 2005, and he's had a major impact on the quality of our product. And I'm looking forward to seeing Sasha, one of our group's interns from last summer, return soon.

The 2006 intern season is in full swing, and I am looking forward to learning what this summer's crowd will bring.

[Updated at BWK's request to replace "co-creator of C" with "co-author of K&R"]

Having tried it, I am free to knock it

2006-03-31T20:49:00.000-05:00

Ask anyone who has stayed awake for thirty-seven hours consuming nothing but Coke and Snickers bars and staring into a green CRT screen, if there is anything glamorous about the world of computer programming. Look deep into his bloodshot eyes, and try to detect any signs of joy among the red streaks. Then, just for kicks, ask him why he does what he does, despite all the pain it's causing. The most positive answer you'll get is, "it feels so good when it stops."

-- The Life of a Computer Science Major

I just changed projects at work, moving from a C++ codebase to a Java codebase. When I joined to Google, all the "hard core" development happened in C++, so, against my better judgement, I sucked it up and tried to be productive in C++. Three years later, the environment at Google is different, so there's real, meaningful work being done in Java, and now I'm taking part in it. I feel like I've finally come to my senses. The only question now is, what took me so long?

I'm not interested in making religious war about languages, but the contrast is too stark not to make note of. Here are some of the joys I've encountered in the past week:

Automated refactoring (including source control integration, for automated checkouts of affected files)
In-editor highlighting of compile-error typos
Code navigation that's unparalleled by emacs
Collection APIs that make sense! Holy cow!

And that was just in making one small change to fix my first bug.

It amazes me that anyone on the planet still insists on using a 30-year old language. Java is C++ with all the boneheaded mistakes taken out, and the difference it makes is amazing.

But my point isn't just that Java is better than C++ and you're stupid if you disagree with me. My point is that using Java instead of C++ lets you focus on the more interesting and challenging problems of building great software, so why waste your time with C++? Why waste the time to invoke a compiler to find out that your code has a typo when the editor can just tell you right away? Why manually slog through every usage of a particular function to add a parameter when you can just have a tool do it for you?

Everyone's favorite argument against Java is performance, but that's based on a write-once-run-forever model of software development. Sure, if you were just going to invest a fixed amount of development time in a bit of code and then let it eat CPU cycles forever, you'd want the highest-performance solution you could find (which, by the way, is probably assembly, or at least straight C, but not C++).

But no software ever gets launched and then runs statically for eternity. It gets extended and built upon and edited by dozens of different developers, all of whom have to be able to figure out what you were smoking when you wrote the code in the first place.

The real challenge of software isn't building the first version, it's building the tenth version on a 5-year-old codebase that has been pushed way beyond its original design parameters and no longer fits the mold, written by someone who left the company two years ago and may or may not have written this code on his best day. In C++ codebases, the state-of-the-art for dealing with code staleness is to rewrite the whole thing from scratch. This never works as well as you'd hope -- sure, it seems good for a while, but give your fresh rewrite a year or two to get stale, and you'll find your completely unable to manipulate the code in the face of changing requirements. In the language of essay cited above, the complete rewrite is a Jello Hour solution.

When I started writing Java code, it took about 3 months before I felt that I was really very facile with it. I wrote C++ code for 3 years, and I still feel like every line of code was a matter of poking myself in the eye with a sharp stick.

The best thing I can say about my time writing C++ code is that it feels so good to have stopped.

Shameless self-promotion

2006-02-28T20:40:00.000-05:00

A couple people have independently told me recently that they saw my AdWords ad on [Nelson Minar], and asked me why I was running the ad. I had two main motivations:

It's universally understood at Google that everyone should experiment with an AdWords campaign. Google gives everyone $1/day to spend on advertising.
I want people to discover my blog, and Google ads work.

As for using Nelson's name, well, I drew my inspiration from Google's own advertising campaign. We've long used ads on the names of well-known engineers, especially in information retrieval, to draw attention to our job listings. Why not use them same technique for my own purposes?

Still, the ad text I was using was a little lame, so I've just rewritten it to:

Shameless Self-Promotion
Read my blog about how we can
improve the quality of software.
jonathanbetz.blogspot.com

While I was in there, I updated the list of people I'm using for as keyword triggers. For the record, the list is:

- Brian Kernighan
- Peter Weinberger
- Nelson Minar
- Jeremy Zawodny
- Josh Bloch
- Steve McConnell
- Joel Spolsky
- Rob Pike

An unintended quiet period

2006-02-26T11:47:00.000-05:00

So, it's coming up on March, and the last post here is dated early November. What was I doing?

- Celebrating the holidays
- Taking on a new responsibility at work
- Trying to figure out how to build a long, successful career at Google
- Reading Valleywag

In late November my boss asked me to be the engineering liason to Google's new Pittsburgh engineering office. We hired a couple managers to run the show there, but I was asked to come in and help them learn their way around Google.

I'm proud to say that they are off to a running start. Google intends to hire 90 engineers in Pittsburgh within 2 years, and things look good so far. We still haven't announced our permanent location in Pittsburgh, but I'm optimistic that we'll have a great location to announce soon.

As far as building a long, successful career at Google goes, I can't say that I've figured out the best answer, but I'm at least pushing into some new directions. Ideally it won't be another 4 months before I post about that.

The Myth of Open Seating

2005-11-09T00:00:00.000-05:00

Sadly, there are still some people who believe that the best way to build great software is to put everyone into a giant open cube farm -- an "open seating plan." These people are flat-out wrong.

This has been a topic of discussion at work lately, as we just moved into a space that combines the worst of all possible seating strategies – some of us are in a large, wide-open space that lets noise carry for miles, and those who were given offices are packed so tightly that you can't actually walk to your desk if your officemates are already seated.

Coincidentally, Joel Spolsky recently had a post in which he reconsidered his long-held tenet that software organizations work best when everyone has a private office; the four interns at Fog Creek worked together well by sharing a common space, and it made him briefly reconsider if private offices are the way to go.

This idea that open seating fosters interaction and creates better software is an unsubstantiated myth that falls apart under the simplest examination.

At Google, you can talk to the senior engineering executives and quickly understand that their commitment to open seating is a reaction to experience in academia and industrial research labs. All of these guys have some story about places like Bell Labs or PARC, where everyone just came into work, went into their private office, worked all day, and went home without ever really interacting with the people around them. This only leads to the kind of culture where ideas never turn into products, where everyone pursues their own agenda without challenge, and you eventually turn into a dinosaur that gets acquired by one of your more successful spinoffs. The reaction at Google is to use open seating to make sure that you're exposed to the people around you.

If you're hiring the kind of people who are happy to go into their office and not interact with anyone all day long, the problem is your hiring criteria, not your office layout.

Building great software is an interactive process, and the ideas from a group are frequently better than the ideas from any one individual. People who have to be physically forced to interact with others are not good team players, and even if you do force them to sit in an open environment, you can't force them to listen to the world around them. Send these people to your biggest competitor.

It is factually correct to say that an open floor plan does create more casual conversations, but that ignores everything else that comes along with open floor plans. It's an extremely fragile environment. In every cube farm, there's always a spoiler. There's the person who leaves their cell phone on their desk at the highest volume, then walks away for three hours. There's the person who carries on a
conversation, from their desk, with the person seated 30 feet away. For some time last year, I became a spoiler without even realizing by playing with a little marble toy in the office – someone eventually worked up the nerve to tell me I was driving 20 people stark raving mad. And, of course, there's always the person who simply lacks the ability to modulate their own volume according to the situation.

And what happens when you're trying to get work done with a spoiler around? You put on headphones. Everybody does it. Walk around the Google engineering office in New York, and you'll probably see ten to thirty percent of the engineers wearing headphones. And putting on headphones does a lot more than block out the loud talkers and the abandoned cell phones: it blocks out all those wonderful casual
conversations that are supposed to be the benefit of open seating. Plus, now you're listening to music, which is only a distraction from what you're really trying to do: writing software.

The worst thing about this debate is that DeMarco & Lister did the hard work years ago to demonstrate that the most productive developers are in organizations that provide private offices. They didn't demonstrate causation but they have convincing data that great developers tend to wind up at the organizations that give them a private office with a door that closes. And the cube farm advocates love to question the details of that study without providing anything other than anecdotal evidence in favor of support of their opinion.

Is it the end of the world to hire the best talent available and then put them in sub-par work enviroments? No. But is a sub-par work environment going to cut into your ability to hire the best talent available? Yes. That's the real message here.

Donde esta Q&A

2005-09-21T10:09:00.000-04:00

When I took my post-college trip to Europe, I was accompanied by one guy who knew a little French, and another guy who knew a little German (my contribution was that I knew a little Latin). Early one morning, we found ourselves in northern Spain trying to find our way from the train station on one side of town to the bus station on the other side of town. Our entire Spanish lexicon consisted of the knowledge that "donde esta" means "where is".

As they say, hilarity ensued. The people of that town were quite helpful, but "donde esta bus station" wasn't quite enough to carry us through. Still, we were in high spirits, and after 15 or so minutes of walking around lost, we concluded that the funniest joke ever was to say "donde esta ... us?"

All of that makes this morning's new feature on google.com all the more interesting. Google Q&A, the product for which I serve as Tech Lead, is now live in Spanish. See what I mean?.

Look what I made!

2005-09-13T17:44:00.000-04:00

Today, Google launched a special search site for Hurricane Katrina information. I worked on the "Search for People" feature. We're collecting data from all over the web, and making it possible for people to find their friends and relatives with one search.

Anyway, I just wanted to take this opportunity to brag a little about the cool stuff I get to do at work. It's pretty rare that writing software gets you so palpably close to doing good things for the world, so it's a real high when it happens.

Random Apple Bugs...

2005-09-09T16:40:00.000-04:00

Interesting flaws, especially UI flaws, in Mac software are rare enough that they're worth noting. I recently ran into two of them.

My wife was trying to export a large group of photos from iPhoto (aside: she was using "export" because the "burn CD" feature isn't advised if the CD is meant to be read on windows. I thought we outgrew os-specific removable media a decade ago).

So, every time she tries to do the export, we get an error message about "there is not enough space available". That's crap -- the finder ui reports 12 GB of free space, and she's only trying to export 100 mb of data. We do the obligatory Google search for the error message.

The only hit is on some message board, and it's talking about a bug that's supposedly already fixed in our version of iPhoto. We try the suggested workaround for kicks, and, no surprise, it doesn't apply.

Eventually I have the inspiration to empty the trash and try again. Aha! Now we have enough space. But the finder ui does nothing to hint us toward looking at the trash.

OK, the problem is solved, but why should we have wasted our time on this? A few simple fixes would make the whole situation a lot easier to resolve:

1. When you report that you don't have enough space, tell the user how much you think you have and how much you think you need. More generally, if you have an error condition because X is less than Y, report your perceived values of X and Y.

2. When you report any resource as available, make some kind of modification to your message if the resource isn't completely available. Banks have long since mastered this style of reporting, maybe we can learn a thing or two from them.

The second of my two recent interface flaws is even harder to excuse. It's actually so bad that I've considered not posting about it here, because I can hardly believe I'd be able to reproduce it. Still:

I use the iLife Backup app. It seems decent enough, and it's definitely better than anything I ver found on windows. But the last time I did a backup, I wanted to be really sure that it had completed successfully. When it looked as though it had finished, I checked all the 'last backed up' times. And they hadn't been updated from the prior backup. Clearly, my backup had failed! So, I ran the backup again. And the 'last backed up time' still hadn't updated.

This is where I had my flash of insight, and closed the backup app. Aha! Upon reopening it, the 'last backed up' timestamp now reflected the backup I had just done. To be clear: the software was doing everything except confirming for me that it had actually backed up my software. Gah!

For the most part, the folks in Cupertino have user interaction down pat; like I said, that's what makes it interesting to note these flaws.

When Bad Software Happens to Good People

2005-08-08T19:05:00.000-04:00

My goal for the day was to take a flight with my wife and daughter from Newark to San Francisco. We booked a nice, early 7am United flight so that I could get to California by 10:30am, local time, and put in a reasonable day's work. We showed up at the airport at 6am, and breezed through the line to the self-service checkin.

Now, my first sign that United's software systems are broken should have come weeks earlier -- we were booking one ticket with miles, and the others would be paid for. When the agent booking the mileage ticket was physically unable to to book the other two tickets, I should have realized that United was using a booking system that had long outlived its usefulness.

Sadly, I'm just not smart enough to read an obvious sign when I see it.

So we check in my wife (the mileage ticket) and then attempt to check in me and my daughter. The checkin kiosk doesn't know who I am. We ask the ticketing agent, and she tells us that, although there's a record of the reservation, the tickets were never issued.

The next 45-or-so minutes were enlightening, from the perspective of software anthropology. The key insight for me came from the following exchange:

Ticketing counter agent: The original booking agent put an f-dash in [some field], and the automatic queueing system never picked that up and issued the ticket, because it looks for [some other code].

Ticketing counter manager: They should know not to put an f-dash in that field, we haven't used that code for years.

I can only deduce from this exchange that there are two separate mechanisms, and one of them (the reservation UI) allows input that isn't recognized by the other (the queueing system that issues the tickets from a reservation); as a result, some number of tickets are never actually issued, and this morning's episode gets repeated from time to time at United ticket counters around the country.

Don't feel bad for me. I made it to San Francisco only two hours later than originally intended, I got to see the Denver airport, and I had plenty of time to finish reading Cinderella Man (I recommend that you read it). But think about this from United's perspective:

From the time we realized there was a problem with our tickets, until the time we got on our first flight, we took the time of no fewer than seven United employees. At checkin, there were two ticket agents, their manager, and a person they called at their ticketing "help desk". The two ticketing agents were completely consumed with our case for about an hour. At the gate, another three gate agents were eventually involved in helping re-route us. And, if we're lucky, United will eventually offer us a few free tickets to accommodate us for the inconvenience. All of this is going to cost them a non-trivial amount of money. And there's nothing in the entire chain of events that was unique to our reservation -- this is a systemic bug that can be invoked by any reservation agent inputting the discontinued "f-dash" into the mysterious forbidden field. This magic code has supposedly been discontinued "for years", but the reservation system still accepts it, and the booking agents still use it, so the problem isn't going away any time soon.

I have some theories about possible root causes of United's software problems. I'll try to get to these in a later post.

The Disconnect Between People and Process

2005-07-28T16:54:00.000-04:00

Earlier today, someone mailed me a link to the Construx course schedule. Construx is the company founded by Steve McConnell, author of Code Complete and The Software Project Survival Guide, along with several other great titles on software development. The courses offered by Construx have titles like:

Code Complete Intensive: Practical Software Construction
Software Project Management Boot Camp

I'm interested in software project management, so I checked out that link. The course covers topic areas like:

Defining software project success
Using a software project plan template
Controlling change
Effective conversion planning

I'm reading this and thinking to myself, "What does this do for me?" Ask yourself -- of software projects you've been on, when did you think "Gosh, this project could be a wild success if we could just define success! Maybe we need a software project plan template?" Not very often, I'll wager.

On the other hand, how many times have you thought "This project could be a success if..."

... I could just convince <team member X> to {stay focused|test his own code|stop playing ping-pong all the time}
... I could show my boss that we just need a few more {weeks|people|machines}
... we had a real {testing plan|development environment|business model}
... our company could attract the most talented developers in the world.

Where is that material in courses on software engineering management? Process is nice, but people matter. We have to start admitting that getting the best people is hard, important, and relevant to software success. If you want to create a process that captures that, all the better, but please stop talking about process as though it can abstract away all impact of people.

Trivial Pursuit is exactly that

2005-06-28T10:50:00.000-04:00

Michael Liedtke of the AP has posted a review comparing the performance of major search engines on a randomly selected set of Trivial Pursuit questions. I'm all about getting the right answer to the top of the page every time, but think for a second about the words "Trivial Pursuit". Those questions aren't chosen because they're likely to be things people want to know, they're chosen because they're likely to be things that people don't know.

A real way to do this review would be to find a way to sample the questions that people actually want to answer with search engines, and compare Yahoo, Google, MSN, et al on that set.

No upside for talking to reporters

2005-06-20T12:59:00.000-04:00

Fellow Googler Joe Beda has a great post detailing why there's no upside in talking to reporters. He might be taking the right idea a little too far, but it's definitely the right idea to start with -- when you blog, your message gets out to the world, unfiltered. Reporters, intentionally or not, can twist your words around.

James Fallows, throw me a bone

2005-06-13T11:10:00.000-04:00

The Times published a rant by James Fallows about question answering vs. keyword search. Google Q&A isn't yet at the level of question answering he describes ("tell me the per capita spending on education in California over the last 50 years"), but he also goes the entire column without mentioning Google Q&A.
For the record, Google Q&A has been live for over two months, and it's a lot more like question answering than vivisimo's result clustering...

My kingdom for a decent personal finance app

2005-06-07T21:52:00.000-04:00

About six months ago, my wife and I made the big switch to a mac. Great! No more spyware! When I don't know how to navigate the GUI, I can drop into the CLI and fix things there! The switch was nearly a complete success.

Until we switched from MS Money to Quicken Mac.

I soon learned that this app is almost universally reviled, for good reason. Trying to describe the ways in which it is broken would take hours. At the root, the problem is that the interface is just a series of co-existing application windows with no defined interaction. Trying to find the right window for a given task requires examing all the windows, and when you get there, you find that the transaction you want to work with hasn't made it to that window yet.

I've looked into some of the alternatives like MoneyDance, but nothing is all that appealing -- especially given the overhead of transitioning among data file formats. My best bet at this point might be to start running MS Money on Virtual PC, thereby undoing much of the benefit of the switch.

None of which is to say that that Intuit appears to have noticed. In all the forums with rants and raves against Intuit, there's no indication that they've tried to address any of it. One might think they'd at least want to address the fray on the amazon comments page, but they haven't bothered yet.

I have tried unsuccessfully to track down the email address for the product manager for Quicken Mac at Intuit (I think the culprit is Adam Samuels). If you have it, please leave a comment. Someone needs to make his life painful enough that something starts to change.

Intro

2005-06-07T21:39:00.000-04:00

I'm Jonathan Betz. I work at Google; I'm the tech lead for our
question answering service (aka "Google Q&A"). Read my blog or don't read my blog, it's up to you. You're not going to get the scoop on the next great Google product, but you might get some insight into life at a great software company, and you'll definitely get an earful of my ideas about modern software development.