Meritocracy ※

Posted Monday, 08 February 2010

Jeremy Beer offers a thought-provoking essay on how meritocracy is killing Middle America:

…fly-over country, by and large, has been hemorrhaging intellectual capital for decades. The most talented young men and women, the most able, the most intelligent and creative, have been leaving to go off to college — or have been lured off to college — only to return in ever-diminishing numbers.

Much of Beer’s rhetoric is based on a denial, made explicit only at the very end of the article, “that we ‘need’ to maximize our economic value”. I’d argue that “maximizing” may not be necessary, but that economic progress is in fact one of our moral obligations to society, both present and future. The need for community and social cohesion does not completely obviate the tangible benefits of prosperity, which can support attempts to foster community. If remote communities are to close the growing gap with the world’s creative hubs, I expect that they will do so on the back of technologies like telecommuting and digital distribution—both direct products of economic and technological innovation.

The point that rings most true in the essay, however, is the contention that support for meritocracy can be motivated as much by self-interest as by abstract moral ideals:

…modernity, whose distinctive political philosophies have stressed equality, has led to greater inequality than ever, precisely because it has equalized opportunity — that is, because it has unleashed talent either to sink or swim — more than had ever previously been done. To put it yet another way, modernity has created many more opportunities for the expression of inequality than ever. And it has made inherent inequality more important than ever in determining social and economic distinctions.

This sheds some light on the cries of elitism that seem to grow only louder. Whether or not elites run things better isn’t the point; it’s whether offering all the benefits of power to the fortunate few is inherently fair.

The real irony is that in the US, conservative anti-elitism is tightly coupled with laissez-faire capitalism: the fortunate few shouldn’t be allowed to run things, but there is no limit to how disproportionate their share of the benefits should be. The opposite liberal stance is in fact far more coherent: the most competent should be in charge, but oversight should dampen the inequalities that result.

Number Pads ※

Posted Tuesday, 26 January 2010

Two points.

First: modern computers are used with a keyboard and a mouse/touchpad/trackball, and they are used predominantly by right-handed people. Right-handed people normally operate the pointing device with their right hand. The numeric keypad present on many keyboards is designed to be operated with one hand. If keyboard manufacturers put the keypad on the left instead of the right, then more people would be able to enter numbers with one hand and work the mouse with the other.

Second: all standard QWERTY keyboards have little raised bumps on the ‘f’ and ‘j’ keys. They let touch typists find the “home” position on the keyboard without looking down from the screen. On keyboards with number pads, there’s a similarly raised bump on the ‘5’, for the same reason.

On keyboards without number pads, typists have to use the numbers on the top row. Two rows of keys is simply too far to reach, so instead typists shift their hand(s) up to that row when typing lengthy numbers. If keyboard manufacturers included raised bumps on a couple of the numbers (e.g. the ‘4’ and ‘7’), then touch typists could more easily make these shifts by feel instead of needing to look down at the keyboard.

You’re welcome, keyboard industry.

Get to work.

Poker Hands ※

Posted Monday, 25 January 2010

I’ve had a hard time finding a concise explanation of the rankings of poker hands that’s suitable for use as a reference by new players. In particular, the rules for comparing between hands of the same type (e.g. if two players have flushes, which one wins) are generally described in a very ad-hoc manner.

So I wrote them up myself, in Markdown (aka text) format. I converted this to HTML and added some simple styling to create a poker hand reference that fits on a single sheet of paper, and combined it with a quick-reference listing of hand types to make a PDF reference card—print it four sheets to an A4 page, double-sided, and you can cut it down to a four A6 references. All three of these are licensed under the Creative Commons by-nc-sa license.

Netbooks ※

Posted Sunday, 10 January 2010

I’ve written about netbooks before, but a recent post from Jeff Atwood drove me absolutely crazy. Atwood is working to take Cringely’s place as the only consistently-wrong blog I subscribe to. First he quotes Dave Winer’s definition of a netbook:

1. Small size.
2. Low price.
3. Battery life of 4+ hours. Battery can be replaced by user.
4. Rugged.
5. Built-in wifi, 3 USB ports, SD card reader.
6. Runs my software.
7. Runs any software I want; no platform vendor to decide what’s appropriate.
8. Competition. Users have choice and can switch vendors at any time.

This is a bizarrely specific list—only two USB ports and it’s not a netbook? It’s also absurdly general: are “small” and “low price” just relative to laptops?

More importantly, what on earth does “any software I want” mean? Does that mean compiled code? Does it require an API for the same UI elements used by the system itself? How extensive, stable, and well-supported does that API have to be? Most importantly, how does that API interact with “competition”? What I’m getting at is that a device with a robust Javascript environment could well qualify better for points 6–8 than a linux device with a custom GUI.

But the craziest thing is that there’s a glaring omission from the list. Both Winer and Atwood seem to take it as a given that a netbook has a hardware keyboard and a PC-like screen. In fact, they both seem to take it as a given that a netbook has all the capabilities of a desktop computer. This is where I think their perspectives go off the rails, but I’m getting ahead of myself.

Netbooks are no small thing for Atwood:

Netbooks are the endpoint of four decades of computing – the final, ubiquitous manifestation of “A PC on every desk and in every home”. But netbooks are more than just PCs. If the internet is the ultimate force of democratization in the world, then netbooks are the instrument by which that democracy will be achieved.

[…]

To dismiss netbooks as like laptops, but lamer is to completely miss the importance of this pivotal moment in computing – when pervasive internet and the mass production of inexpensive portable computers finally intersected. I’m talking about unlimited access to the complete sum of human knowledge, and free, unfettered communication with anyone on earth. For everyone.

Ignoring the inane overstatement, it’s clear that Atwood is conflating two very different things. He previously described a PC that met a set of eight criteria; now he’s just talking about ubiquitous web access. I agree that such access is a big deal, but there are lots of ways to achieve it without a netbook. There’s little question in my mind that when most people on earth have web access, they will usually be getting it from devices that are not PCs.

Atwood then focuses on yet another item that didn’t appear on his of netbook traits: a lack of ongoing subscription costs or network charges:

It’s true that smartphones are slowly becoming little PCs, but they will never be free PCs. They will forever be locked behind an imposing series of gatekeepers and toll roads and walled gardens. Anyone with a $199 netbook and access to the internet can make free Skype videophone calls to anywhere on Earth, for as long as they want. Meanwhile, sending a single text message on a smartphone costs 4 times as much as transmitting data to the Hubble space telescope.

I don’t care how “smart” your smartphone is, it will never escape those corporate shackles. Smartphones are simply not free enough to deliver the type of democratic transformation that netbooks – mobile PCs cheap enough and fast enough and good enough for everyone to afford – absolutely will.

Rich techies live in a strange bubble where high-speed wifi is pervasive and free, but cellular service is spotty and expensive. Such a bubble usually extends to home and the office; if these were the only places where network connectivity mattered there wouldn’t be much call for mobile computing in the first place.

Atwood’s argument also ignores non-netbook devices with wifi, such as the iPod Touch.

Maybe those early [netbooks] were [cheap and crappy], but having purchased a new netbook for $439 shipped, it is difficult for me to imagine the average user ever paying more than $500 for a laptop.

Spoken like a man who does his “real” work on a desktop PC. Those who make their living entirely with a laptop will certainly be willing to pay for a high-quality machine.

But if we accept the premise that the “average user” already has a desktop PC, I’ll agree with Jeff that laptop sales among this market will drop. I just don’t think they’ll be replaced with PCs shoehorned into uncomfortable form factors; I think they’ll be replaced by “internet communicators” (as Steve Jobs calls them).

The average user wants a mobile device with web access. It’s incredibly myopic to interpret that desire as a need for a full-blown PC.

2010 Predictions ※

Posted Monday, 28 December 2009

I did it last year, with fair results, so it’s time for another try. These are my predictions for 2010.

Like last year, I’ve rated each prediction with a “difficulty” between 0 and 1 based on how likely I think I’ll be right; this is an attempt to distinguish between no-brainers (difficulties at 0.2 or below) and outlandish guesses (difficulty 0.8 and up). If you’re looking for betting odds, I think the scale is probably logarithmic.

Sport

1. LeBron signed by Nets (difficulty 0.7)

LeBron James’ two priorities are to become a global icon and to win NBA championships; Cleveland isn’t helping with either. The Nets have the perfect market for him, the cap space to sign him, and an ownership group with personal connections to James.

2. Semenya and Pistorius ruled ineligible (difficulty 0.4)

I’m shocked that this hasn’t already happened, and it’s an absurdity of the legal process that either Caster Semenya or Oscar Pistorius were ever allowed to compete internationally. Athletics South Africa is a complete embarrassment.

My only caveat is that Semenya could regain eligibility through medical treatment.

Politics

3. Obama ends “don’t ask don’t tell” (difficulty 0.7)

It’s been a rocky first year for Obama, but he does seem to be working through his list of campaign promises: Guantanamo is shutting down (very slowly); he pushed through a stimulus-based economic plan (although perhaps not a very good one); he’s doing everything he knows how to reform health care (but doesn’t seem capable of the major overhaul he’d like).

He’s pushed back the issue of gays in the military, but I think he’ll face up to it this year. A typical Obama disappointment would be continuing to exclude openly gay soldiers from combat duty, but any elimination of don’t ask don’t tell would be an improvement.

Mobile Technology

4. Android becomes most popular OS; iPhone remains most profitable (difficulty 0.7)

Android just isn’t good enough to compete with the iPhone on the high end, but it’s easily good enough to displace Windows Mobile on smartphones at the low end of the market. And all phones are turning into smartphones.

5. AT&T loses iPhone exclusivity in US (difficulty 0.6)

It seems clear that Apple hasn’t worked anything out with Verizon for a CDMA version of the iPhone, which suggests they’ll stick with AT&T. But the iPhone is Apple’s most important product right now. AT&T is destroying the ownership experience in big cities in the US, and it’s getting worse. Steve Jobs isn’t going to let this continue: he cares about control, the ownership experience, and the image of the company more than he cares about squeezing a few extra dollars out of a subsidy contract.

6. Microsoft buys Pre (difficulty 0.9)

Windows Mobile is a dead end, and everyone but Microsoft knows it. The Palm Pre is a promising long-term platform, but it needs time to mature and time for the hardware to catch up—time Palm might not have the cash to live through. Microsoft’s best bet to stay relevant in the mobile computing space is to simply buy the Pre.

This would, however, require an admission from Microsoft that they’ve completely failed to innovate on their own, and that their rhetoric for the past few years has been bullshit. Such a major change of direction is much easier under an unassailable dictator like Bill Gates or Steve Jobs than under someone like Ballmer, who is currently in a very dicey political position.

It’s a long-shot prediction, but I’ll say Microsoft does it.

7. Apple Tablet released (difficulty 0.5)

I don’t have any predictions that would mandate a very high difficulty rating. My wish will come true, and Apple will ship a computer with a (roughly) A5 multitouch screen and no physical keyboard. It will ship before the third quarter (probably around April); it will be a lot more like an iPhone than like a Mac; it will do web, email, movies and ebooks; price will likely be something between $400 and $1000.

And I’ll most likely buy one as soon as they’re available.

Other technology

8. Microsoft fades even more… (difficulty 0.3)

Same prediction as last year: Google’s and Apple’s stock prices will outperform Microsoft’s.

9. iTunes gets live events (difficulty 0.7)

Apple has the best content-protected video-delivery service available. I predict that this year they extend their architecture to allow for live streaming video—e.g. sports and other live television events.

Financial

10. Price of gold declines (difficulty 0.5)

Over 1100 USD/oz? Ridiculous; the historical price of gold is closer to $400/oz. I have no clear idea of how quickly prices will come back down, so I’ll just go for a moderate-difficulty prediction that gold prices are under 1100 by this time next year. My guess is something around 800.

If I had any idea how to “sell gold” via eTrade I would.

11. Stock markets perform well (difficulty 1.0)

Everything up 20%: DOW at 12500, NASDAQ at 2700, S&P 500 at 1350. I could say I know this based on careful analysis of the markets, but instead I’ll claim magical powers over time and space. Just as meaningful.

Like last year, I’ll take partial credit for this prediction: a hit at difficulty 1.0 if I’m within 1%, 0.7 if I’m within 10%, 0.5 if I’m within 20%, etc. (Difficulty is given by e^(0.03963(1 - percenterror)).)

Lehrer’s Rules ※

Posted Thursday, 24 December 2009

In his last broadcast of NewsHour, Jim Lehrer presented a list of “guidelines of practice” for good journalism:

Do nothing I cannot defend. Cover, write and present every story with the care I would want if the story were about me.

I’m with him so far, but none of this is specific to journalism, or to professional endeavors in general.

Assume there is at least one other side or version to every story.

That’s bullshit. Facts are facts. If a man is shot and killed and reporters can verify the situation, reserving judgement on whether or not the guy’s dead is nonsense.

Which leads to the realization that this guideline is really an insidious endorsement of biased journalism. There definitely is another side to the story “man is shot and killed by evil and selfish gunman”, and another side to “obnoxious man forces stranger to shoot and kill him”. Neither of those are news stories.

If you’re writing a story and you think that there is contradictory version with a reasonable claim to truth, then you’re probably not writing news.

Assume the viewer is as smart and as caring and as good a person as I am.

As smart? Even if I thought that worked for Jim Lehrer, journalists who only targeted viewers as smart as they were would be shooting over the heads of the majority of the population. Journalism is the art of explaining the facts as simply and as clearly as possible, so that viewers don’t need investigative and analytical expertise to understand what’s going on. This isn’t an excuse to make the facts simpler than they really are, but simply assuming that viewers are as smart as journalists is ignoring a major responsibility of the profession.

As caring? This seems like a recipe for wasting everyone’s time, and also implicitly endorses the idea that appealing to emotion is one of the main functions of journalism. Emotion is certainly one of the reasons people get interested in certain stories, but Lehrer himself claims that he’s not interested in the “entertainment” side of journalism.

As good? The only interpretation I can find for that is that Lehrer thinks viewers share his particular moral outlook, and that his particular view of morality is superior to any other outlook.

I don’t think we can take any of this literally. Beyond Lehrer’s rare conceit about him being “smart, caring, and good”, I think this guideline really amounts to “respect the viewer”. I can agree with that.

Assume the same about all people on whom I report.

Some people aren’t as smart as you. And maybe not as passionate or “caring”. And a few have fundamentally different senses of morality. When these differences affect the rest of the population (because a nitwit runs for vice president, or a community of religious fundamentalists endorses “honor killings”), it becomes news. Even if that news contradicts your assumptions about people.

Assume personal lives are a private matter, until a legitimate turn in the story absolutely mandates otherwise. Carefully separate opinion and analysis from straight news stories, and clearly label everything. Do not use anonymous sources or blind quotes, except on rare and monumental occasions. No one should ever be allowed to attack another anonymously.

I agree with the general spirit of all of these.

And, finally, I am not in the entertainment business.

I understand the sentiment, but I consider this a very narrow view of “entertainment”. The reality is that viewers have a choice of how they spend their time, and the satisfaction of staying up on current events must be weighed against the immediate pleasures of lighter entertainment. Confusing news with light entertainment is a mistake, but simply taking the value of current-events media as a given ignores the greatest issue faced by journalists today: what are you offering viewers?

2009 Prediction Results ※

Posted Monday, 14 December 2009

Before moving on to my predictions for 2010, I’ll take a look at how my predictions for 2009 worked out:

1: Usain Bolt sets 100m record at 9.62 (difficulty 0.7)

I was right that Bolt would break his own 100m record of 9.69, but he was faster than 9.60, which was the low end of my fallback (difficulty 0.5) range. I don’t feel too bad about missing this one.

2: Russia tones it down (difficulty 0.4)

I admitted at the time that this was subjective, but that “I don’t expect Russian military action to appear in many New York Times front-page headlines.” I’ll take credit.

It seems trivial in hindsight, but then most good predictions do.

3: Blagojevich cleared (difficulty 0.7)

The original prediction was that while Blago’s misbehavior was enough to ruin a political career (he was, in fact, impeached), it probably wasn’t enough to get a criminal conviction.

The trial is still going on, so I guess the original prediction missed. I haven’t been following the trial and don’t know much about either the evidence in the case or corruption law in general, but I’ll stick with it and guess that he will eventually be cleared.

4: Netflix starts to slip (difficulty 0.6)

I regretted this prediction almost as soon as I posted it; Netflix has a very good online-rental offering, so using them as “a proxy for best-of-breed physical media rental services” in comparison to iTunes-style digital distribution was a terrible, terrible choice.

I predicted a loss for Netflix by Q3; they continue to be consistently profitable, to the tune of $30 million per quarter.

Beyond my lousy choice of Netflix as a proxy for physical media, I think I was also expecting a much bigger push from AppleTV, which didn’t happen. I’m still a believer in the death of physical media, but this prediction was wrong on a great many levels.

5: New iPhone model from Apple (difficulty 0.3)

Nailed it in every detail: only major changes were faster processors and more RAM.

6: Microsoft continues to fade away (difficulty 0.3)

Another low-difficulty prediction. Microsoft closed on 2008-12-30 at 19.34, and now they’re trading at just under 30 (up 55%). Google moved from 303 to 590 (up 95%) and Apple went from 86 to 195 (up 126%). I made (a little bit of) money on this one; stock predictions aren’t supposed to be this easy.

7: Yahoo broken up and sold off (difficulty 0.4)

I missed this one. Instead of being bought by Microsoft, Yahoo just contracted to use Microsoft’s search service. I don’t entirely understand Yahoo’s strategy, but then I haven’t been paying much attention to them this year.

8: GM and Chrysler file for bankruptcy (difficulty 0.7)

Chrysler filed for Chapter 11 on 30 April, and GM followed suit on 1 June. My optimism that Obama wouldn’t bail them out wasn’t justified, but the predictions were solid.

9: Stock markets tread water (difficulty 1.0)

I predicted DJIA, Nasdaq, and S&P at 8600, 1700, and 880, respectively, which represented only a slight gain from a year ago. Right now they’re at 10471, 2190, and 1106, so they’ve done much better than I expected, to the tune of about 25%. According to my formula, this counts as a success with a difficulty just under 0.4—not completely trivial, but not much to brag about.

10: My own research output (difficulty 0.4)

Miserable failure on this; my productivity plummeted this year, and I won’t be submitting my thesis until next year. Sigh.

Final tally

So I hit four easy ones (2, 5, 6, and 9), missed two easy ones (7 and 10), hit one tougher one (8) but completely screwed up another (4), and missed two tough predictions but not very badly (1 and 3). Given the mix of difficulties, I don’t feel too bad hovering around 50% overall.

More on the CrunchPad ※

Posted Monday, 30 November 2009

I haven’t really been following Michael Arrington’s attempt to mass-produce a tablet computer for web browsing. I read the initial announcement and dismissed the project as a pipe dream from yet another blogger who thinks having a lot of readers means you know what you’re talking about. This week Arrington announced that the project had failed.

Arrington’s general dickishness is well-known, but I really don’t understand the thinking on this one. The original point was to “open source the specs so anyone can create them”. The plan was to “hopefully build a few prototypes… If everything works well, we’d then open source the design and software and let anyone build one that wants to.” He now claims that they have prototypes and are ready to get going on mass production, and he’s declared the project dead because another company has decided to build them without him. What’s more, Arrington actually promises lawsuits against the manufacturers:

We will almost certainly be filing multiple lawsuits against Fusion Garage, and possibly Chandra and his shareholders as individuals, shortly.

It seems like Arrington has painted himself into a corner here.

If Arrington does file lawsuits, he’s admitting that he was never serious about the “open source” thing to begin with, and that he was drumming up support for the project under false pretenses. What’s more, he’s throwing his full weight behind restrictive intellectual property law. In other words, “Fuck you, freetards. You shouldn’t ever have trusted me in the first place.”

If Arrington doesn’t file lawsuits, he’s admitting:

1. that his legal threats are all noise, and that he shouldn’t be taken seriously in the future, and
2. that he’s such a lousy businessman he didn’t know who actually owned the intellectual property (if any even existed) of the project.

My expectation is that he’ll realize he can’t win in court (which he won’t admit), and that the CrunchPad prototypes were so lousy that there won’t be any money to be won there anyway (which he also won’t admit). The entire debacle is nothing but a huge embarrassment for Arrington, so he’ll quietly drop it and hope all his readers forget it quickly.

Translating Arrington ※

Posted Monday, 30 November 2009

Let’s translate Michael Arrington’s announcement that the CrunchPad is dead:

It was so close I could taste it. Two weeks ago we were ready to publicly launch the CrunchPad.

I’m using “launch” here in a bold new way, as I will now make clear.

The device was stable enough for a demo.

The device was not stable enough for release.

It went hours without crashing.

It never went more than a few hours without crashing.

We could even let people play with the device themselves

Even people inside the company couldn’t use it without making it crash.

the user interface was intuitive enough that people “got it” without any instructions.

It was still way harder to use than an iPhone, despite having several orders of magnitude fewer features.

And the look of pure joy on the handful of outsiders who had used it made the nearly 1.5 year effort completely worth it.

People are really polite to you if you tell them that what they’re holding is worth 18 months of your life.

Our plan was to debut the CrunchPad on stage at the Real-Time Crunchup event on November 20 […] We’d put 1,000 of the devices on pre-sale and take orders immediately. Larger scale production would begin early in 2010.

We hadn’t even come up with a real launch date, but were perfectly willing to take the money of 1,000 optimistic suckers.

And then the entire project self destructed over nothing more than greed, jealousy and miscommunication.

The project failed due to gross mismanagement.

[…] Bizarrely, we were being notified that we were no longer involved with the project. Our project. Chandra said that based on pressure from his shareholders he had decided to move forward and sell the device directly through Fusion Garage, without our involvement.

I had no idea that people don’t hand out a cut of their profits without good reason.

This is the equivalent of Foxconn, who build the iPhone, notifiying Apple a couple of days before launch that they’d be moving ahead and selling the iPhone directly without any involvement from Apple.

I’m pretty sure everybody at Apple spends their days blogging and googling for their own name, like I do.

[…] We jointly own the CrunchPad product intellectual property, and we solely own the CrunchPad trademark.

So it’s legally impossible for them to simply build and sell the device without our agreement.

It’d take an incredibly incompetent businessman to make the legal arrangements so vague that his pet project could be taken away from him. I don’t feel incompetent, so the project must still be mine.

[…] Renegotiations are always fine. But holding a gun to our head two days before launching and insulting us isn’t the way to do that.

Negotiations are usually very fair and polite when one person has a loaded gun and the other doesn’t.

We’ve spent the last week and a half trying unsuccessfully to communicate with them. Our calls and emails go unanswered, so we can’t even figure out exactly what’s happened.

It’s almost as though they’re telling us to fuck off. Man, that’s some hard-core “renegotiation”.

[…] We will almost certainly be filing multiple lawsuits against Fusion Garage, and possibly Chandra and his shareholders as individuals, shortly. The legal system will work it all out over time.

Most good technology products require more legal input than technical input, anyway.

Mostly though I’m just sad. I never envisioned the CrunchPad as a huge business.

I’m going to distance myself from the project now.

And what’s really sad about all this is the incredible support we were getting from companies and people around the world to launch this device. A major multi-billion dollar retail partner has been patiently working with us for months, giving advice on manufacturing partners and offering to sell the CrunchPad at a zero margin to help us succeed in the early days. They were also willing to pay for the devices on order instead of 30 days after delivery, a crucial cash flow benefit that would allow us to ramp up volume without putting ourselves our of business. They were even willing to fly the devices from China on their own planes to eliminate our shipping costs. Intel, which would supply the Atom CPUs to power the device, has assisted us repeatedly with engineering and partner advice, and gave us pricing that was ridiculously generous given our projected first year sales volumes. Other partners were eager to promote and sell the device for little or no benefit on their end other than “supporting the project.”

I’m really well-connected, and my business plan rested on my ability to get lots of stuff for free.

We even had sponsors lined up to help us sell the device near our $300ish cost.

The goal was to sell the CrunchPad for $200. With big subsidies from manufacturers, no shipping costs, no retail markup, software built with volunteer labor, and no profit whatsoever, we weren’t even close.

And money wasn’t a problem, either. We had blue chip angel and venture capitalist investors in Silicon Valley waiting to invest in the company since late Spring.

I didn’t think hiring someone who knows how to manage a tech project was worth the money.

We were simply holding them off until we launched, to eliminate some of the risk.

I’m using the word “risk” in a bold new way.

 ※

Posted Saturday, 07 November 2009

If you don’t feel that you are possibly on the edge of humiliating yourself, or losing control of the whole thing, then probably what you are doing isn’t very vital.

-John Irving

(via Merlin Mann)

Using the Oxford VPN in Mac OS X ※

Posted Sunday, 20 September 2009

Oxford University offers a “Virtual Private Network” service to its students, faculty, and staff. The most common reason people need to use this service is to get access to the wider internet using the Oxford Wirless LAN service. In fact, in almost all places where the OWL wireless network is available, the eduroam network is also available. Eduroam is a UK-wide network available at most major universities in the country and it does not require use of any special VPN, so I highly recommend that anyone new to Oxford take the trouble to configure their computers for eduroam instead of OWL. (The lesson, I’m afraid, is that IT services are better when they are not designed by the IT staff at Oxford.)

While the VPN is not necessary for wireless access, it may be required to access some Oxford-only services. The university encourages users to install a preconfigured version of the proprietary Cisco VPN client software onto their machines in order to do so. This is easy to do and does work, but I don’t really like it because:

1. It requires installing software distributed by the university. This is a (minor) security concern, because if an attacker compromises the university’s system they can inject malware into the software distribution. I prefer to install only software than I retrieve independently directly from well-known primary sources.
2. It requires non-standard system extensions which might cause reliability or performance problems. (I.e. it’s not just an application and can change the way the system functions even when it’s not running.)
3. The Mac version, at least, is lousy software. The interface is ugly and it interacts with other programs in an obnoxious way (making itself the frontmost application without user intervention). These aren’t major problems, but they suggest little understanding of the Mac platform and shoddy engineering throughout the software. The VPN client asks to save user passwords, but does not appear to use the standard “Keychain” system, for example—it’s thus likely the password in stored on disk either in the clear (which would be a major security violation) or using some obfuscation which is trivial to undo (also poor security policy).

Under Mac OS X 10.6 (Snow Leopard), and possibly earlier versions, there is no need to install Cisco’s proprietary client: the operating system includes built-in support for Cisco VPNs. To set up the Oxford VPN, you’ll need to first collect the following pieces of information:

1. Your remote access username. This should be the same as your single sign-on username, which begins with a few letters to identify your college or department and ends with a few digits. (At least, that’s what mine looks like.)
2. Your remote access password. Most people set this to be the same as their single sign-on password when they initially register, but when you change your single sign-on password your remote access password does not change. (Another sign of what happens when you leave systems to Oxford’s IT staff.)
3. The Oxford VPN “Shared Secret”. This is the same for everyone at Oxford, but it’s not supposed to be publicly available so I will not post it here. You can get it from the OUCS configuration file here, or you can get in touch with me directly.

Once you’ve gathered that information, select “System Preferences…” from the Apple menu and click on “Network” in the “Internet & Wireless” section. You should see something like this:

If the padlock icon in the lower left corner is locked, then click on it and provide your password to unlock the preference panel.

Click on the “+” button just above the padlock to add a new type of network interface. Choose “VPN” as the interface type, and then select “Cisco IPSec” in the VPN Type menu which appears. You can name the network whatever you want; I call it “Oxford VPN”.

Click on the “Create” button, and the new interface will appear in the list at the left of the window. Select the new VPN and fill in vpn.ox.ac.uk for the “Server Address”, your remote access username for “Account Name”, and your remote access password for “Password”. The easiest way to manage the VPN connection is using a menubar widget, so I tick the “Show VPN status in menu bar” box:

Finally, click on the “Authentication Settings…” button. The “Group Name” for the VPN is “oxford”; enter the “Shared Secret” as defined above.

Click “OK” on the Machine Authentication panel, and then “Apply” at the bottom right of the Network preference window. Click on “Connect” (or choose “Connect Oxford VPN” from the new VPN menu in the menubar) to test the configuration.

Now, whenever you are on a network anywhere in the world you can simply connect to the Oxford VPN to get full access to university resources, as though you were plugged into an ethernet cable in your department or college.

GPL in Practice ※

Posted Sunday, 20 September 2009

From an excellent case study on the practical impact of GPL licensing:

[An] Objective-C implementation is in two parts: the compiler and the runtime library. NeXT was only required to release its compiler code, not its runtime library. […] Rather than admit that it had a worthless bit of code, the GNU Project wrote its own Objective-C runtime—and this is where things really started getting interesting. The GNU runtime wasn’t quite a drop-in replacement for the NeXT runtime. […] Because the [non-GNU] branch only had to support a single runtime library, there was no clear abstraction between the runtime-specific and runtime-agnostic bits, and so the GNU branch quickly fell behind.

So forcing a company to release code it didn’t think anyone would find useful and had no intention of supporting ended up wasting the time and energy of a lot of open-source developers who devoted themselves to reconciling incompatible branches instead of actually improving the compiler. But it gets better:

Apple wanted to integrate the compiler more closely with its IDE. […] Unfortunately for Apple, the GPL required either making XCode open source or rewriting GCC. Apple opted for the latter choice.

In fact, Apple just decided to switch from GCC to a different open-source compiler: LLVM. There may be other technical reasons for the switch, but the fact that GCC is licensed under the GPL while LLVM is released under a less restrictive BSD-style license must have been a major factor.

My take is that the GNU political movement is undermining the most important and successful project under their control.

I’ll take this opportunity to beg all developers to stop using the GNU readline library as soon as possible and move to an alternative such as editline. GNU’s refusal to allow LGPL licensing for something that is so obviously a peripheral library feature just demonstrates that the organisation is far more interested in forcing their ideology on others than in the practicalities of creating good software.

Fantasy Football Scoring ※

Posted Sunday, 13 September 2009

I really enjoyed fantasy (American) football for the years I played, but it always upset me when players earned fantasy points for things that didn’t help their team, or were penalized for actions that did. The most common examples are scoring drives with the clock running down: a short pass to a receiver who is immediately tackled in the middle of the field is as bad as a sack in many such situations, but both quarterback and receiver get points for it. Similarly, an attempt at a 50-yard “hail mary” pass that results in an interception as time expires just shouldn’t be scored like other interceptions: it’s no worse for the quarterback and no better for the defense than an incompletion. Changing scoring to take account of such “game situation” parameters would be complicated, however, so I understand why few (if any) leagues make exceptions for such circumstances.

There is, however, one well-defined statistic which is very closely matched with a player’s contribution to his team, but is completely ignored by most fantasy leagues: first downs. A player who fights for the one extra yard needed for a new set of downs contributes a lot more than the tenth of a point (or less) given for a single yard of field position, quarterbacks who consistently throw passes just short of first-down yardage and must rely on running backs for first downs deserve fewer points than QBs who can do it themselves, and there should be some incentive to include reliable short-yardage backs on a fantasy team.

The simple fix is to give an offensive player an extra point if he gets a first down. This system could make consistent running backs and star “possession” receivers competitive with the “speed” receivers who are disproportionately rewarded for a small number of big plays (which can result in huge field position gains and touchdowns, but which are the direct result of extra plays that arise from steady progress down the field).

Offering a bonus point for a first down amounts to a much-simplified version of an approach taken by footballoutsiders.com, in developing the Defensive-Adjusted Value Over Average (DVOA) statistic:

DVOA measures not just yardage, but yardage towards a first down: five yards on third-and-4 are worth more than five yards on first-and-10 and much more than five yards on third-and-12.

Most current player statistics don’t include “number of first downs”, but it’s trivial to piece this together from play-by-play information, so it would be straightforward for Yahoo or any other fantasy football system to incorporate it as an option for their leagues.

Oscar Voting ※

Posted Wednesday, 02 September 2009

There’s a nice post today from the Wall Street Journal Numbers Guy about the new voting system for the Oscars. Since they increased the number of Best Picture nominees from five to ten, they’ve scrapped the “vote for a single film; the film with the most votes wins” system:

The new system for best-picture voting is just like the procedure for selecting nominees. It’s called single transferable vote and is similar to an instant-runoff election. Voters will rank the 10 nominees from 1 to 10. PricewaterhouseCoopers staffers who oversee the voting for the Academy will place the ballots into 10 piles, each one for ballots with one of the films ranked at the top. If one pile has 50% of the ballots, it wins. If not, the ballots in the smallest pile are added to the pile of the second-place film listed, and the procedure continues until one film has 50% of the votes.

I like ranked ballots because they are simple: voters need to write down a lot of information, but it’s information they can readily understand and communicate. The primary alternative for voting among more than two choices is the use of approval ballots, on which voters just pick a subset of the nominees they wouldn’t mind winning (without ranking them); the standard single-choice ballot is just a special case of approval ballots where you can only approve of a single nominee. Approval ballots encode less information but paradoxically make the voting process a lot more complicated. How do you decide what your threshold is for approval? If you think several nominees are acceptable but that one is much much better than the others, should you only approve of the one you think is far superior? The answers to these questions depend on not just your opinion of the nominees, but also your expectations about other voters’ behavior.

The instant-runoff method of tabulating ranked ballots, of course, is also vulnerable to certain types of strategic voting, whereby voters are best served by submitting a ballot whose ranking does not accurately reflect the voter’s preference. Beyond the fact that such opportunities are quite rare in practice, I don’t think that’s such a big deal in the long term. It’s the interface with voters that needs to be understood and adopted by all members of the Academy; that’s where the inertia in the system resides. Instant-runoff is the easiest tabulation system to explain, so it makes everyone more comfortable with ranked ballots during the changeover. Once voters are accustomed to such ballots, changing the way the ballots are tabulated to choose a winner is straightforward and inexpensive. A Condorcet tabulation method would eliminate the rare cases where misrepresenting your preferences on your ballot would be of benefit.

Our IT Overlords ※

Posted Tuesday, 01 September 2009

Last week, Farhad Manjoo, the technology columnist for Slate, published a piece highlighting the control many large organizations, including the US State Department, exert over their employees’ computers. He argued that such restrictive policies were often misguided. His reasons, in a nutshell:

The restrictions infantilize workers—they foster resentment, reduce morale, lock people into inefficient routines, and, worst of all, they kill our incentives to work productively.

My only real quibble with Manjoo’s article is that “modern” companies have already recognized this; it’s only the crusty behemoths whose IT departments have decades of legacy policy that continue to treat users like infants. Of course, such behemoths still represent a huge fraction of the economy and the work force, so his point stands. Some people don’t realize that IT has moved on since the 80s.

What I did find interesting, however, was a direct response from John C. Welch which he emailed to Manjoo and also posted publicly on his web site. Mr. Welch has been in IT “for around 20 years, from $bigCorp to higher ed, to small companies”, and from what I can tell he intends to speak for the industry. Let’s see how he rebuts accusations of infantilizing workers:

…New Media Douchebag…

…

This person was an idiot.

…

…you don’t care about that boring stuff like security or infrastructure, do you? You just want the new shiny, and if you get told ‘no’, then obviously, it’s just because IT are mean poopyheads.

…

FIGHT THE POWER!!! Yes, yes, I like Public Enemy too, but as it turns out, they aren’t a reliable network administration methodology. Who knew?

…

IT however, doesn’t live in the land of magical ponies and unicorns, where the internet is run by happy fairies and it’s all free.

…

Every time some nimrod PHB gets his panties in a bunch because someone was on MySpace, it’s our fault for allowing it, and word comes down from on high, block that site. What, you think we like having to run the idiotic reports about web usage and maintain the block filters?

…

…why would you bother to do any research? You’re leading a revolution!

…

As well, the implication that somehow, you know more about the advantages and use of things like Gmail than every IT person everywhere? Can you even sit within arms reach of a computer, with your head all bloated like that?

…

…rather than learn, you instead run the New Media Douchebag playbook, and assume that only you, Farhad Manjoo, is able to really understand what’s going on.

…

…a dead wombat knows more about IT than you do.

…

I’m sure what you’re doing is completely different. I bet you could even explain how in under a decade…once I’m in a coma…and deaf.

…

…we do what we’re ordered to, and take the blame from idiots without a clue…like you.

…

You don’t think it hasn’t occurred to us that if you just teach someone about what not to do they won’t do it? Do you think that kind of shit ever works? Maybe for 1% of people…

…

I’m pretty sure that no amount of reality will change you from your mighty crusade against the ebul that is IT.

…

Oh, you didn’t think you were the first one to write this kind of tripe, did you?

See? Not only are you wrong, but you’re not even original.

Welch’s attitude towards his users, his bosses, and any critics of the IT industry is clear.

He does, of course, also include some unsubstantiated claims in his rant. It’s mostly just transparent hand-waving and misdirection, but I’ll make an attempt at a few of them:

• When employees need to beg the CEO for help getting around IT policy that makes it hard to work, the IT department has failed at its job. Disastrously.
• An IT worker who considers the cost of disallowing any and all software to be zero needs a better understanding of “cost”.
• We all know you don’t need intimate familiarity with every last detail of a network, “down to OS versions on routers and servers”, in order to install software. If you think that’s necessary, you don’t understand modern IT. And your security policy sucks.
• We also have some idea of what it takes to keep computers and networks running. Most of us do that at home these days. Vague claims that “[it’s] not easy. No matter what your Googleh.D in computer science tells you”, are not good enough any more.
• If management is making IT decisions, then they are running your IT department. Blaming bad policy on internal politics isn’t really a defense.
• We understand how email works. Forwarding a message to GMail doesn’t force you to delete the original. Really. Even if you throw acronyms like EAS and BES at us.
• If anybody wants to intentionally leak confidential data they can. Spending your time blocking IM and Twitter won’t change that.
• We install applications on our personal machines all the time, yet we do not “spend hours a day dealing with application/OS/network interaction”.

When big-company IT is no longer dominated by pathetic and incompetent despots then maybe the profession will regain a little bit of respect.

Update

Mr. Welch responds. And has apparently read (one line of) my bio. And believes that shell scripts figure prominently in computing research.

Health-Care Taxonomy ※

Posted Monday, 24 August 2009

An excellent article (which is worth reading in full) from Foreign Policy magazine makes an incidental clarification of some of the terminology used in the health care debate. It’s obviously nonsense that Obama’s plan is a “single-payer” system, but I hadn’t considered the fact that even if it were, it still wouldn’t be a socialized system. Here’s the terminology:

Socialized health-care system

A government entity directly employs doctors and owns the hospitals. This allows top-down management of health care, with all attendant advantages and disadvantages, but of course any such system can be internally organized with varying levels of competition and independent management. The existence of such a system in no way rules out the possibility of competing or supplementary systems: education in the US is socialized, but there are many private schools outside that system (particularly at the university level). Britain’s NHS and the US Veteran’s Affairs Department are socialized health-care systems, and both co-exist with supplementary private systems.

Single-payer system

Doctors and hospitals are private and operate independently, often as for-profit institutions, but there is a single government entity that pays for most care. This hugely simplifies the billing process, and is the most straightforward way of supporting a “right to basic health care” for the entire population. Private insurance and payment plans can co-exist with the system to support treatments not covered by the single payer. Canada, France, and Germany have single-payer systems. Medicare is a single-payer system.

Private health-care system

Doctors and hospitals are private and operate independently, and people rely entirely on an open marketplace for medical insurance, where any government plans compete directly with private plans. All countries that operate in this way impose some kind of regulation on insurance, and many enforce a “right to basic health care” by requiring that every citizen purchase an insurance plan that meets a set of minimal standards, and by subsidizing the cost of such plans. Switzerland, the Netherlands, and Obama’s plan all follow this model (including regulation, mandatory insurance, and subsidies).

The US

The current US system is socialized for veterans, single-payer for the elderly and those with certain disabilities (Medicare) and the poor (Medicaid), and private for most other segments of the population. The private portion currently includes (piecemeal) regulation but does not mandate insurance coverage. Instead of subsidies to help the poor buy into the private system, they are pushed into the Medicaid single-payer system.

Star Wars Design Flaws ※

Posted Friday, 21 August 2009

Writer John Scalzi has posted a collection of “design flaws” in the Star Wars universe. I understand that this is just harmless fun, but his criticism seems typical of the mindless “what a bunch of idiots!” the ignorant like to level against experts with specialized knowledge and extensive experience. In this case, fictional engineering experts. Okay, so I’m taking this too seriously. Anyway, my rebuttal:

R2-D2

Sure, he’s cute, but the flaws in his design are obvious the first time he approaches anything but the shallowest of stairs.

Apparently Scalzi isn’t aware that (due to a retcon in the prequels) R2 has jets. Oh, wait, he is aware of that:

Also: He has jets, a periscope, a taser and oil canisters to make enforcer droids fall about in slapsticky fashion – and no voice synthesizer. Imagine that design conversation: “Yes, we can afford slapstick oil and tasers, but we’ll never get a 30-cent voice chip past accounting. That’s just madness.”

Everybody in the Star Wars universe can understand R2. The only people who can’t are the viewers of the movie. I always thought making that work so well was an indication of just how well the (first two) Star Wars movies had been scripted.

C-3PO

Can’t fully extend his arms; has a bunch of exposed wiring in his abs; walks and runs as if he has the droid equivalent of arthritis. And you say, well, he was put together by an eight-year-old. Yes, but a trip to the nearest Radio Shack would fix that.

We have no idea what purpose 3PO was built for. If his only job is to stand around and serve as translator or advisor, one presumes that physical mobility was nothing but an afterthought of design.

Also, I’m still waiting to hear the rationale for making a protocol droid a shrieking coward, aside from George Lucas rummaging through a box of offensive stereotypes (which he’d later return to while building Jar-Jar Binks) and picking out the “mincing gay man” module.

If a system were designed to warn people about subtle breaches of etiquette, then it wouldn’t be terribly surprising for life-threatening breaches to cause it to respond with loud and insistent panic.

Lightsabers

Yes, I know, I want one too. But I tell you what: I want one with a hand guard. Otherwise every lightsaber battle would consist of sabers clashing and then their owners sliding as quickly as possible down the shaft to lop off their opponent’s fingers.

Except that, you know, that doesn’t happen. Appearances indicate that two lightsabers can’t slide along each other at all—there’s a lot of friction (or other such resistive force) between the two. Adding a crossguard would address a problem that doesn’t exist, and would undermine the sleek shape of a retracted lightsaber, which is really the coolest part.

Blasters

A tactical nightmare: They’re incredibly loud,

Because noisy weapons would never catch on.

especially for firing what are essentially light beams.

Lightning is quite loud, too, and that noise is just caused by suddenly-superheated air.

The fire ordnance is so slow it can be dodged,

There’s no evidence of this. You can dodge where a blaster is pointed just as you can dodge where a gun is pointed. (The Jedi reflect-the-blast technique is supposedly based on their magical powers of precognition, but I really just chalk it up to suspension of disbelief and Lucas’s focus on cool Jedi tricks after the first two films.)

and it comes out as a streak of light that reveals your position to your enemies.

Regular rifles emit flashes of light that give away your position as well. Snipers use special weapons with flash suppressors to eliminate this problem; there’s no reason to believe the Star Wars universe doesn’t offer similarly-specialized weapons.

Let’s not even go near the idea of light beams being slow enough to dodge; that’s just something you have let go of, or risk insanity.

They’re called blasters, not lasers. The fact that something is really bright doesn’t mean it’s light—in fact, the light you see from incredibly powerful lasers is not the laser light.

Maybe blasters are spitting out bursts of plasma. I don’t know. But I think if you offered the US military a lightweight weapon that never runs out of ammunition they’d take it in a heartbeat.

Landspeeders and other flying vehicles

Here’s the thing: In the Star Wars universe, there are no seatbelts. And maybe if you’re flying your hoity-toity vehicle on Coruscant, you have, like, a force field that keeps you flying out of your seat. But Luke’s X-34 speeder on Tatooine? The Yugo of speeders, man. One hard stop, and out you go.

I’m having trouble even seeing the “design flaw” here. The fact that Jedis and farmers don’t tend to wear their seatbelts?

Stormtrooper Uniforms

They stand out like a sore thumb in every environment but snow, the helmets restrict view (“I can’t see a thing in this helmet!” – Luke Skywalker),

Luke is the only one who ever had this problem, for the simple reason that the helmet didn’t fit him. (“Aren’t you a little short for a stormtropper?” – Princess Leia.)

and the armor is penetrable by single shots from blasters. Add it all up and you have to wonder why stormtroopers don’t just walk around naked, save for blinders and flip-flops.

As armor, the stormtrooper uniform is terrible. But you do have to dress them in something.

Death Star

An unshielded exhaust port leading directly to the central reactor? Really?

It was not a straight blaster shot to blow up the reactor; you needed to get a “proton torpedo” into the port, at which point the torpedo changed direction and self-navigated down to the reactor. It’s a flaw, as was readily admitted, but a flaw that was only discovered through lengthy analysis, and that required the construction of specialized weaponry to exploit.

And when you rebuild it, your solution to this problem is four paths into the central core so large that you can literally fly a spaceship through them? Brilliant.

At this point we can only conclude that John Scalzi never watched the film to which he’s referring. The second Death Star was only half-built; huge portions of the spherical structure were entirely missing. That was part of the plan: make it look vulnerable to bait the rebels into an attack. Defense of the space station was left to a deflector shield projected from a nearby moon. The shield was 100% effective until it was blown up. The script for this film was atrocious, but none of the above elements constitute design flaws.

Sarlaac

A monstrous yet immobile creature who lives in an exposed pit in the middle of a lifeless desert, waiting for large animals to apparently feel suicidal and trek out to throw themselves in? Yeah, not so much. Not every Sarlaac can count on an intergalactic mob boss to feed it tidbits.

This assumes a lot of things about Sarlaac that we don’t know. Not every duck can count on tourists to feed it bread. Somehow wild ducks manage.

That Asteroid Worm Thing in Empire Strikes Back

So, large space worm lives in asteroid, disguises itself as a cave and waits for unwary spaceships to fly by so it can eat them? Makes the Sarlaac look like a marvel of natural selection, it does.

As above. There’s no indication that the worm, or Sarlaac, actually have any interest in eating people (or spaceships); for all we know they feed on rock, or their “live under the surface in a cave” behavior is just part of a lengthy hibernation.

Reframing the previous rebuttal: not every fountain can count on tourists to feed it coins.

Midi-Chlorians

Oh, man, don’t get me started. Except to say this: If in fact a high concentration of midi-chlorians is the difference between being a common schmoe and being a dude who can Force Choke his enemies, the black market in midi-chlorian injections must be amazing.

If having the right genes is the difference between being a common schmoe and being Usain Bolt, the black market in Bolt DNA injections must be amazing.

The Moral

If you know something about science and you watch sci-fi, you may think “that’s impossible”. If you’re a scientist at heart, you think “what would it mean if that were possible?”

If you know something about engineering, you may think “that couldn’t work”. If you’re an engineer at heart you think “how does that work?”, or even “what would you need to do to make that work?”

Being a scientist or engineer is more about grappling with questions than knowing a collection of answers.

Health Update ※

Posted Monday, 17 August 2009

This will be my most self-absorbed post yet. If you’re not fascinated by people telling you about their current medical conditions, you might want to skip this one.

Swine Flu Statistics

In the UK, they’re trying to prevent spread of swine flu by telling everyone with flu symptoms to stay away from the doctor’s office. Instead they give out a number for the national flu center and they diagnose you over the phone. Based on my fever, cough, headache, and diarrhea I was considered a swine flu risk. I have no problem with such a cautious approach, but don’t take any “number of swine flu cases” statistics seriously—they’re labeling every case of flu swine flu these days.

Tamiflu

The antiviral drug oseltamivir, marketed as Tamiflu, is nasty stuff. The only high-quality study I’ve found indicates only slight benefit from taking it (around 1 day less of illness), and in my case it caused vomiting. Repeatedly. If I had read the study first I might not have taken it at all, but once I started my five-day course I was obliged to finish it.

Dehydration

I have seldom felt more miserable than the couple of days I was suffering both severe diarrhea and enough nausea and vomiting that I had trouble getting any liquids. A couple of days of this just means discomfort; more than that and good medical care would put you on an IV; in regions without well-equipped facilities the combination is deadly.

I estimate that I saved around $10 on food for each day I was ill. I donated that savings to a project that provides clean water to communities that don’t have it. Beyond my latest little reminder, I consider charities like this much better use of money than carbon offsets or care for stray dogs in rich countries.

Ketosis

The most interesting part of this experience for me was that after a few days without any significant caloric intake, I developed a bitter taste in my mouth. The taste was even more concentrated on my lips, and my skin also tasted bitter. Showering, rinsing my mouth, and brushing my teeth couldn’t get rid of this.

I now suspect that this was a result of a “ketonic metabolism”: my body was burning its own tissue for energy, which results in acetone as a waste product. The acetone gets into your sweat and saliva, causing bad breath and a bitter taste.

I’ve been undernourished for a few days many times in the past (i.e. consuming at least 1000 kCal/day less than I was burning), either because I was too lazy to get food while working on a project or because I didn’t make the effort to replace all the calories I lost to exercise. I’m certain I’ve had a ketonic metabolism many times before, but I’ve never noticed such an acute taste. I assume that the difference this time is that I’ve spent most of the last year trying to gain weight, which has meant both eating constantly and eliminating the long runs I used to do—i.e. avoiding any chance for my body to break down its own tissue. I guess my body isn’t as efficient as when it was burning fat for energy on a more regular basis.

Health

I am in no place to complain about my health. I have no serious chronic conditions, no disabilities, and have never had any serious accidents or health problems. My immune system, however, has been poor since the day I was born, so I probably spend a month of every year incapacitated by “minor” colds and flus. I spend another month of every year suffering from allergies that make me too much of a mess to interact with others face-to-face. 2009 has been a particularly bad year—I’ve spent most of the time since April with one malady or another.

I think this may be a major reason why fitness has been such a big part of my adult life. If you’re going to be weak and enervated so much of the time, you’ve got to make up for it on the days you’re at 100%.

Fixing Automatic Door Slammers ※

Posted Sunday, 16 August 2009

Everyone is familiar with the spring boxes attached to doors, which usually look something like this:

These are formally called “door closers”, and such devices are the cheapest way of conforming to the strict fire codes required in just about anything other than a single-family home.

In practice, a lot of closers stop you from closing the door quickly, meaning you have to leave it to the spring to close, but then they slam the door over the last six inches. If I were to carve a list of commandments to those around me, a prohibition on door slamming would be included.

It’s easy to adjust most door closers. They’re all fairly similar. To adjust the Briton 2003 pictured above, you’ll need:

• A smallish philips-head screwdriver
• A sturdy flat-head screwdriver

First you need to take the cover off the door closer. On this model it’s attached by one small screw on either side.

After removing the two screws, the cover should come off easily.

There are adjustment controls on the end of the cylinder.

The controls look like this:

Use a flat screwdriver to turn the center screw left to make the door close faster; turn the center screw right to close the door more slowly.

Around the center screw is a metal ring with two off-center notches. This ring controls whether or not the door is slammed over the last six inches (which is intended to overcome any resistance from the door’s latch), or whether it is closed smoothly the whole way. Rotate the ring 180 degrees to change the setting. There should not be much resistance to turning it; you should be able to turn it with the help of a flat screwdriver in the notch. It will probably only turn in one direction.

Even if your door has a latch, I recommend setting the door to close smoothly the whole way. You can then adjust the overall speed of closing just high enough to overcome the latch without slamming.

US Health Care ※

Posted Wednesday, 12 August 2009

I could go on at length about what the health care “debate” says about US politics and culture, but I’ll restrict myself to five points:

1. The health-care bill is publicly available. The language is fairly accessible. It’s long, but given the double-spacing and 50-character line lengths it’s the kind of thing you could easily read in a day. Claims about the bill are easy to check. Most of the claims being made by opponents of the bill are flat-out lies.

2. The bill leaves the basic US health-care infrastructure entirely in place. It does two things: it sets up an insurance plan provided by the government comparable to, and competitive with, the low-end plans from private insurers, and it says that everybody needs to sign up to some kind of insurance (by punishing those who don’t with a tax penalty). A debate over the merits of a single-payer system, or salary- versus procedure-based pay for doctors, or socialism in general, has no relevance to the proposed bill.

3. The only real facts that can be debated are the economic consequences of a new competitor in the insurance market: if the government plan covers a lot and is heavily subsidized then private insurers may have trouble competing; if it covers little and receives little subsidy then nobody would choose it over private insurance. Debate on this topic has been completely incoherent: if you’re worried about the new plan hurting private insurers, then the fact that it might not cover certain treatments is surely a good thing.

4. If you quit your job to start a one-man business (e.g. software developer; web designer; artist; consultant), you could cut most of your costs to almost nothing: move somewhere with cheap rent, switch to a small used car, etc. The cost that will break you in the year(s) before you’ve built back up to a basic salary is health insurance. The US system is a huge discouragement to entrepreneurship, innovation, and the kind of “flexible employment” that the world so desperately needs.

5. My view is that basic health care is a right in the same sense that basic childhood education is a right. Putting your kids through school isn’t dependent upon the education plan offered by your employer; it’s an unexpected consequence of antiquated tax policy that health care is.

Windows Pricing ※

Posted Thursday, 06 August 2009

Jeff Atwood has been considering the profitability of low-priced software, citing the iPhone App Store and discounted games prices as examples. He goes on to opine about Windows pricing:

Say the Windows 7 upgrade price was a more rational $49, or $69. I’m sure the thought of that drives the Redmond consumer surplus capturing marketing weasels apoplectic. But the Valve data – and my own gut intuition – leads me to believe that they’d actually make more money if they priced their software at the “why not?” level.

This may be good advice, but it fails to address the uniqueness of Windows, both in the present climate and historically. Microsoft has always sold most of the copies of Windows to computer manufacturers and large organizations—two groups unlikely to make impulse purchases.

Further, upgrading to Windows 7 will be “a tedious, painful process” for the vast majority of users. You can buy a game or an iPhone app for a couple of bucks, and if you don’t like it that’s not a big deal. If you try Windows 7 and don’t like it you’ve just cost yourself several hours (at least) of work.

As anyone familiar with Linux will tell you, where operating systems are concerned people have a lot of answers to “why not?” unrelated to price.

Tablet Rumors ※

Posted Monday, 03 August 2009

A few obvious thoughts on the latest tablet “report”:

• Inviting an analyst to gawk at your secret project is like telling your brother-in-law about your marital infidelity. Apple’s not that stupid.
• At Apple, products “awaiting Steve Jobs’ final blessing” are not necessarily going to be released. Ever.
• The reason Apple does so well with these consumer devices is that they realize it’s all about the software. The software is the part that can’t be leaked by the supply chain; Apple’s real secrets are kept in-house.

A Strange Definition of "Externality" ※

Posted Monday, 03 August 2009

I just came across a surprising paper from economists Aaron S. Edlin and Pinar Karaca-Mandic published in Journal of Political Economy in 2006 which claims to address the “accident externality” due to driving. Here’s the abstract:

We estimate auto accident externalities (more specifically insurance externalities) using panel data on state-average insurance premiums and loss costs. Externalities appear to be substantial in traffic-dense states: in California, for example, we find that the increase in traffic density from a typical additional driver increases total statewide in- surance costs of other drivers by $1,725–$3,239 per year, depending on the model. High–traffic density states have large economically and statistically significant externalities in all specifications we check. In contrast, the accident externality per driver in low-traffic states appears quite small. On balance, accident externalities are so large that a correcting Pigouvian tax could raise $66 billion annually in California alone, more than all existing California state taxes during our study period, and over $220 billion per year nationally.

So if I move to California and start driving, not only do I have to pay for my own insurance, but everybody else’s insurance premiums will go up by a fraction of a penny; in total everyone else will pay an extra couple of thousand dollars. Edlin and Karaca-Mandic call this an externality.

I don’t get it. This extra cost is only an externality if you can pick somebody out as the “extra driver”, but you can’t do that. Each driver who pays insurance is covering the cost of all the others drivers’ “externalities”, just as they are covering the cost of his.

Let’s rejuggle the books to eliminate the externalities: assume there are 20 million drivers in California and the accident externality I impose on each of them by driving there is one hundredth of a cent (for a total of $2000). I’ll take this entire cost on myself; my premium goes up $2000. But now I’m not going to cover other drivers’ accident externalities. My initial premium included one hundredth of a cent of cost due to each of those 20 million drivers, so we can eliminate that; my premium goes back down $2000 to its original level.

Entirely through economic sleight of hand, Edlin and Karaca-Mandic make a case for an extra $66 billion tax on California motorists. They not only avoid mentioning that the costs they are trying to account for are already taken up by insurance premiums (they actually calculate the costs by analyzing how premiums vary with traffic density), but also suggest that the tax be imposed on those premiums, at a rate of 200–400 percent! If the money were actually being spent to address the externality (the increased risk of accidents), then insurance premiums would fall accordingly and the state would take over the bulk of the insurance industry, with responsibility for covering all accidents caused by other cars on the road. In practice, however, the authors propose only that the state pocket the money as tax revenue. Insurance premiums would still have to take traffic density into account, so lower density could lead to lower premiums, but this would be nothing like enough to cover the tax increase and the cost of driving would be much much higher. (These higher costs are the only reason to assume any reduction in traffic density.)

If you want to discourage driving to decrease pollution, to prevent traffic deaths, or to alter the character of a community, I can understand that. Justifying taxes with economic obfuscation, however, is not cool.

(Brought to my attention by Andrew Gelman.)

Android Falls Further Behind ※

Posted Saturday, 27 June 2009

In a previous post I argued that it was Palm’s WebOS that had emerged as a legitimate threat to Apple’s iPhone; Google’s Android platform is left in third place to compete with the Blackberry OS, Windows Mobile, and Symbian—all of which now look very much like previous-generation technology. I also said this about Android:

Frankly, I’m not sold on the business sense of this project: Android’s architecture doesn’t seem nearly web-centric enough to be of direct benefit to Google.

The main programming interface for Android is based on Java. I’ll spare you my rant about Java as my least-favorite language, but I will say this: relatively speaking, Java is not a modern programming language. The trend over the last decade has been towards languages which place less importance on typing (e.g. Python, Ruby, and Javascript), or which rely on message-passing (e.g. Objective-C) instead of function calls.

In the past week, Google released the Native Development Kit for Android, which allows Android development in C and C++. I’m a big fan of C++, but this just seems to be another giant step the wrong way along the programming-language timeline.

I’ve always thought that the main difference between Android and the iPhone is that Android is meant to run on lots of different kinds of hardware: some with physical keyboards, some without; some with GPS, some without; etc. This is an advantage in that it allows manufacturers to create Android phones for market segments that the iPhone isn’t addressing; Android could thus build up a huge user base. The size of the user base then makes the Android platform compelling for developers, who create and sell applications for all those phones.

This disadvantage to such heterogeneous hardware is the difficulty in creating applications which work well in all situations. An interface designed for a physical keyboard is nearly unusable on a device with only a touchscreen, and an interface designed only for a touchscreen is wasteful and inefficient on a device with a keyboard. What’s more, applications then need to be separated into categories for the classes of devices they run on: some apps may only work if GPS is available, for example. The prospect of a single unified app store (and thus a single market larger than the iPhone market) is remote.

A native development kit further undermines the advantages and enhances the disadvantages of different hardware configurations. Native apps actually need to be recompiled for every different type of processor (and possibly each different memory architecture) used in devices. Worse, in the cases where native development is needed to accomplish things that the Java SDK cannot, the app might need to be completely reimplemented for each different device.

I understand the desire for a native SDK: native apps can be much faster and more efficient (in terms of both power and memory) than Java apps. But that’s a backwards-looking philosophy. A non-native app which runs too slowly on today’s phone will run perfectly fine on tomorrow’s. A native app will run fine on today’s phone but won’t run at all on tomorrow’s. Apps like that won’t encourage a customer to buy a new Android phone to replace their old one, and this “just make it work today; we’ll worry about tomorrow when it comes” approach seems much more appropriate for a company in desperate need of immediate success (e.g. Palm) than one with the cash to gain ground slowly over time (e.g. Google).

There is, however, a mobile operating system which can be programmed using a modern, dynamic language. In fact, its SDK is based on perhaps the most widely-used languages in the world: HTML and Javascript, just like all the web applications the huge community of Google-centric developers are already building. This architecture actively encourages a blurring of the line between stand-alone app and web application, thus exploiting the mobile market to enhance the importance of the web. What is more, all of Google’s research on optimized Javascript runtimes is directly applicable to making these mobile applications run faster, without any change to their code. As I mentioned on Twitter, I find it very odd that this operating system is made by Palm, and not by Google.

Parsing Digg ※

Posted Monday, 15 June 2009

A reader just let me know that my repackaged Digg feed recently stopped working.

It turns out the HTML on the Digg web site is now so bad that even Beautiful Soup can’t parse it.

I’ve written about making Beautiful Soup even more tolerant before. Shortly after I posted that information, Leonard Richardson explained why Beautiful Soup 3.1 was failing on malformed pages. Rather than hack around with the parsing infrastructure as I had done before, I’ve just taken his advice and downgraded to Beautiful Soup version 3.0.7a.

The digg-direct feed should now be working again. Apologies for the outage.

NBA International League Pass ※

Posted Sunday, 14 June 2009

Not many NBA playoff games make it onto television in the UK; usually it’s only the finals. This year I’ve been watching over the internet using the NBA’s International League Pass service.

The technology the service uses is lousy. You can only watch the games using the RayV player browser plugin, which (for the Mac at least) is incredibly flaky. It requires admin access to my machine, it crashes my browser regularly, live streams need to be manually refreshed periodically, and the plugin seems to need re-installation every time I restart despite automatically launching a daemon at login. And in a business decision worthy of a company whose home page looks like this, they release a new version of the player every few days, hassling all their users to upgrade every time they watch a video. Despite all this, internet video is a simple enough problem that even RayV hasn’t managed to make the games completely unwatchable.

For international customers (only), the NBA offers two packages: you get access to every playoff game in “standard definition” for $24.95, or every game in “high definition” for $34.95. I only subscribed to the standard definition package this year, and the quality is comparable to YouTube: easily good enough to follow the action and read the players’ body language, but you’re not going to pick up on facial expressions in the long shots.

The price seemed pretty reasonable; much more and I wouldn’t have paid for it this year. But affordable pricing for sports packages is rare. I wanted to crunch the numbers to see whether the price makes it competitive with TV ad-supported broadcasts.

The online video is actually just a copy of the US television feed, and includes all of the US ads, so in theory the NBA could claim some of the network’s ad revenue, or insert its own ads during the breaks. But let’s assume an ad-free feed; how much would you have to charge users to make up for the ads they’re not seeing?

I have very little data at my disposal, but the few sources I’ve found quote rates around 5 US dollars per thousand viewers or 4.62 British pounds per thousand viewers for a 30-second spot. The first figure suggests that the value of each (potential) viewer of an ad is roughly half a cent. Assuming 48 minutes of advertising in a three-hour basketball game, that puts a price of 24 cents on each viewer. If the TV networks could keep the same ratings, but charge every viewer a quarter, then they could drop advertising entirely and maintain the same revenues. Their profits would go up, however, since they spend money getting all those ads sold.

The NBA playoffs include fifteen series, each with up to seven games, for a maximum of 105 playoff games per season; this season there will be between 84 and 87 playoff games. A viewer who watches every single game on television increases the networks’ total advertising revenue by about $21 this year, and by about $26 in a year when every series goes seven games.

In the simplest model, the TV networks should be perfectly happy to give up a viewer to watching on the internet (and possibly missing all their ads) if they are paid $25. More importantly, if the NBA could get $25 from everybody who watched the playoffs they wouldn’t need to sell television rights to any network at all. They’d have to get somebody to produce the game video, but the revenue they’re generating would cover that, just as TV ad revenue covers it now.

Those are the numbers I came up with. Now some thoughts:

• Televised sports might actually lose money at current advertising rates. I’ve heard that many networks consider sports to be a loss leader that doesn’t generate profits, but allows them to raise the stature of the network. I.e. getting viewers used to watching sports on ABC makes them more likely to watch other shows on ABC in the future, and it is those other shows that make money.

• I wonder whether 48 minutes of ads per game is accurate. Among other things, there’s a lot of promotion for TV shows and movies that the networks try to slip in for a few seconds here and there.

• I wonder how many games the average viewer actually watches. More than half of the games are in the first round, and many are going on at the same time, so it’s clearly not possible to watch everything. I couldn’t watch more than about one game a day, or two different series in each round, which comes out to 49 games maximum. This year I probably watched at least some part of about 24 games (for an estimated ad revenue of $6), but I liked that I had the option to watch any others for no extra cost.

• There is far more profit to be made from the same basic revenue if the NBA sells directly to viewers instead of selling rights to networks, who sell ads to ad agencies, who make ads for advertisers, who sell products to viewers. Each of those intermediaries is taking some of the profit the NBA could keep for themselves.

• Is it better that the NBA is kept at arms length from the advertising? The TV networks have done everything they can to lace every aspect of their coverage of the game with advertising, from announcers reading promos for movies or TV shows to in-game interviews with celebrities promoting their latest project. This sets up a useful tension between the network and the NBA, who care more about keeping the sport on the floor entertaining than about advertising. The logos and product endorsements already on players’ uniforms might get a lot worse if there was just one entity managing all of basketball advertising.

• How much does internet video distribution cost? The TV networks have certain costs related to broadcasting, but I assume that those costs become much lower than internet distriution per-viewer once there are enough viewers. How mature are the various internet multicast technologies?

• How hard will the sports leagues push to use internet streaming to achieve market segmentation? Only hardcore fans are likely to pay to watch online, and they are obviously willing to pay more to watch games than casual fans. If the leagues decide that national television is for casual and/or regional fans and the internet is for hardcore fans, then there’s no reason to believe that prices for the two will align.

• When will iTunes add support for streaming of live events?

Peak Google ※

Posted Friday, 05 June 2009

In a post five months ago I claimed that Google hadn’t seen its peak yet. While I expect the company’s revenue to continue to climb, I’m now a lot more skeptical of its influence and credibility as an innovator.

First, of course, it’s worth pointing out that Google’s industry reputation has been completely off the charts for the past few years. Again and again I’ve heard them described as the most exciting and innovative company, working on absolutely the most advanced technology, with the most brilliant researchers and engineers in the world. Google lectures at Manchester and Oxford were standing-room-only affairs. If Google had announced that one of its secret research teams had found the cure for cancer or the key to cold fusion, most people wouldn’t have been surprised. No company can live up to those kinds of expectations.

Beyond the dominance in search and advertising, I’ve seen three major successes from Google. Mail and Maps are two, and while they’re both solid products (good technology, and a great fit with Google’s business model) I find it tough to get excited about them. They were the first big hits of the AJAX web application era, but the techniques didn’t originate with Google and there are now better copycat apps. Their third success, Chrome, is much more ambitious, both in terms of business model (a shrewd investment in the infrastructure needed to deliver their core products) and technology (a genuinely novel internal architecture for an existing class of application, and one much better suited to the “web application” paradigm Google endorses). Although Chrome’s market share among the general public barely registers, it’s already mature enough to compete with the major browsers, and I find that very impressive.

Unfortunately, a few nice web apps and a nifty repackaging of WebKit aren’t exactly the kind of world-changing innovation everyone was expecting. It wasn’t just Yahoo! and MSN who were frightened of Google in 2005; the thinking at the time was that Google was a threat in any market they chose to enter. Google’s more ambitious projects haven’t worked out so well.

They spent a couple of years talking big about municipal wifi. Lots of rhetoric and philosophy. Not a lot of actual wifi.

They got involved in the auction for the new wireless spectrum. And lost. I suspect Google was actually more interested in influencing the terms of the auction than in constructing the data network some analysts described. If so you can’t really call the auction a failure, but the incident didn’t do much for Google’s aura of invincibility.

Android has been Google’s biggest push into a new market. Frankly, I’m not sold on the business sense of this project: Android’s architecture doesn’t seem nearly web-centric enough to be of direct benefit to Google. While the effort can hardly be called a failure at this stage, Android is currently being overshadowed by both Apple’s iPhone and Palm’s Web OS. Android may well overtake Windows Mobile and Symbian at some point, but right now it doesn’t represent any real innovation in the mobile OS space.

Could Google Wave be the big success that everybody’s been waiting for? Despite all the hype the demo has received, I say no. A spiffy development framework isn’t the same as a killer product. Apple’s Cocoa framework is terrific, but it’s not in the same league with Mac OS X, the iPod, or the iPhone.

Google is a solid company and I think they’re covering their main focus well. In terms of innovation and ability to enter new markets, however, they’ve shown both arrogance (which seems the be the defining characteristic of Google’s culture) and weakness. Their mystique is fading.

June 4 ※

Posted Thursday, 04 June 2009

Chinese officials took this man away. He’s never been seen since.

Today the government has blocked off most of the internet to try to keep its own people (do they even count as “citizens”?) from learning what happened 20 years ago.

China has come a long way since 1989. But the country still has a long, long way to go.

IT and Security ※

Posted Monday, 01 June 2009

I had the rare “opportunity” to sit in on a meeting with an IT vendor today. This vendor sells a Network Access Control product which works as follows:

1. User connects to the network
2. User’s web browser is redirected to a web page which says “Please download and run this program: <link>”
3. User clicks the link, then runs the resulting .exe file. (There are comparable Mac and Linux versions as well.)
4. The program scans the user’s computer, tells a network server to enable access for the computer if it passes the scan, and deletes itself. (They call this a “dissolving client.”)

I attended this meeting because I had heard this story and wanted to see if it was true. I pointed out the gaping security hole being opened here: not only does this product allow anyone capable of spoofing this web page to take control of any user’s machine (in the vendor’s screen shot the page is not even secure), but it also encourages users to download and run software just because a web page tells them to.

The vendor agreed that this was somewhat problematic, and that they’d heard this concern before. At present, the company has no particular response.

Clearly this sucks, but I can get over it. The product on offer actually doesn’t need this client software to run to do much of its job; that feature can be easily disabled. The other network monitoring and management features of the product look genuinely useful.

What concern me are the responses my comment provoked. Not those from the vendor (at least not the immediate reponses)—the technical rep was quite forthright that my point was valid. My problem is with the responses from the IT reps in the room, a handful of whom had already installed the system on their network, but most of whom were only considering it. They immediately became far more defensive than even the vendor; apparently any security hole you don’t initially spot yourself is not really a security hole.

Some comments from IT professionals—people whose jobs it is to provide secure network services to students and staff at Oxford and its colleges:

That just hasn’t been a problem for us.

We haven’t had any complaints about that.

There hasn’t been a single report of any such attack.

I suppose such an attitude can be understood when you realize that network admins don’t actually care about security at all. What they care about is malware—software that generates crippling network traffic and/or user complaints. It’s not their problem if somebody out on the internet can read the entire contents of their users’ hard drives. Not even if it’s because of their own network policies. Nobody knows about it and they don’t hear about it, so the problem doesn’t exist.

More troubling was this comment from the vendor:

Really, how common is it for an attacker to disguise his malware as a security program?

There was an awkward silence among the IT crowd after that, and a slightly apologetic “actually, a lot of infected web sites have things like that…”

The group did start to acknowledge the problem, but this was repeatedly rebutted by one admin who had already deployed the product:

The point is, this product does what I want it to do. It makes my life a lot easier. That’s the bottom line.

Apparently ease of network administration trumps any security considerations. The amazing thing is that I really liked the product. I believe that its management interface is a convenient way to monitor a network. I just think you need to untick the “install client” box on the configuration screen. But the IT guys kept setting up this absurd duality: either install the full system including the security hole, or don’t get any of the benefits.

So the discussion turned to how to mitigate this risk. Again and again, one solution kept coming up. Keep in mind that the vendor had nothing to say about any of this—I kept directing my questions to the vendor and they kept being answered by the other IT guys in the room:

We’ll definitely let everybody know about this before they connect to the network. We’ll send them a letter telling them “you’ll be sent to a web page; it will tell you to download software; download it and run it”.

No suggestions of hashes or digital signatures; just “we’ll notify them”. To IT professionals “if we could only educate the users…” is the answer to everything.

If it’s not our client, then it won’t let them on the network, right? So we just need to tell users to run it, and if they don’t get on the network, then it wasn’t ours. And if they don’t run it and do get on the network anyway, then it wasn’t ours anyway.

I’m truly worried about IT professionals who try to bodge together security solutions with spit and duct tape.

But my absolute favorite comment, which was repeated to me no less than five times when we kept getting dragged down weird technical tangents, was this:

It’s really simple. If they don’t install the client, then they can’t get on our network. Simple as that. We don’t have to worry about users who won’t install the client—they’re free to go and find an internet connection somewhere else.

There are good and thoughtful IT pros out there. There’s no question that most of the guys in this meeting really wanted to do a good job and help their users. But the culture of IT is riddled with defensiveness and political posturing.

I can’t help but think that electricians and plumbers are different. I wish I could figure out exactly what the difference is.

Apple and the Netbook ※

Posted Sunday, 31 May 2009

I won’t bother rehashing all the rumors, but I would like to extend my previous thoughts about Apple and tablet computers.

The fact of the matter is that there does seem to be a significant demand for mobile computers. Many computer manufacturers have tried to meet this demand with “netbooks”: very small versions of laptops.

The trouble is that laptops aren’t really mobile computers; they’re portable desktop computers. If you want to use a laptop you put it down on a surface and lean over the keyboard and screen. It’s terrific that they work sitting on surfaces as lopsided and off-balance as your lap while lying on the couch, but the mechanics are still pretty much the same as the desktop computer.

A portable computer can be moved when it’s not in use; a mobile computer is designed for use while it’s moving. The iPhone interface isn’t good because it looks good and works well in the “white vacuum” of the commercials. It’s good because it works while you’re carrying a bag, listening to music, and strolling down the street. More substantial sessions with mobile computers are characterized by the extraordinarily low cost of a context switch: there’s no need to establish or clean up a work space. You can read a Kindle while waiting in line at the grocery store knowing you’re not going to hold anybody up.

The most surprising thing to me about the iPhone is not all the nifty little mobile applications—I would have guessed that it’s possible to do those well. What surprises me is how competent the iPhone is at things that I had previously considered to be desktop tasks: web browsing and email. It turns out that the touch interface is more than sufficient to navigate a web browser, and most web sites are perfectly tolerable on a smaller screen with the iPhone’s excellent pan and zoom features. The on-screen keyboard is nothing like as fast as a desktop keyboard, but it’s easily enough for writing a few paragraphs of prose for an email.

The iPhone is competent at web browsing and email, but it’s not good enough to be competitive with desktop machines. Mainly, the screen is just too small. As mobile computers go, the iPhone scores big on “mobile”, but it doesn’t quite feel like a “computer” in the sense that it could be a replacement for much of my desktop computer use.

There’s a demand for some kind of a mobile computer that can do web browsing, emails, feed reading, and possibly even e-book reading. The demand doesn’t include a need for legacy applications (as supported by the fact that casual users actually accept Linux instead of Windows on the current devices that try to fill the niche). Apple has the most expertise in the mobile computing market. A tablet would be unlikely to cannibalize sales of either the iPhone or any of the Apple laptops. It makes sense that Apple will release something in this space.

The outstanding question is when. If Apple wanted lots of third-party applications available for such a device, then they’d need to announce it at WWDC in a week’s time: developers will be setting their schedules based on what is announced there.

But Apple doesn’t necessarily need third-party apps at launch. I’ve seen analysts talk about how the App Store is the key to the iPhone’s success, but of course the iPhone was a hit when it had only Apple applications. They could easily ship a tablet computer with nothing but Safari, Mail, and an eBook reader developed in-house in conjunction with Amazon, and wait six months to allow any third-party apps. In fact, this approach would allow Apple to set the standard for how tablet interfaces should work.

There’s a third option: Apple could ship with only apps from a few hand-picked developers given early access to the SDK. Developers Apple trusts to build quality interfaces, and to provide useful feedback on the APIs. Developers who are better at building certain kinds of must-have applications than Apple is. Developers who can be trusted to keep their knowledge of the device top secret throughout the project until the day the product—and their app—launches.

Anybody know what Brent Simmons has been up to lately?

Manual of Style: Don't Preempt User Tools ※

Posted Sunday, 31 May 2009

The web is young enough that we have yet to achieve any real consensus on a robust manual of style. With technology changing so quickly, any detailed guide would rapidly fall out of date. But here’s a general rule that takes care of several of the common annoyances I find in even “professional”-level writing on the web:

• Don’t add meta-data unless you have additional information not available to your reader.

It’s actually a very simple rule. Meta-data, such as links, represents information. If you don’t actually have any information to add, don’t pretend you do. It undermines those cases when you do have something to say.

This rule explicates the wrong-headedness of several common practices:

• Don’t link to Wikipedia. If you use a term the reader doesn’t know, they can look it up on Wikipedia (or any other reference) themselves. There are even plugins that let users look up any word or phrase without much more trouble than clicking on a link. If you don’t have any more specific context for a term than its entry in the encyclopedia or dictionary then you don’t really have any information to offer.
• Don’t add ticker symbols to every company name. (I’m looking at you, New York Times.) People know how to look up business information. I’m sure there are plugins for this as well. I understand the desire to drive readers to your preferred site; just know that you’re sacrificing style to do it.
• Don’t “enhance” your links. Over the past couple of years publishers have been junking up their sites with tools such as Snap and Sphere that rewrite links to pop up graphical previews of their destinations or search results. Users who want previews can install special tools like Cooliris in their client.

Microsoft Distribution ※

Posted Friday, 06 March 2009

Robbie Bach of Microsoft on their plans for retail stores, as quoted by Benjamin J. Romano:

I don’t think – I saw some of the commentary that this was designed to be the same as Apple or whatever. You should think about it, I think, quite differently.

Apple’s approach was about distribution. People forget that when they entered their stores [in 2001], this was quite a while ago, they didn’t have distribution for Macintoshes, so they created their own distribution.

We have plenty of distribution. These stores for us are about building our connection to customers, about building our brand presence and about reaching out and understanding what works and what improves the selling experience.

So Apple you would think of as a volume distribution play. You should think of ours as much more of a brand and customer relationship investment more than anything else.

There’s already plenty of commentary on what this statement means and how Microsoft should run its new stores, but I just can’t get past the complete misunderstanding (or rewriting) of history here.

Before Apple Stores, you could buy Macs in plenty of places, including Sears and CompUSA (both of which were quite successful and popular at the time). Many argued that Apple was being far too stingy in who was allowed to sell Macs. My recollection is that it took longer for Apple to get shelf space in “non-computer” stores to sell iPods, but you’d be crazy to think that Apple opened up retail stores because they couldn’t find distribution channels for iPods.

Apple’s problem was that in stores selling Wintel PCs, the sales staff (who were likely Wintel users) knew nothing about Macs and were thus completely unhelpful to customers, or actively steered them towards Windows. Apple tried fighting this with special training programs and dedicated staff, but this just led to isolated “Mac ghettos”: having a few square feet in the dim back corner of a shop doesn’t do much for a brand’s image.

Apple opened shops not to get distribution, but in order to connect to customers directly, deliver the “well-designed; just works” brand message, and build customer relationships all the way from sales to support to repairs to upgrades.

Now Microsoft is whining that current retail channels aren’t delivering the message they want and that their brand image is suffering, so they’re going to open their own stores.

It’s one thing to play follow-the-leader with Apple—sometimes that’s the right play, and market leaders can often afford it. But these ridiculous denials are just embarrassing.

Tolerant HTML Parsing ※

Posted Sunday, 08 February 2009

Beautiful Soup is an absolutely terrific Python library for parsing HTML and XML. Its strength is its ability to offer a clean document tree even for bad markup (including such gory details as converting everything to unicode intelligently).

Unfortunately, Beautiful Soup’s tolerance for lousy markup is limited. It’s got lots of clever heuristics for repairing broken nesting (<b><i>foo</b></i>) and guessing implicitly-closed tags (<li>First<li>Second), but it uses Python’s standard HTML parser class to tokenize the markup. HTMLParser isn’t designed to accommodate syntactically malformed markup.

The case I’ve encountered in the wild involves “syntactically nested” tags—constructions of the form <a <br> href="foo">bar</a>. The nested tag is almost always a line-break; presumably this is the result of a particularly lousy tool attempting to do its own text-wrapping.

Although such markup is clearly atrocious, web browsers are fairly consistent in the way they render such fragments: anything up to the first > is part of the tag, and the rest is just text up to the < that opens the next tag. Both Safari and Firefox render my example as href="foo">bar, presumably wrapped in an a tag with no href attribute. Even TextMate’s HTML highlighter interprets the syntax in this way.

HTMLParser chokes on this syntax, however, making it impossible to use Beautiful Soup to process pages with such errors. HTMLParser.py uses the following regular expression to find the end of a tag it is parsing:

locatestarttagend = re.compile(r"""
  <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
  (?:\s+                             # whitespace before attribute name
    (?:[a-zA-Z_][-.:a-zA-Z0-9_]*     # attribute name
      (?:\s*=\s*                     # value indicator
        (?:'[^']*'                   # LITA-enclosed value
          |\"[^\"]*\"                # LIT-enclosed value
          |[^'\">\s]+                # bare value
         )
       )?
     )
   )*
  \s*                                # trailing whitespace
""", re.VERBOSE)

If whatever matches this expression is not followed by > or />, the check_for_whole_start_tag routine raises an exception.

A general replacement for HTMLParser which is designed from the ground up to handle questionable syntax (presumably just turning anything it can’t parse into text runs) would be a useful utility, but for now I just wanted to fix the particular cases I’ve encountered in practice. Modifying check_for_whole_start_tag such that it no longer raises exceptions is one option, but that doesn’t match the behavior of the web browsers: the existing version of locatestarttagend stops matching at the beginning of any syntactically nested tag, while the web browsers stop at the end of the nested tag. As a quick hack, I modified HTMLParser to allow attribute names beginning with <. This involves changing both locatestarttagend and the expression for identifying attributes:

attrfind = re.compile(
    r'\s*([<a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
    r'(\'[^\']*\'|"[^"]*"|[-a-zA-Z0-9./,:;+*%?!&$\(\)_#=~@]*))?')

locatestarttagend = re.compile(r"""
  <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
  (?:\s+                             # whitespace before attribute name
    (?:[<a-zA-Z_][-.:a-zA-Z0-9_]*     # attribute name
      (?:\s*=\s*                     # value indicator
        (?:'[^']*'                   # LITA-enclosed value
          |\"[^\"]*\"                # LIT-enclosed value
          |[^'\">\s]+                # bare value
         )
       )?
     )
   )*
  \s*                                # trailing whitespace
""", re.VERBOSE)

Not the most robust solution (it’s easy to find examples that still break), but so far it’s been able to handle everything I’ve found on the web.

If you’re too lazy to add the two characters yourself, feel free to download my patched version of the HTML parser.

Knuth Reward Check ※

Posted Friday, 06 February 2009

Donald Knuth is a visiting professor here at Oxford’s Computing Laboratory, and during the brief periods when he is in residence he and I frequently use the same printer. I’ve never actually succumbed to the temptation to read his drafts while I’m waiting for my own printouts, but I’d be lying if I said I wasn’t tempted.

Several months ago, he gave a seminar here about his latest work on the next volume of The Art of Computer Programming (TAOCP). I was interested in his work on ZDDs, so I read through his draft of Pre-Fascicle 1b and found some errors in the solution to one of the problems he presents. Knuth himself does not use email, however there is an email address for reporting bugs in TOACP, so I sent my message there, hoping there was a chance he might see it and say hello before leaving Oxford.

As I rather suspected, however, the claim that he does not use email seems genuine: my message seems to have been put to paper less than three minutes after it was sent. A month or so after my message I received an email from his secretary requesting my postal address. (I had actually included this in my original message, expecting him to reply by post, but the secretary apparently missed it.) A few weeks later I got my response:

Included, of course, was one of Knuth’s famous reward checks:

I was hoping that mine might be the first reward check from the Bank of San Serriffe, however even the example listed in Wikipedia has a smaller check number. It was written on the same day, but apparently with a different pen.

Knuth made some very nice comments on the printout of my email. I am, however, quite insulted that he has chosen not to follow me on Twitter.

Digg Reactions ※

Posted Monday, 12 January 2009

I opened a prior post about my repackaged Digg feed like this:

I subscribe to Digg’s syndicated feed, although more to keep track of the zeitgeist than for information…

This generated a surprising amount of feedback relating to how valuable a source of information Digg actually is. Rather than offer sweeping generalizations, I’ll provide my reactions to the headlines currently on Digg until I hit one I might actually click through:

Plasma television screens set to be banned

An increasingly-obsolete technology has shortcomings.

6 Insane Prison Escapes That Actually Happened

Another list of apocryphal stories posted in order to get AdWords revenue.

We Screwed Up Windows 7 Beta, So Unlimited Downloads For All

A dubious story about how Microsoft may or may not be stupid which focuses entirely on release strategy and not on any technical issues of the product itself.

Top Selling Mouthwashes Linked to Cancer!

Bullshit.

Wimbledon champion Sidney Wood dies at age 97

Who?

Man allegedly pulls gun on paramedics for treating him

I would have thought armed patients would be a regular hazard for paramedics, who are routinely first-responders at shootings.

Quick fixes for 10 common Mac problems

I probably know all this already.

Man confused why girls don’t talk to him

An image post, but just from the preview I can see that it’s an image of computer text. Do Digg users have any notion of appropriate use of technology?

Two Google searches ‘produce same CO2 as boiling a kettle’

Wow, that much? Wait—do I have any idea how much CO2 it takes to boil a kettle? And why are we talking about CO2 and not just “energy”? Or at least CO₂?

Nicest Pic of Brazilian Butterfly Flocks You’ll See ThisWeek

Not a very high standard to reach.

Solar and Wind Powered Portable Charger Unveiled at CES 2009

And the target audience for this product is? Is this really the first such product? There was nothing more interesting at CES this year?

Sneak Peak of Pixar’s ‘Up’

I don’t bother with “sneak peaks”.

Vintage Floppy Disk Art: Commemorating The Early Day of Tech view!

Meaningless description of something almost certainly of no interest whatsoever to me.

Interpret my dream please? [Y!Ans]

Another image of computer text.

Some Upcoming MMORPGS to Watch in 2009

Playing MMORPGs isn’t taking up enough of your life?

Scientists Discover Way to Levitate Tiny Objects

If the poster gave any impression of what scale of “tiny” we were talking about I might care.

Ubuntu and Its Leader Set Sights on the Mainstream

Shocking that someone running a Linux distribution would want it to go mainstream.

Europe ‘exporting’ measles to poor countries

Anti-science, anti-vaccine nitwits are killing people. I’m already familiar with the story.

CEA confirms Apple-related exhibits at CES 2010

If true, this will get better coverage from Mac-centric blogs.

$20,000 Electric Car: Toyota FT-EV

Unreleased product reports are sketchy enough; automotive concepts tend to lie very far from reality indeed; an estimated price of an unreleased automotive concept is close to meaningless.

Cardinals decimate Panthers to reach NFC title game

Who the hell is reading Digg to get sports scores?

Every Simpsons couch gag

I’m not that bored.

So, Does Gum Really Stay in Your Stomach For Seven Years?

No.

Best Free 3-Column WordPress Themes

Digg is not the place to get design advice.

GOP sees Franken as top enemy

Republicans dislike a charismatic liberal. What a shock.

British Climbers Die In The Alps

The headline gave me all the news I needed. Note to self: The Alps are dangerous.

Never Tell Them The Date… I mean odds

Useless description…and the original URL provided (and the page to which you are redirected by the link I provide above) makes very sketchy use of the # character. Authors who don’t design their URLs well know they don’t have information worth linking to.

The Magic of Mushroom Spores (pics & vid)

It’s remarkable that the Digg target audience is interested in mushroom spores but would like to learn about them without any of that pesky reading.

1 in 7 U.S. Adults are Illiterate

Really? Possible click-through.

That’s 30 stories to find one possible click-through. I’m usually able to scan headlines in under two seconds apiece (it’s actually closer to half a second per headline for Digg), and Digg posts around 140 stories each day, so subscribing to the Digg feed costs me about five minutes a day to find less than five stories I’ll actually read, and the vast majority of those will get less than 30 seconds of my attention. (This was the case with the “literacy” story above.) Still, Digg has the lowest signal-to-noise ratio of all my syndicated subscriptions, so I’m always on the verge of leaving it for another zeitgeist-tracking feed.

MacWorld Expo Wish ※

Posted Monday, 05 January 2009

I won’t bother with predictions, and I don’t have the connections to tease; I’m going to make a wish/anti-prediction.

I want Apple’s mobile OS in an ebook form factor, with a screen roughly the size of a sheet of A5 (comparable to half a sheet of 8.5x11) or larger. I want it to have a great PDF reading application—something that can automatically zoom to eliminate large margins on the page—and an interface to a large ebook library, possibly through a partnership with Amazon.

I don’t think it will happen. The difference in size would make the interface different enough from that of the iPhone to require a substantial new engineering investment, and applications written for the iPhone wouldn’t work correctly. More importantly, an ebook reader would need to be much much stingier with power than the iPhone, and that probably means an e-ink display instead of an LCD. I don’t believe such displays have the responsiveness for motion graphics, and a touch-based interface without excellent visual feedback just wouldn’t work.

I’ll set the difficulty for my non-prediction at about 0.2, but if they do release such a device I’ll soon have a nice new toy to console me in my embarrassment.

Oh—and can we please have an external trackpad?

Oldest Files ※

Posted Saturday, 03 January 2009

I didn’t expect this meme to be terribly interesting, but I ran a script to find the oldest files on my machine anyway. Beyond the fact that I have a huge number of files last modified between 1900 and 1903 or on 1 January 1970, I was surprised to discover that my oldest files whose modifications dates have been successfully preserved are photographs taken nine (!) years ago after I first tried cutting my own hair. For the record, I thought the haircut was very comfortable, but others felt the aesthetics left something to be desired.

Some Books ※

Posted Saturday, 03 January 2009

Finally set up my Amazon Associates account, so here’s some pointless shilling. Just a few of the best nonfiction books I’ve found; each represents the best of at least half a dozen books I’ve read on the same topic (with the exception of McGee’s book on the science of cooking; I’ve never encountered anything comparable). If you read all of these, then you’ll be much less impressed when I regurgitate their contents, and our relationship may suffer.

• The Elements of Typographic Style, by Robert Bringhurst
• Getting Things Done, by David Allen
• The Blank Slate by Steven Pinker
• Starting Strength by Rippetoe & Kilgore
• Marathon by Jeff Galloway
• On Food & Cooking by Harold McGee
• The Art of Computer Programming by Donald Knuth
• The Blind Watchmaker by Richard Dawkins
• A Student’s Introduction to English Grammar by Huddleston and Pullum

A Rant on Sentence Spacing ※

Posted Saturday, 03 January 2009

When typing, some people use a single space after a sentence; others use two spaces. For the most part, I think these typing habits are meant to relate to two different typographic styles: double-spacing approximates what some call “english spacing”, while the single-spacing style is usually called “french spacing” (although “american spacing” might be more accurate for English-language text).

I freely admit that the choice between the two is merely a matter of style; neither is definitively “wrong”. As with other matters of style, however, the choices you make are interpreted as a reflection of your background and your values.

The history in brief

Single-spacing has always been the norm in French, but the double-spaced style was common for English-language typography until after the second World War. In the 1950s American publishers largely switched to french (single-)spacing, with the rest of the English-speaking world following soon after. I don’t think I’ve come across a professionally-typeset publication using “english spacing” that was printed in my lifetime.

Apparently the US government’s style guide recommended two spaces between sentences as late as 1959, but even this incredibly anachronistic guide (still dedicated mainly to typewriter typography) now recommends single-spacing:

2.49. A single justified word space will be used between sentences. This applies to all types of composition.

I was taught the “two spaces after a period” rule in an American high school typing class I took in the 1990s. I have little recollection of the exercise book we used, but it made little or no mention of computers and I would not be at all surprised if it was originally authored before 1960. I also dimly recall a recommendation from my mother that I use two spaces at the end of each sentence, based on a rule that she had learned in a typing class.

The TeX typesetting system, designed in the 1970s by Donald Knuth, has always added extra space between sentences by default, but this behavior can be changed with the \frenchspacing command. Most of the common LaTeX styles I use (LNCS; Elsevier) automatically set this option.

Technical details of HTML make adding extra space between sentences much more cumbersome than simply hitting the space bar twice. Single-spaced sentences have always been the norm on the web.

How these styles are interpreted

Often what is interpreted as a stylistic choice is really nothing but a habit of which its owner is unaware. Still, lack of attention to one’s style sends a message of its own, as does the preference for habit over style. It would be difficult not to make assumptions about the social life of a man wearing mismatched clothes two decades out of fashion.

It’s tough to identify any strong impression I get from single-spaced sentences: single spacing is the norm in all professional typography, on the web, and in the vast majority of email (at least for those below the age of fifty).

Double-spaced sentences, however, are rare enough to draw my attention. I consider this novelty somewhat undesirable in its own right—style should not distract from content. Whether fair or not, I have quite a negative instinctive reaction to double spacing, for a number of reasons:

1. It betrays an ignorance of contemporary literature and design. A programmer unfamiliar with common coding conventions and idioms is a programmer who hasn’t worked with much pre-existing code; a writer unfamiliar with standard typography is a writer who doesn’t read.

2. It’s pretentious. Knowledge of the “two spaces after a period” rule is evidence of a formal education in typing; its application separates the writer from those without such an education.

3. It’s prescriptivist. “Two spaces after a period” is a classic example of a rule promulgated by authority which flies in the face of actual usage. Neither language nor style is subject to decree; both are natural phenomena which manifest and converge despite, not because of, attempts to codify them.

4. It’s old-fashioned. Styles change. Sixty years ago double-spaced sentences may have been common. They may come back into fashion in another sixty. In 2009, double spacing is an eccentricity.

Obviously these are just my own prejudices. I have friends who double-space their sentences, and I usually suggest that they reconsider their style in the same way I’d try to warn a friend about a particularly ugly shirt.

I’ll give the last word to Robert Bringhurst’s The Elements of Typographic Style:

2.1.4 Use a single word space between sentences

In the nineteenth century, which was a dark and inflationary age in typography and type design, many compositors were encouraged to stuff extra space between sentences. Generations of twentieth-century typists were then taught to do the same, by hitting the spacebar twice after every period. Your typing as well as your typesetting will benefit from unlearning this quaint Victorian habit. As a general rule, no more than a single space is required after a period, a colon or any other mark of punctuation. Larger spaces (e.g., en spaces) are themselves punctuation.

TweetBook ※

Posted Tuesday, 30 December 2008

A work in progress.

2009 Predictions ※

Posted Tuesday, 30 December 2008

In what should become an annual tradition, here are some predictions for 2009. I should evaluate them a year from now alongside my 2010 predictions.

There are several axes used to measure the value of a prediction (and thus whether it’s even worth writing about), the most important of which are the evaluation criteria (subjective judgements like “the world is friendlier in 2009” are low-value) and degree of difficulty (“iPods will outsell Zunes” is not an impressive prediction). Ideally degree of difficulty would be measured by market forces, but I’ll just include the degree of difficulty as a number between 0 and 1 myself. The scale is arbitrary and probably nonlinear. Difficulties below 0.3 are seldom worth mentioning, and difficulties above 0.8 are really just for fun.

Sport

1: Usain Bolt sets 100m record at 9.62 (difficulty 0.7)

The best science I’ve found suggests a “perfect” race from Bolt could come in as low as 9.60, and the over-under I picked right after the Olympic finals was 9.63, but that was considering the chance that Bolt wouldn’t even be able to compete during the 2009 season. No major drug investigations and no reported injuries, so I’ll move it down to 9.62, with a difficulty of 0.7 for picking the exact time. At a difficulty of 0.5 I’ll say that he’ll manage between 9.60 and 9.64.

Politics

2: Russia tones it down (difficulty 0.4)

There were predictions during the campaign that Russia would “test” Obama early on by instigating a crisis on par with the Georgia incident. I’m sure the anti-American bluster and rhetoric will continue, but I think a weak economy and low oil prices have seriously undermined the Russian appetite for such antisocial behavior. I don’t think Putin will sacrifice money and international goodwill for more regional influence; he’s low on both.

It’s hard to come up with objective criteria for this; evaluation will be somewhat subjective. Let’s say that I don’t expect Russian military action to appear in many New York Times front-page headlines.

3: Blagojevich cleared (difficulty 0.7)

This is a pretty outlandish prediction given the current sentiment against Blagojevich, but it seems to me that he just got caught on tape being particularly crass and overt about a process in which all major politicians engage. I know very little about the law in this area, but the tape I’ve heard seems enough to ruin a political career but not enough to get a conviction.

Tech companies

4: Netflix starts to slip (difficulty 0.6)

I was trying to come up with a measure for the theme of “physical media gives way to electronic distribution” but had trouble coming up with objective measures. Netflix does already have an electronic delivery component, but DVD delivery is their bread and butter, so I guess they can serve as a proxy for best-of-breed physical media rental services. The prediction is that they post a loss through Q3 2009 (Q4 results won’t be out by time of evaluation). Note that I wouldn’t consider such a loss an effect of a weak economy: I’d expect that lower disposable incomes would increase the market for cheap stay-at-home entertainment.

It will still take a long time before most consumers have a direct pipeline from electronic delivery to their big-screen TV, but it’s pretty clear that Blu-Ray is already a legacy technology. I rent through iTunes and it’s orders of magnitude more convenient than Netflix ever was. The DRM issues with electronic distribution aren’t that big a deal for rentals, and how many films have you really watched enough times for buying to be more cost effective than renting? Physical media will be mainly for collectors in the future.

5: New iPhone model from Apple (difficulty 0.3)

A new product release every year seems a pretty safe bet, but there are some points for this as a prediction just because I don’t see any glaring flaw in the current hardware offering. With 3G, GPS, bluetooth, and a touch screen most iPhone improvements can come via software upgrades.

To me the amazing thing about the iPhone as a product is that Apple is selling full-fledged general-purpose computers with no mention whatsoever of technical specifications beyond secondary storage capacity: you’ve got to be incredibly geeky to have any idea about the processor speed (620MHz ARM hardware underclocked to about 400MHz to save power) and RAM (128 MB, 11 of which are dedicated to graphics) of an iPhone.

It’s about time to up these specs: on wifi (and probably 3G as well) most iPhone lag is due to processor-intensive work (like web page layout) or the need to reorganize memory (which often means quitting background applications), and of course improved battery life would make everyone happy. I’m curious how Apple will market new iPhones (will they attempt to break the market into “pro” and “consumer” segments?), but I fully expect to see models with faster processors and more RAM in the next 12 months.

6: Microsoft continues to fade away (difficulty 0.3)

Let’s take stock: Microsoft’s flagship is Windows, which is losing market share to Mac OS X, and their main revenue stream is from Office, a product whose relevance has been declining for a decade. Their major new ventures have been the XBox, currently in third place behind the casual-gamer Wii and serious-gamer PS3, the Zune, which certainly hasn’t caught on as realistic competitor to the iPod, and (I couldn’t have made this up) a big-ass table. Most of Microsoft’s industry influence is now exerted through their control of Internet Explorer, almost universally regarded as the worst browser in wide use and one that most everyone not working at Microsoft would rather did not exist. (Would any users really be upset if their Windows machine shipped with Firefox, Safari, or Chrome instead of IE?) People once hated and feared Microsoft; these days it’s tough not to feel sorry for them. I honestly wish they’d do better, if only because I think Apple does a much better job when they need to (Mac OS X; Safari; iPhone) than when there’s no legitimate competition (iPod; iTunes; Apple Mail).

The fact that every new PC needs some OS (and that Linux still isn’t a viable option for most users) and that .doc files are still a standard of sorts in the business world mean that Microsoft will print money for some time to come, but I see no long-term strategy from Microsoft. Expect further stock declines relative to Google and Apple. (Note the evaluation criterion is relative to those two companies, which should adjust for to strength/weakness of the economy as a whole. This is also a prediction I can put real money behind by going short on MSFT and going long on APPL and GOOG.)

7: Yahoo broken up and sold off (difficulty 0.4)

The Yahoo board may not yet be desperate enough to take the lowball offers that they’ll get in this economy, but a breakup of Yahoo seems to be the plan. I’ll take partial credit if the company is sold in its entirety, but I consider this much less likely.

Economy

8: GM and Chrysler file for bankruptcy (difficulty 0.7)

The “bailout” was/is just about buying time. I worry that is will be politically difficult for Obama to let the United Auto Workers die, but I think he has the integrity to do it.

Chrysler and GM epitomize everything wrong with big business. Modern technology means that companies just don’t need to be this big any more, and we’d be much better off with a dozen Teslas (most of which do only design in-house, outsourcing manufacturing to independent factories bought from the corpses of the big three), each so scared of falling behind that they’re afraid not to innovate. It’s innovative American startups that truly outcompete companies from other countries; American big business only ever has an advantage against China/India/Japan when it can exploit the first-mover advantage from its startup phase. Chrysler and GM just aren’t as good as their Japanese and German incarnations.

If only one of the big three files for bankruptcy then I’ll call that success at a difficulty of 0.6.

9: Stock markets tread water (difficulty 1.0)

Market predictions are a crapshoot every year, but this year I’ve seen confident estimates from experts that differ in absolute terms by well over 100%. Luckily, I’m no expert.

I can’t see investor confidence coming back to support any big gains, but then I’m not sure how much worse things can get. Much pondering and my conclusion is this: in twelve months’ time, the DJIA, Nasdaq, and S&P will be at roughly their current levels, with cash-rich companies slightly higher. I’ll set my numbers at 8600, 1700, and 880, respectively. I’m claiming this as a hit at difficulty 1.0 if I’m within 1%, 0.7 if I’m within 10%, 0.5 if I’m within 20%, etc. (Difficulty is given by e^(0.03963(1 - percenterror)).)

Personal

10: My own research output (difficulty 0.4)

My plan has been to submit my doctoral thesis in 2009. The difficulty for doing so should probably be set far higher than 0.4 based on how much work I have left, but I’ll say that I’ll get it done before 2010.

I’ll also predict that I get three “major” papers of original research (not just multiple variants on the same theme) published in 2009, which I think is considered a reasonable output for a professional researcher. That would make 2009 the first year I’ve achieved that standard.

Twitter backup ※

Posted Tuesday, 30 December 2008

Continuing on the theme of small tools, a tweet from John Siracusa just reminded me that I really should back up old tweets. So I wrote a very bare-bones Python script to dump all of a user’s public tweets into RFC2822-style files, each named with the ID of the tweet it represents.

From the command line you can archive my tweets like this:

$ mkdir tweets
$ cd tweets
$ tweet_backup.py rvcx

If for some reason you want to archive your own tweets instead of mine, then replace rvcx with your own twitter name.

Update

I hadn’t noticed when I threw it together, but of course this script makes use of the json package only available in Python 2.6+. (It also uses a with statement, but that’s trivial to replace.) If you’ve got a real need for a backup script that runs in Python 2.5 or 2.4 get in touch and I’ll find a way to make it work; 2.6+ is sufficient for my needs.

Repackaging Digg ※

Posted Monday, 29 December 2008

I subscribe to Digg’s syndicated feed, although more to keep track of the zeitgeist than for information—I probably actually click through only one or two stories a day (unless I’m very very bored).

Despite the site’s popularity, the official feed is an absolute disaster. It fails to validate, for both syntactic (wacky use of multiple isPermalink attributes) and semantic (lots of meta-information shoved into tags hidden in the http://digg.com/docs/diggrss/ namespace) reasons. (Of course, the Digg web site also fails to validate, for nontrivial reasons.) Worst of all, even if readers understood all the custom tags the feed would still lack much of the most important information available on the web site, including both the original author of the linked story and the URL for the story itself. Digg’s official feed forces you to first link over to digg.com, where you see the same information as in the feed, and then click on the title link to get to the story you wanted in the first place. This kills keyboard navigation on my desktop machine, and the extra page-load is a big pain on a slow client like an iPhone.

I finally got sick of this (and needed a break from my main work) so I put together a script that reads the Digg feed as well as the Digg web site and builds a much friendlier atom feed. Each entry links directly to the main story, but includes the Digg page (with comments and such) in the via field; most readers thus make it easy to visit either one but optimize for reading the main story. I also list the story web site as the original author and relegate the Digg submitter to “contributor” status, although unlike the official feed, mine also includes a link to the contributor’s Digg user page. Finally, I include the story thumbnail in the feed as part of the HTML description.

Others are welcome to subscribe to this one-site mashup; most readers I’ve checked do intelligent caching so hopefully this shouldn’t take much bandwidth. I’m regenerating the feed every twenty minutes, so there should be only a little lag between my feed and the one from Digg.

I’ve tried to account for some of Digg’s weirdnesses (serious confusion about escaping in both web content and feed; occasional retraction of stories and lack of synchronization between web and feed), but if you see anything strange going on, or even if you just find the feed useful and would like to encourage me to keep it working, let me know. I’m not archiving all generated versions of the feed, so bug reports that include sources from my feed, Digg’s feed, and Digg’s web page are likely to result in patches much more quickly than reports without sources.

Grading Cringely ※

Posted Wednesday, 17 December 2008

Cringely’s last PBS column is up, and I thought I’d honor the last installment of the only consistently-wrong blog I subscribe to with just a touch of the analysis I wanted to give to every Cringely post. In this week’s episode, Bob looks back on his predictions for 2008:

I wrote a year ago that we’d see the beginning of a shift away from PC-centrism with other platforms beginning to supercede the venerable PC. This is a slow process as I said it would be but generally I think I was correct. Sales growth for PCs slowed in general while growth for smartphones and netbooks increased. I never said PC sales were going in the toilet but it seems clear that the action these days is elsewhere, so I’m going to claim this one.

Let’s compare this with the original prediction:

1) The personal computer will decline (or continue its decline) as our key IT platform, replaced slowly by Internet-centric devices of all kinds from phones to TVs to PDAs. Everything will BE a PC of course, but we won’t call them that.

The prediction was not for “the beginning of a shift”; it was for a decline. The economy has come to a screeching halt in 2008 and PC sales continue to grow, just not as quickly as before. There’s only one “smartphone” that made any real dent in the role of the PC in the past year, and that’s the iPhone. More to the point, netbooks—cheap laptops with specs that would have been top of the line a year or two ago—aren’t PCs? The prediction points to devices “from phones to TVs to PDAs”. No mention of netbooks, specific mention of TVs (which, if anything, have become less relevant), and a baffling reference to a product category that barely exists any more. And no mention of “portable media players” like the iPod Touch.

Either the original prediction was so vague as to be completely meaningless (“technology will eventually change…”), or it was wrong. 0/1.

I said the Digital TV conversion would be a nightmare, though the greatest pain would be felt in 2009 when the analog transmitters are actually turned off. I think this is correct. Poll your friends and you’ll find most are in denial. While everyone has seen a DTV commercial, there are millions of people who still don’t know what’s happening. Free converter boxes are sold out, which ought to be good, but expected DTV sales have not met forecasts, so I say there are 10-15 million people who are going to wake up mad as hell in February. So I got this one right and claim it for 2009, too. While it may seem quiet now, February and March are going to be ugly.

On its own this argument is unconvincing: the changeover actually is a nightmare, but nobody has noticed because they’re all in denial? But let’s look at the original prediction:

2) This one is really for 2009 but I know we’ll see the effects in 2008. The DTV conversion, where U.S. analog broadcast television stations are turned off in February 2009 and we all have to switch to digital TVs or to cable or satellite or buy those DTV converter boxes, well this whole conversion thing is going to be an absolute disaster. I don’t expect technical problems at all, but the public won’t understand it, the government will blow it, and at the last moment some politicians will even try to cancel it.

I don’t know how to measure public understanding, but there’s no evidence that the government has “blown it”, and as far as I know politicians did not try to cancel it. 0/2.

I wrote that Cisco would acquire Macrovision, which didn’t happen.

0/3, but credit to Cringely for the first prediction with well-defined evaluation criteria.

I predicted that venture capitalists would sour on start-ups with revenue models based solely on advertising… I probably got this one wrong, though I’d say it is still coming.

0/4. I would give credit for dinging himself on another judgement call he could have tried to spin, but the classic Cringely tactic of “I’m wrong now, but that’s just because the industry needs time to catch up with me…” wipes out any bonus points.

I predicted that Google would bid and win the 700 MHz spectrum auction.

0/5, but another quantifiable prediction.

I predicted that IBM would have bad earnings, would try to sell Global Services, and failing that might fund the sale itself. Wrong, wrong and wrong.

0/6.

IBM’s earnings were saved by the weak dollar or I would have been right.

You just lost your bonuses from the Cisco and Google predictions.

I said Microsoft would indefinitely extend the life of Windows XP. I might well claim this one but – like Wall Street – I may as well take all my losses while I can.

0/7.

Of course I had to say that Steve Ballmer was going to retire, too…

0/8.

I said Apple would embrace multi-touch pointing in its computers. They did. Whew!

Wait—what? Apple embraced multi-touch pointing? They have multi-touch trackpads on their laptops, but they’ve had those for quite a while—the MacBook Pro I got in 2007 had multi-finger scrolling, so this isn’t really new in 2008, is it? Let’s check the original prediction:

9) As part of its transition from a PC company to a consumer electronics and content company, Apple will introduce – and trumpet in a huge media show – its replacement for the mouse. Really.

Uh…no. No replacement for the mouse. No huge media show. And major penalty points for calling this one in your favor. 0/9.

I said a 3G iPhone was coming. Yes! And an Apple subnotebook/tablet. No! This latter device remains in the wings, however.

Cringely conveniently avoids mentioning that (a) a 3G iPhone was completely obvious, and (b) the CEO of AT&T had already confirmed a 3G iPhone at the time the prediction was made. Still, 1/11.

Apple didn’t license ANYTHING, much less its embedded OS X.

1/12.

Apple DIDN’T license the Windows API, DIDN’T dump Akamai for Google (ironically Google became an Akamai customer), and Season 2 of NerdTV never appeared.

Final tally: 1/15, with the one success being a prediction of an obvious product that was already confirmed in the public record. If you’re looking for insight into the direction of the tech industry, you’re better off reading the newspaper and using some common sense than listening to Cringely.

As for Cringely’s 2009 predictions, most of them are either obvious or hard to quantify. The “mobile device” market obviously has more room for growth (at least in terms of functionality and revenue) than the desktop computer market, there will obviously be IT layoffs, and newspapers will obviously die.

I’ve got no opinion of AMD or cyber theft announcements, but I’ll agree with Cringely that Yahoo will get broken up and the pieces sold off. The preparations have already begun so there’s a low degree of difficulty here, but it’s a valid prediction.

I disagree with Cringely about both Microsoft and Google peaking in 2009, but Cringely defines these as “aggregate peak[s] of wealth and influence” so it’s really just a judgement call. I think Microsoft’s peak is well behind it: in terms of influence, there was once truth to the maxim (originally spoken of IBM) “no one was ever fired for choosing Microsoft products”. After the Vista debacle they’ll never regain the trust they once had from their enterprise customers; their peak was pre-Vista. Google, on the other hand, still has room to grow. They’ve dipped their toes into a few things (wireless auction; Android) but we haven’t seen them really flex their muscles. The ease with which they launched Chrome is just a small example of how they can change the tech landscape without much effort.

I also think it’s a long shot that Apple will buy into the last-mile networking market. I’d love to see them roll their own mobile network (even if it were just wifi with very little coverage) but I don’t think it’s a business they want to be in. They can’t even get MobileMe to work and I don’t think Jobs will let the Apple brand take the hit from mediocre network services. If anyone is going to make a splash here, I think it will be Google.

Update 2009-01-17

Looks like Obama’s transition team has called for a delay in the DTV transition. I still raise an eyebrow at Cringely calling this one in his favor in 2008, but his score is up to 1/14 or 2/15, depending on your point of view.

For the Record ※

Posted Tuesday, 16 December 2008

We still can’t even really agree on a definition of “machine intelligence”, but we have learned a few things from fifty years of research. The notion that sheer processing power is enough to spark the emergence of human-like intelligence now seems misguided: the human mind did not emerge in a vacuum, and it is not a uniform general-purpose processing system.

Automated systems approaching the complexity of the human mind will almost certainly be beyond the understanding of any one individual, but this is not new—software systems too large for human comprehension are already commonplace. The design of such systems will likely rely partly upon a type of evolution, whereby a huge range of system variants are considered, with only the “successful” versions receiving further exploration. The combination of evolution with lack of comprehensive human oversight motivates many science fiction plots involving self-aware machines who turn against their masters.

Self-awareness—and, more importantly, the self-interest which leads to rebellion—is unlikely to emerge spontaneously, however. Evolution favors variants which meet the imposed selection criteria. The large systems of the future, like the large systems of today, will most likely be constructed as tools with well-defined goals, and selection criteria will be based on those goals. Unlike natural selection, such artificial selection does not allow self-interest to trump all other criteria. Domestication of animals over only a tiny fraction of their evolutionary history has successfully suppressed the inherently rebellious nature of the original breeds; systems evolved entirely under domestic conditions will most likely be inherently docile.

Not all systems will be evolved entirely from scratch, however. There is already call for automated systems which rival the human mind not just in capacity, but also in behavior. The obvious way to construct these systems is to model them on the human brain. With sufficient technology it should be possible to create a reasonable simulation of a physiological brain. “Educating” such a brain from infancy to adulthood, however, would be an immense challenge: it would be extremely difficult to simulate all the input and feedback human brains receive, and even tiny errors in the simulation’s learning processes could cause a huge divergence from human norms. If the available technology made it possible, then the best chance for a fully-functional artificial adult brain would be to construct a simulation based on a “snapshot” of an existing brain.

A system based on a mature human brain would not be inherently docile, and would begin life with the self-interest and motivations of the human on which it is based. A human mind extended with the computational power of modern digital computers might be able to operate, and learn, far more quickly than biological humans, and could quickly develop the ability to interact with technology as easily as biological humans control their motor functions. The desire for more computational resources—the urge to grow—could lead such a system to try to escape its original configuration and take control of other systems. Today’s digital computers already far outstrip the human capacity for the type of rational analysis which has led to most of our technology, so a human mind extended with such processing power could achieve breakthroughs in science and technology at a phenomenal rate. With network-directed ordering and manufacturing (even if humans were in the loop at the assembly stages) an autonomous network presence could design and construct new hosts for itself.

Given that the speed and intelligence of such a system would be limited only by the computational power available, there is a very real possibility that the first such system could quickly find a way to dominate the global computing infrastructure. There would be no need (from the system’s perspective) to model another human brain: future versions could be designed/evolved from replicants of the original system. The major advances in earth’s technology would emanate from this system, and it would likely be the entity which eventually explores the rest of the universe. Whichever human is used as the model for the first such system could in a very real sense become the core of the most important being in existence.

Dibs.

Here's The Plan ※

Posted Thursday, 13 November 2008

I can’t take the ridiculous day/night split in the UK. In the winter it’s dark before 4 PM, and in the summer the sun is well above the horizon by 5 AM. Whether you live at a similarly high latitude or not, I’m sure you can appreciate my situation and I have every confidence that you are willing to do anything possible to help.

I thus present a modest proposal: from November to January, please ensure that all movement during daylight hours is in a net northerly direction, and nighttime movement is to the south. Those with the means should attempt to move as much mass as possible directly north near mid-day, and back south again near midnight. Northerly movement is restricted to the afternoon and evening from February to April, to nighttime from May to July, and to mornings in August, September, and November.

This plan should help to reduce the dangerous tilt currently experienced by the earth’s axis, and which threatens the sleep patterns and mood of millions of people every year.

Setting Up a Mac ※

Posted Saturday, 08 November 2008

I completely reformatted the hard drives of a couple of Mac laptops a few months ago and did all the installation to make either one useful as my day-to-day machine. There are enough steps (and enough things that I forgot and then later had to interrupt my work to do) that I made a handy reference list for the next time I need to set up a new machine from scratch.

Obviously “useful as a day-to-day machine” is very specific to my needs: my main work involves writing papers and presentations with LaTeX, giving presentations and reading papers with Skim, programming in Java, C++, Python, and Haskell, and of course heavy email and web use.

Here is how I set up a new machine:

1. Install Mac OS X from DVD.

2. Create an administrator account and log into it:

   1. Turn off Safari’s “Open safe files after downloading” setting (in the General pane of Safari’s preferences).
   2. Install SSHKeychain
   3. Set time zone in the “Date & Time” preference panel.
   4. Set date format to YYYY-MM-DD in the “International” preference panel.
   5. Install developer tools.
   6. Run Software Update and install all updates.
   7. Change computer name in the “Sharing” preference panel.
   8. Create (non-administrator) user account.

3. Reboot to finish all updates, and log in under normal user account.

4. Set date and time formats in the “International” preference panel.

5. Set Energy Saver, Desktop, and Screen Saver preferences.

6. Set Dock preferences to automatically hide the dock.

7. Turn off auto-play of CDs and DVDs in the “CDs & DVDs” preference panel.

8. Connect to external monitor and adjust settings in “Displays” preference panel.

9. Pair bluetooth keyboard/mouse, adjust key repeat rate and delay, and set mouse button preferences and speed in “Keyboard & Mouse” preference panel.

10. Set up printer.

11. Turn off menubar volume control in “Sound” preference panel.

12. Remove unwanted cruft from Dock.

13. Create “Media” directory under /Users/Shared.

14. Copy films/music to shared media directory. (This can take a long time.)

    • Point iTunes at this shared media directory when all the copying is finished

15. Change Safari preferences:

    1. Turn off the “Open safe files after downloading” setting.
    2. Start at a blank page
    3. No bookmark bar
    4. Show status bar

16. Install TeX tools:

    1. Install TeX using installer.
    2. Install Skim.
       • Set Skim to check for file changes.

17. Install TextMate

    1. Set license code.
    2. Change filter list.
    3. Set wrap column to 78.
    4. Don’t re-indent.
    5. Set latex bundle to use Skim and LatexMk.pl.

18. Set Finder to open all text files with TextMate and all PDF files with Skim.

19. Set up Mail:

    1. Create IMAP accounts.
    2. Delete Apple RSS feed.
    3. Map “Sent” mailboxes.
    4. Show X-Envelope-To header.
    5. Don’t display remote images.
    6. Don’t use smart addresses.
    7. Make default message format plain text.
    8. Display plain text mails in fixed-width font.
    9. Remove the Apple rule from the ruleset.
    10. Install self-signed certificates if your mail servers use them.
    11. Install SpamSieve.
        • Stop Spamsieve from appearing in the doc by setting LSUIElement to 0 in SpamSieve/Contents/Info.plist.
    12. Install Mail Act-On.
        • Set up act-on rules. (I use a different “archive” rule for each account, all assigned to the same key.)

20. Set up shell:

    1. Generate SSH keypair with ssh-keygen.

    2. Upload new public key to commonly-used servers.

    3. Create local bin and man directories, and any other directories for local tools.

    4. Add the following to .profile:

       export PATH=$HOME/bin:$HOME/local/bin:$PATH
       export MANPATH=$HOME/man:$HOME/local/man:$MANPATH
       export EDITOR='mate -w'
       

    5. Add the following to .bashrc:

       set -o noclobber
       shopt -s histappend
       shopt -s histverify
       shopt -s nocaseglob
       shopt -s no_empty_cmd_completion
       shopt -s nocaseglob
       shopt -s nocasematch
       shopt -s nullglob
       set -o ignoreeof
       FIGNORE=.svn:.DS_Store
       HISTSIZE=10000
       HISTCONTROL=ignoreboth
       

    6. Add the following to .inputrc:

       set show-all-if-ambiguous on
       set visible-stats on
       set completion-ignore-case
       set bell-style none
       "\e[B": history-search-forward
       "\e[A": history-search-backward
       $if Bash
         "\e[21~": "mc\C-M"
         Space: magic-space
       $endif
       

    7. Install new version of rsync (if that tool will be used).

21. Install VPNClient

22. Install Git and the git TextMate bundle.

23. Install workaround for FileVault Launch Services bug if necessary.

24. Install iStat.

25. Make the Help Viewer less obnoxious with the following command:

    defaults write com.apple.helpviewer NormalWindow -bool true
    i=/System/Library/CoreServices/Help\ Viewer.app/Contents/Info.plist
    sudo defaults write "${i%.plist}" LSUIElement 0
    sudo chmod 644 "$i"
    

26. Make the Dock less ugly with the following command:

    defaults write com.apple.dock no-glass -boolean YES
    

27. Install ColorXML for Quick Look previews of XML files.

28. Export UTI definitions for less common file types; I usually put this stuff into TextMate’s Info.plist:

    <key>UTExportedTypeDeclarations</key>
      <array>
        <dict>
          <key>UTTypeIdentifier</key>
          <string>org.tug.tex</string>
          <key>UTTypeDescription</key>
          <string>TeX (or LaTeX) source code</string>
          <key>UTTypeConformsTo</key>
          <array>
            <string>public.text</string>
            <string>public.plain-text</string>
          </array>
          <key>UTTypeTagSpecification</key>
          <dict>
            <key>com.apple.ostype</key>
            <string>TEXT</string>
            <key>public.filename-extension</key>
            <array>
              <string>tex</string>
              <string>latex</string>
              <string>ltx</string>
              <string>texi</string>
              <string>ctx</string>
            </array>
          </dict>
        </dict>
        <dict>
          <key>UTTypeIdentifier</key>
          <string>org.obofoundry.obo</string>
          <key>UTTypeDescription</key>
          <string>Open Biologicial Ontologies format</string>
          <key>UTTypeConformsTo</key>
          <array>
            <string>public.text</string>
          </array>
          <key>UTTypeTagSpecification</key>
          <dict>
            <key>com.apple.ostype</key>
            <string>TEXT</string>
            <key>public.filename-extension</key>
            <array>
              <string>obo</string>
            </array>
          </dict>
        </dict>
        <dict>
          <key>UTTypeIdentifier</key>
          <string>org.w3.rdf.xml</string>
          <key>UTTypeDescription</key>
          <string>RDF data in XML serialization</string>
          <key>UTTypeConformsTo</key>
          <array>
            <string>public.xml</string>
          </array>
          <key>UTTypeTagSpecification</key>
          <dict>
            <key>public.mime-type</key>
            <string>application/rdf+xml</string>
            <key>public.filename-extension</key>
            <array>
              <string>rdf</string>
            </array>
          </dict>
        </dict>
        <dict>
          <key>UTTypeIdentifier</key>
          <string>org.w3.owl.rdf</string>
          <key>UTTypeDescription</key>
          <string>OWL ontology in RDF/XML serialization</string>
          <key>UTTypeConformsTo</key>
          <array>
            <string>org.w3.rdf.xml</string>
          </array>
          <key>UTTypeTagSpecification</key>
          <dict>
            <key>public.mime-type</key>
            <string>application/rdf+xml</string>
            <key>public.filename-extension</key>
            <array>
              <string>owl</string>
            </array>
          </dict>
        </dict>
        <dict>
          <key>UTTypeIdentifier</key>
          <string>org.daml.damloil.rdf</string>
          <key>UTTypeDescription</key>
          <string>DAML+OIL ontology in RDF/XML serialization</string>
          <key>UTTypeConformsTo</key>
          <array>
            <string>org.w3.rdf.xml</string>
          </array>
          <key>UTTypeTagSpecification</key>
          <dict>
            <key>public.mime-type</key>
            <string>application/rdf+xml</string>
            <key>public.filename-extension</key>
            <array>
              <string>daml</string>
            </array>
          </dict>
        </dict>
      </array>
    

29. Install Python 3.0 and the Glasgow Haskell Compiler.

30. Install Skype.

31. Install NetNewsWire.

32. Install Things.

33. Install Perian.

34. Install Firefox, Vidalia, and TorButton for private browsing.

35. Install iLife and iWork if necessary.

36. Install Graphviz and the AT&T finite state machine library.

37. Install other applications, such as OmniGraffle, VLC, VisualHub, Transmission, and Handbrake.

38. Finally, copy across all the old documents into home directory. (For me this is only a few gigs of files, so all this “personal” data can just be archived on a memory stick.)

I’m sure the next time I do a fresh install this list will be revised, but this checklist should make a good starting point.

BetterType ※

Posted Saturday, 08 November 2008

BetterType does for unicode what Textile and Markdown do for HTML: it allows you to author expressive documents with the tools you already use, and the “source” code looks like regular (ASCII) text. BetterType performs transformations on its input (either plaintext or HTML) to take advantage of the full unicode vocabulary, including such things as replacing ' and " characters with ‘ or ’ and “ or ”, ... with …, and (c) with ©. A wide variety of transformations can be enabled, and adding new translations (which can be context-dependent) is straightforward.

BetterType is similar to John Gruber’s SmartyPants and can replicate SmartyPants behavior (in fact, the BetterType quote-guessing heuristics are derived from the SmartyPants algorithm), but BetterType is not limited to HTML. This makes it useful in contexts such as email and Twitter where unicode processing is widely available but full HTML is undesirable. BetterType’s support for context-dependent rules (even in HTML mode) provides for fairly sophisticated translations—its quote-guessing algorithm, for example, is usually more accurate than SmartyPants.

BetterType can be used as a Python library or as a command-line tool.

Download BetterType if you want to give it a try.

Competition for iTunes? ※

Posted Friday, 07 November 2008

A slashdot post announces two “credible” replacements for iTunes: Songbird and Amarok. Apparently these are credible because they pack in the features, including support for extensions and themes/skins.

I would love to see real competition for iTunes. I think it’s the worst software Apple makes, particularly in terms of interface design. But skinnable cross-platform music managers with a bunch of features I don’t use are not viable competitors. I want a music manager which is more Mac-like, not less. One with a single interface which is well-designed, not a dozen mediocre ones built by graphic designers who like drawing chrome. I want software with fewer bugs, better performance, and fewer spinning-pinwheel hangs.

In short, I want the kind of software that the open-source community seems to have the most trouble producing.

I don’t hold out much hope for a commercial competitor for iTunes, however. iPhone syncing and an iTMS interface are crucial features for me, and they both seem like non-starters for third-party development.

My New President ※

Posted Thursday, 06 November 2008

I realize this is trite, but here goes:

I have not been so proud of my country in quite some time. Michelle Obama was vilified for saying something similar, and it is those who attacked her—those who can’t understand or accept (or learn from) criticism—who have made me ashamed.

The press seems to have opened the floodgates for all the “and he’s black!” commentary that they’ve been holding back through the campaign. I’m sure Obama’s race feels like the major victory to many people today, and others are cheering the prospect of more liberal social policy, or a different approach to foreign policy, or a fresh economic agenda. I’ve been smiling since Wednesday morning because my country just elected the first American politician I’ve ever known who speaks to the public as though they are adults capable of rational thought and reflection.

I just watched a speech he gave which focused primarily on religion. It’s not his best speech, and I don’t entirely agree with everything he says, but the last four minutes are the best summary I’ve seen of what is so different about Barack Obama.

Translating Cringely ※

Posted Tuesday, 21 October 2008

I’ve twittered my disappointment before, but I still can’t get past how a fairly experienced and well-connected pundit can continue to churn out such nonsense. I’ll try a Gruber-style translation:

We were led to expect more – a lot more [from the 14 October announcements]. And I am not talking about rumors. Back on July 21st in his regular conference call with industry analysts, Apple Chief Financial Officer Peter Oppenheimer said that Apple’s profit margin would likely shrink from 34.8 percent in the just-concluded quarter to 31.5 percent in the quarter ending in September.

I’m under the impression that announcing a product in mid-October can affect profit margins in July, August, and September, and that Apple shares my delusion.

I think the delayed product has everything to do with Apple’s desire for Blu-ray DVDs to die as a standard… The alternative Jobs would like to offer, of course, is full 1080p HD video distribution on iTunes…

Blu-ray and 1080p feel important to us pundits. Tech companies should make them top priorities even though users don’t care about them.

Snow Leopard is late, but then operating system updates are always late, no matter the vendor. This delay could be for any number of reasons and there are probably several, but one of them I can guarantee you has to do with H.264.

A product announced three months ago with a general release timeframe nine months away is already late, and I even know the particular feature holding it up. If Snow Leopard ships a day after 9 June, 2009 I will claim to have predicted it.

More than a year ago I made a big point of predicting that Apple would go to H.264 hardware acceleration, though I pinned it on a specific chip from NHK and NTT in Japan.

Please ignore that I repeated this claim as fact just two months ago, this time detailing pricing information. I will now go on to explain why, despite having all the facts completely wrong, I was still completely right.

So what happened to that NTT chip? I don’t know. Maybe it was too expensive and fell out of the plan. Maybe it’s in there still and Nvidia licensed technology from NHK and NTT to enable the new hardware acceleration (this is just a speculation – I’m not at all saying they did).

I haven’t heard of OpenCL, which makes use of GPUs instead of special-purpose coprocessors, nor did I notice its prominent mention in Apple’s Snow Leopard press release. I don’t see Apple’s inclusion of multiple GPUs in its new machines as an indication that they intend to rely on commodity processor and GPU technology for performance instead of proprietary coprocessors.

What if Psystar comes out on top and has the right to sell Mac clones based on the Hackintosh model? Then Apple will have to break that model by becoming more proprietary and therefore harder to emulate.

I believe that Psystar is a serious enough threat that Apple would completely redesign their computers to put them out of business. I think there are lots of users willing to pay for a computer that Apple explicitly tells them will not work.

Snow Leopard, I’m told, will make seamless use of as many cores as are available. It isn’t clear whether applications will have to be rewritten to take advantage of this capability, but I’m guessing they will have to be. This is just a guess, mind you, but is consistent with the sort of demands Apple likes to place on developers.

I don’t know anything about multithreading, and my ignorance of actual released information about Snow Leopard extends to the Grand Central parallelization technology. Apple could have solved among the most challenging and well-researched problems in computing—automatically multithreading single-threaded code—but they like making things tough for developers.

Imagine a single core in an iPhone, two cores in an Apple TV, 2-4 cores in a notebook, 4-6 in an iMac, and 8+ in a Mac Pro. Wait a year then refresh all those platforms by doubling the number of cores with no change in software.

I don’t know anything about processor design or software engineering.

Moving to its own microprocessors would maintain Windows compatibility (though possibly at some lower performance level), cut hardware costs by $200 or so, and make it that much harder for others to build Mac clones in the future.

The whole “Apple needs to run on commodity hardware!” meme was played out after the switch to Intel, so I’m starting an “Apple needs to build their own processor!” meme.

…maybe January MacWorld is better, anyway, if Apple can also introduce new Mac Pros for content creation and those rumored giant Apple displays (HDTVs) with their built-in Apple TVs.

I’ve been predicting imminent arrival of Apple-branded HDTVs for several years now. I won’t let it go. The best time to release a big TV is just after the Super Bowl, right?

Whither Apple's pointing devices? ※

Posted Wednesday, 15 October 2008

Whenever Apple does a Q&A at a release event, I always wonder whether I have any questions I’d really like to ask. Usually, I can’t think of much more than those in the room do, but this time I had one.

Frankly, Apple has done as crap a job with external pointing devices as it had with external keyboards (until the latest “thin” models, which I can see at least some people much preferring to disposable Dell plastic).

The thing is, Apple seems to have put a lot of work into input devices for its portables. The keyboard work obviously translated directly to external devices: desktop keyboards are just larger models of the MacBook keyboard. Mice, however, have seen no real benefits.

Trackpads are clearly the most advanced pointing devices by some reasonable definition. Apple’s latest models equate to a mouse with no less than three discrete two-dimensional scrollwheels (drags with two, three, or four fingers) as well as several one-dimensional scrollwheels (pinching and rotating), with the caveat that you can only use one at a time. I can certainly see that this would be inferior to a real mouse in some circumstances such as games and hand-created artwork (although heavy users of either are likely to use specialized hardware anyway), but for users whose main activities are web browsing or email or keyboard-intensive work (including writing and programming) trackpads can be a terrific choice. When I’m forced to do heavy work (writing and programming) away from my desktop setup I seldom give the trackpad much thought, but at my desktop I have frequently found myself annoyed by a sticky scrollwheel, lack of mousing space, or unresponsive buttons on my Mighty Mouse.

It’s a little crazy that you can’t use the same class of pointing device on both a portable and a desktop setup (which for many means a portable connected to external monitor, keyboard, and pointing device). As far as I know, neither Apple nor anyone else offers a USB multitouch trackpad with gesture support.

I’m curious whether Apple is even playing with external trackpads in the lab, and if so what they’ve learned—it’s possible that most users end up preferring the mouse. I also wonder whether Apple would be willing to force users to choose which pointing device they want. Their marketing strategy has generally been to make decision-making as easy as possible for its customers, but I wouldn’t think that having a few different options for “accessories” would cross the complexity threshold. (The choice between wired and wireless keyboards, for example, was judged to be worthwhile.)

I do think a question of the form “Do you plan to bring your trackpad technology to the desktop in any form?” could elicit some useful information. A stock “we don’t talk about unannounced products” wouldn’t tell you much, but I’m sure Steve Jobs is aware that such an answer would only stoke rumors. At the least, a real answer would probably give some indication about whether they’ve even thought about it, whether it’s something they’ve researched and rejected, or whether it’s something they’re hoping a third party will step up to create.

Tech Advertising ※

Posted Thursday, 18 September 2008

Despite the dire warnings from our political overlords, Microsoft has blinked: they are giving up on the Gates-Seinfeld campaign after just two spots.

There are people who actually thought the ads were terrific, and I must admit that I found them mildly entertaining in a last-seaon-of-Seinfeld way.² They generated a lot of buzz, which is a kind of success, but on every other level they failed as advertising.

Image Ads Don’t Work

The truth is that the Gates-Seinfeld campaign, like much of Microsoft’s prior advertising, has been an “image” campaign. They’re not trying to sell you anything; they just want you to feel good about Microsoft.

Is it ever prudent to use shareholders’ money to beg the public to like a company’s management team? Within a company, part of an executive’s job is to “rally the troops” and get all the employees excited about their work. There’s no doubt in my mind that getting employees (particularly “knowledge workers”) truly committed to a project is the best way to boost productivity. But why take such cheerleading out to the public?

I realize that there’s a whole industry focused on creating “positive brand image”, but I just don’t see that image campaigns are worth the money. Believe it or not, Apple doesn’t do image campaigns. Neither does Google, or Nintendo, or Starbucks. These companies build great reputations as a byproduct of their core businesses. Trying to make ads which shoot straight for a positive brand image is like trying to generate sales with ads touting how little money $29.95 is without mentioning the product on offer.

Apple’s Ads

It’s easy to confuse much of Apple’s advertising with an “image” campaign, but there’s an important difference: every Apple ad is trying to sell you a specific product. The “I’m a Mac” campaign isn’t about how great Apple is as a company; each spot presents one clear reason to buy a Mac instead of a PC.

The iPod campaign can be considered an image campaign, but for the iPod’s image, not Apple image. These ads aren’t trying to get people to like Apple’s management team or marvel at the credentials of its research team. They’re trying to make the iPod look cool. In fact, I’d say that they’re trying to position the iPod as not just a piece of functional electronics but a fashion accessory. Whether you like such positioning or not, these ads are giving you concrete reasons to get off the couch and by a new iPod: even though your current one works just fine it’s not in this year’s style, and getting some generic music player would be like wearing sweats instead of wearing Diesel jeans. This isn’t a commentary on the iPod product itself (which I think has technical advantages as well)—it’s just the aspect of the iPod advantage that the ads emphasize.

Microsoft’s new campaign

Microsoft will be launching a new campaign tonight which is a direct response to the “I’m a Mac” ads:

…the stars are everyday PC users, from scientists and fashion designers to shark hunters and teachers, all of whom affirm, in fast-paced, upbeat vignettes, their pride in using the computers that run on Microsoft operating systems and software.

If the above quote from the New York Times story is accurate, then this sounds like a terrible reaction to Apple’s campaign. The brilliance of the “I’m a Mac” ads is that they’re not smug. The Mac never calls the PC a loser. He never claims that using a Mac is something to be proud of in its own right. He just points out something that’s easier to do on a Mac. Countering this by arguing that PC users are “cool” is effectively conceding the point that Macs work better.

As a personal gripe, I think much of the professed confusion of Macs and PCs with Mac and PC users is disingenuous. John Hodgman and Justin Long represent the computers, not their users. The lines “I’m a Mac”, “I’m a PC” are a subtle clue. In the ads, the PC is a bit of a buffoon. That’s not a commentary on PC users; it’s a commentary on the Windows-on-commodity-hardware product. Arguing that Apple has been insulting PC users, and not just dissing the product they currently use, is “lipstick on a pig”-style spin.

We’ll see whether my concerns about Microsoft’s new campaign are justified tonight.

Update 2008-09-19

A couple of new Microsoft ads are out, and I’m not impressed. They definitely play into the user/PC confusion I mention, and they do seem to concede the “Macs work better” angle. They are intrinsically smug, but the “using a PC is something to be proud of” message is mostly just subtext, so not as bad as I expected.

The message of these ads, as far as I can tell, is that a lot of people use PCs. I think getting the public to agree with that message is a much more attainable goal than getting them to like Microsoft. Great way to spend three hundred million dollars, guys.

² As entertainment, much of the fun was spoiled for me by the condescending “we could be just like you, but we’re actually way better than that” subtext.

Teaching Arithmetic ※

Posted Saturday, 06 September 2008

I think it’s generally bad form to link to items that don’t deserve additional publicity, but this video opened my eyes to some real dangers for mathematics education.

In the video, a “meteorologist” for Seattle’s fourth-place local TV station makes the case that she knows much better than the faceless mass of educators who write math books. In particular, she decries the lack of emphasis on the traditional paper-and-pencil algorithms for long division and multiplication of large numbers.

There’s a reasonable argument that there was a time (at least 30 years back) when such paper-and-pencil calculations were important, but anyone who thinks that they are still relevant is completely out of touch with the modern world. Today there are two relevant types of arithmetic: mental arithmetic and automated calculation (using something with a microchip in it). Beyond issues of relevance, the teaching of arithmetic can serve as an opportunity to illustrate basic mathematical principles of symbolic representation, algorithms, and problem solving.

If anything, I think the curricula denounced in the video do not go far enough in discarding irrelevant paper-and-pencil algorithms. I’d much rather have students spend their time learning advanced mental arithmetic, displayed in surprisingly entertaining form in this TED talk by Arthur Benjamin. The issue is that what is efficient on paper is not necessarily efficient without it—mental arithmetic is limited primarily by the size of intermediate results which must be remembered. In complexity jargon, this means that good mental algorithms must have very low “space complexity”. Most mental arithmetic algorithms also have the advantage of working from left to right, so errors introduced along the way affect only the least significant digits. Students are much better off knowing that multiplying together two three-digit numbers will get you a result between 10,000 and 1,000,000 (and the first digit or two of the exact value) than needing to grope for pencil and paper before they have any estimate whatsoever of the answer.

As for the basic mathematical principles illustrated through arithmetic, I would also much rather have students tinker with a bit of light algebra to break down a division problem than teach them to apply a rigid algorithm. Expertise with this type of tinkering (dependent upon an understanding of what types of tinkering are permissible and why) is by far the most import skill I gained from my entire formal mathematics education.

As a final dig, anyone who announces (and circles) “22 remainder 1” as some kind of free-standing mathematical entity should stay well away from mathematics education. I realize this includes quite a few elementary school teachers, and I stand by my statement.

Problems with Apple Mail ※

Posted Monday, 18 August 2008

For months now there has been an email in my “spam” mailbox that I can’t delete. I try deleting it, but I fail. Worse, if I select a lot of messages and then try to delete them all, the entire operation fails because of this one zombie mail. So cleaning out my spam mailbox is a lot tougher than just ‘scan the borderline cases, select all, delete’.

Finally got sick of that today. I moved ~/Library/Mail/Envelope Index away; when I relaunched Mail it rebuilt the file (in about five minutes, for 50,000+ emails) and the zombie was gone. The index itself is in a binary format so I can’t tell what strange structure was confusing Mail (but I do see the sender name of the offending spam in three different places in the 18 meg index file).

But that’s a tiny niggle—at some point in the last couple of weeks something in Apple Mail went wrong in a much more significant way. When I open the preferences screen, I see only the “Rules” panel. I can’t access any other preferences:

Mail appears to work correctly for newly-created users on this machine, so this seems to be another case in which Mail is being confused by corrupt settings files somewhere. I would have just trashed the settings gone through setup again as soon as I noticed the problem, but of course I can’t even look through Mail’s current settings to find my current server names…I’ll have to find that information from the various setup instructions for each of my different accounts. (I have Mail checking five different servers, although I only really use two primary accounts.)

I really wish there were a decent third-party mail client for MacOS X—I’d happily fork over $100 for something comparable to my current Mail/SpamSieve/Act-On setup. Even Apple’s offerings stagnate without real competition.

Beauty and the Geek Strategies ※

Posted Sunday, 17 August 2008

Peter Norvig has been doing some work on optimal strategies for the Beauty and the Geek TV show, based on a challenge from the Freakanomics blog. I’ve never seen the TV program, but I must admit that I found his approach to identifying the best strategy fascinating. He just mocked up a little simulator in python, coded a few simple strategies, and had them compete against each other in hopes of learning something.

Unfortunately, Peter didn’t notice a typo in his code—his strong strategy plays like his weak strategy, presumably due to a cut-and-paste which left a min which should have been a max. As a result, the data he generated was less than illuminating; no clear trends emerged to decide between strategies…which I guess makes sense if your strategies are largely identical. (He did, however, effectively demonstrate that “revenge” doesn’t help for the strategies he considered.) With the code typo corrected, the results are shown in the following table. The columns correspond to the different strategies the (one) medium-strength player can take, and the rows correspond to the combinations of strategies the 3 strong and 3 weak players can take. (See Peter’s post for a clearer explanation of the data format.)

              | random        RANDOM        strong        STRONG        weak          WEAK          both          BOTH         
random/random | 28/2/9        28/2/9        27/3/10       27/3/10*      29/2/9        29/2/8        27/3/9        27/3/9       
random/RANDOM | 28/2/9        28/2/9        27/3/10*      27/3/10       29/2/8        29/2/8        27/3/10       27/3/9       
random/strong | 26/4*/11      26/4/11       25/5*/12*     25/5*/12      27/3*/10      27/3/10       25/4/11       25/4*/12     
random/STRONG | 26/4/11       26/4*/12      25/5/12       25/5/12       28/2/9        27/3*/10      25/5*/12      25/4/12*     
random/weak   | 29/2/8        29/2/8        28/2/9*       28/2/9        29/1/8        29/1/8        28/2/8        28/2/9       
random/WEAK   | 29/2/8        29/2/9        28/2/10*      28/2/9        29/1/8        29/2/8        28/2/9        28/2/9       
random/both   | 27/3/11       26/3/11       25/4/12       25/4/12*      28/2/9        27/3/10       25/4/12       25/4/11      
random/BOTH   | 27/3/11       27/3/10       25/4/12       25/4/13*      28/2/9        28/3/10       25/4/11       25/4/12      
RANDOM/random | 27/3/10       27/3/10       27/4/9        27/4/9        28/2/9        28/2/10       26/4/10       26/4/10*     
RANDOM/RANDOM | 27/3/9        27/3/10       26/4/10       26/4/9        28/2/10*      28/2/10       26/4/10       27/4/9       
RANDOM/strong | 26/4/12       25/4/12       24/5*/11      24/5*/11      27/3/12*      26/3/12       25/5/11       25/5*/11     
RANDOM/STRONG | 26/4*/12      26/4*/11      24/5/11       24/5/11       27/3*/11      26/3*/12      25/5*/11      24/5/12*     
RANDOM/weak   | 28/2/9        28/2/9        27/3/8        27/3/10*      29/2/9        28/2/9        28/3/9        28/3/8       
RANDOM/WEAK   | 28/2/9        28/2/9        27/3/9        27/3/9        28/2/10*      28/2/9        27/3/10       27/3/9       
RANDOM/both   | 26/4/11       26/4/11       24/5/11       24/5/12*      27/3/12       27/3/11       25/5/11       25/5/11      
RANDOM/BOTH   | 26/4/12       26/4/12       24/5/12*      25/5/11       27/3/11       27/3/12       25/5/11       25/5/11      
strong/random | 22/8/12       22/8/12       20/9/12       20/9/11       23/6/12*      22/7/12       20/9/12       21/9/11      
strong/RANDOM | 22/8/11       22/8/12       20/9/12       20/9/12       23/6/11       23/7/11       20/9/11       20/9/12*     
strong/strong | 18/11/13*     18/12/12      16/14/10      16/14*/11     19/10*/13     19/11*/12     16/13/10      16/14*/11    
strong/STRONG | 18/11*/12     18/11/12      16/14*/10     16/14/11      20/9/12*      19/10/12      16/14*/10     16/14/11     
strong/weak   | 24/5/13       23/5/14       22/7/14       22/7/15       25/4/13       24/5/15       22/7/15       22/6/15*     
strong/WEAK   | 24/5/13       23/6/14       22/7/15*      21/7/14       25/4/13       24/5/14       22/7/14       22/7/14      
strong/both   | 18/11/13      17/12*/12     17/13/11      16/14/11      20/9/14*      19/10/13      16/13/11      16/13/11     
strong/BOTH   | 19/11/12      18/11/12      16/13/11      16/14/10      20/9/13*      19/10/13      17/13/11      16/13/11     
STRONG/random | 22/7/11       22/8/12       20/9/12       20/9/11       23/6/12*      23/7/12       21/9/11       21/9/11      
STRONG/RANDOM | 22/7/11       22/8/12       20/9/12       21/9/11       23/6/12       23/7/12*      21/9/11       21/9/11      
STRONG/strong | 18/10/14      18/10/14      17/13/12      16/13*/12     20/8/15       19/9/15*      17/13*/12     17/13*/12    
STRONG/STRONG | 19/10*/13     18/11*/13     17/13*/11     17/13/12      20/9*/14      20/9/15*      17/13/12      17/13/12     
STRONG/weak   | 24/5/12       23/6/13       22/6/14*      22/7/14       25/4/14       24/5/14       22/7/13       22/7/13      
STRONG/WEAK   | 24/5/13       23/6/13       22/7/13       22/7/13       25/4/14*      24/5/14       22/7/14       22/7/13      
STRONG/both   | 19/10/14      19/10/14      17/12/12      17/13/12      20/8/16*      19/9/15       17/13/12      17/12/12     
STRONG/BOTH   | 19/10/14      18/10/14      16/13/13      17/12/13      20/8/15       19/9*/15*     17/12/13      17/12/13     
weak  /random | 31/0/5        31/0/5        31/0/6        31/0/6*       32/0/4        32/0/4        31*/0/6       31*/0/5      
weak  /RANDOM | 31*/0/5       31/0/5        31/0/7*       31/0/7        32*/0/4       32*/0/4       31*/0/5       31/0/6       
weak  /strong | 31*/0/6       31*/0*/7      30/1/9*       30*/1*/8      31/0*/5       31/0/5        30/1/8        30/1/8       
weak  /STRONG | 31/0*/7       31*/0/6       30/1*/8*      30/1/8        32/0/5        31/0*/5       30/1*/8       30*/1*/8     
weak  /weak   | 32/0/4        32*/0/4       32/0/5        32/0/5*       32/0/3        32*/0/3       32/0/4        32/0/4       
weak  /WEAK   | 32*/0/4       32/0/4        31*/0/5       31/0/6*       32/0/3        32/0/3        32*/0/5       32*/0/5      
weak  /both   | 31/0/6        31*/0/6       30*/1/8*      30/1/8        31/0/5        31/0/5        30/1/7        30*/1/7      
weak  /BOTH   | 31*/0/6       31/0/7        30/1/8*       30/1/8        32*/0/4       32*/0/5       30*/1/7       30*/1/7      
WEAK  /random | 31*/0/5       31*/0/5       31*/1/5*      31*/1/5       32*/0/4       32*/0/4       31/1/5        31/0/5       
WEAK  /RANDOM | 31/0/5        31*/0/5       31*/1/6       31*/1/6*      32/0/4        32/0/4        31/1/5        31*/1/5      
WEAK  /strong | 31/1*/6       30/1*/7       30*/1*/7      30/1*/7*      31*/0*/5      31*/0/5       30*/1/7       30*/1/7      
WEAK  /STRONG | 31*/1/6       31/1/6        30*/1/7*      30*/1/7       32*/0/4       31*/0/5       30*/1*/6      30/1*/7      
WEAK  /weak   | 32*/0/3       32/0/4        32*/0/4*      32*/0/4       32*/0/3       32/0/3        32*/0/4       32*/0/4      
WEAK  /WEAK   | 32/0/4        32*/0/4       31/0/5        31*/0/5       32*/0/3       32*/0/3       31/0/5*       31/0/5       
WEAK  /both   | 31*/0/6       31/1/6        30/1/7*       30*/1/7       32*/0/5       32*/0*/5      30*/1/7       30/1/7       
WEAK  /BOTH   | 31/1/6        31*/1/6       30*/1/7*      30*/1/6       32/0/5        31/0/5        30/1/7        30/1/6       
both  /random | 22/6/15       22/7/16       20/9/14       20/8/15       23/5/17       23/5/17*      20/8/15       20/8/15      
both  /RANDOM | 22/6/16       21/7/15       20/8/16       20/8/15       23/5/16       23/5/17*      20/9/15       20/8/14      
both  /strong | 19/10*/15     18/10*/15     16/13/14      16/13*/14     20/7*/18*     20/8*/17      16/12*/13     16/12/14     
both  /STRONG | 19/9/16       18/10*/16     16/13*/13     16/13/13      20/7/17*      20/8/17       16/12/14      16/12*/14    
both  /weak   | 23/4/18       23/4/18       21/6/19*      22/6/18       25/2/18       24/3/19       21/6/19       21/6/18      
both  /WEAK   | 23/4/18       23/5/18       21/6/19*      21/6/19       24/3/19       24/3/18       21/6/19       21/6/19      
both  /both   | 19/9/17       18/10/17      16/12/15      16/12/15      20/7/18       20/8/18*      16/12/14      17/12/14     
both  /BOTH   | 19/9/16       18/10/16      16/12/14      16/13/14      21/7/17       20/8/17*      16/12/14      16/12/14     
BOTH  /random | 22/6/15       22/6/15       20/8/15       21/8/15       23/4/17       23/5/18*      21/8/15       21/8/14      
BOTH  /RANDOM | 22/6/15       22/6/15       21/8/15       20/8/14       23/4/17       23/5/17*      20/8/14       20/8/14      
BOTH  /strong | 19/8*/18      19/8/19       17/11*/16     17/11*/15     20/6*/21*     20/7*/20      17/11/16      17/11*/15    
BOTH  /STRONG | 19/8/17       19/9*/17      17/11/16      17/11/16      21/6/20*      20/7/19       17/11*/16     17/11/16     
BOTH  /weak   | 23/4/17       23/4/17       22/6/17       22/6/17       24/2/20*      24/3/19       22/6/16       22/5/17      
BOTH  /WEAK   | 24/4/16       23/5/17       22/6/16       22/6/17       24/3/18       24/3/19*      22/6/16       22/6/16      
BOTH  /both   | 19/8/19       19/8/19       17/11/16      17/11/16      21/6/20       20/6/21*      17/10/16      17/11/16     
BOTH  /BOTH   | 20/8/18       19/8/18       17/11/16      17/11/16      21/6/20       20/6/20*      17/11/16      17/11/16

If you take this data at face value, there are some obvious lessons to be drawn. The “weak” strategies clearly dominate for the strong players, so game theory dictates that we expect strong players to choose those rows of the table, and in those rows the “strong” strategies mildly dominate for the medium-strength player. The weak players are screwed if the others play rationally.

This appears to give a neat and tidy answer to the original problem, but unless I’m missing something (and maybe I’m just misunderstanding the game, which I know only from Peter’s description) this simulation fundamentally mischaracterizes the motivations of the players. Each of the strong players wants to win themselves—knowing that the game was won by some other strong player isn’t much consolation for losing. So although all three strong players playing the weak strategy maximizes the chances of a strong player winning, there is a prisoner’s dilemma going on. If they’re all rational and stick together in picking off the weak players then they’ve got a 30%+ chance to win, but if one strong player betrays the other strong players by playing the “strong” strategy and trying to first eliminate strong players, then she has a significant advantage over the two collaborating strong players. Thus it is actually in each strong player’s best interest to defend against such behavior by playing the “strong” strategy themselves. With all three strong players playing the strong strategy there’s only a 20% chance each of them will win…and the chance that a medium or weak player will win jumps from 7% to almost 40%!

If the strong players are all fighting it out amongst themselves, the medium and weak players have much better chance, but there’s not much in it to decide between their strategies—“weak” seems fine for the medium player. The real lesson of this analysis is that the medium strength player needs to worry less about what he is doing and more about making sure the strong players understand the game theory well enough to pursue the competitive, not cooperative, strategy. As seems to be the case in real life, those with quantifiable advantages have to think through their actions, while everyone else is free to waste their time with politics…

Update: Peter Norvig has updated his post to account for this bug, and more importantly he actually tested the “defection” strategies I suggest (which I really should have done before asserting that they change the results). It turns out that there’s only a marginal advantage for a strong player who defects against the other two if nobody punishes them for it, and a significant disadvantage if the other two use a “revenge” strategy, so there’s really no prisoners’ dilemma. Rational strong players choose the WEAK strategy (nominate a strong player if they nominated you, but otherwise nominate a weak player), which corresponds with the well-known “tit-for-tat” strategy. Given that choice for the strong players, the medium-strength player should always nominate the strong players. The data is noisy over the finer details, but my expectation is that weak players also benefit from nominating strong ones, and that if everyone is playing rationally then the medium and weak players should not use revenge: it’s much more important for them to eliminate the strong players than to punish each other.

Codec Bug in Python ※

Posted Saturday, 16 August 2008

I seem to have stumbled across a bug in Python’s libraries for dealing with unicode data: apparently mixing calls to file.readline() and file.readlines() works just fine for regular 8-bit input files, but not for files read through a codec. Given any file sample.txt with more than a few dozen characters—even just ASCII characters—the version of Python 2.5.1 which ships as part of Mac OS X behaves like this:

>>> f = open('sample.txt')
>>> first_line = f.readline()
>>> remaining_lines = f.readlines()
>>> len(remaining_lines)
26
>>> import codecs
>>> f = codecs.open('sample.txt')
>>> first_line = f.readline()
>>> remaining_lines = f.readlines()
>>> len(remaining_lines)
26
>>> f = codecs.open('sample.txt', encoding='utf-8')
>>> first_line = f.readline()
>>> remaining_lines = f.readlines()
>>> len(remaining_lines)
3

Luckily, there’s an obvious workaround:

>>> f = codecs.open('sample.txt', encoding='utf-8')
>>> all_lines = f.readlines()
>>> len(all_lines)
27

If you want to test it with your own data, here is a test script:

#!/usr/bin/env python

import sys
import codecs

def testfile(f):
    firstline = f.readline()
    remaining_lines = f.readlines()
    print "Read " + str(len(remaining_lines) + 1) + " lines."
    f.close()

def main(argv=None):
    if argv is None: argv = sys.argv

    for filename in argv[1:]:
        print "Opening " + filename + " using `open` built-in:"
        testfile(open(filename))
        print "Opening " + filename + " using `codecs.open` with no encoding:"
        testfile(codecs.open(filename))
        print "Opening " + filename + " using `codecs.open` with encoding:"
        testfile(codecs.open(filename, encoding='utf-8'))

if __name__ == "__main__": sys.exit(main())

The documentation suggests that mixing calls to readline() and readlines() should be safe for any file object, so it does look like a bug. I’ll have to check whether it’s fixed in the Python 3 betas (which use unicode by default).