Before the hack day, Jean-Paul Calderone came to the office and gave some talks about twisted. I asked him if there were any tools for finding the cause of reactor pauses. He said he’d built his own little script a while back, and that the key was to use SIGALRM. That was enough to get me on the right track.

Here’s the result. It’s a pretty simple script. Every 100 milliseconds, it cancels any pending SIGALRM and calls setitimer, so that the OS will send SIGALRM to the process in 500ms. It also has a handler for SIGALRM that logs a traceback and adds the offending function to a stats dict.[2]

If the reactor is paused for more than 500ms, the pending SIGALRM won’t be cancelled, so it will be sent to the process. Then the handler will print out the traceback and update stats. Pretty handy.

My tool can give false positives on a heavily-loaded system. This is because I’m calling setitimer with ITIMER_REAL instead of ITIMER_VIRTUAL. Using virtual time won’t catch stuff like sleep()s in the main thread, since sleeping doesn’t count toward execution time. Using real time will fire the SIGALRM after 0.5 seconds even if the process has gotten zero time on the CPU. I got a few false positives on my overloaded VM, but this turned out to be beneficial. The tracebacks went all the way to select(). I mentioned this fact to Paul and he reminded me that we should be using the epoll reactor.

That’s two production issues identified and fixed because of a silly hack day project. Booyah.[3]

Footnotes:

[1] We got around performance issues by running multiple instances of these services. The problems were never fixed because they were in really old code that was written before anyone at Cloudkick knew how to write twisted.

[2] A stats defaultdict, to be precise. If you haven’t used defaultdict before, check it out. It will save you from writing some dumb boilerplate code.

[3] I wasn’t the only person to do a hack day project. Here’s the full list.

1. Google
2. Stack Overflow
3. Actually reading the docs
4. Asking someone who knows more about it than you

…usually in that order. That’s not a terrible way of going about it, but there’s an oft-overlooked option. Instead of scouring documentation or bugging a coworker, sometimes it’s best to run experiments.

This code…

#!/usr/bin/env python

blah = 1

def foo():
    blah = 0
    print "foo:blah", blah

print "global:blah", blah
foo()
print "global:blah", blah

…prints…

global:blah 1
foo:blah 0
global:blah 1

While adding a global in foo…

#!/usr/bin/env python

blah = 1

def foo():
    global blah
    blah = 0
    print "foo:blah", blah

print "global:blah", blah
foo()
print "global:blah", blah

…prints…

global:blah 1
foo:blah 0
global:blah 0

Without global, foo’s blah is not the same blah. With global, it is.

These sorts of code blurbs are great for learning. They’re also great for solving disagreements about code behavior. Warning: You will be wrong sometimes. You will look stupid in front of your peers. On the bright side, these embarrassing moments will stick in your mind. You won’t forget what you were wrong about.

Code experiments can be much trickier, and much more useful, than above. For example…

…at work we recently switched a project’s MySQL engine from MyISAM to InnoDB. After the switch, we encountered some weird errors. A process would save an object to the DB, but some services couldn’t find the newly-created object[2]. I had a hunch that transactions were responsible for the weirdness; my reasoning being that MyISAM lacks transaction support, and it had worked fine.

So I did science. I opened up two MySQL consoles. In console #1, I ran start transaction;. Then in console #2, I ran:

mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)

mysql> insert into inventory_nodeaddress (node_id, ip, ip_version, type) values (NULL, '31.22.190.54', 4, 0);
Query OK, 1 row affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from inventory_nodeaddress where node_id is NULL;
+-----+---------+-------------------------------------+------------+------+
| id  | node_id | ip                                  | ip_version | type |
+-----+---------+-------------------------------------+------------+------+
| 106 |    NULL | 50.57.96.184                        |          4 |    0 |
| 107 |    NULL | 10.182.67.171                       |          4 |    1 |
| 147 |    NULL | 31.22.190.54                        |          4 |    0 |
+-----+---------+-------------------------------------+------------+------+
3 rows in set (0.00 sec)

mysql>

OK, the data’s committed. I even double-checked that it was there by selecting it. Everything is fine, right?

Nope. Back in console #1, I ran:

mysql> select * from inventory_nodeaddress where node_id is NULL;
+-----+---------+-------------------------------------+------------+------+
| id  | node_id | ip                                  | ip_version | type |
+-----+---------+-------------------------------------+------------+------+
| 106 |    NULL | 50.57.96.184                        |          4 |    0 |
| 107 |    NULL | 10.182.67.171                       |          4 |    1 |
+-----+---------+-------------------------------------+------------+------+
2 rows in set (0.00 sec)

mysql>

Once I ended the transaction in console #1 (either through a rollback or a commit), the new row showed up in selects. After some Googling I finally found the relevant MySQL documentation. Once I changed the transaction isolation from REPEATABLE-READ to READ-COMMITTED, selects inside transactions showed recently-inserted rows.

The experiment plus Brad‘s knowledge of Django helped solve the mystery. Django only runs commit when writing to the DB. This sucks for any long-running service that never writes. The service will start up, connect to MySQL, run start transaction and do selects without ever ending the transaction. With the default InnoDB configuration, these services will see an ever-older version of the database. Not fun.

After I added transaction-isolation = READ-COMMITTED to the my.cnf chef template, everything worked swimmingly. Hooray for science.

The next time you’re stumped, try some experiments. As a programmer, you have immense power over the program’s universe. Your code runs on a perfectly deterministic machine[3]. With the right software tools, you can stop time. You can read or change any part of memory. You can step through, one line at a time, to see exactly what’s happening.

Of course, this isn’t real science. These apparati make programming a cakewalk compared to real science.

Footnotes:

[1] You forgot this fact not because you suck at Python, but because you usually write clean code with no globals. At least, that’s what you keep telling yourself.

[2] Just so nobody freaks out: This thing is non-customer-facing and currently under heavy development. I’m also over-simplifying the process. The actual changes happened in a development branch and weren’t merged until things were hunky-dory.

[3]Cosmic rays notwithstanding.

Improving performance is not always easy, but it is simple:

1. Find the slowest part of the program.
2. Make that part faster.
3. Repeat until it’s fast enough or you go insane.

There are lots of profiling tools and programmers often argue about which is the best. I use gprof, callgrind, and Instruments.app. Which profiler you use doesn’t matter as much as actually using one. They all have their advantages and disadvantages, but for this post I’ll only cover Valgrind’s callgrind. Using callgrind doesn’t require special compilation. Just invoke it with your program’s name and it will generate profiling data for callgrind_annotate to analyze.

Here’s a typical profiling run for Ag:

$ make clean && ./build.sh
(snip)
$ time valgrind --tool=callgrind ./ag --literal abcdefghijklmnopqrstuvwxyz ../
(snip)
real	1m34.709s
user	1m33.206s
sys	0m1.492s
$ callgrind_annotate --auto=yes callgrind.out.10361
--------------------------------------------------------------------------------
Profile data file 'callgrind.out.10361' (creator: callgrind-3.6.1-Debian)
--------------------------------------------------------------------------------
I1 cache:
D1 cache:
LL cache:
Timerange: Basic block 0 - 798409857
Trigger: Program termination
Profiled target:  ./ag --literal abcdefghijklmnopqrstuvwxyz ../ (PID 10361, part 1)
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       99
Include dirs:
User annotated:
Auto-annotation:  on

--------------------------------------------------------------------------------
           Ir
--------------------------------------------------------------------------------
3,068,387,924  PROGRAM TOTALS

--------------------------------------------------------------------------------
           Ir  file:function
--------------------------------------------------------------------------------
1,764,541,095  src/util.c:ag_strnstr [/home/geoff/code/the_silver_searcher/ag]
  386,020,821  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch [/lib/x86_64-linux-gnu/libc-2.13.so]
  226,548,868  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strcmp.S:__GI_strncmp [/lib/x86_64-linux-gnu/libc-2.13.so]
  181,861,517  src/util.c:is_binary [/home/geoff/code/the_silver_searcher/ag]
  123,211,270  /build/buildd/eglibc-2.13/posix/fnmatch.c:fnmatch@@GLIBC_2.2.5 [/lib/x86_64-linux-gnu/libc-2.13.so]
  104,867,805  src/print.c:print_file_matches [/home/geoff/code/the_silver_searcher/ag]
   77,058,570  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strlen.S:__GI_strlen [/lib/x86_64-linux-gnu/libc-2.13.so]
   60,030,629  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch'2 [/lib/x86_64-linux-gnu/libc-2.13.so]
   44,019,376  src/ignore.c:filename_filter [/home/geoff/code/the_silver_searcher/ag]
   27,072,821  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/memchr.S:memchr [/lib/x86_64-linux-gnu/libc-2.13.so]
    9,329,984  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strcmp.S:__GI_strcmp [/lib/x86_64-linux-gnu/libc-2.13.so]
    7,803,075  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
    7,040,644  /build/buildd/eglibc-2.13/posix/../locale/weight.h:internal_fnmatch
    6,062,124  /build/buildd/eglibc-2.13/string/../string/memmove.c:__GI_memmove [/lib/x86_64-linux-gnu/libc-2.13.so]
    4,384,383  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../memcpy.S:__GI_memcpy [/lib/x86_64-linux-gnu/libc-2.13.so]
    3,951,640  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_free [/lib/x86_64-linux-gnu/libc-2.13.so]
    3,779,300  /build/buildd/eglibc-2.13/dirent/../sysdeps/unix/readdir.c:readdir [/lib/x86_64-linux-gnu/libc-2.13.so]
    3,181,118  /build/buildd/eglibc-2.13/malloc/malloc.c:malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
(snip)

I snipped out the annotated source code. You can see the full output here.

This profiling info tells me that I’m spending all my time in strnstr(). I did some research on string-matching and found out about the Boyer-Moore algorithm. After some more reading, I decided to go with a simplified version of Boyer-Moore called Boyer-Moore-Horspool.

Here’s the data after I implemented Boyer-Moore-Horspool strstr:

$ time valgrind --tool=callgrind ./ag --literal abcdefghijklmnopqrstuvwxyz ../
real	0m32.429s
user	0m31.034s
sys	0m1.324s
$ callgrind_annotate --auto=yes callgrind.out.11921
--------------------------------------------------------------------------------
Profile data file 'callgrind.out.11921' (creator: callgrind-3.6.1-Debian)
--------------------------------------------------------------------------------
I1 cache:
D1 cache:
LL cache:
Timerange: Basic block 0 - 228181262
Trigger: Program termination
Profiled target:  ./ag --literal abcdefghijklmnopqrstuvwxyz ../ (PID 11921, part 1)
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       99
Include dirs:
User annotated:
Auto-annotation:  on

--------------------------------------------------------------------------------
           Ir
--------------------------------------------------------------------------------
1,139,437,344  PROGRAM TOTALS

--------------------------------------------------------------------------------
         Ir  file:function
--------------------------------------------------------------------------------
386,014,011  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch [/lib/x86_64-linux-gnu/libc-2.13.so]
181,870,097  src/util.c:is_binary [/home/geoff/code/the_silver_searcher/ag]
123,209,345  /build/buildd/eglibc-2.13/posix/fnmatch.c:fnmatch@@GLIBC_2.2.5 [/lib/x86_64-linux-gnu/libc-2.13.so]
104,867,805  src/print.c:print_file_matches [/home/geoff/code/the_silver_searcher/ag]
 76,747,163  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strlen.S:__GI_strlen [/lib/x86_64-linux-gnu/libc-2.13.so]
 63,421,170  src/util.c:boyer_moore_strnstr [/home/geoff/code/the_silver_searcher/ag]
 60,028,609  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch'2 [/lib/x86_64-linux-gnu/libc-2.13.so]
 44,018,667  src/ignore.c:filename_filter [/home/geoff/code/the_silver_searcher/ag]
 27,072,637  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/memchr.S:memchr [/lib/x86_64-linux-gnu/libc-2.13.so]
  8,312,570  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strcmp.S:__GI_strcmp [/lib/x86_64-linux-gnu/libc-2.13.so]
  7,803,075  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
  7,040,534  /build/buildd/eglibc-2.13/posix/../locale/weight.h:internal_fnmatch
  6,061,868  /build/buildd/eglibc-2.13/string/../string/memmove.c:__GI_memmove [/lib/x86_64-linux-gnu/libc-2.13.so]
  4,384,383  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../memcpy.S:__GI_memcpy [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,951,640  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_free [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,779,220  /build/buildd/eglibc-2.13/dirent/../sysdeps/unix/readdir.c:readdir [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,181,118  /build/buildd/eglibc-2.13/malloc/malloc.c:malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,089,135  src/main.c:search_dir'2 [/home/geoff/code/the_silver_searcher/ag]
  2,095,514  /build/buildd/eglibc-2.13/malloc/malloc.c:free [/lib/x86_64-linux-gnu/libc-2.13.so]
  2,018,298  /build/buildd/eglibc-2.13/dirent/../sysdeps/wordsize-64/../../dirent/scandir.c:scandir [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,941,992  /build/buildd/eglibc-2.13/string/strcoll_l.c:strcoll_l [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,889,859  /build/buildd/eglibc-2.13/stdlib/msort.c:msort_with_tmp.part.0'2 [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,704,553  /build/buildd/eglibc-2.13/malloc/malloc.c:malloc_consolidate.part.3 [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,644,688  src/ignore.c:ignorefile_filter [/home/geoff/code/the_silver_searcher/ag]
  1,601,628  /build/buildd/eglibc-2.13/dirent/../sysdeps/unix/sysv/linux/getdents.c:__getdents [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,582,620  src/util.c:strlcat [/home/geoff/code/the_silver_searcher/ag]
(snip)

For the curious, full output of callgrind_annotate is here.

That’s a 3x overall speedup and a 27x speedup in string matching. Impressive! Now Ag is spending most of the time figuring out whether or not it should search a file. It’s clear where I need to optimize next.

Valgrind isn’t perfect though. It makes programs run 25-50x slower than they normally would, so you won’t notice if you’re spending all your time waiting for network or disk I/O. In the case of Ag, this turned into a 20% performance improvement in my benchmarks.

Getting more useful data requires switching from an instrumenting profiler to a sampling profiler. Both Instruments.app and gprof are sampling profilers, but this post is already too long. I’ll cover them some other time.

Worst of all, you have to write documentation.

This stuff isn’t fun, but it’s necessary if you want others to use your project.

Because you made it, the various dependencies and quirks are obvious to you. For the poor soul who clones your repository, the same is not true. Even compiling is a challenge for a newbie. What build system does your project use? Make? Scons? Ant? What build dependencies does it have? Does it check for them or print out a useful error message if any are missing? Is there a helpful README?

If you want other people to use (and possibly one day improve) your work, you need to polish the build scripts and write documentation. Think of it like a sales funnel. Perhaps 100 people download your code, 80 get it to build, 75 run it, 50 use it regularly, 5 make modifications, and finally, 2 contribute patches back. You can increase the numbers. Making those steps easier will grow your user base and contributions.

So that’s what I’ve done with Ag. Over the past couple of weeks I’ve added a man page, a wiki, accepted pull requests to clean up the build, and even improved the –help output.

I’ll be the first to admit that fixing trivial inconveniences is boring. But frequently, boring work is the difference between a personal project and a community of users. Too often, trivial inconveniences stop projects from reaching critical mass. So if you want to build something for others, grit those teeth and go schlep.

So I built it. I call it The Silver Searcher, or Ag for short. Ag is like ack, but better. It’s fast. It’s damn fast. The only thing faster is stuff that builds indicies beforehand, like Exuberant Ctags.

Don’t believe me? Here are some benchmarks. I ran them multiple times and grabbed the median for each.

ggreer@carbon:~/cloudkick/reach% du -sh
250M	.

ggreer@carbon:~% time grep -r -i SOLR ~/cloudkick/reach | wc -l
     617
11.06s user 0.81s system 96% cpu 12.261 total

ggreer@carbon:~% time ack -i SOLR ~/cloudkick/reach | wc -l
     488
2.87s user 0.78s system 97% cpu 3.750 total

ggreer@carbon:~% time ag -i SOLR ~/cloudkick/reach | wc -l
     573
1.00s user 0.51s system 95% cpu 1.587 total

Here’s Ag with some extra ignores, similar to how ack ignores many files by default:

ggreer@carbon:~% cat ~/cloudkick/reach/.agignore
extern
release
fixtures
ggreer@carbon:~% time ag -i SOLR ~/cloudkick/reach | wc -l
     499
0.35s user 0.15s system 94% cpu 0.528 total

That’s the same as git grep:

ggreer@carbon:~/cloudkick/reach% time git grep -i SOLR ~/cloudkick/reach | wc -l
     489
0.32s user 0.58s system 161% cpu 0.556 total

…except git grep only works in git repos. And it doesn’t ignore stuff in the repository like extern or generated files.[3]

The bottom line: Grep’s output was the least useful. It dutifully reported matches in .pyc files and other things I don’t care about. Ack’s results were better and faster than grep. Ag had more results than ack, but took half as long. With a couple of clever ignores (like the extern directory), Ag took a mere half-second and gave even more pertinent results.

I can already hear someone saying, “Big deal. It’s only a second faster. What does one second matter when searching an entire codebase?” My reply: trivial inconveniences matter. Using Ag is like having a faster computer; you don’t realize how slow things were until you’ve experienced fast. The difference is big enough that I can’t go back to ack, just like ack users can’t go back to grep.

Since it behaves like ack, Ag can be used by many fancy ack GUI front-ends. This makes searching convenient as well as fast. After I got Ag sorta-working, I forked AckMate so that I could use Ag in my favorite editor. My fork bundles both AckMate’s Ack and my own Ag. You can switch between them with a simple check box. The tmbundle is on the downloads page. Be warned: it replaces your current AckMate.

There’s still plenty of stuff I want to add,[4] but it’s good enough for my own daily use so I figured I should tell others about it. And of course, patches are welcome!

 

 

 
Footnotes:

[1] The decision was made to put all python dependencies into extern/ instead of using pip. A good call, in my opinion.

[2] At least not without a bunch of pipes and find and xargs. Yes I know there are aliases but it’s annoying to keep those up-to-date.

[3] Yes, I know it’s bad form to put generated files in revision control.

[4] Ctags support, for one. Also inverted matching, accepting piped input, and basic stuff like retrying a search with fewer ignores and no case-sensitivity.

By our standards, ancient Rome was not a nice place. I’m not just talking about problems due to inferior technology or sanitation or disease. Even a hypothetical ancient Rome that fixed those things would be worse than the modern world. Slaves would still exist. Racism and discrimination would still be rampant. The citizenship of one’s parents would still determine your lot in life. Simply put, ancient Roman morality was worse than ours.

It is my claim that over time, most civilizations have moved along a moral trajectory. Slowly -ever so slowly- people have become nicer. We’ve recognized more humans as worthy of moral value. Eventually we decided slavery was wrong. We began to treat women and men equally. Currently we’re moving toward treating homosexuals and heterosexuals equally.

The fact that morals have improved over time should strike people as extraordinary. Think about what must have happened for slavery to go from right to wrong. At some point, a person went from thinking “slavery is right” to thinking “slavery is wrong”, without initially wanting to change his mind! I wish I knew how to trigger that sort of thinking. It’s as valuable as vaccines.

It’s great that people have become more moral over the millennia. Now what are the odds that after thousands of years, we suddenly got morality right? What are the chances that we won’t improve upon our current ideas about right and wrong?

Yeah, pretty damn small.

OK, so if we’re wrong, we should ask the question, “What are some things that future societies might condemn us for?”

Stop here and think about them. I don’t want to contaminate your initial ideas.

 

 

 
…

 

 

 

There are some obvious ones that are already in political discourse. Recreational drugs. LGBT rights. Border and immigration laws. Religious indoctrination of children. Other things that You Can’t Say… yet.

One day morality will even go beyond those things. What about eating meat? It’s absolutely indefensible that we breed, kill, chop up, and eat animals. It’s not the eating of meat that’s wrong. If we could grow meat in vats, there would be no problem. The problem is that animals experience pain much like we do. While animals are not open to the same range of experiences as humans, they still seem to have some moral value. All else equal, I think we’d prefer that animals not suffer and die.

Future technologies could improve morality as well. Like education and nutrition today, in the future it could be considered abusive to not give your child anti-aging and intelligence-enhancing treatments. And that’s just the start.

Does this future scare you? Well so would the present to those who lived in the past. Eutopia is Scary.

I don’t own an iPad for the same reason I don’t have a cable subscription: Because it encourages shallow content consumption.

Don’t misunderstand. I don’t think content consumption is bad. I enjoy it and spend lots of time doing it. But there are different ways to consume it, different types to consume, and differing amounts of time one can spend on it.

So what is shallow content? The shallowest content consists of action movies and chick flicks. These are equivalent to sitting in front of a machine that pulls levers in your brain to trigger emotional reactions. Shallow content is easily accessible, but uninformative and in the long run, less rewarding.

Deep content is complex. Profound. Memorable. It is organized on multiple levels. It rewards re-reading (or re-watching). Gödel, Escher, Bach: An Eternal Golden Braid is the prototype of deep content.

Shallow content is cotton candy and Coca-Cola. Deep content is raspberry cheesecake and Riesling.

When it comes to content consumption, different devices encourage different behaviors. For example, a television is the ultimate device for consuming shallow content. You select a channel. The pictures and sounds elicit emotional responses. Changing the channel is easy, so TV shows are selected for stickiness and addictiveness. TV shows can’t be as complex as other media.[1] They have to be accessible to people who start watching in the middle.

Whether or not a device is meant for content consumption is not a binary attribute; it’s more of a sliding scale. A television is solely for consuming. An iPad is more general-purpose, but encourages consumption over production, and shallow content over deep. A laptop or desktop computer is fully general-purpose. A good indicator of a device’s purpose is its input interface. A TV’s input is simple: channel selection. An iPad has a touch-screen that can show a virtual keyboard for occasions when you need to write something. A computer has a physical keyboard, the fastest brain-machine interface currently available.

Of course, even with a computer you can waste your days on Reddit and IRC (consuming and producing shallow content) or you can do something more rewarding.

[1] Also, the cost to make a TV show is much greater than the cost to make a book. So to be profitable, a TV show must appeal to a greater population than a book. This limits the ideas and opinions on television.

Before, my information diet consisted of countless hours reading short, forgetful pieces linked from IRC, Twitter, Google Reader, or Hacker News. I would try to read longer articles on my laptop, but it was hard to avoid distractions such as e-mail and IM clients.

Reading full-length books was also a hassle, although I didn’t realize it at the time. I’d order a book from Amazon and wait two days. Then I had to carry a chunk of dead tree around. When travelling, I usually took multiple books with me, since I’d finish more than one on a trip.

Right after I got my Kindle, Ryan Phillips told me about Instapaper. Now I skim or ignore short things, and use Instapaper to mark a couple of large gems for evening reading. Instead of habitually refreshing Hacker News, I load it maybe once a day.

Fan fiction such as Harry Potter and the Methods of Rationality have fan-created .mobi and .epub files, but many stories aren’t popular enough to warrant such devotion. Fortunately I found FanFiction Downloader, a semi-automated way to get FanFiction.net stories onto my Kindle. The app isn’t perfect though. It runs out of memory on longer books.

The only major annoyance I’ve encountered is from flight attendants asking me to turn off my Kindle during takeoff and landing. I smile and wait for them to move on, then continue reading. Flipping the power switch wouldn’t do much anyway; Kindles don’t really turn off. They wake up to check for new content periodically, and there’s no way to remove the battery. I’m surprised the “no electronic devices” rule has lasted so long. If I were a conspiracy theorist, I’d rant about Amish Illuminati controlling the FAA. Googling for that idea doesn’t return much. I guess conspiracy theorists aren’t very creative.

E-mailing me my password tells me several things about your company. It tells me that you store passwords in plaintext instead of hashing them. If anyone gets ahold of a DB dump, they’ll have passwords and e-mail addresses. Lots of people use the same password everywhere, making their e-mail vulnerable. E-mailing my password also tells me that you don’t either don’t know or don’t care about the dangers of sending secrets via e-mail. E-mail isn’t always encrypted and messages are often relayed through many servers. Anyone with access to one of those servers could see my password.

The password thing was a big red flag, but I didn’t want to give up so easily. I booted a server and started screwing around with it. I renamed my server in the VPS.net dashboard. Suddenly, my ssh session died. It turns out that renaming a server reboots it without warning. Frustrated, I gave up and decided to try again when I had more patience.

I woke up and saw my inbox contained an invoice for $1.00. Yes, VPS.net sends an invoice every day. Worse, after a week, VPS.net started warning me that my invoices were overdue. I tried to log in and pay the measly $10. I was confronted with a login page asking me to enter my username, password, and answer some security questions. They noticed I was trying to log in from a different IP address and threw some security questions at me. I finally managed to get enough answers correct to log in.

That “Pay Now” button is actually a “try to pay $1 and show a big failure message, but mark the invoice as paid if there have been no payment attempts in the past few hours” button. I had to click it once for each invoice, waiting 3-4 hours between tries if I wanted them to work. Later I noticed the charges actually showed up on my card.

There were other things I noticed, such as sequential instance IDs. Did you know VPS.net has only booted a total of 33,000 instances? Anyway, after the invoice thing I wrote VPS.net off as amateurs and tried out Linode. I’ve had no problems with Linode. Their stuff works without annoying the hell out of me. If you want a server in the UK, go with them.

[1] The litmus test I use is, “Can I make an API call and get a booted server in under 5 minutes?” If not, then it’s not really cloud computing. It’s more of a reasonably fast VPS provider.

[1] Bottlenecked by 142Mhz front side bus.

There are a dozen things that don’t show up in the numbers. The iBook’s keyboard is spongy. It has a smaller trackpad that only supports two-finger scrolling. The iBook’s display is much worse, although that may be due to age. The iBook’s trackpad is plastic instead of glass. Even with a fresh install of Leopard, it feels slow. Yet somehow I used it as my main computer for two years.

In case you forgot, that skinny ethernet port on the iBook is a 56k modem. It’s like ethernet, but data goes 1,800 times slower and makes angry noises.

I wonder what I’ll compare my Air to in 2015.

For example: A Samsung NC20 goes for around $500. A completely maxed-out 11.6″ MacBook Air costs $1500. Most people look at those two prices and think, “Wow, I could save $1,000 by taking a small hit in performance and features.” Or worse, “$1500 is too much money for a computer.” That is the wrong way to think about it.

Here’s why: According to RescueTime, I average 10 hours a day on my laptop. Assuming I upgrade every 18 months, that’s 5,400 hours of use. Amortized over its life, a $1,500 laptop costs me 28 cents per hour. A $500 laptop would be 9 cents per hour. The important factor I haven’t mentioned yet is the amount of value I can create per hour. Let’s be extremely pessimistic and say I create $5 worth of value per hour on average. Let’s also say the $1500 laptop makes me 10% more efficient than the $500 laptop. That means I would create $5.50 worth of value each hour on the expensive laptop. Subtracting the amortized cost of the expensive laptop still leaves me with an extra 31 cents per hour.

The expensive laptop costs 3x as much and only improves my performance by 10%, but it still comes out ahead purely for economic reasons. This is because the amount of value you extract or create using a computer is much greater than the hourly cost of that computer. I’m not even factoring in more subjective things like ease-of-use or aesthetics. You shouldn’t be worried about spending too much on your computer. You should be worried about not spending enough!

I’ve followed my own advice here and bought a maxed-out 11″ MacBook Air. I’m using it as my sole development machine. It’s amazing.

OK, I admit it. This entire post was just an excuse to brag about my MacBook Air.

Today, there are three main computing devices that people use: Laptops, which fit in a backpack and are fast enough for most stuff; smartphones, which fit in a pocket and are a tradeoff of size and speed/features; and desktops, which can have more storage and better graphics than laptops.

Now which one of those three things can an individual assemble and customize?

That’s why nerds/hackers are using the same hardware as everyone else. It used to be that almost everyone had desktops. Alpha geeks built custom machines from parts. Now, people use laptops and smartphones. Alpha geeks don’t have the resources to build or modify such tightly-integrated devices. Instead, they buy whatever best satisfies their needs. Lately, those products have been Macs and iPhones.

So are alpha geeks done with hardware tinkering? I doubt it. I think a new type of device will show up: the wearable. Recent advances in microdisplays and embedded computing have made wearable computers a borderline-practical idea. And individuals can build them for about the cost of a laptop. A few people have already hacked their own together from video goggles (like the Myvu Crystal) and single-board computers (like the BeagleBoard).

An often-heard objection to wearables is, “I already carry a computer with me. It’s called an iPhone and I can use it any time.” That’s true, but most people don’t realize how much trivial inconveniences can affect their behavior. Every time you want to look something up, you have to pull the phone out of your pocket and unlock it. You lose eye contact with anyone you were conversing with. It takes time for the phone to come out of hibernation and run whatever app you want to use. The screen is visible to those nearby, so you can’t message a friend, “What’s the name of this person next to me?”, or search your e-mail archives for a forgotten message that has come up in discussion.

Two major issues with wearables are fashion and software. Fashion will probably be resolved as technology gets better and people become more used to seeing wearables. Software is going to be a bigger problem. Wearables have to be extremely responsive. The idea is to completely integrate the computer with your life. Smartphone software has a similar constraints, but not to the same degree.

A wearable with good software would be outrageously useful. Imagine having a local cache of Wikipedia, e-mail, personal notes, and other data sources. Add a camera to the mix to get lifelogging and augmented reality. While AR is more of a toy, people completely underestimate the utility of lifelogging. I know of one case where a lifelogger had video of the first time he met his wife. A lifelog would also come in handy if you were witness to a crime or accident. To get some idea of what you do all day, you could put the lifelog data into something like RescueTime. If automated transcribing gets better you could build a searchable database of all your conversations. With these new sources of data and forms of interaction, the possibilities are quite vast.

Note: This post is expanded from a comment I made on Hacker News.

You need around 8 hours a day for sleep. That leaves 5,840 hours of wakefulness.

A full-time job is around 2,000 hours a year. 3,840 hours left.

In addition to work you’re paid for, you have to run errands and do chores (shop for groceries, go to the bank, yard work, vacuum, cook, wash dishes, etc). Let’s say you average 90 minutes a day on that, which is 548 hours per year. 3,292 hours remaining.

How long is your commute? The average is 25 minutes each way, so just over 208 hours a year. 3,084 hours left.

How long do you take to get ready in the morning? 40 minutes? 240 hours. 2,844 left.

Do you exercise? Maybe you don’t work out every day. 1 hour a day, 3 times a week? 156 hours. 2,688 left.

Now let’s say you’re 25 years old and healthy. According to most actuarial tables, that gives you even odds of sticking around for another 50 years.

2,688 hours * 50 = 134,400 hours. That’s 15 years. I admit it’s a very rough estimate. I left out a lot of factors (such as anything related to raising children). People work less when they’re older, but that’s partially offset by more time spent being sick. I’d also weight calculations toward free time in younger years, since youth usually means fewer responsibilities and greater physical ability. Later years are spent undergoing general cognitive and physical decline.

What wisdom do I take away from this calculation? Pretty straightforward stuff: Find a job you enjoy. Use a tool like RescueTime to ensure you spend your free time wisely. Finally, you spend a significant chunk of your life using your bed, office chair, and/or computer. Don’t skimp on those things.

Why?

Today is the 27th anniversary of when Stanislav Petrov chose not to destroy the world. When an early warning system reported the Americans had launched ICBMs, Petrov told his superiors it was a malfunction. Had he decided incorrectly, the Soviets would have launched their nukes, causing the United States to actually launch their nukes. Billions would have died.

I am continually amazed by how few know about this incident. The average person could probably tell you the latest celebrity gossip or which team won some sporting competition, but ask them who prevented a gigadeath event and they wouldn’t have a clue.

Lieutenant Colonel Petrov, your name isn’t as famous as it should be, but thank you nonetheless. The gratitude owed to you simply cannot be conveyed.

Flavors ranged from boring vanilla and chocolate to a surprisingly tasty mix of chili pepper and graham cracker. The heat of the peppers and the cold of the ice cream complemented each other, or something. I’m no culinary expert.

The art was pretty impressive too. I’ve thrown a few photos up on Flickr and a few more in my gallery. My favorite piece was a 40 foot tall metal statue of a woman.

Not only is the statue visually appealing, it’s an impressive engineering feat. There are no guy-wires to keep it standing. It’s balanced on a small pedestal hidden under the playa dust. The whole weight of the statue is supported by one foot.

Oh and it lights up at night. There are LEDs throughout the metal latticework, cycling through most of the color spectrum.

On Sunday afternoon I got a call: the San Francisco Police Department found my bike! It took a few hours to sort out paperwork and run UPDATE VEHICLES SET STOLEN=FALSE WHERE LICENCE_PLATE='ABC4567'; It was DMV-esque: talking to different cogs in the machine; each one converting time into frustration. Finally, I was ready to pick up my bike.

Patience waning, I walked across the street to the impound lot, where the real adventure began. I was told it would be $200 to get my bike out of impound. The fee would have been $400, but the city of San Francisco gave me a we’re-sorry-your-bike-got-stolen-but-not-so-sorry-that-we-won’t-charge-you discount. I had no choice. I paid the money. The sleepy lady behind the bulletproof glass then said my bike wouldn’t be available until tomorrow morning.

Now that, I will not abide.

I took a breath. The words I wanted to say would not help me get my bike back. Had I said them, the end result would have been to annoy a sleepy lady and force me to come back to the impound lot in the morning.

I thought up some new words that would help me get my bike back. I exaggerated my plight. “Lady, I have a flight in the morning and I won’t be back for a week. That bike is my only vehicle. You’re the only person who can help me right now.” This was… somewhat true. I had a flight, but it was in the afternoon. I own a car, but I haven’t driven it in weeks. Really, I just wanted my bike.

The sleepy lady relented. My bike would be at the impound lot in an hour. Where was it coming from? Why couldn’t I take a cab to where it was now? Again I bit my tongue.

Finally, done with bureaucratic bullshit. I had spent a good chunk of my Sunday to get to this moment. Carrying my helmet everywhere I went. Fiddling with the key in my jacket pocket. Imagining the engine sound, the smell of fuel-rich exhaust, the exhilaration of twisting the throttle, dropping the clutch and accelerating quicker than my mind could comprehend. Buoyed by these images, I strode through the impound lot. Smiling. Anticipating. Key in hand, ready to ride a celebratory wheelie all… the way… home.

Except, wait a second. Where’s the keyhole in the ignition?

The thief made a few modifications to my bike.

Keyless ignition!

The thief removed the ignition. There’s no place to insert my key. Now if I want to ride, I just flip the kill switch and hit the starter.

Passenger pegs!

I don’t have a picture of these yet, but the pegs were attached with wood screws.

Aftermarket gas cap!

The original gas cap had a key lock. The thief did not have a key. Improvisation ensued.

Broken turn signals!

Fender eliminator kit!

(The thief hacksawed off part of the fender and moved the license plate up.)

Other minor changed included removing the tool pouch, removing all reflectors, and adding 500 miles to the odometer.

This morning I noticed one more thing…

The thief wrecked my bike and cracked part of the engine case. He “fixed” it with some JB weld. It leaks oil when running.

Last I heard from the cops, they have a suspect and he is in jail. I guess that’s a happy ending for everyone involved. Except the thief, who is in jail. And me, since I’m out $200 and my bike is fucked.

Many of my friends, family, and coworkers know this already. If you didn’t know, you’re probably wondering, “Why?”

The decision to pursue cryonics really comes down to answering two questions: “How likely is it that cryonics will work?” and “Is life extension desirable and moral?” Others have answered the second question better than I can, so I’ll just talk about the current feasibility.

Accurately evaluating cryonics requires knowledge scattered across many disciplines. Even so, the basic principle can be understood by anyone: Frozen meat lasts a long time.

I first heard about cryonics in high school. I knew that water expanded when frozen, and wrote off cryonics as a crackpot idea that would never work. It was obvious: ice crystals damaged cells irreparably. There was no way anyone could revive a frozen body. Case closed.

Until early 2009, when I stumbled on Overcoming Bias. A couple of posts discussed cryonics, and caused me to reconsider it. After seeing electron micrographs of cryopreserved brain tissue, I read more. I realized the idea wasn’t so crazy.

This post on Less Wrong got me to start the paperwork. Signing up is rather involved. Major steps include becoming a member of a cryonics provider, securing funding (usually a life insurance policy), and notarizing and witnessing paperwork. For me, the whole process took several months.

While Eliezer Yudkowsky’s post convinced me to sign up for cryonics, the most persuasive writing in the world won’t convince reality that cryonics works. Cryonics either works or it doesn’t. Finding the answer requires examining evidence. So, bring on the evidence!

This presentation by Brian Wowk does a good job showing the state of cryopreservation technology:

For those who don’t want to spend 25 minutes watching the video, the main points are:

• The common conception of freezing damage is backwards. Ice doesn’t cause cells to expand and burst. Ice crystals form between cells, dehydrating and squishing cells as the crystals grow.
• Cryoprotectants can limit or even eliminate ice formation at liquid nitrogen temperatures. Instead of freezing, the solution increases in viscosity as temperature drops. Around -120ºC, the solution reaches its glass transition temperature and becomes an amorphous solid.
• Cryoprotectant research is ongoing. A decade ago, cryoprotectants needed cooling/warming rates of 70ºC per second to prevent ice formation. Cell survival rates were around 50% in tissue samples. Today’s cryoprotectants can revive 90% of cells with cooling/warming rates of less than 1ºC per second.
• Wowk’s team used their current best cryoprotectant (M22) to successfully cryopreserve, thaw, and transplant rabbit kidneys.
• Based on evidence from victims of hypothermia, barbituate overdose, and cardiac arrest, silencing electrical activity in the brain doesn’t erase any long-term memories. Cryopreserving the brain should preserve all aspects of self.
• SEM micrographs of cryopreserved brain tissue show fine structure is preserved. Individual axons are unharmed. Even synaptic vesicles are preserved.
• Cooling cryoprotectant solution too far below the glass transition temperature will cause fracturing from thermal stresses. While they make revival harder, the fractures are few in number and typically cause simple translations of tissue. The information destroyed is negligible.
• Despite limited funding, cryopreservation techniques have improved substantially. Cryopreservation and transplantation of large organs is expected in 10 years.

Alcor uses M22 to cryopreserve the brains of their customers. The Cryonics Institute uses a similar solution. If Alcor or CI gets to you in time, the structure of your brain will be accurately preserved down to the nanometer scale. Everything we know about chemistry, biology, and neuroscience tells us that this process preserves all the information required to reconstruct a human mind.

The biggest unknown in cryonics is the revival process. Right now it’s completely speculative. People have thrown some ideas around, but all require major advances in current technologies. Some even require inventing new technologies (such as molecular nanotech). So today, a few unaugmented, 21st-century, biological humans have given it some thought but haven’t built revival technology. But the information in cryopreserved brains will wait patiently for a solution. Cryonics organizations pay for indefinite maintenance with the interest from low-risk investments. Failure requires that no future intelligence ever finds a way to revive people.

Based on this evidence, I’m very sure the preservation process works. My doubts are about the future. I assign a low probability to being revived: around 5%, since a lot of things have to go right. The biggest thing being that humanity has to not destroy itself. While 5% is a small likelihood, it’s still worth paying for. Any civilization with the means and desire to revive people will be a very nice place to live. And members of that civilization would live a very long time. Quality of life would probably be better than most current depictions of heaven. I am not kidding.

When I told people, reactions varied. My mother laughed. My father and siblings thought it was weird and sort of interesting. I don’t think any of them took it seriously. I guess I’m too timid to say, “It’s your funeral.”

Well, too timid to say it face-to-face.

I’m 5’6″, so the bike is pretty tall for me. I can barely touch my toes to the pavement when I straddle it. But that height comes in handy; I can easily see over cars in traffic. In fact, I’m pretty sure I could ride over the roofs of those cars. The ridiculous suspension lets me ride over curbs and baby strollers without worries. It’s practical as well. Insurance is cheap. It’s slower (and probably safer) than my SV-650/S. It’s ugly in an endearing way. And it doesn’t have expensive fairings to fix after it falls.

Finally, it can do this:

OK, maybe it’s not safer.

Bug #11334: MASTER Copy-Paste doesn’t work if the source is closed before the paste (2004)

I don’t know why this one has stuck around for so long. Fortunately, most of apps have been updated to work around the bug, but there is no good excuse for having clipboard functionality broken in a modern OS.

Bug #20284: Keep Desktop icons automatically arranged (2005)
Upstream: Bug 172128 – Autosorting desktop icons
Related: Bug #12608: ‘keep aligned’ does not work right

If you delete or add icons (for example, by inserting or removing a USB drive), the desktop icons will not stay sorted. New icons will often overlap others. Deleted icons will leave empty spaces on the desktop. There is an option to align icons to a grid, but the grid is smaller than the icons, so icons still overlap.

Bug #22007: no ‘Settings’ button in gnome-screensaver (2005)
Upstream: Bug 316654 – no ability to configure the different screensavers

Have you ever wanted to change the text of GLText? It used to be as easy as selecting that screensaver and clicking on the settings button. Now there is no way to change screensaver settings short of manually editing some obscure XML files. The comment by the gnome-screensaver developer is particularly infuriating: Resolution: WONTFIX I don’t have any plans to support this. My view is that any screensaver theme that requires configuration is inherently broken.

Bug #10550: DVDs with restrictive permissions are unreadable for normal user (late 2004)

This one is especially silly, considering a fix probably involves passing some extra params to mount.

Bug #16454: Sound still comes out of speakers when PCM volume is set to 0 (2005)

What? A bug in ALSA? What a surprise! I think all ALSA and PulseAudio developers are deaf, because they certainly haven’t heard users screaming about Linux sound issues for the past decade.

Bug #18666: Lots of seemingly redundant fonts (2005)

*sigh*

What do these issues have in common? All of them are old, so chances are good they’ll stick around in future Ubuntu releases. A decent portion of them are caused by Gnome upstream. Most are rationalized as not worth fixing because they are rarely encountered by typical users. Another common excuse is that each annoyance is minor. But an OS with a lot of small annoyances makes for a terrible user experience. The chances of a user getting annoyed in a given time period are much higher in Ubuntu than in an OS with a few huge bugs. Lots of little bugs make expectation management impossible. New users can’t learn about all the annoyances before they try the product. If there were only a couple of large deficiencies, users could learn about them beforehand and decide not to use the product. At the very least, their expectations of the product would be closer to reality, and the short list of deficiencies would be easier to remember and work around.

I’ve been bit by every single bug I’ve mentioned. But those are only the oldest, most common annoyances. I’m not including newer stuff like Nvidia drivers breaking in 10.04, or static in the right channel of my headphone jack after upgrading to 10.04, or having to screw with fstab, xorg.conf, and building my own .deb files just to get my system to boot again.

There’s a reason Ubuntu is free: Nobody in their right mind would pay money for it.

With some help from friends (Ben Lowery and Bjorn Tipling), I cleaned up the styling and made sure that the credit card details were never sent across the wire.

I submitted the link to Hacker News, where it was rather well-received. Everyone got the joke. Some even offered suggestions to improve the text of the warning message.

The site quickly became popular. People on Twitter talked about it.

That’s when things started going south. A decent number of people didn’t get the joke. They thought it was a real phishing site. Among them was this guy, supposedly a researcher at antivirus company Trend Micro. Google Analytics shows about half of the visitors actually clicked the submit button. I’m guessing the other half didn’t see the “This was a test. You failed.” message and assumed it was a phishing site.)

I normally don’t care about idiots on the Internet, but enough Firefox users reported it as a phishing site that it got blacklisted. Now if you try to visit it in any modern browser, you’ll get a giant warning. Even worse, sites like Twitter use the Firefox phishing blacklist to filter links, so nobody can link to the site now.

In a final bit of ridiculousness, UK Yahoo News reported that the Anti-Phishing Working Group was responsible for creating ismycreditcardstolen.com. This misunderstanding wouldn’t have happened if the reporter read the about page.

What have I learned from this?

• People on Twitter are dumber than I thought.
• A small group can add any relatively-unpopular site to the Firefox phishing blacklist.
• Some “reporter” couldn’t be troubled to click the about link on the front page of the site he wrote an article about.

Quite a few people have reported the site as being incorrectly flagged, but it hasn’t done any good. I doubt it will ever be removed from the phishing blacklist. Oh well, it was fun while it lasted. And since Paul bought the domain I’m not out any money.