Geoff's Site

A Primer on Programmer IRC Etiquette

2012-05-19T18:12:50-07:00

IRC is a great way to get hands-on help when learning a new language or framework. Experts in tech-related IRC channels are amazing. In the real world, they could charge insane rates for consulting, but they give advice freely on IRC and mailing lists. That said, it’s important to note that IRC veterans find some common newbie behaviors very annoying. Getting help from these experts can be frustrating for a newbie, but the rewards are great.

If you’re new to IRC, here is a short list of guidelines:

Before asking for help, use Google and read the docs. People will get annoyed if you ask questions that are answered in documentation. It shows you didn’t try very hard to find the answer before wasting their time.
Don’t ask to ask, just ask.
Explain the problem in detail. Saying, “It doesn’t work.” will earn you the ire of everyone in the channel. How doesn’t it work? What error messages do you see? What do the logs say? Be specific. People need to know the answers to these questions so they can help. Mention what you think should happen and what is actually happening.
Gist any error logs or source code you have a problem with. Pasting large chunks of text in IRC is frowned upon. It’s called flooding, and it makes conversation difficult for others. Some channels will kick you for flooding. If your IRC client doesn’t rate-limit, the server can even disconnect you.
Explain what you’re trying to do. Often there’s an easier way to get what you want out of a language or framework. Others might know of a useful module or library that does much of the heavy lifting for you.
Finally, don’t be a dick. Don’t call a project stupid. Every piece of software has stupid parts. Granted, some have more stupidity than others, but name-calling isn’t going to help fix the problems. More importantly, name-calling won’t get you help. It will probably get you kick-banned.

Following these rules will make life much more pleasant for everyone involved. You’ll be more likely to get your problem solved and the channel regulars will have less stress in their lives.

I was motivated to write this after an inexperienced person joined #node.js on Freenode. I failed to find a similar rule list that I read years before. I thought about linking to ESR’s How to Ask Questions the Smart Way, but it’s too general, too long, and insults the reader. Now that I’ve written this, I’m ready for next time.

Chile Trip

2012-03-31T20:58:46-07:00

I went with Paul and Justin to Chile and Argentina. Most of the trip consisted of touring wineries, eating gourmet meals, and staying in fancy hotels. Even Hedonismbot would blush at this past week.

Work Patterns on a Plane

2012-03-22T21:01:17-07:00

I’ve found that air travel provides a good opportunity for me to get stuff done. The isolation[1] helps me avoid distractions and forces my mind to stay focused on the job at hand. I’m also buoyed by the knowledge that I will have minimal interruptions for the duration of the flight.

That said, there are some caveats.

Having a small laptop is crucial. Large laptops won’t fit on your lap if someone reclines their seat.

And I can’t do just any sort of work on a plane. Anything requiring Internet access is right-out. Learning a new programming language isn’t going to happen. There are just too many tools I’d have to download and newbie error messages that I’d have to google. Writing something new in Twisted Python is fine, since I know the language and I can save all the necessary documentation beforehand. Writing C is also fine, since most of its documentation is in manpages. And anything I can’t figure out from docs I can usually figure out with science.

Another issue: If I travel with any sort of shallow content, I end up wasting time (and battery life) watching movies instead of coding.

Speaking of battery life: it’s a bit of a problem. My 11” Air lasts 2-7 hours depending on what I’m doing[2]. That’s good enough for most domestic flights, but the poor thing didn’t stand a chance on my recent Japan trip. If my Air dies, I switch to my backup netbook. It’s not as functional, but the spare battery can give it over 12 hours of run-time.

I got a ton of work done on The Silver Searcher while flying. I managed to add support for pipes, refactor the searching code, clean up some particularly ugly printing code, and fix some bugs related to printing matches. In total I changed over 1,000 lines on the trans-pacific flights. I accomplished more in those 20 hours than I did in the month previous. I also felt an unusually large sense of accomplishment.

As Russell says, “If you want to find time to hack on a side project, get on an airplane.”

Lack of Internet access, phone, or friends nearby.
I realize this is a huge range. The CPU and screen seem to be the biggest power hogs. If I don’t pay any attention to power consumption and just run the screen bright, the battery will last about 4 hours. Keeping brightness low and CPU usage to a minimum can almost double that.

Japan Trip

2012-03-19T04:54:22-07:00

I was close to my earned-time-off cap, so I took the month of March off. The first half of the month consisted of a trip to Japan with Paul, Shawn, Brad, and Brad’s wife Caity. It was quite enjoyable. We managed to have a pretty good balance of calm time and party time. I also paid off some of my sleep debt.

Photos are here. Apologies for the terrible gallery software. Replacing it is on my todo list.

My favorite part of the entire trip was going to the 52nd floor of the Park Hyatt in Tokyo. We drank expensive cocktails while admiring the Blade Runner-esque view. There was a magnitude 6.1 earthquake right after we ordered drinks. I knew we were perfectly safe, but some primitive part of my brain panicked and released copious quantities of adrenaline. The building took a long time to stop swaying.

A close second was running in the hills above Kyoto as snow fell. The dusting of snow over the trees made for beautiful scenery.

I stopped in San Francisco for the weekend (and my birthday, also known as St. Patrick’s Day). Now it’s off to Chile and Argentina until the end of the month.

From Wordpress to Jekyll

2012-02-21T09:56:46-08:00

I rewrote my site in Jekyll this weekend. Apologies for the extra items in the RSS feed. Unfortunately I misconfigured the site URL, which changed every post ID, which made RSS readers think I had a bunch of new posts.

I’m still going back through old posts to fix formatting errors. Also, comments will come back at some point in the future.

I hope you like the new design. It’s a blatant rip-off of mnmlist.

Edit: Everyone’s comments are back.

Profiling with Gprof

2012-02-08T02:36:31-08:00

I said I’d post about gprof, so here goes.

Valgrind and gprof are two very different tools. Valgrind is an instrumenting profiler. Gprof is a sampling profiler. Gprof spends most of its time doing nothing. Then every 100,000,000 clock cycles or so, it looks at the instruction pointer to see what function your program is in. It collects that data enough times to end up with a good idea of where your program is spending its time. The advantage of this approach is that your program runs almost at full speed. This gives you a better idea of how much time your program spends waiting for things like disk or network I/O.

My typical profiling experience with gprof looks like this:

(sets CFLAGS=-pg in Makefile.am)
$ make clean && ./build.sh
(snip)
$ time ./ag --literal abcdefghijklmnopqrstuvwxyz ../ | wc -l
271

real    0m1.144s
user    0m0.792s
sys     0m0.340s

$ gprof -bp ag gmon.out 
Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  ms/call  ms/call  name    
 36.09      0.22     0.22    20953     0.01     0.01  is_binary
 32.81      0.42     0.20    16567     0.01     0.01  boyer_moore_strnstr
 13.12      0.50     0.08       63     1.27     1.27  print_file_matches
 11.48      0.57     0.07                             filename_filter
  3.28      0.59     0.02    26377     0.00     0.00  strlcpy
  1.64      0.60     0.01    52754     0.00     0.00  strlcat
  1.64      0.61     0.01        1    10.01   540.36  search_dir
  0.00      0.61     0.00    40160     0.00     0.00  log_debug
  0.00      0.61     0.00    40160     0.00     0.00  vplog
  0.00      0.61     0.00      213     0.00     0.00  add_ignore_pattern
  0.00      0.61     0.00       63     0.00     0.00  print_path
  0.00      0.61     0.00       16     0.00     0.00  load_ignore_patterns
  0.00      0.61     0.00        1     0.00     0.00  cleanup_ignore_patterns
  0.00      0.61     0.00        1     0.00     0.00  generate_skip_lookup
  0.00      0.61     0.00        1     0.00     0.00  init_options
  0.00      0.61     0.00        1     0.00     0.00  parse_options
  0.00      0.61     0.00        1     0.00     0.00  set_log_level

Some caveats: for gprof to work, you need to add -pg to your CFLAGS. Also, gprof is broken on OS X, so run it on a linux server. If you want a sampling profiler on OS X, I recommend Instruments.app.

My Twisted Hack Day Project: Why is the Reactor Pausing?

2012-02-04T23:04:44-08:00

Last week we had a twisted hack day at work. The project I work on has over a dozen twisted services, so this was right up my alley. I knew a couple of services were doing dumb things (like DB calls in the main thread) but nobody had gotten around to fixing the issues.[1] It’s pretty easy to find most instances of Django calls in the main thread, but there are many other ways to hang the reactor. I wanted to find every instance of pausing, so that’s what I decided to make for my hack day project.

Before the hack day, Jean-Paul Calderone came to the office and gave some talks about twisted. I asked him if there were any tools for finding the cause of reactor pauses. He said he’d built his own little script a while back, and that the key was to use SIGALRM. That was enough to get me on the right track.

Here’s the result. It’s a pretty simple script. Every 100 milliseconds, it cancels any pending SIGALRM and calls setitimer, so that the OS will send SIGALRM to the process in 500ms. It also has a handler for SIGALRM that logs a traceback and adds the offending function to a stats dict.[2]

If the reactor is paused for more than 500ms, the pending SIGALRM won’t be cancelled, so it will be sent to the process. Then the handler will print out the traceback and update stats. Pretty handy.

My tool can give false positives on a heavily-loaded system. This is because I’m calling setitimer with ITIMER_REAL instead of ITIMER_VIRTUAL. Using virtual time won’t catch stuff like sleep()s in the main thread, since sleeping doesn’t count toward execution time. Using real time will fire the SIGALRM after 0.5 seconds even if the process has gotten zero time on the CPU. I got a few false positives on my overloaded VM, but this turned out to be beneficial. The tracebacks went all the way to select(). I mentioned this fact to Paul and he reminded me that we should be using the epoll reactor.

That’s two production issues identified and fixed because of a silly hack day project. Booyah.[3]

We got around performance issues by running multiple instances of these services. The problems were never fixed because they were in really old code that was written before anyone at Cloudkick knew how to write twisted.
A stats defaultdict, to be precise. If you haven’t used defaultdict before, check it out. It will save you from writing some dumb boilerplate code.
I wasn’t the only person to do a hack day project. Here’s the full list.

Programming: We can do Science

2012-01-30T08:11:28-08:00

Let’s say you’re messing with some Python and you forget how the global keyword changes scoping[1]. To find the answer you might try…

Google
Stack Overflow
Actually reading the docs
Asking someone who knows more about it than you

…usually in that order. That’s not a terrible way of going about it, but there’s an oft-overlooked option. Instead of scouring documentation or bugging a coworker, sometimes it’s best to run experiments.

This code…

#!/usr/bin/env python

blah = 1

def foo():
    blah = 0
    print "foo:blah", blah

print "global:blah", blah
foo()
print "global:blah", blah

…prints…

global:blah 1
foo:blah 0
global:blah 1

While adding a global in foo…

#!/usr/bin/env python

blah = 1

def foo():
    global blah
    blah = 0
    print "foo:blah", blah

print "global:blah", blah
foo()
print "global:blah", blah

…prints…

global:blah 1
foo:blah 0
global:blah 0

Without global, foo’s blah is not the same blah. With global, it is.

These sorts of code blurbs are great for learning. They’re also great for solving disagreements about code behavior. Warning: You will be wrong sometimes. You will look stupid in front of your peers. On the bright side, these embarrassing moments will stick in your mind. You won’t forget what you were wrong about.

Code experiments can be much trickier, and much more useful, than above. For example…

…at work we recently switched a project’s MySQL engine from MyISAM to InnoDB. After the switch, we encountered some weird errors. A process would save an object to the DB, but some services couldn’t find the newly-created object[2]. I had a hunch that transactions were responsible for the weirdness; my reasoning being that MyISAM lacks transaction support, and it had worked fine.

So I did science. I opened up two MySQL consoles. In console #1, I ran start transaction;. Then in console #2, I ran:

mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)

mysql> insert into inventory_nodeaddress (node_id, ip, ip_version, type) values (NULL, '31.22.190.54', 4, 0);
Query OK, 1 row affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from inventory_nodeaddress where node_id is NULL;
+-----+---------+-------------------------------------+------------+------+
| id  | node_id | ip                                  | ip_version | type |
+-----+---------+-------------------------------------+------------+------+
| 106 |    NULL | 50.57.96.184                        |          4 |    0 |
| 107 |    NULL | 10.182.67.171                       |          4 |    1 |
| 147 |    NULL | 31.22.190.54                        |          4 |    0 |
+-----+---------+-------------------------------------+------------+------+
3 rows in set (0.00 sec)

mysql>

OK, the data’s committed. I even double-checked that it was there by selecting it. Everything is fine, right?

Nope. Back in console #1, I ran:

mysql> select * from inventory_nodeaddress where node_id is NULL;
+-----+---------+-------------------------------------+------------+------+
| id  | node_id | ip                                  | ip_version | type |
+-----+---------+-------------------------------------+------------+------+
| 106 |    NULL | 50.57.96.184                        |          4 |    0 |
| 107 |    NULL | 10.182.67.171                       |          4 |    1 |
+-----+---------+-------------------------------------+------------+------+
2 rows in set (0.00 sec)

mysql>

Once I ended the transaction in console #1 (either through a rollback or a commit), the new row showed up in selects. After some Googling I finally found the relevant MySQL documentation. Once I changed the transaction isolation from REPEATABLE-READ to READ-COMMITTED, selects inside transactions showed recently-inserted rows.

The experiment plus Brad’s knowledge of Django helped solve the mystery. Django only runs commit when writing to the DB. This sucks for any long-running service that never writes. The service will start up, connect to MySQL, run start transaction and do selects without ever ending the transaction. With the default InnoDB configuration, these services will see an ever-older version of the database. Not fun.

After I added transaction-isolation = READ-COMMITTED to the my.cnf chef template, everything worked swimmingly. Hooray for science.

The next time you’re stumped, try some experiments. As a programmer, you have immense power over the program’s universe. Your code runs on a perfectly deterministic machine[3]. With the right software tools, you can stop time. You can read or change any part of memory. You can step through, one line at a time, to see exactly what’s happening.

Of course, this isn’t real science. These apparati make programming a cakewalk compared to real science.

You forgot this fact not because you suck at Python, but because you usually write clean code with no globals. At least, that’s what you keep telling yourself.
Just so nobody freaks out: This thing is non-customer-facing and currently under heavy development. I’m also over-simplifying the process. The actual changes happened in a development branch and weren’t merged until things were hunky-dory.
Cosmic rays notwithstanding.

Making Ag Faster: Profiling with Valgrind

2012-01-23T10:47:36-08:00

These days, a lot of software is written to be “fast enough”. Since code bases can be very large, there’s no such thing as “fast enough” for The Silver Searcher. In fact, my main goal with Ag is speed.

Improving performance is not always easy, but it is simple:

Find the slowest part of the program.
Make that part faster.
Repeat until it’s fast enough or you go insane.

There are lots of profiling tools and programmers often argue about which is the best. I use gprof, callgrind, and Instruments.app. Which profiler you use doesn’t matter as much as actually using one. They all have their advantages and disadvantages, but for this post I’ll only cover Valgrind’s callgrind. Using callgrind doesn’t require special compilation. Just invoke it with your program’s name and it will generate profiling data for callgrind_annotate to analyze.

Here’s a typical profiling run for Ag:

$ make clean && ./build.sh
(snip)
$ time valgrind --tool=callgrind --dsymutil=yes ./ag --literal abcdefghijklmnopqrstuvwxyz ../
(snip)
real	1m34.709s
user	1m33.206s
sys	0m1.492s
$ callgrind_annotate --auto=yes callgrind.out.10361 
--------------------------------------------------------------------------------
Profile data file 'callgrind.out.10361' (creator: callgrind-3.6.1-Debian)
--------------------------------------------------------------------------------
I1 cache: 
D1 cache: 
LL cache: 
Timerange: Basic block 0 - 798409857
Trigger: Program termination
Profiled target:  ./ag --literal abcdefghijklmnopqrstuvwxyz ../ (PID 10361, part 1)
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       99
Include dirs:     
User annotated:   
Auto-annotation:  on

--------------------------------------------------------------------------------
           Ir 
--------------------------------------------------------------------------------
3,068,387,924  PROGRAM TOTALS

--------------------------------------------------------------------------------
           Ir  file:function
--------------------------------------------------------------------------------
1,764,541,095  src/util.c:ag_strnstr [/home/geoff/code/the_silver_searcher/ag]
  386,020,821  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch [/lib/x86_64-linux-gnu/libc-2.13.so]
  226,548,868  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strcmp.S:__GI_strncmp [/lib/x86_64-linux-gnu/libc-2.13.so]
  181,861,517  src/util.c:is_binary [/home/geoff/code/the_silver_searcher/ag]
  123,211,270  /build/buildd/eglibc-2.13/posix/fnmatch.c:fnmatch@@GLIBC_2.2.5 [/lib/x86_64-linux-gnu/libc-2.13.so]
  104,867,805  src/print.c:print_file_matches [/home/geoff/code/the_silver_searcher/ag]
   77,058,570  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strlen.S:__GI_strlen [/lib/x86_64-linux-gnu/libc-2.13.so]
   60,030,629  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch'2 [/lib/x86_64-linux-gnu/libc-2.13.so]
   44,019,376  src/ignore.c:filename_filter [/home/geoff/code/the_silver_searcher/ag]
   27,072,821  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/memchr.S:memchr [/lib/x86_64-linux-gnu/libc-2.13.so]
    9,329,984  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strcmp.S:__GI_strcmp [/lib/x86_64-linux-gnu/libc-2.13.so]
    7,803,075  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
    7,040,644  /build/buildd/eglibc-2.13/posix/../locale/weight.h:internal_fnmatch
    6,062,124  /build/buildd/eglibc-2.13/string/../string/memmove.c:__GI_memmove [/lib/x86_64-linux-gnu/libc-2.13.so]
    4,384,383  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../memcpy.S:__GI_memcpy [/lib/x86_64-linux-gnu/libc-2.13.so]
    3,951,640  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_free [/lib/x86_64-linux-gnu/libc-2.13.so]
    3,779,300  /build/buildd/eglibc-2.13/dirent/../sysdeps/unix/readdir.c:readdir [/lib/x86_64-linux-gnu/libc-2.13.so]
    3,181,118  /build/buildd/eglibc-2.13/malloc/malloc.c:malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
(snip)

I snipped out the annotated source code. You can see the full output here.

This profiling info tells me that I’m spending all my time in strnstr(). I did some research on string-matching and found out about the Boyer-Moore algorithm. After some more reading, I decided to go with a simplified version of Boyer-Moore called Boyer-Moore-Horspool.

Here’s the data after I implemented Boyer-Moore-Horspool strstr:

$ time valgrind --tool=callgrind ./ag --literal abcdefghijklmnopqrstuvwxyz ../
real	0m32.429s
user	0m31.034s
sys	0m1.324s
$ callgrind_annotate --auto=yes callgrind.out.11921
--------------------------------------------------------------------------------
Profile data file 'callgrind.out.11921' (creator: callgrind-3.6.1-Debian)
--------------------------------------------------------------------------------
I1 cache: 
D1 cache: 
LL cache: 
Timerange: Basic block 0 - 228181262
Trigger: Program termination
Profiled target:  ./ag --literal abcdefghijklmnopqrstuvwxyz ../ (PID 11921, part 1)
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       99
Include dirs:     
User annotated:   
Auto-annotation:  on

--------------------------------------------------------------------------------
           Ir 
--------------------------------------------------------------------------------
1,139,437,344  PROGRAM TOTALS

--------------------------------------------------------------------------------
         Ir  file:function
--------------------------------------------------------------------------------
386,014,011  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch [/lib/x86_64-linux-gnu/libc-2.13.so]
181,870,097  src/util.c:is_binary [/home/geoff/code/the_silver_searcher/ag]
123,209,345  /build/buildd/eglibc-2.13/posix/fnmatch.c:fnmatch@@GLIBC_2.2.5 [/lib/x86_64-linux-gnu/libc-2.13.so]
104,867,805  src/print.c:print_file_matches [/home/geoff/code/the_silver_searcher/ag]
 76,747,163  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strlen.S:__GI_strlen [/lib/x86_64-linux-gnu/libc-2.13.so]
 63,421,170  src/util.c:boyer_moore_strnstr [/home/geoff/code/the_silver_searcher/ag]
 60,028,609  /build/buildd/eglibc-2.13/posix/fnmatch_loop.c:internal_fnmatch'2 [/lib/x86_64-linux-gnu/libc-2.13.so]
 44,018,667  src/ignore.c:filename_filter [/home/geoff/code/the_silver_searcher/ag]
 27,072,637  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/memchr.S:memchr [/lib/x86_64-linux-gnu/libc-2.13.so]
  8,312,570  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../strcmp.S:__GI_strcmp [/lib/x86_64-linux-gnu/libc-2.13.so]
  7,803,075  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
  7,040,534  /build/buildd/eglibc-2.13/posix/../locale/weight.h:internal_fnmatch
  6,061,868  /build/buildd/eglibc-2.13/string/../string/memmove.c:__GI_memmove [/lib/x86_64-linux-gnu/libc-2.13.so]
  4,384,383  /build/buildd/eglibc-2.13/string/../sysdeps/x86_64/multiarch/../memcpy.S:__GI_memcpy [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,951,640  /build/buildd/eglibc-2.13/malloc/malloc.c:_int_free [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,779,220  /build/buildd/eglibc-2.13/dirent/../sysdeps/unix/readdir.c:readdir [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,181,118  /build/buildd/eglibc-2.13/malloc/malloc.c:malloc [/lib/x86_64-linux-gnu/libc-2.13.so]
  3,089,135  src/main.c:search_dir'2 [/home/geoff/code/the_silver_searcher/ag]
  2,095,514  /build/buildd/eglibc-2.13/malloc/malloc.c:free [/lib/x86_64-linux-gnu/libc-2.13.so]
  2,018,298  /build/buildd/eglibc-2.13/dirent/../sysdeps/wordsize-64/../../dirent/scandir.c:scandir [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,941,992  /build/buildd/eglibc-2.13/string/strcoll_l.c:strcoll_l [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,889,859  /build/buildd/eglibc-2.13/stdlib/msort.c:msort_with_tmp.part.0'2 [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,704,553  /build/buildd/eglibc-2.13/malloc/malloc.c:malloc_consolidate.part.3 [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,644,688  src/ignore.c:ignorefile_filter [/home/geoff/code/the_silver_searcher/ag]
  1,601,628  /build/buildd/eglibc-2.13/dirent/../sysdeps/unix/sysv/linux/getdents.c:__getdents [/lib/x86_64-linux-gnu/libc-2.13.so]
  1,582,620  src/util.c:strlcat [/home/geoff/code/the_silver_searcher/ag]
(snip)

For the curious, full output of callgrind_annotate is here.

That’s a 3x overall speedup and a 27x speedup in string matching. Impressive! Now Ag is spending most of the time figuring out whether or not it should search a file. It’s clear where I need to optimize next.

Valgrind isn’t perfect though. It makes programs run 25-50x slower than they normally would, so you won’t notice if you’re spending all your time waiting for network or disk I/O. In the case of Ag, this turned into a 20% performance improvement in my benchmarks.

Getting more useful data requires switching from an instrumenting profiler to a sampling profiler. Both Instruments.app and gprof are sampling profilers, but this post is already too long. I’ll cover them some other time.

Building for Others

2012-01-20T08:14:57-08:00

I like to write new code. Unfortunately, writing new code is only a small part of building something useful. The Pareto principle applies. Once you’ve written 80% of the code in a short, fun-filled period, you spend much longer finishing up little things. You have to debug some odd edge case, clean up messy stuff that mostly works, and get it to build and run on some obscure Linux distribution.

Worst of all, you have to write documentation.

This stuff isn’t fun, but it’s necessary if you want others to use your project.

Because you made it, the various dependencies and quirks are obvious to you. For the poor soul who clones your repository, the same is not true. Even compiling is a challenge for a newbie. What build system does your project use? Make? Scons? Ant? What build dependencies does it have? Does it check for them or print out a useful error message if any are missing? Is there a helpful README?

If you want other people to use (and possibly one day improve) your work, you need to polish the build scripts and write documentation. Think of it like a sales funnel. Perhaps 100 people download your code, 80 get it to build, 75 run it, 50 use it regularly, 5 make modifications, and finally, 2 contribute patches back. You can increase the numbers. Making those steps easier will grow your user base and contributions.

So that’s what I’ve done with Ag. Over the past couple of weeks I’ve added a man page, a wiki, accepted pull requests to clean up the build, and even improved the –help output.

I’ll be the first to admit that fixing trivial inconveniences is boring. But frequently, boring work is the difference between a personal project and a community of users. Too often, trivial inconveniences stop projects from reaching critical mass. So if you want to build something for others, grit those teeth and go schlep.

The Silver Searcher: Better than Ack

2011-12-27T16:36:40-08:00

A lot of my time spent “writing” code is actually spent reading code. And a decent chunk of my time spent reading code is actually spent searching code. Lately I’ve started working with a larger codebase.[1] Both grep and ack take a non-negligible amount of time to search it. Both are slow, but for different reasons. Grep is fast, but doesn’t ignore files.[2] Ack is very good at ignoring files, but it’s written in Perl instead of C. What I really want is something that’s fast and ignores files.

So I built it. I call it The Silver Searcher, or Ag) for short. Ag is like ack, but better. It’s fast. It’s damn fast. The only thing faster is stuff that builds indicies beforehand, like Exuberant Ctags.

Don’t believe me? Here are some benchmarks. I ran them multiple times and grabbed the median for each.

ggreer@carbon:~/cloudkick/reach% du -sh
250M	.

ggreer@carbon:~% time grep -r -i SOLR ~/cloudkick/reach | wc -l
     617
11.06s user 0.81s system 96% cpu 12.261 total

ggreer@carbon:~% time ack -i SOLR ~/cloudkick/reach | wc -l
     488
2.87s user 0.78s system 97% cpu 3.750 total

ggreer@carbon:~% time ag -i SOLR ~/cloudkick/reach | wc -l
     573
1.00s user 0.51s system 95% cpu 1.587 total

Here’s Ag with some extra ignores, similar to how ack ignores many files by default:

ggreer@carbon:~% cat ~/cloudkick/reach/.agignore 
extern
release
fixtures
ggreer@carbon:~% time ag -i SOLR ~/cloudkick/reach | wc -l
     499
0.35s user 0.15s system 94% cpu 0.528 total

That’s the same as git grep:

ggreer@carbon:~/cloudkick/reach% time git grep -i SOLR ~/cloudkick/reach | wc -l
     489
0.32s user 0.58s system 161% cpu 0.556 total

…except git grep only works in git repos. And it doesn’t ignore stuff in the repository like extern or generated files.[3]

The bottom line: Grep’s output was the least useful. It dutifully reported matches in .pyc files and other things I don’t care about. Ack’s results were better and faster than grep. Ag had more results than ack, but took half as long. With a couple of clever ignores (like the extern directory), Ag took a mere half-second and gave even more pertinent results.

I can already hear someone saying, “Big deal. It’s only a second faster. What does one second matter when searching an entire codebase?” My reply: trivial inconveniences matter. Using Ag is like having a faster computer; you don’t realize how slow things were until you’ve experienced fast. The difference is big enough that I can’t go back to ack, just like ack users can’t go back to grep.

Since it behaves like ack, Ag can be used by many fancy ack GUI front-ends. This makes searching convenient as well as fast. After I got Ag sorta-working, I forked AckMate so that I could use Ag in my favorite editor. My fork bundles both AckMate’s Ack and my own Ag. You can switch between them with a simple check box. The tmbundle is on the downloads page. Be warned: it replaces your current AckMate.

There’s still plenty of stuff I want to add,[4] but it’s good enough for my own daily use so I figured I should tell others about it. And of course, patches are welcome!

The decision was made to put all python dependencies into extern/ instead of using pip. A good call, in my opinion.
At least not without a bunch of pipes and find and xargs. Yes I know there are aliases but it’s annoying to keep those up-to-date.
Yes, I know it’s bad form to put generated files in revision control.
Ctags support, for one. Also inverted matching, accepting piped input, and basic stuff like retrying a search with fewer ignores and no case-sensitivity.

The Moral Trajectory

2011-12-12T00:20:11-08:00

Apologies in advance. This post is about philosophy and morality; way out of my area of expertise. You have been warned.

By our standards, ancient Rome was not a nice place. I’m not just talking about problems due to inferior technology or sanitation or disease. Even a hypothetical ancient Rome that fixed those things would be worse than the modern world. Slaves would still exist. Racism and discrimination would still be rampant. The citizenship of one’s parents would still determine your lot in life. Simply put, ancient Roman morality was worse than ours.

It is my claim that over time, most civilizations have moved along a moral trajectory. Slowly -ever so slowly- people have become nicer. We’ve recognized more humans as worthy of moral value. Eventually we decided slavery was wrong. We began to treat women and men equally. Currently we’re moving toward treating homosexuals and heterosexuals equally.

The fact that morals have improved over time should strike people as extraordinary. Think about what must have happened for slavery to go from right to wrong. At some point, a person went from thinking “slavery is right” to thinking “slavery is wrong”, without initially wanting to change his mind! I wish I knew how to trigger that sort of thinking. It’s as valuable as vaccines.

It’s great that people have become more moral over the millennia. Now what are the odds that after thousands of years, we suddenly got morality right? What are the chances that we won’t improve upon our current ideas about right and wrong?

Yeah, pretty damn small.

OK, so if we’re wrong, we should ask the question, “What are some things that future societies might condemn us for?”

Stop here and think about them. I don’t want to contaminate your initial ideas.

…

There are some obvious ones that are already in political discourse. Recreational drugs. LGBT rights. Border and immigration laws. Religious indoctrination of children. Other things that You Can’t Say… yet.

One day morality will even go beyond those things. What about eating meat? It’s absolutely indefensible that we breed, kill, chop up, and eat animals. It’s not the eating of meat that’s wrong. If we could grow meat in vats, there would be no problem. The problem is that animals experience pain much like we do. While animals are not open to the same range of experiences as humans, they still seem to have some moral value. All else equal, I think we’d prefer that animals not suffer and die.

Future technologies could improve morality as well. Like education and nutrition today, in the future it could be considered abusive to not give your child anti-aging and intelligence-enhancing treatments. And that’s just the start.

Does this future scare you? Well so would the present to those who lived in the past. Eutopia is Scary.

Consume Less Shallow Content

2011-12-04T15:21:13-08:00

Many of my friends and coworkers own iPads. I do not. I don’t have anything against Apple, in fact I love their products. I agree that the iPad is an engineering marvel, both of hardware and software. It’s elegant. It’s responsive. It’s just plain fun to play with. But I don’t own one.

I don’t own an iPad for the same reason I don’t have a cable subscription: Because it encourages shallow content consumption.

Don’t misunderstand. I don’t think content consumption is bad. I enjoy it and spend lots of time doing it. But there are different ways to consume it, different types to consume, and differing amounts of time one can spend on it.

So what is shallow content? The shallowest content consists of action movies and chick flicks. These are equivalent to sitting in front of a machine that pulls levers in your brain to trigger emotional reactions. Shallow content is easily accessible, but uninformative and in the long run, less rewarding.

Deep content is complex. Profound. Memorable. It is organized on multiple levels. It rewards re-reading (or re-watching). Gödel, Escher, Bach: An Eternal Golden Braid is the prototype of deep content.

Shallow content is cotton candy and Coca-Cola. Deep content is raspberry cheesecake and Riesling.

When it comes to content consumption, different devices encourage different behaviors. For example, a television is the ultimate device for consuming shallow content. You select a channel. The pictures and sounds elicit emotional responses. Changing the channel is easy, so TV shows are selected for stickiness and addictiveness. TV shows can’t be as complex as other media.[1] They have to be accessible to people who start watching in the middle.

Whether or not a device is meant for content consumption is not a binary attribute; it’s more of a sliding scale. A television is solely for consuming. An iPad is more general-purpose, but encourages consumption over production, and shallow content over deep. A laptop or desktop computer is fully general-purpose. A good indicator of a device’s purpose is its input interface. A TV’s input is simple: channel selection. An iPad has a touch-screen that can show a virtual keyboard for occasions when you need to write something. A computer has a physical keyboard, the fastest brain-machine interface currently available.

Of course, even with a computer you can waste your days on Reddit and IRC (consuming and producing shallow content) or you can do something more rewarding.

Also, the cost to make a TV show is much greater than the cost to make a book. So to be profitable, a TV show must appeal to a greater population than a book. This limits the ideas and opinions on television.

Information Dieting with a Kindle

2011-03-14T00:44:35-07:00

I got a Kindle a few months ago, and it has completely changed how I read. It’s yet another example of trivial inconveniences affecting behavior.

Before, my information diet consisted of countless hours reading short, forgetful pieces linked from IRC, Twitter, Google Reader, or Hacker News. I would try to read longer articles on my laptop, but it was hard to avoid distractions such as e-mail and IM clients.

Reading full-length books was also a hassle, although I didn’t realize it at the time. I’d order a book from Amazon and wait two days. Then I had to carry a chunk of dead tree around. When travelling, I usually took multiple books with me, since I’d finish more than one on a trip.

Right after I got my Kindle, Ryan Phillips told me about Instapaper. Now I skim or ignore short things, and use Instapaper to mark a couple of large gems for evening reading. Instead of habitually refreshing Hacker News, I load it maybe once a day.

Fan fiction such as Harry Potter and the Methods of Rationality have fan-created .mobi and .epub files, but many stories aren’t popular enough to warrant such devotion. Fortunately I found FanFiction Downloader, a semi-automated way to get FanFiction.net stories onto my Kindle. The app isn’t perfect though. It runs out of memory on longer books.

The only major annoyance I’ve encountered is from flight attendants asking me to turn off my Kindle during takeoff and landing. I smile and wait for them to move on, then continue reading. Flipping the power switch wouldn’t do much anyway; Kindles don’t really turn off. They wake up to check for new content periodically, and there’s no way to remove the battery. I’m surprised the “no electronic devices” rule has lasted so long. If I were a conspiracy theorist, I’d rant about Amish Illuminati controlling the FAA. Googling for that idea doesn’t return much. I guess conspiracy theorists aren’t very creative.

VPS.net is Really Annoying

2010-11-20T03:55:10-08:00

I’m writing this because I am annoyed at VPS.net. I wanted a server in the UK so I could use things like Spotify and BBC iPlayer. Cloudkick supports Linode and VPS.net. Both are closer to normal virtual private server providers than “true” cloud providers.[1] Linode billed monthly and VPS.net offered daily billing, so I went with the latter. The signup process was typical, except that I was asked to fill out four security questions. I entered stupid nonsensical answers, since while many people know my mother’s maiden name, nobody knows my current passwords and I am unlikely to forget them. Not long after signing up, I received a confirmation e-mail… containing my password in plaintext.

E-mailing me my password tells me several things about your company. It tells me that you store passwords in plaintext instead of hashing them. If anyone gets ahold of a DB dump, they’ll have passwords and e-mail addresses. Lots of people use the same password everywhere, making their e-mail vulnerable. E-mailing my password also tells me that you don’t either don’t know or don’t care about the dangers of sending secrets via e-mail. E-mail isn’t always encrypted and messages are often relayed through many servers. Anyone with access to one of those servers could see my password.

The password thing was a big red flag, but I didn’t want to give up so easily. I booted a server and started screwing around with it. I renamed my server in the VPS.net dashboard. Suddenly, my ssh session died. It turns out that renaming a server reboots it without warning. Frustrated, I gave up and decided to try again when I had more patience.

I woke up and saw my inbox contained an invoice for $1.00. Yes, VPS.net sends an invoice every day. Worse, after a week, VPS.net started warning me that my invoices were overdue. I tried to log in and pay the measly $10. I was confronted with a login page asking me to enter my username, password, and answer some security questions. They noticed I was trying to log in from a different IP address and threw some security questions at me. I finally managed to get enough answers correct to log in.

That “Pay Now” button is actually a “try to pay $1 and show a big failure message, but mark the invoice as paid if there have been no payment attempts in the past few hours” button. I had to click it once for each invoice, waiting 3-4 hours between tries if I wanted them to work. Later I noticed the charges actually showed up on my card.

There were other things I noticed, such as sequential instance IDs. Did you know VPS.net has only booted a total of 33,000 instances? Anyway, after the invoice thing I wrote VPS.net off as amateurs and tried out Linode. I’ve had no problems with Linode. Their stuff works without annoying the hell out of me. If you want a server in the UK, go with them.

The litmus test I use is, “Can I make an API call and get a booted server in under 5 minutes?” If not, then it’s not really cloud computing. It’s more of a reasonably fast VPS provider.

Five Years of Progress in Laptops

2010-11-16T01:07:52-08:00

The last iteration of the iBook G4 came out in September of 2005 and was sold until mid-2006.

	11.6″ MacBook Air	14″ iBook G4
Processor	1.6Ghz Core 2 Duo	1.42Ghz PowerPC G4
Memory	4GB 1067Mhz DDR3 RAM	512MB 333Mhz DDR2 RAM ^[1]
Storage	128GB SSD	60GB 4200RPM HDD
Battery	5-7 hours	3-4 hours
Weight	2.3 lbs	5.9 lbs
Dimensions	0.11-0.68 x 11.8 x 7.6	1.35 x 12.7 x 10.2
Cost	$1500	$1700

[1] Bottlenecked by 142Mhz front side bus.

There are a dozen things that don’t show up in the numbers. The iBook’s keyboard is spongy. It has a smaller trackpad that only supports two-finger scrolling. The iBook’s display is much worse, although that may be due to age. The iBook’s trackpad is plastic instead of glass. Even with a fresh install of Leopard, it feels slow. Yet somehow I used it as my main computer for two years.

In case you forgot, that skinny ethernet port on the iBook is a 56k modem. It’s like ethernet, but data goes 1,800 times slower and makes angry noises.

I wonder what I’ll compare my Air to in 2015.

Expensive Computers are Worth the Price

2010-10-30T08:37:15-07:00

You should not be afraid to spend lots of money on a computer. All too often developers skimp on their computers. They balk at high price tags and try to save money by buying a less expensive model.

For example: A Samsung NC20 goes for around $500. A completely maxed-out 11.6” MacBook Air costs $1500. Most people look at those two prices and think, “Wow, I could save $1,000 by taking a small hit in performance and features.” Or worse, “$1500 is too much money for a computer.” That is the wrong way to think about it.

Here’s why: According to RescueTime, I average 10 hours a day on my laptop. Assuming I upgrade every 18 months, that’s 5,400 hours of use. Amortized over its life, a $1,500 laptop costs me 28 cents per hour. A $500 laptop would be 9 cents per hour. The important factor I haven’t mentioned yet is the amount of value I can create per hour. Let’s be extremely pessimistic and say I create $5 worth of value per hour on average. Let’s also say the $1500 laptop makes me 10% more efficient than the $500 laptop. That means I would create $5.50 worth of value each hour on the expensive laptop. Subtracting the amortized cost of the expensive laptop still leaves me with an extra 31 cents per hour.

The expensive laptop costs 3x as much and only improves my performance by 10%, but it still comes out ahead purely for economic reasons. This is because the amount of value you extract or create using a computer is much greater than the hourly cost of that computer. I’m not even factoring in more subjective things like ease-of-use or aesthetics. You shouldn’t be worried about spending too much on your computer. You should be worried about not spending enough!

I’ve followed my own advice here and bought a maxed-out 11” MacBook Air. I’m using it as my sole development machine. It’s amazing.

OK, I admit it. This entire post was just an excuse to brag about my MacBook Air.

On Alpha Geeks and Gadgets

2010-10-25T03:43:39-07:00

Earlier this month, Benjamin Stein observed that “alpha geeks” are now using the same hardware as normal people. I have a hypothesis for why this is.

Today, there are three main computing devices that people use: Laptops, which fit in a backpack and are fast enough for most stuff; smartphones, which fit in a pocket and are a tradeoff of size and speed/features; and desktops, which can have more storage and better graphics than laptops.

Now which one of those three things can an individual assemble and customize?

That’s why nerds/hackers are using the same hardware as everyone else. It used to be that almost everyone had desktops. Alpha geeks built custom machines from parts. Now, people use laptops and smartphones. Alpha geeks don’t have the resources to build or modify such tightly-integrated devices. Instead, they buy whatever best satisfies their needs. Lately, those products have been Macs and iPhones.

So are alpha geeks done with hardware tinkering? I doubt it. I think a new type of device will show up: the wearable. Recent advances in microdisplays and embedded computing have made wearable computers a borderline-practical idea. And individuals can build them for about the cost of a laptop. A few people have already hacked their own together from video goggles (like the Myvu Crystal) and single-board computers (like the BeagleBoard).

An often-heard objection to wearables is, “I already carry a computer with me. It’s called an iPhone and I can use it any time.” That’s true, but most people don’t realize how much trivial inconveniences can affect their behavior. Every time you want to look something up, you have to pull the phone out of your pocket and unlock it. You lose eye contact with anyone you were conversing with. It takes time for the phone to come out of hibernation and run whatever app you want to use. The screen is visible to those nearby, so you can’t message a friend, “What’s the name of this person next to me?”, or search your e-mail archives for a forgotten message that has come up in discussion.

Two major issues with wearables are fashion and software. Fashion will probably be resolved as technology gets better and people become more used to seeing wearables. Software is going to be a bigger problem. Wearables have to be extremely responsive. The idea is to completely integrate the computer with your life. Smartphone software has a similar constraints, but not to the same degree.

A wearable with good software would be outrageously useful. Imagine having a local cache of Wikipedia, e-mail, personal notes, and other data sources. Add a camera to the mix to get lifelogging and augmented reality. While AR is more of a toy, people completely underestimate the utility of lifelogging. I know of one case where a lifelogger had video of the first time he met his wife. A lifelog would also come in handy if you were witness to a crime or accident. To get some idea of what you do all day, you could put the lifelog data into something like RescueTime. If automated transcribing gets better you could build a searchable database of all your conversations. With these new sources of data and forms of interaction, the possibilities are quite vast.

Note: This post is expanded from a comment I made on Hacker News.

Time Management

2010-10-10T21:07:33-07:00

Each year has 365 days. Each day has 24 hours. That’s 8,760 hours a year. Sounds like a lot, but how much of it is spent doing stuff you have to do versus what you want to do?

You need around 8 hours a day for sleep. That leaves 5,840 hours of wakefulness.

A full-time job is around 2,000 hours a year. 3,840 hours left.

In addition to work you’re paid for, you have to run errands and do chores (shop for groceries, go to the bank, yard work, vacuum, cook, wash dishes, etc). Let’s say you average 90 minutes a day on that, which is 548 hours per year. 3,292 hours remaining.

How long is your commute? The average is 25 minutes each way, so just over 208 hours a year. 3,084 hours left.

How long do you take to get ready in the morning? 40 minutes? 240 hours. 2,844 left.

Do you exercise? Maybe you don’t work out every day. 1 hour a day, 3 times a week? 156 hours. 2,688 left.

Now let’s say you’re 25 years old and healthy. According to most actuarial tables, that gives you even odds of sticking around for another 50 years.

2,688 hours * 50 = 134,400 hours. That’s 15 years. I admit it’s a very rough estimate. I left out a lot of factors (such as anything related to raising children). People work less when they’re older, but that’s partially offset by more time spent being sick. I’d also weight calculations toward free time in younger years, since youth usually means fewer responsibilities and greater physical ability. Later years are spent undergoing general cognitive and physical decline.

What wisdom do I take away from this calculation? Pretty straightforward stuff: Find a job you enjoy. Use a tool like RescueTime to ensure you spend your free time wisely. Finally, you spend a significant chunk of your life using your bed, office chair, and/or computer. Don’t skimp on those things.

Stanislav Petrov Day

2010-09-26T00:34:28-07:00

Many people say Thanksgiving is the fourth Thursday in November. I disagree. If there is any day of the year on which to give thanks, it is September 26th.

Why?

Today is the 27th anniversary of when Stanislav Petrov chose not to destroy the world. When an early warning system reported the Americans had launched ICBMs, Petrov told his superiors it was a malfunction. Had he decided incorrectly, the Soviets would have launched their nukes, causing the United States to actually launch their nukes. Billions would have died.

I am continually amazed by how few know about this incident. The average person could probably tell you the latest celebrity gossip or which team won some sporting competition, but ask them who prevented a gigadeath event and they wouldn’t have a clue.

Lieutenant Colonel Petrov, your name isn’t as famous as it should be, but thank you nonetheless. The gratitude owed to you simply cannot be conveyed.