Simulate Slow Network in Firefox, Chrome, etc

I assumed it would be difficult to set up network delays at the kernel level so I was looking around at browser extensions. Firefox has Tamper Data and Firefox Throttle, but the former only delays HTTP requests and the latter only works on Windows. Not much available for Chrome yet.

But actually, it is very easy to simulate delays at the kernel level:

$ sudo tc qdisc change dev lo root netem delay 400ms
$ ping localhost
PING violet (127.0.0.1) 56(84) bytes of data.
64 bytes from violet (127.0.0.1): icmp_seq=1 ttl=64 time=800 ms

Increasing latency is the easiest way to see results, but netem can also simulate throttling, packet loss, and other kinds of problems. Way cool!

• netem examples
• tc man page
• tc-htb man page

Google Chrome Smooth Scroll

Chromium Wheel Smooth Scroller -- Must have addon for Chrome... I dunno why they left smooth scrolling out of the core, but this guy works great. I love the way it bounces if you go past the end of a page!

This is why we never have enough memory

It's always fun to casually notice a minor extension used to make Gmail your default email client is using more ram than several of your earliest computers had, combined. When I think of the future, I envision each checkbox and radio button requiring 1 gigabyte of ram or more. Now that's progress!

Page Fault Hell: The Worst Thing About Linux On The Desktop

Pagefaulting. Endlessly. You know, thrashing.

In theory, having 12,582,912 times as many bits of memory as the Apollo Guidance Computer would mean that I could run 12 million tasks roughly as complex as navigating a rocket to the moon and back simultaneously without breaking a sweat. In practice, things aren't quite so rosy.

It's happened to me maybe four times in the last year. I'll be minding my own business cranking out some code or whatever, when suddenly the whole system gets a little slow. I look at the memory usage indicator and it's just gone to solid dark green, indicating 100% usage. Crap. Some buggy background process has suddenly decided it desperately needs 4.5GB of ram for god knows what. Once it was vlc, once some lame-o toolbar widget, once it was a random python process... Ugh.

By now the GUI is totally locked. I desperatly mash ctrl-alt-f1, knowing that if I can just get to the console in time I can kill the process before everything I'm working on has been paged out to disk. If the kernel is in a particularly bad mood it may be some number of minutes before I can kill the evil process.

Apparently there is an OOM killer in the kernel which is sometimes activated in these situations, which aims to:

 * 1) lose the minimum amount of work done
 * 2) recover a large amount of memory
 * 3) don't kill anything innocent of eating tons of memory
 * 4) kill the minimum amount of processes (one)
 * 5) try to kill the process the user expects us to kill

Of course, a lot of the server guys don't like it. But it sounds like the perfect solution for a desktop system. If only the UI wouldn't block so much before it had a chance to work, the user might not even notice there had been a problem.

I found a bug in the Ubuntu repository about this, but it hasn't gotten much attention. My guess is most Ubuntu users don't abuse their machines enough to see this very often. The kernel maintainers argue that this isn't a bug, that the kernel is just doing the best it can with what it's told. Well, OK. Maybe a desktop system is a special use-case that deserves special behavior. But what exactly are we supposed to tell the kernel? Ideally, I'd say something like "If a process suddenly (within 1 minute) eats all the free memory and is consuming enough that there's pressure on the rest of the system, don't page out anything I'm working on that has been resident for days. Reduce this new process's priority for at least 5 minutes or so to give me time to analyze the situation and decide if it needs to be killed.

Now, there's ulimit and the various sysctl parameters like vm.swappiness that can all be tweaked and adjusted. Some of these look promising, but I'm reluctant to change any of them for fear I will make the problem worse or add a new, unrelated problem.

At any rate, this is a place where Ubuntu really falls down compared the commercial competition. I can't speak for OS X, but I know the Windows 7 UI, which you can use to shut down misbehaving processes, remains responsive even in the face of extreme memory pressure. Sure, it might take longer to finish the memory-hungry (and in all likelyhood, buggy and broken) process than a linux box would, but having the UI remain responsive at the cost of slower background tasks is just a good design choice.

Interesting old post mostly about Lisp/C at JPL

Interesting old post mostly about Lisp/C at JPL. A timeless story, except nowadays it's more commonly people arguing futilely against Java in favor of Ruby or Python or ...

Python: Iterating over a list vs. iterating over dict.items()

While cleaning up some code today, I was wondering: how efficient is Python's dict.items()? If we're usually iterating over a list of items but don't particularly care about their ordering, is it worth it to use a dict instead so we can benefit from an occasional key-based lookup? Here's the code:

# Build a long list and copy each member
a=range(30000)
[x for x in a]

# Build a list, put each member into a dict,
# and copy each member of the dict
a=range(30000)
b=dict((x,x) for x in a)
c=b.values()
[x for x in c]

Iteration over the dict's items is just a little bit slower:

$ timeit.py -s 'a=range(30000)' '[x for x in a]'
1000 loops, best of 3: 1.35 msec per loop
$ timeit.py -s 'a=range(30000)'\
            -s 'b=dict((x,x) for x in a)'\
            -s 'c=b.values()'\
               '[x for x in c]'
1000 loops, best of 3: 1.37 msec per loop

But obviously lookups in the dict will be much faster. There's extra memory usage too, of course. Still, not a bad deal. Of course, you don't want to put dict_name.values() inside the loop or the extra lookup on each pass will make performance much worse:

$ timeit.py -s 'a=range(30000)'\
            -s 'b=dict((x,x) for x in a)'\
               '[x for x in b.values()]'
100 loops, best of 3: 2.5 msec per loop

Yay for Apple?

Apple is a business first and so their actions do not generally correlate with "the good of the open web," but in this case their refusal to include Flash on the iPhone and the iPad (probably because it is an interpreted environment) puts a ton of very healthy pressure on Adobe. I like it.