Humboldt Distillery Spiced Rum

Sara and I check in to a hotel in the sleepy town of Eureka, California. The Carter House Inn in the Old Town neighborhood, to be exact.

The concierge let us know that a free glass of wine is waiting for us at the bar. We freshen up and bee-line it to the bar to enjoy our free libations.

Matthew, our bartender, was awesome. He tended to us nicely, especially when I asked:

“Can I take a look at that bottle of spiced rum you have?”


Humboldt Distillery Spiced Rum

The bottle had clean lines, clear glass with a honey coloured liquor inside. I opened the wood stopper and took a whiff.

Wow! Hints of vanilla and, is that? yes. Cloves.

I normally shy away from spiced rums. I feel that the spice always overpowers the base notes of the rum itself. But, I could tell that this rum was different.

The bartender explained that it was distilled in a local distillery just south of the city. This was the distillery’s first batch of rum. They also make a Vodka as well.

I ordered a glass to sample the beverage.

The colour was light. Not a true white rum, but not dark enough to be a dark rum. I suspect that the colouring is mainly from the vanilla and spices. While sipping the generous pour, I came to two conclusions:

Firstly, this is the best tasting spiced rum I’ve ever had and I need to share this with others. Second, and most importantly: this will be the first rum to officially enter my new rum collection.

I’ve decided to start collecting rum. These bottles will represent the various places I’ve visited around the world. If I ever needed a reminder of a feeling or place I’ve been to, I will have the luxury of sampling a rum from my travels to take me back.

This rum will always remind me of the best oysters I’ve ever had and my visit to Portland for our 34th and 32nd birthdays.

Velocity New York

In one week, I’ll be in New York City for the latest instalment of the Velocity Conference. This time around I will be working the Fastly booth! Come by, talk nerdy to us, and we can give you a first hand demo of our awesome CDN.

I’ll also have the opportunity to see Barbara Bermes’ talk on evaluating the performance of third party scripts. Barbara and I worked together at CBC back in the day and highly recommend that you see her talk if you’re going to be at the conference.

If you’re going to be at Velocity, drop me a line and we can get together.

Why I Love Working at Fastly

I’m lucky that I get to work with a lot of great people at Fastly. I was reminded of this during an “emergency ops” issue that occurred yesterday.

We use IRC for internal communications between all of our remote workers and teams. A message came into the ops channel:

< edmund> hey ops, i broke the coffee machine. the stack trace is "internal error in file qa_fifo, line 81"

I stared at the screen in disbelief. I had to double check that this was no joke:

< bcrosby> edmund: is the coffee machine really showing that error? if so... omg
< edmund> yes, it is.

Sure enough, the machine was reporting an error on line 81 of ‘qa_fifo’.


In typical IT fashion, the jokes and quips started flying:

< bcrosby> I just took a pic.. omg
< DocWilco> a pic of what?
< edmund> the coffee machine, i assume, kernel panicking
< brendan> did you run chef on it? 
< brendan> should have run barista instead...
* brendan finds coat
< mbyerly> roll to the coffee maker failed, roll her back
< bcrosby> barista-client ?
< aspires_> fifo?
< aspires_> is our coffee maker on a stack language?
< mmay> did you try turning it off and on again?
< bcrosby> I didn't.. cause I figured the Coffee Ops guys would want to preserve the logs ;)
< edmund> i held down a button because i thought it was the right thing to do
< edmund> maybe i overfilled a buffer somewhere with inputs
< edmund> coffee instructions unclear; broke machine.
< bcrosby> hehe
< mmay> It's probably written with windows embedded lol
< bcrosby> the fact that the file has "qa" in it.. scares me
< bcrosby> this entire time we were brewing coffee on a QA version of the code!!!!
< mmay> lol
< mmay> or their process is that once a file is QA'ed, prepend filename with qa_*
< bcrosby> maybe there is a dip switch we need to change from qa to production?

Then the bomb dropped. A co-worker found the exact bit of code that had caused the machine to crash:

< edmund> hmmm
< bcrosby> true.
< mmay> lolz
< bcrosby> edmund: rofl
< mmay> lets send debug info to them
< edmund>   maybe this was it
< mmay> I bet we can hook up JTAG to it
< edmund> asserts on overflow, should use a ring buffer. noobs

In the end, the machine was fixed by turning it off and then on again.

An Introduction to HPACK: Compressing Headers in HTTP 2

One of the problems with HTTP 1.1 is header overhead. HTTP headers are not compressed, even though your content may be. This results in situations where the HTTP headers can actually be larger than the body of the request! (Small side note: I’ve encountered a website whereby the developer decided to store HTML in a cookie. The header ended up being over 5KB in size!)

HTTP 2 solves this problem by compressing the headers using a specification called HPACK. 

HPACK compresses headers using two main methods:

Headers are now stored as explicit key value pairs. On the first request, a full compliment of headers are sent. On subsequent requests, only the difference from the headers in the first request are sent. Take a look at an example:

Let’s say you need to fetch two objects: /index.html and /logo.jpg. Here are an example of what the request headers would look like:


  HTTP 1.1 HTTP 2
# 1
GET /index.html HTTP/1.1
:method: GET
:scheme: http
:path: /index.html
accept-encoding: gzip
# 2
GET /logo.jpg HTTP/1.1
:path: /logo.jpg

A few things you will notice:

  • HTTP 2 has a new header value called “scheme”. This will be either http or https.
  • Instead of sending all of the same headers again in the second request, the client only sends the new ones. The method (GET), scheme (http), and host ( didn’t change.

This is known as differential encoding — where only the differences are sent along.

The second method that is used to reduce the size of headers is using a compression algorithm called Huffman Coding. This will futher compress the binary data.

Both request and response headers are compressed this way. This means that traditionally large headers, such as cookies, are compressed both from the client (when a cookie has already been set) or from the server (when you want to set a cookie).

This vastly reduces the chances of header overhead exceeding the size of object body. Not only will this improve the speed of browsing the web, it also means that companies will pay less to deliver their content!

Top “Non-Browser” User Agents

A simple question came to mind the other day: “What are the top HTTP libraries or applications that are used to access web content?”

What I mean by that is: What are the top user agents that are not browsers (IE, Firefox, Chrome, etc..) that web developers use to access content via HTTP. A few come to mind: libwww for Perl, libcurl for PHP, and CFNetwork for iOS.

Is this information useful to anyone? Usually these libraries are used by applications to fetch content to be handed off to another layer for presentation. Developers do no need to worry about which “non-browser” user agent to tailor their output for, like they do for IE vs. Chrome for example.

I decided to take a small sample of traffic to see if there were any interesting patterns to be found.


I took a sampling of approximately one million hits from four different geographical locations: Los Angeles, New York, Amsterdam, and Sydney. For a total of 3,147,379 samples.

All hits were filtered for user-agents that started with “Mozilla” (using the regular expression: ^User-Agent: Mozilla.*) at capture time. I later realized this didn’t filter out requests from the Opera browser, which I did in post processing. 


The user agent field is a free text field. This means that any value can be used in this field by the application or web developer. I’m assuming that most libraries do not try to pretend to be a user agent it is not by spoofing the header.

Data was captured at the same time across all geographic locations, regardless of local time. There is a direct correlation between time of day or day of week and user agent usage. I’m assuming that is not the case for “non-browser” user agents.

There may be a bias as the data was sampled from Fastly customers. 


There is no surprise that the top to user agents are CFNetwork and Dalvik. These are the agents that are used for iOS and Android applications respectively. Does this count as a “non-browser” user agent? I would argue yes. However, there is no way to determine if the requests were from the app or from an in-app browser.

BTWebClient comes in third place. This user agent is used by bit torrent clients to download torrent files from websites as well as updating trackers using HTTP.

libwww-perl comes in fourth place, followed by Java. 

Facebook External Hit is the next most popular user agent. Whenever a user shares a link on Facebook, the site goes out and fetches a copy of that link to display on the users wall. 


Rank User Agent
1 CFNetwork
2 Dalvik
3 BTWebClient
4 libwww-perl
5 Java
6 facebookexternalhit
7 Apache-HttpClient
8 HttpComponents
9 Parsoid
10 Python-urllib

I’m surprised that cURL didn’t make the top ten.


There was a signifigant location bias. For example: The Sydney data contained a lot of requests from the “FoxTel Guide” user agent. However, that agent was not found in any of the US sample data. This makes sense as users in the US do not care about a service that isn’t available in that country. 

Although this information is interesting, I can’t see any useful need to generate recurring reports or analysis over time. It’s certainly not as interesting as the ongoing browser war

The CDN Manifesto

I wasn’t able to attend this years Velocity conference. So I’m catching up now by watching videos that are available online.

A lot of people ask me: “why did you want to work at Fastly?”. It’s an innocent, but complex question. My answer usually varies based on the audience. The explanation I give non-technical people is most likely: “Because I want to make the internet faster”.

However, Fastly is doing much more than just making websites faster. They are much more than a CDN, they are an extension of your application or website. No longer is the CDN a black box that you just place between your origin and your audience.

A friend asked me the other day: “Give me the 10 second elevator pitch on why Fastly is better than any other CDN”. I thought for a split second and answered with:

“There are three main differentiating factors that sets Fastly apart from the rest of CDN field:

  • Real Time Log Delivery
  • Instant purging / invalidation
  • Full programmatic API interface

Other CDNs don’t have all three capabilities”.

These three items are very powerful to web developers. It’s allows you to fully control your content and gain visibility into what your users are doing.

An excellent talk by my co-worker, Hooman Beheshti, touches on these very points. His entertaining and informative talk at Velocity is a manifesto of what every CDN should be moving forward.

Don’t let the fact that this is a sponsored talk turn you off. It’s not a sales pitch at all.

Getting Lost in 302s

Web properties that have been around for a while probably have a lot of old links, dead ends, and redirects. There is a fear amongst content owners that users are not going to be able to find their site if a URL changes.

“What about everyone’s bookmarks?!” cries the content owner. The bookmark is something from the 90s web (web 1.0 if you will). Nobody uses them anymore.

This was a challenge I was up against at my previous job. It wasn’t until I illustrated the complexities and unscalability of keeping every URL around forever, did change happen.

The mobile landscape changes quickly. This shaped the url structure and was the main cause of the many redirects for the CBC’s mobile website. Over the course of a week and after digging through Apache configurations, Akamai config files, and “meta refresh” html files, the following flow chart was born.

Click for larger

Click for larger

Thankfully there were no redirect loops! However, there were some pretty serious issues. For example: -> -> ->

Yes. you would be redirected three times to the final URL! Not ideal, especially if you are on a mobile device!

A lot of these have since been removed. However, it wasn’t until this diagram was presented to the web developers and management did everyone realize the gravity of the situation. 

It’s true what they say: A picture speaks a thousand words. In this case: A Visio diagram improved web performance!

IPv6 and Web Performance

After reading the first 60 or so pages of Ilya’s excellent book, my mind started racing. How does IPv6 (v6) affect packet size, round trip times, and overall web performance versus IPv4 (v4)?

I decided to set up a simple test from my Windows desktop and my personal webserver hosted by Linode. Both have native IPv6 connectivity. No tunneling.

The setup:

  • Client: Windows 7 & Chrome 35.0.1916.114
  • Server: Centos 6.5 Linux (Kernel: 2.13.7) & Apache 2.2.15

I wanted to keep things as simple as possible to better understand the low level effects of IPv6 on the typical HTTP transaction. As such, I made sure to try to keep the conditions the same for both v6 and v4 requests. My test urls were:

The hostname is the same number of characters and the fetched object is exactly the same. In order to ensure that only v6 and v4 packets were being sent, I disabled support for each one in Windows before doing the test.

You can follow along the packet streams if you like at cloudshark: 


My v6 packets take a different route versus my v4 packets. This results in a lower latency for v4 traffic over the v6 traffic (average over twenty packets is 82.1ms vs. 87.4ms repsectively).

Windows doesn’t have mtr, so I used my Macbook Pro on the same network instead. [Edit: Thanks to @jpaulellis for pointing me to:]

v6 routing:

Blakes-mbp:~ bcrosby$ sudo /usr/local/sbin/mtr -n -c 20 -r
HOST: Blakes-mbp                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2601:9:8480:1196:76d0:2bf  0.0%    20    1.1   1.2   1.0   1.9   0.2
  2.|-- ???                       100.0    20    0.0   0.0   0.0   0.0   0.0
  3.|-- 2001:558:82:213b::1        0.0%    20    9.7  18.6   8.9 184.2  39.0
  4.|-- 2001:558:80:14a::1         0.0%    20   12.7  12.2  10.2  14.3   1.1
  5.|-- 2001:558:80:cf::2          0.0%    20   15.7  12.9  10.4  20.6   2.5
  6.|-- 2001:558:0:f6cb::1         0.0%    20   14.4  14.4  11.8  23.3   2.3
  7.|-- 2001:558:0:f5e8::2         0.0%    20   18.1  17.1  15.4  18.2   0.8
  8.|-- 2001:559::502              0.0%    20   20.3  17.0  14.7  21.8   1.9
  9.|-- 2001:590::4516:8f75        0.0%    20   14.2  15.7  13.9  40.4   5.8
 10.|-- 2001:590::4516:8fa6        0.0%    20   14.7  15.1  14.4  16.3   0.4
 11.|-- 2001:590::4516:8e00        0.0%    20   33.0  45.6  31.5 185.6  40.2
 12.|-- 2001:590::4516:8e3b        0.0%    20   34.6  36.4  31.0  66.4   9.6
 13.|-- 2001:590::4516:8e65        0.0%    20   64.5  66.8  64.3  99.9   7.8
 14.|-- 2001:590::4516:8e4b        0.0%    20   85.5  85.0  82.0 109.0   5.9
 15.|-- 2001:590::451f:22b2        0.0%    20   84.5  87.1  82.7  95.8   4.1
 16.|-- 2001:518:1001:1::2         0.0%    20   93.5  88.0  82.9  94.5   4.5
 17.|-- 2001:518:2800:3::2         0.0%    20   84.2  84.2  82.8  86.5   1.0
 18.|-- 2600:3c03::f03c:91ff:fe6e  5.0%    20   83.0  87.4  82.8 121.1   9.7

V4 routing:

Blakes-mbp:~ bcrosby$ sudo /usr/local/sbin/mtr -n -c 20 -r
HOST: Blakes-mbp                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--                0.0%    20    0.7   1.3   0.7   2.5   0.6
  2.|--                0.0%    20    9.2   9.3   8.3  10.5   0.8
  3.|--              0.0%    20   10.8  10.0   8.7  14.2   1.5
  4.|--               0.0%    20   12.2  13.5  10.3  38.0   6.0
  5.|--               0.0%    20   11.3  12.7   9.7  20.7   2.6
  6.|--               0.0%    20   14.2  14.1  11.0  25.9   3.1
  7.|--              55.0%    20   38.6  57.6  36.8 203.4  54.8
  8.|--               0.0%    20   39.9  43.9  38.2 102.2  13.8
  9.|--               0.0%    20   64.9  64.2  61.9  69.4   1.8
 10.|--               0.0%    20   90.9  83.2  79.9  90.9   2.4
 11.|--              0.0%    20   80.6  81.8  80.5  88.4   1.9
 12.|--             0.0%    20   84.7  85.9  81.7  93.6   4.2
 13.|--               0.0%    20   82.4  82.8  81.0  91.6   2.3
 14.|--             5.0%    20   83.0  82.1  80.9  84.0   0.9

DNS Requests

The main difference betwen looking up a v4 IP address versus a v6 address is the record name. v4 Addresses use an “A” record, while v6 addresses use a “AAAA” record. v6 addresses are also much larger at 16 bytes (versus 4), so the response will always be larger than a standard v4 response.

In this particular test, both v4 and v6 responses fit into a single packet. So the number of round trips are the same. The client is going to my router to do the DNS resolution using UDP. So there is no TCP handshake overhead.


Version # of Packets Total Size RTT
v4 2 176 bytes 0.092ms
v6 2 228 bytes 0.089ms



The v6 request is ~30% larger than the v4 request. However, round trip time will be the same (under my test conditions)

TCP Handshake

All v6 packets will be larger due to the increase in size of IP headers. v6 headers have a 40 byte size, while v4 headers only have a 20 byte header.

One thing I did notice was that the MSS size was different in the initial SYN packet from the client to the server. the v4 MSS was set to 1260 bytes, while the v6 one was set to 1460 bytes. 

The v4 SYN/ACK response from the server reset the MSS to 1460 and the same v6 SYN/ACK response reset it to 1420.


Version # of Packets Total Size RTT
v4 3 186 bytes 0.081ms
v6 3 246 bytes 0.085ms




There is no difference between v4 and v6 when fetching the object. HTTP header and body sizes are exactly the same. 


IPv6 is a new version of the Internet Protocol. It doesn’t change the way TCP or HTTP behaves. Your packets (although a little larger with v6) are routed the same way. This means that it’s latency and not available bandwidth that affects v6 performance, just like with v4.

Overall you will be pushing more bits over the wire, however the number of round trips made to the server to make a simple HTTP request with v6 is the same as with v4.


Version # of Packets Total Size RTT
v4 12 2236 bytes 0.337ms
v6 15 2848 bytes 0.342ms



What’s this? 3 extra packets with the v6 conversation? Yes. For some reason the server decided to change the MSS to 1420 from 1460 before returning the HTTP response. After the response was sent, another MSS change from 1420 back to 1460 was made. I have no idea why. But this accounts for the 3 extra packets. [Edit: My coworker pointed out that this additional TCP handshake was Chrome being proactive with setting up a new TCP session. The browser does this exepecting to download more data when you click on a link or perform another action]

v6 will eventually be the de facto IP version used on the internet. The good news is that all of the advancements in web performance such as front end optimization and HTTP/2 won’t have to change when v6 becomes ubiquitous.

v6 should result in better web performance overall. Mainly due to the fact that:

  • v6 routers don’t perform packet fragmentation.
  • Routers don’t need to perform checksums on v6 packets (like they do on v4)
  • Routers aren’t required to compute packet time in queues

The above points may be moot now with today’s fast routers and specialized cpus. However, when it comes to web performance, every little bit counts.