Humboldt Distillery Spiced Rum

Sara and I check in to a hotel in the sleepy town of Eureka, California. The Carter House Inn in the Old Town neighborhood, to be exact.

The concierge let us know that a free glass of wine is waiting for us at the bar. We freshen up and bee-line it to the bar to enjoy our free libations.

Matthew, our bartender, was awesome. He tended to us nicely, especially when I asked:

“Can I take a look at that bottle of spiced rum you have?”

1472933_10152807145378793_9142179495725529995_n

Humboldt Distillery Spiced Rum

The bottle had clean lines, clear glass with a honey coloured liquor inside. I opened the wood stopper and took a whiff.

Wow! Hints of vanilla and, is that? yes. Cloves.

I normally shy away from spiced rums. I feel that the spice always overpowers the base notes of the rum itself. But, I could tell that this rum was different.

The bartender explained that it was distilled in a local distillery just south of the city. This was the distillery’s first batch of rum. They also make a Vodka as well.

I ordered a glass to sample the beverage.

The colour was light. Not a true white rum, but not dark enough to be a dark rum. I suspect that the colouring is mainly from the vanilla and spices. While sipping the generous pour, I came to two conclusions:

Firstly, this is the best tasting spiced rum I’ve ever had and I need to share this with others. Second, and most importantly: this will be the first rum to officially enter my new rum collection.

I’ve decided to start collecting rum. These bottles will represent the various places I’ve visited around the world. If I ever needed a reminder of a feeling or place I’ve been to, I will have the luxury of sampling a rum from my travels to take me back.

This rum will always remind me of the best oysters I’ve ever had and my visit to Portland for our 34th and 32nd birthdays.

Velocity New York

In one week, I’ll be in New York City for the latest instalment of the Velocity Conference. This time around I will be working the Fastly booth! Come by, talk nerdy to us, and we can give you a first hand demo of our awesome CDN.

I’ll also have the opportunity to see Barbara Bermes’ talk on evaluating the performance of third party scripts. Barbara and I worked together at CBC back in the day and highly recommend that you see her talk if you’re going to be at the conference.

If you’re going to be at Velocity, drop me a line and we can get together.

Why I Love Working at Fastly

I’m lucky that I get to work with a lot of great people at Fastly. I was reminded of this during an “emergency ops” issue that occurred yesterday.

We use IRC for internal communications between all of our remote workers and teams. A message came into the ops channel:

< edmund> hey ops, i broke the coffee machine. the stack trace is "internal error in file qa_fifo, line 81"

I stared at the screen in disbelief. I had to double check that this was no joke:

< bcrosby> edmund: is the coffee machine really showing that error? if so... omg
< edmund> yes, it is.

Sure enough, the machine was reporting an error on line 81 of ‘qa_fifo’.

Bwo7nWBCYAAgw__

In typical IT fashion, the jokes and quips started flying:

< bcrosby> I just took a pic.. omg
< DocWilco> a pic of what?
< edmund> the coffee machine, i assume, kernel panicking
< brendan> did you run chef on it? 
< brendan> should have run barista instead...
* brendan finds coat
< mbyerly> roll to the coffee maker failed, roll her back
< bcrosby> barista-client ?
< aspires_> fifo?
< aspires_> is our coffee maker on a stack language?
< mmay> did you try turning it off and on again?
< bcrosby> I didn't.. cause I figured the Coffee Ops guys would want to preserve the logs ;)
< edmund> i held down a button because i thought it was the right thing to do
< edmund> maybe i overfilled a buffer somewhere with inputs
< edmund> coffee instructions unclear; broke machine.
< bcrosby> hehe
< mmay> It's probably written with windows embedded lol
< bcrosby> the fact that the file has "qa" in it.. scares me
< bcrosby> this entire time we were brewing coffee on a QA version of the code!!!!
< mmay> lol
< mmay> or their process is that once a file is QA'ed, prepend filename with qa_*
< bcrosby> maybe there is a dip switch we need to change from qa to production?

Then the bomb dropped. A co-worker found the exact bit of code that had caused the machine to crash:

< edmund> hmmm http://www.state-machine.com/qp/qpc/qa__fifo_8c.html
< bcrosby> true.
< mmay> lolz
< bcrosby> edmund: rofl
< mmay> lets send debug info to them
< edmund> http://gitorious.digitalstrom.org/ds485-stack/ds485-core/blobs/df6acce3f736e2731177748f2965361c642f05c2/src/qpc/qf/source/qa_fifo.c#line69   maybe this was it
< mmay> I bet we can hook up JTAG to it
< edmund> asserts on overflow, should use a ring buffer. noobs

In the end, the machine was fixed by turning it off and then on again.

An Introduction to HPACK: Compressing Headers in HTTP 2

One of the problems with HTTP 1.1 is header overhead. HTTP headers are not compressed, even though your content may be. This results in situations where the HTTP headers can actually be larger than the body of the request! (Small side note: I’ve encountered a website whereby the developer decided to store HTML in a cookie. The header ended up being over 5KB in size!)

HTTP 2 solves this problem by compressing the headers using a specification called HPACK. 

HPACK compresses headers using two main methods:

Headers are now stored as explicit key value pairs. On the first request, a full compliment of headers are sent. On subsequent requests, only the difference from the headers in the first request are sent. Take a look at an example:

Let’s say you need to fetch two objects: /index.html and /logo.jpg. Here are an example of what the request headers would look like:

 

  HTTP 1.1 HTTP 2
# 1
GET /index.html HTTP/1.1
Host: www.foo.com
Referer:http://www.example.com/
Accept-Encoding:gzip
:method: GET
:scheme: http
:host: www.foo.com
:path: /index.html
referer: http://www.example.com/
accept-encoding: gzip
# 2
GET /logo.jpg HTTP/1.1
Host: www.foo.com
Referer:http://www.foo.com/index.html
Accept-Encoding:gzip
:path: /logo.jpg
referer: http://www.foo.com/index.html

A few things you will notice:

  • HTTP 2 has a new header value called “scheme”. This will be either http or https.
  • Instead of sending all of the same headers again in the second request, the client only sends the new ones. The method (GET), scheme (http), and host (www.foo.com) didn’t change.

This is known as differential encoding — where only the differences are sent along.

The second method that is used to reduce the size of headers is using a compression algorithm called Huffman Coding. This will futher compress the binary data.

Both request and response headers are compressed this way. This means that traditionally large headers, such as cookies, are compressed both from the client (when a cookie has already been set) or from the server (when you want to set a cookie).

This vastly reduces the chances of header overhead exceeding the size of object body. Not only will this improve the speed of browsing the web, it also means that companies will pay less to deliver their content!

The Cost of Communication

I just got off of a 2 hour and 15 minute call with my wife, via Skype, from San Francisco to Toronto. The audio and video quality was impeccable. Modern technology is great!

This got me thinking. How much would it have cost to make the same duration call if this was 1985? What about 1950?

After some digging on the internet for historical Long Distance rates (wow, haven’t head that term in a while) I came up with the following price breakdown.

Length of call: 135 minutes.

Year $/min 2014$/min Total Cost Notes
1930 $2.75 $39.25 $5,298.75 Probably really bad audio quality
1940 $1.40 $23.83 $3,217.05
1950 $0.60 $5.93 $800.55
1960 $0.55 $4.43 $598.05 Touch tones!
1980 $0.40 $1.16 $156.60
1990 $0.25 $0.46 $62.10
2014 $0.0009132 $0.0009132 $0.12 Real time video!

I pay $40 a month for internet access divided by 43,800 minutes a month and you get the 2014 rate above.

So essentially, my call to Sara was free, included real time video and audio, and was cheaper than sending her a written letter with photographs by 1st class airmail. That letter would have cost me $1.15 and taken 5-7 days to arrive.

Top “Non-Browser” User Agents

A simple question came to mind the other day: “What are the top HTTP libraries or applications that are used to access web content?”

What I mean by that is: What are the top user agents that are not browsers (IE, Firefox, Chrome, etc..) that web developers use to access content via HTTP. A few come to mind: libwww for Perl, libcurl for PHP, and CFNetwork for iOS.

Is this information useful to anyone? Usually these libraries are used by applications to fetch content to be handed off to another layer for presentation. Developers do no need to worry about which “non-browser” user agent to tailor their output for, like they do for IE vs. Chrome for example.

I decided to take a small sample of traffic to see if there were any interesting patterns to be found.

Methodology

I took a sampling of approximately one million hits from four different geographical locations: Los Angeles, New York, Amsterdam, and Sydney. For a total of 3,147,379 samples.

All hits were filtered for user-agents that started with “Mozilla” (using the regular expression: ^User-Agent: Mozilla.*) at capture time. I later realized this didn’t filter out requests from the Opera browser, which I did in post processing. 

Assumptions

The user agent field is a free text field. This means that any value can be used in this field by the application or web developer. I’m assuming that most libraries do not try to pretend to be a user agent it is not by spoofing the header.

Data was captured at the same time across all geographic locations, regardless of local time. There is a direct correlation between time of day or day of week and user agent usage. I’m assuming that is not the case for “non-browser” user agents.

There may be a bias as the data was sampled from Fastly customers. 

Results

There is no surprise that the top to user agents are CFNetwork and Dalvik. These are the agents that are used for iOS and Android applications respectively. Does this count as a “non-browser” user agent? I would argue yes. However, there is no way to determine if the requests were from the app or from an in-app browser.

BTWebClient comes in third place. This user agent is used by bit torrent clients to download torrent files from websites as well as updating trackers using HTTP.

libwww-perl comes in fourth place, followed by Java. 

Facebook External Hit is the next most popular user agent. Whenever a user shares a link on Facebook, the site goes out and fetches a copy of that link to display on the users wall. 

 

Rank User Agent
1 CFNetwork
2 Dalvik
3 BTWebClient
4 libwww-perl
5 Java
6 facebookexternalhit
7 Apache-HttpClient
8 HttpComponents
9 Parsoid
10 Python-urllib

I’m surprised that cURL didn’t make the top ten.

Conclusion

There was a signifigant location bias. For example: The Sydney data contained a lot of requests from the “FoxTel Guide” user agent. However, that agent was not found in any of the US sample data. This makes sense as users in the US do not care about a service that isn’t available in that country. 

Although this information is interesting, I can’t see any useful need to generate recurring reports or analysis over time. It’s certainly not as interesting as the ongoing browser war

The CDN Manifesto

I wasn’t able to attend this years Velocity conference. So I’m catching up now by watching videos that are available online.

A lot of people ask me: “why did you want to work at Fastly?”. It’s an innocent, but complex question. My answer usually varies based on the audience. The explanation I give non-technical people is most likely: “Because I want to make the internet faster”.

However, Fastly is doing much more than just making websites faster. They are much more than a CDN, they are an extension of your application or website. No longer is the CDN a black box that you just place between your origin and your audience.

A friend asked me the other day: “Give me the 10 second elevator pitch on why Fastly is better than any other CDN”. I thought for a split second and answered with:

“There are three main differentiating factors that sets Fastly apart from the rest of CDN field:

  • Real Time Log Delivery
  • Instant purging / invalidation
  • Full programmatic API interface

Other CDNs don’t have all three capabilities”.

These three items are very powerful to web developers. It’s allows you to fully control your content and gain visibility into what your users are doing.

An excellent talk by my co-worker, Hooman Beheshti, touches on these very points. His entertaining and informative talk at Velocity is a manifesto of what every CDN should be moving forward.

Don’t let the fact that this is a sponsored talk turn you off. It’s not a sales pitch at all.

Getting Lost in 302s

Web properties that have been around for a while probably have a lot of old links, dead ends, and redirects. There is a fear amongst content owners that users are not going to be able to find their site if a URL changes.

“What about everyone’s bookmarks?!” cries the content owner. The bookmark is something from the 90s web (web 1.0 if you will). Nobody uses them anymore.

This was a challenge I was up against at my previous job. It wasn’t until I illustrated the complexities and unscalability of keeping every URL around forever, did change happen.

The mobile landscape changes quickly. This shaped the url structure and was the main cause of the many redirects for the CBC’s mobile website. Over the course of a week and after digging through Apache configurations, Akamai config files, and “meta refresh” html files, the following flow chart was born.

Click for larger

Click for larger

Thankfully there were no redirect loops! However, there were some pretty serious issues. For example:

cbc.ca/mobile -> m.cbc.ca -> cbc.ca/m -> cbc.ca/m/rich

Yes. you would be redirected three times to the final URL! Not ideal, especially if you are on a mobile device!

A lot of these have since been removed. However, it wasn’t until this diagram was presented to the web developers and management did everyone realize the gravity of the situation. 

It’s true what they say: A picture speaks a thousand words. In this case: A Visio diagram improved web performance!