Earlier this year, I got the chance to speak with Google’s Ilya Grigorik about HTTP/2 for the 1.10 episode of the TTL Podcast. It was a great primer for me on how HTTP/2 works and what it means for how we build the web, but it wasn’t until more recently that I started to think about what it means for how we build the web — that is, how we generate and deploy the HTML, CSS, and JS that power web applications.

If you’re not familiar with HTTP/2, the basics are simultaneously simple and mind-boggling. Whereas its predecessors allowed each connection to a server to serve only one request at a time, HTTP/2 allows a connection to serve multiple requests simultaneously. A connection can also be used for a server to push a resource to a client — a protocol-level replacement for the technique we currently call “inlining.”

This is everything-you-thought-you-knew-is-wrong kind of stuff. In an HTTP/2 world, there are few benefits to concatenating a bunch of JS files together, and in many cases the practice will be actively harmful. Domain sharding becomes an anti-pattern. Throwing a bunch of

The manifest contains information about the other resources that the scout will request, and can even be used by the server to determine what to push alongside the HTML.

A manifest could provide these instructions:

module.exports = {
  baseUrl : 'https://mysite.com/static/',
  resources : {
    vendor : {
      version : 'vendor-d41d8cd98f.js',
      pushWith : [ 'scout' ]
    },
    application : {
      version : 'application-a32e3ec23d.js',
      pushWith : [ 'scout' ]
    },
    secondary : {
      version : 'secondary-e43b8ad12f.js',
      pushWith : [ ]
    }
  }
};

Processing this manifest would require intelligence on the part of the CDN; it may be necessary to replace s3 storage with an actual server that is capable of making these decisions, fronted by a CDN that can intelligently relay responses that include server push.

The elephants in the room

There are two notable challenges to the rapid transition to an HTTP/2 world: the continued existence of legacy browsers, especially on mobile; and the requirement that HTTP/2 connections be conducted over TLS. Thankfully, the latter provides a reasonable opportunity to address the former. Let’s, then, talk about the TLS requirement first.

HTTP/2 is a new protocol, and as such, it is greatly confusing to a large segment of the existing internet: proxies, antivirus software, and the like. During the development of HTTP/2 and SPDY before it, engineers observed that traffic that was transported on an insecure connection would frequently fail. The reason? The proxies, the antivirus software, and all the rest had certain expectations of HTTP traffic; HTTP/2 violated those expectations, and so HTTP/2 traffic was considered unsafe. The software that thwarted insecure HTTP/2 traffic didn’t have the ability to inspect secure traffic, and so HTTP/2 traffic over a secure connection passed through just fine. Thus was born the requirement — which is a browser implementation detail, and not part of the HTTP/2 spec — that HTTP/2 web communication be conducted using TLS.

The Let’s Encrypt project aims to eliminate the high cost of obtaining the certificate that enables secure HTTP communication; there will still be technical hurdles to using that certificate, but those should be surmountable for anyone who cares enough to engineer a performant HTTP/2 deployment.

In order for a browser and a server to communicate using HTTP/2, the browser and the server must first agree that they can. The TLS handshake that enables secure communication turns out to be the ideal time to negotiate the communication protocol, as well: no additional round trip is required for the negotiation.

When a server is handling a request, it knows whether the browser understands HTTP/2; we can use this information to shape our payload. We can send a legacy browser an HTML file that includes an inlined scout file, and that inlined scout file can include the manifest. The manifest can provide information about how to support legacy browsers:

module.exports = {
  baseUrl : 'https://mysite.com/static/',
  resources : {
    // ...
  },
  legacyResources : {
    legacyMain : {
      initialLoad : true,
      version : 'legacy-main-c312efa43e.js'
    },
    legacySecondary : {
      version : 'legacy-secondary-a22cf1e2af.js'
    }
  }
};

For Consideration: HTTP/2-friendly deployments with HTTP/1.1 support

Putting the pieces together, we arrive at a deployment process that does the following:

• Generates files that contain one or more modules, grouped by likelihood of changing, functionality, or another strategy. The file grouping strategy must persist across builds; new groupings would need a new, unique name that had not been used by earlier builds.
• Generates legacy files, where those files contain modules that are grouped according to their likelihood to change, and according to whether they are required for initial load.
• Names all files with a content hash.
• Generates a manifest for the build, where the manifest includes:
  • a baseUrl property whose value is a string that should be used as the base for generating a full URL to a resource, using the pattern /
  • a resources property whose value is an object that, for each file, provides:
    • the most recent changed version
    • a list of individual files which, when any of the files is requested, should trigger a push of the bundle
  • a legacyResources property whose value is an object that, for each legacy bundle, provices:
    • the most recent changed version
    • an optional initialLoad property whose value is true if the resource should be loaded immediately by the scout
• Generates an HTTP/2 scout file* that provides the ability to load resources, and that loads a manifest.
• Generates an HTTP/1 scout file* that provides the ability to load resources, and that includes the manifest.
• Uploads the static resources.
• Updates a delivery mechanism (such as a server or a CDN) based on the data in the new manifest.

The versioning and caching of the resources would be as follows:

• manifest Unversioned. Short cache time, e.g. 10 minutes, to allow for the rapid uptake of new resources for HTTP/2 browsers.
• scout Unversioned. Medium cache time, e.g. one day, assuming the contents of this file are considered relatively stable.
• legacy-scout Unversioned. Short cache time, e.g. 10 minutes, to allow for the rapid uptake of new resources for legacy browsers.
• application and vendor files Versioned. Long cache time, e.g. one year, given that new versions will be picked up when a new manifest is loaded.

* In applications that a) control the initial HTML payload, and b) only use the scout to load other resources, it may not make sense to have a separate scout; it might be sufficient to just load those resources via