This post is about the Web Application Framework in use by some teams at the BBC. It is not strictly a framework in that it specifies the contracts between components, rather than providing concrete implementations of the components. For this reason, I prefer to think of it as the Web Application Specification.
At the beginning of 2015, a group of developers and technical architects from around the BBC got together with the goal of designing a system for sharing web page components between teams. This came from an acceptance that most of the BBC’s public-facing web products have a similar look & feel, and a desire to improve efficiency through sharing rather than building similar things over and over again.
In some organisations, technologies are standardised which makes sharing components between teams a trivial task. At the BBC, teams are free to use whichever technologies they like. This means that any component-sharing solution we come up with can’t be tied to a template language, data structure, build system, or even a CSS preprocessor. This meant that building a component library like Lonely Planet’s Rizzo or CloudFlare’s CFUI was out of the picture. We also felt that something like the FT’s Origami would require too much effort on the users of the components.
We ended up going back to basics and designed a solution that is based on two premises:
What we came up with is called the WAF, or Web Application Framework. It’s a surprisingly simple framework, and is built on top of three core principles:
- Components as endpoints
- The envelope contract
- Page composition
Components as endpoints
For components to be shared successfully, they need to be easy to distribute, easy to install, and easy to use. For this, we turned to a delivery mechanism that most developers are familiar with: the web.
Every component is available at an HTTP endpoint. How that endpoint is built is unimportant, and varies from team to team. Some teams deploy dedicated applications that render specific components. Others make use of an internal component-as-an-endpoint service. Regardless of the implementation, to integrate successfully into the WAF, every component endpoint must to respond to an HTTP GET request with a rendered component that is ready to be inserted into a page.
The envelope contract
Having components as endpoints only works if all of the components are returned in the same format. To solve this, we designed the envelope format which imposes a contract on both the component endpoints and the consumers of those endpoints. The contract for component endpoints is this:
- Component endpoints MAY accept parameters which vary the output.
- Component endpoints MUST return a JSON object with the following properties:
head— an array of strings intended to be inserted into the document
<head>element. Usually stylesheets, metadata, and blocking scripts.
bodyInline— a string intended to be inserted somewhere in the document
bodyLast— an array of strings intended to be inserted at the end of the document
<body>. Usually non-blocking scripts.
- Component endpoints SHOULD atomise the
bodyLastvalues as much as possible.
Here is an example envelope response:
"<link rel=\"stylesheet\" href=\"https://m.files.bbci.co.uk/framework.css\">"
"bodyInline": "<div class=\"nw-c-top-stories\">[... lots of markup ...]</div>",
"<script async defer src=\"https://m.files.bbci.co.uk/modules/bbc-news-front-page/2.0.8/init.js\"></script>"
Now that we have web page components available in a standardised format over HTTP, we need a way to assemble the components into a page. To this end, there is a contract for systems that receive envelopes, which we call page composition systems:
- Page composition systems MUST render components in the order they are specified.
- Page composition systems MUST de-duplicate all
- Page composition systems SHOULD fetch components in parallel.
The de-duplicating aspect of a page composition system is important, because page components may share many of the same dependencies. For example, if many components were built using the Bootstrap CSS framework, they would all return a reference to
bootstrap.css in their
head. Rather than render several redundant references to this CSS, the page composition system must ensure that it is rendered only once.
Many BBC teams have implemented their own page composition systems within their existing web applications. BBC News have developed a page composition service called Mozart which provides a simple configuration-driven way to assemble pages while transparently handling the difficult problems like scaling, logging, and monitoring.
Piecing it all together
This method of assembling web pages is being used successfully in production by several teams at the BBC. The most-used component is a live-updating news feed, initially built for live coverage of sports events on BBC Sport, but now used for live reporting pages on BBC News, and the World Service. It has also seen some use as a way to display long feeds of content like on the BBC News topic pages. Soon BBC News will roll out their new front page using the WAF, which is a good opportunity for us to stress-test the framework and expand it if necessary.
One of the expansions we’re currently looking at is a mechanism for components to declare their cacheability. This will help to reduce the load on component endpoints and on page composition systems as well.