Wildly Inaccurate

Web performance review: TradeMe

Wed, 22 Aug 2018 00:00:00 +1200

Most of the English-speaking world has Amazon and eBay. In New Zealand we have TradeMe - an online auction & classified advert website where Kiwis go to buy & sell general items, cars, and property. If the numbers are to be believed, then pretty much every Kiwi has a TradeMe account. I don’t think it’s an understatement to say that TradeMe is an itegral part of modern New Zealand culture.

Back in 2016, TradeMe announced that it was working on a new “smartphone-optimised” responsive website that will eventually also replace the desktop website. Then halfway through 2018, they announced that the new responsive website was being rolled out to real users.

Being the ever-curious developer, I wanted to give the new TradeMe website a try. After a couple of weeks using the new website on my phone (a Nokia 7 Plus), I became frustrated with how much time I spent staring at a loading spinner, and how sluggish the UI interactions were. So I went back to the old website.

The loading screen for the new TradeMe website

This bothered me, because I know that the other TradeMe experiences are fast. Why should this new website be so slow? I wanted to dig deeper, so I’ve taken the opportunity to conduct a detailed performance review.

Defining the test parameters
The benchmark
Identifying performance bottlenecks
- Main thread activity
- Delayed primary image request
Some performance experiments
Causes for celebration 🎉
- Caching improves the experience on repeat visits
- Code splitting is already implemented
Recommendations
Wrapping up

Defining the test parameters

One of the first things I like to do when I’m assessing the performance of a website is set a benchmark – what do users expect from this website? In most cases it makes sense to use a competing website as the benchmark. When we were improving the performance of BBC News, we used The Guardian and Financial Times as our benchmark because they are two of the fastest websites in the industry. However, it isn’t fair to use eBay or Amazon as a benchmark for TradeMe because they are both multi-billion dollar companies with thousands of developers. Instead, I’m using the much older (but much faster) TradeMe Touch website as a benchmark.

The next thing I like to do is find some performance metrics that are representative of the user experience. Given that TradeMe is mostly used by buyers (as opposed to sellers), I focused on visual completeness (when can I see the thing I’m trying to buy?) and interactivity (when can I scroll and place a bid?). Both of these things are very easy to measure with WebPageTest:

I measure visual completeness by looking for the point at which the primary image is loaded using hero element times (which are currently hidden behind the heroElementTimes flag).
I measure interactivity by using WebPageTest’s Time to Interactive metric, which is calculated as the point when the main thread has not been blocked for 5 seconds.

The final parameter is the test configuration. I wanted to emulate the performance of a mid-range smartphone on a mobile data connection, so for all of these tests I used the following WebPageTest configuration:

Test Location: Sydney, Australia - EC2
Browser: Chrome
Connection: 3G (1.6 Mbps/768 Kbps 300ms RTT)
Emulate Mobile Browser: iPhone 6/7/8
CPU Throttling: 2.6 (specified using the throttle_cpu flag)

I ran all of these tests through SpeedCurve, which uses WebPageTest under the hood. I did this because SpeedCurve simplifies the process of running multiple tests and comparing data across different page variations. It also has some nice charts & views that make it easier to interpret the data.

The benchmark

First up is the old TradeMe Touch website. In the screenshot below, you can see two metrics are highlighted: First Interactive and Largest Image Render. The page is considered interactive at 7.2 seconds, and the primary image is rendered at 14.9 seconds. The interactive time is not bad for a mid-range mobile device, but having to wait nearly 15 seconds to see the primary content is not something that most users will be happy about. I know I said that I wouldn’t compare TradeMe to its better-resourced international competitors, but just for reference: a similar page on eBay renders the primary image at 5.3 seconds.

A waterfall chart showing the timeline of a TradeMe Touch listing page

Now let’s see how the new website performs: interactive at 11.6 seconds, and the primary image is rendered at 30.4 seconds. No wonder I felt frustrated using this new website! On the bright side, there’s plenty of good performance lessons we can learn from this review. So let’s dig a bit deeper.

A waterfall chart showing the timeline of a TradeMe Preview listing page

Identifying performance bottlenecks

Main thread activity

The first thing that draws my eye is this 2-second slice where the main thread is busy evaluating JavaScript. During these 2 seconds, the browser is doing nothing else. No layouts, no painting, not even any network requests.

JavaScript is blocking the main thread for 2 seconds at this point in the timeline

When I open the network view, it’s obvious what’s causing this: the browser is evaluating the 716KB of JavaScript that it just downloaded. This number is the bytes that were sent over the network, and doesn’t represent the true cost of this JavaScript. Once uncompressed, these JavaScript files weigh in at over 2.7MB. (Tip: in the Network tab of Firefox DevTools, you can choose to show both the “Transferred” and “Size” columns so that you can see both the bytes over the wire and the raw uncompressed resource size).

Large JavaScript requests in the network view

This 2 second period is just the tip of the iceberg. By the time the page has finished loading, over 2.2MB of JavaScript has been downloaded. This expands to more than 5.1MB of uncompressed code that the browser must parse, compile, and execute. According to the Coverage tool in Chrome DevTools, 3MB of this code is never even executed during the page load.

Coverage report in Chrome DevTools

Delayed primary image request

The second thing that draws my attention is that the network request for the primary image does not even start until the 30 second mark. It’s request #107, out of 195 total requests. 17 other large images are downloaded before the primary image, which quickly uses up the limited bandwidth of this simulated mobile connection.

Waterfall chart showing the number of requests before the primary image is rendered

Some performance experiments

Looking at waterfall charts and dissecting JavaScript bundles is interesting, but what we really want to get out of a performance review is a list of actionable tasks with a rough estimate of how each task will impact the performance of the website. One of my favourite ways to get this list is by running some A/B performance tests by making some optimisations by hand and measuring their impact.

Prioritise loading the primary image

One of the obvious bottlenecks is the primary image being downloaded so far through the page load. We can improve the user experience by prioritising the primary image. I simulated this optimisation in two steps:

Prime the cache for the primary image. While this is not strictly possible in the real world, the effect should be similar to using <link rel=preload> to request the primary image as soon as possible.
Block all of the other image requests so that the primary image isn’t contending for network bandwidth.

Here are the results:

Metric	Original	Optimised	Difference
Primary image render	30.4s	21s	9.4s (30%)

Viewing the filmstrip of the original page versus the optimised page really shows how much faster this is:

A filmstrip comparison of the original page (top) versus the optimised page (bottom). Frame interval: 2500ms

Defer loading third party assets

While the majority of the blocking assets belong to TradeMe, there is still a considerable amount of third party content being downloaded – about 350KB, most of which is JavaScript. Deferring this third party content until later in the page load could improve the user experience by prioritising the first party content.

I simulated this optimisation by blocking requests from all third party domains. Here are the results:

Metric	Original	Optimised	Difference
First Interactive	11.6s	9.5s	2.1s (18%)

Render page content on the server

Unfortunately the new TradeMe site neither works without JavaScript enabled, nor does it work offline despite utilising a caching service worker. One reason for this is that the page content is fetched and rendered by JavaScript rather than being contained in the initial document response. Rendering some basic content on the server not only enables the page to be used offline or without JavaScript, it also has significant performance benefits since the browser can begin to render content well before it has finished downloading the other page resources.

I simulated this optimisation by creating a static version of the page with the content pre-rendered. I also moved the <script> tags to below the main content to prevent them from blocking rendering. Since this version ran in a different environment, I adjust the numbers to account for the difference in backend times. Here are the results:

Metric	Original	Optimised	Difference
Heading render	21.2s	5.7s	15.5s (73%)
Primary image render	30.4s	11.7s	18.7s (61%)

Like the primary image optimisation test, viewing the filmstrip of the original page versus the optimised page shows how much impact this optimisation really has.

A filmstrip comparison of the original page (top) versus the optimised page (bottom). Frame interval: 2500ms.

Note that First Interactive is not comparable for this test, since I changed the position of the <script> tags.

Causes for celebration 🎉

It’s far too easy in a performance review to focus on the things that a website gets wrong. I always like to try and find some things that a website is doing right.

Caching improves the experience on repeat visits

On the first page load, over 4.2MB of data is transferred. On subsequent page loads, that number is more like 570KB thanks to a caching service worker and sensible caching headers.

Code splitting is already implemented

Despite the massive bundle size, it looks like code splitting has already been implemented. Code splitting is recommended to ensure that your users only download the code that they need. Code splitting can be hard, but the fact that it is already set up on TradeMe should reduce the amount of effort required to optimise the JavaScript delivery.

Recommendations

Test on representative devices and networks

Building great user experiences requires a large dose of empathy, and one of the easiest ways to get this empathy is by using your product in the same way that your users do. Pick up a mid-range Android phone or an Intel Celeron laptop and see how your product feels on those devices. The great thing about using the same devices as your users is that they are cheap - often 10-20x cheaper than the MacBook Pro used by many developers.

Render the core content on the server

I’ve always been an advocate for having a good core experience. If you’re not familiar with the concept of a core experience, I define it as follows:

A core experience should be made up of these five things, in order of importance:

The page content.

The markup required to make the page accessible.

Minimal styling to make the content easily readable – grids, typography, etc.

Minimal styling to brand the page – logo, colours.

The means to enhance the page where appropriate.

A core experience should be just these things. Everything else is an enhancement.

This might sound familiar if you know about progressive enhancement, since having a core experience is a prerequisite for doing any progressive enhancement.

Not only does a good core experience enable users to view your product on practically any device or browser, it can also be hugely beneficial to page performance. TradeMe is a perfect example of this, with my experiment showing a 15-19 second reduction in the time to render core content.

Preload important resources

Even though browsers are becoming smarter and better at loading your page resources, they lack the context required to know which resources are the most important to your users. Thankfully, developers can give the browser some hints using <link rel="preload"> to kick-start the loading of certain resources. This is a great way to speed up the loading of primary images and dynamic content. As well as preloading resources, the upcoming Priority Hints API will give developers the ability to set an importance flag on resources.

Reduce the amount of JavaScript you ship

JavaScript is a hot topic in 2018. It is by far the biggest contributor to slow user experiences on the web – if you haven’t already read Addy Osmani’s The Cost Of JavaScript in 2018 then I would recommend doing so.

In the last few years, the web developer ecosystem has come leaps and bounds. With tools like npm and Webpack, it’s easier than ever before to create interactive user experiences and bundle them up to be delivered on the web. An unfortunate downside to this is that it’s easier than ever before to create huge JavaScript bundles that don’t perform well.

Right now there are no easy answers to The JavaScript Problem. The best thing that developers can do today is to remain diligent and balance ease of development with user experience.

Use performance budgets

Before you even start working on a project, set yourself budgets on key user experience metrics like First Interactive, Hero Element Render Time, and JavaScript bundle size. Keep an eye on your budgets and always fix problems that break your budgets. Performance budgets are becoming a de facto feature of performance monitoring tools. We used SpeedCurve’s performance budgets at the BBC to set performance targets and receive alerts when we broke the budgets.

Wrapping up

I hope this has been a useful insight into how I conduct performance reviews. I’ll be doing more of these in the future, so if there’s a part of the review that you think I should focus on more, or a particular website that you’d like me to review, let me know!

Becoming a team lead: a survival guide

Thu, 14 Jun 2018 00:00:00 +1200

This post is a transposition of my Fluent talk Becoming a team lead: a survival guide. It’s been edited to better fit a blog post format. If you prefer to see the original format, you can watch the video (30 minutes + Q&A) or view the slides with speaker notes.

A survival guide is a strange thing to call this post, since survival guides are usually very serious books that prepare you for extreme situations. Building shelter in the wilderness, foraging for food, avoiding being eaten by bears - that sort of thing. I’m never sure when is the right time to read a survival guide, since being lost in the wilderness or being hunted by bears is not something you really plan for. If you read the survival guide too early, you forget most of it. If you read it too late, it’s no use.

Anyway, the reason this is a survival guide is because after spending my entire career as a software engineer, I was thrown into a team lead role without much warning. I had no idea what I was doing and had very little support. This is the survival guide I wish I had. It documents my experiences as an inexperienced team lead –as an engineer transitioning to a team lead– and my learnings & mistakes along the way. Hopefully this can act as a survival guide for you, too.

This is a fairly long guide, so I’ve split it into sections. Feel free to skip sections that aren’t relevant to you.

Introduction: the accidental leader
Part I: Before you apply
- Understanding the team lead role
- Setting yourself up for the role
Part II: You got the job!
Further reading

Introduction: the accidental leader

In 2014 I was lucky enough to be offered a job at the BBC in London. The team I ended up working on was made up of 10 people, 7 of whom were software engineers. We had a fantastic team lead (Hi, Pete!) who kept things running smoothly and –more importantly– kept the team happy. I learned a lot from Pete.

Then, disaster struck. Within the space of only a few months, most of the team leads in our department left. Including Pete. Up until this point, I had naïvely been trying to climb the career ladder and attain more power for myself within the department. So I had essentially already been digging my own grave (okay, I’m being a little dramatic here) and it only made sense that our head of engineering pushed me into it by asking if I would temporarily fill a team lead role.

Despite my desire for more decision-making power, I was apprehensive about taking a team lead role. I liked writing code, I didn’t like responsibility, and I really didn’t want to become a manager. I was assured, however, that my day-to-day work would hardly change, and that I wouldn’t have to do any management.

Looking back on this moment, I feel a bit stupid. Of course I took the role. It sounded perfect: all the power of a team lead with none of the responsibility. It was exactly what I wanted. If only I knew back then that things would not pan out like that at all. In hindsight I would say that the first few months of my new role as team lead were some of the most stressful and challenging months of my career.

This is really common.

Having spoken to many other team leads since then, I’m amazed at just how common this scenario is. Almost all of the team leads that I’ve met in software development sort of “accidentally” fell into their leadership roles despite having no prior experience, and almost all of them found it difficult to get any support after taking the role – regardless of whether they worked for a small startup or a multinational organisation. It seemed like in contrast to their previous career progression, becoming a team lead was vastly more difficult and overwhelming.

I think that in general the team lead role is misunderstood, and it is also oversold. Our industry is really good at pretending that software delivery is all about writing code, but in all of my experiences this is never the case. Some of the hardest problems in software are either people problems, organisational problems, or cultural problems. Yet we continue to push technically brilliant software engineers into leadership roles, with no training in communication or management, and we wonder why they struggle to deliver or why they burn out.

Part I: Before you apply

Understanding the team lead role

The best way to avoid becoming an overwhelmed, burnt-out team lead is to not become a team lead. No, seriously. It’s important to be able to recognise when you’re not ready for a team lead role, or when to say no if you find yourself being pushed into one.

There are a lot of myths about leadership, and one of the biggest is that you gain power simply by becoming a leader. I thought this. A lot of people think this. My reasons for wanting to become a team lead were along the lines of:

I love writing code. Team leads should be strong coders, since they’ll be taking on all the hard problems.
I’m an experienced developer. I should be the one to steer things in the right direction.
I know what’s best for the project. Things would go so much smoother if we would just switch to this framework, or work on these features first.
My team will self-manage. Everyone on the team is so easygoing and chilled out, there’s really no need for anybody to “manage” them.
I just need the power to make change. I know exactly what to do, I just need the power and the authority to make all of the changes.

I learned pretty quickly that my reasons were all wrong. If you find yourself agreeing with any of them, then I seriously urge you to delay taking on a team lead role because it might not live up to your expectations.

But if those are the wrong reasons to become a team lead, then what are the right reasons? I would sum them up like this:

I want to share my knowledge. One of the core parts of your job as a team lead is to grow your team. You do this best by sharing your knowledge, rather than being someone who puts their knowledge into practice by writing code. Writing some code is important, because it keeps you in touch with the team’s day-to-day work. But in general the more code you write, the more the team relies on you, and the bigger bottleneck you become.
I enjoy mentoring others. Being one of the most experienced people on the team will put you in a prime position to be able to mentor others. Sharing your experiences with your team will help them learn and progress. It will also improve the quality of your team’s output.
I prefer to advise and mediate. You might think you know what’s best for the project, but taking an authoritarian approach to leadership can completely undermine any trust the team has in you. It’s better to step back and let the team make decisions, acting only as a mediator or tie-breaker. (Sometimes it’s okay to make executive decisions, but these should be rare and not be taken lightly).
I’m ready to be a manager. Not all team lead roles come with management responsibilities, but I don’t think it’s possible to completely avoid management. You will be the first person your team comes to when they have a problem. You should be ready to deal with conflicts, personal or private issues, as well as menial things like holiday and sick leave. On top of that, you should know how and when to deal out both praise and criticism.
I know how to influence change. Being a leader doesn’t enable you to force change. What it does is buy you a little bit of extra influence. Making change is all about learning how to use the influence you have to convince people to want the same change as you. Change happens easiest when everybody wants it.

The good news is that if that sounds like you, then you might find that a team lead role suits you well!

Setting yourself up for the role

Assuming you do want to become a team lead, how do you set yourself up for the role so that you’re prepared well in advance?

You likely already work in a contributor role, which will mean doing this preparation in your own time. If you’re lucky enough, you might be able to prepare in your company’s time. Either way, this time will be a good investment because it will help you to hit the ground running and become a more effective team lead.

Do some research

You’ll save yourself many headaches by researching the team lead role in advance. Learn what to expect from the role, and what skills you need to sharpen. The fact that you’re reading this is a great start (there’s a further reading section at the end of this post too). I would recommend investing time in improving your communication skills and learning about different leadership techniques.

I also recommend learning how to identify and deal with mental health issues because they are extremely prevalent in our industry. Being able to identify issues early on in yourself and the people around you can make a huge difference in preventing burnout.

Fix things that cause pain for the team

The aim of this is simple: get used to the idea of being someone who removes roadblocks for the team rather than being a core contributor. Start by volunteering to attend boring meetings, or optimise people’s workflow by making the build faster. Removing pain from the rest of your team will keep them happier and allow them to work with fewer interruptions. I like to think of this part of the role as being a sort of team janitor.

Take every opportunity to mentor

Mentorship is a great way to transfer knowledge and grow your team. It also helps to reduce the team’s reliance on you. You should pair program with junior team members to help them get through their workload, and regularly invite other people to sit with you whenever you are working through a problem.

Being a good mentor is something that I believe really only comes with experience. Getting the experience early on will make you a much better mentor when you move into a lead role.

Focus on the bigger picture

You’ll likely be at least partly responsible for delivery as a team lead. It’s good practice to zoom out and think about the optimal way to get to the end of the project. In particular, think about how to avoid roadblocks in your upcoming work. Sometimes it’s better to have one person work on something totally unrelated to your current sprint so that the rest of the team can work unimpeded in the next sprint.

Be present

Software folk are very good at putting their headphones on and living in their own little bubbles. This reinforces the idea that software development is some sort of mythical or sacred practice that should never be interrupted, and it makes people second guess whether they should ask for help. This isn’t what you want in a team. You want people to be comfortable asking for help – that can be the difference between a successful & productive team who ask for help as soon as they’re stuck, and an unproductive team who struggle with their work in silence.

Part II: You got the job!

So, you’re a new team lead - congratulations! The following sections are your handbook to being an effective team lead in your daily work.

Avoid being overwhelmed

Being a team lead can be overwhelming at times, especially during the first few months while you’re still getting a grasp of everything. If you feel yourself becoming overwhelmed at any point, don’t ignore it. Humans are quite good at picking up on each other’s emotions, and as the team lead you are likely to pass your own stress on to the rest of the team.

Stop writing code

My number one tip for dealing with being overwhelmed is to stop writing code. It’s strange for me to suggest this because for me, coding is quite cathartic - it’s a form of stress relief. But coding takes a lot of your time, and it puts you in a mindset that makes you unavailable to the rest of the team.

If you’re working on something important, see if you can pass it onto a team member. Otherwise just put it to one side for a few days.

Learn to delegate

Learning how to delegate can be difficult, especially for team leads who are transitioning from a full-time contributor role. Often the problem is that the team lead is still considered to be (or considers themselves to be) the de facto owner of certain parts of a codebase. Explicitly passing this ownership on to somebody else can help make it easier to ask other people to work on that code.

Delegation is easier said than done, but doing it correctly can be an extremely effective way of reducing your stress levels. Pick a member of the team you know is capable, make sure they have low stress levels, and (politely!) ask them to complete a task for you.

Say no

This is a personal favourite of mine, and it made a few of my senior stakeholders very unhappy at times. Saying “no” when you or your team is asked to take on more work is a totally valid way of dealing with being overwhelmed. Of course there are more polite ways to say “no”, like “can you ask me again next week” or “is there anybody else available to do that”. However you say it, it’s always better to say no than to disappoint someone by being too busy to fulfil their request.

Saying no has this other benefit which is that it signals to your boss or stakeholders that you or your team have too much to do.

The best case scenario for saying “no” is that your team is given some extra help to deal with the workload. The worst case scenario is that you’re told to do the work anyway, in which case…

Ask for help

We’re not very good at this in the software development industry. It’s as if there’s a bizarre sense of achievement or credibility in struggling through a problem for days when it could have been trivially solved by someone else in minutes. “Help” can come in many forms: someone helping you get through your work, mentorship from your boss or another team lead, or just someone to talk to over lunch. Whatever form it comes in, don’t be afraid to put your hand up and ask for help when you’re feeling the pressure. You’ll thank yourself later.

Similarly to saying “no”, asking for help sends a clear signal upwards that your team isn’t coping. This doesn’t indicate any failure on your part! If anything, it indicates a failure with your stakeholders or upper management – they’re not managing their people (“resources”) well enough.

Conducting effective one-on-one meetings

It feels strange to have a section in this guide about meetings. If you’ve ever had the (dis)pleasure of working with me, then you might know that I’m not very fond of meetings. I think I wrote too many meetings :( on a post-it note in pretty much every team retrospective I attended.

Unfortunately, this dislike of meetings carried over when I became a team lead. And since I spent the first few months of the role just panicking and stressing about my workload, I had the genius idea of saving time by avoiding one-on-one meetings. I assumed that if anyone needed anything, they would just approach me.

This was a really big mistake. One-on-one meetings are one of the best ways to keep in touch with your team because they offer a private & safe space in which to have open & honest conversations that you would not normally have in front of the rest of the team. They might feel awkward at first, but eventually these meetings should feel as natural and casual as a daily stand-up.

Have them regularly

The first piece of advice I have around one-on-one meetings is to have them, and have them regularly. I only mention this because I know from experience how easy it is to avoid them. It doesn’t matter how tight your deadlines are, or how much fun your current workload is, meeting with your team should be at the top of your priority list.

Having them every 2 weeks is a good place to start. If 2 weeks feels too regular, you can slowly increase the time between meetings until they feel just right. Remember to set up recurring calendar events and treat them like you would any other important meeting. Never skip them!

Use a template

I found it really useful to have a template for my one-on-one meetings. I took Lara Hogan’s Questions for our first 1:1 template and modified it to suit my team.

A template gives your meeting some structure and ensures that you don’t miss out anything important. You can stray from the template and adjust it as you go along - it’s just there as a guide.

Take notes

Take notes of everything that’s discussed, because you’ll want to refer to them later. Get the consent of the person that you’re meeting with first, and reassure them that the notes are completely confidential, and only used for your own reference. I usually start every meeting by showing the other person my notes from last time, to make sure we’re on the same page.

Prepare beforehand

Give yourself 10 minutes before each of your one-on-ones to prepare yourself. Think about what you want to say. Remind yourself what each person has been working on recently. Go over your notes from the last meeting and make sure you’ve done everything you said you would. If you haven’t, well, you’ve got 10 minutes to think of some good excuses.

Ask and listen

Probably the most important bit of advice regarding one-on-one meetings is to remember that your role in these meetings is to listen. After you ask a question, just sit back and listen. Long stretches of silence are okay - they give you both a chance to think.

Don’t just listen, either. Really take in what the other person is saying – regardless of how important you think the matter is, or whether you agree or not. This person is telling you something because they think it’s important, and it’s your job to take everything they say very seriously.

Don’t over-promise

Remember how I said that you should do everything that you said you would in the last meeting? It helps if you don’t put yourself in an awkward position in the first place by promising things that you can’t deliver. It’s fine to promise that you’ll order a new keyboard, or that you’ll raise an issue to your manager’s attention. But don’t ever promise things like promotions or pay rises. Over-promising will only lead to disappointment and dissatisfaction.

Dealing with pressure from stakeholders

Let’s talk about stakeholders. No matter the size of your company, you probably have at least a few. They’re the people who are interested in the outcome of your project, and they probably set the overall direction of your work by coming up with the project requirements and various tweaks along the way.

At some point you will find yourself in a position where your stakeholders are asking your team to do too much, and with too little time. How you handle these situations can have a tremendous impact on the happiness and productivity of your team, so approach them with caution.

Always be honest

Always be honest about how your project is going. Especially if you’re behind schedule. In business-speak, this is called “managing expectations”, and it’s vital that you and your stakeholders have the same expectations.

… Unless you’re ahead of schedule

There is one exception to the “always be honest” rule, which is that you should not tell anyone that you’re ahead of schedule. The only time you’re really ahead of schedule is when you finish the entire project early. Until that point, you’re just one unexpected delay away from being behind schedule.

Compromise early & often

Since you’re going to be telling your stakeholders when you’re behind schedule, it’s useful to learn how to compromise. Compromising can be an awkward and painful process where you basically ask your stakeholders to either cut back features or give you more time. You should try to compromise as soon as you know the deadline is tight, and you should do it as often as you need to. Compromise until the deadline feels truly achievable. Just remember that there is no “us vs. them” – everybody wants the same outcome, which is a high quality product delivered in a timely manner.

Don’t pass the pressure onto your team

There will be times where you truly are behind schedule and there are no compromises to be made. During these times, a good team lead will shield their team from any pressure, and generally protect them from the harsh realities of the world. When the pressure is on, the last thing you want to do is stress the team out. Even teams that work well under pressure will eventually burn out and become unproductive.

Ask for help

A common theme in this guide is to ask for help. A strategy I’ve used successfully in the past when I’ve been really close to a deadline is to borrow some time from another team. Then I can return the favour when that team needs help in the future. This is effective because having somebody else work on little things like bug fixes can really take the pressure off everybody else while they focus on finishing major features.

Grow & support your team

I said before that a core part of the team lead role is growing the people on your team. I think it’s actually the most important part. I’ve heard people say that delivering value to the business is the most important part of your role. I disagree with this. To be fair, growing your team and delivering value are two things that go hand-in-hand. You’ll find that as people grow, they learn new skills, and they become happier & more productive. The more skilled your teams are, the more value they end up delivering (and the more they should be paid).

People grow in different ways, and part of your role is to identify how everyone on your team grows best. Some people like to be mentored, and told what to do. Some like to be given reading material or tutorials to follow. One thing that is common to most people I’ve worked with is that they grow when they’re challenged.

Allow pet projects

Sometimes your team’s day-to-day work isn’t challenging enough to stimulate growth. This is why I’m quite fond of pet projects. You don’t have to set aside dedicated time for pet projects, like Google’s now-defunct 20% time. Instead you can allocate time for pet projects when they’re most appropriate. For example, someone on your team wants to learn about React. Wait until you have some downtime, and then ask them to spend a day or two building a simple clone of something they worked on recently using React. Afterwards, have them report back to you (and the team, if they’re comfortable with that) on the pros and cons.

Encourage pairing

For the times when the day-to-day work is challenging enough, you want to make sure that everyone on the team is actually progressing with their work enough to really learn and grow from it. I was given a tip to help with this:

Nobody is allowed to work on a task by themselves for more than one day.

I think this is a fantastic tip. It ensures that people spend less time being stuck on simple problems, and more time progressing with their work in a way that challenges them and allows them to grow. It also works really well for tasks that you know will take longer than a day, because you end up with more than one person having knowledge about these complex tasks. This reduces your reliance on your more senior team members who would normally take on these complex tasks by themselves.

Getting together on a regular basis to talk about things you’ve all learned recently can be a good way to keep everybody up-to-date on the whole project. You can even use these sessions as a way to talk about non-work things, like what you’re all planning to do over the weekend. This “not-work” sharing might not be appropriate for every team, but I think it’s a nice way to make the meetings less formal and get everyone to know each other as people.

These sessions are also the perfect opportunity to invite other teams along to encourage cross-pollination of ideas.

Spend time together

I’m not a particularly social person. I used to be the person who would skip team lunches because I’d rather eat by myself. Everybody relaxes and has fun in different ways, which is why it’s so important to enable the team to spend time together in a variety of settings. Have lunch together in a restaurant, or in the office, or in a park. Go to meetups or conferences together. Start a book club. Use techniques like swarming –where you all work together on the same task– to bring everyone together.

Swarming is a particular favourite of mine. Firstly because it’s a great way to trick people into spending time together (they all think they’re doing work but in fact they’re bonding and being social – ha, punk’d!) and also because it’s just a genuinely good way of getting work done. I’ve found that swarming works particularly well for tasks that are big but easy & monotonous – things like testing a single feature on many configurations, or writing documentation.

Remember: you are a team!

Every team is different. As the team lead, your job is to celebrate everyone’s differences and make every effort to ensure that each person on the team has an equal opportunity to grow and achieve their goals. I promise you that focussing on the growth and happiness of your team will not only make you a better team lead, it will make your team happier and proud to work together. It will make your team more productive and more successful. If you’re lucky, it will even inspire members of your team to become great teams leads in the future – just like you.

Introducing Second: a framework for mostly-static React applications

Sat, 05 Aug 2017 00:00:00 +1200

TL;DR

Second is a framework for building React applications where the data is fetched on the server and most of the components do not change after the first render.
Components can declare their data dependencies using a container similar to Facebook’s Relay.
By default components are only rendered on the server; client-side rendering is opt-in by using the component dehydrator.
Second works with any React-like library including Preact.

Why build this framework?

While using React to build the BBC News front page and several other mostly-static pages, a common theme emerged: only a small number of components on these pages require client-side JavaScript to function. Rendering every component in the browser does not provide an optimal experience — especially for users on low-powered devices or low-speed connections. Instead, selectively bundling components for the browser reduces bundle sizes and minimizes CPU overhead without sacrificing React’s event system and stateful components.

Second is not a large or complex framework. It is the result of combining several simple and well-tested techniques—

Server-side rendering
Container components for declarative data requirements
Selective client-side rendering

—and combining them into a single package that can be easily reused across multiple applications.

How does it work?

Server-side rendering

At its simplest, Second is just a wrapper around server-side rendering:

import VDom from 'react'
import VDomServer from 'react-dom/server'
import second from 'second'
import MyComponent from './components/my-component'

second.init({ VDom, VDomServer })

const props = {
  // ...
}

second.render(MyComponent, props).then((result) => {
  console.log('Rendered HTML:', result)
})

Data-driven components

second-container is a higher-order container component inspired by Facebook’s Relay that allows components to declare their data requirements. Unlike Relay which works only with GraphQL, second-container works with regular HTTP requests out of the box and supports custom data fetchers as well.

const ComponentWithData = second.createContainer(MyComponent, {
  data: (props) => ({
    // Each key is provided to the wrapped component as a prop
    queryResponse: {
      // Props can be used to dynamically build API endpoints
      uri: `http://api.duckduckgo.com/?q=${props.query}&format=json`
    },

    events: {
      // Requests can be marked as optional to avoid failing
      // the entire render when one request fails
      mustSucceed: false,
      uri: `https://api.github.com/users/${props.username}/events`
    }
  })
})

Containers can be used at any level of the render tree, allowing leaf nodes to fetch their own data. For static render trees, this typically scales better than fetching data at the root and passing it as a prop down through the entire tree.

Second’s renderer integrates seamlessly with the containers. By keeping track of outgoing requests, it ensures every mandatory request has returned successfully before fulfilling the rendered tree.

Partial, opt-in browser rendering

second-dehydrator is a higher-order component that serialises (“dehydrates”) a component’s props and renders them into an inline <script> tag. The component can then be hydrated in the browser with its original props, and rendered as a regular interactive React component.

Take this “top stories” BBC News component, for example. It is static except for the live-updating promo in the bottom-right corner.

BBC News top stories component, containing a live-updating promo

This component can be bundled efficiently by dehydrating just the live promo:

import { createDehydrator } from 'second-dehydrator'
import StaticPromo from './components/static-promo'
import LivePromo from './components/live-promo'

const dehydrate = createDehydrator(React.createElement)
const DehydratedLivePromo = dehydrate(LivePromo)

const TopStories = (props) =>
  <div className="nw-c-top-stories">
    <StaticPromo item={props.items[0]} />
    <StaticPromo item={props.items[1]} />
    <StaticPromo item={props.items[2]} />
    <StaticPromo item={props.items[3]} />
    <StaticPromo item={props.items[4]} />
    <DehydratedLivePromo item={props.items[5]} />
  </div>

To hydrate this component in the browser, second-dehydrator takes a component map and a callback. The hydrator scans the DOM for dehydrated components and their corresponding props. It hydrates each component with its props and passes them to the callback function, where they can be rendered with React.

import React from 'react'
import ReactDOM from 'react-dom'
import { hydrate } from 'second-dehydrator'

const renderComponent = (HydratedComponent, props, containerElement) => {
  ReactDOM.render(
    React.createElement(HydratedComponent, props),
    containerElement
  )
}

const componentMap = {
  LivePromo: () => import(/* webpackMode: "eager" */ './components/live-promo')
}

hydrate(renderComponent, componentMap)

What next?

Second has worked well for some of the projects that I’ve worked on this year. These projects have mostly been content-driven pages, with a couple of interactive form-based applications. I’d like to use Second in a wider range of projects to get a feel for whether the abstractions are at the right level and whether the APIs are flexible enough.

By opening up the framework, I’m hoping that the wider React community can have some discussions about whether selective client-side rendering is a useful pattern, and if it is, whether a framework like Second is the right way to implement the pattern.

Introducing a faster BBC News front page

Tue, 04 Apr 2017 00:00:00 +1200

Web performance is something I care deeply about both as a developer whose work affects millions of people around the world, and as a user who often accesses the web on slow & unreliable connections. I have regularly and loudly complained that the BBC News website is unnecessarily slow, so when I was given the opportunity to help rebuild one of the most visited pages of BBC News—the front page—I jumped at the chance.

That was April 2016. Now, a whole year later, we’re ready to begin a phased rollout of the new front page. Starting with a small percentage of users in the UK, we will gradually move everybody to the new front page over the course of several weeks. (Update: as of June 2017, the new front page is rolled out to all users).

Quick facts about the new front page

It is lighter and faster than the old one:
- First meaningful paint happens up to 50% sooner on mobile devices^[1].
- Page enhancements like lazy-loaded images load 150% faster on mobile, and 70% faster on desktop.
- The total bytes downloaded is 50% less on mobile and 75% less on desktop.
- CPU busy time has been reduced by 30% on mobile and by 50% on desktop.
- Performance monitoring has been automated with SpeedCurve from the beginning of the project.
It is available over HTTPS, and we have plans to redirect insecure traffic to HTTPS in the not-so-distant future.
The page is built from React components that are styled with the BBC’s CSS framework, Grandstand.
Each component is a horizontal “slice” of the page that fetches its own data. This makes it easy for us to reuse slices on any page.
The React components are individually rendered by the BBC’s React-component-as-an-API-endpoint service and assembled into a page by our page-assembly-as-a-service system.
React is only used on the server. We do not load it in the browser^[2].
The development team consists of 5 developers and 1 tester, but we have collaborated with over 60 other developers and testers from all around the BBC to build this page.

What’s next?

The version of the front page we’re rolling out is an MVP, a phase one. We will be changing it considerably over the next several months. Here’s an idea of what you can expect to see:

Performance improvements

While we have managed to improve the performance of the front page considerably, there is still a lot of work to do:

The first meaningful paint time is still too high. We can improve it by loading the core CSS sooner.
We still send too many bytes to the client. A good portion of this is from inline styles that are only used on IE8. (Update: We already have a pull request open to drop the IE8-only styles, which should reduce the amount of inline styles by about a third).
Style recalculations and layouts take too long on low-powered devices. This still needs some investigation.
We are essentially hamstrung by the “white BBC bar” at the top of the page. This bar contains components from other parts of the BBC like Search, Notifications, and BBC ID. All of these components load their own blocking CSS & JavaScript before any of the BBC News assets. While this is unlikely to change in the short term, we’re hoping to work with the teams that own these components to reduce their impact.

Design enhancements

In order to ship the new front page sooner, we made a lot of compromises with both the UX and editorial teams around the design of the page. Once we’re finished with the rollout, we will be improving the visual treatment of story cards (promos) to highlight correspondent stories and feature pieces. The current designs look something like this:

The plain promo design for the MVP launch

And below is one of the proposals for how we might display other types of promo:

An example of potential visual treatments for special types of promo

React in the browser

We decided early on in the project that using React in the browser was overkill for a page that is predominantly static text and images. The performance impact of bundling so much JavaScript and executing it in the browser is also unacceptably high: Even making use of server-side rendering, emulated mobile devices spend nearly 4 times as long executing scripts and performing layouts & paints when React was run on the page.

A timeline of the server side rendered page without React in the browser

A timeline showing the impact of running React in the browser

Our current approach to running JavaScript in the browser is to build a good ol’ fashioned bundle, completely separate from the React components. However, we realise that this isn’t going to scale for very long, and that eventually we will have to find a way to run our React components in the browser without impacting the user experience. Solutions that we’re looking into include:

Using Preact in place of React.
Converting our components to stateless functional components where possible to reduce their size.
Smarter code splitting so that we can load non-essential code on-demand.

Acknowledgements

Firstly, to everybody involved: thank you. Rarely have I had the chance to work with so many talented, patient, dedicated, and caring people. From documentation tweaks to requirements gathering, from bug fixes to building entire components; regardless of the size of your contribution, we wouldn’t have reached this point without you. So again: thank you, and congratulations on reaching this milestone.

To the design team and editorial staff: thank you for helping us find a balance between perfection and a fast launch.

To the project managers, business analysts, and product owners: I don’t think you get enough credit. Thank you for working extraordinarily hard to smooth out all of the bumps in this project, and for providing the development team with a clear path.

And finally, to my team: You are amazing. I’m so proud of what we’ve built together. On a more personal level, you’ve made coming into work feel like coming home to a second family, and I’m eternally grateful for that. The donuts are on me 💜.

Footnotes

Mobile device testing was performed using the “Mobile - Emerging Markets” setting on WebPagetest (Chrome Beta on a Motorola G (gen 4) tested from Dulles, Virginia on a 400 Kbps 3G connection with 400ms of latency).
BBC News does not load React in the browser, but some other page components (like the BBC-wide search bar in the top-right) do load React.

Web development technologies to adopt in 2017

Thu, 16 Feb 2017 00:00:00 +1300

I started 2016 feeling quite overwhelmed by the sheer number of new technologies that were being introduced. This year I feel like many of those technologies have matured, so I have collated a list of the ones that I think deserve your attention. My focus for the last couple of years has been on performance, so I’ve made an effort to ensure that all of the technologies mentioned are either “performance-friendly” or are directly related to performance.

Preact — preactjs.com

A fast 3kB alternative to React with the same ES6 API. For projects that use React’s other APIs, preact-compat exists as a compatibility layer allowing Preact to be a complete drop-in replacement for React.

As well as being smaller than React, Preact is also much faster. I measured the impact of migrating to Preact in three projects and saw a 3-4x reduction in JavaScript execution times in all of them (some data and profiler screenshots for one project).

Webpack — webpack.js.org

Bundle your scripts, images, styles, assets… I was initially put off Webpack because I thought it did too much. With the help of Pete Hunt’s webpack-howto I realised that Webpack can be as simple or as complex as you want it to be.

These days I recommend that people use Webpack even for trivial projects because it’s easy to set up, the defaults are good, and it opens up a huge number of possibilities for optimising your application.

offline-plugin — github.com/NekR/offline-plugin

A highly-configurable Webpack plugin that provides an offline experience for your application using ServiceWorker, with AppCache as a fallback.

If there’s one thing on this list that you consider adopting, it should be this. The number of people around the world whose sole means of accessing the Internet on a mobile phone is increasing. We can improve their experience on the web immeasurably by utilising technologies like ServiceWorker & AppCache, and this plugin is a ridiculously easy way to do that.

Tachyons — tachyons.io

Functional CSS for humans. Tachyons is a mobile-first responsive CSS framework with a focus on accessibility. Its single-purpose class structure is scalable and practically removes the need to write any custom CSS.

Tachyons has a relatively small footprint, and is extremely modular so you can easily include only what you need. It’s a good, powerful alternative to heavier frameworks like Bootstrap or Foundation.

Lighthouse — github.com/GoogleChrome/lighthouse

Built by some of the industry’s foremost web performance evangelists, Lighthouse is an automated tool for analysing your web application’s performance and providing insights on developer best practices.

As well as using it as a once-in-a-while test, I also recommend running Lighthouse on a regular basis — perhaps part of a daily build. Its insights are broad enough to cover a wide range of issues from accessibility to performance, but specific enough that any issues are easily actionable.

How we assemble web pages at BBC News

Mon, 13 Feb 2017 00:00:00 +1300

This post is about the Web Application Framework in use by some teams at the BBC. It is not strictly a framework in that it specifies the contracts between components, rather than providing concrete implementations of the components. For this reason, I prefer to think of it as the Web Application Specification.

At the beginning of 2015, a group of developers and technical architects from around the BBC got together with the goal of designing a system for sharing web page components between teams. This came from an acceptance that most of the BBC’s public-facing web products have a similar look & feel, and a desire to improve efficiency through sharing rather than building similar things over and over again.

Cards or 'promos' from BBC Programmes, BBC Sport, BBC News, and CBBC.

In some organisations, technologies are standardised which makes sharing components between teams a trivial task. At the BBC, teams are free to use whichever technologies they like. This means that any component-sharing solution we come up with can’t be tied to a template language, data structure, build system, or even a CSS preprocessor. This meant that building a component library like Lonely Planet’s Rizzo or CloudFlare’s CFUI was out of the picture. We also felt that something like the FT’s Origami would require too much effort on the users of the components.

We ended up going back to basics and designed a solution that is based on two premises:

All of our web pages are built out of HTML, CSS, and JavaScript.
Most pages have three main parts: the head, which contains metadata and styling; the body, which contains content; and the part at the end of the body, which contains JavaScript.

What we came up with is called the WAF, or Web Application Framework. It’s a surprisingly simple framework, and is built on top of three core principles:

Components as endpoints
The envelope contract
Page composition

Components as endpoints

For components to be shared successfully, they need to be easy to distribute, easy to install, and easy to use. For this, we turned to a delivery mechanism that most developers are familiar with: the web.

Every component is available at an HTTP endpoint. How that endpoint is built is unimportant, and varies from team to team. Some teams deploy dedicated applications that render specific components. Others make use of an internal component-as-an-endpoint service. Regardless of the implementation, to integrate successfully into the WAF, every component endpoint must to respond to an HTTP GET request with a rendered component that is ready to be inserted into a page.

The envelope contract

Having components as endpoints only works if all of the components are returned in the same format. To solve this, we designed the envelope format which imposes a contract on both the component endpoints and the consumers of those endpoints. The contract for component endpoints is this:

Component endpoints MAY accept parameters which vary the output.
Component endpoints MUST return a JSON object with the following properties:
- head — an array of strings intended to be inserted into the document <head> element. Usually stylesheets, metadata, and blocking scripts.
- bodyInline — a string intended to be inserted somewhere in the document <body>.
- bodyLast — an array of strings intended to be inserted at the end of the document <body>. Usually non-blocking scripts.
Component endpoints SHOULD atomise the head and bodyLast values as much as possible.

Here is an example envelope response:

{
    "head": [
        "<link rel=\"stylesheet\" href=\"https://m.files.bbci.co.uk/framework.css\">"
        "<style>.nw-c-top-stories{padding-bottom:2rem}</style>",
    ],
    "bodyInline": "<div class=\"nw-c-top-stories\">[... lots of markup ...]</div>",
    "bodyLast": [
        "<script async defer src=\"https://m.files.bbci.co.uk/modules/bbc-news-front-page/2.0.8/init.js\"></script>"
    ]
}

Page composition

Now that we have web page components available in a standardised format over HTTP, we need a way to assemble the components into a page. To this end, there is a contract for systems that receive envelopes, which we call page composition systems:

Page composition systems MUST render components in the order they are specified.
Page composition systems MUST de-duplicate all head and bodyLast values.
Page composition systems SHOULD fetch components in parallel.

The de-duplicating aspect of a page composition system is important, because page components may share many of the same dependencies. For example, if many components were built using the Bootstrap CSS framework, they would all return a reference to bootstrap.css in their head. Rather than render several redundant references to this CSS, the page composition system must ensure that it is rendered only once.

Many BBC teams have implemented their own page composition systems within their existing web applications. BBC News have developed a page composition service called Mozart which provides a simple configuration-driven way to assemble pages while transparently handling the difficult problems like scaling, logging, and monitoring.

Piecing it all together

This method of assembling web pages is being used successfully in production by several teams at the BBC. The most-used component is a live-updating news feed, initially built for live coverage of sports events on BBC Sport, but now used for live reporting pages on BBC News, and the World Service. It has also seen some use as a way to display long feeds of content like on the BBC News topic pages. Soon BBC News will roll out their new front page using the WAF, which is a good opportunity for us to stress-test the framework and expand it if necessary.

One of the expansions we’re currently looking at is a mechanism for components to declare their cacheability. This will help to reduce the load on component endpoints and on page composition systems as well.

What it's like to work as a developer at BBC News

Mon, 26 Dec 2016 00:00:00 +1300

The BBC is a pretty large organisation. Today it employs around 20,000 people (actually around 35,000 when you include part-time and fixed-term contract employees) across a huge number of divisions. The BBC Careers website typically has over 100 vacancies posted on any given day. Before I joined the BBC, I found the sheer scale of it a bit intimidating. Usually I can get an idea of what it’s like to work for a company by reading their job advertisements and their engineering blogs, but with the BBC I was almost completely clueless. In this post I hope to shed some light on what it’s like to work as a developer or tester for BBC News.

Just a small disclaimer first: from an engineering perspective, the BBC is not like most other companies — it’s more like dozens of smaller companies, each with their own engineering department, working towards a common goal. News, Sport, Programmes, iPlayer, Radio… As digital products, these are all built mostly independently of each other. I work for BBC News, so a lot of what I’ve written may not apply outside of BBC News.

Teams & projects
Learning & personal development
Working hours & annual leave
Working environment
Technologies
Other perks & local discounts
The Joel Test

Teams & projects

Note: When I say “BBC News”, I mean “BBC News and BBC World Service”. The 30 or so World Service sites run on the same Responsive News codebase as BBC News. For this reason, the World Service and News teams consider themselves to be part of the same overarching team.

There are about 6 BBC News development teams at any one time. Teams tend to form around projects which can last anywhere between a month (building a small routing service for the BBC’s RSS feeds) to 9 months (rebuilding the BBC News front page with a new technology stack). As well as developers, each team typically also has a dedicated tester, business analyst, project manager, and product owner.

In the past, teams were relatively static and people didn’t move around much between projects. More recently though, the teams have been shuffling a lot more. We don’t have a formal team rotation system in place at the moment, but it’s easy to move between teams if you want to work on something new. I’ve worked on 3 different teams over the last two years.

As well as collaboration between these BBC News teams, we also regularly collaborate with teams from all around the BBC. In the last 3 months alone, my team has worked with:

The Local News team in Cardiff.
The Sport and Live teams, and the Accessibility Champions group in Manchester.
The Worldwide team in West London.
The Weather, Programmes, and Radio teams in Central London.

Learning & personal development

BBC staff can attend BBC Academy courses for free. There are so many courses available covering subjects across journalism, broadcasting, technology, management, and personal development — just to name a few. You can even earn a MSCS (Masters of Science in Computer Science) by completing Academy courses over a 22-month period.

We give a Pluralsight license to every new starter.

We have an office library, and an annual budget for purchasing new books.

We run a developer gathering every month where developers and testers from BBC News and other teams around the BBC give technical talks and open the floor to discussions.

We run a similar but less technical town hall every fortnight where people from all around BBC News (including editorial staff) give 5 minute lightning talks about what’s going on in their world.

There is a budget for attending external events & conferences.

Most teams have at least one dedicated learning day every two weeks where everybody is encouraged to spend their time learning something new.

On top of learning days, we encourage everybody to dedicate a couple of days each month to personal development.

We have regular personal development reviews to make sure everybody has what they need to achieve their personal goals.

Working hours & annual leave

The BBC has a “core hours” system where you are required to work between 10am and 4pm. Other than that, you can pretty much work whichever hours you like so long as you work all of your contracted hours (usually 35 hours per week). I come in at 8am every day and leave at 4pm. Some of my colleagues come in at 10am and leave at 6pm.

Working from home is common, and is done at your manager’s discretion. Many people work from home one day each week.

Overtime is actively discouraged. No project is more important than your well-being!

Taking annual leave is encouraged, with enough notice. If you work on a project with a hard deadline like an election or sporting event, your manager might ask you to avoid taking leave in the weeks leading up to the deadline, but this is not mandatory.

Working environment

All BBC News engineering staff are located in New Broadcasting House. The entire building is accessible, and has the usual amenities including showers, toilets, and tea points, which are all also accessible.

Most of the web & apps engineering teams work in an open plan office space alongside the UX teams. There’s plenty of natural light, and there are a few adjustable standing desks available. Everything is set up for hot desking, but in reality people tend to sit in the same area to work with their team.

If you find open spaces are detrimental to your productivity, there are plenty of high-backed sofas and (mostly) sound-proofed enclosures scattered around the area. There are also several semi-enclosed “train carriages” that seat up to 6 people which are great for collaborative working.

Technologies

Most of our products are web-based and are written in Ruby, JavaScript, and PHP alongside the usual HTML & CSS (usually Sass). We also have some non-web projects which tend to be written in Java and Scala. Teams have the freedom to use whichever technologies they like, provided there is sufficient justification. Here’s a sample of technologies which I’ve personally worked with so far at BBC News:

Responsive News is a PHP application which uses a range of interesting techniques to support all of the World Service languages and scripts.
The BBC News AMP project and many of the worldwide elections components are powered by an open-source broker-renderer Ruby framework called Alephant.
The BBC News front page is built with React and Node.js on a platform developed by BBC Sport.
One of our page composition systems is powered by services written in Ruby and Go.

Other perks & local discounts

Being just down the road from Oxford Circus puts us somewhat dangerously close to dozens of really great shops, restaurants, cafés, and bars. Many of them offer a discount to BBC employees.

The BBC runs a centralised benefits scheme which offers things like:

A great pension plan.
Dental, health, and travel insurance.
Childcare vouchers.
Access to Cyclescheme.
The ability to buy and sell annual leave.

Secure bike storage is available, or you can just lock your bike to the railing outside the building.

The Joel Test

Here’s how BBC News scores on The Joel Test:

Do you use source control?
Yes. We use Git for almost everything, and SVN for a very small number of legacy systems.
Can you make a build in one step?
Yes, all applications can be built in one step. Older applications have a separate deployment process, while newer applications are deployed to integration environments automatically.
Do you make daily builds?
We practice continuous integration, so there are usually hundreds of builds happening every day.
Do you have a bug database?
Yes. We aim to keep the backlog tidy, and will often close minor bugs which are not likely to be fixed.
Do you fix bugs before writing new code?
Not always. Most teams will prioritise bugs which affect more than 1% of users. Other bugs are typically left at the bottom of the backlog and eventually closed.
Do you have an up-to-date schedule?
Yes. Schedules range from broad 1-2 year goals to month-by-month feature delivery.
Do you have a spec?
We have Business Analysts who work within our teams to define specifications and acceptance criteria.
Do programmers have quiet working conditions?
Not by default, but quiet areas are available.
Do you use the best tools money can buy?
Yes.
Do you have testers?
Yes.
Do new candidates write code during their interview?
Yes.
Do you do hallway usability testing?
Yes. UI changes are reviewed by our UX teams, and Accessibility Champions are embedded in each team.

Redefining the BBC News core experience

Fri, 22 Jul 2016 00:00:00 +1200

TL;DR: Over the last 4 years, the BBC News core experience has been transformed from a speedy 21KB page into a slow & bloated 685KB monster. This was in part due to a lack of performance monitoring and 4 years of feature creep, but also due to a lack of performance-oriented culture throughout the business.

I created a lightweight prototype of the BBC News core experience which demonstrates that focusing on the content first and foremost can result in an extremely fast page. I want the BBC and other websites to rethink what the core experience means, and experiment with giving users the power to define their own experience.

In the beginning of 2012 the BBC Responsive News team wrote about how they provide a “core experience” for users by default, and then progressively enhance the page if the browser cuts the mustard. At the time, this was cutting edge. They were able to build pages that worked on practically any browser without compromising the experience for users on modern browsers. To quote directly from the Responsive News blog:

The first tier of support we call the core experience. This works on everything. I’ve seen it work on a Nokia E65, a Blackberry OS4, Kindle 1, a HTC Touch 2 running Win Mobile 6.5, a Samsung U900 Soul, a Commodore Vic20, my nan’s slipper and a toaster just sellotaped to a TV. Likewise, GoogleBot, text-browsers like Lynx, folks that disable JavaScript and so on are all assured a good level of service.

This technique is still in use today, and is an integral part of the front-end strategy for all modern BBC News pages. In 2012 it allowed the team to provide a fast and lightweight experience for users on low-end devices. 7 HTTP requests totalling 21KB was all it took to load the core experience of the BBC News front page. All users benefited from this fast initial page load, with modern browsers progressively enhancing the rest of the page after the content was loaded.

It has been over 4 years since the BBC News core experience was first built, and a lot has changed since then. Today, the core experience consists of 91 HTTP requests totalling 685KB – over 32x heavier than the original core experience. With JavaScript disabled this can be reduced to 137KB – still over 6x heavier.

The BBC News team is aware of their website’s shortcomings. Back in May 2015 I conducted a huge performance review which I’ve spoken about extensively both internally and externally (video). A lot of work has been done over the last year, and many of the issues mentioned in those slides have already been addressed. Despite this, the elephant in the room is still the core experience.

That’s why when BBC News ran an internal hack day (where people can form teams to work on whatever they like) I took the opportunity to totally redefine the BBC News core experience.

Why have a core experience?

If you’re not already convinced that having a core experience is a good idea, I’ll attempt to sell it to you now.

Of the 3.6 billion Internet users today, over 2 billion (65%) are accessing the web from developing countries. This percentage will continue to increase over the coming years, as Internet usage in developing countries is growing over 100x faster than the rest of the world. Yes, you read that right – 100x! In India alone, over 108 million people accessed the Internet for the first time in the last year. That was a 30% increase over the year before.

Internet growth in India is accelerating at an enormous rate

Despite huge Internet growth in developing countries, device capabilities and connection speeds are not improving. A typical smartphone sold in India in 2016 has 512MB of RAM, or about one sixth of a typical 2016 smartphone sold in developed countries. Fast connections are still a precious commodity, with nearly 70% of users in developing countries relying on cellular data to access the Internet. What’s interesting is that while this number is much lower in developed countries (10-20%), it has actually been increasing –not decreasing– over the last few years.

So what does all of this have to do with a core experience?

If the majority of your users access the Internet on a mobile connection, then (hopefully) the last thing you want to do is send them bytes they don’t need. Not only can this be costly for your users (it takes 17 hours of minimum wage work in India to purchase 500MB of mobile data), it can also be costly for your business. At BBC News, the difference between a page loading in 4 seconds versus the same page loading in 8 seconds correlates to a 30% increase in users abandoning the page. This is a big deal for content providers like the BBC, and an even bigger deal for online stores.

Providing a core experience is about being able to reach users all around the world, regardless of connection speed and device capabilities. It’s about thinking beyond the USA & Europe, and welcoming the other 65% of the Internet to your website.

The definition of a core experience

A core experience should be made up of these five things, in order of importance:

The page content.
The markup required to make the page accessible.
Minimal styling to make the content easily readable – grids, typography, etc.
Minimal styling to brand the page – logo, colours.
The means to enhance the page where appropriate.

A core experience should be just these things. Everything else is an enhancement.

The way this has been achieved so far at the BBC has been to create a small core.css stylesheet containing just the styles for points 3 and 4. Then, provided the browser cuts the mustard, some JavaScript inserts an enhanced.css stylesheet into the page that contains the rest of the styling. This JavaScript is also responsible for downloading and initialising other JavaScript enhancements, like live updates and localisation services.

The problem with the BBC News core experience

Cutting the mustard to provide an enhanced experience is great from a web developer’s point of view: browsers that use enhanced.css can use modern (in browser terms) standards like CSS3. Likewise, modern JavaScript can be used without the need for polyfills or compatibility libraries. But there’s a problem with this approach: the user has no say in which experience they get.

The fact that a user has a modern browser does not mean that they have a fast or reliable connection. Providing a core experience is useless if users are going to be forced onto the enhanced experience anyway. Loading the enhanced BBC News front page on a mobile device will cost about 500KB. Opening an article page from there will cost another 370KB. That’s a cost of nearly 1MB of data (or 20 minutes of minimum wage work) just to read a single article.

The BBC News core experience on a wide screen

As well as being relatively heavy, the BBC News core experience was designed with small screens in mind. The content is forced into a single column layout, and the width of the primary image is hard-coded to 200px. It looks fine on low-resolution smartphones and feature phones, but on anything larger it starts to look stretched and at the same time somewhat empty.

How did we get here?

The BBC is a relatively unique organisation in that its digital divisions don’t always act like a cohesive unit. The various digital “products” within the BBC (like News, Sport, and Weather) are developed independently and share very little in the way of code or overall vision. If you look closely at what makes up the BBC News core experience, you might notice that a large portion of the CSS, the JavaScript, and the images on the page come from BBC products that are not BBC News.

Most of these products live in the “white BBC bar” at the top of the page. This is an area of the page which used to have a very good core experience, but has become bloated over the last year or so in order to accommodate new features. When I look through the network requests of a core page load, there are three products that stand out from the crowd:

BBC iD

The ID service allows you to sign in so that you can do things like post comments and receive notifications. It adds 13 HTTP requests to the page –8 of which are scripts– totalling 25KB.

The BBC iD button

Notifications

The notifications service is the little bell icon at the top. When you’re signed in, it will notify you when your favourite TV shows and radio programmes are updated. Notifications adds 7 HTTP requests to the page, with a weight of 37KB. 2 of these requests are scripts, and 4 are individual SVG icons.

Clicking the icon results in another 2 requests being made, including a 47KB font.

The BBC Notifications popover

Search

Search is the simple text input at the top-right of the page. Only, it’s not simple. When you start to type into the input, you’re presented with a lookahead-esque search interface. The BBC Search interface adds 4 HTTP requests to the page, totalling 14KB.

Beginning to type into the search input triggers app.min.js to be loaded in the page, weighing in at a hefty 86KB – or 320KB without compression! Furthermore, two HTTP requests at 4KB each are made on the (debounced) onchange event. Simply typing the word “bears” resulted in my browser making 10 additional requests totalling 43KB.

The BBC Search interface

Perhaps I’m trivialising it, but the answer to how the core experience became so bloated really boils down to this: feature creep, and a lack of monitoring. 4 years ago we set the standard for performance on BBC News, but we never put the controls in place to ensure that the standard stayed the same. Without proper performance monitoring, the BBC News team were completely blind to the deterioration of the website’s performance. The standard has been slipping for years, and we didn’t even know.

I cannot stress enough how important monitoring is for performance. Good monitoring enables realistic performance budgets, which in turn enable automated warnings and performance alarms. Provided there is a good culture around performance, these things can be used as leverage to prioritise performance and user experience over features and deadlines.

Good performance monitoring needs to include both synthetic testing and real user monitoring. Today the BBC uses SpeedCurve to run over 22,000 synthetic performance tests each month. We mostly use these to provide a high-level “state of performance”, and to track whether releases have any affect on page performance. We have also previously used mPulse real user monitoring, which allowed us to get a much more detailed view of how our pages were performing in the real world.

Starting from scratch

Making a slow website fast again is really hard. Sometimes there are easy pickings, like removing unused styles and scripts. Usually though, the problems are like a metastatic cancer: they are spread out far and thin, and you can’t remove them without risking damage to the rest of the system.

That’s why earlier this week, I started thinking about how I would approach the core experience for BBC News if I were given the opportunity to build it from scratch. No overhead, no requirements, no third-party products on the page.

What I came up with was a hand-rolled CSS framework using Sass mixins from Bootstrap v4. This produces about 3.5KB of CSS (1.5KB gzipped), which is small enough to inline. It also enables the markup to be fairly concise, with very few extraneous elements. From a performance perspective, I think the prototype is a success:

A 7KB HTML document containing all the core content and inlined styles – 80% smaller than the current 36KB core experience document.
3 HTTP requests totalling 39KB – 88 fewer requests and 95% fewer bytes.
100ms first paint time – 150ms faster (60%) than the current core experience.
460ms first paint time – 440ms faster (50%).
A much better CPU profile.

The CPU profile of the lightweight prototype

The CPU profile of the current BBC News core experience

While these metrics are interesting, the real proof is in how the prototype performs on a simulated 2G connection: the first paint time is 4.5 seconds, and it is fully loaded at 8.5 seconds. For comparison, the BBC News core experience has a first paint time of 19.5 seconds, and is fully loaded at 52 seconds.

I think the prototype is also an improvement on the user experience. The layout is fully responsive and works at all device widths. It also supports responsive images through the use of srcset, with a sensible medium-quality src fallback.

Here is the finished product on a wide screen:

The lightweight core experience prototype

As well as creating a truly lightweight core experience, I also wanted to think about how we can put people back in control of their own web experience. We can give people a core experience when we think they need it –for example by detecting screen width or connection speed– and then give them the power to enhance the experience for themselves, should they want to.

I’ve started to explore these ideas by implementing two controls:

A button that (lazily) loads the remaining images on the page. This requires basic JavaScript support, so can be used by most users.
Buttons to prefetch article content. This requires a relatively modern browser with service worker support.

I think there’s certainly more to explore in this area, especially around minimising the amount of interaction required to receive the optimal or desired experience.

If you’d like to inspect it further, the code for this prototype is on GitHub, and a live demo is available at https://wildlyinaccurate.com/news-core-experience/.

Where to from here?

After all of that, the point I really want to make is this: for a website like BBC News, content is the only thing that matters.

With current trends, very soon the majority of Internet users will be browsing the web on a mobile connection. The lucky ones will have 3G-like speeds, and the others will be lucky to even get 2G-like speeds. As website producers, we need to make it easier for these people to get to the content that they want. By far, the easiest way for us to do that is to give them just the content. Nothing else.

BBC World Service is expanding to reach a wider audience by 2020, and most of the users in that audience will come from developing countries. This is a fantastic opportunity to reinvent the BBC’s web experience and tailor it for the users we have (people using underpowered phones on mobile connections), rather than the users we want (people using MacBooks on cable connections).

At BBC News we understand this, and we’re in the middle of building our front page from the ground up with performance at the front of our minds. We’re using the small & efficient Grandstand CSS framework developed by our colleagues in BBC Sport. We’re also investing in performance monitoring up-front, and trying to drive performance as a key feature rather then an afterthought.

With HTTPS everywhere being rolled out across the BBC later this year, the BBC’s web teams have a whole new set of opportunities before them. One of the things I’m most excited about is the ability to start using the service worker API to provide a better and more resilient web experience.

Using A/B testing to prioritise performance optimisations

Fri, 20 May 2016 00:00:00 +1200

Back in December 2015 I spoke at LDNWebPerf alongside Peter Chamberlin about how web performance is not a technical problem (slides with speaker notes). One of the things I talked about was how we used multivariate testing (MVT) at BBC News to prioritise performance optimisations. The gist of it was that our stakeholders had already bought into the idea that performance has a strong correlation to business metrics, and they wanted to dedicate some development time to improving performance. The catch was that they didn’t want to spent too much time on it.

Our predicament, then, was that we needed to know which optimisations had the biggest impact on performance without actually spending the time to make the optimisations. For example, we had a hunch that inlining the critical rendering path CSS would improve our start render time, but with over 1MB of CSS and a complicated application architecture, implementing this was much easier said than done.

This is where the idea to A/B test performance came from: we could easily make the performance optimisations by hand on a single page, and then benchmark each of the optimisations to find out which had the biggest impact.

The method

Setting up A/B tests for performance is much simpler than you might think — all you need is a page to use as the baseline, a page for each optimisation, and a consistent way to measure performance.

We took a snapshot of the BBC News front page to use as the baseline, and made a variation of that snapshot by hand for each performance optimisation. The focus of this post isn’t the optimisations themselves, but to give some context, they were things like:

Lazy loading images below the fold.
Delaying JavaScript execution until after the content has rendered.
Removing unused CSS.
Loading adverts asynchronously.

Some of the hand-made performance optimisations

We deployed these optimised variations alongside the baseline, and used WebPagetest to benchmark their performance. If you’re not familiar with WebPagetest, it’s a really useful tool to have in your web performance toolkit. It loads a page in real web browsers and records information about its performance. It has a bunch of useful features like connection throttling, video capture, and also lets you script interaction with the page.

The WebPagetest interface

One of the other features that WebPagetest offers is the ability to visually compare two pages. We used this a lot while we were benchmarking the variations because it’s great way to show the impact of optimisations to non-technical stakeholders.

Visual comparison of the baseline snapshot against an optimised variation

Interpreting the results

Interpreting the results of A/B tests is usually very circumstantial and requires some forethought about your own goals and circumstances. Our goal was to identify which performance optimisations gave us the largest improvement to start render time relative to how long they took to implement. We chose this goal because data from our real user monitoring (RUM) suggested there was a strong correlation between start render time and business metrics like bounce rate, session length, and pages per session. Your goal will likely be completely different.

To simplify the decision making process, we came up with a scoring system that works by multiplying the improvement to start render time by a difficulty rating:

Optimisation	SR improvement (ms)	Difficulty (1 = hardest; 10 = easiest)	Score
Only load JS required for page	250	8	2000
Only load CSS required for page	1000	2	2000
Remove blocking scripts from <head>	1500	1	1500
Lazy load images below the fold	80	10	800

It’s by no means a perfect system, but it did help us to prioritise the optimisations. There are many other factors that come into play, for example lazy loading images is one of the first optimisations we made, despite it having a low score. The rationale behind this was that it was very easy to do and had a noticeable real-world benefit: it reduced the amount of data for users on mobile connections.

Validating the results

The final part of an A/B test is to validate whether the variants had the predicted impact in real world conditions. For performance optimisations, this is best done using real user monitoring like SpeedCurve LUX or SOASTA mPulse. However synthetic testing tools like WebPagetest can also work to validate your optimisations.

We plotted our performance and business metrics around the time that our optimisations were deployed, and (thankfully) saw the results that we expected.

A chart showing a reduction in page load time and an increase in session length

A chart showing a reduction in load time, DOM ready time, and DOM content loaded time

Wrapping up

Taking the time to A/B test performance optimisations can help your team prioritise work and ensure that you don’t waste time making optimisations that have no impact. You don’t need to use complex or expensive software to get meaningful results from your tests — it’s probably sufficient to make optimisations by hand and measure the results using free software like WebPagetest or your browser developer tools.

How can we fix open source culture?

Mon, 28 Mar 2016 00:00:00 +1300

The recent kerfuffle around the NPM #unpublishgate and the Greenkeeper bot impersonation has got me thinking about the open source community and its culture.

Sometimes the open source community feels like a wonderful, cooperative, welcoming place. There have been times when maintaining an open source project has given me an enormous sense of satisfaction and well-being. On the best days, complete strangers offer valuable feedback and even actively contribute to my projects.

On the worst days I feel drained, unappreciated, and even abused. Stephan describes this more concisely than I could right at the bottom of Your “just” considered harmful:

The reactions to the npm #unpublishgate showed me once more just how far spread entitled and toxic behaviour is in our community. This has to change and being silent or accepting won’t help.

This is the part of the open source culture that we need to fix. Entitled and toxic are not words that I associate with welcoming and inclusive communities. Yet they are completely apt descriptions of behaviour which is common within the open source community.

I don’t have any solutions to offer. I’m merely venting some frustrations which have been building up for quite some time. But we need to fix this. I don’t want to see this toxic behaviour cause another friend, colleague, or community member to suffer.

How can we fix open source culture?

Wildly Inaccurate

Web performance review: TradeMe

Defining the test parameters

The benchmark

Identifying performance bottlenecks

Main thread activity

Delayed primary image request

Some performance experiments

Prioritise loading the primary image

Defer loading third party assets

Render page content on the server

Causes for celebration 🎉

Caching improves the experience on repeat visits

Code splitting is already implemented

Recommendations

Test on representative devices and networks

Render the core content on the server

Preload important resources

Reduce the amount of JavaScript you ship

Use performance budgets

Wrapping up

Becoming a team lead: a survival guide

Introduction: the accidental leader

Part I: Before you apply

Understanding the team lead role

Setting yourself up for the role

Do some research

Fix things that cause pain for the team

Take every opportunity to mentor

Focus on the bigger picture

Be present

Part II: You got the job!

Avoid being overwhelmed

Stop writing code

Learn to delegate

Say no

Ask for help

Conducting effective one-on-one meetings

Have them regularly

Use a template

Take notes

Prepare beforehand

Ask and listen

Don’t over-promise

Dealing with pressure from stakeholders

Always be honest

… Unless you’re ahead of schedule

Compromise early & often

Don’t pass the pressure onto your team

Ask for help

Grow & support your team

Allow pet projects

Encourage pairing

Run knowledge sharing sessions

Spend time together

Remember: you are a team!

Further reading

Introducing Second: a framework for mostly-static React applications

TL;DR

Why build this framework?

How does it work?

Server-side rendering

Data-driven components

Partial, opt-in browser rendering

What next?

Introducing a faster BBC News front page

Quick facts about the new front page

What’s next?

Performance improvements

Design enhancements

React in the browser

Acknowledgements

Footnotes

Web development technologies to adopt in 2017

Preact — preactjs.com

Webpack — webpack.js.org

offline-plugin — github.com/NekR/offline-plugin

Tachyons — tachyons.io

Lighthouse — github.com/GoogleChrome/lighthouse

How we assemble web pages at BBC News