• I'm a full stack developer at Fasterize
• twitter : @AnthoBarre

• 
• Fasterize is an all-in-one website accelerator.
• It is an automated web performance optimization service (SAAS) used by many high traffic websites.
  Fasterized websites are already SPDY and SSL enabled and take fully part of the next gen web.

The testing is run on servers in datacenters using a throttled connection to try and mimic the conditions an average user might experience. Pages are loaded in real web browser and performance metrics collected.

It works by measuring how much of the above-the-fold content is visually complete over time until it is 100% complete. Smaller score is better.
Best correlation to the user exeperience (render time)

The purpose of a performance budget is to make sure you focus on performance throughout a project.

Integration : API for webpagetest (node library) | grunt-perfbudget

Answers the question : how fast should I be ?

How bad is the UX if a third party is down ?

A javascript is injected on each page and reports on the page load data for every request that is made for each page.
There’s no need to pre-define your important use cases.

RUM tools leverage the Navigation Timing.

The Navigation Timing is not implemented by Safari (IOS < 9) and IE <= 8

The Resource Timing API allows you to collect complete timing information related to resources in a document.

Resources fetched from a third-party must provide an additional HTTP header (Timing-Allow-Origin: *) to allow the site to gather detailed network timing data.

The script can capture conversion rate and other business data.

“Synthetic monitoring shows you how you relate to your competitors, RUM shows you how you relate to your customers.” – Buddy Brewer (Soasta)

WPT : take the median (of the SpeedIndex) of 10 runs

RUM : remove all the noise that comes with the measure

Steve Souders compared the data from a RUM tool and Webpagetest on his website.

	Chrome 23	Firefox 16	IE 9
Synthetic First View (secs)	4.64	4.18	4.56
Synthetic Repeat View (secs)	2.08	2.42	1.86
Synthetic 50/50 (secs)	3.36	3.30	3.21
RUM (secs)	9.94	8.59	6.67
RUM data points	94	603	89

Conclusion : be cautious about only showing your synthetic page load times – the real user experience is likely quite a bit slower.

Note : At the end of the day the two sources of numbers are different and have very different purposes. Neither of them is wrong or right – they are just different tools.

Correlation Business / performance

Test A/B are better than test before / after.

Fasterize will optimize only pages of a given population.

Analyse an impact on business with enough data (months may be required)

No, RUM and Synthetic can measure only the first page load and can not measure secondary loads via AJAX.

You should consider the User Timing API.
Modern synthetic and Rum tools can collect those metrics.

window.performance.mark('mark_fully_loaded');
window.performance.measure('measure_load_from_dom', 'domComplete', 'mark_fully_loaded');
var items = window.performance.getEntriesByType('measure');