Generating value from smart meter data

Making the most of the smart meter roll-out

Project duration: December 2011 to December 2012

By 2020, smart meters will replace all existing electricity and gas meters, providing a means of automatically recording and communicating energy consumption data to the energy supplier.

Smart meters will generate a vast amount of data. But how can this be used to benefit both the industry and the consumer?

That’s the key question that this project set out to answer, part-funded by the Technology Strategy Board through their 'Harnessing Large and Diverse Sources of Data' competition for research and development funding.

Why the industry needs smart meter data

For more than half of the electricity consumed in the UK market, we only have meter readings which are taken, on average, every six months. Not only is this frequency of data incompatible with the way electricity is traded in the market (which operates on a half-hourly basis), but these readings provide no information about the amount of electricity being used at different times of day.

Any small improvements in understanding patterns of electricity demand could unlock significant economic value, which would benefit both the industry and consumer. The advent of a fully smart-metered electricity system is the first step. But making these improvements is also heavily dependent on our ability to store, manage, process and extract useful information from the smart meter data. This project aimed to address these gaps through building a prototype system capable of managing this class of 'Big Data' and developing novel techniques for analysing and extracting patterns from the data.

Project manager Joshua Thumim explained:

“The electricity industry is reliant on balancing electricity generation and demand – and thus being able to predict peak-demand periods to avoid distribution network failures. To do this, they currently have to make a series of generalisations about the patterns of demand of different consumers to address potential gaps. With an ageing infrastructure, an ongoing investment in renewables and a need to shift demand so it is spread more evenly over the day, smart meter data could prove invaluable.

“The challenge for CSE was to extract, analyse and present this data in a way that can be done promptly and cost effectively."

Crunching the numbers
Programmers from CSE and the University of Bristol set out to develop a new computational system that would allow the extraction of commercially valuable patterns from smart meter data. They came up with a prototype 'Big Data' platform called 'Smart Meter Analytics, Scaled by Hadoop' (SMASH).

SMASH is a highly scalable distributed file system and parallel computing platform which can store, process and retrieve data from very large datasets – the team tested it with datasets up to 20 Terabytes in size (equivalent to smart meter data from about 13 million households per year).

In parallel, a data mining team from the University of Bristol applied new, experimental techniques to a sample of real smart electricity meter data to identify interesting subgroups of consumers with statistically different consumption patterns. Scottish and Southern Energy and Western Power Distribution were partners in this project, providing an invaluable perspective from the electricity industry, and confirmation that the tools and techniques being developed had real business relevance for them. One key application they identified was that the tools would enable energy suppliers and District Network Operators to generate more accurate profiles of consumption, where before they were forced to generalise.

While developing SMASH, the project team also overcame many technical hurdles to produce an innovative web-based user interface, making it easier for industry clients to use and interpret the results.

Benefits for infrastructure, the environment and the fuel poor
Being able to manage and analyse smart meter data using this system has many potential benefits. It could be used to develop flexible time-of-use energy tariffs, encouraging consumers to use energy at 'quieter' times of day, rather than peak times. If electricity consumption could be spread more evenly throughout the day, peak demand could be brought down which would reduce the amount of reserve capacity required. This would increase the efficiency and reliability of the system, leading to a better service, and benefitting the environment.

Improved tariffs and services based on the analysis of electricity demand profiles also have the potential to help vulnerable and fuel-poor households save money on their bills.


This project was a partnership between CSE, Western Power Distribution, Scottish and Southern Energy, and the University of Bristol.

For further information contact:

Joshua Thumim | 0117 934 1439

Stay up to date with our work - sign up for our fortnightly newsletter