Managing and ’mining’ smart meter data at scale

CSE Project Showcase seminar in Westminster

8 July 2013

At a well-attended seminar at One Birdcage Walk in Westminster, CSE's Joshua Thumin and Tom Wilcox demonstrated the techniques CSE has developed to harness the value of smart meter data – at scale.

Who was there?
Attendees at this free event came from over 30 leading organisations from the energy and technology sectors, including IBM, DECC, British Gas, Telefonica O2, Ofgem, Technology Strategy Board, National Physical Laboratory, National Grid and Elexon (full list below).

The event was chaired by Judith Ward, who leads Sustainability First’s GB Electricity Demand Project and featured a 'drive-through' of our new Big Data platform ‘SMASH’ (Smart Meter Analytics Scaled by Hadoop).

SMASH was created by CSE in partnership with the University of Bristol, Western Power Distribution and SSE, and part-funded by the Technology Strategy Board’s ‘Harnessing Big Data’ programme. The 18-month project sought to develop and prove techniques to

  • Manage smart meter data for analytical purposes, taking account of the huge volumes of data that will result from the routine use of smart meters
  • 'Mine' the smart meter data to find statistically valid new customer profiles, and enable real-time data aggregation for user-defined customer groups via the internet

At the seminar we described and demonstrated the highly scaleable distributed file system and parallel computing platform which we developed to store, process and retrieve data from a very large smart meter dataset (tested up to 20 Terabytes to date).

Plus we shared results from our data-mining, demonstrate the prototype web interface and explore potential applications.

You can download Joshua Thumim's presentation (3MB Powerpoint presentation) here.

And you can read some background to this project here.

CSE's Joshua Thumim, who led the project team, said: “It’s great to have had this opportunity to share what we’ve learned so far about managing and mining the massive datasets that will emerge from the smart meter roll-out. We’re very interested in exploring how this work can be taken forwards in collaboration with other experts.

"A key challenge is to make sure that there are effective techniques developed for extracting the value from all that smart meter data – and that these are applied to serve consumer needs and the public interest in a smarter, low-carbon energy system.”


Organisations attending the seminar
Ablescroft EI, British Gas, British Gas Connected Homes, Cambridge Architectural Research, DataBarta Ltd, DECC, EDF Energy, Electricity North West, Energy & Climate Change, Energy for London, Energy Saving Trust, Frontier Economics, Hitachi Europe, IBM, Inside Out Data, Institute for Sustainability, Low Carbon London, Low Carbon Networks Fund , Mandarin Research, National Grid, National Physical Laboratory, NESTA, Ofgem, Power Plus Communications, Scottish & Southern Energy, Strathclyde University, Sustainability First, Technology Strategy Board, Telefonica O2, UCL Energy Institute, Utility Partnership Limited, Western Power Distribution.


This was the first in a new series of CSE Project Showcases in which we aim to share our experience and learning from key projects, and explore how this learning can be valuable to others in their work.

Stay up to date with our work - sign up for our fortnightly newsletter