Why BEIS and Ofgem will need smart energy data to do their jobs

25 July 2019

Access to detailed anonymised smart meter energy data will fast become a necessity for Ofgem and BEIS and other public interest actors as they design the policies and regulations to deliver smarter energy markets. CSE’s Simon Roberts explains why.


Access to smart-meter data for public-policy purposes is essential to avoid policy makers ‘flying blind’ into the energy transition. This is the lead conclusion of the first phase report of the Public Interest Advisory Group on smart meter energy data, which we co-convened with Sustainability First, published on 22 July 2019. The report can be downloaded here.

This multi-stakeholder, deliberative exercise over the last 20 months has explored a wide range of issues, producing eleven detailed stimulus papers to underpin the final findings. All of the papers can be downloaded at www.smartenergydatapiag.org.uk.

PIAG’s work included identifying a range of use-cases for anonymised smart meter energy data to be realised in the public interest. These use-cases provide thoughtful answers to the questions ‘what could we do with smart meter data in the public interest?’ (or ‘what could we do better?’). But we also needed to ask the question ‘what won’t we be able to do if we don’t have access to smart meter data to serve the public interest?’

Considering what will not be possible without access to anonymised smart meter data for public interest actors is instructive. Not least because this is the current path being taken. Unlike every other country with smart meters, the UK has no central repository or database of smart meter data. And the government has no plans to access or capture the data for their own policy-making needs or any wider public interest purposes (though see PIAG Stimulus Paper 7 for how this might be done).

As explained below, the analysis done by Sustainability First and CSE with PIAG reveals what could become significant shortcomings in the efficacy of policy-making and regulatory practices, the effectiveness of local infrastructure planning and the quality of innovation ecosystems for smart energy. These shortcomings risk creating substantial detriment.

This is because the PIAG use-cases from anonymised smart meter data are not simply the ‘nice to have’ extras - the ‘icing on the cake’ - for those keen to have access to better data ‘because it’s there’. The public interest use-cases include important and valuable activities, the absence of which could prove serious as the energy system gets smarter. Indeed, the use-cases are important and perhaps vital applications of smart meter data in the public interest precisely because smart meters have been installed.

Smart meters enable changes which create the need for public interest access to smart meter data

Smart meters are part of the enabling technology for an involving, consumer-oriented, ultra-low carbon smarter energy system. The data which smart meters provide is a necessary component of the smarter energy system as envisaged by government, regulators and many other stakeholders. The data which smart meters provide to enable smarter system operation therefore inevitably has the additional effect of:

  • Changing the context in which policy making needs to take place and in which regulators need to design and oversee markets and regulated practices for the smarter energy system.
  • Altering the nature of the practices and technical infrastructure required to optimise system operation nationally and locally.
  • Stimulating the need and opportunity for innovation to realise in full the anticipated benefits of a smarter energy system and to reduce the costs of achieving them.

So the very introduction of smart meter data into the system to enable the system to become smarter itself creates conditions:

  • Where system design and governance will need to use temporally fine grain (e.g. half-hourly) data to understand fully the impact of its proposed policies and regulatory practices (particularly how the costs and benefits distribute between different types of consumer).
  • Where system planning will need to use temporally and spatially fine grain data to ensure infrastructure planning and service development planning is based on a refined understanding of local and national system characteristics.
  • Where innovations to improve the system (and potentially address otherwise negative consumer impacts) will need fine grain data to develop and test their approaches prior to release.

Of course, a range of commercial interests, such as energy suppliers, will have access to this smart meter data in line with a carefully designed set of privacy rules and customer opt-ins and opt-outs (the Data Access and Privacy Framework). Others, such as network companies and the settlement system, will have controlled access to enable them to fulfil the functions they serve within the energy system and its associated markets. This access will enable them to design their own operations; it will also enable them to use the data to support their own innovation.

It is in the public interest for them to do so, provided the design and regulation of the system and its associated markets create decent alignment between their commercial interests and the public interest. This design and regulation also need to ensure that the incumbency of these interests does not give them an inappropriate advantage over others currently ‘outside’ the system who may have better ideas which, if realised, could generate greater benefits, better serve consumer needs, further reduce system costs and/or enable participation by a wider range of consumers (particularly the more vulnerable). 

The emerging information imbalance between the regulator and the regulated

A serious information imbalance will occur if that fine grain smart meter data is not available to any of the following: (a) those regulating these commercial interests in this changing system; (b) those designing policies and markets within which these interests act, and; (c) those seeking to innovate or respond to market needs or opportunities as an ‘outsider’.

This information imbalance would create a situation in which those, particularly BEIS and Ofgem, with responsibility to represent and reflect and regulate for the public interest in the energy system are having to operate and make decisions as if they had their eyes closed. For example:

  • Without access to fine grain smart energy data, BEIS and Ofgem are likely to find it particularly difficult to understand the distributional impacts of proposed policy, regulatory and market design choices (including charging regimes, licence conditions etc) in a smarter system with a wide range of time of use tariffs, local energy trading, flexibility and storage markets etc. Similarly, they will struggle to track the distributional impacts of emerging practices by licensed and unlicensed actors, from retailers to aggregators to network operators.

As a result, there is a serious risk of regulators and policy-makers making poorly-informed decisions which fail to drive market behaviour as intended and which create unforeseen but unfair outcomes. Without the data, their ability to understand how to promote the public interest within policy design and regulatory practices will be severely compromised in the face of commercial interests who do have access to the data.

Limitations for local energy planning and investment

The lack of access to smart meter energy data will also limit the potential of those seeking to plan local energy infrastructure to respond to new conditions and opportunities. For example:

  • Without access fine grain smart energy data, a local authority like Bristol which is seeking to implement a significant 10-year programme of £1bn of energy infrastructure investment in the city – from heat networks to storage to renewable energy – for the benefit of its citizens, will either have to depend on low grade modelled data or be dependent on incumbents who do have the data but whose interests may be poorly aligned with those of the city.

Favouring incumbents by sustaining high barriers to entry for early stage innovators

Having effective innovation to improve services and lower costs is in the public interest and, as explained in PIAG Stimulus Paper 5, could justify establishing a ‘test-bed’ of representative, anonymised smart meter energy data in the public interest. Without such an innovation test-bed, there will be higher barriers to entry for early stage innovators. They will be unable to secure the low cost access to smart meter data enjoyed by incumbents to test their prototype innovations. For example:

  • Without access to fine grain smart energy data, innovators at the early stages of designing new smart energy solutions will either have to bear the considerable costs of recruiting customers or finding an academic partner who has a dataset (without knowing if their idea is potentially viable in the ‘real world’) or rely on developing a partnership with an incumbent (which can be a dampener on positive but disruptive innovation). These are barriers to entry. As the promoters of open data routinely highlight, the best innovation occurs when the data is made available without cost so that barriers to entry are as low as possible.

Much of the discourse in PIAG has rightly been about how the public interest could be positively served by enabling the use of fine grain smart meter data to realise the use cases. But our analysis also suggests that the public interest risks being significantly undermined if, as currently planned by government, that fine grain smart meter data is not made available as described.

As PIAG concluded at the end of its first phase, the very introduction of smart meter data into the energy system changes the data that public interest actors need to have available to them so that they are able to shape and regulate and manage that smarter energy system in the public interest. That will be a central focus of the second phase of PIAG which gets under way in September.

Stay up to date with our work - sign up for our fortnightly newsletter