Data is the New 'Oil': Driving Data Ecosystems in the Energy Sector

The energy market is changing exponentially – as is the data landscape that underpins this rapid and unrelenting disruption. This was underlined in the Energy Security Board, NEM Data Strategy Consultation Paper, which stated “…data will play a crucial in supporting future security, reliability and affordability as the system moves to more variable, decentralised generation and innovation drives new technologies and services.”

Clearly, businesses of every size and in every sector are looking to capitalize on their data assets, with the knowledge that actionable, real-time data underlies their ability to drive efficient markets. However, the regulatory, business and technical barriers to supporting industry and organization, digital and data transformation, particularly while using conventional approaches, seems impossible – too cumbersome, complex and prohibitively expensive. That said, there is a solution in sight.

Efficient markets need real-time data interoperability

For an evolving market to perform more efficiently and innovatively, removing data friction (equating to inaccurate and untimely data, incompatible data formats and security controls) and increasing transparency is an imperative. Solving this with optimized, seamless data interoperability between DER (Distributed Energy Resources), DES (Distributed Energy Systems), Operational Technology and IT systems in real-time is the goal.

Organizations struggle to securely integrate data within, let alone between, organizations and entities

Organizations are increasingly understanding the value of data for delivering seamless end-user experiences, supply chain integration and driving innovation and new business models. But, they face the challenges of:

Myriad systems, processes and applications, legacy and new technologies, all filled with data and none of it working together;
Traditional and current technology systems which aren’t built to deal with the exponential increase in unstructured data;
An inability to create a real-time, unified view of a customer or a company;
Increasing demands to securely share data within and between organisations, entities and technologies in real-time; and
Data security, cyber security and data regulatory compliance issues.

In addition to this, the realities of business complexities include, but are not limited to:

The integration of data following mergers and acquisitions;
Business partners in one venture being competitors in another;
Third-party vendor networks;
Non-linear relationships; and
Blurring of the boundaries between customer and supplier

This means that companies are spending significant money and time in their, more often than not, unsuccessful attempts to manage and control their data. A recent IDC's Data Integration and Integrity End-User Survey indicates that, currently “80% of time is spent on data discovery, preparation, and protection, and only 20% of time is spent on actual analytics and getting to insight.” - IDC PlanScape: Data Intelligence Software for Data Governance

The new energy value chain is a complex web of inter-relating data points

Specific to the energy sector, there are far more players in the no-longer linear, energy value chain; with buyers and sellers interacting in a complex web which is, to date, not being capitalized on.

This is an ecosystem of exponentially expanding connections and digitized data points; with supply and demand requirements from a distributed network of energy producers, consumers and prosumers, along with an increasing insistence for transparent service switching provisioning.

Increasingly complex, sophisticated and advanced demands are placed on organizations

All this comes along with demands for:

Hybrid products and services merging between energy organisations and smart home technology providers;
Hybrid-multi-cloud Governance, Risk and Compliance (GRC) platforms for unified governance and regulatory compliance;
Transparent, end to end energy consumption data and demand patterns to enhance energy requirements planning and management across fragmented cloud services; and
Digital transformation through disruptive technologies - which that are not limited to Quantum Computing, Artificial Intelligence (AI), distributed ledgers, Automation, Robotic Process Automation (RPA), cloud services, mobile devices, wireless networking and Internet of Things (IoT) - all of which must, optimally, work securely and to scale with embedded, decentralised, legacy, latest and future-facing operational and IT systems.

A recent McKinsey & Company report, Critical infrastructure companies and the global cybersecurity threat, found, “40 percent of industrial sites have at least one direct connection to the public internet, and 84 percent of industrial sites have at least one remotely accessible device.” Given the exponential increase in unstructured/semi-structured data from smart sensors, IoT and connected devices as well as the modestly growing amount of structured data from business systems, the headaches arrive in attempting to achieve actionable, real-time insights, through decentralised data assets, at the point of interaction and demand. The need for unified identity and device authorization and access management cannot be underplayed here as, without them, the ability to control critical assets is severely curtailed.

To meet new market demands, data must be seamlessly combined in real-time

The solution lies in: forming data ecosystems that seamlessly integrate the fragmented and dynamic nature of devices, systems and applications both within and between organisations and technologies; and in enabling the data within these ecosystems to be combined, in real-time, to create a unified data view for actionable use - specifically personalised to the individual person/role/company/component that is requesting the data, regardless of data format or what system it came from.

These distinct, unified data views need to be updatable (with permission) in real-time, as the demand is presented and solution expected, in real-time. This two (or more)-way data-flow within the ecosystem, enables a new breed of distributed applications and business models to be created, while transparently dealing with data complexity, compliance, security, provenance and integrity issues.

The provision of chain-of-evidence for a) the provenance of the data, b) the access controls rules around the data, c) automatic audit record generation for all data access and d) highly secure business/transactions/logging will provide a secure, trusted unified data view which can then be utilised by auditors and compliance monitors to empirically improve risk management.

Real-time actionable data insights power the digitized energy sector

As we have seen from the massive investments organisations are making in data lakes and associated analytics/Machine Learning /AI – the refinement of data is where the value is. To capitalise on this requires taking the ‘oil’ of data and refining it into distinct, real-time, actionable data insights which will drive innovation, efficiencies, integrated supply chains, streamlined services and new business models and opportunities along with enhanced, end-user engagements.

The scale and complexity of this challenge leaves traditional approaches stuck at the starting line. Distributed ledger alternatives have managed to begin the journey, but are incapable of dealing with the volume and real-time nature of the challenges; thus making them, ironically, unsustainable for the energy sector given the associated energy consumption they incur.

Given the challenges listed in the introduction of this article, a new engine had to be invented. It has been, and it is now on the road.