Automated Data and Machine Learning Pipeline for Cost-effective Energy Demand Forecasting in Sector Coupling

Challenge
Energy demand forecasting is the foundation for efficient sector coupling, and it is a fundamental business problem in the energy sector. High accuracy of energy demand forecasting is essential to realizing sector coupling solutions.

Increasing the accuracy of a forecasting model is difficult and highly depends on the quality of training data, the selection of machine learning algorithms, the speed of training processes, and the evaluation metrics.

The current State-of-the-Art methods are based on expert knowledge or prior experiences to select the data initially and data analysis using correlation coefficients or similar methods which are often labor-intensive and time-consuming.

Solution
Therefore, automating the data pipeline that can identify and select the most relevant data, and train the AI models will reduce time and labor costs dramatically.

This project aims to develop, test and deploy a data and machine learning pipeline that can automatically select relevant input data, best-fit machine learning algorithms and evaluation metrics for creating energy demand forecasting models and evaluating their performance.

The project will utilize the open-source OpenDataHub platform from Red Hat. The OpenDataHub platform can be installed locally and allows the developed pipeline to be deployed and tested on-premises. The OpenDataHub platform is designed to be easily scalable for future scenarios and does not require partners to develop any new software infrastructure to support the project, only task-specific extension modules related to the energy domain need to be developed and added to the platform.

This project is a continuation of the Lighthouse South phase one project “AI-based forecasting framework for sector coupling between electricity grids and district heating” and will integrate the experiences learned from the manual development of AI-based forecasting models into the creation of the automated data pipeline.

Espected result/effect
The automated data pipeline will be available to industry partners after the project ends.

Inuatek A/S: The developed solution is expected to reduce labor costs and increase the efficiency of data processing and analysis which are highly demanded by its customers. Therefore, it will create additional sales of services. Hence, Inuatek A/S expects to increase its annual service sales and decrease costs equivalent to 1 million a year after the project ends, and 4 million after 3 years.

Picca A/S: The developed automated pipeline service is possibly integrated into Picca A/S existing automation solutions for the energy and water sector and part of the company’s data interoperability and system integration services in the energy sector. Hence, Picca A/S expects to increase its annual service sales to 1 million a year after the project ends, and 3 million after 3 years.

The proposed automated data and machine learning pipeline will reduce costs and increase efficiency significantly compared to manual data pipelines.

It costs around 3,6 million DKK a year for a team of data engineers to build and maintain data pipelines3. Roughly 80% of an average data engineer’s time is spent constructing data pipelines4. Automated data pipelines can be 90% faster and significantly reduce time compared to manual data pipeline creation and maintenance. The global data pipeline tools market size was valued at 49 billion DKK in 2021 and is estimated to expand at a CAGR of 24.5% from 2022 to 20306.

According to TREFOR’s estimation, grid reinforcement will cost TREFOR 6,25 billion DKK to update all its networks.  Highly accurate energy demand forecasting can be used for congestion management and will significantly reduce the grid overload and cost of grid reinforcement.

Problem owners

• Assens Forsyning
• DinForsyning
• Sønderborg Forsyning (SONFOR)
• TREFOR Infratruktur

Problem solvers

• Syddansk Universitet
• Inuatek
• Picca