KR20220039325A - Apparatus for analyzing meter data of ami - Google Patents

Abstract

Disclosed is an apparatus for analyzing the meter data of an AMI The apparatus comprises: an OLAP spark cluster for profiling, validating, and aggregating meter data from an MDMS system in a distributed parallel manner; an online query module connected to a user terminal based on the Web to extract, analyze, and visualize the meter data stored in the MDMS system; a professional client analysis module which accesses the MDMS system, provides a script to a developer terminal connected thereto based on the Web, and provides an API for advanced analysis of the meter data to support sending and receiving files; and a connection module connected to an operator terminal based on the Web to provide Hadoop data of the MDMS system to other systems and perform configurations. Therefore, users can analyze big meter data without additional coding.

Description

AMI weighing data analysis device

The present invention relates to an AMI metering data analysis device, and more particularly, an online query type analysis tool in a WEB environment, an expert analysis environment, and other systems so that the AMI metering data collected through an intelligent power metering infrastructure can be directly analyzed. It relates to an AMI weighing data analysis device that can provide data in connection with the AMI.

Recently, management of metering data collected according to the construction of an advanced metering infrastructure (AMI) is required.

Since metering data is fundamental data for creating a new energy industry, such as calculating electricity rates, providing customers with real-time usage and rates, and analyzing Demand Response (DR), high reliability is required in the management of metering data.

Currently, about 10 million units have been built as a system for managing metering data, and the head office metering data management system (MDMS: Metering Data Management System) collects, processes, and manages nationwide metering big data.

We are actively promoting AMI construction overseas as well, and AMI measurement data is not only a standard for charging, but also electricity quality (voltage, current) management, violation/conduction/loss search, electric load pattern analysis by region/section, and demand It is important data that is the basis for various tasks and services in the distribution/sales/customer service fields such as forecasting, real-time electricity usage information customer provision, and electricity consumption consulting.

The background technology of the present invention is disclosed in Korean Patent Publication No. 10-2018-0033961 (published on April 4, 2018, a weighing data management system and a computer-readable recording medium).

In recent years, the role of MDMS big data analysis solutions and services to increase the utilization value of AMI measurement data is growing in line with the increase in the supply of AMI and the 4th industrial revolution technology such as big data and artificial intelligence (AI).

A data analysis and visualization solution that can create new value by comprehensively analyzing AMI data is needed in order to enhance the investment effectiveness of the AMI construction project and to strengthen the technological competitiveness of overseas exports.

KEPCO MDMS applies big data technology to load and process weighing data in Hadoop file system, NoSQL database, and relational database (RDB) depending on the purpose and purpose.

However, the amount of data of 10 million AMI instruments collected from MDMS is more than 600 million cases (450 GByte) per day, and it is difficult for business managers, managers, data scientists, and algorithm developers to access and analyze data in a general way.

The present invention has been devised to improve the above problems, and an object of the present invention according to one aspect is to directly analyze the AMI metering data collected through the intelligent power metering infrastructure. It is to provide an AMI measurement data analysis device that can provide data in connection with an expert analysis environment and other systems.

AMI metering data analysis apparatus according to an aspect of the present invention, OLAP spark cluster for profiling, verifying, and aggregating metering data from an MDMS system in distributed parallel; an online query module that is connected to a web-based user terminal and extracts, analyzes, and visualizes weighing data stored in the MDMS system; A professional client analysis module that provides a script by accessing the MDMS system for a web-based connected developer terminal, and provides an API for advanced analysis of weighing data to support sending and receiving files; and a web-based connection module for providing and setting Hadoop data of the MDMS system to other systems by being connected to an operator terminal.

In the present invention, the online query module includes: a small relational DB for managing a model for attribute information on an analysis target attribute in the form of metadata; a data model management unit that manages a model for attribute information on a small relational DB or HDFS source's analysis target attribute through a small relational DB; a weighing data inquiry unit for inquiring weighing data in a data model format managed by the data model management unit; an analysis visualization unit that analyzes and visualizes the weighing data retrieved through the weighing data inquiry unit in parallel through an OLAP (Online Analytical Processing) spark cluster; and an environment setting unit for setting an environment for the user terminal to access the data model management unit, the measurement data inquiry unit, and the analysis visualization unit.

In the present invention, the metering data inquiry unit selects a data model, stores a column of the data model and a search time range, and inputs a search condition through a search query.

In the present invention, the search query term is characterized in that data is filtered through a where conditional clause and an operator.

In the present invention, the weighing data inquiry unit searches by using one or more of a statistical function, a group function, and an alignment function.

In the present invention, the analysis visualization unit provides a pivot analysis function by combining each field of the search result with any one or more of a row, a column, and a value.

In the present invention, the specialized client analysis module includes: a script providing unit that connects a developer terminal to a web-based MDMS system, manages R packages, and provides a script template to download scripts; A proxy server that supports downloading the HDFS file of the MDMS system or inquiring weighing data from the developer's terminal; an authentication server for authenticating a developer terminal connected to the MDMS system; and an API providing unit that authenticates the R API, accesses and downloads Hadoop files, inquires metadata, and provides APIs for querying metering data.

In the present invention, the proxy server is connected to the NFS gateway of the MDMS system to download the HDFS file.

In the present invention, the link module is characterized in that it manages and provides any one or more of a list and history inquiry setting for metering data provision in the Hadoop cluster of the MDMS system.

The AMI metering data analysis device according to an aspect of the present invention connects with the online query type analysis tool of the WEB environment, the expert analysis environment and other systems so that the AMI metering data collected through the intelligent power metering infrastructure can be directly analyzed. Data can be provided, so users can access and query Hadoop big data of long-term history data stored in the MDMS system in the web environment, and provide pivot analysis and various visualization methods to analyze quantitative big data without separate coding. Analysts can easily access and analyze quantitative big data by providing a dedicated analysis environment and API.

1 is a block diagram showing an AMI measurement data analysis apparatus according to an embodiment of the present invention.
2 is a block diagram specifically illustrating an online query module of the AMI metering data analysis apparatus according to an embodiment of the present invention.
3 is a flowchart for explaining the operation of the online query module of the AMI metering data analysis apparatus according to an embodiment of the present invention.
4 is an exemplary diagram illustrating an HDFS directory structure for loading an analysis target in the AMI metering data analysis apparatus according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a search screen for weighing data in the AMI weighing data analysis apparatus according to an embodiment of the present invention.
6 is an exemplary view showing a pivot analysis screen in the AMI measurement data analysis apparatus according to an embodiment of the present invention.
7 is an exemplary diagram illustrating a chart analysis screen in the AMI weighing data analysis apparatus according to an embodiment of the present invention.
8 is a block diagram specifically illustrating a specialized client analysis module of the AMI metering data analysis apparatus according to an embodiment of the present invention.
9 is a block diagram illustrating an execution process of a professional client analysis module of an AMI metering data analysis apparatus according to an embodiment of the present invention.
10 is a list of APIs provided by the specialized client analysis module of the AMI metering data analysis apparatus according to an embodiment of the present invention.
11 is a view for explaining the HDFS access and file download process in the professional client analysis module of the AMI metering data analysis apparatus according to an embodiment of the present invention.

Hereinafter, an AMI measurement data analysis apparatus according to the present invention will be described with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

1 is a block diagram illustrating an AMI measurement data analysis apparatus according to an embodiment of the present invention, and FIG. 2 is a block diagram illustrating an online query module of the AMI measurement data analysis apparatus according to an embodiment of the present invention. 3 is a flowchart for explaining the operation of the online query module of the AMI measurement data analysis apparatus according to an embodiment of the present invention, and FIG. 4 is an analysis target in the AMI measurement data analysis apparatus according to an embodiment of the present invention. It is an exemplary view showing the HDFS directory structure for loading, and FIG. 5 is an exemplary view showing a weighing data search screen in the AMI weighing data analysis apparatus according to an embodiment of the present invention, and FIG. 6 is an embodiment of the present invention. It is an exemplary view showing a pivot analysis screen in the AMI metering data analysis apparatus according to the present invention, Figure 7 is an exemplary view showing a chart analysis screen in the AMI metering data analysis apparatus according to an embodiment of the present invention, Figure 8 is an embodiment of the present invention It is a block diagram specifically showing the professional client analysis module of the AMI metering data analysis apparatus according to an example, and FIG. 9 is a block diagram showing the execution process of the professional client analysis module of the AMI metering data analysis apparatus according to an embodiment of the present invention 10 is a list of APIs provided by the specialized client analysis module of the AMI measurement data analysis apparatus according to an embodiment of the present invention, and FIG. 11 is a professional client analysis of the AMI measurement data analysis apparatus according to an embodiment of the present invention. It is a diagram to explain the HDFS access and file download process in the module.

As shown in FIG. 1 , the AMI metering data analysis device 20 according to an embodiment of the present invention includes an OLAP spark cluster 23 , an online query module 21 , an expert client analysis module 25 , and a link module 27 . ) can be provided.

In this embodiment, the MDMS system 10 collects weighing data from the Head-End System of each regional headquarters, stores the original data in a Hadoop Cluster, and loads the NoSQL database to create a Spark Cluster. Through distributed parallel profiling, verification, and aggregation processing, the processing results are stored in a relational database (RDB) and a Hadoop cluster. In addition, the history data storage range can be stored for 10 years in Hadoop, 13 months in NoSQL DB, and 4 months in relational DB for cost efficiency.

The OLAP (Online Analytical Processing) spark cluster 23 may profile, verify, and aggregate measurement data from the MDMS system 10 in distributed parallel.

Here, the Spark Cluster 23 supports an independent system method and is composed of a Spark master and a worker. In order to query a large amount of weighing data, a large number of high-performance Spark servers are required, so analysis performance can be improved through parallel data processing. can be improved

The online query module 21 may be connected to a user terminal based on the web to extract, analyze, and visualize the weighing data stored in the MDMS system 10 .

That is, when a keyword or search condition to search for HDFS (Hardoop Distributed File System) or Oracle is input in the user terminal 30 accessed through the web-based online query module 21, the desired Data can be searched and provided, and desired analysis results can be visualized and provided by selecting fields, rows, and columns of the searched results.

The professional client analysis module 25 can access the MDMS system 100 for the web-based developer terminal 40 and provide an API for advanced analysis of weighing data to support transmission and reception of files, HDFS files required for advanced analysis can be downloaded to the R client, allowing easy advanced analysis and visualization.

The linkage module 27 is connected to the operator terminal based on the web to provide and set Hadoop data of the MDMS system 10 to other systems, so weighing data for providing the weighing data loaded in HDFS to other systems You can manage the setting function, extraction and provision status, etc.

More specifically, as shown in FIG. 2 , the online query module 21 includes a small relation DB 210 , a data model management unit 212 , a weighing data inquiry unit 214 , an analysis visualization unit 216 , and an environment setting unit. (218).

The small relation DB 210 may manage a model for attribute information on an analysis target attribute in the form of metadata.

Here, the small relation DB 210 is small, and MariaDB manages the data necessary for the AMI metric data analysis device 21, such as information on meta data for the data model and user information, and the Mongo DB used as the NoSQL DB is SON. It can process and manage dynamic schema-type document data such as

The data model management unit 212 may manage the model for attribute information on the analysis target attribute of the small relation DB 210 or the HDFS source.

The measurement data inquiry unit 214 may search for measurement data in a data model format managed by the data model management unit 212 and support a search for specific keywords or statistical analysis.

The analysis visualization unit 216 may analyze and visualize the measurement data inquired through the measurement data inquiry unit 214 in parallel through the OLAP spark cluster 23 .

Here, the analysis visualization unit 216 may provide a pivot function and visualize data to be analyzed through an analysis type and a chart.

The environment setting unit 218 may set an environment for the user terminal 30 to access the data model management unit 212 , the measurement data inquiry unit 214 , and the analysis visualization unit 216 .

3 is a flowchart for explaining the operation of the online query module of the AMI metering data analysis apparatus according to an embodiment of the present invention.

As shown in FIG. 3 , the online query module 21 may provide an environment in which a specific keyword can be searched or statistical analysis can be performed from measurement data.

First, the online query module 21 creates the connection information of the RDB or HDFS of the MDMS system 10, loads the analysis target data into the small relational DB 210, selects the analysis target table, file, or folder at the same time You can select multiple files to create a data model.

Here, the analysis target data is data in CSV file format using the MongoDB export utility so that the weighing data loaded in the NoSQL cluster (MongoDB) of the MDMS system 10 can be utilized in the AMI weighing data analysis device 20 according to this embodiment. can be exported. You can then load the metering data file into HDFS using Apache Nifi.

In addition, for the metering data provision function, it is possible to create Hive Meta Data data and provide an environment for querying by connecting the files loaded in HDFS in the form of JDBC (Java Database Connectivity).

Here, the metering data Hadoop cluster HDFS directory structure can store data as shown in FIG. 4 as regional headquarters, meter reading code, Meter_key (2 digits), year, month, date, and metering data file.

Afterwards, the online query module 21 may select an existing stored data model to search for weighing data, select a column of the data model to reduce the search range, designate a search time range, and search through a search query You can search by entering conditions.

Here, the data model can be created by linking the data stored in the loaded analysis target HDFS file or RDBMS as shown in FIG. 4, and the previously created data model can be stored and managed and displayed as a list on the weighing data search screen shown in FIG. .

To create a new data model, first set the data model to be searched for and a search period, and search terms can be entered directly or with the help of a recommendation function in the search conditions.

When a search word is input in the search word input field, the online query module 21 displays a search word recommendation window according to the inputted content, and shows the previous search history according to the entered content. can

In this case, Command Syntax help may be provided for the convenience of writing keyword grammar or commands. Previous search history can be selected by using the up or down keys on the keyboard or by clicking the mouse. When selected, the previous search information is immediately entered into the input box.

Here, as in Structured Query Language (SQL), data filtering is possible through where conditional clauses and operators, and statistical functions, group functions, sort functions, etc. can be used.

As shown in FIG. 5, the search results are divided into three tabs such as event, statistics, and visualization according to the search conditions. The event tab is a tab where an event list is basically output during all searches, and the statistics tab and visualization tab are search results. It can be displayed only when searching using commands that obtain statistical results such as stats and pivot in the bar.

In the data model column on the left, field information of the data model selected by default can be displayed, and in the search screen, the role of controlling the columns of the result event grid and statistical information of the fields for the search results can be additionally provided.

In addition, by using the checkbox in each field item in the field list, you can display or hide columns with the same name in the search result grid in the Event tab. Count information can be called.

In addition, when a specific field is selected, the field statistics information window pops up and detailed information about the top 10 values can be retrieved from the server and displayed to indicate which type of value represents what distribution within the search result.

In the data tab, information about basic search results is displayed, in the timeline graph, how many pieces of data are in each time period in a bar graph, and in the search result grid, a part of the data can be output in the form of a grid view.

In the Statistics tab, the results of commands such as stats and pivot are displayed in a grid view, and in the Visualization tab, the results can be expressed as a chart. , you can change the style of the chart, such as x/y axis labels and whether to display data values, through the option dialog, and you can check the changes in events that occur over time through the timeline chart.

In particular, it is easy to recognize abnormal situations that cause the number of events to rapidly increase or decrease, so if detailed analysis of these abnormal situations is required, click a specific bar or specify an area by dragging, and then based on the specified time range You can perform the search again.

In addition, the online query module 21 provides a pivot analysis function for easily and quickly deriving a desired analysis result by appropriately combining each field of a search result by an analyst through the user terminal 30 into a row, column, value, etc. It can be provided through the pivot analysis screen as shown in .

Here, the online query module 21 selects a data model for pivot analysis and adds a filter to set a target range for pivot execution. That is, the user can limit the data to be pivoted by setting the range of values of a specific field.

As shown in FIG. 6 , data of a specific field can be limitedly analyzed by using the filter setting at the top of the pivot setting, and operators applicable according to the character type or numeric type of the field property can be used.

For example, in the weighing data, the meter reading value of the LP (Load Profile) section may be set to a value of 1 kW or more, or a filter may be set only for data for which the power recovery time is not received in the power outage data.

In addition, the online query module 21 allows a user to select an analysis type, select a data model, and designate and execute an analysis column based on the web in the user terminal for visualization, so that each field, row, column, etc. of the search result is executed. In order to quickly and easily derive desired analysis results on the GUI screen, unlike pivot, an intuitive user interface optimized for charts can be provided as shown in FIG. 7 by focusing on visualization.

That is, various chart settings and visualization options such as Line Plot, Scatter Plot, Bar Chart, Pie Chart, Histogram, Sankey Chart, and Heatmap can be provided.

The professional client analysis module 25 may include a script providing unit 250 , a proxy server 252 , an authentication server 254 , and an API providing unit 256 as shown in FIG. 8 .

The script providing unit 250 may connect the developer terminal 40 to the MDMS system 10 based on the web to provide a script template to manage the R package and download the script.

In the process of advanced analysis and visualization in the client PC environment of the developer terminal 40, as shown in FIG. 9, the R client and R package shared by web-based access to the MDMS system 10 are downloaded and installed, and the R studio to load the R sample script, then modify the R script according to the purpose of analysis and execute the R script to enable visualization and analysis.

The proxy server 252 is connected to the NFS gateway of the MDMS system 10 and provides a file sharing service to the server host using the network file system (NFS) protocol for the files loaded in HDFS, and from the developer terminal to the MDMS system 10 ) to download the HDFS file or to inquire weighing data.

The authentication server 254 may be configured as a keycloak to process single sign-on (SSO) with the MDMS system 10 to authenticate the connection of the developer terminal 40 .

The API providing unit 256 may provide an API for authenticating the R API, accessing and downloading the Hadoop file, inquiring metadata, and querying measurement data.

As illustrated in FIG. 10, the API list for expert analysis includes login, authentication processing, weighing data download, weighing data provision meta information, weighing data query, LP data loading for time series data analysis, preprocessing code, exploratory analysis code, APIs such as outlier and missing corrections and analysis codes can be provided.

Therefore, it is possible to easily access and analyze the HDFS of the MDMS system 10 by using an API through the developer terminal 40 in a manner in which authentication security is applied.

For example, if we describe in more detail about Hadoop Access and HDFS file download API among APIs, as shown in Fig. 11, authentication is performed through Web API and Keycloadk authentication server 254, and then, proxy server 252 and NFS Services can be provided through the Gateway.

Here, the NFS Gateway is located in the Hadoop Cluster of the MDMS system 10 and mounts the HDFS file system as a Linux server file system, and the MDA.loadData API internally performs MDMS SSO processing to check the authentication token. .

In addition, in order to limit the inquiry of a large amount of data at once, a limited period is managed with the weighing data table, and an error message can be output if a download request is made beyond the relevant period.

The proxy server 252 is connected to the MDMS system 10 through the NFS Gateway, and when downloading the HDFS file system, the download can be processed using the sftp function of the NFS Gateway.

In addition, the download information for each user is recorded in the RDB of the MDMS system 10 through the Web API, the downloaded CSV format HDFS file can be checked through a separate viewer, and the PC using the R client CSV API After loading into memory, analysis and visualization can proceed.

The developer terminal 40 may correct outliers and missing values by using an analysis script for the downloaded weighing data in this way.

The link module 27 may manage and provide any one or more of a list for providing metering data and a history inquiry setting in the Hadoop cluster of the MDMS system 10 on a web-based basis in the operator terminal 50 .

It is possible to automatically link data items and link system information to a legacy system that requires large/large-capacity AMI measurement data stored in the MDMS system 10 to automatically link or provide a download function to a client PC.

Here, by using the function to manage the list, history inquiry, setting, etc. of the weighing data provision, load the setting contents through a batch operation in the spark cluster, connect to Hive of the Hadoop cluster, create and execute the weighing data inquiry query Weighing data can be selected. The provision history can be managed by transferring the data set extracted from the spark cluster according to the query condition to the legacy system and storing it in the Hadoop cluster as well.

As described above, according to the AMI metering data analysis device according to the embodiment of the present invention, an online query type analysis tool of the WEB environment and dedicated analysis for experts so that the AMI metering data collected through the intelligent power metering infrastructure can be directly analyzed Data can be provided in connection with the environment and other systems, so users can access and query Hadoop big data of long-term history data stored in the MDMS system in the web environment, and provide pivot analysis and various visualization methods to measure big data without separate coding. can be analyzed, and analysts such as developers can easily access and analyze quantitative big data by providing a dedicated analysis environment and API.

Implementations described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementations of the discussed features may also be implemented in other forms (eg, as an apparatus or program). The apparatus may be implemented in suitable hardware, software and firmware, and the like. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a computer, a microprocessor, a processing device, including an integrated circuit or programmable logic device, or the like. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDA”) and other devices that facilitate communication of information between end-users.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely an example, and those skilled in the art to which various modifications and equivalent other embodiments are possible. will understand

Therefore, the true technical protection scope of the present invention should be defined by the following claims.

10: MDMS system 20: AMI weighing data analysis device
21: Online Query Module 23: OLAP Spark Cluster
25: specialized client analysis module 27: linkage module
30: user terminal 40: developer terminal
50: operator terminal 210: small relation DB
212: data model management unit 214: weighing data inquiry unit
216: analysis visualization unit 218: environment setting unit
250: script providing unit 252: proxy server
254: authentication server 256: API provider

Claims (9)

OLAP Spark cluster for distributing and parallel profiling, validation, and aggregation of weighing data from MDMS systems;
an online query module for extracting, analyzing, and visualizing the measurement data stored in the MDMS system connected to a web-based user terminal;
a professional client analysis module that accesses the MDMS system for a web-based connected developer terminal to provide a script, and provides an API for advanced analysis of the metering data to support sending and receiving files; and
AMI metering data analysis device comprising a; a web-based connection module for connecting to an operator terminal to provide and set Hadoop data of the MDMS system to other systems.
The method of claim 1, wherein the online query module comprises:
A small relational DB that manages the model for the attribute information on the analysis target attribute in the form of metadata;
a data model management unit for managing a model for attribute information on the analysis target attribute of the small relation DB or the HDFS source through the small relation DB;
a weighing data inquiry unit that inquires about measurement data in a data model format managed by the data model management unit;
an analysis visualization unit that analyzes and visualizes the measurement data retrieved through the measurement data inquiry unit in parallel through the OLAP (Online Analytical Processing) spark cluster; and
and an environment setting unit for setting an environment for the user terminal to access the data model management unit, the measurement data inquiry unit, and the analysis visualization unit.
The apparatus of claim 2, wherein the metering data inquiry unit selects a data model, stores a column of the data model and a search time range, and inputs a search condition through a search query.
The apparatus of claim 3, wherein the search query word filters data through a where conditional clause and an operator.
The apparatus according to claim 2, wherein the weighing data inquiry unit searches using one or more of a statistical function, a group function, and an alignment function.
The apparatus of claim 2, wherein the analysis visualization unit provides a pivot analysis function by combining each field of a search result with any one or more of a row, a column, and a value.
According to claim 1, wherein the professional client analysis module,
a script providing unit that connects a developer terminal to the MDMS system based on a web, manages the R package, and provides a script template to download the script;
a proxy server supporting the developer terminal to download the HDFS file of the MDMS system or to inquire the metering data;
an authentication server for authenticating the developer terminal connected to the MDMS system; and
AMI metering data analysis device comprising: an API providing API for authenticating R API, accessing and downloading Hadoop files, inquiring metadata, and querying metering data.
The apparatus of claim 7, wherein the proxy server is connected to an NFS gateway of the MDMS system to download an HDFS file.
The apparatus of claim 1, wherein the link module manages and provides any one or more of a list for metering data provision and a history inquiry setting in the Hadoop cluster of the MDMS system.

Images