CN115269692A - Intelligent enhanced data analysis method, system, equipment and storage medium - Google Patents

Abstract

The application discloses an intelligent enhanced data analysis method, a system, equipment and a storage medium, comprising the following steps: establishing a power grid knowledge base by accessing data of a power production operation full-service line; extracting data from a power grid knowledge base according to the requirement description to generate an analysis requirement data set as a model training set; training a natural language model by combining data of an existing knowledge base, and identifying a model training set by the trained natural language model to generate a regular expression; identifying data of a power grid knowledge base through a regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets; establishing a plurality of kinds of visual display models, matching the visual display models according to the dimensionality and the index of the data set in the data model, and displaying the data set through the matched visual display models. The problem of current relatively high data of complexity assemble and the application efficiency is lower when analyzing the demand is solved.

Description

Intelligent enhanced data analysis method, system, equipment and storage medium

Technical Field

The present application relates to the field of big data technologies, and in particular, to an intelligent enhanced data analysis method, system, device, and storage medium.

Background

The power grid business departments and leaders have great requirements on data analysis, but lack data and effective support of tools, and report requirements cannot be responded in time. Meanwhile, data conversation between the leader or demand department and data analysts is deficient, and the analysis period is long. Even if the enterprise passes through the bottom data platform, integrate each data source. Business personnel and high management are still left without the right hand for new data, and do not know how to better present the data when drawing. In the process of data exploration, the process is tedious and inefficient, and valuable business clues are difficult to find from the chart.

Disclosure of Invention

The application provides an intelligent enhanced data analysis method, system, equipment and storage medium, which are used for solving the technical problem of low application efficiency when the prior art meets the requirements of high-complexity data aggregation and analysis.

In view of this, a first aspect of the present application provides an intelligent enhanced data analysis method, including:

establishing a basic data support warehouse as a power grid knowledge base by accessing data of a power production operation full service line;

extracting data from the power grid knowledge base according to the requirement description to generate an analysis requirement data set as a model training set;

creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying the model training set through the trained natural language model to generate a regular expression;

and identifying the data of the power grid knowledge base through the regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets.

Optionally, the identifying, by the regular expression, the data of the power grid knowledge base to obtain a plurality of data sets, and establishing a data model for intelligent analysis based on the plurality of data sets, and then further includes:

establishing a plurality of kinds of visual display models, matching the visual display models according to the dimensionality and the index of the data set in the data model, and displaying the data set through the matched visual display models.

Optionally, the visualization display model specifically includes: an icon category suggestion model and an aggregation function suggestion model.

Optionally, the data accessed to the full service line of power production operation constructs a basic data support warehouse as a power grid knowledge base, and specifically includes:

after a data link with a power grid database is established, data in the power grid database are extracted, the extracted data are subjected to standardization processing, the data subjected to standardization processing are collected into a knowledge base, and a basic data support warehouse is obtained and serves as a power grid knowledge base.

A second aspect of the present application provides an intelligent enhanced data analysis system, the system comprising:

the building unit is used for accessing data of a power production operation full service line to build a basic data support warehouse as a power grid knowledge base;

the extraction unit is used for extracting data from the power grid knowledge base according to the requirement description to generate an analysis requirement data set which is used as a model training set;

the generating unit is used for creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying and processing the model training set through the trained natural language model to generate a regular expression;

and the identification unit is used for identifying the data of the power grid knowledge base through the regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets.

Optionally, the method further comprises: a display unit;

the display unit is used for establishing a plurality of kinds of visual display models, matching the visual display models according to the dimensionality and indexes of the data sets in the data models, and displaying the data sets through the matched visual display models.

Optionally, the visualization display model specifically includes: an icon category suggestion model and an aggregation function suggestion model.

Optionally, the building unit is specifically configured to:

after a data link with a power grid database is established, data in the power grid database are extracted, the extracted data are subjected to standardization processing, the data subjected to standardization processing are collected into a knowledge base, and a basic data support warehouse is obtained and serves as a power grid knowledge base.

A third aspect of the present application provides an intelligent enhanced data analysis device, the device comprising a processor and a memory:

the memory is used for storing program codes and transmitting the program codes to the processor;

the processor is configured to perform the steps of the intelligent enhanced data analysis method according to the first aspect, according to instructions in the program code.

A fourth aspect of the present application provides a computer-readable storage medium for storing program code for executing the intelligent enhanced data analysis method of the first aspect.

According to the technical scheme, the method has the following advantages:

the application provides an intelligent enhanced data analysis method, which comprises the following steps: establishing a basic data support warehouse as a power grid knowledge base by accessing data of a power production operation full service line; extracting data from a power grid knowledge base according to the requirement description to generate an analysis requirement data set as a model training set; creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying and processing a model training set through the trained natural language model to generate a regular expression; the data of the power grid knowledge base are identified through the regular expression, a plurality of data sets are obtained and stored, and a data model for intelligent analysis is established based on the data sets. And further establishing a plurality of kinds of visual display models, matching the visual display models according to the dimensionality and indexes of the data set in the data model, and displaying the data set through the matched visual display models.

Compared with the prior art, the method and the device have the advantages that the optimal chart type and aggregation function suggestion can be recommended according to the dimensionality and the index, the user is helped to better display data, the operation efficiency is improved, the use threshold is reduced, the chart recommendation can automatically draw various charts by utilizing the data when the data are not available, and an analyst can conveniently and quickly master the data condition. Therefore, the technical problem that the application efficiency is low when the existing manual work needs to assemble and analyze data with high complexity is solved.

Drawings

Fig. 1 is a schematic flowchart of an embodiment of an intelligent enhanced data analysis method provided in an embodiment of the present application;

fig. 2 is a schematic flowchart of an embodiment of an intelligent enhanced data analysis system provided in the embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Referring to fig. 1, an intelligent enhanced data analysis method provided in an embodiment of the present application includes:

step 101, accessing data of a power production operation full service line to construct a basic data support warehouse as a power grid knowledge base;

specifically, after a data link with a power grid database is established, data in the power grid database are extracted, the extracted data are subjected to standardization processing, and the standardized data are collected into a knowledge base, so that a basic data support warehouse is obtained and used as a power grid knowledge base.

It should be noted that, in order to implement data source management, management configuration is performed on a relational database, a non-relational database, and a file system data source, and first, a data link with the data source is established according to different database types using JDBC, ODBC, and other technologies. The link between the communication and the data source is mainly divided into the following steps: (1) loading a data source driver; (2) acquiring database connection; (3) acquiring a transmitter; (4) Transmitting the sql to a database by using a transmitter for execution, and returning an execution result; (4) processing the result; and (5) releasing the resources. Secondly, according to a configured data source link, relying on ETL operation of model data of a large data platform technology, extracting data through a cdc technology, an sqoop technology and an ftp technology, storing the data in databases such as a platform HIVE database and an HABSE database, and performing data management, wherein the management process is to standardize the managed data to form a standard data set; the method comprises the following steps of performing ETL (extract transform and load) on data to form a standardization process, establishing standards of data types, lengths, data dimensions, data ranges and service attributes, setting the standards as data verification factors, performing quality inspection according to standard specifications after data access, and performing standardization processing on nonstandard data to form a standard data set. And finally, collecting the data after the data management standardization into a knowledge base to obtain a basic data support warehouse which is used as a power grid knowledge base.

CDC technology: CDC is generally called Change Data Capture (Change Data acquisition), and in a broad concept, as long as it is a technology capable of capturing Data changes, it may be called CDC.

Sqoop technique: the method is mainly used for transferring data between Hadoop (Hive) and traditional databases (MySQL, postgresql.), and can lead data in a relational database (such as MySQL, oracle, postgres and the like) into the HDFS of Hadoop and also can lead data in the HDFS into the relational database.

ftp technique: english is abbreviated as filetransfer protocol (file transfer protocol), and chinese is abbreviated as "file transfer protocol". The method is used for bidirectional transmission of control files on the Internet.

Further to the extraction of data, it should be noted that:

(1) If data requiring real-time performance needs to be synchronized instantly, the CDC is used for real-time data synchronization and incremental updating, the CDC is used for performing INSERT, UPDATE or DELETE operations on the source table, and meanwhile, the data can be extracted, and the changed data is stored in a change table of the database. (2) Data enters a storage layer from a service source through Sqoop, the Sqoop is a tool for transferring data in Hadoop and a relational database mutually, data in the relational database (such as MySQL, oracle, postgres and the like) can be imported into an HDFS of the Hadoop, and data of the HDFS can also be exported into the relational database. (3) The File Transfer Protocol is a File Transfer Protocol, and is used for bidirectional Transfer of control files on a network. At the same time, it is also an Application (Application). The big data platform is transmitted through the FTP file, has a breakpoint resume function aiming at network abnormity, and has two forms by using FTP access: firstly, uploading and downloading unstructured files for service inquiry and use; and secondly, FTP uploading, converting and loading the semi-structured file into a specified database.

102, extracting data from a power grid knowledge base according to requirement description to generate an analysis requirement data set as a model training set;

it should be noted that the data is extracted from the power grid knowledge base (basic data support warehouse) by the technologies of word vector model word2vec, sentence segmentation, part of speech tagging POS, named entity recognition NER, dependency parsing DP, recurrent neural network RNN, etc. to form a data set as a model training set.

103, creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying a model training set through the trained natural language model to generate a regular expression;

it can be understood that a usable regular expression is generated by creating a Natural Language Processing (NLP) model, performing natural language model training in combination with data of an existing knowledge base, and performing recognition processing on the model training set obtained in step 102 through the trained model. It should be noted that a Regular Expression, also called a Regular Expression, (Regular Expression, often abbreviated as regex, regexp or RE in code) is a text pattern, including common characters (e.g., letters between a and z) and special characters (called "meta characters"), which is a concept of computer science. Regular expressions use a single string to describe, match a series of strings matching a certain syntactic rule, and are typically used to retrieve, replace, text that conforms to a certain pattern (rule).

Step 104, identifying data of the power grid knowledge base through a regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets;

it should be noted that the data of the power grid knowledge base are identified through the regular expression to obtain and store a plurality of data sets, a data model for intelligent analysis is established based on the data sets, and the data model can be mounted to an intelligent analysis system in practical application.

And 105, establishing a plurality of types of visual display models, matching the visual display models according to the dimensionality and indexes of the data set in the data model, and displaying the data set through the matched visual display models.

It should be noted that various visualization display models are built in, the visualization display models are matched according to different dimensionalities and different indexes of the data set, an interface is successfully returned to display correspondingly, and different forms of display of the data through manual switching of the visualization models are supported.

Further, in application, the optimal chart type and aggregation function suggestion can be recommended according to the dimensionality and the index, so that the user can be helped to better display data, the operation efficiency is improved, and the use threshold is reduced.

Wherein the chart categories suggest: and analyzing the data quantity and the number of dimension measurement of the generated table, and automatically selecting an optimal display mode. The aggregation function suggests: a depth migration algorithm is used. Through dimensions and the names of the measures, the aggregation functions acting on the measures are automatically recommended.

The above is an embodiment of an intelligent enhanced data analysis method provided in the embodiment of the present application, and the following is an embodiment of an intelligent enhanced data analysis system provided in the embodiment of the present application.

Referring to fig. 2, an intelligent enhanced data analysis system provided in an embodiment of the present application includes:

the building unit 201 is used for accessing data of a power production operation full service line to build a basic data support warehouse as a power grid knowledge base;

the extraction unit 202 is used for extracting data from the power grid knowledge base according to the requirement description to generate an analysis requirement data set as a model training set;

the generating unit 203 is used for creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying and processing a model training set through the trained natural language model to generate a regular expression;

the identification unit 204 is used for identifying the data of the power grid knowledge base through a regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets;

the display unit 205 is configured to establish a plurality of types of visual display models, match the visual display models according to the dimensionality and the index of the data set in the data model, and display the data set through the matched visual display models.

Further, an embodiment of the present application further provides an intelligent enhanced data analysis device, where the device includes a processor and a memory:

the memory is used for storing program codes and transmitting the program codes to the processor;

the processor is used for executing the intelligent enhanced data analysis method according to the instruction in the program code.

Further, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium is used for storing a program code, and the program code is used for executing the intelligent enhanced data analysis method described in the foregoing method embodiment.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" for describing an association relationship of associated objects, indicating that there may be three relationships, e.g., "a and/or B" may indicate: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c may be single or plural.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, which are essential or part of the technical solutions contributing to the prior art, or all or part of the technical solutions, may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims (10)

1. An intelligent enhanced data analysis method, comprising:

establishing a basic data support warehouse as a power grid knowledge base by accessing data of a power production operation full service line;

extracting data from the power grid knowledge base according to the requirement description to generate an analysis requirement data set as a model training set;

creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying the model training set through the trained natural language model to generate a regular expression;

and identifying the data of the power grid knowledge base through the regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets.

2. The intelligent enhanced data analysis method according to claim 1, wherein the identifying the data of the power grid knowledge base through the regular expression obtains a plurality of data sets, and establishes a data model for intelligent analysis based on the plurality of data sets, and then further comprises:

establishing a plurality of kinds of visual display models, matching the visual display models according to the dimensionality and the index of the data set in the data model, and displaying the data set through the matched visual display models.

3. The intelligent enhanced data analysis method according to claim 2, wherein the visualization display model specifically comprises: an icon category suggestion model and an aggregation function suggestion model.

4. The intelligent enhanced data analysis method according to claim 1, wherein the building of a basic data support warehouse for the data accessed to the whole service line of the power production operation as a power grid knowledge base specifically comprises:

after a data link with a power grid database is established, data in the power grid database are extracted, the extracted data are subjected to standardization processing, the data subjected to standardization processing are collected into a knowledge base, and a basic data support warehouse is obtained and serves as a power grid knowledge base.

5. An intelligent enhanced data analysis system, comprising:

the building unit is used for accessing data of a power production operation full service line to build a basic data support warehouse as a power grid knowledge base;

the extraction unit is used for extracting data from the power grid knowledge base according to the requirement description to generate an analysis requirement data set as a model training set;

the generation unit is used for creating a natural language processing model, training the natural language model by combining data of the existing knowledge base, and identifying the model training set through the trained natural language model to generate a regular expression;

and the identification unit is used for identifying the data of the power grid knowledge base through the regular expression to obtain and store a plurality of data sets, and establishing a data model for intelligent analysis based on the data sets.

6. The intelligent enhanced data analysis system of claim 5, further comprising: a display unit;

the display unit is used for establishing a plurality of kinds of visual display models, matching the visual display models according to the dimensionality and indexes of the data sets in the data models, and displaying the data sets through the matched visual display models.

7. The intelligent augmentation data analysis system of claim 6, wherein the visualization presentation model specifically comprises: an icon category suggestion model and an aggregation function suggestion model.

8. The intelligent enhanced data analysis system of claim 5, wherein the construction unit is specifically configured to:

after a data link with a power grid database is established, data in the power grid database are extracted, the extracted data are subjected to standardization processing, the data subjected to standardization processing are collected into a knowledge base, and a basic data support warehouse is obtained and serves as a power grid knowledge base.

9. An intelligent enhanced data analysis device, the device comprising a processor and a memory:

the memory is used for storing program codes and transmitting the program codes to the processor;

the processor is configured to execute the intelligent enhanced data analysis method of any one of claims 1-4 according to instructions in the program code.

10. A computer-readable storage medium for storing program code for performing the intelligent enhanced data analysis method of any of claims 1-4.

Images