TWI798170B - Data analysis method and device - Google Patents

Abstract

The present invention provides a data analysis method and device. By constructing a processing model corresponding to each processing node in the data analysis process, the processing models are connected in sequence according to the dependency relationship, and the raw data to be processed is processed by the connected processing model to obtain target data. In the present invention, each processing model is directly connected according to the dependency relationship, the output of the previous processing model can be directly input into the next processing model, and the intermediate data is no longer landed, which saves resources, and omits the loading process of the intermediate data, which improves data analysis efficiency.

Description

Data analysis method and device

The invention belongs to the field of data processing, in particular to a data analysis method and device.

The process of data analysis often includes a large number of processing nodes such as data cleaning, feature smoothing, feature normalization, feature extraction, and feature selection. In traditional data analysis, for each processing node, corresponding intermediate data will be generated, and the generated intermediate data needs to be processed on the ground. Since the intermediate data is used as the upstream data dependency of the next processing node, the next data node needs to be loaded The intermediate data after a data node lands is used as its input.

In traditional data analysis, the upper and lower processing nodes are linked through the landing intermediate data. Every time the data passes through a processing node, it needs to be loaded and landed once. For massive data or complex data analysis process, the data of intermediate data The huge amount not only leads to the waste of computing resources and input and output (IO) resources, but also seriously affects the efficiency of data analysis when data loading and landing.

The present invention provides a data analysis method and device, which are used to solve the problem that in the existing data analysis, data needs to be loaded and landed once every time the data passes through a processing node, which not only causes resource costs, but also affects the efficiency of data analysis.

In order to achieve the above object, the present invention provides a data analysis method, including: constructing processing models corresponding to each processing node in the data analysis process; connecting each processing model in sequence according to the dependency relationship to generate processing modules; The processed raw data are processed to obtain the target data.

In order to achieve the above object, the present invention provides a data analysis device, including: a construction module, used to construct the processing models corresponding to each processing node in the data analysis process; a connection module, used to connect each processing model according to the dependency relationship ; The processing module is used to use the connected processing model to process the raw data to be processed to obtain the target data.

In the data analysis method and device provided by the present invention, by constructing the processing models corresponding to each processing node in the data analysis process, each processing model is connected in sequence according to the dependency relationship, and the raw data to be processed is processed by the connected processing model to obtain the target material. In the present invention, each processing model is directly connected according to the dependency relationship, and the output of the previous processing model can be directly input Into the next processing model, the intermediate data is no longer landing, saving resources, and omitting the loading process of intermediate data, improving the efficiency of data analysis.

101~103‧‧‧Steps

201~206‧‧‧Steps

Step 301~307‧‧‧

11‧‧‧Building modules

12‧‧‧connection module

13‧‧‧Processing module

14‧‧‧Acquiring modules

15‧‧‧analysis module

16‧‧‧Effect verification module

17‧‧‧Floor Module

131‧‧‧Data verification unit

132‧‧‧Acquisition unit

133‧‧‧Format conversion unit

134‧‧‧processing unit

Fig. 1 is a schematic flow diagram of the data analysis method of the first embodiment of the present invention; Fig. 2 is a schematic flow diagram of the data analysis method of the second embodiment of the present invention; Fig. 3 is a DAG diagram of the second embodiment of the present invention; Fig. 4 is the implementation of the present invention One of the schematic diagrams of the application example of the data analysis method of Example 2; FIG. 5 is a schematic flow chart of the data analysis method of the third embodiment of the present invention; FIG. 6 is a schematic structural diagram of the data analysis device of the fourth embodiment of the present invention; FIG. Schematic diagram of the structure of the data analysis device of the fifth embodiment.

The evaluation index acquisition method and device provided by the embodiments of the present invention will be described in detail below with reference to the drawings.

Embodiment one

As shown in FIG. 1 , it is a schematic flowchart of the data analysis method in Embodiment 1 of the present invention. The data analysis method includes the following steps:

S101. Construct a processing model corresponding to each processing node in the data analysis process.

First, analyze the set data analysis process, and obtain each processing node in each data analysis process. After obtaining each processing node, the corresponding processing model of each processing node can be constructed according to the function of each processing node.

For example, the data analysis process includes processing nodes such as feature smoothing, feature normalization, feature extraction, and feature selection. These processing nodes have specific processing functions, and these processing nodes can process the input data to obtain a corresponding The result, which is the intermediate data in the data analysis process. For example, feature normalization is used to normalize the original data based on the mean and standard deviation of each feature, and the normalized data is the intermediate data of the feature normalization. In this embodiment, in order to avoid the generation of intermediate data, the processing node of feature normalization is modeled. The feature normalization processing model has a data conversion function for the input data, and the processing model can record the mean value and standard deviation of each feature. , capable of transforming the original data.

S102. Connect each processing model according to dependency relationship.

When analyzing the data analysis process, it is necessary to obtain the dependencies of each processing model. After constructing the processing models corresponding to each processing node, according to the dependencies between processing nodes, the processing models are sequentially connected in series. . In order for each processing model to be directly connected, it is necessary to set the Data interface. In this embodiment, the data interfaces of each processing model are unified. After the data interface connects each processing model in sequence according to the dependency relationship, the data analysis process can be transformed into an orderly execution logic.

S103. Use the connected processing model to process the raw data to be processed to obtain the target data.

When the processing models are connected, the logic can be executed in an orderly manner, and the raw data to be processed can be input into the connected processing models. The raw data first enters the processing model at the top of the execution logic, and then passes through the processing model The final data enters the next processing model in turn until it enters the processing model at the end of the execution logic, and the final output data of the processing model is the target data.

In the data analysis method provided by this embodiment, the processing models are connected according to the dependency relationship, the output of the previous processing model can be directly input into the next processing model through the data interface, and the intermediate data is no longer landed, saving resources, and because the previous The intermediate data generated by the processing model directly enters the next processing model, which avoids the loading process of intermediate data and improves the efficiency of data analysis.

Embodiment two

As shown in FIG. 2 , it is a schematic flowchart of the data analysis method in Embodiment 2 of the present invention. The data analysis method includes the following steps:

S201. Obtain a non-loop directed DAG graph of the data analysis process.

The data analysis process is composed of a series of processing nodes. By analyzing the characteristics of the data analysis process, the infinite Directed Acyclic Graph (DAG graph for short), DAG graph can connect a series of orderly processing nodes.

S202. Analyze the DAG graph to obtain relevant information about the data analysis process.

Wherein, the relevant information includes: logical functions of processing nodes, dependencies among processing nodes, and storage addresses of each processing model.

Relevant information of the data analysis process is obtained by analyzing the DAG graph, wherein the relevant information includes the logical functions of the processing nodes included in the data analysis process, dependencies between processing nodes, and storage addresses of each processing model. The relevant information may also include input data information, output data information, user configuration parameters, and the like. These relevant information can generate an Extensible Markup Language (Extensible Markup Language, XML) file with node dependencies, save the XML file to the database backup and submit it to the background.

S203. Construct a corresponding processing model according to the logical function of each processing node in the related information.

After the logical function of each processing node is acquired, a corresponding processing model is constructed according to the logical function of the processing node. For example, a data scaling processing node is used to reduce the data larger than the set range, and enlarge the data smaller than the preset range. According to the logic function of this processing node, a corresponding data scaling model can be constructed.

S204. Connect each processing model through a data interface according to the dependency relationship between the processing nodes in the relevant information.

After the processing models corresponding to each processing node are generated, according to the dependencies between the processing nodes in the relevant information, each processing model is faces are directly connected. Specifically, after the background receives the XML file obtained by parsing the DAG graph, it can obtain the dependency relationship between the processing nodes in the analyzed DAG graph. The background automatically assembles the code programs of each processing model according to the dependencies between processing nodes, that is, the background performs DAG on the code programs of each processing model according to the dependencies between processing nodes, and saves and compiles the assembled code programs into executable file. Then, based on the data interface that has been designed, the codes of each processing module are assembled sequentially. After the assembly is completed, each processing model is initialized. In order to realize the direct connection of the processing model, the data interface is unified, so that a series of codes can be easily combined. After the series is connected according to the dependency relationship, the data analysis process is converted into an orderly execution at the program level. logic.

In general, the recombination of code programs will introduce new program defects (bugs), which will lead to greater risks during deployment and require retesting, resulting in duplication of resources. In this embodiment, the data analysis process is DAGized at the code level, which can reduce the number of program defects, and can integrate and deploy the executable code programs of each processing model directly to the online environment, which will greatly reduce online deployment. risks of.

S205. Store each processing model on the ground according to the storage address of each processing model in the relevant information.

After the processing model is generated, in order to avoid repeated calculation of the processing model, each processing model can be stored on the ground according to the storage address of each processing model in the relevant information, so as to improve the reuse rate of the processing model. In practical applications, the size of the processing model is much smaller than the size of the intermediate data, which not only saves resources, but also facilitates the efficiency of data analysis.

S206. Input the raw data to be processed into the connected processing model for processing to obtain the target data.

After generating and connecting the processing models, input the raw data to be processed, and process the connected processing models to obtain the final target data. In this embodiment, the process of processing the original data is completed in the memory, and there is no need to land the intermediate data of each processing model.

In order to better understand the data analysis method provided in the above-mentioned embodiment, the following examples are used for illustration:

The data analysis pipeline for raw data includes the following processing nodes: data normalization, data scaling, and data smoothing. Analyze the data analysis process to obtain the DAG diagram of the data analysis process, as shown in Figure 3, the data output by each processing node in the DAG diagram is intermediate data, for example, the intermediate data output by the data scaling processing node is scaling After data, the intermediate data output by the data smoothing node is the smoothed data.

By analyzing the DAG graph, relevant information between processing nodes in the data analysis process can be obtained, and the relevant information includes: logical functions of processing nodes, dependencies between processing nodes, and storage addresses of each processing model. In this example, the dependencies between processing nodes are: data specification depends on data smoothing, and data smoothing depends on data scaling.

In order not to generate intermediate data during the data analysis process and to avoid landing and loading of intermediate data, according to the logical functions of each processing node in the data analysis process, a corresponding processing model is constructed for each processing node. These processing models have corresponding data conversion function. Specifically include: data scaling model, data smoothing model and data specification model. Further, according to processing The dependencies between the nodes connect each processing model through the data interface, as shown in Figure 4. After the processing models are connected, an orderly execution logic is formed, and the original data is input into the connected processing model, so that the execution logic formed by the above connected processing models starts to process the original data to obtain the final target data. The processing of raw data by each processing model can be completed in the memory, avoiding the generation of intermediate data. Furthermore, these constructed processing models can be implemented and stored according to the storage address configured by the user, so as to facilitate the reuse of these processing models.

The data analysis method provided in this embodiment, by obtaining the DAG diagram of the data analysis process, analyzes the DAG diagram, constructs each processing model according to the analysis result, and connects each processing model according to the dependency relationship, and uses the connected processing model to be processed The raw data are processed to obtain the target data. In this embodiment, the processing models are directly connected, the output of the previous processing model can be directly input into the next processing model, the intermediate data is no longer landed, saving resources, and the loading process of intermediate data is omitted, which improves the efficiency of data analysis .

Embodiment Three

As shown in FIG. 5 , it is a schematic flowchart of the data analysis method in Embodiment 3 of the present invention. On the basis of the above embodiments, using the connected processing model to process the raw data to be processed to obtain the target data includes the following steps:

S301. Perform data verification on the original data.

After obtaining the original data, it is necessary to verify the original data. First, check whether the user has a corresponding processing model. If there is a corresponding processing model, determine whether the original data has not been changed. Specifically, according to the relevant For the input data information and output data information in the information, if the original data has not been changed, it means that there is no need to update the stored processing models, and only need to directly obtain the stored processing models, then perform step S302, otherwise perform step S302 S303.

S302. When the original data passes the data verification, acquire each processing model according to the storage address of each processing model in the relevant information.

When the original data passes the data verification, it means that the stored processing model does not need to be updated, and each processing model can be obtained according to the storage address of each processing model in the relevant information

S303. When the original data does not pass the data verification, rebuild each processing model.

When performing data verification on the original data, when it is determined that the original data has been changed, it is necessary to update the stored processing models. If the stored processing model is not detected, analyze the data analysis process of the original data , to construct the corresponding processing model.

S304. Connect each processing model according to the dependency relationship.

Connect each processing model according to the dependency relationship between processing nodes to form an orderly execution logic.

S305. Convert the format of the original data to obtain the input data.

In order to ensure the uniformity of the data interface, it is necessary to convert the mechanical energy format of the original data to generate input data with a unified format. In this embodiment, the original The original data is uniformly converted into input data in Vector or Matrix format.

S306. Input the input data into the connected processing model to obtain the target data.

The processing models are connected according to the dependencies to form an orderly execution logic, and the original data is input into the connected processing model, so that the above execution logic composed of the connected processing models starts to process the original data, and obtains final target data.

S307. Perform effect verification on the target data according to preset effect verification conditions.

Users can preset certain effect verification conditions according to their own needs, and verify the processing effect of the target data according to the set effect verification conditions. For example, the effect comparison before and after data processing is provided, and the user can very intuitively see the effect comparison of the data before and after processing.

Embodiment Four

As shown in FIG. 6 , it is a schematic structural diagram of a data analysis device according to Embodiment 4 of the present invention. The data analysis device includes: a construction module 11 , a connection module 12 and a processing module 13 .

Among them, the construction module 11 is used to construct the processing models corresponding to each processing node in the data analysis process.

The connection module 12 is used to connect each processing model according to the dependency relationship.

The processing module 13 is used to utilize the connected processing model to be processed The raw data are processed to obtain the target data.

The data analysis device provided in this embodiment connects the processing models according to the dependency relationship, the output of the previous processing model can be directly input into the next processing model through the data interface, the intermediate data is no longer landed, saving resources, and because the previous The intermediate data generated by the processing model directly enters the next processing model, which avoids the loading process of intermediate data and improves the efficiency of data analysis.

Embodiment five

As shown in FIG. 7 , it is a schematic structural diagram of a data analysis device according to Embodiment 5 of the present invention. In addition to the construction module 11, the connection module 12 and the processing module 13 in the fourth embodiment, the data analysis device also includes: an acquisition module 14, an analysis module 15, an effect verification module 16 and a floor model Group 17.

The obtaining module 14 is used to obtain the non-loop directed DAG graph of the data analysis process; the analysis module 15 is used to analyze the DAG graph to obtain relevant information of the data analysis process.

Wherein, the relevant information includes: logical functions of processing nodes, dependencies among processing nodes, and storage addresses of each processing model.

An optional structure of the processing module in this embodiment includes: a data verification unit 131 , an acquisition unit 132 , a format conversion unit 133 and a processing unit 134 .

Wherein, the data verification unit 131 is used to carry out data verification on the original data material verification.

The obtaining unit 132 is configured to obtain each processing model according to the storage address of each processing model in the relevant information when the original data passes the verification.

A format conversion unit 133, configured to perform format conversion on the original data to obtain input data.

The processing unit 134 is configured to sequentially input the input data into the connected processing model for processing to obtain the target data.

Further, the data analysis device also includes: an effect verification module 16 .

The effect verification module 16 is used to verify the effect of the target data according to the preset effect verification conditions.

The construction module 11 is specifically used to construct the corresponding processing model according to the logical function of each processing node in the relevant information.

The connection module 12 is specifically used to connect each processing model through a data interface according to the dependency relationship between the processing nodes in the related information.

Further, the data analysis device also includes: a landing module 17 .

The landing module 17 is configured to store each processing model on the ground according to the storage address of each processing model in the relevant information.

In this embodiment, by obtaining the DAG diagram of the data analysis process, the DAG diagram is analyzed, each processing model is constructed according to the analysis results, and each processing model is connected according to the dependency relationship, and the original data to be processed is processed using the connected processing model. processed to obtain the target data. In this embodiment, the processing models are directly connected, the output of the previous processing model can be directly input into the next processing model, and the intermediate data is no longer landed, saving It saves resources and omits the loading process of intermediate data, which improves the efficiency of data analysis.

Furthermore, these constructed processing models can be implemented and stored according to the storage address configured by the user, so as to facilitate the reuse of these processing models.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by program instructions related hardware. The aforementioned programs can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (14)

A data analysis method, characterized in that it includes: constructing a processing model corresponding to each processing node in the data analysis process; connecting each processing model according to the dependency relationship between the processing nodes; using the connected processing model to process the original data Performing processing to obtain target data, wherein, using the connected processing model to process the to-be-processed original data to obtain target data includes: performing data verification on the original data. According to the method described in item 1 of the patent application, wherein, before constructing the processing model of each processing node in the data analysis process, it includes: obtaining the non-loop directed DAG graph of the data analysis process; analyzing the DAG graph to obtain the data Relevant information of the analysis process; wherein, the relevant information includes: logical functions of processing nodes, dependencies between processing nodes, and storage addresses of each processing model. According to the method described in item 2 of the scope of the patent application, wherein, the processing of the raw data to be processed by using the connected processing model to obtain the target data also includes: when the original data passes the data verification, according to the relevant information The storage address of each processing model obtains each processing model; converts the format of the original data to obtain the input data; inputs the input data into the connected processing model for processing to obtain the target data. According to the method described in item 3 of the patent application, after inputting the input data into the connected processing model to obtain the target data, it further includes: verifying the effect of the target data according to the preset effect verification conditions. According to the method described in any one of items 1-4 of the scope of patent application, wherein the processing model corresponding to each processing node in the construction data analysis process includes: constructing according to the logical function of each processing node in the relevant information corresponding to the processing model. According to the method described in any one of items 1-4 of the scope of patent application, wherein, connecting each processing model according to the dependency relationship between processing nodes includes: according to the dependency relationship between processing nodes in the relevant information Connect each processing model through the data interface. According to the method described in any one of items 1-4 of the scope of the patent application, after constructing the processing models corresponding to each processing node in the data analysis process, it includes: storing each processing model according to the relevant information. The model is stored on the floor. A data analysis device, characterized in that it includes: a construction module, used to construct a processing model corresponding to each processing node in the data analysis process; a connection module, used to connect each processing model according to a dependency relationship; The processing module is used to use the connected processing model to process the raw data to be processed to obtain the target data, wherein the processing module includes: a data checking unit, used to check the original data. According to the device described in item 8 of the scope of the patent application, it also includes: an acquisition module, which is used to obtain the non-loop directed DAG graph of the data analysis process; an analysis module, which is used to analyze the DAG graph to obtain the data analysis Relevant information of the process; wherein, the relevant information includes: logical functions of processing nodes, dependencies between processing nodes, and storage addresses of each processing model. According to the device described in item 9 of the scope of the patent application, wherein the processing module further includes: an acquisition unit, which is used to acquire each processing model according to the storage address of each processing model in the relevant information when the original data passes the verification; the format The conversion unit is used to convert the format of the original data to obtain the input data; the processing unit is used to sequentially input the input data into the connected processing model for processing to obtain the target data. According to the device described in item 10 of the scope of the patent application, it also includes: an effect verification module, which is used to test the target according to the preset effect verification conditions. data to verify the effect. According to the device described in any one of items 8-11 of the scope of patent application, the construction module is specifically used to construct the corresponding processing model according to the logical function of each processing node in the relevant information. According to the device described in any one of items 8-11 of the scope of patent application, wherein the connection of each processing model according to the dependency relationship between the processing models includes: according to the dependency relationship between the processing nodes in the relevant information Connect each processing model through the data interface. According to the device described in any one of items 8-11 of the scope of patent application, it also includes: a landing module for storing each processing model on the ground according to the storage address of each processing model in the relevant information.

Images