CN112100201A

CN112100201A - Data monitoring method, device, equipment and storage medium based on big data technology

Info

Publication number: CN112100201A
Application number: CN202011062233.6A
Authority: CN
Inventors: 章志容; 李实�; 彭添才; 吴联波
Original assignee: Dongguan Mengda Plasticizing Science & Technology Co ltd
Current assignee: Dongguan Mengda Plasticizing Science & Technology Co ltd
Priority date: 2020-09-30
Filing date: 2020-09-30
Publication date: 2020-12-18
Anticipated expiration: 2040-09-30
Also published as: CN112100201B

Abstract

The application relates to a data monitoring method and device based on big data technology, computer equipment and a storage medium. The method comprises the following steps: acquiring a data set; determining an incidence relation between data fields in the data set; generating a data blood relationship graph formed by connecting relationship nodes according to the incidence relationship; when data abnormal alarm information is received, acquiring an abnormal data item from the abnormal alarm information; and in the data blood relationship graph, determining an early warning route graph corresponding to a field to which the abnormal data item belongs so as to determine abnormal relation nodes according to the early warning route graph. The method can improve the monitoring efficiency.

Description

Data monitoring method, device, equipment and storage medium based on big data technology

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data monitoring method and apparatus based on a big data technology, a computer device, and a storage medium.

Background

With the development of computer technology, a big data technology appears, which has the characteristic that huge data volume is the big data technology, and in the face of increasingly huge data volume, how to monitor abnormal data when the data is abnormal to find the source of the abnormal data is a problem that needs to be solved urgently in the application of the big data technology.

In the conventional technology, data of each node is searched in a manual mode to determine a relation node with a problem, and when the data volume is large, the efficiency of searching an abnormal relation node is low.

Disclosure of Invention

In view of the above, it is necessary to provide a data monitoring method, apparatus, computer device and storage medium based on big data technology, which can improve monitoring efficiency.

A data monitoring method based on big data technology, the method comprising:

acquiring a data set;

determining an incidence relation between data fields in the data set;

generating a data blood relationship graph formed by connecting relationship nodes according to the incidence relationship;

when data abnormal alarm information is received, acquiring an abnormal data item from the abnormal alarm information;

and in the data blood relationship graph, determining an early warning route graph corresponding to a field to which the abnormal data item belongs so as to determine abnormal relation nodes according to the early warning route graph.

In one embodiment, the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; the determining the association relationship between the data fields in the data set includes:

acquiring an operation statement for operating data corresponding to each data field in the data set;

acquiring the operation algorithm according to the operation statement adopted by each operation node;

determining an incidence relation between the source field and the target field according to the operation algorithm; the target field is a field resulting from the operation of the source field based on the operation algorithm.

In one embodiment, the method further comprises:

for the target fields in the relation nodes, acquiring the operation time for generating the data in the target fields;

acquiring a target operation node corresponding to an operation account generating the data according to the latest operation time;

determining a source field operated by an operation account corresponding to the target operation node;

and determining the incidence relation between the source field and the corresponding target field according to the operation algorithm, wherein the operation algorithm is the algorithm stored in the target operation node.

In one embodiment, the relationship node is an analysis node storing a corresponding source field, analysis algorithm, and target field; the determining the association relationship between the data fields in the data set includes:

acquiring an analysis statement for analyzing each data field in the data set;

acquiring the analysis algorithm according to the analysis statement adopted at each analysis node;

determining an incidence relation between the source field and the target field according to the analysis algorithm; the target field is a field that is analyzed for the source field based on the analysis algorithm.

In one embodiment, the method further comprises:

for the target fields in the relation nodes, acquiring analysis time for generating data in the target fields;

acquiring a target analysis node for generating the data according to the latest analysis time;

determining a source field corresponding to the target analysis node;

and determining the incidence relation between the source field and the corresponding target field according to the analysis algorithm, wherein the analysis algorithm is the algorithm stored in the target analysis node.

In one embodiment, the association relationship includes an operation association relationship between data fields generated by operating the data fields according to an operation statement and an analysis association relationship between data fields generated by operating the data fields according to an analysis statement;

the generating of the data blood relationship graph formed by connecting the relationship nodes according to the incidence relationship comprises the following steps:

and generating a data blood relationship graph formed by connecting relationship nodes based on the operation association relationship and the analysis association relationship.

In one embodiment, the relationship node stores a corresponding source field, processing algorithm, and target field; the determining, in the data blood relationship map, an early warning roadmap corresponding to a field to which the abnormal data item belongs includes:

determining a first sub-blood relationship graph corresponding to a field to which the abnormal data item belongs in the data blood relationship graph;

acquiring a first abnormal source field of the abnormal data item from a relation node of the first sub-blood relationship graph and generating a first processing algorithm of the abnormal data item based on the first abnormal source field;

verifying an algorithm relationship between the abnormal data item and the first abnormal source field according to the first processing algorithm;

when the verification fails, determining that the first abnormal source field is the end point of the early warning route map;

and determining the early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the method further comprises:

when the verification is passed, acquiring a second sub-blood-margin map to which the first abnormal source field belongs;

a second processing algorithm for obtaining a second abnormal source field from the relationship node of the second sub-kinoform and generating the first abnormal source field based on the second abnormal source field; the first exception source field is derived from the second exception source field;

verifying the algorithm relation between the first abnormal source field and the second abnormal source field according to the second processing algorithm until the verification fails, and determining the end point of the early warning route map;

In one embodiment, the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; after determining abnormal relationship nodes according to the early warning roadmap, the method further comprises:

determining account identification of an operation account corresponding to the abnormal relation node;

and sending early warning information to the operation account and the associated account according to the account identification.

A data monitoring device based on big data technology, the device comprising:

an acquisition module for acquiring a data set;

the determining module is used for determining the incidence relation among the data fields in the data set;

the production module is used for generating a data blood relation graph formed by connecting the relation nodes according to the incidence relation;

the acquisition module is further used for acquiring abnormal data items from the abnormal alarm information when the data abnormal alarm information is received;

the determining module is further configured to determine, in the data blood relationship map, an early warning route map corresponding to a field to which the abnormal data item belongs, so as to determine an abnormal relationship node according to the early warning route map.

In one embodiment, the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; the determining module is further configured to:

In one embodiment, the apparatus further comprises:

the acquisition module is used for acquiring the operation time for generating the data in the target field for the target field in each relationship node; acquiring a target operation node corresponding to an operation account generating the data according to the latest operation time;

the determining module is further configured to determine a source field operated by an operation account corresponding to the target operation node; and determining the incidence relation between the source field and the corresponding target field according to the operation algorithm, wherein the operation algorithm is the algorithm stored in the target operation node.

In one embodiment, the relationship node is an analysis node storing a corresponding source field, analysis algorithm, and target field; the determining module is further configured to:

acquiring an analysis statement for analyzing each data field in the data set;

In one embodiment, the apparatus further comprises:

the acquisition module is further used for acquiring analysis time for generating data in the target field for the target field in each relationship node; acquiring a target analysis node for generating the data according to the latest analysis time;

the determining module is further configured to determine a source field corresponding to the target analysis node; and determining the incidence relation between the source field and the corresponding target field according to the analysis algorithm, wherein the analysis algorithm is the algorithm stored in the target analysis node.

In one embodiment, the association relationship includes an operation association relationship between data fields generated by operating the data fields according to an operation statement and an analysis association relationship between data fields generated by operating the data fields according to an analysis statement; the generation module is further configured to:

In one embodiment, the relationship node stores a corresponding source field, processing algorithm, and target field; the determining module is further configured to:

In one embodiment, the apparatus further comprises:

the obtaining module is further configured to obtain a second sub-sphygmogram to which the first abnormal source field belongs when the verification is passed; a second processing algorithm for obtaining a second abnormal source field from the relationship node of the second sub-kinoform and generating the first abnormal source field based on the second abnormal source field; the first exception source field is derived from the second exception source field;

the verification module is used for verifying the algorithm relation between the first abnormal source field and the second abnormal source field according to the second processing algorithm until the verification fails, and determining the end point of the early warning route map;

the determining module is further configured to determine the early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; the device further comprises:

the determining module is further configured to determine an account identifier of an operation account corresponding to the abnormal relationship node;

and the sending module is used for sending early warning information to the operation account and the associated account according to the account identification.

A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the big data technology based data monitoring method when executing the computer program.

A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the big data technology based data monitoring method.

In the above embodiment, the server first determines the association relationship between the data fields in the data set, and generates the data blood relationship graph according to the association relationship between the data fields. And when the server receives the data abnormal alarm information, acquiring an abnormal data item from the abnormal alarm information, and determining an early warning route map corresponding to a field to which the abnormal data item belongs in the data blood relationship map. And the server traces back the data fields having the association relation with the fields to which the abnormal data items belong according to the early warning route map, and determines the abnormal relation nodes by judging whether the relation between the data fields is the same as the association relation represented by the corresponding relation nodes in the early warning route map. The server automatically monitors and analyzes the problem source of the abnormal data item, and the abnormal relation node can be determined without manual intervention, so that the working efficiency of monitoring the data quality is improved.

Drawings

FIG. 1 is a diagram of an application environment of a data monitoring method based on big data technology in one embodiment;

FIG. 2 is a schematic flow chart of a data monitoring method based on big data technology in one embodiment;

FIG. 3 is a flow diagram illustrating an embodiment of determining associations between source fields and target fields;

FIG. 4 is a flow diagram illustrating the process of determining a source field of a destination field in one embodiment;

FIG. 5 is a flow diagram illustrating a process for determining associations between source fields and target fields according to another embodiment;

FIG. 6 is a flow diagram illustrating the process of determining a source field of a target field in another embodiment;

FIG. 7 is a schematic flow chart for determining an early warning route map in one embodiment;

FIG. 8 is a schematic diagram of an alert roadmap in one embodiment;

FIG. 9 is a block diagram of a data monitoring device based on big data technology in one embodiment;

FIG. 10 is a block diagram of a data monitoring device based on big data technology in another embodiment;

FIG. 11 is a diagram illustrating an internal structure of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The data monitoring method based on the big data technology can be applied to the application environment shown in fig. 1. The server 104 communicates with the service platform device 102 through a network, acquires data in a data set from the service platform device 102, and determines an association relationship between data fields in the data set. In one embodiment, the server 104 may also generate data in the data set according to its own business program, and determine an association relationship between data fields in the data set. The server 104 generates a data blood relationship graph according to the association relationship. When the server receives the data abnormal alarm information, an abnormal data item is obtained from the abnormal alarm information, and an early warning route map corresponding to a field to which the abnormal data item belongs is determined in the data blood margin map, so that the early warning route map is pushed to the analysis terminal 106, and the analysis terminal 106 determines an abnormal relation node according to the early warning route map. In one embodiment, the server may also determine the abnormal relationship node according to the early warning roadmap.

The server 104 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.

In one embodiment, as shown in fig. 2, a data monitoring method based on big data technology is provided, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:

s202, the server acquires a data set.

Wherein a data set is a collection of data. The data in the dataset may be structured data or unstructured data. The data may be, for example, financial data, order data, logistics data, etc., or e-mail data, logs, audio-visual data, etc. When the data in the dataset is unstructured, the server may convert the unstructured data to structured data.

In one embodiment, the server obtains data from the service platform through a data collection technology, and the obtained data form a data set. The server obtains data generated by the operation triggered in the service platform through a data acquisition technology. The data generated by the operation triggered in the service platform includes a data item as an operation object, an identification of an operation account operating on the data item, an operation algorithm operating on the data item, and the like. For example, the commodity purchaser adds data items such as an order amount, a commodity quantity and the like by triggering a purchase operation, an operation mode corresponding to the data items is added, and the account identifier of the operation account may be an account ID of the operation account. The server records the source of the data items by acquiring the data items such as the order amount, the commodity quantity and the like, and the operation algorithm, the account identification of the operation account and the like corresponding to the data items.

In one embodiment, the server obtains the data set by executing a program preset by itself. For example, a program for adding the data field a and the data field B is preset in the server, and the server adds the data field a and the data field B to obtain a newly added data field C. The server records the source of the data field C by recording the data field C and the algorithm that generated the data field C.

S204, the server determines the association relation among the data fields in the acquired data set.

The association relationship between the data fields refers to the relationship between the data fields from generation to operation processing, transmission and circulation, application to final extinction.

For example, modifying the data field a to generate a data field B, and performing an operation on the data field B to generate a data field C, where the data field a is a source of the data field B, the data field B is a source of the data field C, and there is an association relationship among the data field a, the data field B, and the data field C.

For example, a commodity purchaser generates a purchase order by triggering a purchase operation, the purchase order including an order quantity. After the order processing personnel process the purchase order, an invoice is generated according to the order quantity, and the invoice comprises the delivery quantity. The invoice is then sent to the warehouse manager so that the warehouse manager can deliver the goods according to the invoice. And when the warehouse manager delivers the goods, recording the delivery quantity. Then, the order quantity is a source of the shipment quantity, the shipment quantity is a source of the ex-warehouse quantity, and the order quantity, the shipment quantity, and the ex-warehouse quantity have an association relationship.

For example, the server performs analysis operation on the field a through a preset analysis program to generate a field B, and then performs analysis operation on the field B to generate a field C. And then the server sends the field C to the service platform, and the service platform modifies the field C into the field D through the triggered modification operation. Then, field a is the source of field B, field B is the source of field C, field C is the source of field D, and there is an association between fields A, B, C, D.

For example, after a commodity purchaser purchases a commodity in a web page, a purchase record is stored in a field of the data table a. If the server receives a command to view the sales of the goods in 3 months, the server summarizes data sheet A and other data sheets related to the purchase records according to the program corresponding to the command. During the summarization process, intermediate data may be stored using intermediate table B, resulting in data table C representing sales for 3 months. Then the fields in data table C have an association with the fields in intermediate table B and the fields in data table a.

For example, the server pushes the data field B corresponding to the result of the analysis operation performed on the data field a to the service platform, and the service platform applies the data field B and generates the data field C. And the service platform records the application mode of the data field B and feeds the application mode back to the server. The server can obtain the application destination of the data field B according to the application mode of the data field, and determine that the data field A, the data field B and the data field C have an association relation.

For example, a user performs a service operation through a service platform, and the service platform generates user historical behavior data according to the service operation of the user and sends the data to a data center. And the data center station establishes a prediction model of the user behavior according to the received historical behavior data and sends the prediction model to the service platform so that the service platform provides service for the user according to the prediction model. And after the user operates according to the service provided by the service platform based on the prediction model, the operation data of the user is used as the historical behavior data of the user and is sent to the data center. And after receiving the data, the data center updates the prediction model according to the received data, sends the updated prediction model to the service platform, and the process is repeated.

In one embodiment, the business operation performed by the user on the business platform may be a purchasing operation, and the business platform generates historical behavior data of the user purchasing the goods according to the purchasing operation of the user. And the data center establishes a prediction model purchased by the user according to the historical behavior data of the user, and sends the prediction model to the service platform. For example, after an old user comes in, the prediction model generates a recommended commodity list through the prediction model, and commodities are recommended to the old user through the recommended commodity list. The recommended commodity list browsed by the user is generated through a prediction model, the commodities purchased by the user are in a correlation relationship with the prediction model, and the prediction model is in a correlation relationship with historical behavior data of the user. After the new user comes in, the service platform does not have a prediction model generated according to the historical behavior data of the new user, so that commodities cannot be recommended to the new user according to the prediction model. The commodity list seen by the new user is the default commodity list of the service platform. The commodity order data generated by the user's purchasing behavior is also independent of the predictive model.

The server traces the source of the application data through the prediction models, combines behavior data of actual commodities purchased by the users and prediction accuracy preset for each prediction model by the server, calculates the actual purchase rate based on statistical data of actual recommended commodities purchased by old users by analyzing the data, and obtains the accuracy of the prediction models used by the old users. If the difference between the commodities actually purchased by the old user and the commodities recommended by the prediction model is obtained through analysis, the server monitors the accuracy rate of the prediction model and sets the occurrence of problems or the parameters of the prediction model or the algorithm of the model and the like. The server generates a data blood relationship graph representing the incidence relation among the data fields, generates an early warning route graph according to the data blood relationship graph, and then automatically finds out abnormal relation nodes and causes of the abnormality through the early warning route graph.

And S206, the server generates a data blood relation graph formed by connecting the relation nodes according to the incidence relation.

The data blood relationship graph is an abstract logic graph describing the association relationship between data fields. The data-blood-margin graph records the processing of data fields from generation to passage, and finally the flow and application in time sequence. The data lineage graph can be abstracted as being connected by relational nodes. The incidence relation between the data fields can be obtained through the relation nodes, and the sources of the data fields can be traced through the incidence relation between the data fields.

The relationship node corresponds to a specific storage area and is used for storing a source field corresponding to the relationship node, an algorithm for processing the source field, a target field generated after the source field is processed by the algorithm, and the like. The relation node can store a plurality of source fields and a plurality of target fields, and the corresponding relation between the source fields and the target fields can be determined through an algorithm. For example, the source field is (a)₁，a₂，a₃) By the algorithm f (a)₁，a₂，a₃) To (a)₁，a₂，a₃) Obtaining a target field b after operation₁I.e. f (a)₁，a₂，a₃)＝b₁. For example, the source field is (a)₄，a₅) By algorithm g (a)₄，a₅) To (a)₄，a₅) Obtaining a target field b after operation₂I.e. g (a)₄，a₅)＝b₂。

The source field stored by the relationship node may be generated by operating on one or more other data fields, or may be generated by an operation triggered at the service platform.

The server can operate on the source fields stored by the relation nodes in the data blood relationship graph by running an algorithm in a program to generate the target fields. The program run by the server may be a program called according to an instruction triggered in the service platform, or may be a working program preset by the server. The working program preset by the server can be a program for analyzing and processing the data fields of the data set.

S208, when the data abnormal alarm information is received, the server acquires an abnormal data item from the abnormal alarm information.

The data abnormity alarm information is information for informing the server that the data item is abnormal.

The data abnormality alarm information may be alarm information generated by an alarm instruction triggered in the service platform. For example, the alarm information can be input in an intelligent customer service window of the service platform.

The data abnormal alarm information may be alarm information generated when the server checks each data item based on a monitoring program and finds that the association relationship between the data fields is abnormal. For example, the server may set the data in the data set to be checked once at preset time intervals according to a preset program to determine whether the data in the data field is abnormal at the relationship node.

The abnormal alarm information may include an abnormal data item, time when an abnormality occurs, an identifier of an alarm person who triggers an alarm instruction, and the like.

Wherein the abnormal data item is data in which an abnormality has occurred. For example, it may be 30 shipment quantities in item order A. For example, it may be the best brand name aaa sold in 3 months.

S210, in the data blood relationship graph, the server determines an early warning route graph corresponding to a field to which the abnormal data item belongs so as to determine abnormal relation nodes according to the early warning route graph.

After the server acquires the abnormal data item, determining the field to which the abnormal data item belongs. For example, when the abnormal data item is the total transaction amount of the a commodity in one week, the server determines that the corresponding field is the commodity transaction amount field. Then, the server traces the source field of the field from the field to which the abnormal data item belongs in the data blood relationship graph corresponding to the field to which the abnormal data item belongs, and generates an early warning route graph according to the field to which the abnormal data item belongs and the source field.

For example, the server traces back the source field of the total commodity transaction amount field from the total commodity transaction amount field in the data consanguinity corresponding to the total commodity transaction amount field. And generating a corresponding early warning route map according to the commodity transaction total amount field and the source field thereof. For example, the server traces back to the commodity transaction total amount field at the previous stage as a weekday commodity transaction amount field, and the commodity transaction amount on the weekday at the previous stage as a saturday commodity transaction amount field, sequentially traces back until the commodity transaction amount field on the monday is traced back, and generates an early warning route map according to the traced back fields.

In one embodiment, after the early warning route map is generated, the server pushes the early warning route map to the analysis terminal, so that the analysis terminal performs analysis processing according to the early warning route map and finds the abnormal relationship node.

In one embodiment, the server verifies whether the actual relationship between the data fields in the early warning route map is the same as the incidence relationship represented by the relation nodes in the early warning route map, and determines abnormal relation nodes according to the verification result. For example, it is known from an algorithm stored in the relationship node a in the early warning route map that the relationship between the source field and the target field of the relationship node a is "the value in the target field is 10 times the value in the source field", but the server finds that the value of the data in the source field is 15 and the value of the data in the target field is 27, and since 27 is not 10 times 15, the relationship node a is abnormal and is an abnormal relationship node.

In one embodiment, if the receiving quantity is abnormal, the server traces back from the receiving quantity field according to an early warning route map corresponding to the receiving quantity, and if the tracing back is that the ex-warehouse quantity is inconsistent with the order quantity, the server determines that the ex-warehouse node is an abnormal relational node.

In one embodiment, after determining the abnormal relationship node, the server determines the cause of the abnormality by examining all programs that operate on the corresponding data field in the abnormal relationship node. For example, the cause of the abnormality is that the operation node a performs an erroneous operation on the data field. For example, the reason for the abnormality is that the analysis operation of the analysis node B on the data field is erroneous. For example, the cause of the anomaly is that data in the system is updated. For example, the reason for the exception is that an illegal program attacks the data field in the abnormal relational node.

In one embodiment, the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; as shown in fig. 3, the server determining the association relationship between the data fields in the data set includes the following steps:

s302, acquiring an operation statement for operating data corresponding to each data field in the data set.

S304, acquiring an operation algorithm according to the operation statement adopted by each operation node.

S306, the server determines the incidence relation between the source field and the target field according to an operation algorithm; the destination field is a field that results from operating on the source field based on an operation algorithm.

The operation node is a relation node storing a corresponding source field, an operation algorithm and a target field. And operating the corresponding data in the data field by the operation account corresponding to the operation node through an operation algorithm stored in the operation node. The operation executed by the operation account comprises adding, deleting, browsing, modifying and the like. The data field of the operated account for performing various operations is the source field corresponding to the operation node.

And after the operation account operates the data in the source field, generating a target field, wherein the generated target field can be a source field corresponding to other operation nodes. And generating an operation association relation between the source field and the target field corresponding to the operation node. For example, after the operation account 1 corresponding to the operation node 1 operates the data in the data field a, the data field B is generated, and after the operation account 2 corresponding to the operation node 2 operates the data field B, the data field C is generated. An operational association is generated between the data fields A, B, C.

The operation statement is an executable program statement executed when the corresponding data in the data field is operated. For example, the program statements may be written in SQL, Python, or other programming languages.

And the server acquires an operation algorithm for operating the source field stored by the operation node according to the operation statement adopted by each operation node, and determines the association relationship between the source field and the target field stored by the operation node according to the operation algorithm. For example, at an operation node, the operation node modifies a data a in a data field a by using an update statement written in SQL language to generate a data field B, so that a modification relationship is generated between the data field a and the data field B, and the server can determine that the modified data a is data a by using the modification statement.

In one embodiment, as shown in fig. 4, the method further comprises the steps of:

s402, for the target field in each relation node, obtaining the operation time for generating the data in the target field.

S404, according to the latest operation time, acquiring a target operation node corresponding to the operation account generating the data.

S406, determining a source field operated by the operation account corresponding to the target operation node, and an operation algorithm for operating data in the source field.

S408, determining the incidence relation between the source field and the target field according to an operation algorithm, wherein the operation algorithm is an algorithm stored in the target operation node.

Wherein the operation time is a time when an operation at the operation node is triggered. E.g. at t₁At the moment, the operation account triggers the operation of adding a data field on the service platform, and the service platform records the time t₁。

The data in the same target field can be obtained by operating the data in different source fields by operation accounts corresponding to different operation nodes. The server obtains the operation time of the data in the target field by operating the data in different source fields by different operation nodes, and obtains the corresponding target operation node when the data in the data field is generated by the operation of the operation algorithm for the last time according to the operation time corresponding to each operation node. The server determines a source field operated by an operation account corresponding to the target operation node, and an operation algorithm for operating data in the source field, and determines an association relation between the source field and the target field according to the operation algorithm.

The server acquires the source field and the operation algorithm corresponding to the last time when the data in the data field is generated through the operation of the operation algorithm, and can trace the operation account and the source field corresponding to the data in the target field and the corresponding operation algorithm according to the latest operation time. Therefore, when data in the data field is abnormal, the server can trace the source of the abnormal data according to the incidence relation determined by each relation node so as to search the reason of the abnormal data.

In one embodiment, the relationship node is an analysis node that stores the corresponding source field, analysis algorithm, and target field. As shown in fig. 5, the server determining the association relationship between the data fields in the data set includes the following steps:

s502, an analysis statement for analyzing each data field in the data set is obtained.

S504, an analysis algorithm is obtained according to the analysis sentences adopted by each analysis node.

S506, determining the incidence relation between the source field and the target field according to an analysis algorithm; the target field is a field that is analyzed for the source field based on the analysis algorithm.

The analysis node is a node for performing various different analysis operations on the data fields of the data set according to the analysis statement. For example, the analysis operation may be a cleaning, pruning, statistical or mathematical operation on the data fields.

And the analysis node adopts an analysis algorithm to perform analysis operation on the data field to generate an intermediate field or a result field, and then performs analysis operation on the intermediate field to continue generating the intermediate field or the result field.

The analysis statement is an executable program statement executed when the analysis node analyzes the data field. The analysis node may perform analysis operation on the data field according to a preset script file (in which an analysis statement is recorded).

And after the analysis node analyzes and operates the data in the data field, generating a new data field, and then, analyzing and operating the generated data field by the analysis node to generate the new data field. An analytical relationship is generated between the initial data field and a subsequently generated data field. For example, after the analysis node performs analysis operation on the data in the data field a, the data field B is generated, and then the analysis operation is performed on the data field B, the data field C is generated, and the analysis association relationship is generated between the data fields A, B, C. For example, after the analysis node performs an addition operation on the data fields a and B, a data field C is generated, and then a data item smaller than a threshold in the data field C is modified to generate a modified data field D, so that an analysis association relationship is generated between the data fields A, B, C, D.

In one embodiment, the server stores the data result obtained by performing the analysis operation on the data set into the intermediate data set, and then stores the data result obtained by performing the analysis operation on the intermediate data into the intermediate data set or the result data set. The server can store the data result obtained after the analysis and operation on the data to the intermediate data set or the result data set according to the requirement, so that the data result obtained after each analysis and operation can be managed conveniently.

In one embodiment, as shown in fig. 6, the method further comprises the steps of:

s602, for the target field in each relation node, obtaining the analysis time for generating the data in the target field.

S604, acquiring a target analysis node for generating data according to the latest analysis time.

S606, determining a source field corresponding to the target analysis node.

S608, determining the incidence relation between the source field and the corresponding target field according to an analysis algorithm, wherein the analysis algorithm is stored in the target analysis node.

The analysis time is the time for analyzing the data in the data field according to a preset analysis program. For example, the workflow of the analysis program is at t₁The data in the data field is statistically analyzed at the moment, and the server records the time t₁。

The data in the same target field can be obtained by analyzing the data in different source fields by the analysis accounts corresponding to different analysis nodes. The server obtains the analysis time of the data in the target field by analyzing the data in different source fields by different analysis nodes, and obtains the corresponding target analysis node when the data in the data field is generated through the analysis operation for the last time according to the analysis time corresponding to each analysis node. The server determines a source field analyzed by an analysis account corresponding to the target analysis node, and an analysis algorithm for operating data in the source field, and determines an association relation between the source field and the target field according to the analysis algorithm.

The server acquires the source field and the analysis algorithm corresponding to the data in the data field generated by the analysis operation for the last time, and can trace the analysis account and the source field corresponding to the data in the target field and the corresponding analysis algorithm according to the latest analysis time. Therefore, when data in the data field is abnormal, the server can trace the source of the abnormal data according to the incidence relation determined by each relation node so as to search the reason of the abnormal data.

In one embodiment, the association relationship includes an operation association relationship generated by operating data corresponding to the data field by different operation nodes and an analysis association relationship generated by analyzing the data field by different analysis nodes; the step of generating the data blood relationship graph by the server according to the incidence relation comprises the following steps: and generating a data blood relationship graph based on the operation association relation and the analysis association relation.

And generating operation association relation because part of data fields in the data set are operated by different operation nodes. The data fields operated by the operation nodes may be further analyzed and operated by the analysis nodes to generate analysis association relations. And the server generates a data blood relationship graph based on the operation association relation and the analysis association relation among the data fields.

In one embodiment, the server pushes the data field corresponding to the analysis and operation result to the service platform, and the service platform applies the data field, records the application mode of the data field, and feeds the application mode back to the server. And the server generates a data blood relationship graph according to the operation association relation and the analysis association relation among the data fields and the application mode of the data fields.

In one embodiment, the relationship node stores a corresponding source field, processing algorithm, and target field; as shown in fig. 7, the step of determining, by the server, the early warning route map corresponding to the field to which the abnormal data item belongs in the data blood relationship map includes the following steps:

s702, in the data blood relationship graph, determining a first sub blood relationship graph corresponding to a field to which the abnormal data item belongs.

S704, a first abnormal source field of the abnormal data item is obtained from the relation node of the first sub-blood relationship graph, and a first processing algorithm for generating the abnormal data item based on the first abnormal source field is obtained.

S706, verifying the algorithm relationship between the abnormal data item and the first abnormal source field according to the first processing algorithm.

S708, when the verification fails, determining that the first abnormal source field is the end point of the early warning route map.

And S710, determining an early warning route map according to the abnormal data item and the end point of the early warning route map.

The sub-blood relationship graph is a part of the data blood relationship graph and describes the association relationship between a certain data field and a source field in the data set.

And when the server receives the data abnormal alarm information, acquiring an abnormal data item from the abnormal alarm information, and determining a field to which the abnormal data item belongs according to the abnormal data item. And then, in the data blood relationship graph, determining a sub blood relationship graph corresponding to the field to which the abnormal data item belongs.

The early warning route map is a route map for tracing an abnormal source of the abnormal data item.

After the server acquires the abnormal data item, firstly, the sub-blood relationship graph corresponding to the field to which the abnormal data item belongs is determined, and the relation node in the sub-blood relationship graph is acquired. The server obtains a first source field of the abnormal data item from the relation node and generates a processing algorithm of the abnormal data item based on the source field. The server verifies the algorithm relationship between the abnormal data item and the first source field according to the processing algorithm. When the verification fails, the algorithm relationship between the abnormal data item and the first source field is not consistent with the processing algorithm stored in the relationship node, so that the source field is the end point of the early warning route map. And the server determines an early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, when the verification is passed, the server acquires a second sub-chart to which the first abnormal source field belongs; a second processing algorithm for acquiring a second abnormal source field from the relationship node of the second sub-blood relationship graph and generating a first abnormal source field based on the second abnormal source field; the first exception source field is derived from the second exception source field; verifying the algorithm relation between the first abnormal source field and the second abnormal source field according to a second processing algorithm until the verification fails, and determining the end point of the early warning route graph; and determining an early warning route map according to the abnormal data item and the end point of the early warning route map.

And when the verification is passed, the server continuously traces back the data fields with problems to obtain a second sub-blood relationship graph to which the first abnormal source field belongs. And a second processing algorithm for acquiring a second abnormal source field from the relationship node of the second sub-blood relationship graph and generating the first abnormal source field based on the second abnormal source field. And if the algorithm relationship between the first exception source field and the second exception source field is not consistent with the processing algorithm stored in the relationship node, the second exception source field is the end point of the early warning route graph. If the source fields are consistent with the abnormal data items, the source of the second abnormal source field is continuously traced until a relation node is traced, wherein the algorithm relation between the source field and the target field is inconsistent with the processing algorithm stored by the relation node, and the source field stored by the relation node and corresponding to the abnormal data items is the end point of the early warning route graph.

For example, as shown in fig. 8, in order a, the number of items purchased by the user is 10, and the generated order number 10 is stored in the order number field. And when the user receives 6 commodities, the service platform receives the alarm information sent by the user. And the server acquires the information that the receiving quantity is abnormal according to the alarm information received by the service platform and determines that the abnormal data item is the receiving quantity corresponding to the order A. And the server traces back the source field of the receiving quantity field according to the receiving quantity field to which the receiving quantity of the order A belongs, firstly traces back the logistics quantity field, verifies whether the logistics quantity is consistent with the receiving quantity, and if not, determines that the logistics quantity field is the end point of the early warning route chart. If the number of the shipments is consistent, the number of the shipments is continuously traced back to the field of the number of the shipments, whether the field of the number of the shipments is consistent with the field of the logistics number is verified, and if the field of the number of the shipments is not consistent with the field of the logistics number, the field of the number of the shipments is determined. If the two are consistent, the tracing back is continued.

The server verifies the relationship between the source field and the target field of each relationship node in the data blood relationship graph, judges whether the relationship between the source field and the target field stored by each relationship node is consistent with the processing algorithm stored by the relationship node, and determines the end point of the early warning route graph according to the verification result, so that the speed is high, and the error is small.

In one embodiment, the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; after determining the abnormal relation node according to the early warning roadmap, the method further comprises the following steps: determining account identification of an operation account corresponding to the abnormal relation node; and sending early warning information to the operation account and the associated account according to the account identification.

The early warning information is information that the server notifies the operation account that an error occurs in the process of operating or analyzing the data. The warning information may include the time when the error occurred, the data field in which the error occurred, and the like.

The server can send the early warning information to the operation account and the associated account according to the account identification of the operation account.

After the server sends the early warning information to the operation account, the server can continuously follow up the error correction processing process of the operation account, and feed back the problem processing progress to the user who initiates the alarm, or record the problem processing progress. The problem processing progress is fed back to the user, so that the user can know the problem processing process conveniently, and correct the problem processing process in time when the problem processing process is incorrect. Recording the problem processing progress facilitates the server to manage the reason of the error, the data field of the error and the problem processing process.

It should be understood that although the various steps in the flow charts of fig. 2-7 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-7 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.

In one embodiment, as shown in fig. 9, there is provided a data monitoring device based on big data technology, including: an obtaining module 902, a determining module 904, and a generating module 906, wherein:

an obtaining module 902, configured to obtain a data set;

a determining module 904, configured to determine an association relationship between data fields in a data set;

a generating module 906, configured to generate a data blood relationship graph formed by connecting relationship nodes according to the association relationship;

the obtaining module 902 is further configured to obtain an abnormal data item from the abnormal alarm information when the data abnormal alarm information is received;

the determining module 904 is further configured to determine, in the data blood relationship map, an early warning route map corresponding to a field to which the abnormal data item belongs, so as to determine an abnormal relationship node according to the early warning route map.

In the above embodiment, the server first determines the association relationship between the data fields in the data set, and generates the data blood relationship graph according to the association relationship between the data fields. And when the server receives the data abnormal alarm information, acquiring an abnormal data item from the abnormal alarm information, and determining an early warning route map corresponding to a field to which the abnormal data item belongs in the data blood relationship map. And the server traces back the data fields having the association relation with the fields to which the abnormal data items belong according to the early warning route map, and determines abnormal relation nodes by judging whether the relation between the data fields is the same as the association relation between the data fields represented in the early warning route map. The server automatically monitors and analyzes the problem source of the abnormal data item, the abnormal relation node can be determined without manual intervention, and the work efficiency of monitoring the data quality is improved.

In one embodiment, the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; a determining module 904, further configured to:

acquiring an operation statement for operating data corresponding to each data field in a data set;

acquiring an operation algorithm according to the operation statement adopted by each operation node;

determining an incidence relation between a source field and a target field according to an operation algorithm; the destination field is a field that results from operating on the source field based on an operation algorithm.

In one embodiment, the apparatus further comprises:

the obtaining module 902 is further configured to generate, for a target field in each relationship node, an operation time of data in the target field; acquiring a target operation node corresponding to an operation account generating data according to the latest operation time;

the determining module 904 is further configured to determine a source field operated by the operation account corresponding to the target operation node; and determining the incidence relation between the source field and the corresponding target field according to an operation algorithm, wherein the operation algorithm is an algorithm stored in the target operation node.

In one embodiment, the relationship node is an analysis node storing a corresponding source field, analysis algorithm, and target field; a determining module 904, further configured to:

acquiring an analysis statement for analyzing each data field in the data set;

obtaining an analysis algorithm according to the analysis statement adopted at each analysis node;

determining an incidence relation between a source field and a target field according to an analysis algorithm; the target field is a field that is analyzed for the source field based on an analysis algorithm.

In one embodiment, the apparatus further comprises:

an obtaining module 902, configured to obtain, for a target field in each relationship node, analysis time for generating data in the target field; acquiring a target analysis node for generating data according to the latest analysis time;

a determining module 904, configured to determine a source field corresponding to the target analysis node; and determining the incidence relation between the source field and the corresponding target field according to the analysis algorithm, wherein the analysis algorithm is the algorithm stored in the target analysis node.

In one embodiment, the association relationship includes an operation association relationship between data fields generated by operating on the data fields according to the operation statement and an analysis association relationship between data fields generated by operating on the data fields according to the analysis statement; a generating module 906 further configured to:

and generating a data blood relationship graph formed by connecting the relation nodes based on the operation incidence relation and the analysis incidence relation.

In one embodiment, the relationship node stores a corresponding source field, processing algorithm, and target field; a determining module 904, further configured to:

determining a first sub-blood relationship graph corresponding to a field to which an abnormal data item belongs in the data blood relationship graph;

acquiring a first abnormal source field of the abnormal data item from the relation node of the first sub-blood relationship graph and generating a first processing algorithm of the abnormal data item based on the first abnormal source field;

verifying the algorithm relation between the abnormal data item and the first abnormal source field according to a first processing algorithm;

when the verification fails, determining the first abnormal source field as the end point of the early warning route graph;

and determining an early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, as shown in fig. 10, the apparatus further comprises:

an obtaining module 902, further configured to obtain, when the verification passes, a second sub-limbus map to which the first abnormal source field belongs; a second processing algorithm for acquiring a second abnormal source field from the relationship node of the second sub-blood relationship graph and generating a first abnormal source field based on the second abnormal source field; the first exception source field is derived from the second exception source field;

the verification module 908 is configured to verify an algorithm relationship between the first abnormal source field and the second abnormal source field according to a second processing algorithm, and determine an end point of the early warning route map until the verification fails;

the determining module 904 is further configured to determine an early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; the device still includes:

the determining module 904 is further configured to determine an account identifier of an operation account corresponding to the abnormal relationship node;

and a sending module 910, configured to send warning information to the operation account and the associated account according to the account identifier.

For specific limitations of the data monitoring device based on the big data technology, reference may be made to the above limitations of the data monitoring method based on the big data technology, and details are not repeated here. The modules in the data monitoring device based on big data technology can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 11. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data monitoring data based on big data technology. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data monitoring method based on big data technology.

Those skilled in the art will appreciate that the architecture shown in fig. 11 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring a data set; determining an incidence relation between data fields in the data set; generating a data blood relationship graph formed by connecting relationship nodes according to the incidence relationship; when data abnormal alarm information is received, acquiring an abnormal data item from the abnormal alarm information; and in the data blood relationship graph, determining an early warning route graph corresponding to a field to which the abnormal data item belongs so as to determine abnormal relation nodes according to the early warning route graph.

In one embodiment, the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; the processor, when executing the computer program, further performs the steps of: acquiring an operation statement for operating data corresponding to each data field in the data set; acquiring the operation algorithm according to the operation statement adopted by each operation node; determining an incidence relation between the source field and the target field according to the operation algorithm; the target field is a field resulting from the operation of the source field based on the operation algorithm.

In one embodiment, the processor, when executing the computer program, further performs the steps of: for the target fields in the relation nodes, acquiring the operation time for generating the data in the target fields; acquiring a target operation node corresponding to an operation account generating data according to the latest operation time; determining a source field operated by an operation account corresponding to the target operation node; and determining the incidence relation between the source field and the corresponding target field according to an operation algorithm, wherein the operation algorithm is an algorithm stored in the target operation node.

In one embodiment, the relationship node is an analysis node storing a corresponding source field, analysis algorithm, and target field; the processor, when executing the computer program, further performs the steps of: acquiring an analysis statement for analyzing each data field in the data set; acquiring the analysis algorithm according to the analysis statement adopted at each analysis node; determining an incidence relation between the source field and the target field according to the analysis algorithm; the target field is a field that is analyzed for the source field based on the analysis algorithm.

In one embodiment, the processor, when executing the computer program, further performs the steps of: for the target fields in the relation nodes, acquiring analysis time for generating data in the target fields; acquiring a target analysis node for generating data according to the latest analysis time; determining a source field corresponding to a target analysis node; and determining the incidence relation between the source field and the corresponding target field according to an analysis algorithm, wherein the analysis algorithm is an algorithm stored in the target analysis node.

In one embodiment, the association relationship includes an operation association relationship between data fields generated by operating the data fields according to an operation statement and an analysis association relationship between data fields generated by operating the data fields according to an analysis statement; the processor, when executing the computer program, further performs the steps of: and generating a data blood relationship graph formed by connecting relationship nodes based on the operation association relationship and the analysis association relationship.

In one embodiment, the relationship node stores a corresponding source field, processing algorithm, and target field; the processor, when executing the computer program, further performs the steps of: determining a first sub-blood relationship graph corresponding to a field to which the abnormal data item belongs in the data blood relationship graph; acquiring a first abnormal source field of the abnormal data item from a relation node of the first sub-blood relationship graph and generating a first processing algorithm of the abnormal data item based on the first abnormal source field; verifying an algorithm relationship between the abnormal data item and the first abnormal source field according to the first processing algorithm; when the verification fails, determining that the first abnormal source field is the end point of the early warning route map; and determining the early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the processor, when executing the computer program, further performs the steps of: when the verification is passed, acquiring a second sub-blood-margin map to which the first abnormal source field belongs; a second processing algorithm for obtaining a second abnormal source field from the relationship node of the second sub-kinoform and generating the first abnormal source field based on the second abnormal source field; the first exception source field is derived from the second exception source field; verifying the algorithm relation between the first abnormal source field and the second abnormal source field according to the second processing algorithm until the verification fails, and determining the end point of the early warning route map; and determining the early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; the processor, when executing the computer program, further performs the steps of: determining account identification of an operation account corresponding to the abnormal relation node; and sending early warning information to the operation account and the associated account according to the account identification.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a data set; determining an incidence relation between data fields in the data set; generating a data blood relationship graph formed by connecting relationship nodes according to the incidence relationship; when data abnormal alarm information is received, acquiring an abnormal data item from the abnormal alarm information; and in the data blood relationship graph, determining an early warning route graph corresponding to a field to which the abnormal data item belongs so as to determine abnormal relation nodes according to the early warning route graph.

In one embodiment, the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; the computer program when executed by the processor further realizes the steps of: acquiring an operation statement for operating data corresponding to each data field in the data set; acquiring the operation algorithm according to the operation statement adopted by each operation node; determining an incidence relation between the source field and the target field according to the operation algorithm; the target field is a field resulting from the operation of the source field based on the operation algorithm.

In one embodiment, the computer program when executed by the processor further performs the steps of: for the target fields in the relation nodes, acquiring the operation time for generating the data in the target fields; acquiring a target operation node corresponding to an operation account generating data according to the latest operation time; determining a source field operated by an operation account corresponding to the target operation node; and determining the incidence relation between the source field and the corresponding target field according to an operation algorithm, wherein the operation algorithm is an algorithm stored in the target operation node.

In one embodiment, the relationship node is an analysis node storing a corresponding source field, analysis algorithm, and target field; the computer program when executed by the processor further realizes the steps of: acquiring an analysis statement for analyzing each data field in the data set; acquiring the analysis algorithm according to the analysis statement adopted at each analysis node; determining an incidence relation between the source field and the target field according to the analysis algorithm; the target field is a field that is analyzed for the source field based on the analysis algorithm.

In one embodiment, the computer program when executed by the processor further performs the steps of: for the target fields in the relation nodes, acquiring analysis time for generating data in the target fields; acquiring a target analysis node for generating data according to the latest analysis time; determining a source field corresponding to a target analysis node; and determining the incidence relation between the source field and the corresponding target field according to an analysis algorithm, wherein the analysis algorithm is an algorithm stored in the target analysis node.

In one embodiment, the association relationship includes an operation association relationship between data fields generated by operating the data fields according to an operation statement and an analysis association relationship between data fields generated by operating the data fields according to an analysis statement; the computer program when executed by the processor further realizes the steps of: and generating a data blood relationship graph formed by connecting relationship nodes based on the operation association relationship and the analysis association relationship.

In one embodiment, the relationship node stores a corresponding source field, processing algorithm, and target field; the computer program when executed by the processor further realizes the steps of: determining a first sub-blood relationship graph corresponding to a field to which the abnormal data item belongs in the data blood relationship graph; acquiring a first abnormal source field of the abnormal data item from a relation node of the first sub-blood relationship graph and generating a first processing algorithm of the abnormal data item based on the first abnormal source field; verifying an algorithm relationship between the abnormal data item and the first abnormal source field according to the first processing algorithm; when the verification fails, determining that the first abnormal source field is the end point of the early warning route map; and determining the early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the computer program when executed by the processor further performs the steps of: when the verification is passed, acquiring a second sub-blood-margin map to which the first abnormal source field belongs; a second processing algorithm for obtaining a second abnormal source field from the relationship node of the second sub-kinoform and generating the first abnormal source field based on the second abnormal source field; the first exception source field is derived from the second exception source field; verifying the algorithm relation between the first abnormal source field and the second abnormal source field according to the second processing algorithm until the verification fails, and determining the end point of the early warning route map; and determining the early warning route map according to the abnormal data item and the end point of the early warning route map.

In one embodiment, the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; the computer program when executed by the processor further realizes the steps of: determining account identification of an operation account corresponding to the abnormal relation node; and sending early warning information to the operation account and the associated account according to the account identification.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A data monitoring method based on big data technology is characterized by comprising the following steps:

acquiring a data set;

determining an incidence relation between data fields in the data set;

2. The method of claim 1, wherein the relationship node is an operation node storing a corresponding source field, operation algorithm, and target field; the determining the association relationship between the data fields in the data set includes:

3. The method of claim 2, further comprising:

4. The method of claim 1, wherein the relationship node is an analysis node that stores a corresponding source field, analysis algorithm, and target field; the determining the association relationship between the data fields in the data set includes:

acquiring an analysis statement for analyzing each data field in the data set;

5. The method of claim 4, further comprising:

determining a source field corresponding to the target analysis node;

6. The method of claim 1, wherein the association comprises an operation association between data fields generated by operating on the data fields according to an operation statement and an analysis association between data fields generated by operating on the data fields according to an analysis statement;

7. The method of claim 1, wherein the relationship node stores corresponding source fields, processing algorithms, and target fields; the determining, in the data blood relationship map, an early warning roadmap corresponding to a field to which the abnormal data item belongs includes:

8. The method of claim 7, further comprising:

9. The method of claim 1, wherein the relationship node is an operation node; the operation node stores the account identification of the corresponding operation account; after determining abnormal relationship nodes according to the early warning roadmap, the method further comprises:

10. A data monitoring device based on big data technology, the device comprising:

an acquisition module for acquiring a data set;

11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.