CN117112542A - Alarm method, device and storage medium for abnormal blood edge relationship - Google Patents
Alarm method, device and storage medium for abnormal blood edge relationship Download PDFInfo
- Publication number
- CN117112542A CN117112542A CN202310956885.1A CN202310956885A CN117112542A CN 117112542 A CN117112542 A CN 117112542A CN 202310956885 A CN202310956885 A CN 202310956885A CN 117112542 A CN117112542 A CN 117112542A
- Authority
- CN
- China
- Prior art keywords
- data table
- data
- blood
- program
- edge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000008280 blood Substances 0.000 title claims abstract description 142
- 210000004369 blood Anatomy 0.000 title claims abstract description 142
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 137
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000012546 transfer Methods 0.000 claims abstract description 46
- 238000001514 detection method Methods 0.000 claims abstract description 25
- 238000010586 diagram Methods 0.000 claims abstract description 19
- 238000004458 analytical method Methods 0.000 claims description 103
- 238000012545 processing Methods 0.000 claims description 36
- 230000015654 memory Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 9
- 230000000740 bleeding effect Effects 0.000 claims description 4
- 230000002547 anomalous effect Effects 0.000 claims 2
- 238000004891 communication Methods 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000008719 thickening Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/42—Syntactic analysis
Abstract
The application discloses an alarm method, device and storage medium for abnormal blood-edge relationship, and belongs to the field of communication. The method comprises the following steps: receiving a detection request, wherein the detection request comprises a link address of at least one data table access program, and the at least one data table access program is used for accessing data in a plurality of data tables, and the plurality of data tables comprise abnormal data tables; acquiring at least one data table access program based on the link address; analyzing at least one data table access program to obtain a data blood-edge relation diagram, wherein the data blood-edge relation diagram comprises a plurality of blood-edge relations, the blood-edge relations correspond to a target data table and at least one source data table, and the blood-edge relations are used for describing data transfer from the at least one source data table to the target data table; detecting an abnormal blood edge relation in the data blood edge relation graph based on the abnormal data table, wherein a target data table corresponding to the abnormal blood edge relation is the abnormal data table; and alarming the abnormal blood edge relation. The application can improve the data security.
Description
Technical Field
The present application relates to the field of communications, and in particular, to an alarm method, apparatus, and storage medium for abnormal blood edge relationship.
Background
The data table access program is used to access the data tables, e.g., the data table access program may read data from the data tables, write data to the data tables, and/or transfer data in one or more data tables to other data tables.
For example, assume that there is data table 1, data table 2, and data table 3, and that there is a data table access program describing fusing the first column data of data table 1 and the first column data of data table 2, and saving the fused column data to the first column of data table 3. The data table access program is used to effect transfer of the data in data table 1 and data table 2 into data table 3.
Data table 3 may be an abnormal data table having a security problem, and if data in data table 1 and data table 2 is transferred to data table 3 by running the data table accessing program, a great security problem is brought to data in data table 1 and data table 2.
Disclosure of Invention
The application provides an alarm method, an alarm device and a storage medium for abnormal blood-edge relationship, so as to improve data security. The technical scheme is as follows:
in a first aspect, the application provides a method of alerting of abnormal blood-edge relationships, in which method a detection request is received, the detection request comprising a link address of at least one data table access program to be detected, the at least one data table access program being for accessing data in a plurality of data tables, the plurality of data tables comprising abnormal data tables. The at least one data table access program is obtained based on the link address. Analyzing the at least one data table access program to obtain a data blood-edge relationship graph, wherein the data blood-edge relationship graph comprises a plurality of blood-edge relationships, the blood-edge relationships correspond to a target data table and at least one source data table, the plurality of data tables comprise the at least one source data table and the target data table, and the blood-edge relationships are used for describing data transfer from the at least one source data table to the target data table. Based on the abnormal data table, detecting the abnormal blood edge relation in the data blood edge relation graph, wherein the target data table corresponding to the abnormal blood edge relation is the abnormal data table. And alarming the abnormal blood relationship.
Before the at least one data table access program is operated, the at least one data table access program is analyzed to obtain a data blood edge relationship graph, an abnormal blood edge relationship in the data blood edge relationship is detected based on an abnormal data table, and an alarm is given to the abnormal blood edge relationship. Therefore, the user can obtain the data table access program with data security based on the abnormal blood relationship, and can modify the data table access program or prevent the data table access program from running based on the abnormal blood relationship, so that the data table access program is prevented from transferring data in a normal data table to an abnormal data table when running, and the data security is improved.
In one possible implementation, for each of the at least one data table access program, a program type of the data table access program is obtained, the data table access program being used to access a portion of the data tables of the plurality of data tables. And analyzing the data table access program based on the analysis mode corresponding to the program type to obtain the data transfer condition among the partial data tables. And adding a blood edge relation in the data blood edge relation graph, wherein the blood edge relation is used for describing the data transfer condition among the partial data tables. The data table access program of different program types can be analyzed by the analysis modes corresponding to the different program types, so that the expansibility and flexibility of obtaining the data blood edges are improved.
In another possible implementation manner, when the program type is a structured query language SQL type program, an abstract syntax tree AST of the data table access program is generated based on a syntax rule of the SQL type program, and a data transfer condition between the partial data tables is obtained based on the AST. When the program type is Spark jar type program, analyzing the data table access program by using a Spark engine to obtain an execution plan tree, and obtaining the data transfer condition among the partial data tables based on the execution plan tree. When the program type is a program except the SQL type program and the Spark jar type program, the data table access program is analyzed through a self-defined analysis program, and the data transfer condition among the partial data tables is obtained. The data table access program of different program types can be analyzed by the analysis modes corresponding to the different program types, so that the expansibility and flexibility of obtaining the data blood edges are improved.
In another possible implementation, the abnormal blood-edge relationship is highlighted in the data blood-edge relationship graph; or, the abnormal blood edge relation is bolded and displayed in the data blood edge relation graph; alternatively, an anomaly flag is added to the anomaly blood relationship in the data blood relationship graph. Thus, abnormal blood relationship can be flexibly displayed.
In another possible implementation, for each data table of the plurality of data tables, when the site where the data table is located is an overseas site, or when the data table is a data table in a blacklist, the data table is determined to be an abnormal data table, and the blacklist includes at least one abnormal data table. Thus, the abnormal data table can be flexibly and conveniently checked.
In another possible implementation, the resolution success rate and/or the resolution coverage rate are/is displayed, the resolution success rate is a ratio between the number of successfully resolved data table accessing programs and the total number of the at least one data table accessing program, and the resolution coverage rate is a ratio between the number of data table accessing programs capable of resolving the bleeding edge relationship and the total number of the at least one data table accessing program.
In a second aspect, the present application provides an alarm device for abnormal blood-bearing relations for performing the method of the first aspect or any one of the possible implementations of the first aspect. In particular, the apparatus comprises means for performing the method of the first aspect or any one of the possible implementations of the first aspect.
In a third aspect, the present application provides a cluster of computing devices, the cluster comprising at least one computing device, each of the at least one computing device comprising at least one processor and at least one memory, the at least one memory having computer-readable instructions stored therein; the at least one processor executes the computer-readable instructions to cause the cluster to implement the method of the first aspect or any one of the possible implementations of the first aspect.
In a fourth aspect, the present application provides a computer program product comprising a computer program stored in a computer readable storage medium and loaded by a processor to implement the method of the first aspect or any one of the possible implementations of the first aspect.
In a fifth aspect, the present application provides a computer readable storage medium storing a computer program for loading by a processor to perform the method of the first aspect or any one of the possible implementations of the first aspect.
In a sixth aspect, the present application provides a chip comprising a memory for storing computer instructions and a processor for calling and executing the computer instructions from the memory to perform the method of the first aspect or any one of the possible implementations of the first aspect.
Drawings
Fig. 1 is a schematic diagram of a network architecture according to an embodiment of the present application;
FIG. 2 is a flow chart of an alarm method for abnormal blood-edge relationship provided by an embodiment of the application;
FIG. 3 is a schematic illustration of an interface provided by an embodiment of the present application;
FIG. 4 is a schematic diagram of an abstract syntax tree provided by an embodiment of the present application;
FIG. 5 is a graph of data blood relationship provided by an embodiment of the present application;
FIG. 6 is a schematic illustration of another interface provided by an embodiment of the present application;
FIG. 7 is a schematic diagram of an alarm device with abnormal blood relationship according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a computing device according to an embodiment of the present application;
FIG. 9 is a schematic diagram of a cluster according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of another cluster according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, an embodiment of the present application provides a network architecture 100, where the network architecture 100 includes a blood-edge resolution platform 101, a terminal device 102, and a storage system 103, and the blood-edge resolution platform 101 may communicate with the terminal device 102 and the storage system 103.
The storage system 103 is configured to store at least one data table access program for accessing at least one data table for each of the at least one data table access program.
In some embodiments, the data table access program may implement one or more of the following access operations: reading data from the at least one data table, writing data to the at least one data table, or transferring data between the at least one data table, etc.
For example, one data table access program for accessing two data tables, table1 and table2, is shown below. The access operation realized by the data table access program is as follows: the first column data is read from the data table1, and the first column data is written into the data table2, that is, the data table access program is used for transferring the first column data in the data table1 to the data table2.
insert into table2
select
column1
from table1。
For at least one data table accessed by the data table access program, the at least one data table may include an abnormal data table, the abnormal data table being a data table in a blacklist, or the site where the abnormal data table is located being an overseas site, the blacklist including at least one abnormal data table.
The at least one data table further includes a normal data table, and if the access operation implemented by the data table accessing program includes an access operation to transfer data in the normal data table to the abnormal data table, the data table accessing program transfers data in the normal data table to the abnormal data table when the data table accessing program is operated.
This gives a great security problem to the data security, so that it is possible to detect whether the data table accessing program transfers the data in the normal data table to the abnormal data table before running the data table accessing program in order to improve the data security. And if the data table access program is detected to transfer the data in the normal data table to the abnormal data table, alarming. In this way, a technician may prevent the data table access program from running or modify the data table access program to improve the security of the data.
In order to detect the at least one data table access program, the user may configure the link address of the at least one data table access program on the terminal device 102. The terminal device 102 obtains the link address of the at least one data table access program and sends a detection request to the blood-address resolution platform 101, the detection request including the link address of the at least one data table access program.
A blood edge analysis platform 101, configured to receive a detection request, and obtain the at least one data table access program from the storage system 103 based on a link address included in the detection request; analyzing the at least one data table access program to obtain a data blood-edge relation diagram, wherein the data blood-edge relation diagram comprises a plurality of blood-edge relations, the blood-edge relations correspond to a target data table and at least one source data table, the plurality of data tables comprise the at least one source data table and the target data table, and the blood-edge relations are used for describing data transfer from the at least one source data table to the target data table; detecting an abnormal blood edge relation in the data blood edge relation graph based on an abnormal data table, wherein a target data table corresponding to the abnormal blood edge relation is the abnormal data table; and alarming the abnormal blood relationship.
The data table access program corresponding to the abnormal blood relationship is used for transferring the data in the normal data table to the abnormal data table. In this manner, a technician may prevent the data table access program from running or modify the data table access program.
Referring to fig. 2, an embodiment of the present application provides an alarm method 200 for abnormal blood-related relationships, where the method 200 is applied to the network architecture 100 shown in fig. 1. The method 200 includes the following flow.
Step 201: the terminal device obtains a link address of at least one data table access program to be detected, the at least one data table access program being used for accessing data in a plurality of data tables.
For each of the at least one data table access program, the data table access program may access at least one data table, the plurality of data tables including the at least one data table.
In some embodiments, the terminal device may further obtain a task access procedure for reading the at least one data table access procedure.
In step 201, the terminal device displays a software development kit (software development kit, SDK) package template interface to the user, the SDK package template interface including a link input interface to which the user can input a link address of at least one data table access program to be detected. And the terminal equipment acquires the link address of at least one data table access program to be detected from the link input interface included in the SDK package template interface.
Optionally, the SDK package template interface includes a program input interface to which a user can input a task access program. The terminal equipment acquires the task access program from the program input interface included in the SDK package template interface.
For example, referring to the SDK package template interface shown in fig. 3, the SDK package template interface includes a link input interface and a program input interface that are two input boxes. The link input interface is input box 1, the program input interface is input box 2, the user can input the link address 'http:// file# # #' of the at least one data table access program in the input box 1, the task access program is input in the input box 2, and a command is triggered to the terminal equipment by clicking a confirmation button. The terminal device receives the instruction, acquires the link address of the at least one data table access program from the input box 1 based on the instruction, and acquires the task access program from the input box 2.
In some embodiments, the link address is a uniform resource location system (uniform resource locator, URL) of a program file that includes the at least one data table access program. Optionally, the program file further includes program information of each of the at least one data table access program, the program information of the data table access program including one or more of the following: the data table accesses a program type, a program name, a program identification, or the like of the program.
In some embodiments, the link address is an address of a database that includes the at least one data table access program. Optionally, the database further includes program information for each of the at least one data table access program.
The program file and/or the database are located on a storage system.
Step 202: and the terminal equipment sends a detection request to the blood-edge analysis platform, wherein the detection request comprises the link address of at least one data table access program to be detected.
In some embodiments, the detection request further includes the task access program.
Step 203: and the blood margin analysis platform receives the detection request and acquires at least one data table access program to be detected based on the link address included in the detection request.
In some embodiments, the detection request further includes the task access program, and the blood-margin analysis platform obtains at least one data-table access program to be detected through the task access program based on the link address included in the detection request.
In the case that the link address is a URL of a program file in the storage system, the blood-address resolution platform obtains the program file from the storage system through the task access program based on the URL of the program file, and reads at least one data table access program to be detected from the program file. Optionally, the blood margin analysis platform may also read the program information of each data table access program to be detected from the program file.
In the case that the link address is an address of a database in the storage system, the blood-address resolution platform reads at least one data table access program to be detected from the database in the storage system through the task access program based on the address of the database. Optionally, the blood margin analysis platform may also read the program information of each data table access program to be detected from the database in the storage system.
Step 204: the blood edge analysis platform analyzes the at least one data table access program to obtain a data blood edge relation graph, wherein the data blood edge relation graph comprises a plurality of blood edge relations.
Wherein, for each of the plurality of relationships, the relationship corresponds to a target data table and at least one source data table, the plurality of data tables including the at least one source data table and the target data table. The blood-margin relationship is used to describe the transfer of data from the at least one source data table to the target data table, that is, the transfer of data from the at least one source data table to the target data table can be seen by the blood-margin relationship.
Next, an example is listed in which the data table access program is a structured query language (structured query language, SQL) program, which is:
insert into table2
select
column1
from table1。
The two data tables to be accessed by the data table access program comprise a source data table1 and a target data table2. Parsing the data table access program may result in a blood-address relationship describing the transfer of data from source data table1 to target data table2.
The data table access programs of different program types are analyzed by adopting different analysis modes. For example, see the correspondence between program types and parsing schemes shown in table1 below, which enumerates three program types and three parsing schemes, which are in one-to-one correspondence. For each data table access program in the at least one data table access program, using the program type of the data table access program, a corresponding analysis mode can be matched from the corresponding relation between the program type and the analysis mode shown in the following table1, and at least one blood-edge relation corresponding to the data table access program can be obtained by adopting the matched analysis mode.
TABLE1
The blood edge analysis platform comprises an SQL type program and an SQL analysis mode corresponding to the SQL type program, and a Spark jar type program and a Spark engine analysis mode corresponding to the Spark jar type program. The blood edge analysis platform may store the SQL type program and the SQL analysis method in the correspondence between the program types and the analysis methods shown in the above table1, and store the Spark jar type program and the Spark engine analysis method in the correspondence between the program types and the analysis methods shown in the above table 1. Here Spark is a big data processing engine and jar is a software compression package format.
For other types of programs, the user may customize the resolution program and then upload the customized resolution program to the blood-margin resolution platform. The blood margin analysis platform can receive a user-defined analysis program, define program types except an SQL type program and a Spark jar type program as other program types, and correspondingly store the other program types and the user-defined analysis program in the corresponding relation between the program types and the analysis modes shown in the table 1.
Therefore, the serial blood margin integration flow is convenient for users to realize the blood margin metadata acquisition and blood margin analysis logic, and the error rate is reduced. The automatic registration capability of the blood-edge integration program package is provided, and the blood-edge data processing is automatically carried out by matching with the corresponding program package during operation, so that the data blood-edge of the new large data technology stack is rapidly integrated, the expandability and the flexibility of the data blood-edge are improved, and the integration time and the cost are reduced.
For each of the at least one data table access program, the data table access program is referred to as a first data table access program for ease of explanation, which may access at least one data table.
The first data table access program may have grammatical and/or lexical issues and may not be successfully parsed when the first data table access program is parsed.
The first data table access program may not have grammatical problems or lexical problems, and the first data table access program can be successfully parsed when the first data table access program is parsed. The first data table accessing program may access a data table, and the first data table accessing program may be configured to read data from the data table and/or write data to the data table, and although the first data table accessing program can be successfully parsed, the blood-edge relationship cannot be obtained. Alternatively, the first data table access program may access a plurality of data tables, and the first data table access program may be configured to transfer data between the plurality of data tables, so that one or more blood-lineage relationships between the plurality of data tables can be obtained when the first data table access program is successfully parsed.
In step 204, one or more blood-lineage relationships corresponding to the first data table accessing program may be obtained by the following operations 2041-2043, where it is assumed herein that the program to be accessed by the first data table accessing program is a plurality of data tables.
2041: the blood margin analysis platform obtains the program type of the first data table access program.
In step 203, the blood-source resolving platform reads the program type of the first data table access program from the acquired program file, or reads the program type of the first data table access program from a database in the storage system. The program type of the first data table access program may be an SQL type program, or may be a Spark jar type program, or may be another type program. Other types of programs are programs other than the SQL type program and the Spark jar type program.
For example, assuming that the first data table access program is the above-listed SQL program, the blood-margin analysis platform acquires that the program type of the first data table access program is an SQL type program.
2042: the blood-margin analysis platform analyzes the first data table access program based on an analysis mode corresponding to the program type of the first data table access program to obtain data transfer conditions among a plurality of data tables, wherein the plurality of data tables are used for being accessed by the first data table access program.
In 2042, the blood-source analyzing platform matches the program type of the first data table access program with each program type included in the correspondence between the program type and the analyzing mode shown in the above table 1, and may obtain a matching result 1, a matching result 2 or a matching result 3 as follows.
And (3) matching the result 1, wherein the program type of the first data table access program is the same as the SQL type program, and the analysis mode obtained by the blood-source analysis platform is an SQL analysis mode. The blood margin analysis platform adopts an SQL analysis mode to analyze the process of the first data table access program as follows:
the blood margin analysis platform acquires grammar rules of the SQL type program, wherein the grammar rules define various grammars and/or various lexical contents of the SQL type program; an AST of the first data table access program is generated based on the grammar rule of the SQL type program, and the data transfer condition among the plurality of data tables is obtained based on the AST.
Optionally, the blood-source analyzing platform traverses the AST through a tree traversal access algorithm to obtain the data tables accessed by the first data table access program, and if the number of the data tables accessed by the first data table access program is multiple, the data transfer condition among the multiple data tables can be obtained by traversing the AST.
For example, the first data table access program is the SQL program, and the program type of the first data table access program obtained by the blood-edge analysis platform is the SQL type program. The blood-source analysis platform generates an AST of the SQL program based on the grammar rules of the SQL type program, and the AST is shown in fig. 4. Traversing AST as shown in FIG. 4 by a tree traversal access algorithm, and obtaining that the data table accessed by the SQL program comprises a data table1 and a data table2. Traversing an AST as shown in fig. 4 may result in a data transfer scenario between data table1 and data table2, the data transfer scenario being transferring data from data table1 to data table2.
If the first data table access program has grammar errors and/or lexical errors, the blood-margin analysis platform cannot successfully analyze the first data table access program by adopting an SQL analysis mode.
If the first data table access program has no grammar error or lexical error, the blood-source analysis platform can successfully analyze the first data table access program by adopting an SQL analysis mode, and obtain a data table to be accessed by the first data table access program. When the data table to be accessed by the first data table access program is a data table, the blood-edge analysis platform cannot analyze the blood-edge relationship. When the data table to be accessed by the first data table access program is a plurality of data tables, the blood-margin analysis platform cannot analyze and obtain the blood-margin relation among the plurality of data tables.
And 2, matching the program type of the first data table access program with the Spark jar type program, wherein the analysis mode obtained by the blood edge analysis platform is a Spark engine analysis mode. The process of analyzing the first data table access program by the blood margin analysis platform in a Spark engine analysis mode comprises the following steps:
the blood margin analysis platform uses a Spark engine to analyze the first data table access program to obtain an execution plan tree, and obtains the data transfer condition among the plurality of data tables based on the execution plan tree.
In some embodiments, the blood-margin resolution platform inputs the first data-table access program to the Spark engine and registers in the Spark engine a snoop execution completion event. The Spark engine receives the first data table access program, analyzes the first data table access program, generates an execution plan tree of the first data table access program, and generates an execution completion event when the execution plan tree of the first data table access program is generated. And after the blood margin analysis platform monitors the execution completion event, acquiring an execution plan tree of the first data table access program from the Spark engine. The blood margin analysis platform traverses the execution plan tree through a tree traversal access algorithm to obtain the data tables accessed by the first data table access program, and if the number of the data tables accessed by the first data table access program is multiple, the execution plan tree is traversed to obtain the data transfer condition among the multiple data tables.
If the first data table access program has grammar errors and/or lexical errors, the blood-margin analysis platform cannot successfully analyze the first data table access program by adopting a Spark engine analysis mode.
If the first data table access program has no grammar error or lexical error, the blood-margin analysis platform can successfully analyze the first data table access program by adopting a Spark engine analysis mode, and obtain a data table to be accessed by the first data table access program. When the data table to be accessed by the first data table access program is a data table, the blood-edge analysis platform cannot analyze the blood-edge relationship. When the data table to be accessed by the first data table access program is a plurality of data tables, the blood-margin analysis platform cannot analyze and obtain the blood-margin relation among the plurality of data tables.
And 3, matching the program type of the first data table access program to be different from the SQL type program and the Spark jar type program, namely, when the program type of the first data table access program is a program other than the SQL type program and the Spark jar type program. The analysis mode obtained by the blood margin analysis platform is a user-defined analysis program, and the first data table access program is analyzed through the user-defined analysis program to obtain the data transfer condition among the plurality of data tables.
If the first data table access program has grammar errors and/or lexical errors, the blood-margin analysis platform cannot successfully analyze the first data table access program by adopting a self-defined analysis program.
If the first data table access program has no grammar error or lexical error, the blood-margin analysis platform can successfully analyze the first data table access program by adopting a self-defined analysis program, and obtain a data table to be accessed by the first data table access program. When the data table to be accessed by the first data table access program is a data table, the blood-edge analysis platform cannot analyze the blood-edge relationship. When the data table to be accessed by the first data table access program is a plurality of data tables, the blood-margin analysis platform cannot analyze and obtain the blood-margin relation among the plurality of data tables.
2043: and adding at least one blood-edge relation corresponding to the first data table access program into the data blood-edge relation graph by the blood-edge analysis platform, wherein the at least one blood-edge relation is used for describing the data transfer condition among the plurality of data tables.
In some embodiments, the lineage resolution platform can further mark program information of the first data table access program on the at least one lineage relationship.
For example, referring to FIG. 5, the blood-edge analysis platform adds a node corresponding to data table1 and a node corresponding to data table2 to the blood-edge relationship graph, between the node corresponding to data table1 and the node corresponding to data table2, adding an edge pointing from the node corresponding to the data table1 to the node corresponding to the data table2, wherein the edge is the blood edge relation between the data table1 and the data table2, the source data table of the blood-edge relationship is a data table1, and the target data table of the blood-edge relationship is a data table2. The blood-margin analysis platform may also mark program information of the first data table access program on the blood-margin relation.
And adding the blood-edge relation corresponding to each other data table access program in the data blood-edge relation diagram according to the flow of 2041-2043 for each other data table access program in the at least one data table access program.
In some embodiments, after resolving the at least one data table access procedure, the blood-edge resolution platform further obtains a resolution success rate and/or a resolution coverage rate, the resolution success rate being a ratio between a number of successfully resolved data table access procedures and a total number of the at least one data table access procedures, the resolution coverage rate being a ratio between a number of data table access procedures capable of resolving the blood-edge relationship and the total number of the at least one data table access procedures.
In some embodiments, by parsing the first data table access program, not only can the source data table transfer data to the target data table be parsed, but also the data of certain fields in the source data table can be parsed to be transferred to certain fields of the target data table. For ease of illustration, certain fields in the source data table are referred to as source fields, certain fields in the target data table are referred to as target fields, and the at least one blood-edge relationship corresponding to the first data table access program includes a blood-edge relationship from the source field in the source data table to the target field in the target data table. Where a field refers to a column of data.
In some embodiments, upon resolving the first data table access program, the blood-margin resolving platform further obtains resource specification information consumed by resolving the first data table access program, the resource specification information including one or more of: CPU utilization, memory occupancy, or time-consuming, etc.
Step 205: the blood margin analysis platform identifies an abnormal data table from a plurality of data tables accessed by the at least one data-with-access program.
In step 205, the following two ways are listed to identify the abnormal data table, however, other ways are possible to identify the abnormal data table, and the following details are not listed here.
In one mode, for each of the plurality of data tables, the blood-margin analysis platform obtains a site where the data table is located. Determining whether the site where the data table is located is an overseas site, and if the site where the data table is located is an overseas site, determining that the data table is an abnormal data table; if the site at which the data table is located is an intra-site, i.e., the site at which the data table is located is not an overseas site, it is determined that the data table is not an abnormal data table but a normal data table.
Alternatively, the blood-edge resolution platform may extract the address of the data table from the data-table access program and determine the site where the data table is located based on the address of the data table.
For each of the plurality of data tables, the blood-margin analysis platform checks whether the data table is a data table in a blacklist, the blacklist including at least one abnormal data table, determines that the data table is an abnormal data table if the data table is checked to be a data table in the blacklist, and determines that the data table is not an abnormal data table but a normal data table if the data table is checked to be a data table not in the blacklist.
Step 206: the blood margin analysis platform detects abnormal blood margin relations in the data blood margin relation graph based on the abnormal data table, wherein the target data table corresponding to the abnormal blood margin relations is the abnormal data table.
For each identified abnormal data table, the blood-edge analysis platform can search the blood-edge relation taking the abnormal data table as a target data table from the data blood-edge relation graph, and determine the searched blood-edge relation as the abnormal blood-edge relation. Alternatively, the source data table of the abnormal blood-relationship may be a normal data table or may also be an abnormal data table.
For example, assuming that table2 is an abnormal table, the blood-edge analysis platform may find a blood-edge relationship with table2 as a target table from the data blood-edge relationship graph shown in fig. 5, where the blood-edge relationship is a blood-edge relationship between table1 and table2, and determine the found blood-edge relationship as an abnormal blood-edge relationship. The source data table1 of the abnormal blood-edge relationship is a normal data table.
In some embodiments, for any one of the data tables in the data lineage diagram, the data table is taken as at least one lineage of a source data table, and if data of some fields in the source data table are sensitive data, such as a user name, a user phone number, and/or a user identification number, it may be derived that data of the fields in the source data table is transferred to a target data table, and it may also be derived that an lineage directed to the target data table by the some fields of the source data table is also an abnormal lineage.
Step 207: and the blood margin analysis platform alarms the abnormal blood margin relation.
In step 207, the blood edge resolution platform may display the abnormal blood edge relationship, highlighting the abnormal blood edge relationship in the data blood edge relationship graph; or, the abnormal blood edge relation is bolded and displayed in the data blood edge relation graph; or adding an abnormal mark on the abnormal blood edge relation in the data blood edge relation graph to alarm the abnormal blood edge relation.
In some embodiments, the blood-edge analysis platform may send the data blood-edge relation graph highlighting the abnormal blood-edge relation to the terminal device, or send the data blood-edge relation graph thickening the abnormal blood-edge relation to the terminal device, or send the data blood-edge relation graph adding an abnormal mark on the abnormal blood-edge relation to the terminal device. The terminal equipment receives the data blood-edge relation diagram and displays a first interface, wherein the first interface comprises the data blood-edge relation diagram.
Program information of the data table access program may be marked on the abnormal blood-edge relationship in the data blood-edge relationship graph, and the user determines the data table access program corresponding to the abnormal blood-edge relationship based on the marked program information on the abnormal blood-edge relationship. The user may block the operation of the data table access program or modify the data table access program to avoid the data table access program transferring data in the normal data table to the abnormal data table when the data table access program is operated.
For example, referring to the first interface shown in fig. 6, the abnormal blood-edge relationship is a blood-edge relationship between data table1 and data table2, and the blood-edge relationship between data table1 and data table2 is shown bolded in the data blood-edge relationship shown in fig. 5.
In some embodiments, the blood-lineage resolution platform may also display resolution success rate and/or resolution coverage. Optionally, when implementing, the blood-source analysis platform sends analysis success rate and/or analysis coverage rate to the terminal device. And the terminal equipment receives the resolution success rate and/or the resolution coverage rate and displays the resolution success rate and/or the resolution coverage rate in the first interface.
In some embodiments, the blood-lineage resolution platform may also display resource specification information for each data table access program. Optionally, when implementing, the blood-margin analysis platform sends the resource specification information of each data table access program to the terminal device. The terminal equipment receives the resource specification information of each data table access program and displays the resource specification information of each data table access program in the first interface.
For the user-defined parsing program, the user may use the blood-margin parsing platform to debug the parsing program.
In implementation, a user may use a data table access program with a known blood-edge relationship, and for convenience of explanation, the data table access program is referred to as a second data table access program, and the program type of the second data table access program is a program other than the SQL type program and the Spark jar program type, and the second data table access program is sent to the blood-edge analysis platform. The blood margin analysis platform can be matched with a user-defined analysis program, and the blood margin relation corresponding to the second data table access program is obtained through the analysis program. The user may compare the known blood-lineage relationship with a difference between blood-lineage relationships corresponding to a second data table access procedure output by the blood-lineage resolution platform, debug and modify the resolution procedure based on the compared differences.
And then updating the analysis program on the blood margin analysis platform into a modified analysis program, and obtaining the blood margin relation corresponding to the second data table access program by the blood margin analysis platform through the modified analysis program. The user may continue to compare the known blood-lineage relationship with the blood-lineage relationship corresponding to the second data table access program output by the blood-lineage resolution platform, debug and modify the resolution program based on the compared differences. And repeating the process until the known blood edge relationship has no difference with the blood edge relationship corresponding to the second data table access program output by the blood edge analysis platform. Therefore, the testing capability and the version management capability of the blood margin analysis platform are provided, the development efficiency is improved, and the quality is ensured to be controllable.
In the embodiment of the application, before the at least one data table access program is run, a user can send a detection request to the blood-edge analysis platform through the terminal equipment, wherein the detection request comprises a link address of the at least one data table access program to be detected. The blood margin analysis platform receives the detection request, acquires the at least one data table access program, analyzes the at least one data table access program to obtain a data blood margin relation graph, detects an abnormal blood margin relation in the data blood margin relation based on an abnormal data table, and alarms the abnormal blood margin relation. Therefore, the user can obtain the data table access program with data security based on the abnormal blood relationship, and can modify the data table access program or prevent the data table access program from running based on the abnormal blood relationship, so that the data table access program is prevented from transferring data in a normal data table to an abnormal data table when running, and the data security is improved. And a data security check mechanism of a full link is established, so that the circulation of sensitive data is monitored, recorded, analyzed and retrospectively, and the data compliance and the security circulation are ensured.
Referring to fig. 7, an embodiment of the present application provides an alarm device 700 for abnormal blood-edge relationships, where the device 700 is applied to the blood-edge analysis platform 101 of the network architecture 100 shown in fig. 1, or to the blood-edge analysis platform of the method 200 shown in fig. 2. The apparatus 700 includes:
A receiving unit 701, configured to receive a detection request, where the detection request includes a link address of at least one data table access program to be detected, where the at least one data table access program is configured to access data in a plurality of data tables, and the plurality of data tables include an abnormal data table;
a processing unit 702, configured to obtain the at least one data table access program based on the link address;
the processing unit 702 is further configured to parse the at least one data table access procedure to obtain a data blood-edge relationship graph, where the data blood-edge relationship graph includes a plurality of blood-edge relationships, the blood-edge relationships correspond to a target data table and at least one source data table, the plurality of data tables include the at least one source data table and the target data table, and the blood-edge relationships are used to describe transferring data from the at least one source data table to the target data table;
the processing unit 702 is further configured to detect an abnormal blood edge relationship in the data blood edge relationship graph based on the abnormal data table, where the target data table corresponding to the abnormal blood edge relationship is the abnormal data table;
the processing unit 702 is further configured to alarm the abnormal blood edge relationship.
Optionally, the detailed implementation process of the receiving unit 701 for receiving the detection request refers to the relevant content of step 203 of the method 200 shown in fig. 2, which is not described in detail here.
Optionally, the detailed implementation procedure of the at least one data table accessing program obtained by the processing unit 702 is referred to in the relevant content of step 203 of the method 200 shown in fig. 2, and will not be described in detail here.
Optionally, the detailed implementation process of the processing unit 702 to obtain the data blood relationship diagram is referred to in the relevant content of step 203 of the method 200 shown in fig. 2, and will not be described in detail here.
Optionally, the detailed implementation process of the processing unit 702 for detecting abnormal blood edge relations in the data blood edge relation diagram is referred to in the relevant content of step 206 of the method 200 shown in fig. 2, and will not be described in detail here.
Optionally, the detailed implementation of the process 702 for alerting to abnormal blood-edge relationships is described with reference to step 207 of the method 200 of fig. 2, and will not be described in detail herein.
Optionally, the processing unit 702 is configured to:
for each data table access program of the at least one data table access program, acquiring a program type of the data table access program, wherein the data table access program is used for accessing partial data tables in the plurality of data tables;
analyzing the data table access program based on the analysis mode corresponding to the program type to obtain the data transfer condition among the partial data tables;
And adding a blood edge relation in the data blood edge relation graph, wherein the blood edge relation is used for describing the data transfer condition among the partial data tables.
Optionally, the detailed implementation of the process of the processing unit 702 to obtain the program type of the data table access program is referred to in the relevant content of step 203 of the method 200 shown in fig. 2, and will not be described in detail here.
Optionally, the processing unit 702 parses the detailed implementation procedure of the data table accessing program based on the parsing manner corresponding to the program type, see the relevant content of step 2042 of the method 200 shown in fig. 2, which is not described in detail herein.
Optionally, the detailed implementation process of adding the blood-edge relationship to the data blood-edge relationship diagram by the processing unit 702 is referred to in the relevant content of step 2043 of the method 200 shown in fig. 2, and will not be described in detail herein.
Optionally, the processing unit 702 is configured to:
when the program type is a Structured Query Language (SQL) type program, generating an Abstract Syntax Tree (AST) of the data table access program based on the syntax rules of the SQL type program, and obtaining the data transfer condition among the partial data tables based on the AST;
when the program type is Spark jar type program, analyzing the data table access program by using a Spark engine to obtain an execution plan tree, and obtaining the data transfer condition among the partial data tables based on the execution plan tree;
When the program type is a program except the SQL type program and the Spark jar type program, the data table access program is analyzed through a self-defined analysis program, and the data transfer condition among the partial data tables is obtained.
Optionally, the processing unit 702 generates an abstract syntax tree AST of the data table accessing program, and a detailed implementation procedure of the data transfer situation between the partial data tables based on the AST is referred to in step 2042 of the method 200 shown in fig. 2, which is not described in detail here.
Optionally, the processing unit 702 uses the Spark engine to parse the data table accessing program to obtain an execution plan tree, and the detailed implementation process of obtaining the data transfer situation between the partial data tables based on the execution plan tree refers to the relevant content of step 2042 of the method 200 shown in fig. 2, which is not described in detail herein.
Optionally, the processing unit 702 parses the data table accessing program through a custom parsing program, so as to obtain details about the data transfer situation between the partial data tables, which are referred to in step 2043 of the method 200 shown in fig. 2, and will not be described in detail herein.
Optionally, the processing unit 702 is configured to:
highlighting abnormal blood edge relations in the data blood edge relation graph; or,
Thickening and displaying abnormal blood edge relation in the data blood edge relation graph; or,
an anomaly flag is added on the anomaly blood relationship in the data blood relationship graph.
Optionally, the processing unit 702 is further configured to:
for each of the plurality of data tables, when the site where the data table is located is an overseas site or when the data table is a data table in a blacklist, determining that the data table is an abnormal data table, the blacklist including at least one abnormal data table.
Optionally, the detailed implementation of the processing unit 702 to determine the data table as an exception data table is described with reference to step 205 of the method 200 shown in fig. 2, and will not be described in detail herein.
Optionally, the processing unit 702 is further configured to:
and displaying the analysis success rate and/or the analysis coverage rate, wherein the analysis success rate is the ratio between the number of the data table access programs which are successfully analyzed and the total number of the at least one data table access program, and the analysis coverage rate is the ratio between the number of the data table access programs which can analyze the bleeding edge relationship and the total number of the at least one data table access program.
In the embodiment of the application, before the at least one data table access program is operated, the processing unit analyzes the at least one data table access program to obtain a data blood edge relationship graph, detects an abnormal blood edge relationship in the data blood edge relationship based on the abnormal data table, and alarms the abnormal blood edge relationship. Therefore, the user can obtain the data table access program with data security based on the abnormal blood relationship, and can modify the data table access program or prevent the data table access program from running based on the abnormal blood relationship, so that the data table access program is prevented from transferring data in a normal data table to an abnormal data table when running, and the data security is improved.
Referring to FIG. 8, an embodiment of the application provides a computing device 800. For example, the computing device 800 may be a device included in a blood-edge resolution platform in the network architecture shown in FIG. 1, or a device in a blood-edge resolution platform of the method 200 shown in FIG. 2.
As shown in fig. 8, a computing device 800 includes: bus 802, processor 804, memory 806, and communication interface 808. Communication between processor 804, memory 806, and communication interface 808 is via bus 802. Computing device 800 may be a server or a terminal device. It should be understood that the present application is not limited to the number of processors, memories in computing device 800.
Bus 802 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one line is shown in fig. 8, but not only one bus or one type of bus. Bus 802 may include a path to transfer information between various components of computing device 800 (e.g., memory 806, processor 804, communication interface 808).
The processor 804 may include any one or more of a central processing unit (central processing unit, CPU), a graphics processor (graphics processing unit, GPU), a Microprocessor (MP), or a digital signal processor (digital signal processor, DSP).
The memory 806 may include volatile memory (RAM), such as random access memory (random access memory). The processor 804 may also include non-volatile memory (ROM), such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state disk (solid state drive, SSD).
Referring to fig. 8, a memory 806 stores executable program codes, and a processor 804 executes the executable program codes to implement the functions of the receiving unit 701 and the processing unit 702 in the apparatus 700 shown in fig. 7, respectively, thereby implementing the method provided in any of the above embodiments. That is, the memory 806 has instructions stored thereon for performing the methods provided by any of the embodiments described above. Or,
the communication interface 808 enables communication between the computing device 800 and other devices or communication networks using a transceiver module such as, but not limited to, a network interface card, transceiver, or the like.
The embodiment of the application also provides a cluster. The cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop, notebook, or smart phone.
As shown in fig. 9, the cluster includes at least one computing device 800. The same instructions for performing the methods provided by any of the embodiments described above may be stored in memory 806 in one or more computing devices 800 in the cluster.
In some possible implementations, the memory 806 of one or more computing devices 800 in the cluster may also have stored therein respective portions of instructions for performing the above-described alarm method of abnormal blood-edge relationships. In other words, a combination of one or more computing devices 800 may collectively execute instructions for performing the alert method of any of the embodiments described above to provide an abnormal blood-edge relationship.
In some possible implementations, one or more computing devices in the cluster may be connected through a network. Wherein the network may be a wide area network or a local area network, etc. Fig. 10 shows one possible implementation. As shown in fig. 10, two computing devices 800A and 800B are connected by a network. Specifically, the connection to the network is made through a communication interface in each computing device.
In this type of possible implementation, instructions to perform the functions of processing unit 702 in the embodiment shown in FIG. 7 are stored in memory 806 in computing device 800A. Meanwhile, an instruction for performing the function of the receiving unit 701 in the embodiment shown in fig. 7 is stored in the memory 806 in the computing device 800B.
It should be appreciated that the functionality of computing device 800A shown in fig. 10 may also be performed by multiple computing devices 800. Likewise, the functionality of computing device 800B may also be performed by multiple computing devices 800.
The embodiment of the application also provides another cluster. The connection between the computing devices in the cluster may be similar to the connection of the cluster described with reference to fig. 10. In contrast, the same instructions for performing the methods provided by any of the embodiments described above may be stored in memory 806 in one or more computing devices 800 in the cluster.
In some possible implementations, part of the instructions for performing the methods provided in any of the embodiments described above may also be stored in the memory 806 of one or more computing devices 800 in the cluster, respectively. In other words, a combination of one or more computing devices 800 may collectively execute instructions for performing the methods provided by any of the embodiments described above.
Embodiments of the present application also provide a computer program product comprising instructions. The computer program product may be software or a program product containing instructions capable of running on a computing device or stored in any useful medium. The computer program product, when run on at least one computing device, causes the at least one computing device to perform the method provided by any of the embodiments described above.
The embodiment of the application also provides a computer readable storage medium. The computer readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc. The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided by any of the embodiments described above.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the present application is not intended to be limiting, but rather, any modifications, equivalents, improvements, etc. that fall within the principles of the present application are intended to be included within the scope of the present application.
Claims (15)
1. An alarm method for abnormal blood relationship, the method comprising:
receiving a detection request, wherein the detection request comprises a link address of at least one data table access program to be detected, and the at least one data table access program is used for accessing data in a plurality of data tables, and the plurality of data tables comprise abnormal data tables;
acquiring the at least one data table access program based on the link address;
analyzing the at least one data table access program to obtain a data blood-edge relation diagram, wherein the data blood-edge relation diagram comprises a plurality of blood-edge relations, the blood-edge relations correspond to a target data table and at least one source data table, the plurality of data tables comprise the at least one source data table and the target data table, and the blood-edge relations are used for describing data transfer from the at least one source data table to the target data table;
detecting an abnormal blood edge relation in the data blood edge relation graph based on an abnormal data table, wherein a target data table corresponding to the abnormal blood edge relation is the abnormal data table;
And alarming the abnormal blood relationship.
2. The method of claim 1, wherein said parsing the at least one data table access program to obtain a data blood relationship graph comprises:
for each data table access program in the at least one data table access program, acquiring a program type of the data table access program, wherein the data table access program is used for accessing part of the data tables in the plurality of data tables;
analyzing the data table access program based on the analysis mode corresponding to the program type to obtain the data transfer condition among the partial data tables;
and adding a blood edge relation in the data blood edge relation graph, wherein the blood edge relation is used for describing the data transfer condition among the partial data tables.
3. The method of claim 2, wherein parsing the data table access program based on the parsing scheme corresponding to the program type, comprises:
when the program type is a Structured Query Language (SQL) type program, generating an Abstract Syntax Tree (AST) of the data table access program based on the syntax rule of the SQL type program, and obtaining the data transfer condition among the partial data tables based on the AST;
When the program type is Spark jar type program, analyzing the data table access program by using a Spark engine to obtain an execution plan tree, and obtaining the data transfer condition among the partial data tables based on the execution plan tree;
when the program type is a program except the SQL type program and the Spark jar type program, analyzing the data table access program through a self-defined analysis program to obtain the data transfer condition among the partial data tables.
4. A method according to any one of claims 1-3, wherein said alerting the abnormal blood-edge relationship comprises:
highlighting the abnormal blood edge relationship in the data blood edge relationship graph; or,
the abnormal blood edge relationship is bolded and displayed in the data blood edge relationship graph; or,
an anomaly flag is added to the anomalous blood relationship in the data blood relationship graph.
5. The method of any one of claims 1-4, wherein the method further comprises:
for each data table of the plurality of data tables, when the site where the data table is located is an overseas site or when the data table is a data table in a blacklist, determining that the data table is an abnormal data table, wherein the blacklist comprises at least one abnormal data table.
6. The method of any one of claims 1-5, wherein the method further comprises:
and displaying an analysis success rate and/or analysis coverage rate, wherein the analysis success rate is a ratio between the number of successfully analyzed data table access programs and the total number of the at least one data table access program, and the analysis coverage rate is a ratio between the number of data table access programs capable of analyzing bleeding edge relations and the total number of the at least one data table access program.
7. An alarm device for abnormal blood-stasis, the device comprising:
a receiving unit configured to receive a detection request, where the detection request includes a link address of at least one data table access program to be detected, where the at least one data table access program is configured to access data in a plurality of data tables, and the plurality of data tables include an abnormal data table;
a processing unit, configured to obtain the at least one data table access program based on the link address;
the processing unit is further configured to parse the at least one data table access program to obtain a data blood-edge relationship graph, where the data blood-edge relationship graph includes a plurality of blood-edge relationships, the blood-edge relationships correspond to a target data table and at least one source data table, the plurality of data tables include the at least one source data table and the target data table, and the blood-edge relationships are used to describe transferring data from the at least one source data table to the target data table;
The processing unit is further configured to detect an abnormal blood edge relationship in the data blood edge relationship graph based on an abnormal data table, where a target data table corresponding to the abnormal blood edge relationship is the abnormal data table;
the processing unit is also used for alarming the abnormal blood edge relation.
8. The apparatus of claim 7, wherein the processing unit is to:
for each data table access program in the at least one data table access program, acquiring a program type of the data table access program, wherein the data table access program is used for accessing part of the data tables in the plurality of data tables;
analyzing the data table access program based on the analysis mode corresponding to the program type to obtain the data transfer condition among the partial data tables;
and adding a blood edge relation in the data blood edge relation graph, wherein the blood edge relation is used for describing the data transfer condition among the partial data tables.
9. The apparatus of claim 8, wherein the processing unit is to:
when the program type is a Structured Query Language (SQL) type program, generating an Abstract Syntax Tree (AST) of the data table access program based on the syntax rule of the SQL type program, and obtaining the data transfer condition among the partial data tables based on the AST;
When the program type is Spark jar type program, analyzing the data table access program by using a Spark engine to obtain an execution plan tree, and obtaining the data transfer condition among the partial data tables based on the execution plan tree;
when the program type is a program except the SQL type program and the Spark jar type program, analyzing the data table access program through a self-defined analysis program to obtain the data transfer condition among the partial data tables.
10. The apparatus according to any of claims 7-9, wherein the processing unit is configured to:
highlighting the abnormal blood edge relationship in the data blood edge relationship graph; or,
the abnormal blood edge relationship is bolded and displayed in the data blood edge relationship graph; or,
an anomaly flag is added to the anomalous blood relationship in the data blood relationship graph.
11. The apparatus of any of claims 7-10, wherein the processing unit is further configured to:
for each data table of the plurality of data tables, when the site where the data table is located is an overseas site or when the data table is a data table in a blacklist, determining that the data table is an abnormal data table, wherein the blacklist comprises at least one abnormal data table.
12. The apparatus of any of claims 7-11, wherein the processing unit is further configured to:
and displaying an analysis success rate and/or analysis coverage rate, wherein the analysis success rate is a ratio between the number of successfully analyzed data table access programs and the total number of the at least one data table access program, and the analysis coverage rate is a ratio between the number of data table access programs capable of analyzing bleeding edge relations and the total number of the at least one data table access program.
13. A cluster of computing devices, the cluster comprising at least one computing device, each of the at least one computing device comprising at least one processor and at least one memory having computer-readable instructions stored therein; the at least one processor executes the computer-readable instructions to cause the cluster to perform the method of any of claims 1-6.
14. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of claims 1-6.
15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310956885.1A CN117112542A (en) | 2023-07-31 | 2023-07-31 | Alarm method, device and storage medium for abnormal blood edge relationship |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310956885.1A CN117112542A (en) | 2023-07-31 | 2023-07-31 | Alarm method, device and storage medium for abnormal blood edge relationship |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117112542A true CN117112542A (en) | 2023-11-24 |
Family
ID=88811847
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310956885.1A Pending CN117112542A (en) | 2023-07-31 | 2023-07-31 | Alarm method, device and storage medium for abnormal blood edge relationship |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117112542A (en) |
-
2023
- 2023-07-31 CN CN202310956885.1A patent/CN117112542A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| NL2029881B1 (en) | Methods and apparatus for automatic detection of software bugs | |
| US7103885B2 (en) | Comment driven processing | |
| US8429632B1 (en) | Method and system for debugging merged functions within a program | |
| US11334474B2 (en) | Fast change impact analysis tool for large-scale software systems | |
| CN111124479B (en) | Method and system for analyzing configuration file and electronic equipment | |
| US9898386B2 (en) | Detecting byte ordering type errors in software code | |
| WO2019227641A1 (en) | Application testing method, apparatus, terminal device and medium | |
| EP3676702A1 (en) | Method and apparatus for data compilation, electronic device and computer readable storage medium | |
| CN110727581B (en) | Crash positioning method and electronic equipment | |
| CN110688096A (en) | Method, device, medium and electronic equipment for constructing application program containing plug-in | |
| CN110928720A (en) | Core dump file generation method and device based on Linux system | |
| CN114816993A (en) | Full link interface test method, system, medium and electronic equipment | |
| CN110990346A (en) | File data processing method, device, equipment and storage medium based on block chain | |
| CN108427580B (en) | Configuration pair naming repetition detection method, storage medium and intelligent device | |
| CN114528204A (en) | Method for processing code, method for processing exception and respective device | |
| US9064042B2 (en) | Instrumenting computer program code by merging template and target code methods | |
| CN111046393B (en) | Vulnerability information uploading method and device, terminal equipment and storage medium | |
| CN110727565B (en) | Network equipment platform information collection method and system | |
| CN117112542A (en) | Alarm method, device and storage medium for abnormal blood edge relationship | |
| CN113641702B (en) | Method and device for interactive processing with database client after statement audit | |
| CN110244954B (en) | Compiling method and equipment for application program | |
| CN114416057A (en) | Project code packaging method and device, terminal equipment and storage medium | |
| CN114386045A (en) | Web application program vulnerability detection method and device and storage medium | |
| CN114936368A (en) | Java memory Trojan detection method, terminal device and storage medium | |
| CN112433943A (en) | Method, device, equipment and medium for detecting environment variable based on abstract syntax tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |