CN115129753A - Data blood relationship analysis method and device, electronic equipment and storage medium - Google Patents
Data blood relationship analysis method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115129753A CN115129753A CN202210760713.2A CN202210760713A CN115129753A CN 115129753 A CN115129753 A CN 115129753A CN 202210760713 A CN202210760713 A CN 202210760713A CN 115129753 A CN115129753 A CN 115129753A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- relationship
- fields
- service data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000008280 blood Substances 0.000 title claims abstract description 79
- 210000004369 blood Anatomy 0.000 title claims abstract description 79
- 238000004458 analytical method Methods 0.000 title claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 46
- 230000008569 process Effects 0.000 claims abstract description 29
- 238000004590 computer program Methods 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 13
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 238000003066 decision tree Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000007405 data analysis Methods 0.000 abstract description 2
- 230000002068 genetic effect Effects 0.000 abstract description 2
- 238000012545 processing Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 8
- 238000007726 management method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000008033 biological extinction Effects 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data analysis, and discloses a data blood relationship analysis method, which comprises the following steps: acquiring business data corresponding to a user input data query request from a business system, analyzing field genetic relationship among fields in the business data, storing the business data into a message queue, loading the business data into a local disk, triggering an asynchronous thread on the local disk, running the business data through the asynchronous thread, and analyzing the field inheritance relationship among the fields of the business data in the running process of the asynchronous thread; constructing a field association file of the business data according to the field blood relationship and the field inheritance relationship, and marking identification attributes among the fields in the field association file; and storing the data in the service data into a graph database according to the field association file and the identification attribute so as to obtain the data blood relationship among the data in the service data. In addition, the invention also relates to a block chain technology, and the field association file can store a block chain. The invention can improve the efficiency of data blood relationship analysis.
Description
Technical Field
The present invention relates to the field of data analysis, and in particular, to a method and an apparatus for analyzing data blood relationship, an electronic device, and a computer-readable storage medium.
Background
Data management has entered an intelligent era, accurate tracking and tracing of data can effectively improve positioning of data problems in a big data system and improve data quality, a traditional data blood relationship analysis method is generally that developers manually analyze SQL according to scripts of tasks in a development platform to find out blood relationship of the data, but through the method, the developers need to analyze the scripts of the tasks in the development platform one by one, so that a large amount of time and energy of the developers are consumed, and the data blood relationship analysis efficiency is affected.
Disclosure of Invention
The invention provides a data blood relationship analysis method, a data blood relationship analysis device, electronic equipment and a computer readable storage medium, and mainly aims to improve the efficiency of data blood relationship analysis.
In order to achieve the above object, the present invention provides a data blood relationship analysis method, including:
receiving a data query request input by a user, acquiring service data from a service system according to the data query request, analyzing field blood relationship among fields in the service data by using a computing engine, and storing the service data into a preset message queue;
loading the service data in the message queue to a local disk, and triggering an asynchronous thread on the local disk so as to run the service data through the asynchronous thread and analyze the field inheritance relationship between fields of the service data in the running process of the asynchronous thread;
constructing a field association file of the business data according to the field blood relationship and the field inheritance relationship, and marking identification attributes among fields in the field association file;
and storing the data in the service data into a graph database according to the field association file and the identification attribute so as to obtain the data blood relationship among the data in the service data.
Optionally, the obtaining service data from the service system according to the data query request includes:
identifying a query object and a query field of the data query request;
converting the data query request into a query statement according to the query object and the query field;
and inquiring service data from a background database of the service system according to the inquiry statement.
Optionally, the parsing, by the computing engine, field blood relationship between fields in the business data includes:
capturing the corresponding relation between the input field and the output field of the business data in the query process by utilizing a hook function in the computing engine;
according to the corresponding relation, a decision tree of the input field and the output field is constructed by utilizing a decision algorithm in the computing engine;
calculating the node paths of any two fields in the decision tree by using a depth traversal algorithm;
and determining the association relation of the corresponding fields according to the node path, and taking the association relation as the field blood relationship among the fields in the business data.
Optionally, the loading the service data in the message queue to a local disk includes:
serializing the service data in the message queue to obtain serialized data;
and transmitting the serialized data to a local disk in a first-in first-out order.
Optionally, the analyzing the field inheritance relationship between the fields of the service data in the running process of the asynchronous thread includes:
deserializing the service data to obtain deserialized data, and identifying a call relation between fields in the deserialized data;
according to the calling relation, identifying a parent field and a child field of the deserialized data in the running process of the asynchronous thread;
and generating field inheritance relation between fields in the service data according to the parent field and the subclass field.
Optionally, the constructing a field association file of the service data according to the field consanguinity relationship and the field inheritance relationship includes:
taking the field which has the field blood relationship and the field inheritance relationship in the service data as a clustering center point;
and connecting the rest fields in the service data with the clustering central point by adopting a conceptual relationship model to form a field association file.
Optionally, the marking an identification attribute between fields in the field association file includes:
inquiring field ID and field number between fields in the field association file;
and splicing the field ID and the field number to obtain the identification attribute between the fields in the field association file.
In order to solve the above problem, the present invention also provides a data blood relationship analysis device, including:
the system comprises a blood relationship analysis module, a message queue management module and a message queue management module, wherein the blood relationship analysis module is used for receiving a data query request input by a user, acquiring business data from a business system according to the data query request, analyzing field blood relationship among fields in the business data by using a computing engine, and storing the business data into a preset message queue;
the inheritance relationship analysis module is used for loading the service data in the message queue into a local disk, triggering an asynchronous thread on the local disk, running the service data through the asynchronous thread, and analyzing the field inheritance relationship among fields of the service data in the running process of the asynchronous thread;
the field association file construction module is used for constructing a field association file of the service data according to the field blood relationship and the field inheritance relationship and marking identification attributes among fields in the field association file;
and the blood relationship acquisition module is used for storing the data in the service data into a graph database according to the field association file and the identification attribute so as to acquire the data blood relationship among the data in the service data.
In order to solve the above problem, the present invention also provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to implement the data relationship analysis method described above.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the data relationship analysis method.
It can be seen that, in the embodiments of the present invention, firstly, service data of a data query request input by a user is queried from a service system, and a computation engine is used to analyze a field blood relationship between fields in the service data, so that a source of a field in the service data can be known, thereby ensuring a precondition of tracing back a blood relationship between subsequent service data, and the service data is stored in a preset message queue, thereby implementing asynchronous processing of the service data, improving a data processing speed of a large data volume, and satisfying data processing requirements of different users; secondly, the embodiment of the invention loads the service data in the message queue into a local disk, and triggers an asynchronous thread on the local disk, so as to run the service data through the asynchronous thread, analyze the field inheritance relationship between fields of the service data in the running process of the asynchronous thread, and obtain the calling relationship of the fields in the service data, thereby ensuring the premise of data consanguinity relationship identification between each piece of data in the service data; further, in the embodiment of the present invention, a field association file of the service data is constructed according to the field blood relationship and the field inheritance relationship, an identification attribute between fields is marked in the field association file, and the data in the service data is stored in a graph database in combination with the field association file and the identification attribute to obtain the data blood relationship between the data in the service data, so that excessive manual analysis of the data blood relationship between the data can be avoided, and the efficiency of analyzing the data blood relationship can be improved.
Drawings
Fig. 1 is a schematic flow chart illustrating a data relationship analysis method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a data relationship analysis apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device for implementing a data relationship analysis method according to an embodiment of the present invention;
the implementation, functional features and advantages of the present invention will be further described with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a data blood relationship analysis method. The execution subject of the data relationship analysis method includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiments of the present application. In other words, the data lineage relationship analysis method may be performed by software or hardware installed in a terminal device or a server device, where the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
Fig. 1 is a schematic flow chart of a data relationship analysis method according to an embodiment of the present invention. In an embodiment of the present invention, the data blood relationship analysis method includes:
s1, receiving a data query request input by a user, acquiring service data from a service system according to the data query request, analyzing field blood relationship among fields in the service data by using a calculation engine, and storing the service data into a preset message queue.
In the embodiment of the invention, the data query request is generated based on different user requirements, for example, the requirement of the user a is to query data about car insurance claims in the last year, the requirement of the user B is to query data about life insurance claims in the last year, and the service system can be understood as a data platform for data interaction with the user in an actual service scene, and the data platform comprises an interaction page platform, a data management platform, a data storage platform and the like.
As an embodiment of the present invention, the acquiring service data from a service system according to the data query request includes: and identifying a query object and a query field of the data query request, converting the data query request into a query statement according to the query object and the query field, and querying service data from a background database of the service system according to the query statement.
The query object refers to a query target of the data query request, such as the above "car insurance", the query field is used for representing a query node of the data query request, and the query statement may be converted in SQL programming language to form an SQL statement.
It should be understood that the business data is obtained by querying from the business system through the SQL statement, and there may be many query fields in the SQL statement, and the query fields thereof query corresponding destination fields to form corresponding business data, therefore, in the embodiment of the present invention, the computing engine is used to analyze the blood relationship between the fields in the business data to know the source of the fields in the business data, thereby ensuring the precondition of tracing the blood relationship between subsequent business data. The field consanguineous relation can be understood as a relation similar to human social consanguineous relation formed between fields in the processes of generation, processing, circulation to extinction of the fields, and can represent dependency information between input fields and output fields generated in the process of loading and running business data.
As an embodiment of the present invention, the parsing, by using a computing engine, field blood relationship between fields in the business data includes: capturing the corresponding relation between the input field and the output field of the business data in the query process by utilizing a hook function in the computing engine, constructing a decision tree of the input field and the output field by utilizing a decision algorithm in the computing engine according to the corresponding relation, computing a node path of any two fields in the decision tree by utilizing a depth traversal algorithm, determining the incidence relation of the corresponding fields according to the node path, and taking the incidence relation as the field blood relationship between the fields in the business data.
The input field may be understood as a query field in the data query request, the output field may be understood as a field in the service data, the computation engine may be constructed by a Spark computation framework, the hook function may be a hook function, the decision algorithm may be an AST algorithm, the deep traversal algorithm may be a vertical traversal algorithm, the node path refers to a connection path of any two fields, and the association relationship refers to a field relationship determined according to the node path, and includes a parallel relationship, an inclusion relationship, and the like.
Further, the embodiment of the present invention maintains the service data in a preset message queue to implement asynchronous processing of the service data, increase data processing speed of a large data volume, and meet data processing requirements of different users, where the preset message queue may be a kafka message queue, and the storage of the service data may be implemented by calling an interface of the message queue.
S2, loading the service data in the message queue to a local disk, and triggering an asynchronous thread on the local disk to run the service data through the asynchronous thread, and analyzing the field inheritance relationship between the fields of the service data in the running process of the asynchronous thread.
In the embodiment of the present invention, the local disk refers to a disk used for storing data at a local client, and the asynchronous Thread refers to a minimum unit used for running a service data task, which can be created through a Java language, for example, a Thread created through an inherited Thread class of Java.
In an optional embodiment of the present invention, the loading the service data in the message queue to a local disk includes: and serializing the service data in the message queue to obtain serialized data, and transmitting the serialized data to a local disk by adopting a first-in first-out sequence. The serialization refers to a process of converting the state information of the service data into a form capable of being stored or transmitted, and can be implemented through a Java language.
Further, in the embodiment of the present invention, the asynchronous thread is triggered on the local disk, so that the service data is run through the asynchronous thread to identify the field inheritance relationship between fields in the service data, thereby implementing the premise of identifying the data lineage relationship between data in the service data. The field inheritance relationship can understand the relationship between fields with the same property and behavior.
Further, as an embodiment of the present invention, the analyzing the field inheritance relationship between fields of the service data in the operation process includes: performing deserialization on the service data to obtain deserialization data, identifying a call relation between fields in the deserialization data, identifying a parent field and a child field of the deserialization data in the running process of the asynchronous thread according to the call relation, and generating a field inheritance relation between the fields in the service data according to the parent field and the child field.
The deserialization corresponds to the serialization, which can be understood as a process of recovering a byte sequence in data into a Java object, and the call relation can be realized by an upward transformation method in a Java language.
Based on the identification of the field inheritance relationship, the calling relationship of the fields in the service data can be obtained, so that the blood relationship identification premise among all data in the service data can be guaranteed.
S3, constructing a field association file of the business data according to the field consanguinity relation and the field inheritance relation, and marking the identification attributes among the fields in the field association file.
It should be understood that the field context may determine an association relationship between an input field and an output field in the business data, and the field inheritance relationship may determine a call relationship between fields in the business data, so that according to the field context and the field inheritance relationship, an embodiment of the present invention constructs a conversion relationship of fields in the business data to describe a data context between data in the business data.
As an embodiment of the present invention, the constructing a field association file of the service data according to the field consanguinity relationship and the field inheritance relationship includes: and connecting the rest fields in the service data with the clustering center point by adopting a conceptual relationship model to form a field association file.
Illustratively, the service data includes 1000 fields, and there are 50 fields that include both field blood relationship and field inheritance relationship, so that the 50 fields are used as a cluster center point, the remaining 950 fields are used as points to be connected, and a conceptual relationship model is adopted to connect the cluster center point and the points to be connected according to the relationship between each field in the 1000 fields to form the 1000-field associated file.
Further, in order to ensure the privacy and security of the field association file, the field association file may also be stored in a blockchain node.
Based on the field association file, the connection relation between each data field in the business data can be visually displayed, so that the data blood relationship between the data in the business data can be rapidly judged, and data monitoring is realized.
Furthermore, the embodiment of the invention determines the identity between the fields in the field association file by marking the identification attributes between the fields in the field association file, thereby facilitating the rapid search of the subsequent data relationship. As an embodiment of the present invention, the marking an identification attribute between fields in the field association file includes: and inquiring the field ID and the field number between the fields in the field association file, and splicing the field ID and the field number to obtain the identification attribute between the fields in the field association file. The field ID refers to a unique information identifier of the field, the field number refers to a task number of the field in the asynchronous thread in the service data running process, and the splicing can be realized by splicing characters, such as "-".
And S4, storing the data in the service data into a graph database according to the field association file and the identification attribute so as to obtain the data blood relationship among the data in the service data.
In the embodiment of the present invention, the graph database is a non-relational database, and the graph theory is applied to store relationship information between entities, and the graph database may be a Neo4j database, and further, based on the field association file and the identification attribute, the embodiment of the present invention stores data in the service data into the graph database through a data import tool, so as to achieve acquisition and display of a blood relationship between data in the service data, wherein the data import tool may be a Neo4j-import tool. The data relationship can be understood as a relationship similar to human social relationship formed between data in the processes of generation, processing, circulation to extinction, and can represent dependency information between input data and output data generated in the process of loading and running business data.
Further, after storing the data in the service data into the graph database, the present invention further includes: and setting a read-write separation interface in the graph database to realize read-write separation of subsequent data consanguinity relationship acquisition and improve the quick query of the data consanguinity relationship.
It can be seen that, in the embodiments of the present invention, firstly, service data of a data query request input by a user is queried from a service system, and a computation engine is used to analyze a field blood relationship between fields in the service data, so that a source of a field in the service data can be known, thereby ensuring a precondition of tracing back a blood relationship between subsequent service data, and the service data is stored in a preset message queue, thereby implementing asynchronous processing of the service data, improving a data processing speed of a large data volume, and satisfying data processing requirements of different users; secondly, the embodiment of the invention loads the service data in the message queue into a local disk, triggers an asynchronous thread on the local disk, runs the service data through the asynchronous thread, analyzes the field inheritance relationship between fields of the service data in the running process of the asynchronous thread, and can obtain the calling relationship of the fields in the service data, thereby ensuring the data consanguinity relationship identification premise among each piece of data in the service data; further, in the embodiment of the present invention, a field association file of the service data is constructed according to the field blood relationship and the field inheritance relationship, an identification attribute between fields is marked in the field association file, and the data in the service data is stored in a graph database in combination with the field association file and the identification attribute to obtain the data blood relationship between the data in the service data, so that excessive manual analysis of the data blood relationship between the data can be avoided, and the efficiency of analyzing the data blood relationship can be improved.
FIG. 2 is a functional block diagram of the data blood relationship analysis device according to the present invention.
The data blood relationship analysis apparatus 100 according to the present invention may be installed in an electronic device. According to the realized functions, the data blood relationship analysis device may include a blood relationship analysis module 101, an inheritance relationship analysis module 102, a field association file construction module 103, and a blood relationship acquisition module 104. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and can perform a fixed function, and is stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the blood relationship analysis module 101 is configured to receive a data query request input by a user, acquire service data from a service system according to the data query request, analyze a field blood relationship between fields in the service data by using a computing engine, and store the service data in a preset message queue;
the inheritance relationship analysis module 102 is configured to load the service data in the message queue into a local disk, and trigger an asynchronous thread in the local disk, so that the service data is run through the asynchronous thread, and a field inheritance relationship between fields of the service data in a running process of the asynchronous thread is analyzed;
the field association file construction module 103 is configured to construct a field association file of the service data according to the field consanguinity relationship and the field inheritance relationship, and mark an identification attribute between fields in the field association file;
the blood relationship obtaining module 104 is configured to store the data in the service data into a graph database according to the field association file and the identifier attribute, so as to obtain a data blood relationship between the data in the service data.
In detail, when the modules in the data blood relationship analysis apparatus 100 according to the embodiment of the present invention are used, the same technical means as the data blood relationship analysis method described in fig. 1 above are adopted, and the same technical effects can be produced, which is not described herein again.
Fig. 3 is a schematic structural diagram of an electronic device 1 for implementing the data blood relationship analysis method according to the present invention.
The electronic device 1 may comprise a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may further comprise a computer program, such as a data context analysis program, stored in the memory 11 and executable on the processor 10.
In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, and includes one or more Central Processing Units (CPUs), a microprocessor, a digital Processing chip, a graphics processor, a combination of various control chips, and the like. The processor 10 is a Control Unit (Control Unit) of the electronic device 1, connects various components of the electronic device 1 by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (for example, executing a data blood-related relationship analysis program and the like) stored in the memory 11 and calling data stored in the memory 11.
The memory 11 includes at least one type of readable storage medium including flash memory, removable hard disks, multimedia cards, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disks, optical disks, and the like. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, e.g. a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of a data genetic relationship analysis program, but also to temporarily store data that has been output or is to be output.
The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
The communication interface 13 is used for communication between the electronic device 1 and other devices, and includes a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices 1. The user interface may be a Display (Display), an input unit, such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
Fig. 3 only shows the electronic device 1 with components, and it will be understood by those skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data relationship analysis program stored in the memory 11 of the electronic device 1 is a combination of a plurality of computer programs, and when running in the processor 10, can realize:
receiving a data query request input by a user, acquiring service data from a service system according to the data query request, analyzing field blood relationship among fields in the service data by using a computing engine, and storing the service data into a preset message queue;
loading the service data in the message queue to a local disk, and triggering an asynchronous thread on the local disk so as to run the service data through the asynchronous thread and analyze the field inheritance relationship between fields of the service data in the running process of the asynchronous thread;
constructing a field association file of the business data according to the field blood relationship and the field inheritance relationship, and marking identification attributes among fields in the field association file;
and storing the data in the service data into a graph database according to the field association file and the identification attribute so as to obtain the data blood relationship among the data in the service data.
Specifically, the processor 10 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the computer program, which is not described herein again.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a non-volatile computer-readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The invention also provides a computer-readable storage medium, in which a computer program is stored, which computer program, when executed by a processor of an electronic device 1, enables:
receiving a data query request input by a user, acquiring service data from a service system according to the data query request, analyzing field blood relationship among fields in the service data by using a computing engine, and storing the service data into a preset message queue;
loading the service data in the message queue to a local disk, and triggering an asynchronous thread on the local disk so as to run the service data through the asynchronous thread and analyze the field inheritance relationship between fields of the service data in the running process of the asynchronous thread;
constructing a field association file of the service data according to the field blood relationship and the field inheritance relationship, and marking identification attributes among fields in the field association file;
and storing the data in the service data into a graph database according to the field association file and the identification attribute so as to obtain the data blood relationship among the data in the service data.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a string of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, which is used for verifying the validity (anti-counterfeiting) of the information and generating a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method for data relationship analysis, the method comprising:
receiving a data query request input by a user, acquiring service data from a service system according to the data query request, analyzing field blood relationship among fields in the service data by using a computing engine, and storing the service data into a preset message queue;
loading the service data in the message queue to a local disk, and triggering an asynchronous thread on the local disk so as to run the service data through the asynchronous thread and analyze the field inheritance relationship between fields of the service data in the running process of the asynchronous thread;
constructing a field association file of the service data according to the field blood relationship and the field inheritance relationship, and marking identification attributes among fields in the field association file;
and storing the data in the service data into a graph database according to the field association file and the identification attribute so as to obtain the data blood relationship among the data in the service data.
2. The method for analyzing data consanguinity relationship according to claim 1, wherein said obtaining business data from a business system according to said data query request comprises:
identifying a query object and a query field of the data query request;
converting the data query request into a query statement according to the query object and the query field;
and inquiring service data from a background database of the service system according to the inquiry statement.
3. The method of claim 1, wherein the parsing the field consanguinity relationship between the fields in the business data using a computing engine comprises:
capturing the corresponding relation between an input field and an output field of the business data in the query process by utilizing a hook function in the calculation engine;
according to the corresponding relation, a decision tree of the input field and the output field is constructed by utilizing a decision algorithm in the computing engine;
calculating the node paths of any two fields in the decision tree by using a depth traversal algorithm;
and determining the association relation of the corresponding fields according to the node path, and taking the association relation as the field blood relationship among the fields in the business data.
4. The method for analyzing data consanguinity relationship according to claim 1, wherein the loading the service data in the message queue into a local disk includes:
serializing the service data in the message queue to obtain serialized data;
and transmitting the serialized data to a local disk in a first-in first-out order.
5. The method for analyzing data relationship of blood relationship as claimed in claim 1, wherein said analyzing the field inheritance relationship between fields of said service data during the running process of said asynchronous thread comprises:
deserializing the service data to obtain deserialized data, and identifying a call relation between fields in the deserialized data;
identifying a parent field and a child field of the deserialization data in the running process of the asynchronous thread according to the call relation;
and generating field inheritance relation between fields in the service data according to the parent field and the subclass field.
6. The method for analyzing data consanguinity relationship according to any one of claims 1 to 5, wherein the constructing a field association file of the business data according to the field consanguinity relationship and the field inheritance relationship includes:
taking the fields which have the field blood relationship and the field inheritance relationship simultaneously in the service data as clustering center points;
and connecting the rest fields in the service data with the clustering central point by adopting a conceptual relationship model to form a field association file.
7. The method of data relationship analysis according to claim 1, wherein said marking identification attributes between fields in said field association file comprises:
inquiring field ID and field number between fields in the field association file;
and splicing the field ID and the field number to obtain the identification attribute between the fields in the field association file.
8. A data relationship analysis apparatus, the apparatus comprising:
the system comprises a data query request input by a user, a data relationship analysis module, a calculation engine and a message queue, wherein the data query request is used for acquiring service data from a service system, analyzing field blood relationship among fields in the service data by using the calculation engine, and storing the service data into the preset message queue;
the inheritance relationship analysis module is used for loading the service data in the message queue into a local disk, triggering an asynchronous thread on the local disk, running the service data through the asynchronous thread, and analyzing the field inheritance relationship among fields of the service data in the running process of the asynchronous thread;
a field association file construction module, configured to construct a field association file of the service data according to the field consanguinity relationship and the field inheritance relationship, and mark an identification attribute between fields in the field association file;
and the blood relationship acquisition module is used for storing the data in the service data into a graph database according to the field association file and the identification attribute so as to acquire the data blood relationship among the data in the service data.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and (c) a second step of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data relationship analysis method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a method for data relationship analysis according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210760713.2A CN115129753A (en) | 2022-06-29 | 2022-06-29 | Data blood relationship analysis method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210760713.2A CN115129753A (en) | 2022-06-29 | 2022-06-29 | Data blood relationship analysis method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115129753A true CN115129753A (en) | 2022-09-30 |
Family
ID=83381254
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210760713.2A Pending CN115129753A (en) | 2022-06-29 | 2022-06-29 | Data blood relationship analysis method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115129753A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115934855A (en) * | 2022-11-29 | 2023-04-07 | 广发银行股份有限公司 | Full-link field level blood margin analysis method, system, equipment and storage medium |
| CN117688217A (en) * | 2024-02-02 | 2024-03-12 | 北方健康医疗大数据科技有限公司 | System, method and medium for realizing data blood relationship structure based on directed graph |
-
2022
- 2022-06-29 CN CN202210760713.2A patent/CN115129753A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115934855A (en) * | 2022-11-29 | 2023-04-07 | 广发银行股份有限公司 | Full-link field level blood margin analysis method, system, equipment and storage medium |
| CN115934855B (en) * | 2022-11-29 | 2023-08-25 | 广发银行股份有限公司 | Full-link field-level blood margin analysis method, system, equipment and storage medium |
| CN117688217A (en) * | 2024-02-02 | 2024-03-12 | 北方健康医疗大数据科技有限公司 | System, method and medium for realizing data blood relationship structure based on directed graph |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112134739B (en) | Link state tracking method and device, electronic equipment and computer storage medium | |
| CN111428458A (en) | Universal report generation method and device and computer readable storage medium | |
| CN115129753A (en) | Data blood relationship analysis method and device, electronic equipment and storage medium | |
| CN113704665B (en) | Dynamic service release method and device, electronic equipment and storage medium | |
| CN114979120B (en) | Data uploading method, device, equipment and storage medium | |
| CN112364107A (en) | System analysis visualization method and device, electronic equipment and computer readable storage medium | |
| CN113806434A (en) | Big data processing method, device, equipment and medium | |
| CN113342472A (en) | Micro-service cluster creating method and device, electronic equipment and readable storage medium | |
| CN114881616A (en) | Business process execution method and device, electronic equipment and storage medium | |
| CN111694844A (en) | Enterprise operation data analysis method and device based on configuration algorithm and electronic equipment | |
| CN113051171A (en) | Interface test method, device, equipment and storage medium | |
| CN114398194A (en) | Data collection method and device, electronic equipment and readable storage medium | |
| CN112256783A (en) | Data export method and device, electronic equipment and storage medium | |
| CN114385497A (en) | Test environment generation method and device, electronic equipment and storage medium | |
| CN114519084A (en) | Block chain-based medical data inspection method, device, equipment and medium | |
| CN114185776A (en) | Big data point burying method, device, equipment and medium for application program | |
| CN113434542A (en) | Data relation identification method and device, electronic equipment and storage medium | |
| CN112667244A (en) | Data verification method and device, electronic equipment and computer readable storage medium | |
| CN113918305B (en) | Node scheduling method, node scheduling device, electronic equipment and readable storage medium | |
| CN114911479A (en) | Interface generation method, device, equipment and storage medium based on configuration | |
| CN115102770A (en) | Resource access method, device and equipment based on user permission and storage medium | |
| CN114896164A (en) | Interface optimization method and device, electronic equipment and storage medium | |
| CN112905470A (en) | Interface calling method and device, computer equipment and medium | |
| CN114461531A (en) | Platform adaptability test method, device, equipment and storage medium of test case | |
| CN112527443A (en) | Prompt box display method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |