CN115145919A

CN115145919A - Method, device, equipment and medium for generating data blood relationship between service systems

Info

Publication number: CN115145919A
Application number: CN202210769402.2A
Authority: CN
Inventors: 郑强; 郑成坤
Original assignee: CISDI Chongqing Information Technology Co Ltd
Current assignee: CISDI Chongqing Information Technology Co Ltd
Priority date: 2022-06-30
Filing date: 2022-06-30
Publication date: 2022-10-04

Abstract

The invention relates to the technical field of computers, and particularly discloses a method, a device, equipment and a medium for generating data blood margin between service systems. The method comprises the following steps: acquiring first data, wherein the first data comprises metadata of data items needing data blood margin tracking in a plurality of service systems; registering first data, and selecting a storage mode according to a registration result to store the first data to obtain second data; and acquiring third data at preset time, and establishing a data blood relationship map according to a mapping relation between the second data and the third data, wherein the third data comprises data items obtained by intercepting service call flows among the plurality of service systems. By the method, the updated metadata can be obtained in the interaction process of the business system, and the data blooding margin is generated.

Description

Method, device, equipment and medium for generating data blood margin between service systems

Technical Field

The invention relates to the technical field of computers, in particular to a method, a device, equipment and a medium for generating data blood margin between service systems.

Background

In the processes of data generation, processing fusion, circulation and final extinction among service systems, a relationship is naturally formed among the data, which is called as the blood relationship of the data. The data blood relationship analysis technology becomes an important means for data traceability, data value evaluation, data compliance proof and data quality evaluation.

In the prior art, a data blood relationship is generated mainly by a method based on SQL (Structured Query Language) parsing, a regular SQL statement is extracted by an SQL script, and a blood relationship between SQL statement input and output is obtained through a series of cleaning, syntax analysis and abstract syntax tree generation, which has the following disadvantages: SQL grammars of different database manufacturers are different, however, the existing data blood margin analysis tool generally utilizes an SQL parser provided by a data management platform, cannot completely parse SQL sentences of other manufacturers, and is low in cross-platform performance; the processing range is narrow, and the SQL analysis based method can not be applied to the scene that the processing flow of the data between the business systems is sometimes separated from SQL.

Disclosure of Invention

In order to solve the above technical problems, embodiments of the present application provide a method, an apparatus, a device, and a medium for generating data blood relationship between service systems, which can overcome the problems in the related art that SQL statements have low cross-platform performance and a narrow processing range, and cannot be applied to a scenario of processing flow when the SQL statements depart from the SQL statements.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.

In one embodiment of the present application, a method for generating data blood margin between business systems is provided, including:

acquiring first data, wherein the first data comprises metadata of data items needing data blood margin tracking in a plurality of business systems;

registering the first data, and selecting a storage mode according to a registration result to store the first data to obtain second data;

and acquiring third data at preset time, and establishing a data blood relationship map according to a mapping relation between the second data and the third data, wherein the third data comprises data items obtained by intercepting service call processes among the plurality of service systems.

In an embodiment of the present application, before the acquiring the first data, the method includes:

registering system codes of the plurality of service systems, wherein the system codes comprise unique identification codes preset for the service systems;

if a preset marking tool exists in the business system, marking metadata of a data item needing data blood margin tracking through the preset marking tool to obtain first data;

and if no preset labeling tool exists in the service system, injecting a probe into the service system, and acquiring metadata of a data item needing data blood margin tracking through the probe to obtain the first data, wherein the metadata comprises a data object name, an attribute name, a data type, a natural language remark and extensible information, and the extensible information is used for customizing data information.

In an embodiment of the present application, the obtaining metadata of the data item to be subjected to data blood margin tracking by the probe includes:

comparing the acquired existing metadata with metadata in the plurality of service systems at preset time to obtain a comparison result;

and determining the data items and the metadata which need to be subjected to data blood margin tracking according to the comparison result.

In an embodiment of the present application, the registering the first data includes:

acquiring the state of a data blood margin center server when the first data is transmitted to the data blood margin center server;

if the data consanguinity center server is in a normal state, registering first data;

if the state of the data consanguineous center server is abnormal, caching the first data in the service system, and when the state of the data consanguineous center server is recovered to be normal, acquiring the first data again and registering.

In an embodiment of the present application, when registering the first data, the method further transmits identification information to the data bloodline center server, where the identification information includes:

the system code is used for identifying the unique service system;

a registration time for recording a time at which the first data is registered;

a data item identification for identifying a unique data item;

a data type for identifying a data type of the data item;

remark information for explaining the data items.

In an embodiment of the application, the selecting a storage manner according to the registration result to store the first data and obtain the second data includes:

if the registration result is that the data which is the same as the first data already exists in the data consanguinity center server, the first data is not stored, and the existing storage information is associated with a service system corresponding to the first data;

if the registration result is that the data in the data consanguinity center server is completely different from the first data, storing the first data;

if the registration result is that the data which are the same as part of the first data exist in the data consanguinity center server, calculating the mutual relation between the first data and the part of the same data, and selecting and storing different part contents in the first data according to the mutual relation;

and if the registration result is that data which belongs to the same data item with the first data exists in the data consanguinity center server, updating the data which belongs to the same data item according to the first data.

In an embodiment of the present application, the acquiring the third data at the preset time includes:

when the plurality of service systems call services, acquiring a calling link and data items transmitted in the calling link, and generating a log record according to the data items transmitted in the calling link, wherein the data items transmitted in the calling link at least comprise a service system name, a calling ID, a called system name, a called interface name, calling generation time, calling duration, a data item value and a calling state;

and transmitting the log record to a data consanguinity center server at preset time according to the configuration of the service system to obtain third data.

In an embodiment of the present application, the establishing a data blood-related map according to the mapping relationship between the second data and the third data includes:

taking the third data as a center, and constructing a system calling relationship map according to the mapping relationship between the second data and the third data;

and calculating an old value and a new value of a data item in each node of the system call relation graph to obtain the data blood-related graph, wherein the key calculation process comprises a process of changing the old value of the data item into the new value of the data item.

In an embodiment of the application, after the obtaining the third data at the predetermined time and establishing the data blood-related atlas according to the mapping relationship between the second data and the third data, the method includes:

determining a starting point, an end point and calling link information of the service calling flow according to each service calling flow among the plurality of service systems;

and recording the starting point, the end point and the calling link information in the data blood margin map, and displaying the data blood margin to obtain a data blood margin graph display interface.

In an embodiment of the present application, the determining, according to each service invocation flow among the plurality of service systems, a start point, an end point, and invocation link information of the service invocation flow includes:

obtaining a starting point and an end point of the service calling flow according to the calling information in the service calling flow among the plurality of service systems each time;

and transmitting the call information to a data call flow relation model to obtain call link information, wherein the call link information is divided into a normal data flow and an abnormal data flow according to a call state, a long data flow and a short data flow according to the link length, and a same data flow and a different data flow according to the difference of flow starting nodes.

In an embodiment of the present application, the data call flow relationship model includes:

a data start point for representing the start of a data flow, the data start point storing an initial state of data, an initiator, metadata of the data item, and custom information;

the data intermediate node is used for representing the intermediate process of the data flow, and the data intermediate node stores the intermediate process information of the data, the metadata of the data item and the self-defining information;

the data destination is used for representing an end node of a data flow, and when data in the nodes in the data blood-level graph does not flow to the next node or normal transmission cannot be performed due to abnormity in the data transmission process, the current node is set as the data destination;

the data circulation line is used for representing the flow relation among the nodes in the data blood margin map, and the data circulation line is connected with the data starting point, the data intermediate nodes and the data end point;

and the data annotation is used for storing user-defined data items and node information and assisting data understanding and visual display.

In one embodiment of the present application, a business intersystem data blood margin generation apparatus is provided, including:

the system comprises a first data acquisition module, a second data acquisition module and a data processing module, wherein the first data acquisition module is used for acquiring first data, and the first data comprises metadata of data items needing data blood margin tracking in a plurality of business systems;

the second data acquisition module is used for registering the first data and selecting a storage mode according to a registration result to store the first data to obtain second data;

and the data blood margin map establishing module is used for acquiring third data at preset time, and establishing a data blood margin map according to the mapping relation between the second data and the third data, wherein the third data comprises data items obtained by intercepting service call flows among the plurality of service systems.

In one embodiment of the present application, there is provided an electronic device including:

one or more processors;

a storage device for storing one or more programs that, when executed by the one or more processors, cause the electronic device to implement the business intersystem data consanguinity method as described above.

In an embodiment of the present application, a computer readable medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, is adapted to carry out the method for generating data consanguinity between business systems as described above.

In the technical scheme provided by the embodiment of the application, the data consanguinity generation method between the service systems can be executed in the data consanguinity center server, and the metadata of the data items needing data consanguinity tracking can be obtained; registering and storing the metadata; intercepting data items of a service call flow between service systems; and establishing a data blood relationship map according to the metadata and the data items of the calling process. The method can efficiently generate the data blood relationship among the service systems; compared with the SQL analysis method, the method has no platform limitation; and the data item of the calling flow during data processing circulation between the service systems is combined with the generation of the data blood margin, so that the problem that the data blood margin relation cannot be acquired during the data processing circulation in the prior art is avoided, the updating information of the data item can be acquired in real time, and the data quality is optimized.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings described below are only some embodiments of the application, and that for a person skilled in the art, other drawings can be obtained from these drawings without inventive effort. In the drawings:

FIG. 1 is a schematic illustration of an implementation environment to which the present application relates;

FIG. 2 is a flow chart illustrating a method for generating traffic intersystem data blood margin according to an exemplary embodiment of the present application;

FIG. 3 is a diagram illustrating a nesting relationship of data lineage metadata according to an exemplary embodiment of the present application;

FIG. 4 is a schematic flow chart illustrating the process of labeling metadata of data items to be subjected to data lineage tracking before step S210 in the embodiment shown in FIG. 2;

fig. 5 is an architecture diagram for registering the first data in step S220 in the embodiment shown in fig. 2;

fig. 6 is a schematic flowchart of the process of storing the first data according to the registration result in step S220 in the embodiment shown in fig. 2;

fig. 7 is a schematic structural diagram of a business intersystem data blood margin generation apparatus according to an exemplary embodiment of the present application.

FIG. 8 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

Reference to "a plurality" in this application means two or more. "and/or" describe the association relationship of the associated objects, meaning that there may be three relationships, e.g., A and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Abbreviations and key term definitions referred to herein with this application are introduced:

a service system: the business system refers to business links required by enterprises for achieving positioning, roles played by all partners and modes and contents of cooperation and transaction of stakeholders. When the background among the service systems is docked, the data in the service systems needs to be traced, and the data is realized through the blood margin of the data.

Data item: a data element may consist of several data items (data items), which are the indivisible smallest units of data. The names of the data items comprise numbers, alias names, brief descriptions, the lengths and the types of the data items and the value ranges of the data items. The data item is the most basic, indivisible named data unit in the data record, and is the smallest identification unit with independent meaning.

Data blood relationship: the data generation, processing fusion, circulation and circulation are carried out until the data disappear finally, and a relationship is naturally formed among the data. The relationship between data is expressed by referring to a similar relationship in human society, which is called the blood-related relationship of data. The data bloodline is one of the components of the metadata. It analyzes the table and the table fields for the bloody border path from the data source to the current table, and whether the relationships existing between the bloody border fields are satisfied, the data consistency of interest, and the rationality of the table design. The method can be used for analyzing the influence of the change of the upstream data on the downstream data; and tracking the source of the upstream problem when the downstream data changes. The data bloodline had: attribution, wherein specific data belongs to a specific organization or an individual, and the data has attribution; the data can be generated by processing a plurality of data, and the processing process can be a plurality of processes; the traceability and the blood relationship of the data reflect the life cycle of the data and the whole process from generation to extinction of the data, and the traceability is realized; the hierarchy, the blood relationship of the data is hierarchical, the description information of the data such as classification, induction and summary of the data forms new data, and the description information of different degrees forms the hierarchy of the data.

Metadata: metadata, also called intermediate data and relay data, is data describing data (data about data), and is mainly information describing data attribute (property) for supporting functions such as indicating storage location, history data, resource search, file record, and the like. Metadata is an electronic catalog, and in order to achieve the purpose of creating a catalog, the contents or features of data must be described and collected, so as to achieve the purpose of assisting data retrieval.

And calling a link, wherein the calling link is used for recording relevant data of the whole request link in sequence from the initiation of a service request to the termination of a user or a machine for subsequent query analysis, positioning system bug or performance optimization.

And (3) SDK: software Development Kit, namely, the Software Development Kit. SDKs are a collection of development tools used by software engineers to create application software for a particular software package, software framework, hardware platform, operating system, etc. It may simply be a file that provides an application program interface API for a certain programming language, but may also include complex hardware that can communicate with a certain embedded system. SDKs also often include example code, supporting technical notes, or other supporting documentation to clarify suspicions for basic reference.

The data blood margin generation method, apparatus, device and medium according to the embodiments of the present application relate to the fields of data blood margin, business system and the like described above, and the embodiments will be described in detail below.

Referring first to fig. 1, fig. 1 is a schematic diagram of an implementation environment related to the present application. The implementation environment comprises

terminal devices

101, 102, 103, a network 104 and a server 105. Network 104 may provide a medium for communication links between

terminal devices

101, 102, 103 and servers. Network 104 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

The users may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104, to receive or send messages, etc. The

terminal devices

101, 102, 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, wearable devices, virtual reality devices, smart homes, smart voice interaction devices, smart appliances, vehicle terminals, and the like.

The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the

terminal apparatuses

101, 102, 103. The background management server may analyze and otherwise process the received data such as the request, and feed back the processing result to the

terminal devices

101, 102, and 103.

The server 105 may be an independent physical server, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server that provides basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, web service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), big data, and an artificial intelligence platform, and the like, which is not limited by the present disclosure.

In an embodiment of the present application, the server 105 may be, for example, a data lineage center server configured to track data items between business systems and generate data lineage relationships from metadata of the tracked data items; the server 105 can also perform visual display on the generated data blood relationship to generate a visual data blood relationship chart; the

terminal devices

101, 102, 103 receive the visualized data blood relationship chart transmitted by the server 105 through the network 104, and then show the user.

The data consanguineous center server may, for example, respond to: acquiring first data, wherein the first data comprises metadata of data items needing data blood margin tracking in a plurality of business systems; registering the first data, and selecting a storage mode according to a registration result to store the first data to obtain second data; and acquiring third data at preset time, and establishing a data blood relationship map according to a mapping relation between the second data and the third data, wherein the third data comprises data items obtained by intercepting service call flows among the plurality of service systems. After the data blood margin between the service systems is obtained, the data blood margin central server can display the data blood margin to generate a data blood margin chart for a user to observe and analyze.

Fig. 2 is a flowchart illustrating a method for generating data consanguinity between business systems according to an exemplary embodiment of the present application. The method provided by the embodiment of the present application may be executed by any electronic device with computing processing capability, for example, the method may be executed by the server 105 in the embodiment of fig. 1 described above. In the following embodiments, the data consanguinity center server is taken as an example for illustration, but the disclosure is not limited thereto.

Referring to fig. 2, a business system data blood margin generation method provided by an embodiment of the present application may include the following steps.

In step S210, first data is obtained, where the first data includes metadata of data items in a plurality of business systems that need data consanguinity tracking.

The metadata of the service system data item is marked to obtain first data. Because the data blood relationship of some data items in the service system does not change in the system calling process, or some data items are not target data items that the user needs to perform data blood relationship generation, before the metadata is labeled, the data items that need to perform data blood relationship tracking can be identified. The number of service systems is not limited in the present application.

It should be noted that the data lineage metadata can be infinitely nested to characterize the relationship between data. Referring to fig. 3, fig. 3 is a schematic diagram illustrating a nesting relationship of data lineage metadata according to an exemplary embodiment of the present application. Data items A1, A2, A3 are metadata of data item a, and data item A1 also has corresponding metadata a11, a12; accordingly, the data item A3 also has corresponding metadata a31.

Referring to fig. 4, fig. 4 is a schematic flow chart illustrating the process of marking metadata of a data item requiring data blood-reason tracking before step S210 in the embodiment shown in fig. 2, including:

s410, registering system codes of the plurality of service systems, wherein the system codes comprise unique identification codes preset for the service systems.

In an embodiment of the application, before data consanguinity generation, corresponding unique identification codes are set in each service system in advance, and then the unique identification codes of each system are transmitted to a data consanguinity center server for registration. When the identification code is transmitted, the related information of the system can be transmitted to the data consanguinity center server to be registered together, and the related information of the system can be, for example, system architecture information, system code, and system code modification.

And S420, acquiring first data through a preset marking tool or probe in the service system.

In an embodiment of the application, if a code of a business system can be modified, metadata of a data item needing data blood margin tracking is marked in the code in an annotation mode to obtain first data; and if the code of the business system cannot be modified, injecting a probe into the code, and acquiring metadata of a data item needing data blood margin tracking through the probe to obtain first data.

Wherein, a data item labeling tool, in this embodiment, a labeling SDK (Software Development Kit) is integrated in a code of a part of the service system, and the code of the part of the service system can be modified; and if the SDK is not integrated in the code of the part of the service system, the code of the part of the service system cannot be modified, and a probe is required to acquire the metadata in the part of the service system.

In an embodiment of the present application, the step S420 specifically includes the following steps:

if a preset marking tool exists in the business system, marking metadata of a data item needing data blood margin tracking through the preset marking tool to obtain the first data;

and if no preset marking tool exists in the service system, a probe is injected into the service system, metadata of a data item needing data blood margin tracking is obtained through the probe, and the first data are obtained, wherein the metadata comprise a data object name, an attribute name, a data type, a natural language remark and extensible information, and the extensible information is used for self-defining data information.

The metadata comprises a data object name, an attribute name, a data type, a natural language remark and extensible information, and the extensible information is used for customizing the data information.

In the embodiment, when the labeling SDK is integrated in the service system, the metadata is labeled in an annotation mode; when the service system does not integrate the annotation SDK, the probe is used to obtain metadata. The two metadata acquisition modes are selected for different types of service systems, so that the method is suitable for different platforms, cross-platform operation is realized, and the accuracy of metadata is ensured.

In an embodiment of the present application, the obtaining metadata of the data item that needs to be subjected to data blood margin tracking by the probe specifically includes the following steps: comparing the acquired existing metadata with metadata in the plurality of service systems at preset time to obtain a comparison result;

In the implementation, a probe is used for identifying service call flows and data items among a plurality of service systems; acquiring metadata of data items acquired from a data consanguineous center server at preset time, and comparing the metadata with the metadata of the data items in a plurality of service systems to obtain a comparison result; identifying the data items needing data blood relationship tracking according to the comparison result; and dynamically acquiring the metadata of the data item needing data blood margin tracking when a business system runs through the probe.

In this embodiment, the data items to be subjected to data blood margin tracking are identified by the probe, and then the metadata is acquired.

In S220, the first data is registered, and a storage manner is selected according to the registration result to store the first data, so as to obtain second data.

In an embodiment of the present application, when the business system runs, the metadata of the data item is registered in the data limbus center server, and the data limbus center server receives the metadata of the data item, and then stores and indexes the metadata.

In one embodiment of the present application, referring to fig. 5, fig. 5 is an architecture diagram of registering the first data in step S220 in the embodiment shown in fig. 2. When registering the first data, the method comprises the following steps:

when the business system runs, collecting metadata of data items of the system, and transmitting the metadata of the data items to a data consanguineous center server;

if the data consanguineous center server is abnormal, the service system caches a piece of metadata information, and when the data consanguineous center server is normal, the metadata information is retransmitted to the data consanguineous center server.

In this embodiment, the three systems a, B, and C transmit the data items to the data consanguinity center server for registration, and then return the registration result to the corresponding service system.

Acquiring the state of the data blood margin center server when the first data is transmitted to the data blood margin center server; if the data consanguinity center server is in a normal state, registering the first data; if the data blood margin center server state is abnormal, caching the first data in the service system, and when the data blood margin center server state is recovered to be normal, acquiring the first data again and registering. The method can avoid the situations of data loss and the like caused by registering the metadata when the state of the data edge center server is abnormal.

In one embodiment of the present application, identification information is further transmitted to the data edge center server when registering the first data, and the identification information includes: the system code is used for identifying the unique service system; a registration time for recording a time at which the first data is registered; a data item identification for identifying a unique data item; a data type for identifying a data type of the data item; remark information for explaining the data items.

In an embodiment of the present application, referring to fig. 6, fig. 6 is a schematic flowchart illustrating a process of storing the first data according to the registration result in step S220 in the embodiment shown in fig. 2. Selecting a storage mode to store the first data according to the registration result, and specifically comprising the following steps of:

if the registration result is that the data which are the same as part of the first data exist in the data blood margin center server, calculating the correlation between the first data and the part of the same data, and selecting and storing different part contents in the first data according to the correlation;

In this embodiment, after the data consanguinity center server receives the metadata of the data item, if the center server has the same metadata information of the data item uploaded by other service systems, the center server directly informs the service systems that the metadata information is registered, and associates the system that registers the metadata information of the data item for the first time with the service systems at the same time, indicating that they can process the same data item; after the data blood margin central server receives the data item metadata, if the central server does not have the data item metadata information, the data item metadata information is stored; after the data blood margin central server receives the data item metadata, if the central server has part of the data item metadata information, the relationship among the metadata information is calculated and stored. After the data consanguinity center server receives the data item metadata, if the data item metadata belongs to the same data item, but the data item has an update, the data item metadata information is updated. The data storage manner in this embodiment may be any manner that can express the relationship between the metadata information of the data item and the metadata information.

In step S230, third data is acquired at a preset time, and a data blood-level map is established according to a mapping relationship between the second data and the third data.

And the third data comprises data items obtained by intercepting service call flows among the plurality of service systems.

In an embodiment of the application, in the service system calling process, by intercepting the calling among the service systems, the data items transmitted in the calling link are obtained, and the data items transmitted in the calling link are cached locally and are timely and batch-sent to the data consanguinity center server.

In an embodiment of the application, the obtaining of the third data at the preset time specifically includes the following steps:

when the plurality of service systems call services, acquiring a calling link and data items transmitted in the calling link, and generating a log record according to the data items transmitted in the calling link, wherein the data items transmitted in the calling link at least comprise a service system name, a current calling ID, a called system name, a called interface name, calling generation time, calling duration, a data item value and a calling state;

In this embodiment, a local log is generated according to data transmitted in the call link, and the local log is transmitted to the data consanguinity center server in real time or in batch for processing and storing. When the local logs are abnormally sent in real time or in batch, the local logs are sent again after a period of time to ensure that the data cannot be lost; and after the data consanguinity center server receives the local log, deleting the local log according to a specified deletion strategy within a preset time so as to recover the storage resources and the computing resources.

In an embodiment of the present application, the establishing a data blood relationship map according to a mapping relationship between the second data and the third data specifically includes the following steps:

It should be noted that, in the present application, a mapping relationship exists between the metadata and the data item transmitted in the call link, so that the data blood relationship map can be obtained according to the mapping relationship between the metadata and the data item transmitted in the call link.

In this embodiment, the data consanguinity center server, according to metadata information of data items reported by each system, takes a system name, a current calling ID, a called system, a called interface name, a calling generation time, a calling duration, data items and values, a calling state, and other customizable data information as a center, constructs a system calling relationship graph according to a mapping relationship between metadata and the data items, and calculates an old value, a new value, and a key calculation process of related data in each node of the calling relationship, thereby finally forming a data consanguinity graph;

the relationship of the data blood relationship maps can be stored in relational, non-relational, memory and file databases, and indexes are established for the data blood relationship maps, so that the rapid retrieval is facilitated;

the key calculation process refers to the process of changing the data item from the old value to the new value, and codes, formulas, operations and tasks causing the change of the process are recorded by the data consanguineous center server.

In an embodiment of the application, after the third data is acquired at a preset time and the data consanguinity atlas is established according to the mapping relationship between the second data and the third data, the method further includes displaying the data consanguinity, and specifically includes the following steps:

In an embodiment of the application, the starting point, the end point and the calling link information of the service calling process are obtained according to the calling information in the service calling process among a plurality of service systems each time; displaying the data blood margin according to the starting point, the end point and the calling link information of the service calling process, and generating an index according to the category and the index field to obtain a data blood relationship graph display interface.

In an embodiment of the present application, determining a start point, an end point, and call link information of the service call flow according to each service call flow among the plurality of service systems specifically includes the following steps:

The data call flow relation model is a relation flow model designed for service call and information transmission among systems, and comprises the following steps: the method comprises the following steps of (1) data starting point, data intermediate node, data end point, data circulation line and data annotation;

the data starting point represents the start of a data flow and stores the initial state of data, an initiator, metadata information of the data item and custom information;

the data intermediate node represents the intermediate process of the data flow, and stores the intermediate process information of the data, the metadata information of the data item and the user-defined information;

the data endpoint represents an end node of the data flow, the data blooding margin central server does not analyze that the data flow is transmitted to the next node, or the data is analyzed to be abnormal and cannot be transmitted continuously, and then the data flow is marked to be ended;

the data flow transfer line expresses a flow relation among the data nodes and is connected with a data starting point, a data intermediate node and a data end point;

the data annotation is used for storing user-defined data items and data node information and assisting data understanding and visual display.

In an embodiment of the application, the data blood margin graphical display interface can perform operations of zooming, rotating, labeling and editing; the data blood margin graph display interface can perform operations of drilling up, drilling down data items, data calling and flow information; nodes in the data bloodline graphical presentation interface can be manipulated to move and hide.

In this embodiment, a visualization technique is used to display the data items and the data blood relationship between the data items, so as to generate a graphical interface. The generated graphical interface can be interactively displayed according to the control instruction of the user, and an analysis tool and a customizable analysis tool plug-in are provided for different analysis requirements.

To sum up, the method for generating the data blooding margin among the service systems can efficiently generate the data blooding margin relation among the service systems, displays the data blooding margin relation in a graphical mode, and is convenient for optimizing data quality, tracking and auditing data, checking data safety compliance and visually maintaining and managing data assets.

It should be noted that although the various steps of the methods in this application are depicted in the drawings in a particular order, this does not require or imply that these steps must be performed in this particular order, or that all of the shown steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.

Embodiments of the apparatus of the present application are described below, which may be used to perform the inter-business system data blood margin generation method in the above embodiments of the present application. Referring to fig. 7, fig. 7 is a schematic structural diagram illustrating an inter-business system data blood margin generation apparatus according to an exemplary embodiment of the present application. The device for generating data blood relationship between service systems comprises a first data acquisition module 701, a second data acquisition module 702, a data blood relationship map establishing module 703 and a data blood relationship display module 704.

The first data acquisition module 701 is configured to acquire first data, where the first data includes metadata of data items that need to be subjected to data blood margin tracking in multiple business systems;

a second data obtaining module 702, configured to register the first data, and select a storage manner according to a registration result to store the first data, so as to obtain second data;

a data blood relationship graph establishing module 703, configured to obtain third data at a preset time, and establish a data blood relationship graph according to a mapping relationship between the second data and the third data, where the third data includes a data item obtained by intercepting a service call flow between the multiple service systems;

a data blood margin display module 704, configured to determine a start point and an end point of a service call flow and call link information according to the service call flow among the multiple service systems each time; and recording the starting point, the end point and the calling link information in the data blood margin map, and displaying the data blood margin to obtain a data blood margin graph display interface.

In an embodiment of the application, based on the foregoing scheme, the inter-business system data blood margin generation apparatus further includes a data blood margin labeling module, configured to label a data item to be tracked. The data blood margin labeling module comprises: the system comprises a metadata manual labeling unit, an SDK and a metadata automatic identification labeling unit.

In an embodiment of the present application, based on the foregoing solution, the second data obtaining module 702 includes a metadata storage unit, the metadata storage unit is used for storing metadata of data items and establishing an index, and the metadata storage unit includes a metadata storage sub-unit and a blood-related data index unit.

The inter-service-system data blood-margin generation device provided by the embodiment of the application is applied to the mutual calling interaction among service systems, the modules and the units are used for acquiring the metadata of data items needing data blood-margin tracking in a plurality of service systems, and then registering and storing the metadata; acquiring data items transmitted in a calling link in the calling process of a plurality of service systems; establishing a data blood relationship map according to the metadata and the data items transmitted in the calling link; and recording the starting point, the end point and the calling link information of the service calling process in a data blood margin map so as to visually display the data blood margin for analysis by a user.

It should be noted that the apparatus provided in the foregoing embodiment and the method provided in the foregoing embodiment belong to the same concept, and the specific manner in which each module and unit execute operations has been described in detail in the method embodiment, and is not described again here.

Embodiments of the present application also provide an electronic device, including one or more processors, and a storage device, where the storage device is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device is enabled to implement the inter-business system data blood margin generation method as described above.

FIG. 8 illustrates a schematic structural diagram of a computer system suitable for use to implement the electronic device of the embodiments of the present application.

It should be noted that the computer system 800 of the electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 8, a computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes, such as performing the methods in the above-described embodiments, according to a program stored in a Read-Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for system operation are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An Input/Output (I/O) interface 805 is also connected to the bus 804.

The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. When the computer program is executed by the Central Processing Unit (CPU) 801, various functions defined in the system of the present application are executed.

It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access Memory (R AM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

Yet another aspect of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the business intersystem data consanguinity generation method as before. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist separately without being incorporated in the electronic device.

Another aspect of the application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, so that the computer device executes the method for generating the inter-business system data blood margin provided in the foregoing embodiments.

According to an aspect of an embodiment of the present application, there is further provided a computer system, including a Central Processing Unit (CPU), which can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) or a program loaded from a storage portion into a Random Access Memory (RAM), for example, execute the method in the foregoing embodiment. In the RAM, various programs and data necessary for system operation are also stored. The CPU, ROM, and RAM are connected to each other via a bus. An Input/Output (I/O) interface is also connected to the bus.

The following components are connected to the I/O interface: an input section including a keyboard, a mouse, and the like; an output section including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section including a hard disk and the like; and a communication section including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section performs communication processing via a network such as the internet. The drive is also connected to the I/O interface as needed. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive as necessary, so that a computer program read out therefrom is mounted into the storage section as necessary.

The above description is only a preferred exemplary embodiment of the present application, and is not intended to limit the embodiments of the present application, and those skilled in the art can easily make various changes and modifications according to the main concept and spirit of the present application, so that the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for generating data blood margin between service systems is characterized by comprising the following steps:

and acquiring third data at preset time, and establishing a data blood relationship map according to a mapping relation between the second data and the third data, wherein the third data comprises data items obtained by intercepting service call flows among the plurality of service systems.

2. The method for generating data blooding margin between business systems according to claim 1, wherein before the acquiring the first data, the method comprises:

3. The method for generating data blooding margin between business systems according to claim 2, wherein said obtaining metadata of data items needing data blooding margin tracking through said probe comprises:

4. The method for generating interbusiness intersystem data blooding margin according to claim 1, wherein the registering the first data includes:

if the state of the data consanguineous center server is abnormal, caching the first data in the service system, and when the state of the data consanguineous center server is recovered to be normal, re-acquiring the first data and registering.

5. The method for generating data margin between business systems according to claim 4, wherein identification information is further transmitted to the data margin center server when registering the first data, and the identification information includes:

the system code is used for identifying the unique service system;

registering time for recording the time of registering the first data;

a data item identification for identifying a unique data item;

a data type for identifying a data type of the data item;

remark information for explaining the data items.

6. The method for generating data bloodlines between business systems according to claim 1, wherein the selecting a storage method to store the first data according to the registration result to obtain the second data comprises:

7. The method for generating data blooding margin between business systems according to claim 1, wherein said obtaining the third data at the predetermined time comprises:

8. The method for generating data blood margin between business systems according to claim 1, wherein the establishing a data blood margin map according to the mapping relationship between the second data and the third data comprises:

and calculating an old value and a new value of a data item and a key calculation process in each node of the system call relation graph to obtain the data blood relationship graph, wherein the key calculation process comprises a process of changing the old value of the data item into the new value of the data item.

9. The method for generating data blooding margin between business systems according to claim 1, wherein said obtaining the third data at a preset time, and establishing the data blooding margin map according to the mapping relationship between the second data and the third data, comprises:

10. The method of claim 9, wherein the determining the start point, the end point, and the call link information of the service call flow according to each service call flow between the plurality of service systems comprises:

11. The method of generating data blooding margin between business systems according to claim 10, wherein said data calls a flow relation model, comprising:

12. An inter-business system data blood margin generation apparatus, comprising:

13. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the electronic device to implement the data lineage generation method according to any one of claims 1 to 11.

14. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method of traffic intersystem data consanguinity according to any one of claims 1 to 11.