CN115033639A - Method and related device for generating relation graph for data sharing among clusters - Google Patents
Method and related device for generating relation graph for data sharing among clusters Download PDFInfo
- Publication number
- CN115033639A CN115033639A CN202210391146.8A CN202210391146A CN115033639A CN 115033639 A CN115033639 A CN 115033639A CN 202210391146 A CN202210391146 A CN 202210391146A CN 115033639 A CN115033639 A CN 115033639A
- Authority
- CN
- China
- Prior art keywords
- database cluster
- clusters
- data sharing
- cluster
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000004458 analytical method Methods 0.000 claims description 21
- 238000012216 screening Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 29
- 238000012423 maintenance Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000013523 data management Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/42—Syntactic analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method and a related device for generating a relation graph of data sharing among clusters, wherein the method comprises the following steps: the method comprises the steps that after a link gateway receives an access request of a database cluster, access information is recorded, and operation log information stored by the link gateways corresponding to a plurality of database clusters is obtained on the basis of obtaining the operation log information; analyzing the running log information to obtain and store access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data; and generating a relation graph of data sharing among clusters by accessing the time, the source database cluster and the target database cluster. The method does not need to establish and maintain a relation graph of data sharing among clusters through manual operation, is simpler and more timely, and avoids errors and omissions; based on the method, the efficiency of data sharing among clusters is improved, and data security risks and data access abnormity are avoided.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and a related apparatus for generating a relationship graph for data sharing between clusters.
Background
When an enterprise such as a bank constructs a large data center, a plurality of corresponding database clusters for large-scale parallel analysis are usually established according to a plurality of different service lines. In the process of performing data processing on each service line based on the corresponding database cluster, it is usually necessary to access other database clusters to complete the data processing, that is, each cluster in the plurality of database clusters needs to acquire data of other clusters, which is called inter-cluster data sharing.
In the related art, in order to clarify the path of data sharing among clusters, a relational graph of data sharing among clusters needs to be manually created and maintained according to database cluster configuration files stored by linking gateways corresponding to a plurality of database clusters, so that each cluster can comprehensively know the path of data sharing among clusters based on the relational graph, and further the efficiency of data sharing among clusters is improved.
However, with the increase of the number of clusters, manual operations such as creation and maintenance of a relationship diagram for data sharing among clusters for a long time are tedious in manner, low in timeliness and easy to miss, and the maintenance cost of the relationship diagram is higher and higher, so that the efficiency of data sharing among clusters is low, and even data security risk, data access abnormality and the like exist.
Disclosure of Invention
In view of this, embodiments of the present application provide a method and a related apparatus for generating a relational graph for data sharing between clusters, which do not need to create and maintain the relational graph for data sharing between clusters through manual operations, are simpler and more timely, and avoid errors and omissions; therefore, the efficiency of data sharing among clusters is improved, and data security risks and data access abnormity are avoided.
In a first aspect, an embodiment of the present application provides a method for generating a relationship diagram for data sharing between clusters, where the method includes:
acquiring running log information stored by a linking gateway corresponding to a plurality of database clusters, wherein the running log information is obtained by recording access information after the linking gateway receives an access request of the database clusters;
analyzing the running log information, and acquiring and storing access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data;
and generating a relation graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster.
Optionally, the analyzing the running log information to obtain and store the access time, the source database cluster, and the target database cluster corresponding to the access information specifically includes:
and performing syntax analysis and semantic analysis on the running log information to obtain and store the access time, the source database cluster and the target database cluster.
Optionally, the method further includes:
analyzing the database cluster configuration file stored by the link gateway to obtain first metadata of the source database cluster and second metadata of the target database cluster;
generating a relationship graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster, specifically:
and generating a relation graph of data sharing among the clusters according to the access time, the source database cluster, the first metadata, the target database cluster and the second metadata.
Optionally, the method further includes:
acquiring a query request of a relation graph of data sharing among the clusters;
and controlling and displaying the data sharing relation graph among the clusters according to the query request.
Optionally, the query request carries a preset time period, and the controlling and displaying of the relationship graph of data sharing among clusters according to the query request includes:
screening the access time, the source database cluster and the target database cluster in the inter-cluster data sharing relation graph according to the preset time period to obtain the screened inter-cluster data sharing relation graph;
and controlling and displaying the screened data sharing relation graph among the clusters according to the query request.
Optionally, the method further includes:
counting the path attribute information from the source database cluster to the target database cluster in the relational graph of the inter-cluster data sharing;
and displaying the path attribute information in a relation graph of the inter-cluster data sharing.
Optionally, the path attribute information includes a path access number and/or a path load condition.
In a second aspect, an embodiment of the present application provides an apparatus for generating a relationship graph for data sharing between clusters, where the apparatus includes: the device comprises an acquisition unit, an analysis unit and a generation unit;
the acquisition unit is used for acquiring operation log information stored by a linking gateway corresponding to a plurality of database clusters, wherein the operation log information is obtained by recording access information after the linking gateway receives an access request of the database clusters;
the analysis unit is used for analyzing the running log information, and acquiring and storing access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data;
and the generating unit is used for generating a relation graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster.
Optionally, the parsing unit is specifically configured to:
and carrying out syntax analysis and semantic analysis on the running log information, and obtaining and storing the access time, the source database cluster and the target database cluster.
Optionally, the parsing unit is further configured to:
analyzing the database cluster configuration file stored by the link gateway to obtain first metadata of the source database cluster and second metadata of the target database cluster;
the generating unit is specifically configured to:
and generating a relation graph of data sharing among the clusters according to the access time, the source database cluster, the first metadata, the target database cluster and the second metadata.
Optionally, the obtaining unit is further configured to:
acquiring a query request of a relation graph of data sharing among the clusters;
the device further comprises: a control unit;
and the control unit is used for controlling and displaying the data sharing relation graph among the clusters according to the query request.
Optionally, the query request carries a preset time period, and the control unit is specifically configured to:
screening the access time, the source database cluster and the target database cluster in the inter-cluster data sharing relationship graph according to the preset time period to obtain the screened inter-cluster data sharing relationship graph;
and controlling and displaying the screened data sharing relation graph among the clusters according to the query request.
Optionally, the apparatus further comprises: a counting unit and a display unit;
the statistical unit is configured to count path attribute information from the source database cluster to the target database cluster in the inter-cluster data sharing relationship diagram;
and the display unit is used for displaying the path attribute information in the data sharing relation graph among the clusters.
Optionally, the path attribute information includes a number of path accesses and/or a path load condition.
In a third aspect, an embodiment of the present application provides a computer device, where the computer device includes a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to execute the method for generating the inter-cluster data sharing relationship diagram according to the first aspect, according to instructions in the program code.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing program code, where the program code is configured to, when executed by a processor, perform the method for generating a relationship graph of data sharing between clusters according to the first aspect.
Compared with the prior art, the method has the advantages that:
by adopting the technical scheme of the embodiment of the application, on the basis that the linked gateways record the access information after receiving the access request of the database cluster to obtain the operation log information, firstly, the operation log information stored by the linked gateways corresponding to a plurality of database clusters is obtained; then, analyzing the running log information to obtain and store access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data; and finally, generating a relation graph of data sharing among clusters through the access time, the source database cluster and the target database cluster. Therefore, in the method, the link gateway receives an access request of the database cluster, records access information to obtain running log information, automatically obtains the running log information stored by the link gateway, and automatically analyzes the running log information to obtain access time, a source database cluster and a target database cluster corresponding to the access information, so that a relational graph for sharing data among clusters can be automatically generated, the relational graph for sharing data among clusters does not need to be created and maintained through manual operation, and the method is simpler and more timely and avoids the occurrence of errors and omissions; based on the method, the efficiency of data sharing among clusters is improved, and data security risks and data access abnormity are avoided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the description below are only some embodiments described in the present application, and it is obvious for those skilled in the art that other drawings may be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of a system framework related to an application scenario in an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for generating a relationship diagram of data sharing between clusters according to an embodiment of the present application;
fig. 3 is a schematic diagram of a relationship diagram of data sharing between clusters according to an embodiment of the present application;
fig. 4 is a schematic diagram of a system for generating inter-cluster data sharing according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an apparatus for generating a relationship diagram of data sharing between clusters according to an embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
Currently, each cluster in a plurality of database clusters needs to acquire data of other clusters, which is called inter-cluster data sharing. Generally, in order to clarify the path of data sharing among clusters, a relational graph of data sharing among clusters needs to be created and maintained manually according to database cluster configuration files stored by linking gateways corresponding to a plurality of database clusters, so that each cluster can comprehensively know the path of data sharing among clusters based on the relational graph, and further the efficiency of data sharing among clusters is improved.
However, with the increase of the number of clusters, it is found that, due to the long-term manual operations such as creation and maintenance of the relationship diagram for data sharing among clusters, the manner is cumbersome, the timeliness is low, errors and omissions are easy to occur, the maintenance cost of the relationship diagram is higher and higher, the efficiency of data sharing among clusters is low, and even there are data security risks, data access abnormalities and the like.
In order to solve the problem, in the embodiment of the application, the operation log information stored by the link gateways corresponding to a plurality of database clusters is acquired on the basis that the link gateways record the access information after receiving the access request of the database clusters and obtain the operation log information; analyzing the running log information to obtain and store access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data; and generating a relation graph of data sharing among clusters by accessing the time, the source database cluster and the target database cluster. Therefore, in the method, the link gateway receives an access request of the database cluster, records access information to obtain running log information, automatically obtains the running log information stored by the link gateway, and automatically analyzes the running log information to obtain access time, a source database cluster and a target database cluster corresponding to the access information, so that a relational graph for sharing data among clusters can be automatically generated, the relational graph for sharing data among clusters does not need to be created and maintained through manual operation, and the method is simpler and more timely and avoids the occurrence of errors and omissions; based on the method, the efficiency of data sharing among clusters is improved, and data security risks and data access abnormity are avoided.
For example, one of the scenarios in the embodiment of the present application may be applied to the scenario shown in fig. 1. The scenario includes a plurality of database clusters 101, a linking-gateway 102, and a server 103, where the plurality of database clusters 101 includes a database cluster 1, a database cluster 2, … …, and a database cluster N, where N is a positive integer greater than or equal to 3. The server 103, the plurality of database clusters 101 and the linking gateway 102, use the method for generating a relationship diagram of data sharing between clusters provided in the embodiment of the present application, to generate a relationship diagram of data sharing between clusters.
First, in the application scenario described above, although the actions of the embodiment provided in the embodiment of the present application are described as being performed by the server 103; however, the embodiments of the present application are not limited in terms of executing subjects as long as the actions disclosed in the embodiments provided by the embodiments of the present application are executed.
Next, the above scenario is only one example of the scenario provided in the embodiment of the present application, and the embodiment of the present application is not limited to this scenario.
The following describes in detail a specific implementation manner of the method for generating a relationship diagram of data sharing between clusters and a related apparatus in the embodiment of the present application, with reference to the accompanying drawings.
Exemplary method
Referring to fig. 2, a schematic flowchart of a method for generating a relationship diagram of data sharing between clusters in an embodiment of the present application is shown. In this embodiment, the method may include, for example, the steps of:
step 201: the method comprises the steps of obtaining running log information stored by a link gateway corresponding to a plurality of database clusters, wherein the running log information is obtained by recording access information after the link gateway receives an access request of the database clusters.
Because the database cluster configuration files stored by the linking gateways corresponding to a plurality of database clusters are used in the related technology, the relational graph of data sharing among the clusters is manually created and maintained, and with the increase of the number of the clusters, the manual operations such as creation and maintenance of the relational graph of data sharing among the clusters and the like are performed for a long time, so that the method is complicated, the timeliness is low, mistakes and omissions are easy to occur, the maintenance cost of the relational graph is higher and higher, the efficiency of data sharing among the clusters is low, and even data security risks, abnormal data access and the like exist.
Therefore, in the embodiment of the application, in the process that each cluster in a plurality of database clusters needs to acquire data of other clusters to share data among the clusters, the database cluster needing data sends an access request to a linking gateway corresponding to the plurality of database clusters, after receiving the access request of the database clusters, the linking gateway queries whether a matched database cluster providing data exists in the stored database cluster configuration file, and if so, returns access information so that the database cluster needing data accesses the database cluster providing data through the access information to obtain the needed data.
Correspondingly, after receiving the access request of the database cluster, the linking gateway records the access information accessed to the database cluster providing data by the database cluster needing data at a certain time so as to obtain and store the running log information. On this basis, in order to automatically generate a relationship graph of data sharing between clusters, it is first necessary to automatically acquire the operation log information stored in the above-mentioned linking gateway.
The linking gateway corresponding to the plurality of database clusters can be one linking gateway or a plurality of linking gateways, and under the condition of a plurality of linking gateways, the database cluster configuration file stored by each linking gateway provides service for the plurality of database clusters through a load balancing strategy so as to meet the requirement of high availability of the cluster system.
Step 202: analyzing the running log information, and acquiring and storing access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster that requires data, and the target database cluster is a database cluster that provides data.
In this embodiment of the application, after obtaining the operation log information stored in the link gateway in step 201, the operation log information needs to be analyzed to obtain access time corresponding to the access information, a database cluster that needs data, and a database cluster that provides data, where the database cluster that needs data is used as a source database cluster, and the database cluster that provides data is used as a target database cluster. In addition, the access time, source database cluster, and target database cluster from parsing the log information need to be stored for later use.
In step 202, parsing the operation log information actually includes performing syntax parsing on the operation log information at a syntax level and performing semantic parsing on a semantic level, and fully analyzing a syntax structure of the operation log information and semantic information expressed by the operation log information, so as to parse access time, a source database cluster, and a target database cluster corresponding to the access information. Therefore, in an optional implementation manner of this embodiment of the present application, the step 202 may specifically be, for example: and carrying out syntax analysis and semantic analysis on the running log information, and obtaining and storing the access time, the source database cluster and the target database cluster.
Step 203: and generating a relation graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster.
In the embodiment of the present application, after the access time, the source database cluster, and the target database cluster are obtained through analysis in step 202, the association between the access time, the source database cluster, and the target database cluster expresses a relationship of data sharing among the clusters; based on the method, a relational graph for sharing data among clusters can be generated by accessing time, a source database cluster and a target database cluster.
In addition, in the embodiment of the present application, in order to generate the relational graph for data sharing between clusters, more cluster information about the source database cluster and the target database cluster can be provided, that is, metadata of the source database cluster and the target database cluster, such as a database cluster name and/or a database cluster address, etc., can be known through the relational graph for data sharing between clusters; the database cluster configuration file stored by the linking gateway may also be parsed to obtain metadata of the source database cluster as first metadata and metadata of the target database cluster as second metadata. Based on the method, a relational graph of data sharing among clusters is generated by combining the first metadata and the second metadata on the basis of the access time, the source database cluster and the target database cluster. The inter-cluster data sharing relationship graph represents first metadata of a source database cluster and second metadata of a target database cluster.
That is, in an optional implementation manner of this embodiment of the present application, before the step 203, the method may further include, for example, step a: analyzing the database cluster configuration file stored by the link gateway to obtain first metadata of the source database cluster and second metadata of the target database cluster; correspondingly, the step 203 may specifically be, for example: and generating a relation graph of data sharing among the clusters according to the access time, the source database cluster, the first metadata, the target database cluster and the second metadata.
In addition, in this embodiment of the application, after the relationship diagram for data sharing among clusters is generated in step 203, the user may also query the generated relationship diagram for data sharing among clusters, and after the query request for the relationship diagram for data sharing among clusters is obtained, the generated relationship diagram for data sharing among clusters is controlled to be displayed in response to the query request. Therefore, in an optional implementation manner of this embodiment of the present application, after the step 203, the method may further include, for example, the following steps B to C:
and B: acquiring a query request of a relation graph of data sharing among the clusters;
and C: and controlling and displaying the data sharing relation graph among the clusters according to the query request.
For the query request of the obtained relational graph of the inter-cluster data sharing, it is further required to determine whether the query request carries a preset time period, if so, it indicates that the query request is not a full query request, and it is required to screen the access time, the source database cluster and the target database cluster in the generated relational graph of the inter-cluster data sharing according to the preset time period carried by the query request, and only the access time, the source database cluster and the target database cluster within the preset time period are reserved to obtain the screened relational graph of the inter-cluster data sharing. And on the basis, responding to the query request, and controlling and displaying the screened inter-cluster data sharing relation graph. Therefore, in an optional implementation manner of the embodiment of the present application, the query request carries a preset time period, and the step C may include, for example, the following step C1-step C2:
step C1: screening the access time, the source database cluster and the target database cluster in the inter-cluster data sharing relation graph according to the preset time period to obtain the screened inter-cluster data sharing relation graph;
step C2: and controlling and displaying the screened data sharing relation graph among the clusters according to the query request.
Correspondingly, under the condition that the query request does not carry the preset time period, if the default query request is a full-volume query request, the generated relational graph for data sharing between the clusters needs to be directly controlled and displayed in response to the query request. Therefore, in an optional implementation manner of the embodiment of the present application, the query request does not carry the preset time period, and the step C may specifically be: and directly controlling and displaying the data sharing relation graph among the clusters according to the query request.
In addition, in the embodiment of the present application, in order to show the path attribute information from the source database cluster to the target database cluster on the generated inter-cluster data sharing relationship diagram, for example, the number of path accesses and/or the path load condition, etc.; the path attribute information from the source database cluster to the target database cluster in the relational graph of the inter-cluster data sharing can be counted, and the path attribute information is displayed on the generated relational graph of the inter-cluster data sharing, so that when a user queries the relational graph of the inter-cluster data sharing, the path attribute information from the source database cluster to the target database cluster can be known. Therefore, in an optional implementation manner of this embodiment of the present application, after the step 203, the method may further include, for example, the following steps D to E:
step D: and counting the path attribute information from the source database cluster to the target database cluster in the data sharing relationship graph among the clusters.
Step E: and displaying the path attribute information in a relation graph of the inter-cluster data sharing.
The path attribute information from the source database cluster to the target database cluster may include, for example, the number of path accesses and/or the path load condition. For example, refer to fig. 3, which is a schematic diagram of a relationship diagram of data sharing between clusters according to an embodiment of the present application; and displaying the path access times from the source database cluster to the target database cluster in the data sharing relation graph among the clusters.
Referring to fig. 4, this figure is a schematic diagram of a system for generating inter-cluster data sharing according to an embodiment of the present application. The system for generating the inter-cluster data sharing comprises a log acquisition subsystem, a log analysis subsystem, a generation subsystem, a data management subsystem and a query subsystem. The system comprises a log acquisition subsystem, a log analysis subsystem and a database cluster management subsystem, wherein the log acquisition subsystem is used for acquiring running log information stored by a linking gateway corresponding to a plurality of database clusters and sending the running log information to the log analysis subsystem, and the running log information is obtained by recording access information after the linking gateway receives an access request of the database clusters; the log analysis subsystem is used for analyzing the running log information, obtaining access time, a source database cluster and a target database cluster corresponding to the access information, and sending the access time, the source database cluster and the target database cluster to the generation subsystem and the data management subsystem; the generating subsystem is used for generating a relational graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster and sending the relational graph to the data management subsystem; the data management subsystem is used for storing the access time, the source database cluster, the target database cluster and a relational graph of data sharing among the clusters; and the query subsystem is used for querying the relationship graph of data sharing among the clusters.
According to various implementation manners provided by the embodiment, on the basis that the linked gateways record access information after receiving access requests of the database clusters to obtain the running log information, firstly, the running log information stored by the linked gateways corresponding to a plurality of database clusters is obtained; then, analyzing the running log information to obtain and store access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data; and finally, generating a relation graph of data sharing among clusters by accessing the time, the source database cluster and the target database cluster. Therefore, in the method, the link gateway receives an access request of the database cluster, records the access information to obtain the running log information, automatically obtains the running log information stored by the link gateway, and automatically analyzes the running log information to obtain the access time corresponding to the access information, the source database cluster and the target database cluster, so that a relational graph for data sharing among clusters can be automatically generated, the relational graph for data sharing among clusters does not need to be created and maintained through manual operation, and the method is simpler and more timely and avoids errors and omissions; based on the method, the efficiency of data sharing among clusters is improved, and data security risks and data access abnormity are avoided.
Exemplary devices
Referring to fig. 5, a schematic structural diagram of an apparatus for generating a relationship diagram of data sharing between clusters in the embodiment of the present application is shown. In this embodiment, the apparatus may specifically include: an acquisition unit 501, an analysis unit 502 and a generation unit 503;
the acquiring unit 501 is configured to acquire operation log information stored in a linking gateway corresponding to a plurality of database clusters, where the operation log information is obtained by recording access information after the linking gateway receives an access request of the database cluster;
the analyzing unit 502 is configured to analyze the operation log information, obtain and store access time, a source database cluster, and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data;
the generating unit 503 is configured to generate a relationship graph of data sharing among clusters according to the access time, the source database cluster, and the target database cluster.
In an optional implementation manner of the embodiment of the present application, the parsing unit 502 is specifically configured to:
and carrying out syntax analysis and semantic analysis on the running log information, and obtaining and storing the access time, the source database cluster and the target database cluster.
In an optional implementation manner of the embodiment of the present application, the parsing unit 502 is further configured to:
analyzing the database cluster configuration file stored by the link gateway to obtain first metadata of the source database cluster and second metadata of the target database cluster;
the generating unit 503 is specifically configured to:
and generating a relation graph of data sharing among the clusters according to the access time, the source database cluster, the first metadata, the target database cluster and the second metadata.
In an optional implementation manner of the embodiment of the present application, the obtaining unit 501 is further configured to:
acquiring a query request of a relation graph of data sharing among the clusters;
the device further comprises: a control unit;
and the control unit is used for controlling and displaying the data sharing relation graph among the clusters according to the query request.
In an optional implementation manner of the embodiment of the present application, the query request carries a preset time period, and the control unit is specifically configured to:
screening the access time, the source database cluster and the target database cluster in the inter-cluster data sharing relation graph according to the preset time period to obtain the screened inter-cluster data sharing relation graph;
and controlling and displaying the screened data sharing relation graph among the clusters according to the query request.
In an optional implementation manner of the embodiment of the present application, the apparatus further includes: a statistical unit and a display unit;
the statistical unit is configured to count path attribute information from the source database cluster to the target database cluster in the inter-cluster data sharing relationship graph;
and the display unit is used for displaying the path attribute information in the data sharing relationship graph among the clusters.
In an optional implementation manner of the embodiment of the present application, the path attribute information includes a path access number and/or a path load condition.
According to various implementation manners provided by the embodiment, on the basis that the linked gateways record access information after receiving access requests of the database clusters to obtain the running log information, firstly, the running log information stored by the linked gateways corresponding to a plurality of database clusters is obtained; then, analyzing the running log information to obtain and store access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data; and finally, generating a relation graph of data sharing among clusters by accessing the time, the source database cluster and the target database cluster. Therefore, in the method, the link gateway receives an access request of the database cluster, records access information to obtain running log information, automatically obtains the running log information stored by the link gateway, and automatically analyzes the running log information to obtain access time, a source database cluster and a target database cluster corresponding to the access information, so that a relational graph for sharing data among clusters can be automatically generated, the relational graph for sharing data among clusters does not need to be created and maintained through manual operation, and the method is simpler and more timely and avoids the occurrence of errors and omissions; based on the method, the efficiency of data sharing among clusters is improved, and data security risks and data access abnormity are avoided.
In addition, an embodiment of the present application further provides a computer device, where the computer device includes a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to execute the method for generating the inter-cluster data sharing relationship diagram according to the instruction in the program code.
An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium is configured to store program code, and when the program code is executed by a processor, the computer-readable storage medium is configured to perform the method for generating a relationship graph of data sharing between clusters according to the foregoing method embodiment.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing is illustrative of the preferred embodiments of the present application and is not to be construed as limiting the present application in any way. Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application. Those skilled in the art can now make numerous possible variations and modifications to the disclosed embodiments, or modify equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the claimed embodiments. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present application are within the scope of the protection of the technical solution of the present application, unless the technical essence of the present application deviates from the content of the technical solution of the present application.
Claims (10)
1. A method for generating a relationship graph of data sharing among clusters, comprising:
acquiring running log information stored by a linking gateway corresponding to a plurality of database clusters, wherein the running log information is obtained by recording access information after the linking gateway receives an access request of the database clusters;
analyzing the running log information, and acquiring and storing access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data;
and generating a relation graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster.
2. The method according to claim 1, wherein the analyzing the running log information to obtain and store the access time, the source database cluster, and the target database cluster corresponding to the access information comprises:
and carrying out syntax analysis and semantic analysis on the running log information, and obtaining and storing the access time, the source database cluster and the target database cluster.
3. The method of claim 1, further comprising:
analyzing the database cluster configuration file stored by the link gateway to obtain first metadata of the source database cluster and second metadata of the target database cluster;
generating a relationship graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster, specifically:
and generating a relation graph of data sharing among the clusters according to the access time, the source database cluster, the first metadata, the target database cluster and the second metadata.
4. The method of claim 1, further comprising:
acquiring a query request of a relation graph of data sharing among the clusters;
and controlling and displaying the data sharing relation graph among the clusters according to the query request.
5. The method according to claim 4, wherein the query request carries a preset time period, and the controlling and displaying of the relationship graph of data sharing among clusters according to the query request comprises:
screening the access time, the source database cluster and the target database cluster in the inter-cluster data sharing relation graph according to the preset time period to obtain the screened inter-cluster data sharing relation graph;
and controlling and displaying the screened data sharing relation graph among the clusters according to the query request.
6. The method according to any one of claims 1-5, further comprising:
counting the path attribute information from the source database cluster to the target database cluster in the relational graph of the inter-cluster data sharing;
and displaying the path attribute information in a relation graph of the inter-cluster data sharing.
7. The method according to claim 6, wherein the path attribute information comprises a path access number and/or a path load condition.
8. An apparatus for generating a relationship graph for data sharing between clusters, comprising: the device comprises an acquisition unit, an analysis unit and a generation unit;
the acquisition unit is used for acquiring operation log information stored by a linking gateway corresponding to a plurality of database clusters, wherein the operation log information is obtained by recording access information after the linking gateway receives an access request of the database clusters;
the analysis unit is used for analyzing the running log information, and acquiring and storing access time, a source database cluster and a target database cluster corresponding to the access information; the source database cluster is a database cluster needing data, and the target database cluster is a database cluster providing data;
and the generating unit is used for generating a relation graph of data sharing among clusters according to the access time, the source database cluster and the target database cluster.
9. A computer device, comprising a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to execute the method of generating a relationship graph for data sharing between clusters according to any one of claims 1 to 7 according to instructions in the program code.
10. A computer-readable storage medium for storing program code, which when executed by a processor, is configured to perform the method of generating a relationship graph for inter-cluster data sharing of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210391146.8A CN115033639A (en) | 2022-04-14 | 2022-04-14 | Method and related device for generating relation graph for data sharing among clusters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210391146.8A CN115033639A (en) | 2022-04-14 | 2022-04-14 | Method and related device for generating relation graph for data sharing among clusters |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115033639A true CN115033639A (en) | 2022-09-09 |
Family
ID=83118994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210391146.8A Pending CN115033639A (en) | 2022-04-14 | 2022-04-14 | Method and related device for generating relation graph for data sharing among clusters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115033639A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115269277A (en) * | 2022-09-27 | 2022-11-01 | 山东恒辉软件有限公司 | Intelligent laboratory data collaborative comprehensive management system |
| CN118364033A (en) * | 2024-06-19 | 2024-07-19 | 天津南大通用数据技术股份有限公司 | Data processing method and device among database clusters and related equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102457816A (en) * | 2010-10-21 | 2012-05-16 | 中兴通讯股份有限公司 | Method for managing cluster information, cluster access gateway and corresponding cluster system |
| CN110233875A (en) * | 2019-05-10 | 2019-09-13 | 厦门网宿有限公司 | A kind of data processing method, object storage gateway pond and operation system |
| CN110955662A (en) * | 2019-11-29 | 2020-04-03 | 车智互联（北京）科技有限公司 | Method, computing device and storage medium for maintaining data table association relation |
| CN111737741A (en) * | 2020-06-19 | 2020-10-02 | 中国工商银行股份有限公司 | Distributed database cluster access method and intermediate service layer |
| US20200372026A1 (en) * | 2019-05-20 | 2020-11-26 | Looker Data Sciences, Inc. | Retroreflective clustered join graph generation for relational database queries |
| WO2021189953A1 (en) * | 2020-10-12 | 2021-09-30 | 平安科技（深圳）有限公司 | Log data processing method and apparatus, computer device, and storage medium |
| CN113536054A (en) * | 2021-07-15 | 2021-10-22 | 中国工商银行股份有限公司 | Business monitoring method, device, equipment and storage medium based on graph data |
| CN114143316A (en) * | 2021-11-30 | 2022-03-04 | 招商局金融科技有限公司 | Multi-tenant network communication method and device, container node and storage medium |
-
2022
- 2022-04-14 CN CN202210391146.8A patent/CN115033639A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102457816A (en) * | 2010-10-21 | 2012-05-16 | 中兴通讯股份有限公司 | Method for managing cluster information, cluster access gateway and corresponding cluster system |
| CN110233875A (en) * | 2019-05-10 | 2019-09-13 | 厦门网宿有限公司 | A kind of data processing method, object storage gateway pond and operation system |
| US20200372026A1 (en) * | 2019-05-20 | 2020-11-26 | Looker Data Sciences, Inc. | Retroreflective clustered join graph generation for relational database queries |
| CN110955662A (en) * | 2019-11-29 | 2020-04-03 | 车智互联（北京）科技有限公司 | Method, computing device and storage medium for maintaining data table association relation |
| CN111737741A (en) * | 2020-06-19 | 2020-10-02 | 中国工商银行股份有限公司 | Distributed database cluster access method and intermediate service layer |
| WO2021189953A1 (en) * | 2020-10-12 | 2021-09-30 | 平安科技（深圳）有限公司 | Log data processing method and apparatus, computer device, and storage medium |
| CN113536054A (en) * | 2021-07-15 | 2021-10-22 | 中国工商银行股份有限公司 | Business monitoring method, device, equipment and storage medium based on graph data |
| CN114143316A (en) * | 2021-11-30 | 2022-03-04 | 招商局金融科技有限公司 | Multi-tenant network communication method and device, container node and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115269277A (en) * | 2022-09-27 | 2022-11-01 | 山东恒辉软件有限公司 | Intelligent laboratory data collaborative comprehensive management system |
| CN115269277B (en) * | 2022-09-27 | 2022-12-27 | 山东恒辉软件有限公司 | Intelligent laboratory data collaborative comprehensive management system |
| CN118364033A (en) * | 2024-06-19 | 2024-07-19 | 天津南大通用数据技术股份有限公司 | Data processing method and device among database clusters and related equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11799728B2 (en) | Multistage device clustering | |
| CN111522922B (en) | Log information query method and device, storage medium and computer equipment | |
| US10853124B2 (en) | Managing user data in a multitenant deployment | |
| US11163731B1 (en) | Autobuild log anomaly detection methods and systems | |
| US11809457B2 (en) | Systems and methods for indexing and aggregating data records | |
| US9590880B2 (en) | Dynamic collection analysis and reporting of telemetry data | |
| US11507562B1 (en) | Associating data from different nodes of a distributed ledger system | |
| US10565172B2 (en) | Adjusting application of a set of data quality rules based on data analysis | |
| US20110191394A1 (en) | Method of processing log files in an information system, and log file processing system | |
| US11768776B1 (en) | Evicting data associated with a data intake and query system from a local storage | |
| CN115033639A (en) | Method and related device for generating relation graph for data sharing among clusters | |
| US11362912B2 (en) | Support ticket platform for improving network infrastructures | |
| AU2014400621B2 (en) | System and method for providing contextual analytics data | |
| US20190370138A1 (en) | Compliance testing through sandbox environments | |
| CA2710405C (en) | Data comparison system | |
| CN110990420A (en) | Data query method and device | |
| US11301475B1 (en) | Transmission handling of analytics query response | |
| KR20220027924A (en) | System, apparatuses, and methods of processing and managing web traffic data | |
| WO2022018554A1 (en) | Dynamically determining trust level of end-to-end link | |
| CN113010208B (en) | Version information generation method, device, equipment and storage medium | |
| CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| US11755453B1 (en) | Performing iterative entity discovery and instrumentation | |
| US20200026701A1 (en) | Dynamic visualization of application and infrastructure components with layers | |
| CN116186777A (en) | Audit method and device for MPP database | |
| JP2021506010A (en) | Methods and systems for tracking application activity data from remote devices and generating modified behavioral data structures for remote devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |