CN117827970A - Data synchronization method, device, equipment and storage medium of graph database - Google Patents
Data synchronization method, device, equipment and storage medium of graph database Download PDFInfo
- Publication number
- CN117827970A CN117827970A CN202311849657.0A CN202311849657A CN117827970A CN 117827970 A CN117827970 A CN 117827970A CN 202311849657 A CN202311849657 A CN 202311849657A CN 117827970 A CN117827970 A CN 117827970A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- label
- synchronization
- graph database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013515 script Methods 0.000 claims abstract description 18
- 230000008859 change Effects 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 13
- 238000001914 filtration Methods 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 15
- 238000012544 monitoring process Methods 0.000 claims description 15
- 230000001360 synchronised effect Effects 0.000 claims description 13
- 238000013461 design Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000007726 management method Methods 0.000 abstract description 6
- 238000006243 chemical reaction Methods 0.000 abstract description 3
- 238000013499 data model Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 description 6
- 101100317378 Mus musculus Wnt3 gene Proteins 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013508 migration Methods 0.000 description 3
- 230000005012 migration Effects 0.000 description 3
- 238000009434 installation Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 241000879887 Cyrtopleura costata Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data synchronization method, device and equipment of a graph database and a storage medium, and relates to the technical field of data modular transfer. The method comprises the following steps: collecting a database log in a source database, and sending the database log to a stream data processing server in a json message mode; configuring data synchronization modeling rules according to metadata of a relational database; synchronizing stock data according to rules; consuming json messages according to rules; and executing filtering operation on the json message, converting database data and data change information change contained in the json message into a script executable by the graph database, and transmitting the script executable by the graph database to the graph database for execution. The method is based on metadata management data model conversion synchronization, meets the synchronization requirement of converting objects and relations into edges and nodes of a graph library in a configurable mode, adapts to data quasi-real-time synchronization of a relational database to the graph database, and meets the requirements of multi-element heterogeneous scenes on data instantaneity and integrity of graph topology display.
Description
Technical Field
The present invention relates to the field of data modular migration technologies, and in particular, to a method, an apparatus, a device, and a storage medium for synchronizing data in a graph database.
Background
Under the large background of the national promotion enterprise cloud development strategy, the China telecom informatization enterprise provides new requirements for the query requirement and the query mode of massive data, and data synchronization among databases of different types needs to be realized so as to ensure the real-time performance and usability of the data. The heterogeneous database data synchronization tool has great significance for providing a multi-element heterogeneous query scheme for cloud network resources and a data sharing platform, and the data synchronization on the market at the present stage still has the defects in the technology, such as:
full table based data synchronization cannot support table-to-graph database node and edge model transformations for relational databases: the traditional database synchronization tool is mainly based on physical tables and attribute synchronization data, and can not well support quasi-real-time synchronization of partial data in the tables.
Synchronization of the nodes and edges of the relationship to the gallery cannot be done: the synchronization from the relational database to the gallery requires converting entity data and relational data into nodes and edges of the gallery, and marking the corresponding nodes according to service requirements to meet the service requirements. Existing synchronization tools are temporarily unable to support the above-described functions.
The synchronization efficiency in the mass data scene is not enough: the traditional data synchronization tool reflects high-efficiency data migration performance when migrating a small amount of data, when encountering data synchronization of cloud network resources, data sharing platform resources, mass data and complex scenes, the traditional data synchronization tool has the problems of corresponding untimely application and low data synchronization efficiency, and often cannot timely and efficiently complete the requirements of users.
Disclosure of Invention
The invention aims to: the method, the device, the equipment and the storage medium for synchronizing the data of the graph database are provided, the configurable data synchronization of cloud network resources and a data sharing platform based on metadata management is realized, the synchronization of the data of the relational database to the graph database is supported, and the problems in the prior art are effectively solved.
In a first aspect, a method for synchronizing data in a graph database is provided, including the following steps:
collecting a database log in a source database, and sending the database log to a stream data processing server in a json message mode;
the streaming data processing server receives the json message, and configures a data synchronization modeling rule based on metadata according to metadata of a relational database;
according to the data synchronization modeling rules, automatically or manually synchronizing stock data;
configuring an incremental synchronization task: setting a data source and a written library, and consuming the json message according to a configured data synchronous modeling rule; filtering the json message, converting database data and data change information changes contained in the json message into scripts executable by the graph database, and sending the scripts to the graph database for execution;
the configuration monitoring task monitors the json message pushing process and the increment synchronization process in real time, and the aim of the process is to monitor the whole data synchronization process, ensure the normal synchronization of the data, discover, locate and solve the possible problems in the synchronization process in time.
In a further embodiment of the first aspect, the metadata-based data synchronization modeling rule is configured according to metadata of the relational database, and the metadata-based data synchronization modeling rule includes a configuration gallery configuration table;
the gallery configuration table is used for storing the mapping relation between the meta model and the gallery label and the label attribute, and when the data is modeled, the data is imported into the gallery by taking the gallery configuration table as a reference, and the node label is given.
In a further embodiment of the first aspect, the node labels are classified into a large class label, a specification label, an attribute label, and a custom label, and the label name is set based on a meta model.
In a further embodiment of the first aspect, the generic label, specification label is used to describe a meta-model;
the attribute labels are names of corresponding node attribute dictionary values;
the custom tag is a business requirement or an alias which is convenient for distinguishing data;
the large class labels and the specification labels are default labels, and the attribute labels are selection labels; the selection tag is arranged to be at a specification level or an attribute level according to the granularity of the service demand data.
In a further embodiment of the first aspect, the node labels are further divided into a main label and a non-main label, and are used for modeling data from the PG source database to the gallery;
the main label is the thinnest layer label, one node only has one main label, and other labels are all non-main labels.
In a further embodiment of the first aspect, the gallery configuration table includes a node tag table, a tag hierarchy table, a node attribute table, a relationship tag table, a relationship attribute table;
the node label table is used for storing the mapping relation between the node main label and the meta model;
the label hierarchy table is used for storing the hierarchy relation between the node main label name and the specification label name and the large specification label name; the node attribute table is used for storing the mapping relation between the node attribute and the meta model and all label names of the node; the relation tag table is used for storing the mapping relation between the edge tag and the meta model; the relation attribute table is used for storing the mapping relation between the edge attribute and the meta-model.
In a second aspect of the present invention, a data synchronization device for a graph database is provided, where the data synchronization device for a graph database includes an acquisition unit, a rule design unit, a synchronization unit, an incremental synchronization task configuration unit, and a pushing unit.
The acquisition unit is used for acquiring a database log from the source end database and transmitting the database log to the stream data processing server in a json message mode;
the rule design unit is used for reading json information received by the stream data processing server and configuring a data synchronization modeling rule based on metadata according to metadata of the relational database;
the synchronization unit is used for automatically synchronizing stock data according to the data synchronization modeling rule;
the incremental synchronous task configuration unit is used for setting a data source and a written library, and consuming the json message according to a configuration rule; filtering the json message, converting database data and data change information changes contained in the json message into scripts executable by the graph database, and sending the scripts to the graph database for execution;
the pushing unit is used for configuring a monitoring task to monitor a json message pushing process and an increment synchronization process in real time.
In a third aspect, the present invention proposes an electronic device, characterized in that it comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a method for synchronizing data of a graph database as described in the first aspect.
In a fourth aspect of the invention, a computer readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium, which when run on an electronic device causes the electronic device to perform the method for synchronizing data of a graph database according to the first aspect.
Compared with the prior art, the invention has the following beneficial effects:
(1) Dynamic configuration based on metadata management: the data model conversion synchronization based on metadata management meets the synchronization requirement of converting the object and the relation into the edges and the nodes of the graph library in a configurable way, adapts to the data quasi-real-time synchronization of the relation database to the graph database, and meets the requirements of the multi-element heterogeneous scene on the data instantaneity and the integrity of the graph topology display.
(2) Simplifying the synchronization flow: the service logic is configured, and the long-distance network resource modeling rule is configured and adjusted to change, so that the migrated code program is not required to be repeatedly modified, the demand changing flow is simplified, and the demand changing cost is reduced.
(3) And the synchronization performance is improved: the data migration core algorithm based on the micro-service and container technology improves the data processing efficiency and greatly improves the data performance of heterogeneous data.
Drawings
Fig. 1 is a flowchart of a data synchronization method of the database of fig. 1 according to an embodiment of the present invention.
Fig. 2 is a flowchart of a method of synchronizing data to Neo4j based on kafka in embodiment 2 of the present invention.
Fig. 3 is a data flow relationship diagram of a method based on kafka sync data to Neo4j.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the invention may be practiced without one or more of these details. In other instances, well-known features have not been described in detail in order to avoid obscuring the invention.
Example 1:
the embodiment discloses a data synchronization method of a graph database, as shown in fig. 1, the content of which includes: log changes in the PG source database were monitored using wal Json, which was converted to incremental Json formatted message streams. And sending the Json format message stream to a message queue, and setting a topic to transmit the message. And converting the metadata configuration object and the relationship into a configuration rule of the gallery node and the edge. And configuring a data synchronization process, and converting the message stream in the required data Json format into a gallery storage SQL stream according to the configured conversion rule. And writing the SQL stream into the graph library to complete the data change of the graph database.
And finally, the complex business logic data change from the cloud network resource relational database to the graph data is completed through the steps.
Example 2:
referring to fig. 2, the present embodiment discloses a method for synchronizing data to Neo4j based on kafka, which comprises the following steps:
s1, monitoring and configuring a source database log: and installing wal a json log monitoring plug-in on the source database, and sending the database log to Kafka in a json message mode.
S2, kafka theme installation configuration: the Kafka topic receiving management database log messages are installed and configured.
S3, configuring a synchronization rule: and configuring metadata-based data synchronization modeling rules according to metadata of the relational database, wherein the rules comprise rules of converting an object into a node and converting a relationship into a node or an edge.
S4, initializing stock data: according to the data synchronization modeling rule, the stock data is automatically or manually synchronized (the data synchronization above is real-time data synchronization, the synchronization is data generating variation; the data in the source database is not synchronized in the data synchronization above before the program is run, and the stock data is said to be the data).
S5, incremental synchronous task configuration: setting a data source and a written library, consuming json messages in a Kafka theme according to a configured data synchronous modular transformation rule, filtering the messages, converting the changes of the objects (database data and data change information contained in the json messages) into scripts executable by a graph database, and sending the scripts to the graph database for execution.
S6, monitoring a message pushing and incremental data synchronization process: and (5) configuring a monitoring task monitoring message pushing and incremental synchronization process.
In the step S3, the synchronization rule may be configured according to a gallery configuration table, where the gallery configuration table is a mapping relationship table of a meta model, a gallery label and a label attribute, and when the data is modeled, the data is imported into the gallery and given node and edge label names based on the configuration table.
Node labels are divided into a large class label, a specification label, an attribute label and a custom label, and label names are set by taking a meta model as a basis. The large class label name and the specification label name are respectively the large class specification name and the specification name of the meta model; the attribute label is called a corresponding node attribute dictionary value name; custom labels are referred to as business requirements or aliases that facilitate distinguishing data. The large class labels and the specification labels are default labels, and the attribute labels are selection labels (according to the granularity of service requirement data to a specification level or an attribute level).
The node labels are divided into main labels and non-main labels and are used for converting the PG library into the drawing library modeling data. The main label is the thinnest layer label, one node only has one main label, and other labels are all non-main labels.
The gallery configuration table comprises a node label table, a label hierarchy table, a node attribute table, a relationship label table and a relationship attribute table.
The node label table is used for storing the mapping relation between the node main label and the meta model, and the following table 1 is shown below:
TABLE 1 node tag table
The label hierarchy table is used for storing the hierarchical relationship between the main label name of the node and the specification label name of the large class, and the table 2 is as follows:
table 2 tag hierarchy table
| Field name | Field type | Main key | Can be empty | Field description | Remarks |
| id | int8 | Is that | Main key | ||
| name | varchar(32) | Name of the name | |||
| code | varchar(32) | Encoding | |||
| parent_id | int8 | Is that | Parent node ID | ||
| parents | text | Is that | Parent node ID set | ||
| level | int4 | Node level (-1: main label; 0: superior label) | |||
| update_time | timestamp | Is that | Modification time | ||
| create_time | timestamp | Creation time | |||
| childs | text | Is that | Child node ID set |
The node attribute table is used for storing the mapping relation between the node attribute and the meta-model and all label names of the node, see table 3 below:
TABLE 3 node Property Table
The relationship tag table is used for storing the mapping relationship between the edge tag and the meta model, and the following table 4 is shown below:
table 4 relationship tag table
The relationship attribute table is used for storing the mapping relationship between the edge attribute and the meta-model, see table 5 below:
TABLE 5 relationship Property Table
| Field name | Field type | Main key | Can be empty | Field description | Remarks |
| id | int8 | Is that | Main key | ||
| relation_id | int8 | Relationship ID | |||
| name | varchar(32) | Attribute names | |||
| code | varchar(32) | Attribute encoding | |||
| type | int4 | Type (1 link 2 side 3 external key) | |||
| version | varchar(2) | Version of | |||
| is_valid | int4 | Whether or not to be effective | |||
| domain_id | varchar(2) | Service domain id | |||
| created_time | date | Creation time | |||
| updated_time | date | Is that | Update time | ||
| foreign_code | varchar(50) | Is that | Associated node attribute presentation field name | ||
| foreign_direction | varchar(50) | Is that | Node where added association attribute is located | ||
| data_table | varchar(25) | Data source list | |||
| data_column | varchar(25) | Data source field | |||
| data_node_table | varchar(32) | Is that | Source list of nodes at two ends of link |
The data synchronization method using the graph database disclosed in the above embodiments 1 and 2 can be applied to at least two fields:
1. and synchronizing the IP business data from a backbone network resource library and a space library of the cloud network resource and the data sharing platform to Neo4J.
2. And the IT cloud, the CT cloud and the angel wing cloud service are synchronized to Neo4J from a cloud resource library of the cloud network resource and the data sharing platform.
Example 3:
the present embodiment proposes a data synchronization apparatus 700 of a graph database, see fig. 3, and the data synchronization method of the graph database disclosed in the foregoing embodiment 1 and embodiment 2 can be implemented by using the data synchronization apparatus 700.
The data synchronization device 700 of the graph database comprises an acquisition unit 701, a rule design unit 702, a synchronization unit 703, an incremental synchronization task configuration unit 704 and a pushing unit 705. The acquisition unit 701 is configured to acquire a database log from a source database, and send the database log to a stream data processing server in a json message manner; the rule design unit 702 is configured to read json messages received by the streaming data processing server, and configure metadata-based data synchronization modeling rules according to metadata of the relational database; the synchronization unit 703 is configured to automatically synchronize stock data according to the data synchronization modeling rule; the incremental synchronization task configuration unit 704 is configured to set a data source and a written library, and consume the json message according to a configured rule; filtering the json message, converting database data and data change information changes contained in the json message into scripts executable by the graph database, and sending the scripts to the graph database for execution; the pushing unit 705 is configured to monitor the json message pushing process and the incremental synchronization process in real time by using the monitoring task.
Example 4:
the embodiment provides an electronic device, which comprises a processor, a memory, a communication interface and a communication bus. The processor, the memory and the communication interface perform communication with each other via a communication bus. The memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the data synchronization method of the graph database disclosed in the above embodiment. The method at least comprises the following steps:
s1, monitoring and configuring a source database log: and installing wal a json log monitoring plug-in on the source database, and sending the database log to Kafka in a json message mode.
S2, kafka theme installation configuration: the Kafka topic receiving management database log messages are installed and configured.
S3, configuring a synchronization rule: and configuring metadata-based data synchronization modeling rules according to metadata of the relational database, wherein the rules comprise rules of converting an object into a node and converting a relationship into a node or an edge.
S4, initializing stock data: according to the data synchronization modeling rule, the stock data is automatically or manually synchronized (the data synchronization above is real-time data synchronization, the synchronization is data generating variation; the data in the source database is not synchronized in the data synchronization above before the program is run, and the stock data is said to be the data).
S5, incremental synchronous task configuration: setting a data source and a written library, consuming json messages in a Kafka theme according to a configured data synchronous modular transformation rule, filtering the messages, converting the changes of the objects (database data and data change information contained in the json messages) into scripts executable by a graph database, and sending the scripts to the graph database for execution.
S6, monitoring a message pushing and incremental data synchronization process: and (5) configuring a monitoring task monitoring message pushing and incremental synchronization process.
The electronic device may also communicate with one or more external devices (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device, and/or with any device (e.g., router, modem, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface. And, the electronic device may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with an electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
Example 5:
the present embodiment proposes a computer readable storage medium, in which at least one executable instruction is stored, which when executed on an electronic device, causes the electronic device to perform the operations of the data synchronization method of the graph database according to the foregoing embodiment. More specific examples of the computer readable storage medium in the present disclosure may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
In summary, the above embodiments address the shortcomings of the conventional data synchronization tools, and this patent has the following advantages and improvements:
the data synchronization is realized by using and matching the data, so that the rapid configuration of the data synchronization from the relational database to the graph database is realized;
according to the data configuration, synchronization of the metadata-based data object is realized, the data synchronization range can be not an integral table, and can be partial data quasi-real-time synchronization based on the object and attribute conditions, so that the data volume of data synchronization is reduced, and the data synchronization efficiency is greatly improved.
The data synchronization is monitored comprehensively, and the running states of a source end database, a kafka theme, a target end graph database and an incremental synchronization process are monitored.
As described above, although the present invention has been shown and described with reference to certain preferred embodiments, it is not to be construed as limiting the invention itself. Various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method for synchronizing data of a graph database, comprising:
collecting a database log in a source database, and sending the database log to a stream data processing server in a json message mode;
the streaming data processing server receives the json message, and configures a data synchronization modeling rule based on metadata according to metadata of a relational database;
according to the data synchronization modeling rules, automatically or manually synchronizing stock data;
configuring an incremental synchronization task: setting a data source and a written library, and consuming the json message according to the data synchronous modular transformation rule;
and executing filtering operation on the json message, converting database data and data change information change contained in the json message into a script executable by the graph database, and transmitting the script executable by the graph database to the graph database for execution.
2. The method for synchronizing data in a graph database according to claim 1, wherein the step of converting the database data and the data change information change included in the json message into scripts executable in the graph database and transmitting the scripts to the graph database for execution, further comprises: the configuration monitoring task monitors the json message pushing process and the increment synchronization process in real time.
3. The method for synchronizing data of a graph database of claim 1, wherein: the metadata-based data synchronization modeling rules are configured according to the metadata of the relational database, and the metadata-based data synchronization modeling rules comprise a configuration chart library configuration table;
the gallery configuration table is used for storing the mapping relation between the meta model and the gallery label and the label attribute, and when the data is modeled, the data is imported into the gallery by taking the gallery configuration table as a reference, and the node label is given.
4. A method of synchronizing data of a graph database as claimed in claim 3, wherein: the node labels are divided into a large class label, a specification label, an attribute label and a custom label, and the label names are set by taking a meta model as a basis.
5. The method for synchronizing data of a graph database of claim 4, wherein: the large class labels and the specification labels are used for describing meta-models;
the attribute labels are names of corresponding node attribute dictionary values;
the custom tag is a business requirement or an alias which is convenient for distinguishing data;
the large class labels and the specification labels are default labels, and the attribute labels are selection labels; the selection tag is arranged to be at a specification level or an attribute level according to the granularity of the service demand data.
6. The method for synchronizing data of a graph database of claim 4, wherein: the node labels are also divided into main labels and non-main labels, and are used for modeling and converting data from a source database to a gallery;
the main label is the thinnest layer label, one node only has one main label, and other labels are all non-main labels.
7. A method of synchronizing data of a graph database as claimed in claim 3 wherein the graph library configuration table comprises a node label table, a label hierarchy table, a node attribute table, a relationship label table, a relationship attribute table;
the node label table is used for storing the mapping relation between the node main label and the meta model;
the label hierarchy table is used for storing the hierarchy relation between the node main label name and the specification label name and the large specification label name;
the node attribute table is used for storing the mapping relation between the node attribute and the meta model and all label names of the node;
the relation tag table is used for storing the mapping relation between the edge tag and the meta model;
the relation attribute table is used for storing the mapping relation between the edge attribute and the meta-model.
8. A data synchronization apparatus for a graph database, comprising:
the acquisition unit is used for acquiring a database log from the source end database and sending the database log to the stream data processing server in a json message mode;
the rule design unit is used for reading json messages received by the streaming data processing server and configuring data synchronization modeling rules based on metadata according to metadata of the relational database;
the synchronization unit is used for automatically synchronizing the stock data according to the data synchronization modeling rule;
the incremental synchronous task configuration unit is used for setting a data source and a written library, and consuming the json message according to a configuration rule; filtering the json message, converting database data and data change information changes contained in the json message into scripts executable by the graph database, and sending the scripts to the graph database for execution;
the pushing unit is used for configuring a monitoring task to monitor a json message pushing process and an increment synchronization process in real time.
9. An electronic device, the device comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a method of data synchronization of a graph database as claimed in any one of claims 1 to 7.
10. A computer readable storage medium, characterized in that at least one executable instruction is stored in the storage medium, which executable instruction, when run on an electronic device, causes the electronic device to perform the method of data synchronization of a graph database according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311849657.0A CN117827970A (en) | 2023-12-29 | 2023-12-29 | Data synchronization method, device, equipment and storage medium of graph database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311849657.0A CN117827970A (en) | 2023-12-29 | 2023-12-29 | Data synchronization method, device, equipment and storage medium of graph database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117827970A true CN117827970A (en) | 2024-04-05 |
Family
ID=90511011
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311849657.0A Pending CN117827970A (en) | 2023-12-29 | 2023-12-29 | Data synchronization method, device, equipment and storage medium of graph database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117827970A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118445364A (en) * | 2024-06-28 | 2024-08-06 | 中国人民解放军国防科技大学 | Distributed resource synchronization method, system and equipment |
-
2023
- 2023-12-29 CN CN202311849657.0A patent/CN117827970A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118445364A (en) * | 2024-06-28 | 2024-08-06 | 中国人民解放军国防科技大学 | Distributed resource synchronization method, system and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106611046B (en) | Spatial data storage processing middleware system based on big data technology | |
| CN106708993B (en) | Method for realizing space data storage processing middleware framework based on big data technology | |
| CA2533793C (en) | Automatic and dynamic provisioning of databases | |
| US7487191B2 (en) | Method and system for model-based replication of data | |
| RU2500023C2 (en) | Document synchronisation on protocol not using status information | |
| US7783591B2 (en) | Coordinated data conversion systems and methods | |
| US20080005197A1 (en) | Dynamic report mapping apparatus to physical data source when creating report definitions for information technology service management reporting for peruse of report definition transparency and reuse | |
| US8291310B2 (en) | Delta-saving in XML-based documents | |
| CN111061817A (en) | Adaptive service construction system, method and computer readable medium | |
| CN110019251A (en) | A kind of data processing system, method and apparatus | |
| CN117827970A (en) | Data synchronization method, device, equipment and storage medium of graph database | |
| US20080016086A1 (en) | Abstracted dynamic report definition generation for use within information technology infrastructure | |
| CN114661832B (en) | Multi-mode heterogeneous data storage method and system based on data quality | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN102355499A (en) | Cloud computing system | |
| CN113381870A (en) | Message processing method and device | |
| CN103853612A (en) | Method for reading data based on digital family content under distributed storage | |
| CN104052804A (en) | Method, device and cluster for sharing data streams between different task topologies | |
| US8250104B2 (en) | Life cycle based data coordination | |
| CN111782820B (en) | Knowledge graph creation method and device, readable storage medium and electronic equipment | |
| Zhang et al. | An optimal container update method for edge‐cloud collaboration | |
| WO2024103898A1 (en) | Database cluster management method and apparatus | |
| US20080059506A1 (en) | Method, system and schema for building a hierarchical model schema definition from a flat model definition | |
| US20240036910A1 (en) | Meta-level management system that aggregates information and functionalities of computational-resource management systems and that provides new management functionalities | |
| CN112559495A (en) | System for supporting multi-system label mapping to realize unified label management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |