CN112395363A - Data synchronization method, device, equipment and readable storage medium - Google Patents
Data synchronization method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN112395363A CN112395363A CN202011411156.0A CN202011411156A CN112395363A CN 112395363 A CN112395363 A CN 112395363A CN 202011411156 A CN202011411156 A CN 202011411156A CN 112395363 A CN112395363 A CN 112395363A
- Authority
- CN
- China
- Prior art keywords
- component
- search engine
- mirror image
- database
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000013507 mapping Methods 0.000 claims abstract description 89
- 238000012360 testing method Methods 0.000 claims abstract description 72
- 230000004044 response Effects 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 13
- 238000010586 diagram Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 13
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 238000012827 research and development Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 210000004916 vomit Anatomy 0.000 description 1
- 230000008673 vomiting Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
Abstract
The embodiment of the disclosure provides a data synchronization method, a device, equipment and a readable storage medium, wherein a basic mirror image is deployed on a server, a user defines a mapping rule file aiming at different back-end services, a backup component in the basic mirror image is compiled by using the mapping rule file to obtain a test mirror image, the test mirror image is started to obtain a target container of the back-end services, and the compiled backup component is used in the target container to perform data synchronization on a database component and a search engine component. By adopting the scheme, the data synchronization is carried out on the database component and the search engine component based on the standardized basic mirror, the cost is low, and the coverage rate and the efficiency of the unit test stage are improved to a great extent.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of data processing, and in particular, to a data synchronization method, device, equipment and readable storage medium.
Background
Generally, a back-end service can be brought online after being tested in a development and test stage. After the connection, the back-end service is visible to the external network users, namely after receiving the requests of the external network users, the back-end service processes the requests and returns the processing results to the external network users.
When a back-end service uses a database component and a search engine component, the data synchronization of the database component and the search engine component is needed after the back-end service is on-line. Common synchronization schemes include the Canal-based open source synchronization scheme and the heterogeneous system scheme.
Similarly, in the development and test stage, data synchronization is also required to be performed on the database component and the search engine component of the back-end service. However, the synchronization scheme after online cannot be used in the test phase.
Disclosure of Invention
The embodiment of the disclosure provides a data synchronization method, a data synchronization device and a readable storage medium, which achieve the purpose of performing data synchronization on a database component and a search engine component of a back-end service in a research and development test stage through a simple configuration file and a mapping code.
In a first aspect, an embodiment of the present disclosure provides a data synchronization method, including:
receiving a synchronization request, wherein the synchronization request is used for requesting data synchronization of a database component and a search engine component of a back-end service;
in response to the synchronization request, compiling a backup component in a basic mirror image by using a mapping rule file of the backend service to obtain a test mirror image, wherein the basic mirror image comprises the database component, the search engine component and the backup component, and the mapping rule file is used for defining a business rule of the backend service;
running the test mirror image to obtain a target container of the back-end service;
and performing data synchronization on the database component and the search engine component in the target container by using the compiled backup component.
In a second aspect, an embodiment of the present disclosure provides a data synchronization apparatus, including:
the device comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving a synchronization request which is used for requesting data synchronization of a database component and a search engine component of a back-end service;
a response unit, configured to respond to the synchronization request, compile a backup component in a basic mirror image using a mapping rule file of the backend service to obtain a test mirror image, where the basic mirror image includes the database component, the search engine component, and the backup component, and the mapping rule file is used to define a business rule of the backend service;
the running unit is used for running the test mirror image to obtain a target container of the back-end service;
and the synchronization unit is used for carrying out data synchronization on the database component and the search engine component in the target container by utilizing the compiled backup component.
In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data synchronization method as described above in the first aspect and various possible designs of the first aspect.
In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium having stored therein computer-executable instructions that, when executed by a processor, implement the data synchronization method as described in the first aspect above and in various possible designs of the first aspect.
In the data synchronization method, the device, the equipment and the readable storage medium provided by the embodiment of the disclosure, a base mirror image is deployed on a server, a user defines a mapping rule file for different backend services, the mapping rule file is utilized to compile backup components in the base mirror image to obtain a test mirror image, the test mirror image is started to obtain a target container of the backend services, and the compiled backup components are utilized in the target container to perform data synchronization on a database component and a search engine component. By adopting the scheme, the data synchronization is carried out on the database component and the search engine component based on the standardized basic mirror, the cost is low, and the coverage rate and the efficiency of the unit test stage are improved to a great extent.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic network architecture of a data synchronization method provided by an embodiment of the present disclosure;
FIG. 2 is a flow chart of a data synchronization method provided by an embodiment of the present disclosure;
FIG. 3 is a schematic diagram illustrating a generation process of a target container in the data synchronization method provided by the embodiment of the present disclosure;
fig. 4 is a schematic process diagram of data synchronization in the data synchronization method provided by the embodiment of the present disclosure;
fig. 5 is a block diagram of a data synchronization apparatus according to an embodiment of the disclosure;
fig. 6 is a block diagram of another data synchronization apparatus provided in the embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
Some current back-end services typically use relational database management system components (e.g., MySQL) to store data, and full-text search Engine (ES) components to perform complex combined queries and full-text searches. After the back-end service is on-line, data synchronization needs to be performed on the MySQL component and the ES component. Common synchronization schemes include the Canal-based open source synchronization scheme and the heterogeneous system scheme. In the open source synchronization scheme based on the Canal, a deployment Canal service (server) end is disguised as a MySQL standby machine, and the data synchronization of the MySQL component and the ES component is realized through a customized Canal client (client) end.
In the heterogeneous system scheme, a mapping rule is formulated, and the mapping rule is scheduled through functional as a Service (FaaS), so that data synchronization of the MySQL component and the ES component is flexibly realized.
Generally, a backend service needs to be tested by unit test, joint debugging test and the like before going online, and can be on-line only after the test is passed. The back-end service is on-line, that is, the back-end service faces the external network user and is visible to the external network user. The unit test is a test performed by a developer in a research and development stage on a back-end service which is not on-line. In the unit testing stage, data synchronization is also needed to be carried out on the MySQL component and the ES component.
In the unit testing stage, the MySQL component and the ES component need an isolated environment, and are mainly realized by a container technology at present. If an open-source-canal-based synchronization scheme is used in the container, a canal server needs to be deployed for each service, a customized canal client needs to be developed, and complicated configuration needs to be performed, which is high in cost. For example, two backend services are respectively used for commodity search and user search, and two sets of MySQL components, two sets of canal servers and canal clients, and two sets of ES components need to be deployed. One set of MySQL components is used for storing commodity data, and the other set of MySQL components is used for storing user data. One set of the canal server and the canal client is used for binding business logic for commodity searching, and the other set of the canal server and the canal client is used for binding business logic for user searching. One set of ES components is used for searching by a user, and the other set of ES components is used for searching for commodities. However, the backend services are voluminous. It can be seen from this that: given the use of the canal-based open source scheme, customization costs are high and deployment costs are high.
If a heterogeneous system solution is used, the binary file of the MySQL component needs to be synchronized to the message queue and then written to the ES component through the mapping rules. The process relies on too many intermediate components and is complex.
According to the above, it can be seen that: for unit testing, there is no method for synchronizing the database component and the search engine component at present, that is, the existing unit testing cannot cover the data synchronization of the database component and the search engine component.
The embodiment of the disclosure provides a data synchronization method, a data synchronization device and a readable storage medium, wherein a test mirror image is manufactured for any one back-end service based on a standardized basic mirror image, and then data synchronization is performed on a database component and a search engine component of the back-end service, so that the cost is low, and the coverage rate and the efficiency of a unit test stage are improved to a great extent.
Fig. 1 is a schematic network architecture of a data synchronization method provided in an embodiment of the present disclosure. Referring to fig. 1, the network architecture includes: the electronic device 1 and the server 2, and network connection is established between the electronic device 1 and the server 2. The server 2 has a local environment and a container environment, and the container environment is pre-deployed with a base image which is made by using an open source based database component, a search engine component and a backup component.
The container environment is used for manufacturing a test image and starting the test image. In the local environment for deploying some mapping rules, configuration files, etc. In the data synchronization process, the server 2 compiles the basic mirror image by using the mapping file in the local environment to obtain the test mirror image. A test image may be understood as a program that is not started and not running. After the test image is started, data synchronization in the database component and the search engine component is performed.
In fig. 1, the electronic device 1 is a desktop electronic device such as a television, a computer, etc., or a mobile electronic device such as a mobile phone, a tablet computer, a notebook computer, etc. The server is an independently arranged server or a server cluster formed by a plurality of servers, and the embodiment of the disclosure is not limited.
Fig. 2 is a flowchart of a data synchronization method provided by an embodiment of the present disclosure. This embodiment is explained from the perspective of a server. The embodiment comprises the following steps:
101. a synchronization request is received.
Wherein the synchronization request is used for requesting data synchronization of a database component and a search engine component of the back-end service.
In the embodiment of the present disclosure, the backend service is a backend service having a database component and a search engine component, where the database component is, for example, a relational database management system, such as MySQL, or a distributed database Hbase. The search engine component is, for example, an electronic search Engine (ES), etc., and the embodiments of the present disclosure are not limited thereto.
In the process of research and development testing of the back-end service, a research and development worker sends a synchronization request to a server by using a computer and the like so as to trigger the server to perform data synchronization on a database component and a search engine component of the back-end service. In addition, research and development personnel can also compile some test cases, and the data synchronization is carried out on the database component and the search engine component of the back-end service based on the test case trigger server.
Although step 101 is to trigger synchronization with the front-end, i.e., the electronic device, the embodiments of the present disclosure are not limited thereto, for example, a simple start command is set on the server, and the server can be triggered to synchronize the database component and the search engine component based on the start command.
102. And responding to the synchronous request, and compiling the backup component in the basic mirror image by using the mapping rule file of the back-end service to obtain the test mirror image.
The basic mirror image comprises the database component, the search engine component and the backup component, and the mapping rule file is used for defining business rules of the back-end service.
Illustratively, the mapping rule file is a file customized according to a service, and may also be referred to as mapping. Different backend services correspond to different mapping rule files. For example, if the backend service is commodity search, the mapping rule file is defined according to the commodity search rule; if the back-end service is searched for the user, the mapping rule file is defined according to the user searching rule. It can be seen from this that: the basic mirror image is one, namely, the database component, the search engine component and the canal component deployed on the server are one, and target test containers of different back-end services can be obtained based on the basic mirror image and different mapping rule files, configuration files and the like.
The database component typically backs up the component because the database component may be asked, the purpose of the backup component being to ensure data reliability. In the embodiment of the present disclosure, different backend services have a database component that does not vomit, but different database components correspond to the same backup component, and the backup component is compiled using different mapping rule files, so that a test mirror image of each backend service can be obtained. After compiling, the backup component is disguised as a standby machine of the database component, data synchronization is carried out on the backup component and the database component in real time, therefore, the latest binary file can be obtained, and the data contained in the binary file can be analyzed into the data in the search engine component by analyzing the binary file according to the mapping rule subsequently.
In the embodiment of the present disclosure, the backup component is, for example, an open source data synchronization tool canal component, and may also be other components that can be used as a backup of the database component. The open source data synchronization tool canal component typically includes two parts: canal server and canal client. In this step, the server compiles the canal client by using the mapping rule file, so that the business logic is written into the canal client.
In the embodiment of the disclosure, the database components are initialized based on different first configuration files, so that the database components of different back-end services can be obtained; similarly, the search engine components are initialized based on different first configuration files, and the search engine components with different back-end services can be obtained.
103. And running the test mirror image to obtain a target container of the back-end service.
The test image in step 102 may be understood as a program that is not started and is not running. And running the test mirror image, distributing resources for the test mirror image, and starting the test mirror image. That is, the basic mirror image is compiled, and the target container of the unit test can be obtained by running the test mirror image after the compilation is completed.
104. And performing data synchronization on the database component and the search engine component in the target container by using the compiled backup component.
Illustratively, the server automatically parses the log, i.e. the binlog file, of the database component in the target container by using a compiled open source data synchronization tool, i.e. the canal component, and writes the data in the binlog file into the search engine component according to the user-defined business rules in the mapping rule file. Taking the database component as the MySQL component and the search engine component as the elastic search component as an example, after the test mirror image is started, the server can synchronize the data in the table specified in the MySQL component into the document indicated by the index specified by the elastic search component according to the compiled canal component.
In the data synchronization method provided by the embodiment of the disclosure, a base mirror image is deployed on a server, a user defines a mapping rule file for different backend services, the mapping rule file is utilized to compile backup components in the base mirror image to obtain a test mirror image, the test mirror image is started to obtain a target container of the backend services, and the compiled backup components are utilized in the target container to perform data synchronization on a database component and a search engine component. By adopting the scheme, the data synchronization is carried out on the database component and the search engine component based on the standardized basic mirror, the cost is low, and the coverage rate and the efficiency of the unit test stage are improved to a great extent.
In the above embodiment, the mapping rule file includes at least one mapping rule, and different mapping rules in the at least one mapping rule include different mapping relationships between database identifiers, table identifiers, target types, and indexes.
Illustratively, one mapping rule is the corresponding relationship between the database identifier, the table identifier, the target type and the index. The database identifier is used for indicating a database corresponding to the user read-write operation, the table identifier is used for indicating a table in the database to which the user read-write operation is directed, and the target type is used for indicating a type of structured data, such as a JS Object Notation (JSON). Briefly, the mapping rules are used to instruct the server to parse the data contained in the target table (i.e., the table indicated by the table identification) within the database indicated by the database identification into the structural data indicated by the target type, and write the structural data into the document indicated by the index within the search engine component.
By adopting the scheme, the mapping rule file comprises all business logics for synchronizing the database component and the search engine component, different mapping rule files are customized aiming at different back-end services, and the process of unit testing is simplified.
Fig. 3 is a schematic diagram of a generation process of a target container in the data synchronization method provided by the embodiment of the disclosure. Referring to fig. 3, the server has a container environment and a local environment. The container environment is provided with basic services, the basic services comprise a database component, a search engine component and a backup component, and the backup component comprises a server (canal server) and a client (canal client). The basic mirror image is manufactured by using an open source mirror image based on an open source before the server responds to a synchronization request from the electronic equipment and compiles the basic mirror image by using the mapping rule file to obtain the test mirror image.
Illustratively, the server obtains an open source image containing the database components, and installs the search engine components and the backup components on the open source image, thereby obtaining a base image. The database component, the search engine component, and the backup component in the base image are not initialized or compiled. After the server makes the basic mirror image, the server can make a test mirror image aiming at any back-end service by using the basic mirror image. Therefore, the database component, the backup component and the search engine component are not required to be respectively deployed for each back-end service, and the test mirror images can be made for different back-end services by only utilizing one basic mirror image, so that the components depended by the back-end services are simplified.
Referring to fig. 3 again, in the process of manufacturing a test image for a certain backend service, the server obtains a mapping rule file (mapping. The mapping rule file is a file customized for the back-end service and comprises a plurality of mapping rules, and the mapping rules are used for indicating how to synchronize the data in the database component to the search engine component. For example, one mapping rule is: table1-insert → es.index1, which means that the data inserted in the database component table1 is written into the document with index1 in the search engine component.
After the mapping rule file is obtained, the server compiles the canal client in the backup component by using the mapping rule file to obtain the compiled canal client.
In addition, the server needs to initialize the database components and the search engine components in the base image. In one mode, before a server responds to a synchronization request and compiles a base image by using a mapping rule file to obtain a test image, a first configuration file and a second configuration file are obtained, wherein the first configuration file is used for initializing the database component, and the second configuration file is used for initializing the search engine component. The server then initializes the database component with the first configuration file and initializes the search engine component with the second configuration file.
In this manner, the server obtains a first configuration file for initializing the database component and a second configuration file for initializing the search engine component. For example, the developer sends the first configuration file and the second configuration file to the server through a computer or the like. For another example, the developer stores the first configuration file and the second configuration file in a File Transfer Protocol (FTP), and the server downloads the first configuration file and the second configuration file from the FTP. After the two configuration files are obtained, the server initializes the database component and the search engine component.
By adopting the scheme, the server initializes the database component and the search engine component before compiling the basic mirror image, and the aim of flexibly initializing the database component and the search engine component is fulfilled.
In another mode, after responding to a synchronization request, a server compiles a basic image by using a mapping rule file to obtain a test image, and before running the test image to obtain a target container, the server acquires a first configuration file and a second configuration file, wherein the first configuration file is used for initializing the database component, the second configuration file is used for initializing the search engine component, the first configuration file is used for initializing the database component, and the second configuration file is used for initializing the search engine component.
By adopting the scheme, the server initializes the database component and the search engine component after compiling the basic mirror image and before running the test mirror image, and the aim of flexibly initializing the database component and the search engine component is fulfilled.
In yet another mode, after the server runs the test image to obtain the target container, a first configuration file and a second configuration file are obtained, where the first configuration file is used to initialize the database component, the second configuration file is used to initialize the search engine component, the database component is initialized by using the first configuration file, and the search engine component is initialized by using the second configuration file.
In the above embodiment, when the server uses the compiled backup component in the target container to perform data synchronization on the database component and the search engine component, a binary file is first obtained in the target container, where the binary file is generated by operating the database component. And then, the server queries the mapping rule file of the backup component after query compiling according to the binary file to obtain the target type and the first index. Then, the server analyzes the binary file into the structural data of the target type; and writing the structural data of the target type into the document indicated by the first index, wherein the first index indicates the position of the structural data in the search engine component.
Exemplarily, referring to fig. 4, fig. 4 is a schematic diagram of a data synchronization process in a data synchronization method provided by an embodiment of the present disclosure. Referring to fig. 4, after the database component is initialized, a user performs read/write operations on the database component, such as data insertion (insert), data update (update), and data deletion (delete) on the database. The users are, for example, developers, testers, etc. A user performs read and write operations on a database component to generate a binary file, which is also referred to as a log file, a log, a binlog, and the like. The binary file reaches a canal client (client) end through a canal service (server) end, the canal client analyzes data in the binary file according to a mapping rule in a mapping rule file to obtain structured data of a target type, and writes the structured data of the target type into a document indicated by a first index, wherein the document indicated by the first index is positioned in a search engine component.
Referring again to fig. 4, the user operations indicated by the binary file include: operation 1, insert data (db. table1-insert) in table 1; operation 2, insert data (db. table2-insert) in table 2; operation 3, updating data (db. table1-update) of table 1; operation 4, update data (db. table1-update) to table 2. The server discovers with the compiled canal client: table1 corresponds to a mapping rule of rule 1(db. table1, es. index1) (struct, mapping, doc), and table2 corresponds to a mapping rule of rule 2(db. table2, es. index2) (struct, mapping, doc). Wherein, the struct target type is represented, such as JSON. Therefore, the server synchronizes the data corresponding to the operations 1 and 3 in the binary file into the search engine component according to the rule 1; and synchronizing the data corresponding to the operation 2 and the operation 4 in the binary file into the search engine component according to the rule 2.
By adopting the scheme, the server determines the target type and the index by utilizing the compiled canal component, and further analyzes the binary file into structural data of the target type and writes the structural data into a document indicated by the index, so that data synchronization is realized, and the data synchronization process of the database component and the search engine component is simplified.
Referring to fig. 4 again, after synchronizing the database component and the search engine component, the server also receives a query request from the electronic device, where the query request carries the second index. The server then responds to the query request to return the structural data indicated by the second index in the search engine component to the electronic device.
Illustratively, the second index is any one of a plurality of indexes on the search engine component, the server receives a query request carrying the second index, determines a document indicated by the second index from the index engine component, and returns structured data contained in the document to the electronic device.
By adopting the scheme, the server simulates the action of the back-end service after the back-end service is on line after the data synchronization is carried out on the database component and the search engine component, so that the unit test is more accurate.
In response to the data synchronization method of the above embodiment, fig. 5 is a block diagram of a data synchronization apparatus provided in the embodiment of the disclosure. For ease of illustration, only portions that are relevant to embodiments of the present disclosure are shown. Referring to fig. 5, the apparatus includes: a receiving unit 11, a response unit 12, an operation unit 13 and a synchronization unit 14.
The receiving unit 11 is configured to receive a synchronization request, where the synchronization request is used to request data synchronization of a database component and a search engine component of a backend service.
A response unit 12, configured to respond to the synchronization request, compile a backup component in a basic image using a mapping rule file of the backend service to obtain a test image, where the basic image includes the database component, the search engine component, and the backup component, and the mapping rule file is used to define a business rule of the backend service.
And the running unit 13 is configured to run the test image to obtain a target container of the backend service.
A synchronization unit 14, configured to perform data synchronization on the database component and the search engine component in the target container by using the compiled backup component.
In an embodiment of the present disclosure, the synchronization unit 14 is configured to obtain a binary file in the target container, where the binary file is generated by performing a read-write operation on the database component, determine a target type and a first index according to the binary file and the compiled backup component, parse the binary file into the structural data of the target type, and write the structural data of the target type into a document indicated by the first index, where the first index indicates a position of the structural data in the search engine component.
In one embodiment of the present disclosure, the mapping rule file contains at least one mapping rule, and different mapping rules in the at least one mapping rule contain different mapping relationships of database identification, table identification, target type and index.
Fig. 6 is a block diagram of another data synchronization apparatus according to an embodiment of the present disclosure. The data synchronization apparatus 100 provided in this example is based on the foregoing fig. 5, and further includes:
an initializing unit 15, configured to obtain a first configuration file and a second configuration file, where the first configuration file is used to initialize the database component, the second configuration file is used to initialize the search engine component, the database component is initialized by using the first configuration file, and the search engine component is initialized by using the second configuration file.
Referring to fig. 6 again, in an embodiment of the present disclosure, the data synchronization apparatus 100 further includes:
a processing unit 16, configured to, before the response unit 12 responds to the synchronization request and compiles a basic mirror image by using a mapping rule file to obtain a test mirror image, obtain an open-source mirror image, where the open-source mirror image includes the database component; and installing the search engine component and the backup component on the open source mirror image to obtain the basic mirror image.
In an embodiment of the present disclosure, the receiving unit 11 is further configured to receive an inquiry request of an electronic device after the synchronizing unit 14 performs data synchronization on the database component and the search engine component by using the target container, where the inquiry request carries a second index.
The responding unit 12 is further configured to respond to the query request to return the structural data indicated by the second index in the search engine component to the electronic device.
In one embodiment of the present disclosure, the database component is a MySQL component and the search engine component is a full-text search engine ES component.
The apparatus provided in this embodiment may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again.
Fig. 7 is a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure, where the electronic device 200 may be a terminal device or a server. Among them, the terminal Device may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a Digital broadcast receiver, a Personal Digital Assistant (PDA), a tablet computer (PAD), a Portable Multimedia Player (PMP), a car terminal (e.g., car navigation terminal), etc., and a fixed terminal such as a Digital TV, a desktop computer, etc. The electronic device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, the electronic device 200 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 201, which may perform various suitable actions and processes according to a program stored in a Read Only Memory (ROM) 202 or a program loaded from a storage means 208 into a Random Access Memory (RAM) 203. In the RAM 203, various programs and data necessary for the operation of the electronic apparatus 200 are also stored. The processing device 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input/output (I/O) interface 205 is also connected to bus 204.
Generally, the following devices may be connected to the I/O interface 205: input devices 206 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 207 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 208 including, for example, magnetic tape, hard disk, etc.; and a communication device 202. The communication device 202 may allow the electronic apparatus 200 to communicate wirelessly or by wire with other apparatuses to exchange data. While fig. 7 illustrates an electronic device 200 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device 202, or installed from the storage device 208, or installed from the ROM 202. The computer program, when executed by the processing device 201, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or service. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of Network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first retrieving unit may also be described as a "unit for retrieving at least two internet protocol addresses".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In a first aspect, according to one or more embodiments of the present disclosure, there is provided a data synchronization method, including: receiving a synchronization request, wherein the synchronization request is used for requesting data synchronization of a database component and a search engine component of a back-end service; in response to the synchronization request, compiling a backup component in a basic mirror image by using a mapping rule file of the backend service to obtain a test mirror image, wherein the basic mirror image comprises the database component, the search engine component and the backup component, and the mapping rule file is used for defining a business rule of the backend service; running the test mirror image to obtain a target container of the back-end service; and performing data synchronization on the database component and the search engine component in the target container by using the compiled backup component.
According to one or more embodiments of the present disclosure, the performing, in the target container, data synchronization on the database component and the search engine component by using the compiled backup component includes: acquiring a binary file in the target container, wherein the binary file is generated by performing read-write operation on the database component; determining a target type and a first index according to the binary file and the compiled backup component; analyzing the binary file into the structural data of the target type; writing the structural data of the target type to the document indicated by the first index, the first index indicating a location of the structural data in the search engine component.
According to one or more embodiments of the present disclosure, the mapping rule file includes at least one mapping rule, and different mapping rules in the at least one mapping rule include different mapping relationships of database identifiers, table identifiers, target types, and indexes.
According to one or more embodiments of the present disclosure, the method described above further includes: acquiring a first configuration file and a second configuration file, wherein the first configuration file is used for initializing the database component, and the second configuration file is used for initializing the search engine component; initializing the database component with the first configuration file and initializing the search engine component with the second configuration file.
According to one or more embodiments of the present disclosure, before the compiling the base image by using the mapping rule file in response to the synchronization request to obtain the test image, the method further includes: acquiring an open source mirror image, wherein the open source mirror image comprises the database component; and installing the search engine component and the backup component on the open source mirror image to obtain the basic mirror image.
According to one or more embodiments of the present disclosure, after the data synchronization of the database component and the search engine component by using the target container, the method further includes: receiving a query request of the electronic equipment, wherein the query request carries a second index; and responding to the query request to return the structural data indicated by the second index in the search engine component to the electronic equipment.
According to one or more embodiments of the present disclosure, the database component is a MySQL component and the search engine component is a full-text search engine ES component.
In a second aspect, according to one or more embodiments of the present disclosure, there is provided a data synchronization apparatus including:
the receiving unit is used for receiving a synchronization request, and the synchronization request is used for requesting data synchronization of a database component and a search engine component of a back-end service.
And the response unit is used for responding to the synchronization request and compiling the backup component in the basic mirror image by using the mapping rule file of the back-end service so as to obtain the test mirror image, wherein the basic mirror image comprises the database component, the search engine component and the backup component, and the mapping rule file is used for defining the business rule of the back-end service.
And the running unit is used for running the test mirror image to obtain the target container of the back-end service.
And the synchronization unit is used for carrying out data synchronization on the database component and the search engine component in the target container by utilizing the compiled backup component.
According to one or more embodiments of the present disclosure, the synchronization unit is configured to obtain a binary file in the target container, where the binary file is generated by performing a read-write operation on the database component, determine a target type and a first index according to the binary file and the compiled backup component, parse the binary file into structural data of the target type, and write the structural data of the target type into a document indicated by the first index, where the first index indicates a position of the structural data in the search engine component.
According to one or more embodiments of the present disclosure, the mapping rule file includes at least one mapping rule, and different mapping rules in the at least one mapping rule include different mapping relationships of database identifiers, table identifiers, target types, and indexes.
According to one or more embodiments of the present disclosure, the apparatus described above further includes: the initialization unit is used for acquiring a first configuration file and a second configuration file, wherein the first configuration file is used for initializing the database component, the second configuration file is used for initializing the search engine component, the database component is initialized by using the first configuration file, and the search engine component is initialized by using the second configuration file.
According to one or more embodiments of the present disclosure, the apparatus described above further includes: the processing unit is used for acquiring an open source mirror image before the response unit responds to the synchronous request and utilizes a mapping rule file to compile a basic mirror image so as to obtain a test mirror image, and the open source mirror image comprises the database component; and installing the search engine component and the backup component on the open source mirror image to obtain the basic mirror image.
According to one or more embodiments of the present disclosure, the receiving unit is further configured to receive an inquiry request of an electronic device after the synchronization unit performs data synchronization on the database component and the search engine component by using the target container, where the inquiry request carries a second index; the response unit is further configured to respond to the query request to return the structural data indicated by the second index in the search engine component to the electronic device.
According to one or more embodiments of the present disclosure, the database component is a MySQL component and the search engine component is a full-text search engine ES component.
In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data synchronization method as described above in the first aspect and various possible designs of the first aspect.
In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium having stored therein computer-executable instructions that, when executed by a processor, implement the data synchronization method as described in the first aspect above and in various possible designs of the first aspect.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims (10)
1. A method of data synchronization, comprising:
receiving a synchronization request, wherein the synchronization request is used for requesting data synchronization of a database component and a search engine component of a back-end service;
in response to the synchronization request, compiling a backup component in a basic mirror image by using a mapping rule file of the back-end service to obtain a test mirror image, wherein the basic mirror image comprises the database component, the search engine component and the backup component, the mapping rule file is used for defining a business rule of the back-end service, and the backup component is used for backing up data in the database component;
running the test mirror image to obtain a target container of the back-end service;
and performing data synchronization on the database component and the search engine component in the target container by utilizing the compiled backup component.
2. The method of claim 1, wherein the synchronizing the data of the database component and the search engine component with the compiled backup component in the target container comprises:
acquiring a binary file in the target container, wherein the binary file is generated by performing read-write operation on the database component;
determining a target type and a first index according to the binary file and the compiled backup component;
analyzing the binary file into the structural data of the target type;
writing the structural data of the target type to the document indicated by the first index, the first index indicating a location of the structural data in the search engine component.
3. The method according to claim 1 or 2, wherein the mapping rule file comprises at least one mapping rule, and different mapping rules of the at least one mapping rule comprise different mapping relationships of database identification, table identification, target type and index.
4. The method of claim 1 or 2, further comprising:
acquiring a first configuration file and a second configuration file, wherein the first configuration file is used for initializing the database component, and the second configuration file is used for initializing the search engine component;
initializing the database component with the first configuration file and initializing the search engine component with the second configuration file.
5. The method according to claim 1 or 2, wherein before compiling the base image using the mapping rule file to obtain the test image in response to the synchronization request, the method further comprises:
acquiring an open source mirror image, wherein the open source mirror image comprises the database component;
and installing the search engine component and the backup component on the open source mirror image to obtain the basic mirror image.
6. The method of claim 1 or 2, wherein after the data synchronization of the database component and the search engine component using the target container, further comprising:
receiving a query request of the electronic equipment, wherein the query request carries a second index;
and responding to the query request to return the structural data indicated by the second index in the search engine component to the electronic equipment.
7. The method of claim 1 or 2, wherein the database component is a MySQL component and the search engine component is a full-text search engine ES component.
8. A data synchronization apparatus, comprising:
the device comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving a synchronization request which is used for requesting data synchronization of a database component and a search engine component of a back-end service;
a response unit, configured to respond to the synchronization request, compile a backup component in a basic mirror image using a mapping rule file of the backend service to obtain a test mirror image, where the basic mirror image includes the database component, the search engine component, and the backup component, and the mapping rule file is used to define a business rule of the backend service;
the running unit is used for running the test mirror image to obtain a target container of the back-end service;
and the synchronization unit is used for carrying out data synchronization on the database component and the search engine component by utilizing the compiled backup component in the target container.
9. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data synchronization method of any of claims 1 to 7.
10. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the data synchronization method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011411156.0A CN112395363B (en) | 2020-12-04 | 2020-12-04 | Data synchronization method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011411156.0A CN112395363B (en) | 2020-12-04 | 2020-12-04 | Data synchronization method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112395363A true CN112395363A (en) | 2021-02-23 |
| CN112395363B CN112395363B (en) | 2024-03-26 |
Family
ID=74604377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011411156.0A Active CN112395363B (en) | 2020-12-04 | 2020-12-04 | Data synchronization method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112395363B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113378030A (en) * | 2021-05-18 | 2021-09-10 | 上海德衡数据科技有限公司 | Search method of search engine, search engine architecture, device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102200944A (en) * | 2011-06-16 | 2011-09-28 | 中国联合网络通信集团有限公司 | Test environment cloning method and system for enterprise resource planning (ERP) system |
| US20180336258A1 (en) * | 2017-05-22 | 2018-11-22 | Sap Se | Validating Query Results During Asynchronous Database Replication |
| CN108958927A (en) * | 2018-05-31 | 2018-12-07 | 康键信息技术（深圳）有限公司 | Dispositions method, device, computer equipment and the storage medium of container application |
| CN110399307A (en) * | 2019-07-31 | 2019-11-01 | 网宿科技股份有限公司 | A kind of test method, test platform and destination server |
| CN111679965A (en) * | 2020-04-23 | 2020-09-18 | 平安国际智慧城市科技股份有限公司 | Automatic testing method and device, computer equipment and storage medium |
| CN111813683A (en) * | 2020-07-14 | 2020-10-23 | 深圳前海移联科技有限公司 | Universal automatic software test environment synchronization method |
-
2020
- 2020-12-04 CN CN202011411156.0A patent/CN112395363B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102200944A (en) * | 2011-06-16 | 2011-09-28 | 中国联合网络通信集团有限公司 | Test environment cloning method and system for enterprise resource planning (ERP) system |
| US20180336258A1 (en) * | 2017-05-22 | 2018-11-22 | Sap Se | Validating Query Results During Asynchronous Database Replication |
| CN108958927A (en) * | 2018-05-31 | 2018-12-07 | 康键信息技术（深圳）有限公司 | Dispositions method, device, computer equipment and the storage medium of container application |
| CN110399307A (en) * | 2019-07-31 | 2019-11-01 | 网宿科技股份有限公司 | A kind of test method, test platform and destination server |
| CN111679965A (en) * | 2020-04-23 | 2020-09-18 | 平安国际智慧城市科技股份有限公司 | Automatic testing method and device, computer equipment and storage medium |
| CN111813683A (en) * | 2020-07-14 | 2020-10-23 | 深圳前海移联科技有限公司 | Universal automatic software test environment synchronization method |
Non-Patent Citations (1)
| Title |
|---|
| 曾泽堂: "一种数据同步系统的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, 31 July 2019 (2019-07-31) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113378030A (en) * | 2021-05-18 | 2021-09-10 | 上海德衡数据科技有限公司 | Search method of search engine, search engine architecture, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112395363B (en) | 2024-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112395253B (en) | Index file generation method, terminal device, electronic device and medium | |
| CN110619100B (en) | Method and apparatus for acquiring data | |
| CN110392068A (en) | A kind of data transmission method, device and its equipment | |
| CN111309304B (en) | Method, device, medium and electronic equipment for generating IDL file | |
| CN111338944B (en) | Remote Procedure Call (RPC) interface testing method, device, medium and equipment | |
| US20240095022A1 (en) | Hotfix method, apparatus, device and storage medium | |
| CN113407165B (en) | SDK generation and self-upgrade method, device, readable medium and equipment | |
| CN112395363B (en) | Data synchronization method, device, equipment and readable storage medium | |
| CN112650521B (en) | Software development kit SDK thermal restoration method and device and electronic equipment | |
| CN111813685B (en) | Automatic test method and device | |
| CN111338666A (en) | Method, device, medium and electronic equipment for realizing application program upgrading | |
| CN111414152B (en) | Method, system, readable medium and electronic device for realizing business logic | |
| CN112422648A (en) | Data synchronization method and system | |
| CN111625232B (en) | Method and device for generating application program installation package, readable medium and electronic equipment | |
| CN113761075A (en) | Method, device, equipment and computer readable medium for switching databases | |
| CN111240657A (en) | Method and device for generating SDK and electronic equipment | |
| CN111787043A (en) | Data request method and device | |
| CN110750424A (en) | Resource inspection method and device | |
| CN112445820A (en) | Data conversion method and device | |
| CN116820354B (en) | Data storage method, data storage device and data storage system | |
| CN111209225B (en) | Method, device, medium and electronic equipment for acquiring real address | |
| CN112988528B (en) | Log processing method, device and container group | |
| CN112711400A (en) | View processing method, device, storage medium and computer program product | |
| CN114741132A (en) | Method, device, equipment and medium for managing configuration data | |
| CN117648117A (en) | Code branch management method, system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |