CN108763339A - Synchronous method, device and the equipment of database data - Google Patents
Synchronous method, device and the equipment of database data Download PDFInfo
- Publication number
- CN108763339A CN108763339A CN201810456123.4A CN201810456123A CN108763339A CN 108763339 A CN108763339 A CN 108763339A CN 201810456123 A CN201810456123 A CN 201810456123A CN 108763339 A CN108763339 A CN 108763339A
- Authority
- CN
- China
- Prior art keywords
- mongodb
- node
- copy sets
- data file
- test environment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses a kind of synchronous method of database data, device and equipment, are related to database technical field, can improve MongoDB database in phase efficiency and accuracy.Wherein method includes：Obtain the corresponding compressed data file of database data of MongoDB copy set destination nodes in production environment；Whether the compressed format for detecting the compressed data file is identical as the compressed format for the compressed data file that MongoDB copy sets can start in test environment；If identical, locked for the MongoDB copy sets destination node configurable write；The compressed data file got is copied to each MongoDB copy sets node in the test environment respectively.The application is suitable for the synchronization process of database data.
Description
Technical field
This application involves database technical field, particularly with regard to a kind of synchronous method of database data, device and
Equipment.
Background technology
In order to keep development rapid and flexible, many companies use MongoDB databases, and by some important data
It is stored in MongoDB databases.Wherein MongoDB is a database stored based on distributed document, is compiled by C Plus Plus
It writes, it is intended to provide expansible high-performance data storage solution for WWW (World Wide Web, WEB) application.And
MongoDB copy sets are one group of MongoDB servers, wherein there are one master servers (Primary, also known as host node), are used for
Handle client request；Also multiple backup servers (Secondary, also known as backup node), for preserving master server
Data copy.
In a software project, it will usually have test environment and production environment, sometimes for better test, need by
The database data full dose of MongoDB copy sets node is synchronized in test environment in production environment.The side of synchronization traditional at present
Formula is first to be carried out the database data of the MongoDB copy sets of production environment from hard disk using mongodump backup commands
It decompresses and backs up out, reuse mongorestore reduction order data convert that backup is gone out to test environment MongoDB
In copy set.
However, when needing to synchronize the database data of larger amount of MongoDB copy sets node, by above-mentioned traditional
The method of synchronization can cause full dose to synchronize an overlong time, and then database in phase can be caused less efficient, and if
Database data is changed in synchronizing process, be will also result in data and is synchronized inaccurate problem.
Invention content
In view of this, this application provides a kind of synchronous method of database data, device and equipment, main purposes to be
Solution when needing to synchronize the database data of larger amount of MongoDB copy sets node, passes through above-mentioned traditional synchronization at present
Mode can cause database in phase less efficient, and if database data is changed in synchronizing process, will also result in
Data synchronize inaccurate problem.
According to the one side of the application, a kind of synchronous method of database data is provided, this method includes：
Obtain the corresponding compressed data file of database data of MongoDB copy set destination nodes in production environment；
Detect what whether the compressed format of the compressed data file can start with MongoDB copy sets in test environment
The compressed format of compressed data file is identical；
If identical, locked for the MongoDB copy sets destination node configurable write；
The compressed data file got is copied to each MongoDB copy sets in the test environment respectively
Node.
Preferably, it is described the compressed data file got is copied to respectively it is each in the test environment
Before MongoDB copy set nodes, the method further includes：
Close the MongoDB processes of each MongoDB copy sets node in the test environment, and by the test environment
In the database data of each MongoDB copy sets node emptied；
The each MongoDB pairs compressed data file got copied to respectively in the test environment
This collection node, specifically includes：
The compressed data file got is copied in the test environment respectively after database data empties
Each MongoDB copy sets node.
Preferably, the method further includes：
After the compressed data file replicates completion, start each MongoDB copy sets section in the test environment
The MongoDB processes of point；
The IP address of MongoDB copy sets in the test environment is reconfigured, and forces to initialize so that the test
The MongoDB copy sets of environment re-elect host node and backup node.
Preferably, after the compressed data file replicates completion, the method further includes：
That unlocks the configuration of MongoDB copy sets destination node described in the production environment writes lock.
Preferably, the IP address for reconfiguring MongoDB copy sets in the test environment, specifically includes：
By the IP address of MongoDB copy sets in the test environment, institute is changed to by the IP address of the production environment
State the IP address of test environment.
Preferably, if there are the dsc datas of MongoDB databases, the acquisition to produce ring in the target node memory
The corresponding compressed data file of database data of MongoDB copy sets destination node, specifically includes in border：
Obtain the dsc data of the MongoDB databases in the target node memory；
The dsc data is written in hard disk, and is generated together with the database data of destination node described in the hard disk
New compressed data file；
The each MongoDB pairs compressed data file got copied to respectively in the test environment
This collection node, specifically includes：
The new compressed data file is copied to each MongoDB copy sets section in the test environment respectively
Point.
Preferably, the corresponding data of database data for obtaining MongoDB copy set destination nodes in production environment
Compressed file specifically includes：
Obtain the load state information of each node in the MongoDB copy sets of production environment；
According to the load state information, the node of load pressure minimum in each node is determined；
Obtain the corresponding compressed data file of node database data of the load pressure minimum；
It locks, specifically includes for the MongoDB copy sets destination node configurable write：
It is locked for the node configurable write of load pressure minimum.
Preferably, according to the load state information, the node of load pressure minimum in each node is determined, specifically
Including：
According to the load state information, the backup node of load pressure minimum in each node is determined；
The corresponding compressed data file of node database data for obtaining the load pressure minimum, specifically includes：
Obtain the corresponding compressed data file of backup node database data of the load pressure minimum；
It locks, specifically includes for the node configurable write of load pressure minimum：
It is locked for the backup node configurable write of load pressure minimum.
According to the another aspect of the application, a kind of synchronizing device of database data is provided, which includes：
Acquiring unit, the corresponding number of database data for obtaining MongoDB copy sets destination node in production environment
According to compressed file；
Detection unit, for detect the compressed data file that the acquiring unit obtains compressed format whether with test wrapper
The compressed format for the compressed data file that MongoDB copy sets can start in border is identical；
Dispensing unit, if detecting the compressed format and test environment of the compressed data file for the detection unit
The compressed format for the compressed data file that middle MongoDB copy sets can start is identical, then is the MongoDB copy sets target
Node configurable write is locked；
Synchronization unit, the compressed data file for getting the acquiring unit copy to the test respectively
Each MongoDB copy sets node in environment.
Preferably, described device further includes：Empty unit；
It is described to empty unit, for close the MongoDB of each MongoDB copy sets node in the test environment into
Journey, and the database data of each MongoDB copy sets node in the test environment is emptied；
The synchronization unit, specifically for the compressed data file got is copied to the test environment respectively
Middle database data empty after each MongoDB copy sets node.
Preferably, described device further includes：
Post-processing unit starts each in the test environment after replicating completion in the compressed data file
The MongoDB processes of MongoDB copy set nodes；
The IP address of MongoDB copy sets in the test environment is reconfigured, and forces to initialize so that the test
The MongoDB copy sets of environment re-elect host node and backup node.
Preferably, the post-processing unit is additionally operable to after the compressed data file replicates completion, unlocks the life
The configuration of MongoDB copy set nodes writes lock described in production environment.
Preferably, the post-processing unit is specifically used for the IP address of MongoDB copy sets in the test environment,
The IP address of the test environment is changed to by the IP address of the production environment.
Preferably, the acquiring unit, if specifically for there are the heat of MongoDB databases in the target node memory
Data then obtain the dsc data of the MongoDB databases in the target node memory；
The dsc data is written in hard disk, and is generated together with the database data of destination node described in the hard disk
New compressed data file；
The synchronization unit, specifically for copying to the new compressed data file in the test environment respectively
Each MongoDB copy sets node.
Preferably, the acquiring unit, specifically includes：
Acquisition module, the load state information of each node in the MongoDB copy sets for obtaining production environment；
Determining module, the load state information for being got according to the acquisition module determine in each node
The node of load pressure minimum；
The acquisition module is additionally operable to obtain the corresponding data compression of node database data of the load pressure minimum
File；
The dispensing unit is specifically used for locking for the node configurable write of load pressure minimum.
Preferably, the determining module is specifically used for, according to the load state information, determining in each node and bearing
Carry the backup node of pressure minimum；
The acquisition module is specifically used for obtaining the corresponding number of backup node database data of the load pressure minimum
According to compressed file；
The dispensing unit is specifically additionally operable to lock for the backup node configurable write of load pressure minimum.
According to the application another aspect, a kind of storage medium is provided, computer program, described program are stored thereon with
The synchronous method of above-mentioned database data is realized when being executed by processor.
According to the application another aspect, a kind of entity device of database in phase is provided, including storage medium,
Processor and storage on a storage medium and the computer program that can run on a processor, the processor execution described program
The synchronous method of the above-mentioned database datas of Shi Shixian.
By above-mentioned technical proposal, a kind of synchronous method of database data, device and equipment provided by the present application, with mesh
Preceding traditional method of synchronization is compared, the application MongoDB copy sets destination node database data in detecting production environment
The compressed data file that MongoDB copy sets can start in the compressed format and test environment of corresponding compressed data file
When compressed format is identical, illustrate to be copied directly to survey by the corresponding database data compressed file of the destination node in production environment
In test ring border, the MongoDB copy sets in test environment can also start the compressed data file, therefore the application can should
The compressed data file of destination node is copied directly to each MongoDB copy sets node in test environment.It is not necessarily to so first
It decompresses the compressed data file and backs up out and then restore the process of test environment MongoDB copy sets, it is same when needing
When walking the database data of larger amount of MongoDB copy sets node, MongoDB database data full doses synchronize primary duration
Reduce significantly, the synchronous efficiency of MongoDB database datas can be promoted.And to the mesh of production environment before data synchronization
Mark node, which adds, writes lock, can be changed in synchronizing process to avoid database data, ensure MongoDB database in phase
Accuracy.
Above description is only the general introduction of technical scheme, in order to better understand the technological means of the application,
And can be implemented in accordance with the contents of the specification, and in order to allow above and other objects, features and advantages of the application can
It is clearer and more comprehensible, below the special specific implementation mode for lifting the application.
Description of the drawings
Attached drawing described herein is used for providing further understanding of the present application, constitutes part of this application, this Shen
Illustrative embodiments and their description please do not constitute the improper restriction to the application for explaining the application.In the accompanying drawings：
Fig. 1 shows a kind of flow diagram of the synchronous method of database data provided by the embodiments of the present application；
Fig. 2 shows the flow diagrams of the synchronous method of another database data provided by the embodiments of the present application；
Fig. 3 shows the MongoDB copy set example schematics of two varying environments provided by the embodiments of the present application；
Fig. 4 shows Cross-environment MongoDB database in phase example schematics provided by the embodiments of the present application；
Fig. 5 shows that the profile instance after Cross-environment MongoDB database in phase provided by the embodiments of the present application shows
It is intended to；
Fig. 6 shows a kind of structural schematic diagram of the synchronizing device of database data provided by the embodiments of the present application；
Fig. 7 shows the structural schematic diagram of the synchronizing device of another database data provided by the embodiments of the present application.
Specific implementation mode
The application is described in detail below with reference to attached drawing and in conjunction with the embodiments.It should be noted that not conflicting
In the case of, the features in the embodiments and the embodiments of the present application can be combined with each other.
For at present when needing to synchronize the database data of larger amount of MongoDB copy sets node, by traditional
The method of synchronization can cause MongoDB database in phase less efficient, and if MongoDB databases number in synchronizing process
According to being changed, it will also result in data and synchronize inaccurate problem, present embodiments provide a kind of synchronous method of database data,
MongoDB database in phase efficiency and accuracy can be improved, as shown in Figure 1, this method includes：
101, the corresponding data compression text of database data of MongoDB copy set destination nodes in production environment is obtained
Part.
Wherein, the database data that MongoDB copy sets are preserved in compressed data file, according to MongoDB databases
The database data of mechanism, MongoDB copy sets can be stored in by compression in compressed data file, to save memory space money
Source can read the data preserved in the compressed data file when MongoDB copy sets work automatically, automatic in reading process
It is decompressed.
MongoDB copy sets include multiple nodes (MongoDB servers), wherein there are one host nodes and multiple backups to save
Point, if host node collapses, one of member can be upgraded to new host node by backup node automatically, since backup saves
Data copy of the point for preserving host node, so the MongoDB database datas of these nodes are all identical.In the present embodiment
In, it is specific to choose which node in production environment MongoDB copy sets (utilizes the number on the destination node as destination node
The synchronization of Cross-environment data is carried out according to library compressed data file), selection setting can be carried out according to practical business demand.For example, can
To select a backup node in MongoDB copy sets as destination node, avoids selection host node and cause to influence client
Hold the processing of request.
For the executive agent of the present embodiment can be for Cross-environment MongoDB database in phase device or set
It is standby, it may be implemented to improve database in phase efficiency and accuracy.It should be noted that method provided in this embodiment,
Other than applying in the scene for synchronizing MongoDB database datas to test environment from production environment, it can also be applied to
The data synchronization scenarios of its Cross-environment, such as can be applied to synchronize MongoDB database numbers from test environment to production environment
According to scene, can also be applied to from production environment to development environment synchronize MongoDB database datas scene etc..
102, the compressed format of compressed data file that detection is got whether with MongoDB copy sets energy in test environment
The compressed format of the compressed data file enough started is identical.
Wherein, the compressed format of compressed data file can be judged by the extension name of compressed data file.
For example, an inquiry instruction can be sent to the MongoDB copy sets of test environment, it can be with by the inquiry instruction
Inquiry obtains the compressed format for the compressed data file that MongoDB copy sets can start in test environment, then will get
Production environment in the obtained compressed format of compressed data file of compressed format and inquiry of compressed data file be compared,
If the two compressed format is identical, illustrate the corresponding database data compressed file of the destination node in production environment is directly multiple
It makes in test environment, the MongoDB copy sets in test environment can also directly initiate the compressed data file, and then determine
The MongoDB data of Cross-environment can be directly carried out using the corresponding MongoDB database datas compressed file of the destination node
Library data synchronize；If the two compressed format differs, after the compressed data file for illustrating directly to replicate the destination node, test
MongoDB copy sets in environment can not start the compressed data file, i.e., can not read the data in this document, therefore cannot
The MongoDB database numbers of Cross-environment are directly carried out using the corresponding MongoDB database datas compressed file of the destination node
According to synchronization.
If MongoDB copy sets can start in the compressed format and test environment of the compressed data file 103, got
Compressed data file compressed format it is identical, then locked for MongoDB copy sets destination node configurable write in production environment.
For the present embodiment, directly carried out using the corresponding MongoDB database datas compressed file of the destination node
Before the MongoDB database in phase of Cross-environment, needs to add the destination node and write lock, that is, forbid the destination node
MongoDB databases are written into data, and state is read-only status, so that the destination node is during data synchronize,
The data in the compressed data file replicated will not be changed, it is ensured that the accuracy of MongoDB database in phase,
To ensure the data consistency of Cross-environment data synchronization.
104, the compressed data file got is copied to each MongoDB copy sets section in test environment respectively
Point.
For example, there are four nodes for the MongoDB copy sets in test environment, it, can be by life in data synchronization process
The compressed data file of the destination node in environment is produced, while copying to this four nodes of test environment, and then realizes transannular
The process of border MongoDB database in phase.
The synchronous method of above-mentioned database data through this embodiment works as needs compared with the method for synchronization traditional at present
When Cross-environment synchronizes the MongoDB database datas of 1000GB, traditional approach exports more than 8 hours of time-consuming meeting of data, then
In addition data import the time, the total time will be more than 16 hours, and the present embodiment is without first decompressing compressed data file and backing up
Out and then the processes of test environment MongoDB copy sets is restored, utilizes the data compression of destination node in production environment
File directly carries out the MongoDB database in phase to test environment, and the MongoDB database datas of synchronous 1000GB are only
1 hour is needed, MongoDB database data full doses are greatly reduced and synchronize primary duration, MongoDB databases can be promoted
The synchronous efficiency of data.And the destination node of production environment is added before data synchronization and writes lock, it can be to avoid database number
It is changed according in synchronizing process, ensures the accuracy of MongoDB database in phase.
Further, as the refinement and extension of above-described embodiment specific implementation mode, in order to completely illustrate database
The overall flow of method of data synchronization present embodiments provides the synchronous method of another database data, as shown in Fig. 2, should
Method includes：
201, the load state information of each node in the MongoDB copy sets of production environment is obtained.
Wherein, the load pressure situation of node server is included in load state information.For the present embodiment, when receiving
When needing the instruction of Cross-environment MongoDB database in phase, it can obtain each in the MongoDB copy sets of production environment
The real-time load state information of node is then based on the load state information and determines suitable node as destination node, i.e. profit
Cross-environment data synchronization is carried out with the database data compressed file on the destination node.
202, it according to the load state information got, determines in each node of MongoDB copy sets of production environment and bears
Carry the node of pressure minimum.
For example, there are 3 nodes for the MongoDB copy sets of production environment, the wherein load value of a nodes is 80%, b nodes
Load value be the load values of 40%, C nodes be 60%, then can determine that the load pressure of b nodes is minimum.
The node of load pressure minimum is as target section in each node of MongoDB copy sets by selecting production environment
Point, then using on the destination node database data compressed file carry out Cross-environment data synchronization, can to avoid node by
It is excessive in load pressure and lead to occur the case where node delay machine, it is ensured that data synchronized is normally carried out.Further, it is
It avoids selection host node and leads to the processing for influencing client request, can choose load pressure as a preferred method,
Minimum backup node is as destination node, correspondingly, step 202 can specifically include：According to the load condition letter got
Breath, determines the backup node of load pressure minimum in each node of MongoDB copy sets of production environment.
203, the corresponding compressed data file of node database data of load pressure minimum is obtained.
In order to facilitate data are read, the dsc data in the database data of MongoDB copy sets is stored in sometimes interior
It deposits, wherein dsc data may be considered the popular business datum of newest period, such as can be according to business demand dsc data
Nearest 30 minutes or nearest 1 hour hot data can thus obtain the MongoDB of current hot topic directly from memory
Database data.For the present embodiment, if there are the dsc data of MongoDB databases in the node memory of load pressure minimum,
The database data for synchronizing MongoDB copy sets if necessary to Cross-environment full dose at this time, with this condition, as a kind of preferred side
Formula, corresponding step 203 can specifically include：Obtain the heat of the MongoDB databases in the node memory of load pressure minimum
Data；Then the dsc data is written in hard disk, and generates new data together with the database data of the node in hard disk
Compressed file.
Based on the preferred embodiment of above-mentioned steps 202, correspondingly, step 203 can specifically include：It is minimum to obtain load pressure
The corresponding compressed data file of backup node database data.
204, the compressed format of compressed data file that detection is got whether with MongoDB copy sets energy in test environment
The compressed format of the compressed data file enough started is identical.
The form that the database data of the MongoDB copy sets of production environment preserves in a hard disk is compressed data file,
Since the compressed format of the compressed data file may be different in different practical businesses, in order to enable test environment
MongoDB copy sets can start the MongoDB database datas in production environment, traditional Cross-environment MongoDB at present
The mode of database in phase is that compressed data file is first all unziped to unified file format in memory and exports backup
Out, it then imports in test environment again, although the unified file format can be tested MongoDB copy sets in environment and start,
But this synchronizing process needs to expend longer time.And the present embodiment can be in the middle MongoDB for detecting production environment
The compressed data file that MongoDB copy sets can start in the compressed format and test environment of the compressed data file of copy set
Compressed format it is identical when, each MongoDB pairs for the compressed data file of production environment being copied directly in test environment
This collection node is greatly reduced MongoDB database data full doses and synchronizes primary duration, can promote MongoDB databases
The synchronous efficiency of data.
If MongoDB copy sets can start in the compressed format and test environment of the compressed data file 205, got
Compressed data file compressed format it is identical, then be in production environment load pressure minimum node configurable write lock.
For the present embodiment, carried out in the MongoDB database data compressed files of the node using load pressure minimum
Before the MongoDB database in phase of Cross-environment, needs the node to load pressure minimum to add and write lock, so as in data
In synchronizing process, the MongoDB databases of the node of load pressure minimum is forbidden to be written into data, ensures that Cross-environment data are same
The data consistency of step.
Based on the preferred embodiment of above-mentioned steps 202, if the backup node using load pressure minimum realizes Cross-environment
MongoDB database in phase is then the step of the node configurable write lock of load pressure minimum in production environment in step 205
Suddenly, it can specifically include：It is locked for the backup node configurable write of load pressure minimum.
206, the MongoDB processes of each MongoDB copy sets node in test environment are closed, and will be each in test environment
The database data of a MongoDB copy sets node is emptied.
In the present embodiment, before the MongoDB database in phase for carrying out Cross-environment using compressed data file,
In order to avoid colliding with the MongoDB copy set database datas of test environment, need first to empty each in test environment
Then the database data of MongoDB copy set nodes recycles compressed data file to carry out the MongoDB databases of Cross-environment
Data synchronize.
207, the compressed data file got is copied to respectively each after database data empties in test environment
MongoDB copy set nodes.
Here the strategy that may be used while replicating will get compressed data file while copy in test environment
Database data empty after each MongoDB copy sets node；Or the if speed that test environment internodal data transmits
Comparatively fast, can also the compressed data file got first first be copied into a node in test environment, then by the node
Replicated with environmental data, and then the efficiency of MongoDB database in phase can be further increased, and can also be right
The node of load pressure minimum is unlocked in advance in production environment writes lock, restores the write-in data function of the node in advance, and then drop
The low influence to production environment MongoDB copy sets.
Based on the preferred embodiment in above-mentioned steps 203, it is written to by the dsc data of MongoDB databases in node memory
It, can be by new number in hard disk, and after generating new compressed data file together with the database data of the node in hard disk
It copies to each MongoDB copy sets node in test environment respectively according to compressed file, and then realizes that Cross-environment full dose synchronizes
The database data of MongoDB copy sets.
208, after compressed data file replicates completion, start each MongoDB copy sets node in test environment
MongoDB processes, and unlock MongoDB copy sets load pressure minimum in production environment node configuration write lock.
Due to closing the MongoDB processes of each MongoDB copy sets node in test environment before, in order to restore to survey
The function of MongoDB copy sets in test ring border needs to restart these closings after compressed data file replicates completion
Node M ongoDB processes.And in order to enable in production environment where the compressed data file is former load pressure minimum section
Point restores write-in data function can accordingly unlock so as to the MongoDB database datas of other nodes in synchronous production environment
The node of load pressure minimum writes lock, unlock the node write lock after, which will synchronize other under same environment
The MongoDB database datas of node.
209, the IP address of MongoDB copy sets in test environment is reconfigured, and forces to initialize so that test environment
MongoDB copy sets re-elect host node and backup node.
After starting test environment interior joint MongoDB processes, since compressed data file is answered from production environment
What system came, the IP of all copy sets is the IP address of production environment in test environment, for MongoDB copies in test environment
Collection can normal operation, need to reconfigure the IP address of MongoDB copy sets in test environment, and force to initialize so that
The MongoDB copy sets of test environment re-elect host node and backup node.Test is reconfigured as a preferred method,
In environment the step of the IP address of MongoDB copy sets, it can specifically include：By the IP of MongoDB copy sets in test environment
Address is changed to the IP address of test environment by the IP address of production environment.It, can by above-mentioned this configuration and initialization procedure
So that in test environment MongoDB copy sets can based on from production environment it is copied come compressed data file normal operation
The function of copy set.
In order to facilitate the synchronous method of database data provided in this embodiment is understood, following application scenarios example is provided,
But not limited to this：
It is production environment and test environment respectively for example, as shown in figure 3, there are two MongoDB copy environment, wherein raw
Production environment is the MongoDB copy sets of three nodes, and corresponding MongoDB databases are there are the data of 1000GB, and test environment
It is also the MongoDB copy sets of three nodes, two environment are independently operated.At this time in order to preferably test, need to produce
The database data full dose of MongoDB copy sets node is synchronized in test environment in environment.For the convenience of description, ring will be produced
MongoDB copy sets three nodes in border are respectively labeled as A1, A2, A3；By three node difference of test environment MongoDB copy sets
Labeled as B1, B2, B3, wherein A1 nodes and B1 nodes is host node (Primary nodes), and A2, A3 and B2, B3 are backup node
(Secondary nodes).
First if there are MongoDB database dsc datas in A3 node memories, by dsc data write-in hard disk, and with
The database data of A3 nodes generates new compressed data file together in hard disk.After the completion of write-in, on A3 nodes plus write
Lock, at this time A3 node Ms ongoDB databases will stop writing data toward hard disk, the MongoDB on hard disk in the compressed data file
Database data will remain unchanged, specific as shown in Figure 4；Then in order to avoid the MongoDB copy set data with test environment
Library data collide, and need first to empty the database data of each MongoDB copy sets node in test environment, corresponding long-range
The MongoDB processes of three nodes in test environment are killed, and empty three node M ongoDB database datas of test environment；
After emptying data, the compressed data file of A3 nodes is copied into tri- nodes of B1, B2, B3, specifically three can be used to answer
Task parallelism processed replicates, as shown in figure 4, since the compressed data file is compressed, the MongoDB database numbers of 1000GB
According to the file size for being actually stored in hard disk between 300GB to 400GB, in gigabit networking, reproducible was finished in one hour.
After duplication is fully completed, in order to which A3 nodes restore data write-in functions, to synchronize A1 and A2 nodes
MongoDB database datas, can unlock A3 nodes writes lock, specific as shown in Figure 5；And in order to restore in test environment
The function of MongoDB copy sets can restart tri- node processes of test environment B1, B2, B3, due to MongoDB databases
File is copied next from production environment, and the IP address of all copy sets is the IP address of production environment, needs to reconfigure
The IP address of copy set is the IP address of test environment and forces to initialize, as shown in figure 5, after the completion of forcing to initialize, is surveyed
Test ring border MongoDB copy sets meeting automatic election goes out host node and backup node.
By the above-mentioned this MongoDB database datas Cross-environment method of synchronization, synchronous TB grades can obviously reduce
MongoDB database datas, and can ensure data consistency.
Further, the specific implementation as Fig. 1 methods, the embodiment of the present application provide a kind of synchronization of database data
Device, as shown in fig. 6, the device includes：Acquiring unit 31, detection unit 32, dispensing unit 33, synchronization unit 34.
Acquiring unit 31 can be used for obtaining the database data pair of MongoDB copy set destination nodes in production environment
The compressed data file answered；Acquiring unit 31 is that the major function of MongoDB database corresponding data files is obtained in the present apparatus
Module, detection trigger unit 32 works after obtaining this document.
Detection unit 32, can be used for measurement acquisition unit 31 acquisition compressed data file compressed format whether with survey
The compressed format for the compressed data file that MongoDB copy sets can start in test ring border is identical；Detection unit 32 is the present apparatus
The main functional modules of the compressed format of middle Cross-environment detection data compressed file.
Dispensing unit 33, if can be used for compressed format and test environment that detection unit 32 detects compressed data file
The compressed format for the compressed data file that middle MongoDB copy sets can start is identical, then is MongoDB copy set destination nodes
Configurable write is locked；Dispensing unit 33 is the main functional modules that configuration node writes lock in the present apparatus, should after configuration node writes lock
Node will be unable to that new data are written into hard disk.
Synchronization unit 34 can be used for the compressed data file that acquiring unit 31 is got copying to test environment respectively
In each MongoDB copy sets node.Synchronization unit 34 is the main functional modules that data synchronize in the present apparatus.
It is same in the MongoDB database datas for carrying out Cross-environment using compressed data file in specific application scenarios
Before step, in order to avoid colliding with the MongoDB copy set database datas of test environment, need first to empty in test environment
The database data of each MongoDB copy sets node, accordingly as shown in fig. 7, the present apparatus further includes：Empty unit 35；
Unit 35 is emptied, can be used for closing the MongoDB processes of each MongoDB copy sets node in test environment,
And the database data of each MongoDB copy sets node in test environment is emptied；
Synchronization unit 34, the compressed data file that specifically can be used for get copy to data in test environment respectively
Library data empty after each MongoDB copy sets node.
In specific application scenarios, due to closing each MongoDB copy sets node in test environment before
MongoDB processes, in order to restore the function of MongoDB copy sets in test environment, after compressed data file replicates completion,
It needs to restart these node M ongoDB processes closed.Accordingly as shown in fig. 7, the present apparatus further includes：Post-processing unit
36；
Post-processing unit 36 can be used for after compressed data file replicates completion, start each in test environment
The MongoDB processes of MongoDB copy set nodes；Then the IP address of MongoDB copy sets in test environment is reconfigured, and
Force initialization so that the MongoDB copy sets of test environment re-elect host node and backup node.
In specific application scenarios, in order to enable destination node is extensive in the production environment at the former place of the compressed data file
It makes carbon copies into data function, so as to the MongoDB database datas of other nodes in synchronous production environment, correspondingly, post-processing is single
Member 36 can be also used for after compressed data file replicates completion, unlocks MongoDB copy sets node in production environment and configures
Write lock.
In specific application scenarios, since compressed data file is copied next, the test environment from production environment
In all copy sets IP be production environment IP address, in order in test environment MongoDB copy sets can normal operation, phase
It answers, post-processing unit 36 specifically can be used for by the IP address of MongoDB copy sets in test environment, by production environment
IP address is changed to the IP address of test environment.
In specific application scenarios, in order to realize that Cross-environment full dose synchronizes the database data of MongoDB copy sets, phase
The acquiring unit 31 answered, if specifically can be used in target node memory obtaining mesh there are the dsc data of MongoDB databases
Mark the dsc data of the MongoDB databases in node memory；By dsc data be written hard disk in, and with the number of destination node in hard disk
Generate new compressed data file together according to library data.
Correspondingly, synchronization unit 32, specifically can be used for respectively copying to new compressed data file in test environment
Each MongoDB copy sets node.
In specific application scenarios, as shown in fig. 7, acquiring unit 31, specifically includes：Acquisition module 311, determining module
312；
Acquisition module 311 can be used for the load condition letter of each node in the MongoDB copy sets of acquisition production environment
Breath；
Determining module 312, the load state information that can be used for being got according to acquisition module 311 determine each node
The node of middle load pressure minimum；
Acquisition module 311 can be also used for the corresponding data compression of node database data for obtaining load pressure minimum
File；
Dispensing unit 33 specifically can be used for locking for the node configurable write of load pressure minimum.
The node of load pressure minimum is as target section in each node of MongoDB copy sets by selecting production environment
Point, then using on the destination node database data compressed file carry out Cross-environment data synchronization, can to avoid node by
It is excessive in load pressure and lead to occur the case where node delay machine, it is ensured that data synchronized is normally carried out.
In specific application scenarios, the processing for influencing client request is caused in order to avoid selection host node, it can be with
The backup node of load pressure minimum is chosen as destination node, corresponding determining module 312 specifically can be used for according to load
Status information determines the backup node of load pressure minimum in each node；
Acquisition module 311 specifically can be used for obtaining the corresponding number of backup node database data of load pressure minimum
According to compressed file；
Dispensing unit 33 specifically can be also used for locking for the backup node configurable write of load pressure minimum.
It should be noted that each function list involved by a kind of synchronizing device of database data provided by the embodiments of the present application
Other corresponding descriptions of member, can be with the corresponding description in reference chart 1 and Fig. 2, and details are not described herein.
Based on above-mentioned method as depicted in figs. 1 and 2, correspondingly, the embodiment of the present application also provides a kind of storage medium,
On be stored with computer program, which realizes the same of above-mentioned database data as depicted in figs. 1 and 2 when being executed by processor
One step process.The program can write to obtain using shell tools.
Based on this understanding, the technical solution of the application can be expressed in the form of software products, software production
Product can be stored in a non-volatile memory medium (can be CD-ROM, USB flash disk, mobile hard disk etc.), including some instructions
With so that computer equipment (can be personal computer, server or the network equipment an etc.) execution the application is each
Method described in implement scene.
Based on above-mentioned method and Fig. 6 as depicted in figs. 1 and 2, virtual bench embodiment shown in Fig. 7, in order to realize
Above-mentioned purpose, the embodiment of the present application also provides a kind of entity devices of database in phase, are specifically as follows computer, clothes
Business device or the network equipment etc., which includes storage medium and processor；Storage medium, for storing computer journey
Sequence；Processor, for executing computer program to realize the synchronous method of above-mentioned database data as depicted in figs. 1 and 2.
Optionally, which can also include user interface, network interface, camera, radio frequency (Radio
Frequency, RF) circuit, sensor, voicefrequency circuit, WI-FI modules etc..User interface may include display screen
(Display), input unit such as keyboard (Keyboard) etc., optional user interface can also connect including USB interface, card reader
Mouthful etc..Network interface may include optionally standard wireline interface and wireless interface (such as WI-FI interface).
It will be understood by those skilled in the art that a kind of entity device structure of database in phase provided in this embodiment
The restriction to the entity device is not constituted, may include more or fewer components, either combines certain components or difference
Component arrangement.
Can also include operating system, network communication module in storage medium.Operating system is the above-mentioned database number of management
According to the program of synchronous entity device hardware and software resource, message handling program and other softwares and/or program are supported
Operation.Network communication module for realizing the communication between each component in storage medium inside, and with information processing entities equipment
In communicate between other hardware and softwares.
Through the above description of the embodiments, those skilled in the art can be understood that the application can borrow
It helps software that the mode of necessary general hardware platform is added to realize, hardware realization can also be passed through.Pass through the skill of application the application
Art scheme,.
It will be appreciated by those skilled in the art that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, module in attached drawing or
Flow is not necessarily implemented necessary to the application.It will be appreciated by those skilled in the art that the mould in device in implement scene
Block can according to implement scene describe be distributed in the device of implement scene, can also carry out respective change be located at be different from
In one or more devices of this implement scene.The module of above-mentioned implement scene can be merged into a module, can also be into one
Step splits into multiple submodule.
Above-mentioned the application serial number is for illustration only, does not represent the quality of implement scene.Disclosed above is only the application
Several specific implementation scenes, still, the application is not limited to this, and the changes that any person skilled in the art can think of is all
The protection domain of the application should be fallen into.
Claims (10)
1. a kind of synchronous method of database data, which is characterized in that including：
Obtain the corresponding compressed data file of database data of MongoDB copy set destination nodes in production environment；
Detect the compressed data file compressed format whether with MongoDB copy sets can start in test environment data
The compressed format of compressed file is identical；
If identical, locked for the MongoDB copy sets destination node configurable write；
The compressed data file got is copied to each MongoDB copy sets section in the test environment respectively
Point.
2. according to the method described in claim 1, it is characterized in that, described that the compressed data file got is multiple respectively
Before making each MongoDB copy sets node in the test environment, the method further includes：
The MongoDB processes of each MongoDB copy sets node in the test environment are closed, and will be each in the test environment
The database data of a MongoDB copy sets node is emptied；
The each MongoDB copy sets compressed data file got copied to respectively in the test environment
Node specifically includes：
The compressed data file got is copied to respectively each after database data empties in the test environment
MongoDB copy set nodes.
3. according to the method described in claim 2, it is characterized in that, the method further includes：
After the compressed data file replicates completion, start each MongoDB copy sets node in the test environment
MongoDB processes；
The IP address of MongoDB copy sets in the test environment is reconfigured, and forces to initialize so that the test environment
MongoDB copy sets re-elect host node and backup node.
4. according to the method described in claim 3, it is characterized in that, the compressed data file replicate complete after, it is described
Method further includes：
That unlocks the configuration of MongoDB copy sets destination node described in the production environment writes lock.
5. according to the method described in claim 3, it is characterized in that, described reconfigure MongoDB pairs in the test environment
The IP address of this collection, specifically includes：
By the IP address of MongoDB copy sets in the test environment, the survey is changed to by the IP address of the production environment
The IP address in test ring border.
6. if according to the method described in claim 1, it is characterized in that, there are MongoDB data in the target node memory
The dsc data in library, then it is described obtain production environment in MongoDB copy set destination nodes the corresponding data pressure of database data
Contracting file, specifically includes：
Obtain the dsc data of the MongoDB databases in the target node memory；
The dsc data is written in hard disk, and is generated newly together with the database data of destination node described in the hard disk
Compressed data file；
The each MongoDB copy sets compressed data file got copied to respectively in the test environment
Node specifically includes：
The new compressed data file is copied to each MongoDB copy sets node in the test environment respectively.
7. according to the method described in claim 1, it is characterized in that, MongoDB copy set targets in the acquisition production environment
The corresponding compressed data file of database data of node, specifically includes：
Obtain the load state information of each node in the MongoDB copy sets of production environment；
According to the load state information, the node of load pressure minimum in each node is determined；
Obtain the corresponding compressed data file of node database data of the load pressure minimum；
It locks, specifically includes for the MongoDB copy sets destination node configurable write：
It is locked for the node configurable write of load pressure minimum.
8. a kind of synchronizing device of database data, which is characterized in that including：
Acquiring unit, the corresponding data pressure of database data for obtaining MongoDB copy sets destination node in production environment
Contracting file；
Detection unit, for detect the compressed data file that the acquiring unit obtains compressed format whether in test environment
The compressed format for the compressed data file that MongoDB copy sets can start is identical；
Dispensing unit, if in detecting the compressed format and test environment of the compressed data file for the detection unit
The compressed format for the compressed data file that MongoDB copy sets can start is identical, then is the MongoDB copy sets target section
Point configurable write lock；
Synchronization unit, the compressed data file for getting the acquiring unit copy to the test environment respectively
In each MongoDB copy sets node.
9. a kind of storage medium, is stored thereon with computer program, which is characterized in that realized when described program is executed by processor
The synchronous method of database data described in any one of claim 1 to 7.
10. a kind of synchronizer of database data, including storage medium, processor and storage on a storage medium and can located
The computer program run on reason device, which is characterized in that the processor is realized when executing described program in claim 1 to 7
The synchronous method of any one of them database data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810456123.4A CN108763339A (en) | 2018-05-14 | 2018-05-14 | Synchronous method, device and the equipment of database data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810456123.4A CN108763339A (en) | 2018-05-14 | 2018-05-14 | Synchronous method, device and the equipment of database data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108763339A true CN108763339A (en) | 2018-11-06 |
Family
ID=64010414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810456123.4A Pending CN108763339A (en) | 2018-05-14 | 2018-05-14 | Synchronous method, device and the equipment of database data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108763339A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111159141A (en) * | 2020-01-02 | 2020-05-15 | 中国航空工业集团公司西安航空计算技术研究所 | Decentralized distributed data synchronization method, distributed node and system |
| CN111858762A (en) * | 2020-07-21 | 2020-10-30 | 中冶南方工程技术有限公司 | Data storage method and system of full-process quality management system |
| CN113485869A (en) * | 2021-06-08 | 2021-10-08 | 上海众言网络科技有限公司 | Database upgrading method and device |
-
2018
- 2018-05-14 CN CN201810456123.4A patent/CN108763339A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111159141A (en) * | 2020-01-02 | 2020-05-15 | 中国航空工业集团公司西安航空计算技术研究所 | Decentralized distributed data synchronization method, distributed node and system |
| CN111858762A (en) * | 2020-07-21 | 2020-10-30 | 中冶南方工程技术有限公司 | Data storage method and system of full-process quality management system |
| CN113485869A (en) * | 2021-06-08 | 2021-10-08 | 上海众言网络科技有限公司 | Database upgrading method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107544871B (en) | Virtual machine disk backup method and device | |
| CN102591982B (en) | Perform the method and system of increment SQL server database backup | |
| US9521196B2 (en) | Methods, devices and systems for dynamically managing memberships in replicated state machines within a distributed computing environment | |
| US10055149B2 (en) | Intelligent snapshot tree replication | |
| CN108763339A (en) | Synchronous method, device and the equipment of database data | |
| US10235088B1 (en) | Global replication policy for multi-copy replication | |
| US11917005B2 (en) | Clustered storage system synchronization | |
| CN107783816A (en) | The method and device that creation method and device, the big data cluster of virtual machine create | |
| CN106605217B (en) | For the method and system for being moved to another website from a website will to be applied | |
| CN112364049B (en) | Data synchronization script generation method, system, terminal and storage medium | |
| CN109783533A (en) | Collecting method, device, computer equipment and storage medium | |
| CN109753379B (en) | Snapshot data backup and deletion method, device and system | |
| TW202008147A (en) | Compression processing method and apparatus for block chain data | |
| CN113297159A (en) | Data storage method and device | |
| CN107544851B (en) | Lock operation processing method and device | |
| CN110825758A (en) | Transaction processing method and device | |
| CN113918096B (en) | Method, device and application for uploading algorithm mirror image package | |
| CN114817046A (en) | Pressure testing method, device and system for block chain network | |
| CN114328018B (en) | Snapshot creation method, computing device and storage medium | |
| CN111898161B (en) | Method, system, device and medium for managing authority of HUE | |
| CN103440179A (en) | Method for continuously storing configuration information of storage device | |
| CN113138879A (en) | Method and system for hybrid edge replication | |
| CN118377657B (en) | Data recovery method and device, storage medium and electronic equipment | |
| CN116244040B (en) | Main and standby container cluster system, data synchronization method thereof and electronic equipment | |
| CN116501552B (en) | Data backup method, device, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181106 |
|
| RJ01 | Rejection of invention patent application after publication |