CN110222122A - A kind of method of data synchronization and relevant device of MongoDB - Google Patents
A kind of method of data synchronization and relevant device of MongoDB Download PDFInfo
- Publication number
- CN110222122A CN110222122A CN201910683713.5A CN201910683713A CN110222122A CN 110222122 A CN110222122 A CN 110222122A CN 201910683713 A CN201910683713 A CN 201910683713A CN 110222122 A CN110222122 A CN 110222122A
- Authority
- CN
- China
- Prior art keywords
- export
- data
- mongodb
- synchronization
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Abstract
This application discloses the method for data synchronization of MongoDB a kind of, device, electronic equipment and computer readable storage medium, the method for data synchronization of the MongoDB includes: to export the data of MongoDB to generate export；The synchronous script file of synchronization means is generated with the configuration parameter in preset data warehouse according to the file parameters of export；Synchronous script file is executed to call synchronization means that export is synchronized to preset data warehouse.The application recycles synchronization means to complete data synchronous task first by data after exporting in MongoDB database.Since the data export operation to MongoDB database not will cause a large amount of data query accumulation, the operation and data export of MongoDB database will not be influenced on the data simultaneously operating local rather than online export, therefore, the application can avoid impacting the performance of database, and effectively improve data synchronization efficiency in turn.
Description
Technical field
This application involves data synchronization technology field, in particular to a kind of method of data synchronization of MongoDB, device, electricity
Sub- equipment and computer readable storage medium.
Background technique
MongoDB is the PostgreSQL database system write by C Plus Plus, stored based on distributed document.
MongoDB is it is intended that WEB application provides expansible high-performance data storage solution passes through in high-load situations, this
Add more nodes, it is ensured that server performance.It stores data as a document, data structure by key assignments (key=>
Value) to composition, document is similar to JSON (JavaScript Object Notation, JS object numbered musical notation) object, field value
It may include other documents, array and document array.
In practical applications, as problem of data safety is increasingly important, the data synchronous working of MongoDB has very heavy
The meaning wanted.Mostly in the prior art is to be directly connected to MongoDB with some synchronization means, in a manner of on-line synchronous by data from
Cloud data warehouse is directly synchronized in MongoDB.It is easily understood that first to the data simultaneously operating of MongDB database
Dependent on data query, that is, data acquisition operations of database background application, the data that then will acquire again are uploaded to destination address.
Since the destination address in data synchronization process is mostly the data-storage system of access of need to networking, or even source in some cases
Location is also, therefore data-moving process is not only confined to related hardware performance, is also confined to Internet resources.It is existing as a result,
A large amount of data query often is generated to database when inquiry obtains data to be synchronized from MongDB in technology to accumulate, and is made
It is slow to obtain data no-load speed, and the database performance of MongoDB is enabled sharply to decline, will affect operation system when serious just
Chang Gongneng, or even need to restart database and could restore.
It has been that those skilled in the art need to pay close attention in consideration of it, providing a kind of scheme for solving above-mentioned technical problem
's.
Summary of the invention
The method of data synchronization for being designed to provide a kind of MongoDB, device, electronic equipment and the computer of the application can
Storage medium is read, to effectively improve data synchronization efficiency, and avoid while smoothly completing data synchronous task
The database performance of MongoDB is affected, it is ensured that the normal operation of operation system.
In order to solve the above technical problems, in a first aspect, being wrapped this application discloses the method for data synchronization of MongoDB a kind of
It includes:
The data of MongoDB are exported to generate export；
The synchronous of synchronization means is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
Script file；
The synchronous script file is executed to call the synchronization means that the export is synchronized to described preset
Data warehouse.
Optionally, before the data by MongoDB are exported to generate export, further includes:
The file type of the export is determined according to the data magnitude of MongoDB.
Optionally, the data by MongoDB are exported to generate export, comprising:
Synchronous export is completed in deletion；
The incremental data of MongoDB is exported to generate export.
Optionally, the file parameters according to the export are synchronous with the generation of the configuration parameter in preset data warehouse
The synchronization script file of tool, comprising:
The synchronous of dataX tool is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
Script file.
It is optionally, described to call the synchronization means that the export is synchronized to the preset data warehouse, comprising:
Call the dataX tool that the export is synchronized to MaxCompute.
Optionally, the file parameters according to the export are synchronous with the generation of the configuration parameter in preset data warehouse
The synchronization script file of tool, comprising:
According to the login account of the store path of the export, coding and column set and the preset data warehouse
Number, login password, item number, table number, partition number, generate the synchronization script file of the synchronization means.
Second aspect, disclosed herein as well is the data synchronization units of MongoDB a kind of, comprising:
Export module, for exporting the data of MongoDB to generate export；
Configuration module, it is same for being generated according to the file parameters of the export and the configuration parameter in preset data warehouse
The synchronization script file of step tool；
Synchronization module calls the synchronization means that the export is same for executing the synchronous script file
It walks to the preset data warehouse.
It optionally, further include determining module, for exporting the data export of MongoDB to generate in the export module
Before file, the file type of the export is determined according to the data magnitude of MongoDB.
Optionally, the export module is specifically used for:
Synchronous export is completed in deletion；The incremental data of MongoDB is exported to generate export.
Optionally, the configuration module is specifically used for:
The synchronous of dataX tool is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
Script file.
Optionally, the synchronization module is specifically used for:
Call the dataX tool that the export is synchronized to MaxCompute.
Optionally, the configuration module is specifically used for:
According to the login account of the store path of the export, coding and column set and the preset data warehouse
Number, login password, item number, table number, partition number, generate the synchronization script file of the synchronization means.
The third aspect, disclosed herein as well is a kind of electronic equipment, comprising:
Memory, for storing computer program；
Processor, for executing the computer program to realize following steps:
The data of MongoDB are exported to generate export；According to the file parameters and present count of the export
The synchronization parameter of synchronization means is generated according to the configuration parameter in warehouse；The synchronous script file is executed to call the synchronous work
The export is synchronized to the preset data warehouse by tool.
Optionally, when the processor executes the computer subprogram saved in the memory, can implement with
Lower step: the file type of the export is determined according to the data magnitude of MongoDB.
Optionally, when the processor executes the computer subprogram saved in the memory, can implement with
Lower step: synchronous export is completed in deletion；The incremental data of MongoDB is exported to generate export.
Optionally, when the processor executes the computer subprogram saved in the memory, can implement with
Lower step: the synchronous of dataX tool is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
Script file.
Optionally, when the processor executes the computer subprogram saved in the memory, can implement with
Lower step: call the dataX tool that the export is synchronized to MaxCompute.
Optionally, when the processor executes the computer subprogram saved in the memory, can implement with
Lower step: according to the login account of the store path of the export, coding and column set and the preset data warehouse
Number, login password, item number, table number, partition number, generate the synchronization script file parameter of the synchronization means.
Fourth aspect, disclosed herein as well is a kind of computer readable storage medium, the computer readable storage medium
In be stored with computer program, to realize following steps when the computer program is executed by processor:
The data of MongoDB are exported to generate export；According to the file parameters and present count of the export
The synchronization script file of synchronization means is generated according to the configuration parameter in warehouse；It is described same to call to execute the synchronous script file
The export is synchronized to the preset data warehouse by step tool.
It optionally, can be with when the computer subprogram saved in the computer readable storage medium is executed by processor
Specific implementation following steps: the file type of the export is determined according to the data magnitude of MongoDB.
It optionally, can be with when the computer subprogram saved in the computer readable storage medium is executed by processor
Specific implementation following steps: synchronous export is completed in deletion；By the incremental data export of MongoDB to generate export text
Part.
It optionally, can be with when the computer subprogram saved in the computer readable storage medium is executed by processor
It implements following steps: dataX is generated according to the configuration parameter in the file parameters of the export and preset data warehouse
The synchronization script file of tool.
It optionally, can be with when the computer subprogram saved in the computer readable storage medium is executed by processor
Specific implementation following steps: call the dataX tool that the export is synchronized to MaxCompute.
It optionally, can be with when the computer subprogram saved in the computer readable storage medium is executed by processor
Implement following steps: according to the store path of the export, coding and column set and the preset data warehouse
Login account, login password, item number, table number, partition number, generate the synchronization script file of the synchronization means.
The method of data synchronization of MongoDB provided herein includes: the data export by MongoDB to generate export
File；The synchronous script of synchronization means is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
File；The synchronous script file is executed to call the synchronization means that the export is synchronized to the preset data
Warehouse.
As it can be seen that the application first by data after exporting in MongoDB database, recycles synchronization means to complete data synchronous
Task.Since the data export operation to MongoDB database will not cause a large amount of data query heap to MongoDB database
Product, to the data simultaneously operating local rather than online export also with MongoDB database without direct correlation, will not influence
The operation of MongoDB database and data export, therefore, the application, which can effectively avoid, impacts the performance of database, protects
Hinder the normal operation of business function, and effectively improves data synchronization efficiency in turn.The data of MongoDB provided herein are same
Step device, electronic equipment and computer readable storage medium equally have above-mentioned beneficial effect.
Detailed description of the invention
In order to illustrate more clearly of the technical solution in the prior art and the embodiment of the present application, below will to the prior art and
Attached drawing to be used is needed to make brief introduction in the embodiment of the present application description.
Fig. 1 is a kind of flow chart of the method for data synchronization of MongoDB disclosed in the embodiment of the present application；
Fig. 2 is the flow chart of the method for data synchronization of another MongoDB disclosed in the embodiment of the present application；
Fig. 3 is a kind of flow chart of the data synchronization unit of MongoDB disclosed in the embodiment of the present application；
Fig. 4 is the structural block diagram of a kind of electronic equipment disclosed in the embodiment of the present application.
Specific embodiment
The core of the application is that the method for data synchronization, device, electronic equipment and the computer that provide a kind of MongoDB can
Storage medium is read, to effectively improve data synchronization efficiency, and avoid while smoothly completing data synchronous task
The database performance of MongoDB is affected, it is ensured that the normal operation of operation system.
In order to which technical solutions in the embodiments of the present application is more clearly and completely described, below in conjunction with this Shen
Please attached drawing in embodiment, technical solutions in the embodiments of the present application is introduced.Those of ordinary skill in the art are not having
Every other embodiment obtained under the premise of creative work is made, shall fall in the protection scope of this application.
Currently, when synchronous to MongoDB database progress data, mostly in the prior art is directly to synchronize work using data
Tool connection MongoDB database so can generate a large amount of read operation to MongoDB database, to can produce to export data
Raw a large amount of data query accumulation, causes database performance to substantially reduce, or even will affect the normal function of operation system.In order to
The performance issue from MongoDB database synchronization mass data is solved, this application provides the data sides of synchronization of MongoDB a kind of
Method can effectively solve the above problems.
Shown in Figure 1, the embodiment of the present application discloses the method for data synchronization of MongoDB a kind of, specifically includes that
S11: the data of MongoDB are exported to generate export.
As it was noted above, MongoDB is the open source data write by C Plus Plus, stored based on distributed document
Library system, it is intended to which expansible high-performance data storage solution is provided for WEB application.In high-load situations, this, pass through
Add more nodes, it is ensured that server performance.It should be pointed out that the number of MongoDB provided by the embodiment of the present application
According to synchronous method, specifically first by data after exporting in MongoDB database, recycling synchronization means to complete, data are synchronous to appoint
Business.
It is accumulated, is influenced since using on-line synchronous method a large amount of data query can be generated to MongoDB database
The service operation performance of MongoDB database, therefore, the embodiment of the present application use offline synchronization method: first by MongoDB number
It is exported according to the data in library, operation then is synchronized to derived data again, to avoid influencing MongoDB database
Runnability.Export generated after data export, the server that can be specifically stored in where MongoDB database
On.It is accumulated it is easily understood that data export operation can't generate a large amount of data query to MongoDB database, thus
It can effectively ensure that the service feature of MongoDB database is not affected.
It should also be noted that, the export generated after export is preferably kept in local, i.e. service where MongoDB
Device, to further decrease the influence to MongoDB database performance.
S12: the synchronous of synchronization means is generated with the configuration parameter in preset data warehouse according to the file parameters of export
Script file.
Specifically, export is object to be synchronized, and preset data warehouse is object synchronization address, then the text of export
Part parameter and the configuration parameter in preset data warehouse are the synchronization parameters of synchronization means.By the file parameters of export and
Synchronous script file is written that is, the synchronization parameter to synchronization means configures in the configuration parameter in preset data warehouse.
It should also be noted that, synchronous script file is specifically generated using automatized script Core Generator, synchronous foot
The implementation procedure of step file is the synchronous implementation procedure of data.
Further, synchronous step file is specifically as follows JSON file, and may particularly include two parts: the portion reader
Divide and the part writer.Wherein, the part reader can recorde the file parameters of export, such as store path, coding, column
Set etc., so that synchronization means reads export.The part writer can recorde the configuration parameter in preset data warehouse, such as
It may include login account, login password, item number, table number, partition number etc., so that data are synchronized to present count by synchronization means
According to the target area in warehouse.
It is used as a kind of specific embodiment as a result, step S12 can be with specifically: according to the store path of export, compile
Code and column set and the login account in preset data warehouse, login password, item number, table number, partition number, generate synchronous work
The synchronization script file of tool.
Certainly, those skilled in the art can be selected according to practical situations and voluntarily export file parameters and
The specific type of the configuration parameter in preset data warehouse, and further generate the corresponding synchronous script file of synchronization means, this Shen
Please embodiment this is not further limited.
S13: synchronous script file is executed to call synchronization means that export is synchronized to preset data warehouse.
Synchronization means is called by executing synchronous script file, according to the synchronization parameter set in synchronous script file,
The data of export can be synchronized in the corresponding region in preset data warehouse, to realize that the data of MongoDB are synchronous.
The method of data synchronization of MongoDB provided by the embodiment of the present application includes: to export the data of MongoDB with life
At export；The synchronous foot of synchronization means is generated with the configuration parameter in preset data warehouse according to the file parameters of export
This document；Synchronous script file is executed to call synchronization means that export is synchronized to preset data warehouse.
As it can be seen that the application first by data after exporting in MongoDB database, recycles synchronization means to complete data synchronous
Task.Since the data export operation to MongoDB database will not cause a large amount of data query heap to MongoDB database
Product, to the data simultaneously operating local rather than online export also with MongoDB database without direct correlation, will not influence
The operation of MongoDB database and data export, therefore, the application, which can effectively avoid, impacts the performance of database, protects
Hinder the normal operation of business function, and effectively improves data synchronization efficiency in turn.
Shown in Figure 2, the embodiment of the present application discloses the method for data synchronization of another MongoDB, specifically includes that
S21: the file type of export is determined according to the data magnitude of MongoDB.
It should be pointed out that can first determine that the file type of export before turn-on data synchronous task.Specifically
Both export lattice of ground, MongoDB database support CSV (Comma-Separated Values, character separation value) and JSON
Formula, wherein the former has higher space utilization rate.
In this way, if the data magnitude of MongoDB is larger, it more than for example, ten million rank, then can be by the file of export
Type is selected as csv file.CSV is a kind of general, relatively simple file format, stores table number with plain text
According to by user, business and science extensive use.Csv file most widely application be shift list data between programs, and this
A little programs itself are can be operated on incompatible format.
S22: synchronous export is completed in deletion.
S23: the incremental data of MongoDB is exported to generate export.
It should be pointed out that data synchronous task needs periodically constantly to execute with the continuous generation and update of data.Cause
This, can be just for the incremental data in MongoDB database every time when progress data are synchronous after synchronizing cycle has been determined
It synchronizes.Also, before exporting incremental data, synchronous export can will have first been completed in synchronizing cycle last time and deleted
It removes.
S24: the synchronous of dataX tool is generated with the configuration parameter in preset data warehouse according to the file parameters of export
Script file.
It should be pointed out that synchronization means used in the present embodiment is specifically as follows dataX tool.DataX be Ah
In the off-line data synchronization means of a kind of open source that provides, it can realize including MySQL, Oracle, SqlServer,
The various isomery numbers such as Postgre, HDFS, Hive, ADS, HBase, TableStore (OTS), MaxCompute (ODPS), DRDS
According to data synchronizing function efficient between source.Certainly, other synchronization means also can be selected in those skilled in the art, and the application is to this
It does not further limit.
S25: synchronous script file is executed to call dataX tool that export is synchronized to MaxCompute.
It should be pointed out that preset data warehouse used in the present embodiment is specifically as follows MaxCompute.
MaxCompute is that a kind of big data that Ali provides calculates service, is a kind of EB grade data warehouse solution of quick, complete trustship
Certainly scheme.Certainly, those skilled in the art also can be selected other and synchronize storage service as preset data warehouse, and the application is to this
It does not further limit.
It should also be noted that, above-mentioned steps S21 only need to execute primary initial, and step S22~S25 then can be
It is repeated in synchronizing cycle.Specifically, those skilled in the art can be taken by writing shell script file, and using Linux
The crontab of business device is the task that the timing of shell script design executes, anti-will pass through operation shell script file repeatedly
Step S22~S25 is executed again, realizes the purpose of timed periodic synchrodata.
Shown in Figure 3, the embodiment of the present application discloses the data synchronization unit of MongoDB a kind of, specifically includes that
Export module 301, for exporting the data of MongoDB to generate export；
Configuration module 302, it is same for being generated according to the file parameters of export and the configuration parameter in preset data warehouse
The synchronization script file of step tool；
Synchronization module 303 calls synchronization means to be synchronized to export default for executing synchronous script file
Data warehouse.
The particular content of data synchronization unit about above-mentioned MongoDB, the data that can refer to previously with regard to MongoDB are same
Being discussed in detail for one step process, just repeats no more here.
As it can be seen that the data synchronization unit of MongoDB disclosed in the embodiment of the present application, first by data from MongoDB data
After exporting in library, synchronization means is recycled to complete data synchronous task.Since the data export operation of MongoDB database will not
A large amount of data query is caused to accumulate MongoDB database, to the data simultaneously operating local rather than online export
Also the operation and data export of MongoDB database, therefore, the application be will not influence without direct correlation with MongoDB database
It can effectively avoid and the performance of database is impacted, ensure the normal operation of business function, and it is same to effectively improve data in turn
Walk efficiency.
On the basis of the above, further, the data synchronization unit of MongoDB disclosed in the embodiment of the present application
In a specific embodiment, further includes:
Determining module, for before export module 301 exports the data of MongoDB to generate export, according to
The data magnitude of MongoDB determines the file type of export.
On the basis of the above, further, the data synchronization unit of MongoDB disclosed in the embodiment of the present application
In a specific embodiment, export module 301 is specifically used for:
Synchronous export is completed in deletion；The incremental data of MongoDB is exported to generate export.
On the basis of the above, further, the data synchronization unit of MongoDB disclosed in the embodiment of the present application
In a specific embodiment, configuration module 302 is specifically used for:
The synchronous script of dataX tool is generated with the configuration parameter in preset data warehouse according to the file parameters of export
File.
On the basis of the above, further, the data synchronization unit of MongoDB disclosed in the embodiment of the present application
In a specific embodiment, synchronization module 303 is specifically used for:
Synchronous script file is executed to call dataX tool that export is synchronized to MaxCompute.
On the basis of the above, further, the data synchronization unit of MongoDB disclosed in the embodiment of the present application
In a specific embodiment, configuration module 302 is specifically used for:
According to the store path of export, coding and column set and preset data warehouse login account, log in it is close
Code, item number, table number, partition number generate the synchronization script file of synchronization means.
Shown in Figure 4, the embodiment of the present application discloses a kind of electronic equipment, comprising:
Memory 401, for storing computer program；
Processor 402, for executing the computer program to realize following steps:
The data of MongoDB are exported to generate export；According to the file parameters and present count of the export
The synchronization script file of synchronization means is generated according to the configuration parameter in warehouse；Synchronous script file is executed to call the synchronous work
The export is synchronized to the preset data warehouse by tool.
It in a specific embodiment, can be with when processor 402 executes the computer subprogram saved in memory 401
Specific implementation following steps: the file type of the export is determined according to the data magnitude of MongoDB.
It in a specific embodiment, can be with when processor 402 executes the computer subprogram saved in memory 401
Specific implementation following steps: synchronous export is completed in deletion；By the incremental data export of MongoDB to generate export text
Part.
It in a specific embodiment, can be with when processor 402 executes the computer subprogram saved in memory 401
It implements following steps: dataX is generated according to the configuration parameter in the file parameters of the export and preset data warehouse
The synchronization script file of tool.
It in a specific embodiment, can be with when processor 402 executes the computer subprogram saved in memory 401
Specific implementation following steps: synchronous script file is executed to call the dataX tool to be synchronized to the export
MaxCompute。
It in a specific embodiment, can be with when processor 402 executes the computer subprogram saved in memory 401
Implement following steps: according to the store path of the export, coding and column set and the preset data warehouse
Login account, login password, item number, table number, partition number, generate the synchronization script file of the synchronization means.
Further, the embodiment of the present application also discloses a kind of computer readable storage medium, described computer-readable to deposit
Computer program is stored in storage media, to realize following steps when the computer program is executed by processor:
The data of MongoDB are exported to generate export；According to the file parameters and present count of the export
The synchronization script file of synchronization means is generated according to the configuration parameter in warehouse；Synchronous script file is executed to call the synchronous work
The export is synchronized to the preset data warehouse by tool.
In a specific embodiment, the computer subprogram saved in the computer readable storage medium is by processor
When execution, following steps can be implemented: determine the file type of the export according to the data magnitude of MongoDB.
In a specific embodiment, the computer subprogram saved in the computer readable storage medium is by processor
When execution, can implement following steps: synchronous export is completed in deletion；The incremental data of MongoDB is exported
To generate export.
In a specific embodiment, the computer subprogram saved in the computer readable storage medium is by processor
When execution, following steps can be implemented: be joined according to the configuration in the file parameters of the export and preset data warehouse
Number generates the synchronization script file of dataX tool.
In a specific embodiment, the computer subprogram saved in the computer readable storage medium is by processor
When execution, following steps can be implemented: executing synchronous script file to call the dataX tool by the export text
Part is synchronized to MaxCompute.
In a specific embodiment, the computer subprogram saved in the computer readable storage medium is by processor
When execution, following steps can be implemented: be gathered according to the store path of the export, coding and column and described
The login account in preset data warehouse, login password, item number, table number, partition number generate the synchronization script of the synchronization means
File.
About the particular content of above-mentioned electronic equipment and computer readable storage medium, can refer to previously with regard to MongoDB
Method of data synchronization be discussed in detail, here just repeat no more.
Each embodiment is described in a progressive manner in the application, the highlights of each of the examples are with other realities
The difference of example is applied, the same or similar parts in each embodiment may refer to each other.For equipment disclosed in embodiment
Speech, since it is corresponded to the methods disclosed in the examples, so being described relatively simple, related place is referring to method part illustration
?.
It should be noted that in present specification, the relational terms of such as " first " and " second " etc are used merely to
One entity or operation and another entity or operate is distinguished, without necessarily requiring or implying these entities or
There are any actual relationship or orders between person's operation.In addition, the terms "include", "comprise" or its any other
Variant is intended to non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only
It including those elements, but also including other elements that are not explicitly listed, or further include for this process, method, object
Product or the intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...",
It is not precluded in the process, method, article or apparatus that includes the element that there is also other identical elements.
Technical solution provided herein is described in detail above.Specific case used herein is to this Shen
Principle and embodiment please is expounded, the present processes that the above embodiments are only used to help understand and its
Core concept.It should be pointed out that for those skilled in the art, in the premise for not departing from the application principle
Under, can also to the application, some improvement and modification can also be carried out, these improvement and modification are also fallen into the protection scope of the application.
Claims (10)
1. a kind of method of data synchronization of MongoDB characterized by comprising
The data of MongoDB are exported to generate export；
The synchronous script of synchronization means is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
File；
The synchronous script file is executed to call the synchronization means that the export is synchronized to the preset data
Warehouse.
2. the method for data synchronization of MongoDB according to claim 1, which is characterized in that in the number by MongoDB
According to export to generate export before, further includes:
The file type of the export is determined according to the data magnitude of MongoDB.
3. the method for data synchronization of MongoDB according to claim 1, which is characterized in that the data by MongoDB
Export is to generate export, comprising:
Synchronous export is completed in deletion；
The incremental data of MongoDB is exported to generate export.
4. the method for data synchronization of MongoDB according to claim 1, which is characterized in that described according to the export text
The file parameters of part generate the synchronous script file of synchronization means with the configuration parameter in preset data warehouse, comprising:
The synchronous script of dataX tool is generated with the configuration parameter in preset data warehouse according to the file parameters of the export
File.
5. the method for data synchronization of MongoDB according to claim 4, which is characterized in that described to call the synchronous work
The export is synchronized to the preset data warehouse by tool, comprising:
Call the dataX tool that the export is synchronized to MaxCompute.
6. the method for data synchronization of MongoDB according to any one of claims 1 to 5, which is characterized in that described according to institute
The file parameters for stating export generate the synchronous script file of synchronization means with the configuration parameter in preset data warehouse, comprising:
According to the login account in the store path of the export, coding and column set and the preset data warehouse, step on
Password, item number, table number, partition number are recorded, the synchronization script file of the synchronization means is generated.
7. a kind of data synchronization unit of MongoDB characterized by comprising
Export module, for exporting the data of MongoDB to generate export；
Configuration module, for the file parameters work synchronous with the generation of the configuration parameter in preset data warehouse according to the export
The synchronization script file of tool；
Synchronization module calls the synchronization means to be synchronized to the export for executing the synchronous script file
The preset data warehouse.
8. the data synchronization unit of MongoDB according to claim 7, which is characterized in that further include:
Determining module, for before the export module exports the data of MongoDB to generate export, according to
The data magnitude of MongoDB determines the file type of the export.
9. a kind of electronic equipment characterized by comprising
Memory, for storing computer program；
Processor, for executing the computer program to realize the number such as MongoDB as claimed in any one of claims 1 to 6
The step of according to synchronous method.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer in the computer readable storage medium
Program, to realize such as MongoDB as claimed in any one of claims 1 to 6 when the computer program is executed by processor
The step of method of data synchronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910683713.5A CN110222122A (en) | 2019-07-26 | 2019-07-26 | A kind of method of data synchronization and relevant device of MongoDB |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910683713.5A CN110222122A (en) | 2019-07-26 | 2019-07-26 | A kind of method of data synchronization and relevant device of MongoDB |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110222122A true CN110222122A (en) | 2019-09-10 |
Family
ID=67812668
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910683713.5A Pending CN110222122A (en) | 2019-07-26 | 2019-07-26 | A kind of method of data synchronization and relevant device of MongoDB |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110222122A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061798A (en) * | 2019-12-23 | 2020-04-24 | 杭州雷数科技有限公司 | Configurable data transmission and monitoring method, equipment and medium |
| CN111552746A (en) * | 2020-04-30 | 2020-08-18 | 北京思特奇信息技术股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN112860792A (en) * | 2021-01-29 | 2021-05-28 | 银盛支付服务股份有限公司 | Method and system for synchronizing mongodb cluster data and hive cluster data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317843A (en) * | 2014-10-11 | 2015-01-28 | 上海瀚之友信息技术服务有限公司 | Data synchronism ETL (Extract Transform Load) system |
| CN105847378A (en) * | 2016-04-13 | 2016-08-10 | 北京思特奇信息技术股份有限公司 | Big data synchronizing method and system |
| WO2018126964A1 (en) * | 2017-01-04 | 2018-07-12 | 中兴通讯股份有限公司 | Task execution method and apparatus and server |
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
| CN109460439A (en) * | 2018-11-14 | 2019-03-12 | 泰康保险集团股份有限公司 | A kind of data processing method, device, medium and electronic equipment |
-
2019
- 2019-07-26 CN CN201910683713.5A patent/CN110222122A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317843A (en) * | 2014-10-11 | 2015-01-28 | 上海瀚之友信息技术服务有限公司 | Data synchronism ETL (Extract Transform Load) system |
| CN105847378A (en) * | 2016-04-13 | 2016-08-10 | 北京思特奇信息技术股份有限公司 | Big data synchronizing method and system |
| WO2018126964A1 (en) * | 2017-01-04 | 2018-07-12 | 中兴通讯股份有限公司 | Task execution method and apparatus and server |
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
| CN109460439A (en) * | 2018-11-14 | 2019-03-12 | 泰康保险集团股份有限公司 | A kind of data processing method, device, medium and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| 卢至彤等: "一种MongoDB应用优化策略", 《计算机系统应用》 * |
| 卢至彤等: "一种MongoDB应用优化策略", 《计算机系统应用》, no. 05, 15 May 2017 (2017-05-15) * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061798A (en) * | 2019-12-23 | 2020-04-24 | 杭州雷数科技有限公司 | Configurable data transmission and monitoring method, equipment and medium |
| CN111061798B (en) * | 2019-12-23 | 2024-03-08 | 杭州雷数科技有限公司 | Configurable data transmission and monitoring method, equipment and medium |
| CN111552746A (en) * | 2020-04-30 | 2020-08-18 | 北京思特奇信息技术股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN111552746B (en) * | 2020-04-30 | 2023-04-28 | 北京思特奇信息技术股份有限公司 | Data synchronization method, device, electronic equipment and storage medium |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN112860792A (en) * | 2021-01-29 | 2021-05-28 | 银盛支付服务股份有限公司 | Method and system for synchronizing mongodb cluster data and hive cluster data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110222122A (en) | A kind of method of data synchronization and relevant device of MongoDB | |
| CN103902593B (en) | A kind of method and apparatus of Data Migration | |
| Ke et al. | On traffic-aware partition and aggregation in mapreduce for big data applications | |
| CN104572689A (en) | Data synchronizing method, device and system | |
| CN104881466B (en) | The processing of data fragmentation and the delet method of garbage files and device | |
| CN103473287A (en) | Method and system for automatically distributing, running and updating executable programs | |
| CN106445643B (en) | It clones, the method and apparatus of upgrading virtual machine | |
| Labouseur et al. | Scalable and Robust Management of Dynamic Graph Data. | |
| CN112099800B (en) | Code data processing method, device and server | |
| CN105677904B (en) | Small documents storage method and device based on distributed file system | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| CN102779138A (en) | Hard disk access method of real time data | |
| JP2011170667A (en) | File-synchronizing system, file synchronization method, and file synchronization program | |
| CN104765651A (en) | Data processing method and device | |
| Sheng et al. | GraPU: Accelerate streaming graph analysis through preprocessing buffered updates | |
| Liu et al. | On construction of an energy monitoring service using big data technology for smart campus | |
| CN109885535A (en) | A kind of method and relevant apparatus of file storage | |
| CN103778067A (en) | Object processing method and device of Java card and Java card | |
| CN104899161A (en) | Cache method based on continuous data protection of cloud storage environment | |
| Kotilainen et al. | P2PRealm-peer-to-peer network simulator | |
| CN103034655A (en) | Collection method and system of user behavior information and related equipment | |
| CN102685222B (en) | A kind of cloud SRM device for electric power system | |
| CN103970844A (en) | Big data write-in method and device, big data read method and device and big data processing system | |
| CN105023170A (en) | Processing method and device of click stream data | |
| CN109255001A (en) | Maintaining method and device, the electronic equipment in interface instance library |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |