CN110062028A - Data synchronous method, apparatus, computer equipment and computer storage medium - Google Patents
Data synchronous method, apparatus, computer equipment and computer storage medium Download PDFInfo
- Publication number
- CN110062028A CN110062028A CN201910218802.2A CN201910218802A CN110062028A CN 110062028 A CN110062028 A CN 110062028A CN 201910218802 A CN201910218802 A CN 201910218802A CN 110062028 A CN110062028 A CN 110062028A
- Authority
- CN
- China
- Prior art keywords
- data
- paging
- synchrodata
- paged
- synchrodata table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 230000006837 decompression Effects 0.000 claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 19
- 230000006835 compression Effects 0.000 claims description 12
- 238000007906 compression Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 12
- 238000012795 verification Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 9
- 238000013507 mapping Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 241001269238 Data Species 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012216 screening Methods 0.000 description 2
- KCXVZYZYPLLWCC-UHFFFAOYSA-N EDTA Chemical compound OC(=O)CN(CC(O)=O)CCN(CC(O)=O)CC(O)=O KCXVZYZYPLLWCC-UHFFFAOYSA-N 0.000 description 1
- 101100217298 Mus musculus Aspm gene Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 229960001484 edetic acid Drugs 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses a kind of data synchronous method, apparatus, computer equipment and computer storage mediums, are related to cloud storage technical field, and each tables of data after can making paging reduces bandwidth pressure during data are synchronous, improve data synchronization efficiency.The described method includes: obtaining the paged data of business side's configuration；The paged data is compressed in tables of data according to default compressed format, obtains the tables of data to be synchronized of paging compressed file format；The tables of data to be synchronized is synchronized in the target database of server with multithreading paging compressed file format according to pre-set isochronous schedules task, in order to which server carries out decompression processing to the synchrodata table according to default decompression format, each paged data in the synchrodata table is obtained, and paged data separated time journey each in the synchrodata table is written into target database in a manner of ordering load.
Description
Technical field
The present invention relates to cloud storage technical fields, more particularly, to data synchronous method, apparatus, computer equipment and meter
Calculation machine storage medium.
Background technique
The rapid development and information-based propulsion of computer network so that data become the precious resources of many enterprises so that
Data acquisition, storage and processing task are growing day by day, and network-based data sharing has become people and obtains data resource not
The mode that can or lack.
It needs to synchronize shared data in order to which data sharing is better achieved, data synchronize for enterprise to Guan Chong
It wants, for example the server end of enterprise is used to manage the business end for belonging to difference department, enterprise, and is stored in server end
Data are managed, then server end is managed each business end for convenience, while improving the data processing effect at business end
Rate needs to guarantee that the data between server end and each business end are synchronous.
Since the business datum that each business end is contacted is different, it is frequently encountered the very big tables of data of data volume,
And the tables of data very big for data volume, it usually needs very long synchronization time synchronizes entire tables of data, directly affects business
The synchronous efficiency at end.The mode for generalling use separated time journey paging at present carrys out the very big tables of data of synchrodata amount, and will be after paging
Multiple tables of data be concurrently inserted into the object table of server.However, since the very big tables of data of data volume is operationally deposited
In associate feature, although the speed that each data table data after paging synchronizes can increase, however, to ensure that data are same
Efficiency is walked, each tables of data after paging will receive bandwidth limitation during data are synchronous, lead to the number of entire tables of data
It is longer according to synchronization time.
Summary of the invention
In view of this, the present invention provides a kind of synchronous method, apparatus of data, computer equipment and computer storages to be situated between
Matter, it is synchronous in data that main purpose is to solve each tables of data of data volume is very big in presently relevant technology table after paging
During will receive bandwidth limitation the problem of.
According to the present invention on one side, a kind of method that data are synchronous is provided, business end is applied to, this method comprises:
The paged data of acquisition business side configuration；
The paged data is compressed in tables of data according to default compressed format, obtain paging compressed file format to
Synchrodata table；
According to pre-set isochronous schedules task by the tables of data to be synchronized with multithreading paging compressed file format
It is synchronized in the target database of server.
Further, the paged data of the acquisition business side configuration includes:
Need to filter out the synchrodata table that data volume is greater than default value in synchrodata table from business side；
The data being greater than according to default division rule to each data volume in the synchrodata table of default value carry out
Paging processing obtains the paged data of multiple business side's configurations.
Further, the default division rule is that the fixed quantity determined according to transmission bandwidth needs the business side
Data in synchrodata table are divided, described to be greater than default value to each data volume according to default division rule
Data in synchrodata table carry out paging processing, and the paged data for obtaining multiple business side's configurations includes:
Each data volume is counted greater than the data in the synchrodata table of default value, and records the number that statistics obtains
According to amount；
When the data volume that the statistics obtains reaches fixed quantity, the same step number of default value is greater than from the data volume
It is put into paging according to the data for intercepting the fixed quantity in table, and by the data of the fixed quantity, obtains the configuration of business side
Paged data.
According to the present invention on the other hand, a kind of method that data are synchronous is provided, server, the method packet are applied to
It includes:
Receive the synchrodata table of multithreading paging compressed file format；
Decompression processing is carried out to the synchrodata table according to default decompression format, is obtained each in the synchrodata table
Paged data；
Paged data separated time journey each in the synchrodata table is written to target database in a manner of ordering load
In.
Further, after the synchrodata table for receiving multithreading paging compressed file format, the method is also
Include:
Consistency checking is carried out to the synchrodata table according to the file identification carried in the synchrodata table；
It is to have synchronized by the synchrodata list notation if verification result is consistent；If verification result is inconsistent,
It is not synchronize by the synchrodata list notation.
Further, the file identification includes each paging compressed file generates in the synchrodata table multiple points
All paging compressed files summarize the entirety mark of generation in page mark and the synchrodata table, described according to
The file identification carried in synchrodata table carries out consistency checking to the synchrodata table
According to the entirety mark in the synchrodata table, all paging compressed files in the synchrodata table are summarized
Carry out consistency checking；
According to the paging mark in the synchrodata table, corresponding paging pressure is identified to paging in the synchrodata table
Contracting file carries out consistency checking.
Further, described that paged data separated time journey each in the synchrodata table is written in a manner of ordering load
Include: into target database
The target table of the corresponding write-in of each paged data, the mapping directory are searched from the mapping directory being pre-created
In be stored with the location information of each paged data target table in target database in synchrodata table；
Paged data separated time journey each in the synchrodata table is written to each in a manner of order load
The target table of the corresponding write-in of paged data.
According to the present invention on the other hand, a kind of business end is provided, which includes:
Acquiring unit, for obtaining the paged data of business side's configuration；
Compression unit obtains paging pressure for the paged data to be compressed in tables of data according to default compressed format
The tables of data to be synchronized of contracting file format；
Synchronization unit is used for the tables of data to be synchronized according to pre-set isochronous schedules task with multithreading paging
Compressed file format is synchronized in the target database of server.
Further, the acquiring unit includes:
Screening module, for needing to filter out the same step number that data volume is greater than default value in synchrodata table from business side
According to table,；
Pagination module, for being greater than the synchrodata table of default value to each data volume according to default division rule
In data carry out paging processing, obtain the paged data of multiple business sides configuration.
Further, the default division rule is that the fixed quantity determined according to transmission bandwidth needs the business side
Data in synchrodata table are divided,
The pagination module, specifically for counting each data volume greater than the number in the synchrodata table of default value
According to, and record the data volume that statistics obtains；
The pagination module is specifically also used to when the data volume that the statistics obtains reaches fixed quantity, from the number
It is greater than in the synchrodata table of default value according to amount and intercepts the data of the fixed quantity, and the data of the fixed quantity is put
Enter in paging, obtains the paged data of business side's configuration.
According to the present invention on the other hand, a kind of server is provided, which includes:
Receiving unit, for receiving the synchrodata table of multithreading paging compressed file format；
Decompression units obtain described same for carrying out decompression processing to the synchrodata table according to default decompression format
Each paged data in step data table；
Writing unit, for paged data separated time journey each in the synchrodata table to be written in a manner of ordering load
Into target database.
Further, described device further include:
Authentication unit, for it is described receive multithreading paging compressed file format synchrodata table after, according to institute
It states the file identification carried in synchrodata table and consistency checking is carried out to the synchrodata table；
The synchrodata list notation is to have synchronized if consistent specifically for verification result by the authentication unit；
The synchrodata list notation is not if it is inconsistent to be specifically also used to verification result by the authentication unit
It is synchronous.
Further, the file identification includes each paging compressed file generates in the synchrodata table multiple points
All paging compressed files summarize the entirety mark of generation, the authentication unit in page mark and the synchrodata table
Include:
First authentication module, for being identified according to the entirety in the synchrodata table, to institute in the synchrodata table
There is paging compressed file to summarize carry out consistency checking；
Second authentication module, for being identified according to the paging in the synchrodata table, in the synchrodata table points
Page identifies corresponding paging compressed file and carries out consistency checking.
Further, said write unit includes:
Searching module, for searching the object table of the corresponding write-in of each paged data from the mapping directory being pre-created
Lattice are stored with the position of each paged data target table in target database in synchrodata table in the mapping directory
Information；
Writing module, in a manner of order load by paged data separated time each in the synchrodata table
Journey is written to the target table of the corresponding write-in of each paged data.
Another aspect according to the present invention provides a kind of computer equipment, including memory and processor, the storage
The step of device is stored with computer program, and the processor realizes data synchronous method when executing the computer program.
Another aspect according to the present invention provides a kind of computer storage medium, is stored thereon with computer program, institute
The step of stating the method that realization data synchronize when computer program is executed by processor.
By above-mentioned technical proposal, the present invention provides a kind of data synchronous method and device, by business end according to
The paged data that default compressed format configures business side is compressed in tables of data, is not had to batch insertion operation, but will be synchronized
Data are transferred to server with multithreading paging compressed file format, greatly reduce the big table of data volume and carry out what data synchronized
Target database is written the paged data decompressed in server separated time journey, sufficiently in time in a manner of ordering load
The characteristics of using server high concurrent, synchrodata is loaded onto the target data of server at faster speed.With it is existing
The data method of synchronization that each tables of data in technology after paging is limited in bandwidth in data synchronization process is compared, the present invention
Synchrodata is sent to server by the way of paging, high frequency, multithreading compression at business end by embodiment, so that paging
Data afterwards are not bandwidth limited in synchronizing process, improve data synchronization efficiency, and in server using multithreading decompression, life
It enables the mode of load that paged data is written into target data, improves data write efficiency.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows a kind of synchronous method flow schematic diagram of data provided in an embodiment of the present invention；
Fig. 2 shows the synchronous method flow schematic diagrams of another data provided in an embodiment of the present invention；
Fig. 3 shows the synchronous method flow schematic diagram of another data provided in an embodiment of the present invention；
Fig. 4 shows a kind of structural schematic diagram at business end provided in an embodiment of the present invention；
Fig. 5 shows the structural schematic diagram at another business end provided in an embodiment of the present invention；
Fig. 6 shows a kind of structural schematic diagram of server provided in an embodiment of the present invention；
Fig. 7 shows the structural schematic diagram of another server provided in an embodiment of the present invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
The embodiment of the invention provides a kind of synchronous methods of data, apply at business end, every after can making paging
A tables of data reduces bandwidth pressure during data are synchronous, improves data synchronization efficiency, as shown in Figure 1, this method comprises:
101, the paged data of business side's configuration is obtained.
Wherein, the paged data of business side's configuration can be the data of storage in database or operation process generation, for example,
The production data that server generates, the user information data etc. of client input can also be the day of record user behavior in platform
Will data, for example, user browses the data of webpage, user buys the data etc. of commodity, here without limiting.
Since business side needs synchronous data to can be potentially encountered the king-sized tables of data of data volume, it is generally the case that industry
Business side, can be to needing synchronous data to detect when needing synchrodata, and detection needs whether synchronous data volume is greater than
Preset quantity, and then need to filter out the synchrodata table that data volume is greater than default value in synchrodata table from business side, it should
Preset quantity needs be arranged it is higher, such as 1000 or 10000, the business side that can just withdraw deposit in this way needs synchrodata table
Data volume is big, for business side need synchrodata quantity it is larger in the case of, be not suitable for directly carry out data synchronization, need
Paging processing is carried out to data, each data volume is greater than in the synchrodata table of default value according to default division rule here
Data carry out paging processing, the paged data of multiple business sides configuration is obtained, due to each paging generated after paging processing
The data volume that data include is smaller, can alleviate business number formulary according to the data transmission pressure in synchronizing process.
For the embodiment of the present invention, the fixed quantity that default division rule can be determined according to transmission bandwidth needs business side
The data in synchrodata table are wanted to be divided, the synchrodata that specifically can be greater than default value by counting each data volume
Data in table, and the data volume that statistics obtains is recorded, when the data volume that statistics obtains reaches fixed quantity, it is greater than from quantity
The data that fixed quantity is intercepted in the synchrodata table of default value, the data of fixed quantity are put into a paging, are obtained
The paged data of business side's configuration.
It should be noted that above-mentioned default division rule is that the fixed quantity determined according to transmission bandwidth is divided, this
In preset division rule and can also be divided according to data type, can be combined with data type and transmission bandwidth certainly and determine
Fixed quantity divided, for example, preferentially according to data type to needing the data in synchrodata table to divide, so
The data under every kind of data type in tables of data are scanned afterwards, if after the data volume in scan data table reaches fixed quantity,
The data for belonging to the fixed quantity of same book type are put into a paging, guarantee the data type phase of data in each paging
Together, and then multiple paged datas of different types of data are obtained.
102, the paged data is compressed in tables of data according to default compressed format, obtains paging compressed file format
Tables of data to be synchronized.
Specifically default compressed format can be determined by analyzing the feature of paged data, for example, if in paged data
It repeats in the presence of a large amount of in byte, default compressed format can be for the byte-code compression in paged data, using the lossless pressure of RLE
Contracting, by byte-code compression duplicate in the field of paged data into tables of data, if there are a large amount of text in paged data, it can
Think for the Text compression in paged data, using LZ lossless compression, by the Text compression in paged data into tables of data,
Here to the default compressed format of paged data without limiting.
To the embodiment of the present invention, by the way that paged data to be compressed in tables of data, obtained paging compressed file format
Tables of data to be synchronized can significantly improve the original and compressibility of tables of data, and occupied space is smaller, and database is deposited
Store up more tables of data.
103, according to pre-set isochronous schedules task by the tables of data to be synchronized with multithreading paging compressed file
Format is synchronized in the target database of server.
It may be related to different isochronous schedules parameters for the embodiment of the present invention, in isochronous schedules task, specifically may be used
With include synchronization time parameter, isochronous schedules time level parameters and synchronization node weight parameter, synchronization time, parameter was pre-
Estimate business side's synchrodata completion synchronize the required time, for example, configuration 1 hour in finishing service side's synchrodata it is same
Step, isochronous schedules time level parameters are the unit time that business side's synchrodata is dispatched in synchronizing process, for example, 5 minutes
Rank, 10 minutes ranks, 30 minutes ranks, 1 hour rank, 3 hours ranks and 1 day rank, synchronization node weight parameter are
The number of nodes for needing to open simultaneously in synchronizing process, for example, opening 5 servers or 10 servers progress data simultaneously
It is synchronous.
Before sending data synchronous task due to business Fang Xiang server, business side can count the same of synchrodata in advance
Information is walked, such as counts which tables of data needs synchronizes, synchronous deadline limitation etc. is further obtained according to statistics
Synchronizing information isochronous schedules task is set, i.e., completion data synchronize required time, isochronous schedules time rank and same
Node weights parameter business side is walked, for example, business side requires the data synchronous task for completing 15 tables of data in 3 days, here
Setting isochronous schedules parameter be synchronization time parameter be 3 days, isochronous schedules time level parameters be 1 day rank, synchronization node power
Weight parameter be 5 servers, every server distribute daily at least one tables of data complete data synchronous task, further to
It is realized synchrodata table according to the isochronous schedules parameter being arranged in isochronous schedules task with multithreading paging compressed file format
It is synchronized in the target database of server.
The present invention provides a kind of method that data are synchronous, by configuring business side according to default compressed format at business end
Paged data be compressed in tables of data, do not have to batch insertion operation, but by synchrodata with multithreading paging compressed file
Format transmission greatly reduces the big table of data volume and carries out the data synchronous time to server.After paging in the prior art
The data method of synchronization that is limited in bandwidth in data synchronization process of each tables of data compare, the embodiment of the present invention is in business
Synchrodata is sent to server by the way of paging, high frequency, multithreading compression by end, so that the data after paging are same
It is not bandwidth limited during step, improve data synchronization efficiency.
The embodiment of the invention provides the synchronous methods of another data, are applied to server, can use in server
Paged data is written into target data the mode that multithreading decompression, order load, and data write efficiency is improved, such as Fig. 2 institute
Show, which comprises
201, the synchrodata table of multithreading paging compressed file format is received.
For the embodiment of the present invention, due to the particularity of big table handling data, it is synchronous that data directly often are carried out to big table
Efficiency will affect the time of business end synchrodata, and the method for synchronization compressed here using paging, multithreading can greatly shorten
Data synchronization times.
202, decompression processing is carried out to the synchrodata table according to default decompression format, obtained in the synchrodata table
Each paged data.
For the embodiment of the present invention, default decompression format here is corresponding with the default compressed format at business end, equally
Default decompression format can be determined by data characteristics in analysis synchrodata table, further according to default decompression format to same
Step data table carries out decompression processing, obtains each paged data in synchrodata table.
203, paged data separated time journey each in the synchrodata table is written to number of targets in a manner of ordering load
According in library.
Synchrodata is written into the target table of target database the mode for generalling use insertion in the prior art, and
Mode one data of every write-in for data volume insertion is a cpu movement, and every write-in one by way of loading command
Ten thousand datas are a cpu movements, and synchrodata can be improved in separated time journey write-in synchrodata in a manner of loading command here
Write efficiency.
Further, the correct position in target database can be synchronized to for the paged data guaranteed in synchrodata table
It sets, mapping directory can be created in advance in the server for the paged data in synchrodata table, recording in the mapping directory has
The location information of each paged data target table to be written, as the area target database C table c3 is written in paged data 101
In, paged data 201 be written the area target database D table d1 in, in order in server after decompression obtains paged data, can
To pass through the location information for the target table for searching the corresponding write-in of each paged data from the mapping directory table being pre-created, root
Corresponding object table is written into target database in paged data in a manner of ordering load according to the location information of target table
In lattice.
The present invention provides the synchronous methods of another data, pass through the paged data decompressed in server separated time journey
The characteristics of target database is written, makes full use of server high concurrent in a manner of ordering load, will synchronize at faster speed
Data are loaded onto the target data of server.With each tables of data after paging in the prior art in data synchronization process by
The data method of synchronization limited to bandwidth is compared, and the embodiment of the present invention in server using multithreading decompression, order by being loaded
Mode paged data is written into target data, improve data write efficiency.
It should be noted that the repetition in order to avoid synchrodata table is synchronous, multithreading paging pressure is received in server
After the synchrodata table of contracting file format, one can be carried out to synchrodata table according to the file identification carried in synchrodata table
The verifying of cause property, this document mark are able to verify that the consistency of synchrodata table illustrates same step number if file identification is identical
It has been synchronized to server according to table, has been synchronized without repeating, then has been to have synchronized by synchrodata list notation；If file identification is different
If cause, illustrate that synchrodata table is not synchronized to server also, simultaneously operating can be continued to execute, then by synchrodata list notation
Not synchronize.
For the embodiment of the present invention, above-mentioned file identification may include each paging compressed file life in synchrodata table
At multiple paging marks and synchrodata table in all paging compressed files summarize generation one entirety mark, with specific reference to
The realization process that the file identification carried in synchrodata table carries out consistency checking to synchrodata table may include but not office
It is limited to following steps:
301, it is identified according to the entirety in the synchrodata table, to all paging compressed files in the synchrodata table
Summarize carry out consistency checking.
For the embodiment of the present invention, it is generally designated as the synchrodata that all paged datas in synchrodata table summarize generation
The identification information that table carries can be with by identifying corresponding all paging compressed files to whole and summarizing carry out consistency checking
Whether the data in verifying synchrodata table have change, if unanimously, illustrating that the synchrodata table is synchronized and crossing, be not necessarily to
It repeats to synchronize, then stops the simultaneously operating for executing synchrodata table；If inconsistent, illustrate that the synchrodata table does not synchronize, then
Continue to execute the simultaneously operating of synchrodata table.
Usual situation can be generated in synchrodata table to verify the consistency of synchrodata using Encryption Algorithm
Paging mark, MD5 and SHA1 is common Encryption Algorithm at present, and the embodiment of the present invention is not limited Encryption Algorithm, lead to
Crossing Encryption Algorithm can be generated a numerical value, which is equivalent to the paging mark of paged data, by whether detecting the numerical value
Change can be with verify data consistency.
302, it is identified according to the paging in the synchrodata table, corresponding point is identified to paging in the synchrodata table
Page compressed file carries out consistency checking.
For the embodiment of the present invention, each paging number is generated after paging is handled since paging is identified as synchrodata table
According to the identification information of carrying, consistency checking is carried out by identifying corresponding paging compressed file to paging, can be verified each
Whether the data in paged data have change, if unanimously, illustrating that the paged data is synchronized and crossing, synchronize without repeating,
Then stop executing the simultaneously operating of paged data；If inconsistent, illustrate that the paged data does not synchronize, then continue to execute paging number
According to simultaneously operating.
It should be noted that paging mark is for carrying out consistency checking to paged data, whole mark is for summarizing
Synchrodata table carry out consistency checking, if converged by all paging compressed files in whole identity verification synchrodata table
Always meet consistency, then without carrying out consistency checking to each paged data, and passes through paging identity verification synchrodata table
Middle paging compressed file meets consistency, can not illustrate that all paging compressed files summarize and meet consistency checking.Usual feelings
Under condition, carry out consistency checking preferentially can be summarized to paging compressed files all in synchrodata table, if it find that after inconsistent,
Corresponding paging compressed file is identified to paging in synchrodata table again and carries out consistency checking, so which paged data searched
It has been be synchronized that, to execute simultaneously operating for the paged data not being synchronized.
Further, the specific implementation as Fig. 1 the method, the embodiment of the invention provides a kind of business end, such as Fig. 4
Shown, which includes: acquiring unit 41, compression unit 42, synchronization unit 43.
Acquiring unit 41 can be used for obtaining the paged data of business side's configuration；
Compression unit 42 can be used for that the paged data being compressed in tables of data according to default compressed format, obtain
The tables of data to be synchronized of paging compressed file format；
Synchronization unit 43 can be used for the tables of data to be synchronized according to pre-set isochronous schedules task with multi-thread
Journey paging compressed file format is synchronized in the target database of server.
Business end provided by the invention, passes through the paged data for configuring business side according to default compressed format at business end
It is compressed in tables of data, does not have to batch insertion operation, but synchrodata is transferred to multithreading paging compressed file format
Server greatly reduces the big table of data volume and carries out the data synchronous time.With each data after paging in the prior art
The data method of synchronization that table is limited in bandwidth in data synchronization process is compared, and the embodiment of the present invention uses at business end divides
Synchrodata is sent to server by the mode that page, high frequency, multithreading compress, so that the data after paging are in synchronizing process
It is not bandwidth limited, improve data synchronization efficiency.
In specific application scenarios, as shown in figure 5, acquiring unit 41 includes:
Screening module 411 can be used for needing to filter out in synchrodata table from business side data volume greater than default value
Synchrodata table,；
Pagination module 412 can be used for being greater than the same of default value to each data volume according to default division rule
Data in step data table carry out paging processing, obtain the paged data of multiple business side's configurations.
Further, the default division rule is that the fixed quantity determined according to transmission bandwidth needs the business side
Data in synchrodata table are divided,
The pagination module 412 specifically can be used for counting the synchrodata that each data volume is greater than default value
Data in table, and record the data volume that statistics obtains；
The pagination module 412 specifically can be also used for when the data volume that the statistics obtains reaches fixed quantity, from
The data volume, which is greater than in the synchrodata table of default value, intercepts the data of the fixed quantity, and by the fixed quantity
Data are put into paging, obtain the paged data of business side's configuration.
It should be noted that other corresponding descriptions of each functional unit involved by a kind of business end provided in this embodiment,
Can be with reference to the corresponding description of Fig. 1, details are not described herein.
It is stored thereon with based on above-mentioned method as shown in Figure 1 correspondingly, the present embodiment additionally provides a kind of storage medium
Computer program, the program realize above-mentioned data as shown in Figure 1 synchronous method when being executed by processor.
Further, the specific implementation as Fig. 2 and Fig. 3 the method, a kind of server of the embodiment of the present invention are such as schemed
Shown in 6, which includes: receiving unit 51, decompression units 52, writing unit 53.
Receiving unit 51 can be used for receiving the synchrodata table of multithreading paging compressed file format；
Decompression units 52 can be used for carrying out decompression processing to the synchrodata table according to default decompression format, obtain
Each paged data in the synchrodata table；
Writing unit 53 can be used for paged data separated time each in the synchrodata table in a manner of ordering load
Journey is written into target database.
Server provided by the invention orders the side of load by the paged data decompressed in server separated time journey
The characteristics of target database is written in formula, makes full use of server high concurrent, is loaded onto service for synchrodata at faster speed
In the target data of device.The number being limited in bandwidth in data synchronization process with each tables of data after paging in the prior art
It is compared according to the method for synchronization, the embodiment of the present invention in server using multithreading by way of being decompressed, order loads by paging number
According to write-in into target data, data write efficiency is improved.
In specific application scenarios, as shown in fig. 7, described device further include:
Authentication unit 54 can be used for after the synchrodata table for receiving multithreading paging compressed file format,
Consistency checking is carried out to the synchrodata table according to the file identification carried in the synchrodata table；
The synchrodata list notation is by the authentication unit 54 if it is consistent specifically to can be used for verification result
It has synchronized；
The authentication unit 54, if it is inconsistent specifically to can be also used for verification result, by the synchrodata table mark
It is denoted as and does not synchronize.
Further, the file identification includes each paging compressed file generates in the synchrodata table multiple points
All paging compressed files summarize the entirety mark of generation, the authentication unit in page mark and the synchrodata table
54 include:
First authentication module 541 can be used for according to the entirety mark in the synchrodata table, to the synchrodata
All paging compressed files summarize carry out consistency checking in table；
Second authentication module 542 can be used for according to the paging mark in the synchrodata table, to the synchrodata
Paging identifies corresponding paging compressed file and carries out consistency checking in table.
Further, said write unit 53 includes:
Searching module 531 can be used for searching the corresponding write-in of each paged data from the mapping directory being pre-created
Target table is stored with each paged data target table in target database in synchrodata table in the mapping directory
Location information；
Writing module 532 can be used for each paging in the synchrodata table in such a way that the order loads
Data separated time journey is written to the target table of the corresponding write-in of each paged data.
It should be noted that other phases of each functional unit involved by the device that a kind of data provided in this embodiment synchronize
It should describe, can be described referring to figs. 2 to the corresponding of Fig. 3, details are not described herein.
It is deposited thereon based on above-mentioned method as shown in Figure 2 to Figure 3 correspondingly, the present embodiment additionally provides a kind of storage medium
Computer program is contained, which realizes above-mentioned data as shown in Figure 2 to Figure 3 synchronous method when being executed by processor.
Based on this understanding, the technical solution of the application can be embodied in the form of software products, which produces
Product can store in a non-volatile memory medium (can be CD-ROM, USB flash disk, mobile hard disk etc.), including some instructions
With so that computer equipment (can be personal computer, server or the network equipment an etc.) execution the application is each
Method described in implement scene.
Based on above-mentioned method and Fig. 4 as shown in Figure 1 to Figure 3 to virtual bench embodiment shown in Fig. 7, for reality
Existing above-mentioned purpose, the embodiment of the present application also provides a kind of computer equipments, are specifically as follows personal computer, server, net
Network equipment etc., the entity device include storage medium and processor；Storage medium, for storing computer program；Processor is used
In execution computer program to realize the synchronous method of above-mentioned data as shown in Figure 1 to Figure 3.
Optionally, which can also include user interface, network interface, camera, radio frequency (Radio
Frequency, RF) circuit, sensor, voicefrequency circuit, WI-FI module etc..User interface may include display screen
(Display), input unit such as keyboard (Keyboard) etc., optional user interface can also connect including USB interface, card reader
Mouthful etc..Network interface optionally may include standard wireline interface and wireless interface (such as blue tooth interface, WI-FI interface).
It will be understood by those skilled in the art that the synchronous entity device structure of data provided in this embodiment is not constituted pair
The restriction of the entity device may include more or fewer components, perhaps combine certain components or different component cloth
It sets.
It can also include operating system, network communication module in storage medium.Operating system is that the above-mentioned computer of management is set
The program of standby hardware and software resource, supports the operation of message handling program and other softwares and/or program.Network communication mould
Block leads to for realizing the communication between each component in storage medium inside, and between other hardware and softwares in the entity device
Letter.
Through the above description of the embodiments, those skilled in the art can be understood that the application can borrow
It helps software that the mode of necessary general hardware platform is added to realize, hardware realization can also be passed through.Pass through the skill of application the application
Art scheme, compared with currently available technology, the embodiment of the present invention at business end using paging, high frequency, multithreading by being compressed
Synchrodata is sent to server by mode, so that the data after paging are not bandwidth limited in synchronizing process, improves number
Paged data is written into target data by the way of multithreading decompression, order load according to synchronous efficiency, and in server,
Improve data write efficiency.
It will be appreciated by those skilled in the art that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, module in attached drawing or
Process is not necessarily implemented necessary to the application.It will be appreciated by those skilled in the art that the mould in device in implement scene
Block can according to implement scene describe be distributed in the device of implement scene, can also carry out corresponding change be located at be different from
In one or more devices of this implement scene.The module of above-mentioned implement scene can be merged into a module, can also be into one
Step splits into multiple submodule.
Above-mentioned the application serial number is for illustration only, does not represent the superiority and inferiority of implement scene.Disclosed above is only the application
Several specific implementation scenes, still, the application is not limited to this, and the changes that any person skilled in the art can think of is all
The protection scope of the application should be fallen into.
Claims (10)
1. a kind of synchronous method of data, which is characterized in that the described method includes:
The paged data of acquisition business side configuration；
The paged data is compressed in tables of data according to default compressed format, obtains the to be synchronized of paging compressed file format
Tables of data；
It is according to pre-set isochronous schedules task that the tables of data to be synchronized is synchronous with multithreading paging compressed file format
Into the target database of server.
2. the method according to claim 1, wherein the paged data of acquisition business side configuration includes:
Need to filter out the synchrodata table that data volume is greater than default value in synchrodata table from business side；
The data being greater than according to default division rule to each data volume in the synchrodata table of default value carry out paging
Processing obtains the paged data of multiple business side's configurations.
3. according to the method described in claim 2, it is characterized in that, the default division rule is to determine according to transmission bandwidth
Fixed quantity needs the data in synchrodata table to divide the business side, it is described according to default division rule to each
The data volume is greater than the data in the synchrodata table of default value and carries out paging processing, obtains point of multiple business side's configurations
Page data includes:
Each data volume is counted greater than the data in the synchrodata table of default value, and records the data that statistics obtains
Amount；
When the data volume that the statistics obtains reaches fixed quantity, the synchrodata table of default value is greater than from the data volume
The data of the middle interception fixed quantity, and the data of the fixed quantity are put into paging, obtain point of business side's configuration
Page data.
4. a kind of synchronous method of data, which is characterized in that the described method includes:
Receive the synchrodata table of multithreading paging compressed file format；
Decompression processing is carried out to the synchrodata table according to default decompression format, obtains each paging in the synchrodata table
Data；
Paged data separated time journey each in the synchrodata table is written into target database in a manner of ordering load.
5. according to the method described in claim 4, it is characterized in that, receiving the same of multithreading paging compressed file format described
After step data table, the method also includes:
Consistency checking is carried out to the synchrodata table according to the file identification carried in the synchrodata table；
It is to have synchronized by the synchrodata list notation if verification result is consistent；If verification result is inconsistent, by institute
Stating synchrodata list notation is not synchronize.
6. according to the method described in claim 5, it is characterized in that, the file identification includes each in the synchrodata table
All paging compressed files summarize generation in multiple pagings mark of paging compressed file generation and the synchrodata table
One entirety mark, it is described that consistency is carried out to the synchrodata table according to the file identification carried in the synchrodata table
Verifying includes:
According to the entirety mark in the synchrodata table, progress is summarized to all paging compressed files in the synchrodata table
Consistency checking；
According to the paging mark in the synchrodata table, corresponding paging is identified to paging in the synchrodata table and compresses text
Part carries out consistency checking.
7. a kind of synchronous device of data, which is characterized in that described device includes:
Acquiring unit, for obtaining the paged data of business side's configuration；
Compression unit obtains paging compression text for the paged data to be compressed in tables of data according to default compressed format
The tables of data to be synchronized of part format；
Synchronization unit, for being compressed the tables of data to be synchronized with multithreading paging according to pre-set isochronous schedules task
File format is synchronized in the target database of server.
8. a kind of synchronous device of data, which is characterized in that described device includes:
Receiving unit, for receiving the synchrodata table of multithreading paging compressed file format；
Decompression units obtain the same step number for carrying out decompression processing to the synchrodata table according to default decompression format
According to paged data each in table；
Writing unit, for paged data separated time journey each in the synchrodata table to be written to mesh in a manner of ordering load
It marks in database.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists
In the step of processor realizes any one of claims 1 to 7 the method when executing the computer program.
10. a kind of computer storage medium, is stored thereon with computer program, which is characterized in that the computer program is located
The step of reason device realizes method described in any one of claims 1 to 7 when executing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910218802.2A CN110062028A (en) | 2019-03-21 | 2019-03-21 | Data synchronous method, apparatus, computer equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910218802.2A CN110062028A (en) | 2019-03-21 | 2019-03-21 | Data synchronous method, apparatus, computer equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110062028A true CN110062028A (en) | 2019-07-26 |
Family
ID=67315883
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910218802.2A Pending CN110062028A (en) | 2019-03-21 | 2019-03-21 | Data synchronous method, apparatus, computer equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110062028A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110727655A (en) * | 2019-09-10 | 2020-01-24 | 连连银通电子支付有限公司 | Method, device, equipment and medium for building shadow database of block chain |
| CN110727720A (en) * | 2019-10-21 | 2020-01-24 | 腾讯科技（深圳）有限公司 | List display and query method and device, storage medium and computer equipment |
| CN111427698A (en) * | 2020-03-26 | 2020-07-17 | 平安医疗健康管理股份有限公司 | Azakban-based data synchronization method and device and computer equipment |
| CN112162964A (en) * | 2020-10-15 | 2021-01-01 | 苏州交驰人工智能研究院有限公司 | Self-adaptive data synchronization method and device, computer equipment and storage medium |
| CN114363361A (en) * | 2022-03-17 | 2022-04-15 | 武汉中科通达高新技术股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740069A (en) * | 2016-01-29 | 2016-07-06 | 中国电力科学研究院 | Automatic scheduling method for multistage data transformation tasks |
| CN107491565A (en) * | 2017-10-10 | 2017-12-19 | 语联网（武汉）信息技术有限公司 | A kind of method of data synchronization |
| CN107562705A (en) * | 2017-07-25 | 2018-01-09 | 努比亚技术有限公司 | A kind of data export method, service end and computer-readable recording medium |
| CN107609159A (en) * | 2017-09-26 | 2018-01-19 | 恒生电子股份有限公司 | Method, apparatus and computer-readable medium for data loading |
| CN108776710A (en) * | 2018-06-28 | 2018-11-09 | 农信银资金清算中心有限责任公司 | A kind of concurrent stowage and device of database data |
| CN109064248A (en) * | 2018-06-25 | 2018-12-21 | 北京辰森世纪科技股份有限公司 | Order data synchronous method of calibration, device and equipment |
-
2019
- 2019-03-21 CN CN201910218802.2A patent/CN110062028A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740069A (en) * | 2016-01-29 | 2016-07-06 | 中国电力科学研究院 | Automatic scheduling method for multistage data transformation tasks |
| CN107562705A (en) * | 2017-07-25 | 2018-01-09 | 努比亚技术有限公司 | A kind of data export method, service end and computer-readable recording medium |
| CN107609159A (en) * | 2017-09-26 | 2018-01-19 | 恒生电子股份有限公司 | Method, apparatus and computer-readable medium for data loading |
| CN107491565A (en) * | 2017-10-10 | 2017-12-19 | 语联网（武汉）信息技术有限公司 | A kind of method of data synchronization |
| CN109064248A (en) * | 2018-06-25 | 2018-12-21 | 北京辰森世纪科技股份有限公司 | Order data synchronous method of calibration, device and equipment |
| CN108776710A (en) * | 2018-06-28 | 2018-11-09 | 农信银资金清算中心有限责任公司 | A kind of concurrent stowage and device of database data |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110727655A (en) * | 2019-09-10 | 2020-01-24 | 连连银通电子支付有限公司 | Method, device, equipment and medium for building shadow database of block chain |
| CN110727720A (en) * | 2019-10-21 | 2020-01-24 | 腾讯科技（深圳）有限公司 | List display and query method and device, storage medium and computer equipment |
| CN110727720B (en) * | 2019-10-21 | 2023-06-20 | 腾讯科技（深圳）有限公司 | List display and query method and device, storage medium and computer equipment |
| CN111427698A (en) * | 2020-03-26 | 2020-07-17 | 平安医疗健康管理股份有限公司 | Azakban-based data synchronization method and device and computer equipment |
| CN111427698B (en) * | 2020-03-26 | 2022-11-15 | 深圳平安医疗健康科技服务有限公司 | Azakban-based data synchronization method and device and computer equipment |
| CN112162964A (en) * | 2020-10-15 | 2021-01-01 | 苏州交驰人工智能研究院有限公司 | Self-adaptive data synchronization method and device, computer equipment and storage medium |
| CN112162964B (en) * | 2020-10-15 | 2024-04-05 | 苏州交驰人工智能研究院有限公司 | Self-adaptive data synchronization method, device, computer equipment and storage medium |
| CN114363361A (en) * | 2022-03-17 | 2022-04-15 | 武汉中科通达高新技术股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110062028A (en) | Data synchronous method, apparatus, computer equipment and computer storage medium | |
| US9800412B2 (en) | System and method for securing sensitive data | |
| Kolodner et al. | A cloud environment for data-intensive storage services | |
| CN105205154B (en) | Data migration method and device | |
| US20130227085A1 (en) | Terminal and method for using cloud services | |
| CN109725933A (en) | Big data document structure tree method, apparatus, equipment and computer readable storage medium | |
| CN110083579A (en) | Incremental data synchronous method, apparatus, computer equipment and computer storage medium | |
| CN110147201A (en) | Line compression method, apparatus, computer equipment and storage medium | |
| US10783073B2 (en) | Chronologically ordered out-of-place update key-value storage system | |
| US11150981B2 (en) | Fast recovery from failures in a chronologically ordered log-structured key-value storage system | |
| US11086726B2 (en) | User-based recovery point objectives for disaster recovery | |
| US11853284B2 (en) | In-place updates with concurrent reads in a decomposed state | |
| CN106844694A (en) | For the method and apparatus of synchrodata | |
| WO2021226822A1 (en) | Log write method and apparatus, electronic device, and storage medium | |
| CN115470235A (en) | Data processing method, device and equipment | |
| US20210181945A1 (en) | User-based recovery point objectives for disaster recovery | |
| CN110018761B (en) | Method, device and terminal for managing recently used files | |
| CN114297196B (en) | Metadata storage method and device, electronic equipment and storage medium | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| US11163636B2 (en) | Chronologically ordered log-structured key-value store from failures during garbage collection | |
| CN114153732A (en) | Fault scene testing method and device, electronic equipment and storage medium | |
| US20130218851A1 (en) | Storage system, data management device, method and program | |
| CN114968069A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN112433677A (en) | Method for docking multiple storage back ends by cloud mirror image service | |
| CN110275863A (en) | File moving method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190726 |