CN101770515B - Data block comparison based data updating method - Google Patents
Data block comparison based data updating method Download PDFInfo
- Publication number
- CN101770515B CN101770515B CN201010040010XA CN201010040010A CN101770515B CN 101770515 B CN101770515 B CN 101770515B CN 201010040010X A CN201010040010X A CN 201010040010XA CN 201010040010 A CN201010040010 A CN 201010040010A CN 101770515 B CN101770515 B CN 101770515B
- Authority
- CN
- China
- Prior art keywords
- data
- data block
- file
- edition
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data block comparison based data updating method. The method comprises the following steps: (1) respectively dividing data in all versions into a plurality of data blocks, respectively extracting all documents in the data in each version and labeling map information of each data block against the data in each version; (2) establishing a corresponding index document against the data in each version; and (3) figuring out data blocks in the data in the previous version different from that in the data in the next version according to the attribute information of all documents and labeling map information in the index document in the next version when data are updated, and updating the data blocks. The data updating method greatly improves the speed of data updating by data division and specific updating algorithm, and further reduces load of a server by combining with an improved network download structure, and improves the performance of network transmission.
Description
Technical field
The present invention relates to field of computer technology, relate in particular to a kind of updating data method.
Background technology
The online game of domestic hot topic is more than 100 sections; Online game meeting generation content update more than 30% almost all can be arranged every day, and the recreation of the computer the inside of Internet bar also needs to upgrade timely, could be in the demand that satisfies the netizen of Internet bar; On this demand of solution, there are two technological difficulties; One of difficult point is that the machine terminal number of all Internet bars is very huge, about more than 1,000 ten thousand, if all directly carry out game renewal from Internet Server; The performance load of server can not be dealt with at all, also need distribute huge network bandwidth resources to support simultaneously on the internet; Two of difficult point is that the renewal amount of hot game is very huge; If rank from the file renewal; Every day is on average about 40G at present; Huge renewal amount makes that almost game renewal basically can not be in normal operation inside the Internet bar, Internet bar's limited bandwidth, recreation will have been upgraded needs patient hardly time span).In order to overcome the above problems; Prior art also has the means that adopt data increment to upgrade, and the difference after promptly checking Data Update and upgrading is only partly upgraded to this difference during renewal; Generally with logical file as the atom process object; Upgrade diversity ratio with the attribute of file itself, for example document creation time or file modification time judge as basis for estimation whether this file was updated.If file was updated, then whole file as the difference demand, need Network Transmission to reach the purpose of Data Update.
But existing method be with logical file as process object, in practical application, a lot of Data Update just change the one piece of data in the file.It is excessive than granularity that this just causes upgrading diversity ratio, can repeat to upgrade some legacy datas.
If the file that the renewal diversity ratio is come out needs to upgrade, the length of file is exactly the size that needs renewal.Because file is changeable, causing atomic update length also is dynamic change.Can cause the performance of Network Transmission to descend.
Summary of the invention
The invention provides a kind of fast data update method; The speed that has improved Data Update greatly through division and specific update algorithm to data; Combine improved network download framework further to alleviate the load of server in addition, improve the performance of Network Transmission.
The block index of data-updating method of the present invention through using data content and difference are overlapped the binary data stream that complete data (for example all data of a recreation) are regarded a continuous whole as to one to recently realizing.A file is the logic unit of one section binary data stream.A data block also is the logic unit of binary data stream.
Three's relation is: data block is the subclass of file, and file is the subclass of game data.When the whole size of file during less than a definition data block length, the content of a file is exactly a data block.Certainly, most applications is that a file has the plurality of data piece.The length of data block can artificial define, and we can find the optimum data block length of overall performance in global system according to the characteristics of Network Transmission and the characteristics of disk (a large amount of at present disk array of using) buffer memory like this.Rather than passive adaptation file size.
A kind of based on data block data-updating method relatively, comprise the steps:
(1) all edition datas all are divided into the plurality of data piece,, extract respectively in this edition data to each edition data:
A) attribute information of whole edition data and sign map information;
B) attribute information of All Files and sign map information;
C) the sign map information of each data block;
(2) to each edition data, set up corresponding index file, described index file comprises the information of extracting in the step (1) to this edition data;
When (3) carrying out Data Update, contrast edition data formerly and at the index file of back edition data, step is following during contrast:
A) find out the file that does not comprise in the edition data formerly according to the attribute information of All Files in the version index file of back and sign map information, and the file that change is arranged;
For the file that does not comprise in the edition data formerly, then directly upgrade, be about to this document in the edition data of back and copy to formerly in the edition data;
For the file that change is arranged; Find out in the edition data formerly and the data block of change is arranged in the edition data of back according to the sign map information of each data block in the file; Data block to change is arranged is upgraded, and is about to this block copy in the edition data of back in edition data formerly;
B) attribute information according to files all in the version index file formerly finds out the file that in the edition data of back, does not comprise with the sign map information, and the file that change is arranged;
For the file that in the edition data of back, does not comprise, directly delete this document;
For the file that change is arranged, find out the data block that in the edition data of back, does not comprise according to the sign map information of each data block in the file, directly delete this data block.
In the step of the present invention (1), when some edition datas are divided into the plurality of data piece, be to be object with single file; File is divided one by one; Data in the different files can not be divided in the same data block, because the length of data block is preassigned, and file is divided into when last; The length of end block is general can be not just and the equal in length of preassigned data block, and the disposal route for an end of file piece is following so:
The length that is noted that end block must be the length that is less than or equal to preassigned data block; Otherwise will continue to divide; When the length of end block equals the length of preassigned data block; Just divide by predetermined principle, what our emphasis was considered is the length of the length of end block less than preassigned data block.
At first set a threshold value; Calculate the length of end block; Obtain the ratio of the length of this end block length and preassigned data block,, then this end block is merged in its previous data block when ratio during less than threshold value; If ratio during more than or equal to threshold value, then divides this end block into a data block separately.As preferably, threshold value gets 1/3, can reduce fragmentary short length data piece so as far as possible, helps improving the efficient of Network Transmission, also avoids frequent reading and writing data as far as possible.
The length of data block has direct influence to updating data:
If data block length is excessive, distortion takes place when Network Transmission, can cause whole data block to resend, during go to account for a large amount of Internet resources and machine performance resource, the bothersome effort.This also is the weakness of simple processing mode.
If definition data block is moderate, because a blocks of data piece can obtain from a plurality of servers.Assembling again after mass data can obtain respectively from a plurality of servers, whole like this Internet resources will be divided in the bandwidth of each server.Length as the preference data piece adopts 16～64K comparatively suitable, and the most further the length of preference data piece adopts 32K, and can reach taking into account of Network Transmission quality and program feature this moment.
If definition data block is too little, then increased the interaction times between program and the operating system, influence program feature.
There are many methods can obtain the sign map information of data block (one section binary stream) in the prior art.For example: MD5 sign indicating number and CRC32 or the like are arranged.
Compare MD5 sign indicating number and CRC32, the collision rate of MD5 sign indicating number is lower, that is to say, the binary data stream of 2 sections different contents can be distinguished difference with the MD5 sign indicating number, but CRC32 may think that 2 sections streams are the same.The identification information of producing through CRC32 has only 4 bytes, and makes 128 bytes of identification information possible operation through MD5.The time complexity of the two is more or less the same.
No matter what mode of use is set up the sign map information of data block, its purpose all is that the content that needs only the less sign map information replacement data piece of correlation data amount compares.The tentation data block length is the 32K byte, and then we use relatively coming of 4 bytes or 128 bytes, the comparison of replacement 32K byte.
Finally selecting for use CRC32 as the sign map information among the present invention, mainly is that the collision rate of considering CRC32 can be accepted, and the sign map information takes up space for a short time, helps reducing the size of index.
The described index file of step (2) is the set of all data block sign map informations of edition data of record.The index file of the certain corresponding version of the data of different editions.
The effect of index file: program to be updated, through comparing the index file of different editions, produce the difference demand, program to be updated like this as long as download the difference partial data, just can be accomplished edition upgrading.
Content in the index file of the present invention comprises following several:
A) attribute information of whole edition data and sign map information comprise with regard to each edition data:
The type identification of PkgId edition data, dissimilar edition datas adopts different PkgId, and for example the PkgId of the edition data of different recreation is different;
The unit-sized of BlockSize file block (unit byte), the i.e. length of data block;
PkgIdxVer index version, the sign of the edition data that same type but version are different; For example same recreation has different versions, and PkgId is identical between so different versions, but PkgIdxVer is different;
The CRC self checking of SelfCRC index file is used in network transport process verification index file self correct transmission whether;
The tabulation of FileDescList file description information (arrays of a plurality of file description information) has been represented the attribute information of files all in this edition data and has been identified map information.
B) attribute information that has comprised files all in this edition data among the tabulation FileDescList of file description information and sign map information, with regard to each file, its descriptor (FileDescInfo) as follows comprising:
FileAttr file type, type are divided into two kinds of catalogue and files;
The relative path that the FilePathName file name comprises;
The FileModifyTime file modification time;
The CRC check sign indicating number of FileCRC file whole contents;
FileLength file size (unit byte);
The numbering of first data block in the BeginBlockId file; With regard to whole edition data, the numbering of wherein all data blocks all is unique, and the numbering through data block can the position of rapidly locating piece in whole edition data and whole file;
The sign map information of all data blocks in this document has been represented in the tabulation of BlockDescList file data block message (arrays of a plurality of data block information);
C) comprised the sign map information of all data blocks in this document among the tabulation BlockDescList of file data block message, with regard to a data block, its descriptor is (BlockDescInfo) as follows:
The CRC check sign indicating number of BlockCRC data block contents.
In the step (3), contrast edition data formerly and be the difference contrast algorithm that has adopted based on block index during edition data in the back:
Can be divided into two big steps:
One, obtain the index file of two versions after; Earlier with the index file of redaction (edition data) in the back as benchmark, the index file of traversal redaction takes out the file description information (FileDescInfo) of file type; The information of searching this file in the index file of legacy version (edition data formerly); If can not find out, explain that then this file increases newly, produce difference comparative result (this file needs update all).If found, judge whether the FileCRC of FileDescInfo in 2 index is identical, if identical, we think that this file has no change.If inequality, then which the demand data block of searching variation has, and beginning travels through the piece descriptor (BlockDescInfo) of this file in proper order, and when the BlockCRC of new and old correspondence is different, we think that this data block needs to upgrade.
Two, above operation can be handled effect newly-increased and the modification data block.If the file delete of redaction during data block, disposal route is following:
As benchmark, traversal legacy version index file takes out the file description information of file type, in the index file of redaction, searches, if do not have, explains that this file will be deleted with the index file of legacy version.
For the deletion of piece, identical with top flow process.
More than two steps, the difference of generation has 2 types: need down loading updating with need be replaced.This just reflects which position is the data block of upgrading need be placed in the time of will descending which data and data assembling.
And the fileinfo of index file has this information of BeginBlockId (numbering that file is first) in describing; So the traversal fileinfo is described array, length that just can be through data block and side-play amount calculating accurately navigate to arbitrary blocks of data piece and should leave the position in the disk in.
In the step of the present invention (3) when carrying out the renewal of data block; Be between the request end that end and data block are provided of data block, to carry out in fact; The end that provides that is to say data block has been deposited the edition data in the back; Edition data has formerly been deposited in the request end of data block, for the handling capacity and the transmission speed of data volume between the request end that end and data block are provided of improving data block, adopts following method during Data Update:
(1) set up a shared buffer memory (DownCache) at the end that provides of data block, when Updating Information piece, data block the request of end according to the request end of data block be provided, in shared buffer memory, read in the data designated piece; Shared buffer memory all request realization unified managements to client (request end of data block) are convenient in foundation.
(2) data block provide end when receiving that data block is downloaded application, at first read shared buffer memory, if this data block is arranged in the shared buffer memory, then immediately with the data block of request to the transmission of the request end of data block;
If shared buffer memory is this data block not, be not to will begin in a minute to read file, submit to request queue but download application to this data block.
(3) the download application in the submission time sequential processes request queue of when providing of data block held the free time, applying for by download is transmitted the data block of request to the request end of data block.
As preferably, the data block in the shared buffer memory is sorted according to hit rate (hit rate is to read same number of times), when shared buffer memory was piled, the data block that will be in the end cleaned it out from shared buffer memory.
Apply for that for fear of a certain data block download the waiting time in request queue is long; Set the waiting time threshold value; When a certain data block is downloaded the waiting time of application in request queue when surpassing threshold value; Even the end that provides of data block does not have the free time, also this data block of priority processing is downloaded application.Generalized case, time threshold can be made as 15～30 minutes.
Seek time when further reducing reading and writing of files carries out ascending sort for the data block that hit rate in the shared buffer memory is identical according to sequence number.
The technical terms that the present invention mentions is like no specified otherwise, and its implication can be done following understanding:
1, data block: be the set of one group of binary data stream, describe the granularity of network or logical process.
2, index file: describe the complete game data collection of a cover with data block, each data block is made the sign map information, the sign map information of these data blocks and the file of part game attributes information record are called index file.
3, incremental update: be to one type of operation, rely on legacy data, only need to upgrade low volume data, just can reach upgrading purpose process with edition upgrading attribute.
4, IDC (Intemet Data Center): Internet data center
5, CRC32 (Cyclic Redundancy Check): 32 CRCs
6, time complexity: the time resource that the finger counting method need consume.If the scale of a problem is n, the needed time of a certain algorithm of separating this problem is T (n), and it is " time complexity " that a certain function T (n) of n is called this algorithm
7, space complexity: be meant required storage space that takies of algorithm of computer science completion.
8, MD5 sign indicating number (message-digest algorithm 5): md5-challenge.It is technical to be widely used in encryption and decryption, and it can be described as " digital finger-print " of file.
9, reorganization:, be concentrated into the process of continuous data with the data of Discrete Distribution.
10, separate volume: the inverse operation that is reorganization.
Data-updating method of the present invention, the speed through division and specific update algorithm to data have improved Data Update greatly combines improved network download framework further to alleviate the load of server in addition, improves the performance of Network Transmission.
Description of drawings
Fig. 1 is the structural representation of index file of the present invention;
Fig. 2 is the schematic flow sheet of difference contrast algorithm of the present invention;
Fig. 3 is the overall system architecture figure that utilizes data-updating method of the present invention;
Fig. 4 upgrades the passage rack composition for game data;
Fig. 5 is the newly-increased recreation schematic flow sheet in the IDC layer;
Fig. 6 is the Data Update flow process of IDC layer and Internet bar service layer;
Fig. 7 is the Data Update and the program interaction of Internet bar's server layer and Internet bar's computer layer;
Fig. 8 downloads the buffer memory process flow diagram for optimizing;
Fig. 9 is the mutual synoptic diagram of TCPP2P business procedure.
Embodiment
Below in conjunction with concrete hardware and Internet resources explanation data-updating method of the present invention, wherein with the example that is updated to of game data.
The present invention is through setting up three layers of update mode (Internet bar's computer is that terminating layer, Internet bar's server are that Internet bar's layer, Internet Server are the IDC layer); The renewal of recreation is updated to Internet bar's layer final updated to terminating layer from the IDC layer; Through three layers of method for updating; The renewal amount at terminal is pooled to Internet bar's layer, thereby effectively reduces renewal amount, solved the problem of the concurrent bottleneck and the huge renewal amount of great amount of terminals.These three layers of overall architecture can be referring to Fig. 3.
Wherein:
Terminating layer: program name BarClient, operand are the netizens, have the restoring function of penetrating and game renewal function, and this program only and Internet bar's server communication.
Internet bar's layer: program name BarServer, operand are webmaster or the owner of Internet bar, have the BarClient of providing end and upgrade, upgrade game data and network management function from IDC.
IDC layer: the game making of latest edition is provided and BarServer is provided download games.
Referring to Fig. 4, program and framework thereof in the IDC layer comprise:
1. game making instrument (PMUpdate); The function of this program is: with game data reorganization and the corresponding index of making; Promptly with regard to same games, generate different edition datas, and all edition datas all are divided into plurality of data piece (each data block predetermined length 32k size).
To the end block of a file, do to handle as follows:
NBlockCnt=piece number;
The nFileLen=file size;
The unit-sized of nBlockSize=file block;
nBlockCnt＝nFileLen/nBlockSize+ 1
if(nFileLen％nBlockSize＜nBlockSize/MagicBlockGene){--nBlockCnt；}
It is a parameter of setting up for conserve storage that MagicBlockGene gets 1/3; It can prevent effectively that the too small situation that takies the whole blocks size of end block from occurring; When run into end block when too small (less than nBlockSize/MagicBlockGene) end block do not become piece separately, but merge on last, so; Second from the bottom block length will be greater than BlockSize, so block size is different.Except end block, the size of other pieces all equals BlockSize
After calculating piece and counting nBlockCnt, EndBlockId=BeginBlockId+nBlockCnt
Can see that from top explanation the length of data block is not unalterable.2 types length is arranged, and a kind of is designated length, and another kind is the residue length of end block.
To each edition data, extract respectively in this edition data:
A) attribute information of whole edition data and sign map information;
B) attribute information of All Files and sign map information;
C) the sign map information of each data block;
According to the corresponding index file of the information issuing of carrying.Index file can the contents are as follows referring to Fig. 1:
A) attribute information of whole edition data and sign map information comprise with regard to each edition data:
The type identification of PkgId edition data, dissimilar edition datas adopts different PkgId, and for example the PkgId of the edition data of different recreation is different;
The unit-sized of BlockSize file block (unit byte), the i.e. length of data block;
PkgIdxVer index version, the sign of the edition data that same type but version are different; For example same recreation has different versions, and PkgId is identical between so different versions, but PkgIdxVer is different;
The CRC self checking of SelfCRC index file is used in network transport process verification index file self correct transmission whether;
The tabulation of FileDescList file description information (arrays of a plurality of file description information) has been represented the attribute information of files all in this edition data and has been identified map information.
B) attribute information that has comprised files all in this edition data among the tabulation FileDescList of file description information and sign map information, with regard to each file, its descriptor (FileDescInfo) as follows comprising:
FileAttr file type, type are divided into two kinds of catalogue and files;
The relative path that the FilePathName file name comprises;
The FileModifyTime file modification time;
The CRC check sign indicating number of FileCRC file whole contents;
FileLength file size (unit byte);
The numbering of first data block in the BeginBlockId file; With regard to whole edition data, the numbering of wherein all data blocks all is unique, and the numbering through data block can the position of rapidly locating piece in whole edition data and whole file;
The sign map information of all data blocks in this document has been represented in the tabulation of BlockDescList file data block message (arrays of a plurality of data block information);
C) comprised the sign map information of all data blocks in this document among the tabulation BlockDescList of file data block message, with regard to a data block, its descriptor is (BlockDescInfo) as follows:
The CRC check sign indicating number of BlockCRC data block contents.
After this edition data and index file are uploaded on the synchronisation source node (SyncSrcNode), so that BarServer incremental update game data.
2. synchronisation source node (SyncSrcNode); The function of this program is the index file of accepting from the submission of PMUpdate program, and through difference contrast, the data block that causes the demand (needing the data updated piece); Ask for corresponding data block to the PMUpdate program; And with these data blocks, the information according to index inside is placed on the target location exactly.The data that this program is deposited all are the data files through reorganization.SyncSrcNode also provides other the SyncSrcNode data synchronization service of not upgrading this version recreation and the synchronous service of BarServer simultaneously.
Referring to Fig. 2, step is following when carrying out the difference contrast:
A) find out the file that does not comprise in the edition data formerly according to the attribute information of All Files in the version index file of back and sign map information, and the file that change is arranged;
For the file that does not comprise in the edition data formerly,, be about to this document in back edition data and copy to formerly in the edition data then with directly upgrading;
For the file that change is arranged; Find out in the edition data formerly and the data block of change is arranged in the edition data of back according to the sign map information of each data block in the file; Data block to change is arranged is upgraded, and is about to this block copy in the edition data of back in edition data formerly;
B) attribute information according to files all in the version index file formerly finds out the file that in the edition data of back, does not comprise with the sign map information, and the file that change is arranged;
For the file that in the edition data of back, does not comprise, directly delete this document;
For the file that change is arranged, find out the data block that in the edition data of back, does not comprise according to the sign map information of each data block in the file, directly delete this data block.
3. global synchronization source server program (GlobalSSS-GlobalSyncSrcServer); Its effect is the game resource information of distributing newly-increased recreation; For example: distribute recreation ID for newly-increased recreation; Distribute version number to the recreation of upgrading, the attribute of record latest edition recreation, and offer Internet bar's service end demonstration.Its another important task is: the game data of management SyncSrcNode is synchronous.SyncSrcNode has the recreation of which version synchronously, supplies SyncSrcNode or BarServer to go to connect and downloads.Because it is responsible for the distribution of game resource attribute,, be used for the persistence game information so want supporting database.
4. local synchronisation source server program (LocalSSS-LocalSyncSrcServer), this program itself are Agents.BarServer is provided the more index file that needs of new game and the information of the SyncSrcServer node of all latest editions of playing of IDC end.
PMUpdate is the promoter of game renewal, the game data after index file being provided and reorganizing.SyncSrcNode is the keeper and the supplier of game data; In fact BarServer also is a SyncSrcNode; Only BarServer also has other functions, aspect game renewal, can be interpreted as a SyncSrcNode that rank is lower to BarServer.
GlobalSSS has 2 big functions, the game information of the distribution of promptly play ID and version, the persistence of game attributes information and SyncSrcNode synchronously.
Originally has only GlobalSSS; There is not the LocalSSS program; Because the order of magnitude of BarServer does not receive external attack with the positive GlobalSSS of guarantor, has produced LocalSSS, LocalSSS is the function cutting version of GlobalSSS; Up-to-date index file and the SyncSrcNode tabulation of holding this version recreation is provided, and realized the load balancing effect.
Referring to Fig. 5, mutual between more new technological process between the IDC layer and program:
PMUpdate logins GlobalSSS, after logining successfully, and to GlobalSSS request list of games, after the recreation of finding not have on the IDC end to upgrade as the game making personnel, newly-increased this recreation of decision.Through interface alternation PMUpdate the game information that the game making personnel edit is uploaded to GlobalSSS, the recreation ID of its band is 0, representes unallocated recreation ID.
After GlobalSSS receives game attributes information, find that recreation ID is zero, then distribute a new recreation ID to reply to the PMUpdate program automatically, and in the data named list, insert a record, the attribute of this this recreation of record expression.
Receive recreation ID number of GlobalSSS with initial version number after, PMUpdate begins the game data reorganization and begins to make index file, and the index file that completes is sent on the SyncSrcNode of application configuration.After target SyncSrcNode receives index file, under assigned catalogue, create file destination (file of data reorganization form), search the index file of local corresponding recreation ID again, the contrast of beginning difference.In the operation of newly-increased recreation, because SyncSrcNode this locality does not have previous version, so the difference comparing result is to need to upgrade all data.Then SyncSrcNode can want corresponding data block to the PMUpdate program according to the demand data piece tabulation of relatively coming out.After PMUpdate receives the piece demand signalling, from buffer memory or disk, read corresponding data block, be transferred to the SyncSrcNode program.After SyncSrcNode receives data block, write appointed positions.After all data blocks all had been placed into legal position, SyncSrcNode notice PMUpdate upgraded and finishes.PMUpdate then submits to GlobalSSS with index file, local this index file of persistence of GlobalSSS, and relevant information in the index file is modified in that record that inserted just now.
After GlobalSSS accomplishes aforesaid operations, notify all to be registered to the LocalSSS here and ask game download information (comprising up-to-date index tabulates with the SyncSrcNode that can supply download) to him with SyncSrcNode.The SyncSrcNode program finds that index is with local different; Will be to the SyncSrcNode program in the SyncSrcNode tabulation that obtains; The request of this version recreation is downloaded in proposition, begins to download flow process (because this flow process and BarServer request download flow process are similar, will introduce narration below).After LocalSSS received download list, which SyncSrcNode node the BarServer that can notify all downloading this recreation last revision went download the recreation of redaction.
The SyncSrcNode of existing simultaneously latest edition recreation reports oneself information to GlobalSSS, is distributed to the LocalSSS of all registrations by GlobalSSS.
Certain recreation flow process that has existed of PMUpdate upgrading with above similar, difference is that GlobalSSS does not reply the ID that plays, but new version number.
Referring to Fig. 6, the Data Update flow process and the program interaction thereof of IDC layer and Internet bar's server layer are following:
BarServer can regularly obtain list of games after successfully logining LocalSSS.When some recreation be configured to automatic renewal (be IDC when renewal is arranged, need not manual intervention will be automatically the function of game data synchronously) or by webmaster through after the interface program manual intervention, BarServer can ask the download message of this version recreation to LocalSSS.BarServer can contrast new and old index after obtaining this information, carries out the difference contrast, generates the tabulation of demand data piece.SyncSrcNode in download message wants the demand data piece.After downloading and assemble all data blocks of completion, BarServer can notify the lastest imformation of this version recreation of the LocalSSS oneself that is connected.
In fact, synchronously very similar between the SyncSrcNode of IDC layer with this flow process, the BarServer only here can regard as and ask synchronous SyncSrcNode, and GlobalSSS is equivalent to the LocalSSS here.
Referring to Fig. 7, between the game renewal flow process of Internet bar's server layer and Internet bar's computer and program mutual as follows:
BarClient reaches the standard grade and can remove to login BarServer, and behind the BarServer authentication success, BarClient asks list of games to BarServer.After obtaining list of games that BarServer holds, which recreation needs download BarClient judges according to own mechanism.Play request index file to the needs download.After BarServer obtains the index request of certain BarClient, the recreation index of correspondence is sent back to BarClient.BarClient contrasts algorithm according to difference, draws the demand data piece, and to BarServer request demand piece.BarServer replys the content of specified data block according to request.BarClient acquiring demand data block has, and whether certain file of interpretation is complete, if complete then be renamed as official document to this file from temporary file.
After all game data files were all accomplished renewal, just to make preceding format and content just the same with PMUpdate.
By top can finding out to be SyncSrcNode, BarServer or BarClient all used on the incremental update function the realizes data-updating method based on block index and supporting difference contrast of the present invention.Data-updating method of the present invention is in whole three layers of update mechanism.
In addition, for BarServer or other SyncSrcNode application of renewal synchronously, SyncSrcNode can remove to open corresponding APMB package according to their application of sending and also reply.After PMUpdate uploads the SyncSrcNode that plays; Just have various BarServer or SyncSrcNode and download this recreation; Send identical application; Cause identical data block repeat read, just become very low in such event on the efficient, transmission speed also is difficult to up.So, when identical request comes, directly reply the block cache of reading.
Original buffer memory is the buffer memory that each unwraps fixed size in the prior art, the blocks of data of buffer memory oneself separately, but be not that each recreation all is that the download application quantity of awfully hot door is also unequal, will cause the waste on the certain space like this.
Referring to Fig. 8, in the three-tier architecture of the present invention, in carrying out data interaction two-layer,, when data interaction, particularly when the transmission of data block, adopted and optimized the method for downloading buffer memory for handling capacity and the transmission speed that improves data volume, as follows:
(1) set up a shared buffer memory (DownCache) at the end that provides of data block, when Updating Information piece, data block the request of end according to the request end of data block be provided, in shared buffer memory, read in the data designated piece; Shared buffer memory all request realization unified managements to client (request end of data block) are convenient in foundation.
For three layers of frame that the present invention adopts, data block the request end of end and data block is provided the time comparatively speaking, in the adjacent two layers, high-level layer can be used as the end that provides of data block, the low level layer is then as the request end of data block.
(2) data block provide end when receiving that data block is downloaded application, at first read shared buffer memory, if this data block is arranged in the shared buffer memory, then immediately with the data block of request to the transmission of the request end of data block;
If shared buffer memory is this data block not, be not to will begin in a minute to read file, submit to request queue but download application to this data block.
(3) the download application in the submission time sequential processes request queue of when providing of data block held the free time, applying for by download is transmitted the data block of request to the request end of data block.
As preferably, the data block in the shared buffer memory is sorted according to hit rate (hit rate is to read same number of times), when shared buffer memory was piled, the data block that will be in the end cleaned it out from shared buffer memory.
Apply for that for fear of a certain data block download the waiting time in request queue is long; Set the waiting time threshold value; When a certain data block is downloaded the waiting time of application in request queue when surpassing threshold value; Even the end that provides of data block does not have the free time, also this data block of priority processing is downloaded application.Generalized case, time threshold can be made as 15～30 minutes.
Seek time when further reducing reading and writing of files carries out ascending sort for the data block that hit rate in the shared buffer memory is identical according to sequence number.
When the present invention upgrades in data block; In the transmission of data block, adopted the method for optimizing the download buffer memory for a large amount of requests; The end that provides of data block has been followed certain strategy, and the recreation that promptly linking number is many is preferentially downloaded, and has carried out ascending sort according to the sequence number of file and data block; Seek time when further reducing written document is sequential write when also having guaranteed written document.
Because the IDC layer bandwidth is a scarce resource; Cost is higher; Present IDC renewal amount of the present invention is saturated in rush hour, in order to guarantee upgrading demand of Internet bar's server layer, makes full use of the uplink bandwidth resource of Internet bar; Let the mutual shared data of Internet bar's server, also adopt the P2P technology when the present invention carries out Data Update.
TCPP2P uses ICP/IP protocol to carry out the P2P technology of data sharing, and its advantage is that ICP/IP protocol provides reliable and stable connection, guarantees the high speed and the data correctness of Network Transmission.Establishment of connection receives many-sided the influence, so require to use Internet bar's server of TCPP2P must support the UPNP technology.TCPP2P need be through two stages, 1. the 2. data interaction stages of information sharing stage
The information sharing stage:
Referring to Fig. 9, each Internet bar's server of this stage all will report the game information that oneself exists to LocalSSS, and IP and the port information of supporting P2P, supplies other Internet bar's servers to connect.When oneself needs to download, ask for the P2P information that reports this version game information Internet bar service end to the LocalSSS that is connected simultaneously.
For realizing sharing of P2P information, on the basis that keeps the original framework of game renewal passage, increased a class method newly, be called overall P2P information retransmission process (GlobalP2P).Its effect is exactly the P2P information that reports the LocalSSS of registration, is transmitted to other all the registration LocalSSS except that the person of reporting.After BarServer gets access to relevant P2P information, can be connected with Internet bar's service end of target resource automatically.
The data interaction stage:
We introduced the function that BarServer has SyncSrcNode the front, the flow process of swap data each other between 2 BarServer, and the game renewal flow processing mode of introducing with the front is consistent.
The present invention is through setting up three layers of update mode (Internet bar's computer is that terminating layer, Internet bar's server are that Internet bar's layer, Internet Server are the IDC layer); The renewal of recreation is updated to Internet bar's layer final updated to terminating layer from the IDC layer; Through three layers of method for updating; The renewal amount at terminal is pooled to Internet bar's layer, thereby effectively reduces renewal amount, solved the problem of the concurrent bottleneck and the huge renewal amount of great amount of terminals.
The present invention passes through the technology to the block index of game content and difference contrast and P2P distributing network; The renewal amount of recreation is controlled to minimum; Simultaneously also under the band-limited prerequisite of Internet Server; Made full use of the uplink bandwidth resource of Internet bar, shortened the consuming time of game renewal, for Internet bar and netizen provide more happy recreation experience.
Claims (5)
1. a data-updating method that compares based on data block is characterized in that, comprises the steps:
(1) all edition datas all are divided into the plurality of data piece, described data block is the subclass of file, and described file is the subclass of edition data; To each edition data, extract respectively in this edition data:
A) attribute information of whole edition data and sign map information;
B) attribute information of All Files and sign map information;
C) the sign map information of each data block;
When some edition datas are divided into the plurality of data piece, be to be object with single file, by the length of preassigned data block one by one file divide, file is divided into when last, the disposal route of its end block is following:
1) setting threshold;
2) calculate the length of end block, obtain the ratio of the length of this end block length and preassigned data block;
3) when described ratio during, then this end block is merged in its previous data block, if ratio during more than or equal to threshold value, then divides this end block into a data block separately less than threshold value;
(2) to each edition data, set up corresponding index file, described index file comprises the information of extracting in the step (1) to this edition data;
When (3) carrying out Data Update:
A) find out the file that does not comprise in the edition data formerly according to the attribute information of All Files in the version index file of back and sign map information, and the file that change is arranged;
For the file that does not comprise in the edition data formerly, then directly upgrade;
For the file that change is arranged, find out in the edition data formerly and the data block of change is arranged in the edition data of back according to the sign map information of each data block in the file, the data block that change is arranged is upgraded;
B) attribute information according to files all in the version index file formerly finds out the file that in the edition data of back, does not comprise with the sign map information, and the file that change is arranged;
For the file that in the edition data of back, does not comprise, directly delete this document;
For the file that change is arranged, find out the data block that in the edition data of back, does not comprise according to the sign map information of each data block in the file, directly delete this data block.
2. data-updating method as claimed in claim 1 is characterized in that, when carrying out the renewal of data block, the end that provides of data block has been deposited the edition data in the back in the step (3), and edition data has formerly been deposited in the request end of data block:
1) set up a shared buffer memory at the end that provides of data block, when Updating Information piece, data block the request of end according to the request end of data block be provided, in shared buffer memory, read in the data designated piece;
2) data block provide end when receiving that data block is downloaded application, at first read shared buffer memory, if this data block is arranged in the shared buffer memory, then immediately with the data block of request to the transmission of the request end of data block;
If shared buffer memory is this data block not, download application to this data block and submit to request queue;
3) the download application in the submission time sequential processes request queue of when providing of data block held the free time, applying for by download is transmitted the data block of request to the request end of data block.
3. data-updating method as claimed in claim 2 is characterized in that, the data block in the shared buffer memory is sorted according to hit rate, and when shared buffer memory was piled, the data block that will be in the end was removed from shared buffer memory.
4. data-updating method as claimed in claim 3 is characterized in that, carries out ascending sort for the data block that hit rate in the shared buffer memory is identical according to sequence number.
5. data-updating method as claimed in claim 4 is characterized in that, sets the waiting time threshold value, downloads the waiting time of application in request queue when surpassing threshold value when a certain data block, and this data block of end priority processing that provides of data block is downloaded application.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010040010XA CN101770515B (en) | 2010-01-18 | 2010-01-18 | Data block comparison based data updating method |
| MYPI2010005110A MY174951A (en) | 2010-01-18 | 2010-10-29 | A data updating method based on data block comparison |
| SG2010081495A SG173248A1 (en) | 2010-01-18 | 2010-11-03 | A data updating method based on data block comparison |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010040010XA CN101770515B (en) | 2010-01-18 | 2010-01-18 | Data block comparison based data updating method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101770515A CN101770515A (en) | 2010-07-07 |
| CN101770515B true CN101770515B (en) | 2012-01-11 |
Family
ID=42503373
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010040010XA Active CN101770515B (en) | 2010-01-18 | 2010-01-18 | Data block comparison based data updating method |
Country Status (3)
| Country | Link |
|---|---|
| CN (1) | CN101770515B (en) |
| MY (1) | MY174951A (en) |
| SG (1) | SG173248A1 (en) |
Families Citing this family (49)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102480494B (en) * | 2010-11-23 | 2014-04-02 | 金蝶软件（中国）有限公司 | File updating method, device and system |
| CN102122473A (en) * | 2010-12-20 | 2011-07-13 | 上海仙视电子有限公司 | Advertising player system for inquiry and renewal by using 3G (3rd generation) module |
| CN102761861A (en) * | 2011-04-27 | 2012-10-31 | 中国移动通信集团广东有限公司 | Information updating method, server and information updating system |
| CN102810092B (en) * | 2011-05-31 | 2017-07-07 | 腾讯科技（深圳）有限公司 | data read-write method and system |
| CN102355454A (en) * | 2011-08-11 | 2012-02-15 | 杭州顺网科技股份有限公司 | Online game communication acceleration method based on local area network |
| CN102567053A (en) * | 2011-12-20 | 2012-07-11 | 云海创想信息技术（无锡）有限公司 | Method and device for downgrading software version |
| CN102571952B (en) * | 2011-12-31 | 2015-11-25 | 北京金山软件有限公司 | A kind of system and method for transfer files |
| CN102682127B (en) * | 2012-05-16 | 2014-12-03 | 北京像素软件科技股份有限公司 | Data version control method |
| CN102831204B (en) * | 2012-08-09 | 2015-09-30 | 珠海金山网络游戏科技有限公司 | A kind of file packing form possessing Version Control ability |
| CN102915325A (en) * | 2012-08-11 | 2013-02-06 | 深圳市极限网络科技有限公司 | Md5 Hash list-based file decomposing and combining technique |
| CN103678319B (en) * | 2012-09-03 | 2019-03-26 | 腾讯科技（深圳）有限公司 | Resource file update method, device and system and server |
| CN102880681B (en) * | 2012-09-13 | 2016-01-13 | 烽火通信科技股份有限公司 | Based on display and the management method of the big data quantity vector data of grid |
| CN103036964B (en) * | 2012-12-04 | 2015-05-20 | 杭州顺网科技股份有限公司 | Internet bar server data updating method based on peer-to-peer (P2P) |
| US9171027B2 (en) | 2013-05-29 | 2015-10-27 | International Business Machines Corporation | Managing a multi-version database |
| CN103412777A (en) * | 2013-08-13 | 2013-11-27 | 珠海金山网络游戏科技有限公司 | Updating method and system for synchronizing packed files with specified versions |
| CN103955498A (en) * | 2014-04-22 | 2014-07-30 | 北京联时空网络通信设备有限公司 | Search engine creating method and device |
| CN104281717B (en) * | 2014-10-31 | 2017-07-21 | 晶赞广告（上海）有限公司 | A kind of method for setting up magnanimity ID mapping relations |
| CN104573064B (en) * | 2015-01-23 | 2017-12-08 | 四川中科腾信科技有限公司 | A kind of data processing method under big data environment |
| CN104778223B (en) * | 2015-03-25 | 2018-08-21 | 广州多益网络股份有限公司 | The data model method for building up and device in a kind of hand trip house |
| CN104834871A (en) * | 2015-05-12 | 2015-08-12 | 合肥高维数据技术有限公司 | Differentiating method for improving writing rate of hidden storage system |
| CN105335196A (en) * | 2015-11-02 | 2016-02-17 | 深圳市新国都支付技术有限公司 | POS (Point Of Sale) terminal program incremental download method |
| CN107015985B (en) * | 2016-01-27 | 2021-03-30 | 创新先进技术有限公司 | Data storage and acquisition method and device |
| CN106095455A (en) * | 2016-06-17 | 2016-11-09 | 北京小米移动软件有限公司 | A kind of application update method and device |
| CN106131123B (en) * | 2016-06-21 | 2019-03-29 | 福建星网视易信息系统有限公司 | The remote resource synchronous method and system of three-tier architecture |
| CN107545011B (en) * | 2016-06-29 | 2020-04-10 | 高德信息技术有限公司 | Data reading method and device |
| CN106202458B (en) * | 2016-07-15 | 2019-09-17 | 深圳市永达电子信息股份有限公司 | It is a kind of that quickly real-time detection and the method for file in upgrading operation system can be restored |
| KR20180073128A (en) * | 2016-12-22 | 2018-07-02 | 항저우 순왕 테크놀로지 컴퍼니 리미티드 | A data updating method based on data block comparison |
| CN108540511A (en) * | 2017-03-03 | 2018-09-14 | 全球能源互联网研究院 | A kind of method of data synchronization towards Hadoop clusters |
| CN107833637B (en) * | 2017-06-19 | 2020-12-04 | 平安医疗健康管理股份有限公司 | Medicine rule record updating method and device, computer equipment and medium |
| CN107608715B (en) * | 2017-07-20 | 2020-07-03 | 上海寒武纪信息科技有限公司 | Apparatus and method for performing artificial neural network forward operations |
| CN107638694B (en) * | 2017-09-26 | 2021-06-08 | 杭州电魂网络科技股份有限公司 | Development kit downloading method and device and server |
| CN107885526A (en) * | 2017-11-17 | 2018-04-06 | 苏州蜗牛数字科技股份有限公司 | A kind of file updating method and system |
| CN108170460B (en) * | 2017-12-15 | 2020-12-11 | 杭州中天微系统有限公司 | Method and device for upgrading increment of embedded system |
| CN111831297B (en) * | 2019-04-17 | 2021-10-26 | 中兴通讯股份有限公司 | Zero-difference upgrading method and device |
| CN110237535A (en) * | 2019-06-03 | 2019-09-17 | 北京金山安全软件有限公司 | Game data verification method and device, electronic equipment and storage medium |
| CN110493122B (en) * | 2019-08-29 | 2022-01-28 | 北京拉勾网络技术有限公司 | Session information synchronization method and device, computing equipment and storage medium |
| CN110750546B (en) * | 2019-10-21 | 2023-07-25 | 中国民航信息网络股份有限公司 | Database updating method and device |
| CN110841280A (en) * | 2019-11-06 | 2020-02-28 | 福建天晴在线互动科技有限公司 | Method for online dynamic loading and checking of resources of micro-terminal |
| CN111142903B (en) * | 2019-12-23 | 2023-09-22 | 杭州安恒信息技术股份有限公司 | Configuration file interactive updating method and device based on file comparison |
| CN111400279B (en) * | 2020-03-12 | 2021-02-12 | 腾讯科技（深圳）有限公司 | Data operation method, device and computer readable storage medium |
| CN112507100B (en) * | 2020-12-18 | 2023-12-22 | 北京百度网讯科技有限公司 | Update processing method and device of question-answering system |
| CN112905553A (en) * | 2021-02-07 | 2021-06-04 | 全链通有限公司 | Data sharing method, device and system based on block chain |
| CN112799955B (en) * | 2021-02-08 | 2023-09-26 | 腾讯科技（深圳）有限公司 | Method and device for detecting model change, storage medium and electronic equipment |
| CN113094443A (en) * | 2021-05-21 | 2021-07-09 | 珠海金山网络游戏科技有限公司 | Data synchronization method and device |
| CN113377793A (en) * | 2021-06-18 | 2021-09-10 | 西门子（中国）有限公司 | Method and device for updating data block of nonvolatile memory |
| CN113849125B (en) * | 2021-08-30 | 2024-01-09 | 北京东方网信科技股份有限公司 | CDN server disk reading method, device and system |
| CN113760830B (en) * | 2021-09-22 | 2024-01-30 | 国网信息通信产业集团有限公司 | Distributed file storage editable system and method |
| CN114064097B (en) * | 2021-11-26 | 2024-05-10 | 中国联合网络通信集团有限公司 | Software upgrading method, terminal equipment and storage medium |
| CN114936387B (en) * | 2022-07-25 | 2022-11-29 | 深圳市明源云科技有限公司 | Text file reading method, text file writing method, electronic equipment and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1904835A (en) * | 2006-08-08 | 2007-01-31 | 北京飞天诚信科技有限公司 | Method for updating computer software |
| CN101567858A (en) * | 2008-04-21 | 2009-10-28 | 联想(北京)有限公司 | Method and system for synchronizing data |
| CN101582920A (en) * | 2009-06-09 | 2009-11-18 | 中兴通讯股份有限公司 | Method and device for verifying and synchronizing data blocks in distributed file system |
-
2010
- 2010-01-18 CN CN201010040010XA patent/CN101770515B/en active Active
- 2010-10-29 MY MYPI2010005110A patent/MY174951A/en unknown
- 2010-11-03 SG SG2010081495A patent/SG173248A1/en unknown
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1904835A (en) * | 2006-08-08 | 2007-01-31 | 北京飞天诚信科技有限公司 | Method for updating computer software |
| CN101567858A (en) * | 2008-04-21 | 2009-10-28 | 联想(北京)有限公司 | Method and system for synchronizing data |
| CN101582920A (en) * | 2009-06-09 | 2009-11-18 | 中兴通讯股份有限公司 | Method and device for verifying and synchronizing data blocks in distributed file system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101770515A (en) | 2010-07-07 |
| SG173248A1 (en) | 2011-08-29 |
| MY174951A (en) | 2020-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101770515B (en) | Data block comparison based data updating method | |
| US10740315B2 (en) | Transitioning between system sharing types in multi-tenancy database systems | |
| US10621167B2 (en) | Data separation and write redirection in multi-tenancy database systems | |
| CN110019215B (en) | Key pattern management in a multi-tenancy database system | |
| CN109947767B (en) | System sharing type in multi-leasing database system | |
| US10482080B2 (en) | Exchanging shared containers and adapting tenants in multi-tenancy database systems | |
| CN107577427B (en) | data migration method, device and storage medium for blockchain system | |
| KR20200027413A (en) | Method, device and system for storing data | |
| US20190129985A1 (en) | Deploying changes to key patterns in multi-tenancy database systems | |
| CN100553328C (en) | The demand stream media system of intelligent access data | |
| CN102004778B (en) | Text index online updating method in cloud environment | |
| CN104980487A (en) | Service discovery using collection synchronization with exact names | |
| US20020023114A1 (en) | System,method and recording medium for document conversion using an intrmediate computer | |
| JP2001313639A (en) | System and method for managing network configuration data and recording medium | |
| CN106776783B (en) | Unstructured data memory management method and system | |
| CN105243067A (en) | Method and apparatus for realizing real-time increment synchronization of data | |
| CN101350009A (en) | System for writing and compiling cooperated documents | |
| US9021071B2 (en) | Methods of federating applications providing modular data | |
| CN104765840A (en) | Big data distributed storage method and device | |
| KR20040082332A (en) | Network printer connection update scheme for printer clients | |
| CN103164525A (en) | Method and device for WEB application release | |
| CN103207872A (en) | Real-time indexing method and server | |
| JP2020197873A (en) | Information processing system and method for controlling information processing system | |
| CN105677579A (en) | Data access method and system in cache system | |
| CN111798338A (en) | Smart tourism GDS method, system and intermediate server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |