CN110995788A - Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server - Google Patents

Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server Download PDF

Info

Publication number
CN110995788A
CN110995788A CN201911107977.2A CN201911107977A CN110995788A CN 110995788 A CN110995788 A CN 110995788A CN 201911107977 A CN201911107977 A CN 201911107977A CN 110995788 A CN110995788 A CN 110995788A
Authority
CN
China
Prior art keywords
data block
file
stored
data
offset
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911107977.2A
Other languages
Chinese (zh)
Other versions
CN110995788B (en
Inventor
孟令波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Chenhe Quality Inspection Technology Co Ltd
Original Assignee
Guangzhou Chenhe Quality Inspection Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Chenhe Quality Inspection Technology Co Ltd filed Critical Guangzhou Chenhe Quality Inspection Technology Co Ltd
Priority to CN201911107977.2A priority Critical patent/CN110995788B/en
Publication of CN110995788A publication Critical patent/CN110995788A/en
Application granted granted Critical
Publication of CN110995788B publication Critical patent/CN110995788B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/172—Caching, prefetching or hoarding of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • H04L67/1078—Resource delivery mechanisms
    • H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method for realizing breakpoint continuous uploading and file storage of an HTTP server, which specifically comprises the following steps: s1, after receiving the fragments of the uploaded files, the server divides the fragments into smaller data blocks according to the multiple of the memory pages, and marks the data blocks to be completely received and stored through bitmap mapping files; s2, when the received data blocks are continuous and exceed the set number, calculating the MD5 value of the continuous data block set; and S3, after all the fragments of the file are received and stored, the MD5 values are verified by the client, and the stored fragments are combined into a complete file. The invention fragments the uploaded file into smaller data blocks according to rules, maps the file through a bitmap, marks, receives and stores the file, and realizes the problem of repeated coverage of the file through a specific algorithm, thereby realizing the breakpoint continuous uploading of an HTTP protocol.

Description

Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server
Technical Field
The invention relates to the technical field of computers, in particular to a method for realizing breakpoint continuous uploading file storage of an HTTP server.
Background
The HTTP breakpoint resuming refers to that based on the HTTP protocol, when uploading/downloading, a file is divided into several parts, each part is uploaded/downloaded by using a thread, and if a fault interrupt is encountered, the uncompleted part can be uploaded/downloaded continuously from the part that has been uploaded/downloaded, without uploading/downloading from the beginning.
HTTP/1.1 starts to support breakpoint continuous transmission downloading, and through a GET method of an HTTP protocol, a Range and a Content-Range entity header are used for realizing the exchange between a client and a server to acquire a specified file segment.
The URI corresponding to the PUT method of the HTTP protocol is the resource itself to be created or updated. Such as: PUT http: the semantics of// www.example.com/file are to create or update a file. The side effects of multiple PUTs on the same URI are the same as those of one PUT; thus, the PUT method has idempotency. The more correct definition of a PUT is replace (create or update), i.e., if present, a replacement is created, and if not present, it is created.
However, in many current HTTP servers, it is considered that overlapping coverage of files may cause a problem of resource consistency, and a Content-Range entity header under the PUT method is not implemented, so that the PUT method cannot implement breakpoint continuous uploading.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a method for realizing the storage of the breakpoint continuous uploading file of the HTTP server, which divides the uploaded file into smaller data blocks according to rules, maps the file through a bitmap, marks the file for receiving and storing, and realizes the problem of repeated coverage of the file through a specific algorithm, thereby realizing the breakpoint continuous uploading of the HTTP protocol.
In order to achieve the purpose, the invention adopts the following technical scheme:
a method for realizing breakpoint continuous uploading file storage of an HTTP server specifically comprises the following steps:
s1, after receiving the fragments of the uploaded files, the server divides the fragments into smaller data blocks according to the multiple of the memory pages, and marks the data blocks to be completely received and stored through bitmap mapping files;
s2, when the received data blocks are continuous and exceed the set number, calculating the MD5 value of the continuous data block set;
s3, after all the fragments of the file are received and stored, the MD5 values are verified by the client, and the stored fragments are combined into a complete file;
the specific process of step S1 is as follows:
s1.1, a bitmap mapping file FMAP is established by a server according to the total length L of an uploaded file and according to a preset multiple factor F and a memory page size PS, wherein each 2 bits of the bitmap mapping file are mapped into 1 data block, binary 00 represents blank and no data block is stored, binary 11 represents a complete data block, binary 01 or 10 represents that one data block is incomplete and the data length is less than the length of the mapping data block, 01 represents that the tail of the stored data block is offset and aligned with the initial position of the data block with the complete size, and 10 represents that the stored data block starts from the end position of the data block with the complete size;
s1.2, a server receives a fragment SP, an offset SO of the fragment SP relative to the starting position of a complete file and the length SL of the fragment SP; if the length SL of the fragment is less than BS, go to step S1.3, otherwise go to step S1.4;
s1.3, data block merging processing:
s1.3.1, calculating an offset interval according to the offset SO of the slice SP:
offset interval ═ S0, rounded ((SO + PS-1)/PS) ＊ PS ];
s1.3.2, checking whether there is a data block within the offset interval range calculated in step S1.3.1: if the fragment SP does not exist or the fragment SP does not overlap with the existing fragment, the fragment SP and the offset SO thereof are saved; if the fragment SP is overlapped with the existing fragment, comparing the overlapped part, and if the overlapped part is different, performing difference data processing;
s1.3.3, combining the fragment SP with the existing fragment, if the offset after combination is an integral multiple of BS and the length of the combined data block is equal to BS, setting the bit position corresponding to the bitmap mapping file FMAP as 11, which indicates that the corresponding data block is completely stored; finishing the processing;
s1.4, dividing offset SO of SP into piecesConversion to a set of offsets SO1，SO2，SO3...SOn－1，SOnThen calculates the corresponding data size L1，L2，L3，...，Ln－1，LnAnd according to the data size L1，L2，L3，...，Ln－1，LnData block DB is cut out1，DB2，DB3，...，DBn－1，DBnStoring; wherein:
SO1equal to SO;
if S0 is not an integer multiple of BS, then adjust L1To make SO2Is an integer multiple of BS;
SO2＝S01+L1，SO3＝SO2+L2，…，SOn＝SOn－1+Ln－1；
L2，L3，...，Ln－1are the same in size and equal to the BS;
Lnis less than or equal to BS;
SO3＝SO2+BS，SO4＝SO3+BS，…，SOn＝SOn－1+BS；
s1.5, according to the offset set obtained in the step S1.4, sequentially checking bits corresponding to a bitmap mapping file FMAP, if the bits are 11, comparing the stored data block with the corresponding data block to be stored, and if the data are different, performing difference data processing; if the bit is 10, comparing the data block to be stored with the stored data block from the beginning, wherein the comparison data length is the data block with the smaller length, and if the data is different, performing difference data processing; if the bit is 01, comparing the reverse direction from the tail of the data block to be stored with the stored data block, wherein the comparison data length is the smaller of the length of the two data blocks, and if the data are different, performing difference data processing;
the specific process of the difference data processing is as follows:
(1) if the rule is set to overwrite:
if the size of the data block to be stored is the same as that of the stored data block, the stored data block is stored in an overlaying mode;
if the data block to be stored is different from the stored data block, deleting the stored data block, storing the data block to be stored, and updating the offset of the data block relative to the start position of the complete file;
(2) if the rule is set to prohibit overwriting:
ending the file fragmentation processing operation;
s1.6, data block DB obtained by dividing fragment SP1，DB2，DB3，...，DBn－1，DBnStored as a plurality of small files F1、F2....FnRecording and storing the offset of each small file relative to the start position of the complete file; setting a bitmap mapping file FMAP according to the offset set in the step S1.4; wherein:
if the bit of the bitmap mapping file corresponding to the data block to be stored is 00, storing the data block;
if SOn+LnEqual to the full file length, indicates DBnAnd if the data block is the last data block, the bit position of the corresponding bitmap mapping file is 11.
Further, the specific process of step S2 is: and checking the bitmap mapping file, and if the bits in a certain fixed range are all set, namely a set number of continuous data blocks are stored completely, calculating and recording the MD5 value of the continuous data block set.
Further, the set number is 1/N of the total number of the data blocks.
Still further, as a preferable mode, N ═ 10.
Further, the specific process of step S3 is: checking the bitmap mapping file, if all the bit positions are set, checking the MD5 value through the client, sequentially combining all the corresponding data blocks into a large file, calculating the MD5 value of the large file, and finishing uploading after final checking through the client.
And further, after the uploading is finished, clearing the MD5 values of the deleted data blocks, the bitmap mapping file and the continuous data block set.
The invention has the beneficial effects that:
1. the method of the invention decomposes the file fragments into data blocks with fixed size through a specific algorithm and solves the possible overlapping problem of a plurality of file fragments, thereby realizing the breakpoint continuous uploading of HTTP protocol. The invention can ensure that the single file is uploaded and merged and kept consistent after being randomly fragmented.
Detailed Description
The present invention will be further described below, and it should be noted that the present embodiment is based on the technical solution, and a detailed implementation manner and a specific operation process are provided, but the protection scope of the present invention is not limited to the present embodiment.
Example 1
The embodiment provides a method for implementing breakpoint continuous uploading file storage of an HTTP server, which specifically includes:
s1, after receiving the fragments of the uploaded files, the server divides the fragments into smaller data blocks according to the multiple of the memory pages, and marks the data blocks to be completely received and stored through bitmap mapping files;
s2, when the number of the continuous received data blocks exceeds the set number, calculating the MD5 value of the continuous data block set;
and S3, after all the fragments of the uploaded file are received and stored, the client checks all MD5 values, and after the check is passed, the stored fragments are combined into a complete file.
Further, the specific process of step S1 is as follows:
s1.1, a bitmap mapping file FMAP is established by a server according to the total length L of an uploaded file and a preset multiple factor F and a memory page size PS, wherein each 2 bits of the bitmap mapping file are mapped into 1 data block, binary 00 represents blank and no data block is stored, binary 11 represents a complete data block, binary 01 or 10 represents that one data block is incomplete and the data length is smaller than the length of the mapping data block, 01 represents that the tail part of the stored data block is offset and aligned with the initial position of the data block with the complete size, 10 represents that the stored data block starts from the end position of the data block with the complete size, the size of the complete data block is BS (F ＊ PS), and the size of the bitmap mapping file is equal to ((L + BS-1)/F ＊ BS + 3)/4;
s1.2, when the server receives one fragment SP, receiving the offset SO of the fragment SP relative to the starting position of the complete file and the length SL of the fragment SP at the same time; if the length SL of the fragment SP is smaller than BS, turning to step S1.3, otherwise, turning to step S1.4;
s1.3, data block merging processing:
s1.3.1, calculating an offset interval according to the offset SO of the slice SP:
offset interval ═ S0, rounded ((SO + PS-1)/PS) ＊ PS ];
s1.3.2, checking whether the slice in the offset interval range calculated in step S1.3.1 exists, if the slice does not exist or the slice SP does not overlap with the existing slice, saving the slice SP and the offset SO thereof; if the fragment SP is overlapped with the existing fragment, comparing the overlapped part, and if the overlapped part is different, performing difference data processing;
the specific process of the difference data processing is as follows:
(1) if the rule is set to overwrite:
if the size of the data block to be stored is the same as that of the stored data block, the stored data block is stored in an overlaying mode;
if the size of the data block to be stored is different from that of the stored data block, deleting the stored data block, storing the data block to be stored, and updating the offset of the data block relative to the start position of the complete file;
(2) if the rule is set to prohibit overwriting:
ending the file fragmentation processing operation;
s1.3.3, combining the fragment SP with the existing fragment, if the offset after combination is integral multiple of BS and the length of the combined data block is equal to BS, setting the bit position corresponding to the bitmap mapping file FMAP as 11, which indicates that the corresponding data block is completely stored;
s1.4, converting the offset SO of the fragment SP into an offset set SO1，SO2，SO3...SOn－1，SOnThen calculates the corresponding data size L1，L2，L3，...，Ln－1，LnAnd according to the data size L1，L2，L3，...，Ln－1，LnData block DB to be stored is cut out1，DB2，DB3，...，DBn－1，DBnStoring; wherein:
SO1equal to SO;
if SO is not an integer multiple of BS, adjusting L1To make SO2Is an integer multiple of BS;
SO2＝S01+L1，SO3＝SO2+L2，…，SOn＝SOn－1+Ln－1；
L2，L3，...，Ln－1are the same in size and equal to the BS;
Lnis less than or equal to BS; therefore, there are:
SO3＝SO2+BS，SO4＝SO3+BS，…，SOn＝SOn－1+BS；
s1.5, according to the offset set obtained in the step S1.4 and the cut data blocks to be stored, sequentially checking bit positions corresponding to a bitmap mapping file FMAP, if the bit positions are 11, comparing the stored data blocks with the corresponding data blocks to be stored, and if the data are different, performing difference data processing; if the bit is 10, comparing the data block to be stored with the stored data block from the beginning, wherein the comparison data length is the smaller of the two data block lengths, and if the data are different, performing difference data processing; if the bit is 01, comparing the reverse direction from the tail of the data block to be stored with the stored data block, and if the data length is different, performing difference data processing;
the specific process of the difference data processing is as follows:
(1) if the rule is set to overwrite:
if the size of the data block to be stored is the same as that of the stored data block, the stored data block is stored in an overlaying mode;
if the size of the data block to be stored is different from that of the stored data block, deleting the stored data block, storing the data block to be stored, and updating the offset of the data block relative to the start position of the complete file;
(2) if the rule is set to prohibit overwriting:
ending the file fragmentation processing operation;
s1.6, data block DB obtained by dividing fragment SP1，DB2，DB3，...，DBn－1，DBnStored as a plurality of small files F1、F2....FnRecording and storing the offset of each small file relative to the start position of the complete file; setting a bitmap mapping file FMAP according to the offset set in the step S1.4; wherein:
if the bit of the bitmap mapping file corresponding to the data block to be stored is 00, storing the data block;
if SOn+LnEqual to the full file length, indicates DBnAnd if the data block is the last data block, the bit position of the corresponding bitmap mapping file is 11.
Further, the specific process of step S2 is: and checking the bitmap mapping file, and if the bits in a certain fixed range are all set, namely a set number of continuous data blocks are stored completely, calculating and recording the MD5 value of the continuous data block set.
In this embodiment, the set number is 1/N of the total number of data blocks. More specifically, in the present embodiment, N is 10. The reason for taking N to 10 is to assume that the file is divided into 10 shares, because many download tools divide the file into 10 downloads. This value can be modified.
Still further, the specific process of step S3 is: checking the bitmap mapping file, if all the bit positions are set, checking the MD5 value through the client, sequentially combining all the corresponding data blocks into a large file, calculating the MD5 value of the large file, and finishing uploading after final checking through the client.
And further, after the uploading is finished, clearing the MD5 values of the deleted data blocks, the bitmap mapping file and the continuous data block set.
Examples
Assuming that the memory page size is 4096 bytes, the multiple factor is 1, and the size of the complete data block is 4096 bytes; the original file a.zip is 23345, and is divided into 3 fragments; slicing SP0The relative complete file offset is 1 and the length is 5000; SP1The relative full file offset is 5001, length 18000, SP2The relative complete file offset is 23001, and 345 length; the stored tile used to calculate MD5 is 2, the first set of data blocks is 3, and the second set of data blocks is 3.
1. Based on the file length 23345, a bitmap file a.zip.bmap is created, with a size of 2 bytes and a length calculation formula of ((23345+4095)/4096+ 3)/4. Initialize all bits of a.zip.bmap to 0.
2. Processing SP0The offset is 1, length 5000, length greater than 4096 bytes, and the slice is divided into 2 shares, the first share being 4096 bytes and the second share being 904 bytes. Respectively stored as a.zip.p. 1, a.zip.p4097, which corresponds to the 1 st, 2 nd, 3 rd, 4 th bit of the first byte of a.zip.bmap being 0, the 1 st, 2 nd bit of the first byte of a.zip.bmap being set 11, the 3 rd, 4 th bit being set 10.
3. Processing SP1The offset is 5001, the length is 18000, the length is larger than 4096 bytes, the slice is divided into 5 parts, the 1 st part is 3192 bytes, the 2 nd part is 4096 bytes, the 3 rd part is 4096 bytes, the 4 th part is 4096 bytes, and the 5 th part is 2520 bytes; the offsets are 5001, 8193, 12289, 16385, 20481 respectively.
The 2 nd, 3 rd and 4 th offsets are all 4096 times, the length is also 4096 times, the 5 th, 6 th, 7 th and 8 th bit bits of the first byte and the 1 st and 2 nd bit bits of the second byte of a.zip.
The 1 st byte is 3192 bytes and the offset is 5001. checking that the 3 rd and 4 th bits of the corresponding a.zip. bmap first byte are 10 indicates that there is a slice smaller than one data block size from 4097 times. The size of the corresponding fragment is read to be 904 bytes, and it is determined that the 1 st part and the stored fragment can be merged into a complete data block. Setting the 3 rd and 4 th bits of the first byte of the corresponding a.zip.bmap to 11, and merging the storage data blocks.
2520 byte for the 5 th byte, 20481 for the offset, check the bit 3, 4 of the corresponding second byte of a.zip.bmap to 0, set it to 10, and store the slice a.zip.p20481
4. Processing SP2The offset is 23001, the length is 345, the length is smaller than 4096 bytes, the sum of the length and the offset is the size of the complete file, the last fragment is represented, the bit 3 and the bit 4 of the second byte corresponding to the a.zip.bmap are 10, the stored data block is represented, the length is 2520, and the display and SP are respectively shown2Joining with SP2Merge and overwrite storage and set the corresponding bit position 11.
5. Checking a.zip.bmap, completely storing the 1 st data block to the 3 rd data block, and calculating a corresponding MD5 value; the 4 th to 6 th data blocks are completely stored, and the corresponding MD5 is calculated; total 2 MD5 values
6. Checking a.zip.bmap, setting all valid bits, calculating the MD5 value a.zip.md5 of all data blocks, and totaling 1 MD5 value
7. Sending 3 MD5 values of 5 and 6 to a file uploader, after the uploader checks that no errors exist, the server side combines the data blocks into a complete file in sequence, and reserving the MD5 value a.zip.md5
8. The data block, a.zip.bmap, 1 st-3 rd and 4 th-6 th data block MD5 values of the delete slice are cleared.
Various corresponding changes and modifications can be made by those skilled in the art based on the above technical solutions and concepts, and all such changes and modifications should be included in the protection scope of the present invention.

Claims (6)

1. A method for realizing the breakpoint continuous uploading file storage of an HTTP server is characterized by comprising the following steps:
s1, after receiving the fragments of the uploaded files, the server divides the fragments into smaller data blocks according to the multiple of the memory pages, and marks the data blocks to be completely received and stored through bitmap mapping files;
s2, when the received data blocks are continuous and exceed the set number, calculating the MD5 value of the continuous data block set;
s3, after all the fragments of the file are received and stored, the MD5 values are verified by the client, and the stored fragments are combined into a complete file;
the specific process of step S1 is as follows:
s1.1, a bitmap mapping file FMAP is established by a server according to the total length L of an uploaded file and according to a preset multiple factor F and a memory page size PS, wherein each 2 bits of the bitmap mapping file are mapped into 1 data block, binary 00 represents blank and no data block is stored, binary 11 represents a complete data block, binary 01 or 10 represents that one data block is incomplete and the data length is less than the length of the mapping data block, 01 represents that the tail of the stored data block is offset and aligned with the initial position of the data block with the complete size, and 10 represents that the stored data block starts from the end position of the data block with the complete size;
s1.2, a server receives a fragment SP, an offset SO of the fragment SP relative to the starting position of a complete file and the length SL of the fragment SP; if the length SL of the fragment is less than BS, go to step S1.3, otherwise go to step S1.4;
s1.3, data block merging processing:
s1.3.1, calculating an offset interval according to the offset SO of the slice SP:
offset interval ═ S0, rounded ((SO + PS-1)/PS) ＊ PS ];
s1.3.2, checking whether there is a data block within the offset interval range calculated in step S1.3.1: if the fragment SP does not exist or the fragment SP does not overlap with the existing fragment, the fragment SP and the offset SO thereof are saved; if the fragment SP is overlapped with the existing fragment, comparing the overlapped part, and if the overlapped part is different, performing difference data processing;
s1.3.3, combining the fragment SP with the existing fragment, if the offset after combination is an integral multiple of BS and the length of the combined data block is equal to BS, setting the bit position corresponding to the bitmap mapping file FMAP as 11, which indicates that the corresponding data block is completely stored; finishing the processing;
s1.4, converting the offset SO of the fragment SP into an offset set SO1，SO2，SO3...SOn－1，SOnThen calculates the corresponding data size L1，L2，L3，...，Ln－1，LnAnd according to the data size L1，L2，L3，...，Ln－1，LnData block DB is cut out1，DB2，DB3，...，DBn－1，DBnStoring; wherein:
SO1equal to SO;
if S0 is not an integer multiple of BS, then adjust L1To make SO2Is an integer multiple of BS;
SO2＝S01+L1，SO3＝SO2+L2，…，SOn＝SOn－1+Ln－1；
L2，L3，...，Ln－1are the same in size and equal to the BS;
Lnis less than or equal to BS;
SO3＝SO2+BS，SO4＝SO3+BS，…，SOn＝SOn－1+BS；
s1.5, according to the offset set obtained in the step S1.4, sequentially checking bits corresponding to a bitmap mapping file FMAP, if the bits are 11, comparing the stored data block with the corresponding data block to be stored, and if the data are different, performing difference data processing; if the bit is 10, comparing the data block to be stored with the stored data block from the beginning, wherein the comparison data length is the data block with the smaller length, and if the data is different, performing difference data processing; if the bit is 01, comparing the reverse direction from the tail of the data block to be stored with the stored data block, wherein the comparison data length is the smaller of the length of the two data blocks, and if the data are different, performing difference data processing;
the specific process of the difference data processing is as follows:
(1) if the rule is set to overwrite:
if the size of the data block to be stored is the same as that of the stored data block, the stored data block is stored in an overlaying mode;
if the data block to be stored is different from the stored data block, deleting the stored data block, storing the data block to be stored, and updating the offset of the data block relative to the start position of the complete file;
(2) if the rule is set to prohibit overwriting:
ending the file fragmentation processing operation;
s1.6, data block DB obtained by dividing fragment SP1，DB2，DB3，...，DBn－1，DBnStored as a plurality of small files F1、F2....FnRecording and storing the offset of each small file relative to the start position of the complete file; setting a bitmap mapping file FMAP according to the offset set in the step S1.4; wherein:
if the bit of the bitmap mapping file corresponding to the data block to be stored is 00, storing the data block;
if SOn+LnEqual to the full file length, indicates DBnAnd if the data block is the last data block, the bit position of the corresponding bitmap mapping file is 11.
2. The method for implementing HTTP server breakpoint resuming uploading of file storage according to claim 1, wherein the specific process of step S2 is as follows: and checking the bitmap mapping file, and if the bits in a certain fixed range are all set, namely a set number of continuous data blocks are stored completely, calculating and recording the MD5 value of the continuous data block set.
3. The method as claimed in claim 2, wherein the predetermined number is 1/N of the total number of the data blocks.
4. The method of claim 3, wherein N is 10.
5. The method for implementing HTTP server breakpoint resuming uploading of file storage according to claim 2, wherein the specific process of step S3 is as follows: checking the bitmap mapping file, if all the bit positions are set, checking the MD5 value through the client, sequentially combining all the corresponding data blocks into a large file, calculating the MD5 value of the large file, and finishing uploading after final checking through the client.
6. The method for implementing storage of the HTTP server breakpoint continuous upload file according to claim 5, wherein after uploading is finished, MD5 values of deleted data blocks, bitmap mapping files and continuous data block sets are cleared.
CN201911107977.2A 2019-11-13 2019-11-13 Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server Active CN110995788B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911107977.2A CN110995788B (en) 2019-11-13 2019-11-13 Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911107977.2A CN110995788B (en) 2019-11-13 2019-11-13 Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server

Publications (2)

Publication Number Publication Date
CN110995788A true CN110995788A (en) 2020-04-10
CN110995788B CN110995788B (en) 2022-02-22

Family

ID=70084167

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911107977.2A Active CN110995788B (en) 2019-11-13 2019-11-13 Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server

Country Status (1)

Country Link
CN (1) CN110995788B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112073535A (en) * 2020-09-21 2020-12-11 烟台云朵软件有限公司 Bitmap-based data packet fragment transmission method
CN114051152A (en) * 2022-01-17 2022-02-15 飞狐信息技术（天津）有限公司 Video playing method and device, storage medium and electronic equipment
CN115277723A (en) * 2022-07-19 2022-11-01 国能信控互联技术有限公司 Buffer event-based breakpoint continuous transmission method and system for edge acquisition history module

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102710758A (en) * 2012-05-22 2012-10-03 苏州云博信息技术有限公司 Method for continuously transferring breakpoint of data stream based on single chip system
CN103858440A (en) * 2011-08-31 2014-06-11 高通股份有限公司 Switch signaling methods providing improved switching between representations for adaptive http streaming
CN104253816A (en) * 2014-09-25 2014-12-31 武汉传神信息技术有限公司 HTTP (hyper text transport protocol)-based large file page breakpoint resuming method
CN104679830A (en) * 2015-01-30 2015-06-03 乐视网信息技术（北京）股份有限公司 File processing method and device
EP3065341A1 (en) * 2015-03-05 2016-09-07 Mitsubishi Electric R&D Centre Europe B.V. Content classification medthod and device
CN107566540A (en) * 2017-10-31 2018-01-09 云宏信息科技股份有限公司 Big document breakpoint transmission method, apparatus and system
CN108540566A (en) * 2018-04-18 2018-09-14 暴风集团股份有限公司 file uploading method, device, system and client and server
CN109639807A (en) * 2018-12-19 2019-04-16 中国四维测绘技术有限公司 A kind of massive remote sensing image file network transmission method based on slice slice
US10303797B1 (en) * 2015-12-18 2019-05-28 EMC IP Holding Company LLC Clustering files in deduplication systems
US20190236102A1 (en) * 2018-01-29 2019-08-01 Planet Data Solutions System and method for differential document analysis and storage
US10445292B1 (en) * 2016-03-30 2019-10-15 Veritas Technologies Llc Sharing services between deduplication systems

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103858440A (en) * 2011-08-31 2014-06-11 高通股份有限公司 Switch signaling methods providing improved switching between representations for adaptive http streaming
CN102710758A (en) * 2012-05-22 2012-10-03 苏州云博信息技术有限公司 Method for continuously transferring breakpoint of data stream based on single chip system
CN104253816A (en) * 2014-09-25 2014-12-31 武汉传神信息技术有限公司 HTTP (hyper text transport protocol)-based large file page breakpoint resuming method
CN104679830A (en) * 2015-01-30 2015-06-03 乐视网信息技术（北京）股份有限公司 File processing method and device
EP3065341A1 (en) * 2015-03-05 2016-09-07 Mitsubishi Electric R&D Centre Europe B.V. Content classification medthod and device
US10303797B1 (en) * 2015-12-18 2019-05-28 EMC IP Holding Company LLC Clustering files in deduplication systems
US10445292B1 (en) * 2016-03-30 2019-10-15 Veritas Technologies Llc Sharing services between deduplication systems
CN107566540A (en) * 2017-10-31 2018-01-09 云宏信息科技股份有限公司 Big document breakpoint transmission method, apparatus and system
US20190236102A1 (en) * 2018-01-29 2019-08-01 Planet Data Solutions System and method for differential document analysis and storage
CN108540566A (en) * 2018-04-18 2018-09-14 暴风集团股份有限公司 file uploading method, device, system and client and server
CN109639807A (en) * 2018-12-19 2019-04-16 中国四维测绘技术有限公司 A kind of massive remote sensing image file network transmission method based on slice slice

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
WEN YUJIE，TONG JIARONG，CHARLES CHIANG: ""FMAP_a_technology_mapping_algorithm_for_FPGA_with_MUX-LUT_mixed_architecture"", 《ASIC, 2003. PROCEEDINGS. 5TH INTERNATIONAL CONFERENCE ON》 *
李杰绪: ""一种改进的P2P文件传输系统的研究与实现"", 《中国优秀硕士学位论文全文数据库-信息科技辑》 *
王恒亮: ""基于应用层动态特征识别技术的数据流分析软件工具包"", 《中国优秀硕士学位论文全文数据库-信息科技辑》 *
陈威: ""轻量级BitTorrent网络平台设计与实现"", 《中国优秀硕士学位论文全文数据库-信息科技辑》 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112073535A (en) * 2020-09-21 2020-12-11 烟台云朵软件有限公司 Bitmap-based data packet fragment transmission method
CN114051152A (en) * 2022-01-17 2022-02-15 飞狐信息技术（天津）有限公司 Video playing method and device, storage medium and electronic equipment
CN115277723A (en) * 2022-07-19 2022-11-01 国能信控互联技术有限公司 Buffer event-based breakpoint continuous transmission method and system for edge acquisition history module

Also Published As

Publication number Publication date
CN110995788B (en) 2022-02-22

Similar Documents

Publication Publication Date Title
CN110995788B (en) Method for realizing breakpoint continuous uploading and file storage of HTTP (hyper text transport protocol) server
US10725772B2 (en) Electric control device for vehicles, program updating method, and server apparatus
US7546492B2 (en) Remotely repairing files by hierarchical and segmented cyclic redundancy checks
US20120185612A1 (en) Apparatus and method of delta compression
US9619334B2 (en) System and method for merging continuous volume snapshots
US20050010916A1 (en) System for providing software application updates to multiple clients on a network
US10416915B2 (en) Assisting data deduplication through in-memory computation
US10732881B1 (en) Region cloning for deduplication
JPH0778067A (en) Method and system for data compression
US10346256B1 (en) Client side cache for deduplication backup systems
US11169796B2 (en) Methods and systems for remote software update
US20110246763A1 (en) Parallel method, machine, and computer program product for data transmission and reception over a network
US20080275923A1 (en) Method for the expungement of backup versions of files on server targets that are configured to be updated sequentially
US20170060957A1 (en) Optimized and consistent replication of file overwrites
US11797488B2 (en) Methods for managing storage in a distributed de-duplication system and devices thereof
JP2004199121A (en) Difference updating method, program and apparatus
US10802819B2 (en) Binary image differential patching
US11860739B2 (en) Methods for managing snapshots in a distributed de-duplication system and devices thereof
CN114518897A (en) Remote upgrading method and system for communication module
JP4285679B2 (en) Differential transfer method, program and system
US20160037119A1 (en) Video recording apparatus, and video recording method when temporary network failure occurs
US20180060349A1 (en) Method for transferring a difference file
JP2005100007A (en) Data file saving system and allocation method
JP6979987B2 (en) Information processing equipment
WO2017022034A1 (en) Information processing device, information processing method, and information processing program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant