CN111193786A - Method for reading large file in segmentation mode based on Blob object to improve uploading efficiency - Google Patents
Method for reading large file in segmentation mode based on Blob object to improve uploading efficiency Download PDFInfo
- Publication number
- CN111193786A CN111193786A CN201911338003.5A CN201911338003A CN111193786A CN 111193786 A CN111193786 A CN 111193786A CN 201911338003 A CN201911338003 A CN 201911338003A CN 111193786 A CN111193786 A CN 111193786A
- Authority
- CN
- China
- Prior art keywords
- blob
- file
- characteristic
- segmented
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method for reading large files in a segmented mode based on Blob objects and improving uploading efficiency, which comprises the following steps: loading a file to be transmitted into a browser; converting files intoBlobObject, obtainingBlobA characteristic value of the object; segmentationBlobForming an object into a plurality of segments, and acquiring a plurality of segment characteristic values; preservation of the originalBlobThe characteristic value of the object and the segmented fragment is a characteristic object, and the characteristic object is transmitted to a server; transmitting the segmented fragments to a server; the server side checks the consistency of the characteristic value and the data; the server side merges the fragments and checks the original of the characteristic objectBlobAnd (5) the consistency of the characteristic values of the object and the merged file is realized, and the file uploading is completed. By the method, the problem that the data transmission is limited by the size of the file is solved, the integrity of information transmission is realized, the data transmission efficiency is improved, and the data transmission method is suitable for mass data transmissionWebApplications upload data providing the basis.
Description
Technical Field
The invention relates to the technical field of software development, in particular to a method for reading large files in a segmented mode based on Blob objects and improving uploading efficiency.
Background
WebUploading data in an application is a common data transfer operation.
At present, most of direct file uploading methods cause data instability, for example, the data is easily uploaded due to network interruption or overlarge files, so that a user needs to manually upload again, and the working efficiency is affected.
Due to the fact thatHTTPIs emerging, the browser is added withAjaxAndBlobinterface equally such thatWebEnd can useJava ScriptAnd directly uploading the local file.
HTTP(Hypertext transfer protocol) is used inWebBasic protocol for uploading files based onTCP/IPA communication protocol to transfer data, most typically between a browser and a server so that a surfer can browse them. At present, the methodHTTPThe version of the specification document isHTTP/2。
AjaxMeans thatAsynchronous JavaScript+XML(asynchronous)JavaScriptAndXML) It is not a new technology per se, but one was covered in 2005Jesse James GarrettNew terminology is proposed to describe a 'new' method using a set of prior art techniques, comprising:HTML or XHTML,Cascading Style Sheets,JavaScript,The Document Object Model,XML,XSLTand most importantlyXMLHttpRequest object. When using combinations of these techniquesAJAXAfter the model, the web application can quickly present the incremental updates on the user interface without having to reload (refresh) the entire page. This enables the program to go back fasterThe operation of the user is satisfied, and the use efficiency is improved.
BlobThe object represents a class file object of immutable, raw data.BlobIs not necessarily represented byJavaScriptData in native format.FileInterface is based onBlobInherit fromBlobAnd extend it to support files on the user's system.
MD5Information summarization algorithm (english:MD5 Message-Digest Algorithm) A widely used cryptographic hash function produces a 128-bit (16-byte) hash value (16-byte)hash value) And the method is used for ensuring the completeness and consistency of information transmission.MD5By the American cryptologist Rhonid Levister (Ronald Linn Rivest) Design, published in 1992, to replaceMD4And (4) an algorithm. The programming of this set of algorithms is specified in the RFC 1321 standard. After 1996 the algorithm proved to have weaknesses and could be broken, and experts generally recommend changing to other algorithms for data requiring a high degree of security, e.g.SHA-2. In 2004, it provedMD5Algorithm cannot prevent collision: (collision) And is therefore not suitable for security authentication, e.g.SSLPublic key authentication, digital signature, and the like.
In the prior art, the transmission of a large amount of data often fails to upload frequently, frequent uploading is needed, and the user waits overtime and the error of the merged file is serious in the process of merging the files at the server side.
Disclosure of Invention
Aiming at the technical problems in the related art, the invention provides a method for reading a large file in a segmented mode based on a Blob object to improve uploading efficiency, and the defects in the prior art can be overcome.
In order to achieve the technical purpose, the technical scheme of the invention is realized as follows:
a method for reading large files in a segmented mode based on Blob objects to improve uploading efficiency comprises the following steps:
s1: loading a file to be transmitted into a browser;
s2: by usingJavaScriptConverting a document intoBlobObject, obtainingBlobA characteristic value of the object, wherein the characteristic value ismd5；
S3: by usingJavaScriptDivision ofBlobForming an object into a plurality of segments, and acquiring a plurality of segment characteristic values;
s4: preservation of the originalBlobThe characteristic value of the object and the segmented fragment is a characteristic object, and the characteristic object is transmitted to a server side by adopting a browser;
s5: by usingAjaxTransmitting the segmented fragments to a server;
s6: the server side checks the consistency of the characteristic value and the data;
s7: the server side merges the fragments and checks the original of the characteristic objectBlobAnd (5) the consistency of the characteristic values of the object and the merged file is realized, and the file uploading is completed.
Further, the step S1 includes the following steps:
s11: preparing a file to be transmitted;
s12: loading the file into a local file system;
s13: and acquiring the file from the local file system by adopting a browser.
Further, the step S3 includes the following steps:
s31: obtainingBlobThe length of the object data;
s32: setting the length of the segment;
s33: according toBlobObject data length, segmentationBlobThe object is divided into a plurality of segments with corresponding segment lengths;
s34: and searching and recording characteristic values of a plurality of segments.
Further, the step S6 includes the following steps:
s61: acquiring fragment data information;
s62: acquiring characteristic values of the segmentation segments in the characteristic object;
s63: analyzing and comparing feature values of the segmentation segments and the segmentation segments in the feature objects.
Further, the step S7 includes the following steps:
s71: the server merges the fragments into a file according to the sequence;
s72: obtaining an original in the feature objectBlobA characteristic value of the object;
s73: analyzing and comparing the raw in the feature objectBlobThe object and the feature value of the merged file.
The invention has the beneficial effects that: by the method, the problem that the data transmission is limited by the size of the file is solved, the integrity of information transmission is realized, the data transmission efficiency is improved, and the data transmission method is suitable for mass data transmissionWebApplications upload data providing the basis.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic diagram of an embodiment of the present inventionBlobAnd the flow block diagram of the method for reading the large file by the object segmentation to improve the uploading efficiency.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
As shown in fig. 1, the method for reading a large file based on Blob object segmentation to improve uploading efficiency according to the embodiment of the present invention includes the following steps:
s1: loading a file to be transmitted into a browser;
s2: by usingJavaScriptConverting a document intoBlobObject, obtainingBlobThe characteristic values of the object, wherein,characteristic value ofmd5；
S3: by usingJavaScriptDivision ofBlobForming an object into a plurality of segments, and acquiring a plurality of segment characteristic values;
s4: preservation of the originalBlobThe characteristic value of the object and the segmented fragment is a characteristic object, and the characteristic object is transmitted to a server side by adopting a browser;
s5: by usingAjaxTransmitting the segmented fragments to a server;
s6: the server side checks the consistency of the characteristic value and the data;
s7: the server side merges the fragments and checks the original of the characteristic objectBlobAnd (5) the consistency of the characteristic values of the object and the merged file is realized, and the file uploading is completed.
Step S1 includes the following steps:
s11: preparing a file to be transmitted;
s12: loading the file into a local file system;
s13: and acquiring the file from the local file system by adopting a browser.
Step S3 includes the following steps:
s31: obtainingBlobThe length of the object data;
s32: setting the length of the segment;
s33: according toBlobObject data length, segmentationBlobThe object is divided into a plurality of segments with corresponding segment lengths;
s34: and searching and recording characteristic values of a plurality of segments.
Step S6 includes the following steps:
s61: acquiring fragment data information;
s62: acquiring characteristic values of the segmentation segments in the characteristic object;
s63: analyzing and comparing feature values of the segmentation segments and the segmentation segments in the feature objects.
Step S7 includes the following steps:
s71: the server merges the fragments into a file according to the sequence;
s72: obtaining the characteristic objectOf (2) originalBlobA characteristic value of the object;
s73: analyzing and comparing the raw in the feature objectBlobThe object and the feature value of the merged file.
In order to facilitate understanding of the above-described technical aspects of the present invention, the above-described technical aspects of the present invention will be described in detail below in terms of specific usage.
The invention adds a proofreading mechanism on the basis of the prior art, can reduce the uploading operation times when the application transmits a large amount of data, and solves the problems that the user waits overtime in the process of combining the files at the server side and how to ensure the correctness of the combined files.
The data segmentation and uploading method comprises the following steps:
(1) acquiring a file to be transmitted from a local file system by using a browser;
(2) use ofJavaScriptConverting files to be transmitted into filesBlobObject and acquireBlobCharacteristic value of object (md5）；
(3) Use ofJavaScriptHandleBlobDividing the object into a plurality of segments and obtaining all segment feature values (md5）；
We can holdBlobThe data can be analogized to an array [1,2,3,4,5,6,7,8,9 ]]Then dividing the file into fixed lengths according to the length of the dataBlobObjects, i.e. in this example, divide the object into 3 shares, respectively [1,2,3],[ 4,5,6], [7,8,9]；
(4) Storing the characteristic values of the original object and the segmented object as a characteristic object, and transmitting the characteristic object to a server;
the original object is [1,2,3,4,5,6,7,8,9 ]]Its eigenvalue we usemd5The method is obtained, and is assumed to bemd5123456789The eigenvalues of similar segmented objects are also assumed to bemd512，md5456，md5789The transmitted values are:
{
‘file_md5’:’md5123456789’,
‘file_split_md5’:[
’md5123’,
’md5456’,
’md5789’,
]
}
(5) will be dividedBlobTransmitting the object to a server;
use ofAjaxWill be provided withBlobThe file is transmitted to a server;
(6) the server side checks whether the characteristic value is consistent with the data, and if not, the client side is required to send out the corresponding data againBlobAn object;
after receiving data, the server side is allBlobOf documentsmd5Checking if the characteristic value of the file is found to be equal tofile_split_md5In (1)md5If the values are inconsistent, returning the corresponding file to the client, and if the client receives the found incomplete information, retransmitting the corresponding file until all the data are satisfied;
(7) the server merges the files and checksmd5If the two are consistent, the uploading process is finished;
after the server merges the files according to the sequence, the characteristic value sum of the merged files is checkedfile_md5And if the two are consistent, returning error information to enable the client to retransmit. If they are consistent, it indicates that the data is [1,2,3,4,5,6,7,8,9 ]]The result is correct.
The invention adoptsJavaScriptDivide the file and useAjaxThe method utilizes local bandwidth to the maximum, and obviously improves the transmission efficiency and the use characteristic value of data relative to the common method (by adopting the method of (1))md5) The automatic repair method for the file information after secondary proofreading and transmission error finding ensures the integrity of the data in the transmission process.
Md5The method is a universal method for checking the characteristic value of the file, can be used for quickly identifying whether two files are consistent, and can be used for checking whether the file is changed before and after transmission, so that the integrity of the file in the transmission process is ensured.
BlobThe data is obtained by reading files by the browser, but the data can be arbitrarily divided in the program because the data is similar to an arrayThe failure of block and data transmission is caused by network interruption, and the failure of transmission caused by network interruption can be reduced by dividing a large file into small blocks and then combining the small blocks again.
AjaxThe method is provided by the browser for transmitting data, and can support the simultaneous transmission of a plurality of data. This maximizes the use of bandwidth resources. In file transfer, it is possible to have them transferred simultaneously, which is faster than the direct transfer of a single file.
The invention integrates the three aspects and adds a check and retransmission mechanism to improve the uploading efficiency and reduce the operation times. Based onHTTPProtocol, by reading local files from browser, with JavaScriptThe method comprises the steps of dividing a file into a plurality of small blocks, acquiring a characteristic value of the file, transmitting the file and the characteristic value to a server through a browser, if the characteristic value of the transmitted file is different from the characteristic value acquired in advance, sending retransmission data in the browser, and if the characteristic values of all the files are consistent, completing verification operation and file combination operation at the server to finally obtain a correct file.
Using pre-transmission and reception problemsmd5The values and the transmission times serve as evaluation criteria for the transmission results. 3 orders of magnitude different sized files are used. The data transmission is carried out by using a common direct transmission method and a text method respectively. Uploading files from a browser and sending the files to a server, and if the server receives a problem, extracting a characteristic value (md5) Feature values extracted from the source file(s) ((md5) And if the two files are consistent, the two files are considered to be the same, which indicates that the transmission result is correct. If the transmission time is shorter by using the method, the method is considered to improve the file uploading efficiency compared with the common direct transmission method. It has been found that the transmission time of the method is shorter.
In summary, with the above technical solution of the present invention, the method solves the problem of limitation of file size in data transmission, achieves integrity of information transmission, improves data transmission efficiency, and provides massive data for transmissionWebApplications upload data providing the basis.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (5)
1. A method for reading large files in a segmented mode and improving uploading efficiency based on Blob objects is characterized by comprising the following steps:
s1: loading a file to be transmitted into a browser;
s2: by usingJavaScriptConverting a document intoBlobObject, obtainingBlobA characteristic value of the object, wherein the characteristic value ismd5；
S3: by usingJavaScriptDivision ofBlobForming an object into a plurality of segments, and acquiring a plurality of segment characteristic values;
s4: preservation of the originalBlobThe characteristic value of the object and the segmented fragment is a characteristic object, and the characteristic object is transmitted to a server side by adopting a browser;
s5: by usingAjaxTransmitting the segmented fragments to a server;
s6: the server side checks the consistency of the characteristic value and the data;
s7: the server side merges the fragments and checks the original of the characteristic objectBlobAnd (5) the consistency of the characteristic values of the object and the merged file is realized, and the file uploading is completed.
2. The method for improving uploading efficiency based on the segmented reading of the Blob object large file according to claim 1, wherein the step S1 includes the following steps:
s11: preparing a file to be transmitted;
s12: loading the file into a local file system;
s13: and acquiring the file from the local file system by adopting a browser.
3. The method for improving uploading efficiency based on the segmented reading of the Blob object large file according to claim 1, wherein the step S3 includes the following steps:
s31: obtainingBlobThe length of the object data;
s32: setting the length of the segment;
s33: according toBlobObject data length, segmentationBlobThe object is divided into a plurality of segments with corresponding segment lengths;
s34: and searching and recording characteristic values of a plurality of segments.
4. The method for improving uploading efficiency based on the segmented reading of the Blob object large file according to claim 1, wherein the step S6 includes the following steps:
s61: acquiring fragment data information;
s62: acquiring characteristic values of the segmentation segments in the characteristic object;
s63: analyzing and comparing feature values of the segmentation segments and the segmentation segments in the feature objects.
5. The method for improving uploading efficiency based on the segmented reading of the Blob object large file according to claim 1, wherein the step S7 includes the following steps:
s71: the server merges the fragments into a file according to the sequence;
s72: obtaining an original in the feature objectBlobA characteristic value of the object;
s73: analyzing and comparing the raw in the feature objectBlobThe object and the feature value of the merged file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911338003.5A CN111193786A (en) | 2019-12-23 | 2019-12-23 | Method for reading large file in segmentation mode based on Blob object to improve uploading efficiency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911338003.5A CN111193786A (en) | 2019-12-23 | 2019-12-23 | Method for reading large file in segmentation mode based on Blob object to improve uploading efficiency |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111193786A true CN111193786A (en) | 2020-05-22 |
Family
ID=70709339
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911338003.5A Pending CN111193786A (en) | 2019-12-23 | 2019-12-23 | Method for reading large file in segmentation mode based on Blob object to improve uploading efficiency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111193786A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111881108A (en) * | 2020-07-27 | 2020-11-03 | 广州锦行网络科技有限公司 | File transmission method and device based on web ssh |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070292023A1 (en) * | 2006-06-20 | 2007-12-20 | Baer Richard L | Data reduction for wireless communication |
| US20130188926A1 (en) * | 2012-01-20 | 2013-07-25 | Comcast Cable Communications, Llc | Network storage device and method |
| CN105224257A (en) * | 2015-10-16 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | The disposal route of large files and system in a kind of cloud storage system |
| CN105635324A (en) * | 2016-03-17 | 2016-06-01 | 新浪网技术(中国)有限公司 | Big file uploading and continuous uploading method and device for browser or server |
| CN106921682A (en) * | 2015-12-24 | 2017-07-04 | 百度在线网络技术（北京）有限公司 | File uploading method and device |
| CN107609054A (en) * | 2017-08-24 | 2018-01-19 | 郑州云海信息技术有限公司 | Upload, method for down loading and the processing system of big file in a kind of object storage system |
| CN109361754A (en) * | 2018-11-05 | 2019-02-19 | 中国广核电力股份有限公司 | A kind of document transmission method and device based on browser |
-
2019
- 2019-12-23 CN CN201911338003.5A patent/CN111193786A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070292023A1 (en) * | 2006-06-20 | 2007-12-20 | Baer Richard L | Data reduction for wireless communication |
| US20130188926A1 (en) * | 2012-01-20 | 2013-07-25 | Comcast Cable Communications, Llc | Network storage device and method |
| CN105224257A (en) * | 2015-10-16 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | The disposal route of large files and system in a kind of cloud storage system |
| CN106921682A (en) * | 2015-12-24 | 2017-07-04 | 百度在线网络技术（北京）有限公司 | File uploading method and device |
| CN105635324A (en) * | 2016-03-17 | 2016-06-01 | 新浪网技术(中国)有限公司 | Big file uploading and continuous uploading method and device for browser or server |
| CN107609054A (en) * | 2017-08-24 | 2018-01-19 | 郑州云海信息技术有限公司 | Upload, method for down loading and the processing system of big file in a kind of object storage system |
| CN109361754A (en) * | 2018-11-05 | 2019-02-19 | 中国广核电力股份有限公司 | A kind of document transmission method and device based on browser |
Non-Patent Citations (1)
| Title |
|---|
| 古兰精: "利用Blob对象实现大文件分片上传", 《博客园HTTPS://WWW.CNBLOGS.COM/GOLOVING/P/10439718.HTML》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111881108A (en) * | 2020-07-27 | 2020-11-03 | 广州锦行网络科技有限公司 | File transmission method and device based on web ssh |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11501533B2 (en) | Media authentication using distributed ledger | |
| EP3660664B1 (en) | File folder transmission on network | |
| US9792384B2 (en) | Remote retreival of data files | |
| US7860892B2 (en) | Information processing apparatus, history file generation method and program | |
| US20150326667A1 (en) | Peer-to-peer sharing of cloud-based content | |
| US10938575B2 (en) | Signature compression for hash-based signature schemes | |
| US20140071482A1 (en) | Intermediary apparatus, image processing apparatus, computer-readable storage medium storing instructions for intermediary apparatus, computer-readable storage medium storing instructions for image processing apparatus, and communication method | |
| US20080046459A1 (en) | Method for generating metadata and electronic device having metadata delivery function | |
| US20120030224A1 (en) | Enabling active content in messaging using automatic data replacement | |
| US11245537B2 (en) | System and method for a local server with self-signed certificates | |
| CN108512898B (en) | File pushing method and device, computer equipment and storage medium | |
| CN112491989A (en) | Data transmission method, device, equipment and storage medium | |
| CN110046524A (en) | A kind of processing method of data, device, electronic equipment and medium | |
| CN104079623B (en) | Multistage cloud storage synchronisation control means and system | |
| US8005920B2 (en) | Method and apparatus for transmitting arbitrarily large amounts of data over the internet | |
| CN111193786A (en) | Method for reading large file in segmentation mode based on Blob object to improve uploading efficiency | |
| US7987455B1 (en) | System and method of command processing | |
| CN114866533A (en) | File uploading method and device and electronic equipment | |
| CN111339551A (en) | Data verification method and related device and equipment | |
| US9531905B2 (en) | Scanner that is capable of uploading scan data in a target area within a data storage server | |
| US20120030556A1 (en) | Method for enabling interactive content within messaging application | |
| US10956659B1 (en) | System for generating templates from webpages | |
| US6714950B1 (en) | Methods for reproducing and recreating original data | |
| CN115297104B (en) | File uploading method and device, electronic equipment and storage medium | |
| US20230036217A1 (en) | Systems and methods for using a structured data database and for exchanging electronic files containing unstructured or partially structered data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200522 |