CN113079219A

CN113079219A - Large file fragment uploading method and system

Info

Publication number: CN113079219A
Application number: CN202110388121.8A
Authority: CN
Inventors: 李华
Original assignee: Beijing Minglue Zhaohui Technology Co Ltd
Current assignee: Beijing Minglue Zhaohui Technology Co Ltd
Priority date: 2021-04-12
Filing date: 2021-04-12
Publication date: 2021-07-06

Abstract

The invention provides a large file fragment uploading method and a large file fragment uploading system, wherein the technical scheme of the method comprises a file pre-operation step, wherein corresponding fragment sizes are matched for files based on a preset standard according to the sizes of the files; a hash calculation step of calculating a hash value of a file to be uploaded; a hash checking step, namely checking the hash value of the file and acquiring the uploading state of the file according to the checking result; a file processing step, namely performing fragmentation operation on the files which are not uploaded completely according to the uploading state and uploading the fragments; and a file merging step, namely merging the uploaded fragments of the file to obtain the complete file. The invention solves the problem that the prior art is inconvenient in uploading large files.

Description

Large file fragment uploading method and system

Technical Field

The invention belongs to the field of machine learning, and particularly relates to a large file fragment uploading method and system.

Background

In many current business application scenarios, large file uploading is an important interactive scenario, such as uploading large Excel form data into a database, uploading video and audio files, and the like. If the file volume is large or the network condition is not good, the uploading time is long (more messages need to be transmitted and the probability of packet loss retransmission is high), and the user cannot refresh the page and can only wait for the completion of the request with patience. And the transmission is easily interrupted due to the influence of various factors of network instability, and the user can only select to upload the file again.

Disclosure of Invention

The embodiment of the application provides a large file fragment uploading method and system, which at least solve the problem that the prior art is inconvenient in large file uploading.

In a first aspect, an embodiment of the present application provides a large file fragment uploading method, including: a hash calculation step of calculating a hash value of a file to be uploaded; a hash checking step, namely checking the hash value of the file and acquiring the uploading state of the file according to the checking result; a file processing step, namely performing fragmentation operation on the files which are not uploaded completely according to the uploading state and uploading the fragments; and a file merging step, namely merging the uploaded fragments of the file to obtain the complete file.

Preferably, the method further comprises a file pre-operation step, wherein the file is matched with the corresponding fragment size based on a preset standard according to the size of the file before the hash value of the file is calculated.

Preferably, the hash calculation step includes: the hash value of the file is calculated by SparkMD 5.

Preferably, the file processing step includes: judging the number of fragments of the file, uploading the file with the fragment number below a threshold in parallel, and uploading the file with the fragment number exceeding a threshold in sequence.

Preferably, the file merging step includes: and merging the fragments in a Stream mode.

In a second aspect, an embodiment of the present application provides a large file fragment uploading system, which is suitable for the above large file fragment uploading method, and includes: the hash calculation unit is used for calculating the hash value of a file to be uploaded; the hash checking unit is used for checking the hash value of the file and acquiring the uploading state of the file according to the checking result; the file processing unit is used for carrying out fragmentation operation on the files which are not uploaded completely according to the uploading state and uploading the fragments; and the file merging unit is used for merging the uploaded fragments of the file to obtain the complete file.

In some embodiments, the file pre-operation unit is further included, and before calculating the hash value of the file, the file pre-operation unit matches, according to the size of the file, a corresponding fragment size for the file based on a preset criterion.

In some of these embodiments, the hash calculation unit comprises: the hash value of the file is calculated by SparkMD 5.

In some of these embodiments, the document processing unit includes: judging the number of fragments of the file, uploading the file with the fragment number below a threshold in parallel, and uploading the file with the fragment number exceeding a threshold in sequence.

In some embodiments, the file merging unit includes: and merging the fragments in a Stream mode.

Compared with the related art, the large file fragment uploading method provided by the embodiment of the application adopts the fragment uploading technology, and if the network transmission is interrupted, the user reselects the file and only needs to transmit the rest fragments without retransmitting the whole file, so that the retransmission cost is greatly reduced.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is a flowchart of a large file fragment uploading method of the present invention;

FIG. 2 is a block diagram of a large file fragment uploading system according to the present invention;

FIG. 3 is a block diagram of an electronic device of the present invention;

in the above figures:

1. a file pre-operation unit; 2. a hash calculation unit; 3. a hash check unit; 4. a document processing unit; 5. a file merging unit; 60. a bus; 61. a processor; 62. a memory; 63. a communication interface.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Embodiments of the invention are described in detail below with reference to the accompanying drawings:

fig. 1 is a flowchart of a large file fragment uploading method of the present invention, please refer to fig. 1, the large file fragment uploading method of the present invention includes the following steps:

s1: and matching the corresponding fragment size for the file based on a preset standard according to the size of the file.

In specific implementation, a visual operation interface can be set, a file is selected or dragged in the interface, and a proper fragment size is selected, optionally, the proper fragment size can be automatically matched according to the file size, and the fragment size can also be customized; optionally, the interface can be provided with specific information of the display file, and a progress bar is arranged to display the real-time progress, so that friendly operation prompt information is provided.

S2: and calculating the hash value of the file to be uploaded.

Optionally, the hash value of the file is calculated by SparkMD 5.

In specific implementation, the file is segmented according to the segment size by Blob, the content of the file is read according to bytes by FileReader, and is converted into an ArrayBuffer object (binary buffer), and SparkMD5 is used for calculating the final hash of the file.

S3: and verifying the hash value of the file, and acquiring the uploading state of the file according to the verification result.

In specific implementation, the uploading state of the file comprises that the file is uploaded, part of the file is uploaded and the file is not uploaded, and different operations are subsequently performed according to different states.

S4: and according to the uploading state, carrying out fragmentation operation on the files which are not uploaded, and uploading the fragments.

In a specific implementation, for a status of "file uploaded", the upload is prompted.

In a specific implementation, for a state of "file partial upload", a partial fragment is constructed and uploaded.

In a specific implementation, for a state of "file not uploaded", all fragments are constructed and uploaded.

Optionally, the number of fragments of the file is determined, the file with the fragment number below a threshold is uploaded in parallel, and the file with the fragment number exceeding a threshold is uploaded in sequence.

In specific implementation, different fragment data needs to be constructed for transmission according to different application layer transmission protocols. The HTTP requests consume resources relatively, so that the requests with the quantity less than or equal to 10 can be uploaded in parallel, and the requests with the quantity greater than 10 can be executed in sequence by depending on the synchronization characteristic of async awake, thereby reducing the sudden increase of memory consumption of the browser in a short time and preventing uploading breakdown. Furthermore, in a specific implementation, the Websocket API supports the transmission of binary data, but requires placing the Socket in "string mode" or "binary mode", whereas socket.io now supports transmit buffers (node.js), Blob, ArrayBuffer and even File, so native Websocket is not chosen here.

S5: and combining the uploaded fragments of the file to obtain the complete file.

Optionally, the fragments are merged in a Stream manner.

In a specific implementation, file fragment merging is commonly in three forms: adding files, merging Buffer, and merging Stream, optionally, merging Stream is used in this embodiment.

The add-file mode merge refers to merge using fs. The function of apppendfile () is to asynchronously append data, which may be a string or buffer, to a file, which is created if the file does not exist.

Buffer mode merging is a common file merging mode, and the method is to read each fragment file by fs. The method is simple and easy to understand, but has the disadvantages of large size of the read file, large memory occupied by the merging process and low efficiency. Meanwhile, the upper limit of the default buffer size of the Node is 2GB, and once the uploaded large file exceeds 2GB, the method fails to be used.

For the Stream merge approach, a Stream is a collection of data-like an array or string. The difference is that the data in the stream may not be available all at once and there is no need to put all of this data into memory at once, which makes the stream very efficient when handling large amounts of data or when data is sent from an external source in segments.

In other words, when a 2GB file is processed using the buffer method, the occupied memory may be more than 2GB, and when a stream is used to process the file, only tens of M may be occupied. All streams are instances of eventemiter. They issue events that can be used to read or write data. However, in implementations, the data in the stream may optionally be used in a simpler manner using the pipe method.

In a specific implementation, a writeable stream may be created through fs. And then, respectively reading the files after the fragments by using fs. createReadStream (), and then, pouring the read data into the writable stream in a pipe () way like pouring water, and immediately pouring a cup of water until all the water is poured after monitoring that the cup of water is poured. At this point, all files are merged.

It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.

The embodiment of the application provides a large file fragment uploading system, which is suitable for the large file fragment uploading method. As used below, the terms "unit," "module," and the like may implement a combination of software and/or hardware of predetermined functions. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware or a combination of software and hardware is also possible and contemplated.

Fig. 2 is a frame diagram of a large file fragment uploading system according to the present invention, please refer to fig. 2, which includes:

file pre-operation unit 1: and matching the corresponding fragment size for the file based on a preset standard according to the size of the file.

The hash calculation unit 2: and calculating the hash value of the file to be uploaded.

Optionally, the hash value of the file is calculated by SparkMD 5.

Hash check unit 3: and verifying the hash value of the file, and acquiring the uploading state of the file according to the verification result.

The document processing unit 4: and according to the uploading state, carrying out fragmentation operation on the files which are not uploaded, and uploading the fragments.

The file merging unit 5: and combining the uploaded fragments of the file to obtain the complete file.

Optionally, the fragments are merged in a Stream manner.

In addition, a large file fragment uploading method described in conjunction with fig. 1 may be implemented by an electronic device. Fig. 3 is a block diagram of an electronic device of the present invention.

The electronic device may comprise a processor 61 and a memory 62 in which computer program instructions are stored.

Specifically, the processor 61 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 62 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 62 may include a Hard Disk Drive (Hard Disk Drive, abbreviated HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 62 may include removable or non-removable (or fixed) media, where appropriate. The memory 62 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 62 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 62 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (earrom), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.

The memory 62 may be used to store or cache various data files that need to be processed and/or used for communication, as well as possible computer program instructions executed by the processor 61.

The processor 61 may read and execute the computer program instructions stored in the memory 62 to implement any one of the large file fragment uploading methods in the above embodiments.

In some of these embodiments, the electronic device may also include a communication interface 63 and a bus 60. As shown in fig. 3, the processor 61, the memory 62, and the communication interface 63 are connected via a bus 60 to complete communication therebetween.

The communication port 63 may be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.

The bus 60 includes hardware, software, or both to couple the components of the electronic device to one another. Bus 60 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 60 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 60 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

The electronic device can execute the large file fragment uploading method in the embodiment of the application.

In addition, in combination with the large file fragment uploading method in the foregoing embodiments, embodiments of the present application may provide a computer-readable storage medium to implement the method. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any one of the large file fragment uploading methods in the above embodiments.

And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A large file fragmentation uploading method is characterized by comprising the following steps:

a hash calculation step of calculating a hash value of a file to be uploaded;

a hash checking step, namely checking the hash value of the file and acquiring the uploading state of the file according to the checking result;

a file processing step, namely performing fragmentation operation on the files which are not uploaded completely according to the uploading state and uploading the fragments;

and a file merging step, namely merging the uploaded fragments of the file to obtain the complete file.

2. The large file fragment uploading method according to claim 1, further comprising a file pre-operation step of matching the file with a corresponding fragment size based on a preset criterion according to the size of the file before calculating the hash value of the file.

3. The large file fragmentation uploading method according to claim 1, wherein the hash calculation step comprises: the hash value of the file is calculated by SparkMD 5.

4. The large document fragment uploading method of claim 1, wherein the document processing step comprises: judging the number of fragments of the file, uploading the file with the fragment number below a threshold in parallel, and uploading the file with the fragment number exceeding a threshold in sequence.

5. The large file fragmentation uploading method of claim 1, wherein the file merging step comprises: and merging the fragments in a Stream mode.

6. A large file fragmentation upload system, comprising:

the hash calculation unit is used for calculating the hash value of a file to be uploaded;

the hash checking unit is used for checking the hash value of the file and acquiring the uploading state of the file according to the checking result;

the file processing unit is used for carrying out fragmentation operation on the files which are not uploaded completely according to the uploading state and uploading the fragments;

and the file merging unit is used for merging the uploaded fragments of the file to obtain the complete file.

7. The large file fragment uploading system of claim 6, further comprising a file pre-operation unit, wherein before the hash value of the file is calculated, the file is matched with a corresponding fragment size based on a preset criterion according to the size of the file.

8. The large file shard uploading system of claim 6, wherein the hash calculation unit comprises: the hash value of the file is calculated by SparkMD 5.

9. The large document fragment uploading system of claim 6, wherein the document processing unit comprises: judging the number of fragments of the file, uploading the file with the fragment number below a threshold in parallel, and uploading the file with the fragment number exceeding a threshold in sequence.

10. The large file shard uploading system of claim 6, wherein the file merging unit comprises: and merging the fragments in a Stream mode.