CN117240847A

CN117240847A - File fragment transmission method, device, system and equipment

Info

Publication number: CN117240847A
Application number: CN202311320252.8A
Authority: CN
Inventors: 刘健; 杨艳光; 李欣然; 张同虎; 张玉鑫; 刘可昕; 肖鹏涛
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2023-10-11
Filing date: 2023-10-11
Publication date: 2023-12-15

Abstract

The application provides a file fragment transmission method, a device, a system and equipment, which relate to the technical field of big data, and the method comprises the following steps: the cache window conversion service module receives each file fragment corresponding to the target transmission file and fragment metadata corresponding to the file fragment sent by the receiving server; the slicing metadata comprises slicing serial numbers of file slicing; according to the fragment metadata, saving the file fragments to a file cache window which is created in advance; the storage position of the file fragments in the file cache window is related to the fragment serial numbers of the file fragments; and converting the data forms of the file fragments one by one according to the storage positions of the file fragments stored in the file cache window so as to send the file fragments to the target receiving equipment according to the sequence of the file fragments. According to the application, by constructing the file cache window, file fragments arriving at the front end in an out-of-order manner can be converted into stream data form with consistent sequence for transmission, and the complexity of file fragment transmission is reduced.

Description

File fragment transmission method, device, system and equipment

Technical Field

The present application relates to the field of big data, and in particular, to a method, an apparatus, a system, and a device for file fragment transmission.

Background

The hypertext transfer protocol (Hyper Text Transfer Protocol, HTTP) is a request-response protocol commonly used for interactions between a browser and a server. When using a browser, a user may typically download files from a server or upload files to a server using HTTP technology.

In the process of uploading files, after receiving large files locally, the receiving server usually needs to transfer the large files to a remote special storage server due to the limited storage space of the receiving server.

In the related art, the requirements for the above-mentioned storage server are very strict, for example, the storage server must support random access, which results in higher complexity of file fragment transmission.

Disclosure of Invention

The application provides a file fragment transmission method, a device, a system and equipment, which can reduce the complexity of file fragment transmission.

In a first aspect of the present application, a method for transmitting a file fragment is provided, the method comprising:

receiving each file fragment corresponding to a target transmission file and fragment metadata corresponding to the file fragment sent by a receiving server; wherein, the fragment metadata comprises fragment serial numbers of the file fragments;

According to the fragment metadata, saving the file fragments to a file cache window which is created in advance; the storage position of the file fragments in the file cache window is related to the fragment serial numbers of the file fragments;

and converting the data form of each file fragment one by one according to the storage position of each file fragment stored in the file cache window so as to send each file fragment to target receiving equipment according to the sequence of the fragments.

In one embodiment, the storing the file fragments to a file cache window created in advance according to the fragment metadata includes:

determining a sequence relationship between a plurality of storage locations in the file cache window;

determining the storage position of the file fragment in the file cache window according to the sequence relation between the fragment sequence number in the fragment metadata;

and storing the file fragments to the file cache window according to the storage positions.

In one embodiment, the method further comprises:

a data stream conversion unit is created in advance, and the data stream conversion unit is used for providing an input stream and an output stream.

In one embodiment, the converting, one by one, the data form of each file fragment according to the storage location of each file fragment stored in the file cache window includes:

and writing each file fragment into the data stream conversion unit one by one according to the storage position of each file fragment stored in the file cache window.

In a second aspect of the present application, there is provided a file fragment transmission method, including:

receiving each file fragment corresponding to a target transmission file sent by a client;

transmitting the file fragments and fragment metadata corresponding to the file fragments to a pre-created cache window conversion service module; the cache window conversion service module stores the file fragments to a file cache window which is created in advance according to the fragment metadata; the file fragments are stored in the file cache window, wherein the fragment metadata comprises fragment serial numbers of the file fragments, and storage positions of the file fragments in the file cache window are related to the fragment serial numbers of the file fragments.

In one embodiment, each file fragment corresponding to the target transmission file sent by the receiving client includes:

Receiving N file fragments corresponding to the target transmission file sent by the client in a parallel sending mode; and N is a positive integer, and is smaller than or equal to the current maximum number of the transmittable file fragments of the transmitting window of the client.

In one embodiment, the method further comprises:

when the file fragments are received, locking the current file creation operation authority;

judging whether the file cache window is currently created by the cache window conversion service module;

if the file cache window is not currently created by the cache window conversion service module, the cache window conversion service module is instructed to create the file cache window;

unlocking the current file creation operation authority.

In a third aspect of the present application, there is provided a file fragment transmission apparatus, comprising:

the receiving unit is used for receiving each file fragment corresponding to the target transmission file sent by the receiving server and fragment metadata corresponding to the file fragment; wherein, the fragment metadata comprises fragment serial numbers of the file fragments;

the transfer unit is used for storing the file fragments to a file cache window which is created in advance according to the fragment metadata; the storage position of the file fragments in the file cache window is related to the fragment serial numbers of the file fragments;

And the processing unit is used for converting the data form of each file fragment one by one according to the storage position of each file fragment stored in the file cache window so as to send each file fragment to the target receiving equipment according to the fragment sequence.

In one embodiment, the transfer unit is configured to:

In an embodiment, the apparatus further comprises a preprocessing unit for:

In one embodiment, the processing unit is configured to:

In a fourth aspect of the present application, there is provided a file fragment transmission apparatus, comprising:

The receiving unit is used for receiving each file fragment corresponding to the target transmission file sent by the client;

the sending unit is used for sending the file fragments and fragment metadata corresponding to the file fragments to a pre-created cache window conversion service module; the cache window conversion service module stores the file fragments to a file cache window which is created in advance according to the fragment metadata; the file fragments are stored in the file cache window, wherein the fragment metadata comprises fragment serial numbers of the file fragments, and storage positions of the file fragments in the file cache window are related to the fragment serial numbers of the file fragments.

In one embodiment, the receiving unit is configured to:

In an embodiment, the apparatus further comprises a preprocessing unit for:

unlocking the current file creation operation authority.

In a fifth aspect of the present application, a file fragment transmission system is provided, where the system includes a client, a receiving server, and a cache window conversion service module;

the client is used for sending each file fragment corresponding to the target transmission file to the receiving server;

the buffer window conversion service module is used for executing the file fragment transmission method provided by the first aspect;

the receiving server is configured to perform the file fragment transmission method as provided in the second aspect.

In a sixth aspect of the application, there is provided an apparatus comprising: a memory and a processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored by the memory, causing the processor to perform a file-slicing transmission method as provided in the first aspect, or to perform a file-slicing transmission method as provided in the second aspect.

In a seventh aspect of the present application, there is provided a computer-readable storage medium having stored therein computer-executable instructions for implementing the file-slicing transmission method as provided in the first aspect or the file-slicing transmission method as provided in the second aspect when executed by a processor.

According to an eighth aspect of the present application there is provided a computer program product comprising computer program code which, when run on a computer, causes the computer to perform the method of file-slicing transmission as provided in the first aspect, or the method of file-slicing transmission as provided in the second aspect.

According to the file fragment transmission method, device, system and equipment provided by the embodiment of the application, the file cache window is constructed, after the receiving server receives the file fragments, the received file fragments and fragment metadata are handed over to the cache window, and the cache window stores the fragment data in the corresponding positions in the cache window according to the fragment sequence and the metadata; meanwhile, according to the storage positions of the file fragments stored in the file cache window, the data forms of the file fragments are converted one by one, so that the file fragments arriving at the front end in an out-of-order manner can be provided with a single sequential stream data format for the target receiving device in a unified stream output interface mode, and the process can reduce the requirement on the operation complexity of the target receiving device (such as the requirement on random access capability), thereby reducing the complexity of file fragment transmission.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic diagram of one possible scenario provided by an embodiment of the present application;

fig. 2 is a flow chart of a file fragment transmission method according to an embodiment of the present application;

fig. 3 is another flow chart of a file slice transmission method according to an embodiment of the present application;

fig. 4 is an interactive flow diagram of a file fragment transmission method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of another interaction flow of a file fragment transmission method according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a file slice transmission device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of another file slice transmission device according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a server according to the present application.

Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.

At present, the downloading technology of a browser is used for uploading large files, the file uploading speed is low, the size of the files is limited by a gateway, the uploading cannot be continued after the uploading process is interrupted, and the problem of incomplete files caused by the loss of data packets in HTTP transmission is solved. Related technologies propose a scheme of performing slicing processing and transmission on a large file, so as to ensure the integrity and efficient transmission of the file.

In enterprise-level applications, after a large file is received locally at a server side, the large file is dumped to a storage server of a remote special storage, and the dumping process firstly needs to store or cache the large file locally, so that huge additional disk IO and network IO overhead is brought to the server side, the response time to a front-end user is prolonged, and bad experience is caused to the whole transmission process. The related technology further provides real-time dump of the file fragments on the basis that the storage or the cache of the file is not needed at a server side, and the file fragments are dumped to a storage server in real time when the file fragments are received, so that the cost of a data transmission process from the receiving server to the storage server is optimized, and the performance of the whole system is improved again.

However, when the concurrent access request carried by the server side reaches a critical value, the related art architecture cannot support larger-scale storage and concurrent access requests. In other words, the storage server has limited storage space, and when there are a large number of file dump demands on the server or a plurality of server sides all initiate file dump requests to the storage server, the single-point storage server is difficult to meet the file dump demands on the server side, and the throughput performance of the storage server may be seriously affected.

In order to improve the throughput performance and the storage capacity of the whole system, in some embodiments, a scheduling server is newly added, so that the horizontal expansion capacity of a receiving server and a storage server can be improved, corresponding storage servers can be rapidly and orderly distributed for target files, and the situation that the load of some storage servers is too large or too small can not occur, so that the dump requirement of a large number of concurrent access requests of the receiving server side can be met.

However, in the above embodiments, the back-end storage device must be based on a random-accessible storage device, which is too demanding for most scenarios and limits the selectivity of the back-end storage device.

In view of the above technical problems, the present application proposes the following technical ideas:

by using a buffer window mechanism, a buffer window is added, so that files arriving at the front end out of order are fragmented, and a single sequential stream data format can be provided for the back end storage device in a unified stream output interface mode. Therefore, the complexity of the interface requirement of the back-end storage device is reduced, the limitation of similar random access requirements is eliminated, and a simplified operation scheme is provided for carrying out various complex processes on the file.

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals refer to the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of the application. The embodiments described below by referring to the drawings are illustrative and intended to explain the present application and should not be construed as limiting the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

Referring to fig. 1, fig. 1 is a schematic diagram of a possible scenario provided by an embodiment of the present application, and it should be noted that the file fragment transmission method of the present application may be used in the field of financial science and technology. The file fragment transmission method can also be used in any field except the field of financial science and technology, and the application field of the file fragment transmission method is not limited.

As shown in fig. 1, the system comprises a terminal device 110, a plurality of receiving servers 120, a scheduling server 130 and a plurality of storage servers 140, wherein the terminal device 110, the server 120, the scheduling server 130 and the storage servers 140 are connected with each other through a wired or wireless network. The number of the receiving servers 120 and the number of the storage servers 140 may be the same or different, alternatively, this embodiment takes 3 receiving servers and 5 storage servers as examples, where the terminal device 110 is configured to initiate a request for transmitting one or more target files to each receiving server 120, the receiving server 120 is configured to generate a dump request based on the transmission request initiated by the terminal device 110, and request the corresponding storage server 140 to the scheduling server 130, the scheduling server 130 is configured to generate a directory structure table, and each directory in the directory structure table allocates a corresponding storage server according to an information summary value corresponding to each target file of each receiving server after receiving the dump request, and determines a corresponding directory for each target file of each receiving server and allocates a corresponding storage server 140 according to the directory structure table. The receiving server 120 is further configured to record the storage server 140 after acquiring the storage server 140 allocated to the scheduling server 130, and dump the file fragments into the storage server 140 in real time when receiving the file fragments of the terminal device 110.

The terminal device 110 may include, but is not limited to, a computer, a smart phone, a tablet computer, an e-book reader, a dynamic image expert compression standard audio layer 3 (Moving Picture experts group audio layer III, MP3 for short) player, a dynamic image expert compression standard audio layer 4 (Moving Picture experts group audio layer IV, MP4 for short) player, a portable computer, a car-mounted computer, a wearable device, a desktop computer, a set-top box, a smart television, and the like.

The receiving server 120, the dispatching server 130 and the storage server 140 may be independent physical servers, may be a server cluster or a distributed system formed by a plurality of physical servers, and may also be cloud servers for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), basic cloud computing services such as big data and artificial intelligence platforms, and the like.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

The scene diagram of the application is briefly described above, and the file fragment transmission method provided by the embodiment of the application is described in detail below.

Referring to fig. 2, fig. 2 is a flow chart of a file fragment transmission method according to an embodiment of the present application, where the method includes:

step S201, receiving each file fragment corresponding to the target transmission file sent by the receiving server and fragment metadata corresponding to the file fragment; the fragment metadata comprises fragment serial numbers of the file fragments.

In this embodiment, the cache window conversion service module may be pre-established in the distributed system by using a cache window mechanism. The file fragment transmission method can be executed by the cache window conversion service module.

In some embodiments, the client may send each file fragment corresponding to the target transmission file to the receiving server. When receiving a file fragment, the receiving server locks the current file creation operation authority; judging whether a file cache window is established currently by the cache window conversion service module; if the file cache window is not currently created by the cache window conversion service module, the cache window conversion service module is instructed to create the file cache window, and the current file creation operation authority is unlocked; if the buffer window conversion service module has created the file buffer window at present, the current file creation operation authority is directly unlocked.

In some embodiments, the receiving server may transfer the received file fragments and fragment metadata information to the file cache window.

Step S202, according to the fragment metadata, saving the file fragments to a file cache window created in advance; the storage position of the file fragments in the file cache window is related to the fragment serial numbers of the file fragments.

In some embodiments, the cache window transformation service module may first determine a sequential relationship between a plurality of storage locations in a file cache window; and then determining the storage position of the file fragments in the file cache window according to the sequence relation between the fragment serial numbers in the fragment metadata and the sequence relation, and storing the file fragments in the file cache window according to the storage position.

Step 203, according to the storage positions of the file fragments stored in the file cache window, converting the data form of each file fragment one by one, so as to send each file fragment to the target receiving device according to the sequence of fragments.

In some embodiments, the buffer window conversion service module may write each file fragment into the data stream conversion unit one by one according to the storage location of each file fragment stored in the file buffer window, so that the data form of each file fragment may be converted one by one.

According to the file fragment transmission method provided by the embodiment of the application, after the receiving server receives the file fragments, the buffer window conversion service module hands over the received file fragments and fragment metadata to the buffer window, and the buffer window stores the fragment data in the corresponding positions in the buffer window according to the fragment sequence and the metadata; meanwhile, according to the storage positions of the file fragments stored in the file cache window, the data forms of the file fragments are converted one by one, so that the file fragments arriving at the front end in an out-of-order manner can be provided with a single sequential stream data format for the target receiving device in a unified stream output interface mode, and the process can reduce the requirement on the operation complexity of the target receiving device (such as the requirement on random access capability), thereby reducing the complexity of file fragment transmission.

Referring to fig. 3, fig. 3 is another flow chart of a file slice transmission method according to an embodiment of the present application, and in some embodiments of the present application, the method includes:

step S301, each file fragment corresponding to the target transmission file sent by the client is received.

In this embodiment, the above-described file fragment transmission method may be performed by the receiving server.

In some embodiments, the receiving server may receive each file fragment corresponding to the target transmission file sent by the client.

The client can send each file fragment corresponding to the target transmission file to the receiving server in a parallel sending mode. When the client sends each file fragment corresponding to the target transmission file to the receiving server in a parallel sending mode, the receiving server can receive each file fragment corresponding to the target transmission file sent by the client in a parallel receiving mode.

Step S302, sending the file fragments and fragment metadata corresponding to the file fragments to a pre-created cache window conversion service module; and the cache window conversion service module stores the file fragments to a file cache window which is created in advance according to the fragment metadata.

The file fragment metadata comprises fragment serial numbers of the file fragments, and storage positions of the file fragments in the file cache window are related to the fragment serial numbers of the file fragments.

In some embodiments, the receiving server locks the current file creation operation authority when receiving the file fragments; judging whether a file cache window is established currently by the cache window conversion service module; if the file cache window is not currently created by the cache window conversion service module, the cache window conversion service module is instructed to create the file cache window, and the current file creation operation authority is unlocked; if the buffer window conversion service module has created the file buffer window at present, the current file creation operation authority is directly unlocked.

In some embodiments, the receiving server transfers the received file fragments and fragment metadata information to the file cache window, so that the file cache window stores the fragment data in a proper position in sequence according to the metadata.

According to the file fragment transmission method provided by the embodiment of the application, the file cache window is constructed, and after the receiving server receives the file fragments, the received file fragments and fragment metadata are handed over to the cache window, so that the cache window can store the fragment data at the corresponding positions in the cache window according to the fragment sequence according to the metadata, and further the data form of each file fragment is converted one by one according to the storage positions of each file fragment stored in the file cache window, the file fragments which are reached by the front end in an out-of-order manner can be provided with a single sequential stream data format for the target receiving device in a unified stream output interface mode, the requirement on the operation complexity of the target receiving device is reduced, and the complexity of file fragment transmission is reduced.

Referring to fig. 4, fig. 4 is an interactive flow chart of a file fragment transmission method according to an embodiment of the present application, where the method includes:

in step S401, the client sends each file fragment corresponding to the target transmission file to the receiving server.

In some embodiments, the client may be installed in the terminal device 110 shown in fig. 1.

In some embodiments, the client may send, in a parallel sending manner, each file fragment corresponding to the target transmission file to the receiving server.

Specifically, the client may convert the original sequential transmission into parallel transmission with the maximum parallelism of N, detect whether an undelivered file fragment exists before each transmission, and check whether the transmission window has reached an upper limit, and if so, wait for a delay T and then re-circulate the detection. When the transmission window is 0 and there is no unsent file fragment, the next step is entered.

In some embodiments, when the client sends the file fragments to the server, if the server cache window is full, the client returns to the cache full state, and at this time, the client may attempt to request the cache state from the server after waiting for a certain time interval until the window cache is available to stop.

Step S402, the receiving server sends the file fragments and fragment metadata corresponding to the file fragments to the cache window conversion service module.

Alternatively, the file cache window may be a separate component located behind the receiving server, in communication with the receiving server in a particular fashion, and responsive to each file fragment received by the receiving server, performing caching and out-of-order rearrangement.

In some embodiments, the file cache window may be unified with the receiving server to be inherited into one process for execution on the application deployment, or may be two processes with interaction capability.

Optionally, the fragment metadata may include key information such as a verification hash value, a fragment sequence number, and an uploading task identifier of the file fragment.

And step S403, the cache window conversion service module stores the received file fragments to a file cache window which is created in advance according to the fragment metadata.

The storage position of the file fragments in the file cache window is related to the fragment serial numbers of the file fragments.

In some embodiments, a sequential relationship between a plurality of storage locations in a file cache window may be determined first; and then determining the storage position of the file fragments in the file cache window according to the sequence relation between the fragment serial numbers in the fragment metadata and the sequence relation, and storing the file fragments in the file cache window according to the storage position.

Step S404, the buffer window conversion service module converts the data form of each file fragment one by one according to the storage position of each file fragment stored in the file buffer window so as to send each file fragment to the target receiving device according to the fragment sequence.

In some embodiments, a data stream conversion unit for providing an input stream and an output stream may be created in advance.

Among them, outputStream and InputStream are two types of Java for processing data streams, outputStream is used for writing data to various destinations such as files, network connections, etc., and InputStream is used for reading data from various sources such as files, network connections, etc. In general, outputStream and InputStream provide a unified interface for processing data input and output, so that a server can select different data sources and targets to perform read-write operations according to requirements.

By creating the data stream conversion unit in the embodiment of the application, the writing thread can be automatically blocked when the buffer is full and the reading thread can be automatically blocked when the buffer is empty based on the combination of the InputStream and the OutputStream, thereby simplifying the front and rear thread control. In addition, the data stream conversion unit can convert the non-standard data of the file fragments into a standard sequential reading form of the InputStream, and is also beneficial to standardized butt joint by utilizing most of the existing external devices.

In some embodiments, the cache window conversion service module performs the data sequence integrity check in parallel after placing the newly received file fragment into the file cache window. For example, when the first file fragment arrives, the data stream conversion unit is called, data is written into the data stream conversion unit, until the next file fragment of the received file fragment is missing, the writing of the data stream conversion is paused, and the arrival of the next file fragment is waited; when the data stream conversion writing process finds that the full part of the file fragments have been written, the back-end stream conversion interface is stopped, and after all the data is sent to the target receiving device, the data stream conversion unit is turned off.

According to the file fragment transmission method provided by the embodiment of the application, the file cache window is constructed, after the receiving server receives the file fragments, the received file fragments and fragment metadata are handed over to the cache window, and the cache window stores the fragment data in the corresponding positions in the cache window according to the fragment sequence and the metadata; meanwhile, according to the storage positions of the file fragments stored in the file cache window, the data forms of the file fragments are converted one by one, so that the file fragments arriving at the front end in an out-of-order manner can be provided with a single sequential stream data format for the target receiving device in a unified stream output interface mode, and the process can reduce the requirement on the operation complexity of the target receiving device, thereby reducing the complexity of file fragment transmission.

Referring to fig. 5, fig. 5 is another interactive flow chart of a file fragment transmission method according to an embodiment of the present application, where the embodiment integrally describes a client, a receiving server, a cache window conversion service module and a scheduling server, and the flow chart is as follows:

step S501, the user selects a target file at the client.

Step S502, obtaining a target file selected by a user. If the user does not select the target file, the client can acquire the target file selected by the user through the onchange event by utilizing the JS script.

Step S503, the client reads the size, length, etc. of the target file.

In step S504, the client calculates the target file MD5, specifically, using spark-md5. Js.

Step S505, the client creates a unique character string TaskID for the target file.

Step S506, the client sends step 1: the upload/begin request (file transfer request, carrying basic information of the target transfer file) is sent to the receiving server in the form of an http post request.

Specifically, the client initiates an http post request, submits information such as a file name of a target file, an MD5 value of the whole file, a file length and the like, and a process unique identification code TaskID to the receiving server.

Step S507, the receiving server records the basic information of the target file and stores the basic information in the target database, and initiates a dump request to the dispatching server.

In step S508, the cache window conversion service module initiates a server registration request to the scheduling server, which may take the form of form submission.

Step S509, the dispatch server receives the registration request, registers the storage servers, and records each registered storage server.

Step S510, creating a directory structure table for MD5, wherein the directory structure table contains 256×256 directories formed by the possible combinations of the first four bits of the MD5 digest.

And S511, performing association configuration on the catalogue in the target structure table and the registered storage server to obtain configuration information.

It can be understood that the above steps S501-S511 are the beginning stages of file slice transmission.

It should be noted that, there is no order division between the steps S508-S511 and the steps S501-S507, where the storage of the storage server and the allocation of the directory structure table may be before or after the selection of the target file. In this embodiment, S508 is taken as an example.

Step S512, the scheduling server receives the dump request, and matches the target file with the target storage server according to the first 4-bit characters of the information abstract value of the target file and the configuration information.

Step S513, the receiving server acquires and records the information of the target storage server for which the scheduling server matches.

In step S514, the client performs the slicing operation on the target file, and performs initial setting on the slices, for example, the total number of slices is a, and the initial slice number is 0, so as to record the slicing transmission process conveniently.

Step S515, the client sets an execution condition: it is determined whether there is an unsent slice, if so, step S516 is performed, and if not, step S531 is performed.

Step S516, the client judges whether the upper limit of the sending window is reached; if so, after waiting for the time T, the process returns to S515, and if not, the process continues to S517.

In step S517, the client calculates a file slice MD5, a file slice Size, and the like.

Step S518, the client sends step 2: an upload/slice request (file slice transmission request) is transmitted to the reception server in a parallel transmission form.

Step S519, the receiving server performs MD5 verification on the file fragment.

Step S520, after the MD5 value of the file fragment passes the verification, the receiving server locks the current file 'creation operation' authority through a global lock control (such as a redission-lock).

Step S521, the cache window conversion service module confirms the file cache window creation condition.

Step S522, the receiving server judges whether the current file cache window is created according to the file creation condition of the cache window conversion service module, and if not, the step S523 is executed; if created, step S524 is performed.

S523 the cache window conversion service module creates a file cache window,

specifically, the buffer window conversion service module may create a data stream conversion unit while creating a file buffer window, where the data stream conversion unit may be an OutputStream- > InputStream conversion unit.

Step S524, unlocking the creation operation authority of the current file and locking the writing (writing operation) authority of the current file fragment.

The method comprises the steps of locking file creation permission, namely, global file creation permission for locking the digest value of a source file MD 5; unlocking the file creation operation authority, namely unlocking the global file creation authority of the digest value of the source file MD 5.

Step S525, the buffer window conversion service module stores the file fragments transmitted at this time in the file buffer window, and records the situation that all the file fragments reach the file buffer window.

Step S526, the receiving server determines that the file fragment caching is completed, and returns a transmission result of the client.

Step S527, the cache window conversion service module determines whether the next file fragment of the currently transmitted node has arrived, and if yes, step S528 is executed.

S528, the latest arrived file fragments are written into the OutputStream- > InputStream conversion unit, and streaming data is continuously sent to the target receiving equipment.

And S529, the cache window conversion service module judges whether all the file fragments are sent completely, and if yes, the step S530 is executed.

Step S530, turning off the OutputStream- > InputStream conversion unit.

It can be understood that the steps S512-S530 are the stage of transmitting and dumping the fragments.

Step S531, the client sends a request step 3: an upload/finish request (file fragment transfer completion request).

In step S532, the receiving server initiates an instruction message for calculating the MD5 total value of the file fragment to the cache window conversion service module.

Step S533, the cache window conversion service module requests the target receiving device to finish file consistency verification.

Optionally, after the whole transmission is finished, the receiving server may rely on the cache window conversion service module to send a data consistency verification request to the remote target receiving device, and receive a verification interface, and if the remote target receiving device is not a persistent storage device, the verification process may not be performed.

It should be noted that, the foregoing solution provided in this embodiment includes the interaction procedure between the scheduling server side and the receiving server side in the foregoing embodiment, which can correspondingly implement all the method steps implemented in the foregoing embodiment, and can achieve the same technical effects, and detailed descriptions of the same parts and beneficial effects as those of the method embodiment in this embodiment are omitted.

Based on the description in the above embodiments, in some embodiments of the present application, the file cache window uses the data slice cache as a core, and may be divided into four segments, which are respectively: a send segment, a readable segment, a receive segment, and a blank segment.

In some embodiments, a maximum length MAX may be set for the entire cache to limit the upper memory footprint of the cache. In practical implementation, the method can be implemented in an array with a fixed length of MAX, and the start and stop marks and the global start mark of the four segments are defined and cached at the same time. Wherein:

global start flag: and recording the starting sequence number of the whole data segment, wherein the sequence number is initialized to 0 and increases along with the received file fragment sequence number.

Transmitting a fragmentation mark: the transmission segment contains only one file fragment, with a flag that is set to null when no transmission is performed. The file fragments at the transmitting end are set only when the application starts to write the file fragments to the back end, and are withdrawn after the writing is completed.

Readable start/end of segment marker: the readable segment is a continuous array of fragments that can be transmitted directly to the backend system. The readable start tag points to the first file fragment that has been received but has not yet been reclaimed, and when there is no readable data, the start tag points to a null value. The readable end of segment flag points to the last readable file fragment and if there is no readable file fragment, points to a null value. Theoretically, the transmission segment flag is located within the readable segment start end flag.

Receive segment start/end markers: meaning that the data segment is already in transmission, or that a partial file fragment has been transmitted, but there are cases where a preamble fragment has not yet arrived. The segment head file fragment of the segment is generally in a sending state, and the file fragment from the middle to the segment bit is in the sending state or is sent completely. When the transmission ends, the reception segment starts and the end flag is null.

Blank section start/end flag: a blank segment is a segment of data that has not yet stored valid readable data and is not covered by a received segment marker, which can be occupied by a received segment when receiving a new file fragment. The blank segment start/end marks point to the start and end positions, respectively, of the unoccupied data segment.

In some embodiments, the allocation method of the buffer resource of the file buffer window may include:

the initial stage: the whole buffer memory resource is available, and the received data, the readable data and the sending data do not exist. At this time, the sender marks the blank value, the readable start/end marks the blank value, the blank start mark=0, the end mark=max-1, and the global start mark is 0.

Starting a transmission stage: at this time, the readable segment data is empty, and the sending end sends out multiple file fragments to the service end at the same time according to the parallelism N. The control logic will send to the X% MAX location tag in the cache set based on the send sequence number X (0 start) for each file slice. At this time, the transmission segment start flag assigns a minimum value of the plurality of fragment numbers, and the end flag assigns a maximum value of the plurality of fragment numbers. The minimum value of the blank segment=maximum value of the transmission segment +1, and the initial condition, N > =max is not allowed to occur. Therefore, the blank end mark value remains MAX-1 unchanged.

Partial fragment reception completion phase: at this stage, some fragments have been received, and there are the following cases:

case 1: when read-only fragmentation is not present: if no unfinished file fragments exist between the currently received complete file fragments and the global start mark, setting the global start mark to the current fragments as readable segments, setting a readable segment start end mark, triggering a sending event to the back end, starting a sending module, setting a sending segment mark from the global start mark, sending the sending segment marks to the end of the readable segments one by one, and waiting. Meanwhile, the receiving segment start and end marks are recharged, the maximum and minimum values of the data segment serial numbers actually being received at the moment are taken as the receiving segment start and end marks, and if the real-time receiving data does not exist, the receiving segment setting marks are null. And if the incomplete file fragments exist between the received fragments and the global start mark, waiting for the receiving of the corresponding file fragments to be completed.

Case 2: when read-only fragmentation exists: if no unfinished file fragment exists between the current received complete file fragment and the read-only file fragment ending mark, setting the readable fragment ending mark as the current fragment sequence number, triggering the event sent to the back end, judging whether the event is in an idle state by the sending module, and if so, starting writing data to the back end. And if the unfinished file fragments exist between the current received finished file fragments and the read-only file fragment ending mark, waiting for the receiving of the corresponding file fragments to be finished.

Newly added receive fragmentation phase: in this stage, in the receiving process, the client initiates a new file fragment uploading request again, at this time, it should first determine whether there is an available blank space, that is, whether the uploading fragment sequence number is between the blank space start end marks or is at the transmitting end, but the current state is idle. And if the data is not matched with the data, indicating that the buffer is full, and returning a buffer full state to the client.

And if the new fragment sequence number is in the blank section, resetting the end mark of the transmitting end and the start mark of the blank section. Setting the fragments as end sequence numbers of the transmitting end, wherein the sequence number +1 is larger than the end sequence number of the blank section, and setting a blank section start and end mark to be empty, otherwise, setting the blank section start mark to be a receiving fragment sequence number +1; if the new fragment sequence number is at the receiving end, the sequence number position is set as receiving.

And a back-end data transmission stage: this phase belongs to the parallel operation phase with the aforementioned reception phase. At this stage, the transmission procedure starts with the last transmitted fragment number +1 (if the initial value is null, it starts with number 0), searches for transmittable data in the readable section, and waits for wakeup if it does not exist. If there is transmittable data, the slice is set to the transmission slice flag. After the setting is completed, the fragment data is read, written into an OutputStream- > InputStream conversion unit and sent to a rear-end application. If the writing is completed, the sent completion sequence number +1 is marked with a null value at the sending end. Readable fragment reclamation logic is executed at this point. Continuing to detect whether there is a transmissible data fragment, and if so, repeating the transmitting logic.

Readable recovery stage: this stage occurs in the send data completion stage, after triggering reclamation, the system logic determines: if there is a blank segment, the blank segment end flag points to the transmitted completion flag. Then, whether the readable end mark is in the blank section is judged, if so, the readable end mark is given a null value, otherwise, the readable end mark is +1.

According to the file fragment transmission method provided by the embodiment of the application, the file data which arrives in the out-of-order fragments is converted into the simple streaming interface with the consistent sequence through the transmission buffer window technology and the streaming conversion technology of the fragment data, and is exposed to the back-end application, so that the requirement on the operation complexity of the back-end permanent storage equipment (such as random access capability) can be reduced in the process, and meanwhile, the streaming interface can enable the back-end application processing mode to get rid of a single file storage mode, thereby increasing the expansibility of the whole system to file processing.

Based on the description in the foregoing embodiment, the embodiment of the present application correspondingly further provides a file slice transmission device, referring to fig. 6, fig. 6 is a schematic structural diagram of a file slice transmission device provided by the embodiment of the present application, and in some implementations, the file slice transmission device 40 includes:

A receiving unit 601, configured to receive each file fragment corresponding to a target transmission file sent by a receiving server and fragment metadata corresponding to the file fragment; and the fragment metadata comprises fragment serial numbers of the file fragments.

A dump unit 602, configured to save the file fragment to a file cache window created in advance according to the fragment metadata; and the storage position of the file fragment in the file cache window is related to the fragment sequence number of the file fragment.

And the processing unit 603 is configured to convert the data form of each file fragment one by one according to the storage location of each file fragment stored in the file cache window, so as to send each file fragment to the target receiving device according to the sequence of fragments.

In one embodiment, the dump unit 602 is configured to:

In one embodiment, the apparatus further comprises a preprocessing unit, configured to:

In one embodiment, the processing unit 603 is configured to:

Based on the description in the foregoing embodiment, the embodiment of the present application correspondingly further provides a file slice transmission device, referring to fig. 7, fig. 7 is a schematic structural diagram of a file slice transmission device provided in the embodiment of the present application, and in some implementations, the file slice transmission device 70 includes:

a receiving unit 701, configured to receive each file fragment corresponding to a target transmission file sent by a client;

a sending unit 702, configured to send the file fragments and fragment metadata corresponding to the file fragments to a pre-created cache window conversion service module; the cache window conversion service module stores the file fragments to a file cache window which is created in advance according to the fragment metadata; the file fragments are stored in the file cache window, wherein the fragment metadata comprises fragment serial numbers of the file fragments, and storage positions of the file fragments in the file cache window are related to the fragment serial numbers of the file fragments.

In one embodiment, the receiving unit 701 is configured to:

unlocking the current file creation operation authority.

It should be noted that, the device provided in this embodiment can implement the content implemented in the method embodiment and achieve the same technical effects, and specific details of the same parts and beneficial effects as those of the method embodiment in this embodiment are not described here.

Correspondingly, the embodiment of the application also provides a device, referring to fig. 8, fig. 8 is a schematic structural diagram of the device provided by the application, as shown in fig. 8, where the device includes: a memory 801 and a processor 802;

Memory 801 stores computer-executable instructions;

the processor 802 executes computer-executable instructions stored in the memory 801 to implement the file-slicing transmission method described above, wherein the memory 801 and the processor 802 are connected through the bus 803.

It should be noted that, the device provided in this embodiment can implement all the method steps implemented by the buffer window conversion service module or the receiving server in the method embodiment, and can achieve the same technical effects, and detailed descriptions of the same parts and beneficial effects as those in the method embodiment in this embodiment are omitted.

The embodiment of the application correspondingly provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and the computer execution instructions are used for realizing the file fragment transmission method when being executed by a processor.

The embodiment of the application correspondingly provides a computer program product, and the computer program product comprises computer program code which can realize the file slicing transmission method when running on a computer.

The embodiment of the application correspondingly provides a file fragment transmission system, which comprises the following steps: the contents implemented by each part in the system can be specifically referred to the contents shown in fig. 5, and will not be described herein.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, functional modules/units in the apparatus, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media).

The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.

Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.

Claims

1. A method for file fragment transmission, the method comprising:

2. The method of claim 1, wherein the saving the file fragments to a pre-created file cache window according to the fragment metadata comprises:

3. The method according to claim 1, wherein the method further comprises:

4. A method according to claim 3, wherein said converting the data form of each of said file fragments one by one according to the storage location of each of said file fragments stored in said file cache window comprises:

5. A method for file fragment transmission, the method comprising:

6. The method of claim 5, wherein receiving each file fragment corresponding to the target transmission file sent by the client includes:

7. The method of claim 5, wherein the method further comprises:

unlocking the current file creation operation authority.

8. A file fragment transmission device, the device comprising:

9. A file fragment transmission device, the device comprising:

10. The file fragment transmission system is characterized by comprising a client, a receiving server and a cache window conversion service module;

the buffer window conversion service module is used for executing the file fragment transmission method according to any one of claims 1-4;

the receiving server is configured to perform the file fragment transmission method according to any one of claims 5 to 7.

11. An apparatus, comprising: a memory and a processor;

the memory stores computer-executable instructions;

the processor executing computer-executable instructions stored in the memory causes the processor to perform the file-slicing transmission method of any one of claims 1-4, or the file-slicing transmission method of any one of claims 5-7.

12. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are for implementing a file-slicing transmission method according to any of claims 1-4 or a file-slicing transmission method according to any of claims 5-7.

13. A computer program product, characterized in that the computer program product comprises computer program code which, when run on a computer, causes the computer to perform the file-slicing transmission method according to any of claims 1-4 or the file-slicing transmission method according to any of claims 5-7.