CN116389515A - File fragment transmission method, device, server, medium and program product - Google Patents
File fragment transmission method, device, server, medium and program product Download PDFInfo
- Publication number
- CN116389515A CN116389515A CN202310582978.2A CN202310582978A CN116389515A CN 116389515 A CN116389515 A CN 116389515A CN 202310582978 A CN202310582978 A CN 202310582978A CN 116389515 A CN116389515 A CN 116389515A
- Authority
- CN
- China
- Prior art keywords
- file
- storage
- target
- information
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000012634 fragment Substances 0.000 title claims abstract description 199
- 230000005540 biological transmission Effects 0.000 title claims abstract description 95
- 238000000034 method Methods 0.000 title claims abstract description 89
- 238000012546 transfer Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012795 verification Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 abstract description 20
- 230000004044 response Effects 0.000 abstract description 8
- 238000005516 engineering process Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Abstract
The application provides a file fragment transmission method, a device, a server, a medium and a program product, and relates to the technical field of big data, wherein the method comprises the following steps: responding to a file fragment transmission request of a target file, and establishing a secure file transmission protocol SFTP session channel between the target storage server; checking whether the information abstract value of the file fragment is consistent with a preset information abstract value or not after receiving the file fragment corresponding to one target file, and transmitting the file fragment to a target storage server based on an SFTP session channel when the information abstract value of the file fragment is consistent with the preset information abstract value; and in response to the completion of the transmission of all the file fragments of the target file, when the total value of the first information abstracts of all the file fragments is matched with the total value of the second information abstracts of the target file, combining all the file fragments in the target storage server to obtain the target file, and closing the SFTP session channel. In the file dumping process, the local disk IO overhead of the receiving server and the overall network overhead are effectively reduced.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a method, an apparatus, a server, a medium, and a program product for file fragment transmission.
Background
The hypertext transfer protocol (Hyper Text Transfer Protocol, HTTP) is a request-response protocol commonly used for interactions between a browser and a server. When using a browser, a user may typically download files from a server or upload files to a server using HTTP technology.
In the process of uploading files, after receiving the files, particularly for enterprise-level servers, after receiving large files locally at a server side, the server usually needs to dump to a remote special storage service, and in the dumping process, the original files need to be stored in a local disk firstly, then network communication in a machine room is utilized, and the files are integrally sent from the local disk to the remote special storage service for permanent storage. The whole dumping process generates larger disk IO cost, extra time cost, network cost and garbage file management cost of a receiving end.
Disclosure of Invention
In order to solve the problems, namely the problems of high time overhead, high garbage management cost and the like in the process of dumping after file transmission, the application provides a file fragment transmission method, a device, a server, a medium and a program product.
According to a first aspect of the present application, there is provided a file fragmented outgoing method, comprising:
responding to a file fragment transmission request of a target file, and establishing a secure file transmission protocol SFTP session channel between the target storage server;
checking whether the information abstract value of the file fragment is consistent with a preset information abstract value or not after receiving the file fragment corresponding to one target file;
transmitting the file fragments to the target storage server based on the SFTP session channel when the information abstract value of the file fragments is consistent with a preset information abstract value;
and responding to the completion of the file slicing transmission of the target file, when the total value of the first information abstracts of all the file slices is matched with the total value of the second information abstracts of the target file, combining all the file slices in the target storage server to obtain the target file, and closing the SFTP session channel.
In one embodiment, before the file fragments are transferred to the target storage server based on the SFTP session channel, the method further comprises:
judging whether a storage file corresponding to the target file is created in the target storage server or not;
The transmitting the file fragments to a target storage server based on the SFTP session channel comprises the following steps:
when a storage file corresponding to the target file is created in the target storage server, acquiring a storage file name and a storage path corresponding to the storage file;
and based on the SFTP session channel, writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths.
In one embodiment, after determining whether the storage file corresponding to the target file has been created in the target storage server, the method further includes:
when a storage file corresponding to the target file is not created in the target storage server, reading file information of the target file, wherein the file information comprises basic file information and/or file data information, and determining a storage space required by the target file based on the file information;
and sending a first indication message to the target storage server, wherein the first indication message carries storage space information required by the target file, so that the target storage server creates a storage file based on the first indication message, and determines a storage file name and a storage path corresponding to the storage file.
In one embodiment, before the file fragments are transferred to the target storage server based on the SFTP session channel, the method further comprises:
and locking the current file creation permission based on the global lock control, and synchronizing corresponding permission locking information with the target storage server, so that the target storage server creates a storage file based on the indication message when creating the storage file.
In one embodiment, the method further comprises:
and unlocking the current file creation authority and locking the fragment writing authority of the storage file after the target storage server finishes creating the storage file, so that when the file fragments are written into the storage file corresponding to the target storage server, the file fragments are written into the storage file once.
In one embodiment, the storage file is a file created by the target storage server and having a preset file size;
the writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths based on the SFTP session channels comprises the following steps:
Based on the SFTP session channel, writing the file fragments into absolute offset of a storage file corresponding to the target storage server according to the storage file names and the storage paths, wherein the absolute offset is determined according to the actual offset of the storage file and the preset file size.
In one embodiment, the file-slicing transmission request is an HTTP PUT request, and the method further comprises:
when the information abstract value of the file fragments is inconsistent with the preset information abstract value, second indicating information is sent to a request end which initiates the file fragment transmission request, the second indicating information is used for indicating the request end to resend the file fragment transmission request of the corresponding file fragments, and the resend fragment transmission request carries fragment update files corresponding to the file fragments.
According to a second aspect of the present application, there is provided a file fragment transmission apparatus, including:
the channel establishment module is used for responding to a file fragment transmission request of the target file and establishing an SFTP session channel with the target storage server;
the verification module is used for verifying whether the information abstract value of the file fragment is consistent with a preset information abstract value or not when receiving the file fragment corresponding to one target file;
A transmission module configured to transmit the file fragments to a target storage server based on the SFTP session channel when the information summary value of the file fragments is consistent with a preset information summary value;
and the closing module is used for responding to the completion of the transmission of all the file fragments of the target file, combining all the file fragments in the target storage server to obtain the target file when the total value of the first information abstract of all the file fragments is matched with the total value of the second information abstract of the target file, and closing the SFTP session channel.
In one embodiment, the apparatus further comprises:
a judging module configured to judge whether a storage file corresponding to the target file has been created in the target storage server;
the transmission module includes:
an obtaining unit, configured to obtain a storage file name and a storage path corresponding to a storage file when the storage file corresponding to the target file has been created in the target storage server;
and the writing unit is used for writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths based on the SFTP session channels.
In one embodiment, the apparatus further comprises:
the reading module is used for reading file information of the target file when the storage file corresponding to the target file is not created in the target storage server, wherein the file information comprises basic file information and/or file data information, and determining the storage space required by the target file based on the file information;
the first indication module is configured to send a first indication message to the target storage server, where the first indication message carries storage space information required by the target file, so that the target storage server creates a storage file based on the first indication message, and determines a storage file name and a storage path corresponding to the storage file.
In one embodiment, the storage file is a file created by the target storage server and having a preset file size;
the writing unit is specifically configured to write the file fragments into the absolute offset of the storage file corresponding to the target storage server according to the storage file name and the storage path based on the SFTP session channel, where the absolute offset is determined according to the actual offset of the storage file and the preset file size.
According to a third aspect of the present application, there is provided a server comprising: a memory and a processor;
the memory stores computer-executable instructions;
and the processor executes the computer-executed instructions stored in the memory, so that the server executes the file fragment transmission method.
According to a fourth aspect of the present application, there is provided a computer readable storage medium having stored therein computer executable instructions for implementing the file fragment transmission method when executed by a processor.
According to a fifth aspect of the present application, there is provided a computer program product comprising computer program code which, when run on a computer, causes the computer to perform the file fragment transmission method.
It can be understood that, by responding to the file fragment transmission request of the target file, the file fragment transmission method, the device, the server, the medium and the program product provided by the embodiment of the application establish a secure file transmission protocol SFTP session channel with the target storage server; checking whether the information abstract value of the file fragment is consistent with a preset information abstract value or not after receiving the file fragment corresponding to one target file; transmitting the file fragments to the target storage server based on the SFTP session channel when the information abstract value of the file fragments is consistent with a preset information abstract value; and responding to the completion of the file slicing transmission of the target file, when the total value of the first information abstracts of all the file slices is matched with the total value of the second information abstracts of the target file, combining all the file slices in the target storage server to obtain the target file, and closing the SFTP session channel. In the file slicing transmission process, an SFTP session channel is established between the receiving server and the target storage server, and in the large file slicing uploading process, local caching is not needed, and the file slicing is transmitted to the rear-end file storage server in real time, so that the local disk IO (input/output) cost, extra time cost and network cost of the receiving server are reduced, and the garbage file management cost of the receiving end is reduced.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a flow chart of a file fragment transmission method provided in an embodiment of the present application;
fig. 2 is a schematic flow chart of another file fragment transmission method according to an embodiment of the present application;
FIG. 3 is a second flow chart of another method for file slice transmission according to the embodiment of the present application;
fig. 4 is a flow chart of another file slice transmission method according to an embodiment of the present application;
fig. 5 is a flowchart of a file fragment transmission method according to an exemplary embodiment of the present application;
fig. 6 is a schematic structural diagram of a file slice transmission device according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.
Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.
Detailed Description
In one possible application scenario, the system includes a terminal device and a server electrically connected to the terminal device, where in this embodiment, a browser in the terminal device is taken as an example of a request end or a client, and the terminal device performs data interaction with the server through a network. The user can upload large files to the server on the browser through hypertext protocol (Hyper Text TransferProtocol, HTTP) technology. The large file may specifically refer to a file occupying a large storage space, such as a video file, an application software installation file, and the like, which are common in life.
The client may include, but is not limited to, a computer, a smart phone, a tablet computer, an e-book reader, a dynamic video expert compression standard audio layer 3 (Moving Picture experts group audio layer III, MP3 for short) player, a dynamic video expert compression standard audio layer 4 (Moving Picture experts group audio layer IV, MP4 for short) player, a portable computer, a car computer, a wearable device, a desktop computer, a set-top box, a smart television, and the like.
The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), basic cloud computing services such as big data and artificial intelligent platforms, and the like.
In practical life application, when a user uploads a large file to a server from a browser by using an HTTP technology, the uploading speed of the file is often slow, the size of the file is limited by a gateway, the uploading cannot be continued after the uploading process is interrupted, and a series of problems such as incomplete file caused by the loss of data packets in HTTP transmission are solved. In order to solve the problem that large files cannot be uploaded at one time in the prior art, one method adopted is to divide the files into a plurality of small files with the same size and verify the small files to obtain the large files. However, this creates a new problem, and after the receiving end (server) obtains the small file locally and obtains the large file, there is generally a need to dump the large file to the remote storage server, in the dumping process, the original file is first landed on the local disk, and then the internal network communication of the machine room is utilized, so that the file is integrally sent from the local disk to the remote dedicated storage service for permanent storage, resulting in the problems of larger disk IO cost, additional time overhead, network overhead, redundancy of a large amount of garbage files, and the like at the receiving end.
In view of this, the file fragment transmission method, apparatus, server, medium and program product provided in this embodiment, in combination with a secure file transfer protocol (SSH File Transfer Protocol, abbreviated as SFTP) technique, establish a secure file transfer protocol SFTP session channel with a target storage server by responding to a file fragment transmission request of the target file; checking whether the information abstract value of the file fragment is consistent with a preset information abstract value or not after receiving the file fragment corresponding to one target file; transmitting the file fragments to a target storage server based on the SFTP session channel when the information abstract value of the file fragments is consistent with the preset information abstract value; and in response to the completion of the transmission of all the file fragments of the target file, when the total value of the first information abstracts of all the file fragments is matched with the total value of the second information abstracts of the target file, combining all the file fragments in the target storage server to obtain the target file, and closing the SFTP session channel. In the process, in the process of file fragment transmission, an SFTP session channel is established between the receiving server and the target storage server, and the process of transmitting the fragments to the rear-end file storage server in real time after the fragments are uploaded to the local in the large file fragment uploading process is realized, so that the local disk IO (input/output) cost of the receiving server is reduced, and the overall network cost of the system is reduced.
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals refer to the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of the present application. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present application and are not to be construed as limiting the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
It should be noted that the file fragment transmission method of the present application may be used in the field of financial science and technology. The method can also be used in any field except the field of financial science and technology, and the application field of the file fragment transmission method is not limited.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
The application scenario of the present application is briefly described above, and the file fragment transmission method provided in the embodiment of the present application is described in detail below by taking application to a server, specifically, by taking application to a receiving server as an example.
Referring to fig. 1, fig. 1 is a flowchart of a file fragment transmission method according to an embodiment of the present application, including steps S101 to S104.
Step S101, a secure file transfer protocol SFTP session channel is established between the target storage server and the target storage server in response to a file fragment transfer request of the target file.
In this embodiment, the target file is an overall file that needs to be transmitted by the client, that is, a large file, and at least includes two or more file fragments. In one implementation, the file-slicing transmission request may adopt a request mode other than HTTP PUT, so that generation of temporary files can be effectively reduced during slicing transmission.
In this embodiment, the receiving server may use an open-source JSCH component to create an SFTP operation API interface and a corresponding SFTP protocol client, and establish an SFTP channel between the receiving server and the target storage server, and use the SFTP channel to perform SFTP instruction interaction with the target storage server. It is to be appreciated that the JSCH component is an SFTP protocol client component, and in some embodiments other SFTP protocol client components may be used in addition to the JSCH component described above.
It will be appreciated that in response to a condition or state being relied upon to represent an operation being performed, one or more operations being performed may be in real time or with a set delay when the dependent condition or state is satisfied; the plurality of operations performed are not limited in order of execution unless specifically stated.
In an example, before receiving a file fragment transmission request, a receiving server first achieves a consensus of fragment transmission with a client, specifically, the client initiates a file transmission request to the server by adopting a request mode of HTTP POST, and the receiving server acquires and records basic information of a target file, where the basic information includes a full-text MD5 value of the target file, a unique identification code TaskID, and the like, and sends indication information to the client, and the client further generates fragments based on the indication information and initiates a fragment transmission request to the receiving server.
Step S102, checking whether the information abstract value of the file fragment is consistent with a preset information abstract value or not after receiving the file fragment corresponding to one target file.
Specifically, each time the server receives a corresponding file fragment of a target file, the server reads the information abstract value of the file fragment, and checks the file fragment, if the check fails, the server can instruct the client to retransmit the corresponding file fragment.
Optionally, when the terminal device generates the file fragments, the initial information abstract value (i.e. the preset information abstract value) of each file fragment is calculated correspondingly, and when the server side receives the file fragments, the server side checks the received file fragments by using the initial information abstract value, and determines the integrity of the fragment transmission by checking whether the abstract values between the two are the same.
Optionally, the client may transmit in a cyclic manner during transmitting the file fragments, that is, by identifying whether there is a file fragment that is not transmitted in all the generated file fragments, and if so, transmitting a file fragment transmission request corresponding to the file fragment to the server until all the file fragments are transmitted.
And step 103, transmitting the file fragments to the target storage server based on the SFTP session channel when the information abstract value of the file fragments is consistent with the preset information abstract value.
In this embodiment, after the verification of the fragments is completed, instead of combining the fragments of each file to obtain an overall file in the prior art, the overall file is dumped to the storage server, and in the process of file fragment transmission, the file fragments with the consistent verification are dumped to the target storage server directly. In the process, file fragments do not need to be stored or cached in the receiving server, so that the disk IO cost of the receiving server is effectively reduced, additional time expenditure and network expenditure are avoided, and the garbage file management cost is reduced.
Optionally, the client may transmit in a cyclic transmission manner during the process of transmitting the file fragments, that is, by identifying whether there is a file fragment that is not transmitted in all the generated file fragments, if so, the client transmits a file fragment transmission request corresponding to the file fragment to the receiving server, and directly dumps the target storage server until all the file fragments are transmitted.
And step S104, responding to the completion of the transmission of all file fragments of the target file, and when the total value of the first information abstracts of all file fragments is matched with the total value of the second information abstracts of the target file, combining all file fragments in the target storage server to obtain the target file, and closing the SFTP session channel.
In this embodiment, the second summary total value of the target file may be calculated before the file fragments are transmitted, so as to determine whether all the file fragments are transmitted completely by checking the MD5 total value of the file fragments and the MD5 total value of the target transmission file, and when all the file fragments are transmitted completely, the second summary total value of the target file is combined to obtain a large file, and the SFTP channel is closed, so that the file transmission and dumping process is completed.
Referring to fig. 2, fig. 2 is a schematic flow chart of another file slicing transmission method provided in the embodiment of the present application, on the basis of the above embodiment, according to this embodiment, a storage file is created in a target storage server, and file slices are written into corresponding storage files to implement file slicing writing of the file, specifically, before step S103 transfers file slices to the target storage server based on an SFTP session channel, the method may further include step S201, and step S103 is further divided into step S103a and step S103b.
Step S201, determining whether a storage file corresponding to the target file has been created in the target storage server, if the storage file corresponding to the target file has been created in the target storage server, executing step S103a, otherwise ending the flow, or creating the storage file, where the creation manner of the storage file is described in detail later, and will not be repeated here.
Step S103a, obtaining a storage file name and a storage path corresponding to a storage file;
step S103b, when the information abstract value of the file fragment is consistent with the preset information abstract value, writing the file fragment into a storage file corresponding to the target storage server according to the storage file name and the storage path based on the SFTP session channel.
It will be appreciated that the storage file is a file for writing the file fragment corresponding to the target file.
In one implementation, the channel. Stat method is called to determine whether the storage file exists, if yes, the Exception mechanism Exception is returned, and the prompt information is: and judging that the storage file does not exist if the No search file is the No search file, otherwise, judging that the storage file exists.
In one embodiment, as shown in fig. 3, after determining in step S201 whether the storage file corresponding to the target file has been created in the target storage server, the following steps 301 and 302 may be further included.
Step S301, when a storage file corresponding to the target file is not created in the target storage server, reading file information of the target file, wherein the file information comprises basic file information and/or file data information, and determining a storage space required by the target file based on the file information;
step S302, a first indication message is sent to the target storage server, wherein the first indication message carries storage space information required by the target file, so that the target storage server creates the storage file based on the first indication message, and a storage file name and a storage path corresponding to the storage file are determined.
Specifically, the present embodiment further determines the storage file name and path of the storage server by reading file information of the target file, including basic file information (including MD5 digest information), file data information, and the like, and then determining the storage space required for the target file according to the file information, and by creating a storage file of a corresponding size in the storage server.
In this embodiment, step S302 determines a storage file name and a storage path, and step S103b is performed to store the fragmented file into the corresponding storage file according to the storage file name and the storage path.
In one embodiment, by locking the file creation authority to avoid repeated creation of the file when the storage file is created, before the file fragments are transferred to the target storage server based on the SFTP session channel in step S103, it may further include:
and locking the current file creation permission based on the global lock control, and synchronizing corresponding permission locking information to the target storage server, so that the target storage server creates a storage file based on the indication message when creating the storage file.
In one implementation, the global lock control may be a redistributing-lock control of a redistributing lock, and in other implementations, other global lock spaces may be used, which is not particularly limited in this embodiment.
In this embodiment, when the storage server performs file creation, the global lock control is used to lock the file creation authority, and the storage server can only create one storage file, so that repeated creation of the storage file can be effectively prevented, and efficient writing of file fragments is facilitated.
Further, after the file is created, by locking the tile writing authority, so as to effectively avoid the repeated writing of the tiles, specifically, the method may further include the following steps:
And unlocking the current file creation authority and locking the fragment writing authority of the storage file after the storage file is created by the target storage server, so that when the file fragments are written into the storage file corresponding to the target storage server, the file fragments are written into the storage file once.
In this embodiment, after the storage file is created, the file creation permission is unlocked, and the slice writing permission of the storage file is locked at the same time, when the slice writing is performed, only one file slice can be written, so as to avoid the repeated writing of one file slice, wherein the locking of the slice writing permission can also be performed by adopting a redission-rlock control or other controls.
In one embodiment, considering that SFTP does not support random access of a file, only application append access is supported, and each slice is written in a specified offset of a stored file through a form of random access, the embodiment uses append mode to set a negative number offset form of absolute offset=offset-size, so as to change the random access form of the implemented file. Specifically, the storage file is a file with a preset file size created by the target storage server;
Step S103b, based on the SFTP session channel, writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths, specifically comprising the following steps of
Based on the SFTP session channel, the file fragments are written into the absolute offset of the storage file corresponding to the target storage server according to the storage file names and the storage paths, and the absolute offset is determined according to the actual offset of the storage file and the preset file size.
It will be appreciated that the offset offest in the append access may be negative, and the present embodiment uses append mode to set the form of a negative offset of absolute offset=offset-size to change phases to achieve random access to the file. Specifically, by creating a storage file of size greater than 1, and opening the file stream in append mode, the opening offset is set to size-1, and writing an arbitrary byte at this offset, determining the file size, and closing the file. When the slice writing is performed, the created storage file is opened in the addition mode at the position of the absolute offset (actual offset-size), and the slice of the file is additionally written.
In this embodiment, the created file is opened by adding, the offset of the opened file is the (actual offset-size) of the current slice, a negative value is formed with respect to the tail of the file, the uploaded slice data read from the file slice transmission request is written to the storage file in a cycle of 1024 bytes from this offset, and the storage file is written until the slice writing is completed. The process adopts the random access mode of the additional access mode phase change to realize the random access mode of the slice writing.
Referring to fig. 4, fig. 4 is a flow chart of another file slicing transmission method provided in the embodiment of the present application, where, based on the above embodiment, a file slicing transmission request is an HTTP PUT request, and a corresponding MD5 file slicing is updated by sending a slicing update file, so that generation of temporary files can be effectively reduced during slicing transmission, and specifically, the method of the embodiment may further include the following step S401 in addition to the steps S101 to S104.
In step S401, when the information abstract value of the file fragment is inconsistent with the preset information abstract value, second indication information is sent to a request end initiating the file fragment transmission request, where the second indication information is used to instruct the request end to resend the file fragment transmission request corresponding to the file fragment, and the resent fragment transmission request carries a fragment update file corresponding to the file fragment.
In this embodiment, each time a server receives a file fragment, the server reads the information abstract value of the file fragment, and verifies the file fragment, if the verification does not indicate the client to retransmit the corresponding file fragment, the update of the file fragment is implemented, and the HTTP PUT mode is adopted to effectively avoid accumulation of temporary files and relieve the system storage pressure.
To facilitate understanding of embodiments of the present application, the following description is provided in connection with an exemplary embodiment of the present application, as shown in fig. 5, including the following procedures:
step S501, the user selects a target file at the client.
In step S502, if the user does not select the target file, the client may obtain the target file selected by the user through the onchange event by using the JS script.
Step S503, the client reads the size, length, etc. of the target file.
In step S504, the client calculates the target file MD5, specifically, using spark-md5. Js.
It can be understood that steps S501-S504 described above are steps for the client to read and verify the target file.
Step S505, the client creates a unique character string TaskID for the target file.
Step S506, the client sends step 1: the upload/begin request (file transfer request, carrying basic information of the target transfer file) is sent to the receiving server in the form of an http post request.
Specifically, the client initiates an http post request, submits information such as a file name, a full-file MD5 value, a file length and the like, and a process unique identification code TaskID to the receiving server.
Step S507, the receiving server records the basic information of the target file and stores the basic information in the target database.
In step S508, the client performs the slicing operation on the target file, and performs initial setting on the slices, for example, the total number of slices is a, and the initial slice number is 0, so as to record the slicing transmission process conveniently.
Step S509, the client sets an execution condition: it is determined whether there is an unsent slice, if so, step S510 is performed, and if not, step S520 is performed.
Step S510, the client calculates a file fragment MD5, a file fragment Size, and the like.
Step S511, the client sends step 2: an upload/slice request (file slice transfer request) is sent to the receiving server in the form of an http put request.
Specifically, for the segmented fragments, the client may use an http put request to circularly initiate fragment data uploading to the receiving server.
Step S512, the receiving server performs MD5 verification on the file fragments.
Step S513, after the MD5 value of the file fragment is verified, the receiving server locks the current (storage) file creation (operation) authority through a global lock control (e.g. redission-lock).
Step S514, the storage server confirms the file creation condition.
Step S515, the receiving server judges whether the current file is created according to the file creation condition of the storage server, if not, the receiving server instructs the storage server to create the file, and if so, step S517 is executed.
Specifically, the file does not have a specified offset write method that can call SFTP, a specified size file is created, and the storage file size is determined according to the size of the incoming file.
Step S516, the storage server creates a file according to the instruction of the receiving server.
Step S517, unlocking the creation operation authority of the current file, and locking the writing (writing operation) authority of the current file fragment.
The method comprises the steps of locking file creation permission, namely, global file creation permission for locking the abstract value of a source file md 5; unlocking the file creation operation authority, namely unlocking the global file creation authority of the digest value of the source file MD 5.
Step S518, the storage server writes the file fragments in the storage file under the writing authority of the file fragments.
Step S519, unlock the write permission of the current file partition, so as to facilitate writing of the subsequent file partition.
It can be appreciated that the steps S505-S519 are the slice transfer and dump phases.
Step S520, the client sends a request step 3: an upload/finish request (file fragment transfer completion request);
step S521, the storage server calculates the MD5 total value of the file fragment, and returns the successful or failed transmission result of the target file to the client.
Specifically, a scheme for storing the digest calculation of the server-side file MD5 may receive a return value by remotely invoking an MD5sum command and parse the return value. And in the receiving server, the MD5sum command of the storage server can be remotely called through the Jsch component to acquire the MD5 digest total value of the file fragment.
Step S522, the flow is terminated.
It is understood that steps S520 to S522 are the end stage of the flow.
The embodiment of the application correspondingly further provides a file fragment transmission device, as shown in fig. 6, including:
a channel establishment module 61 configured to establish an SFTP session channel with the target storage server in response to a file-fragment transmission request of the target file;
a verification module 62, configured to verify whether the information abstract value of a file fragment corresponds to a preset information abstract value after receiving a file fragment corresponding to a target file;
a transmission module 63 configured to transmit the file fragments to the target storage server based on the SFTP session channel when the information summary value of the file fragments is consistent with the preset information summary value;
And a closing module 64, configured to, in response to the completion of the transmission of all the file fragments of the target file, combine the file fragments in the target storage server to obtain the target file when the first summary total value of all the file fragments matches the second summary total value of the target file, and close the SFTP session channel.
In one embodiment, the apparatus further comprises:
a judging module configured to judge whether a storage file corresponding to the target file has been created in the target storage server;
a transmission module 63 comprising:
an obtaining unit configured to obtain a storage file name and a storage path corresponding to a storage file when the storage file corresponding to the target file has been created in the target storage server;
and the writing unit is used for writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths based on the SFTP session channels.
In one embodiment, the apparatus further comprises:
the reading module is used for reading file information of the target file when the storage file corresponding to the target file is not created in the target storage server, wherein the file information comprises basic file information and/or file data information, and determining the storage space required by the target file based on the file information;
The first indication module is configured to send a first indication message to the target storage server, where the first indication message carries storage space information required by the target file, so that the target storage server creates the storage file based on the first indication message, and determines a storage file name and a storage path corresponding to the storage file.
In one embodiment, the apparatus further comprises:
the first permission locking module is used for locking the current file creation permission based on the global lock control, and synchronizing corresponding permission locking information to the target storage server, so that the target storage server creates a storage file based on the indication message when creating the storage file.
In one embodiment, the apparatus further comprises:
the second permission locking module is used for unlocking the current file creation permission and locking the fragment writing permission of the storage file in response to the completion of creating the storage file by the target storage server, so that when the file fragments are written into the storage file corresponding to the target storage server, the file fragments are written into the storage file once.
In one embodiment, the storage file is a file created by the target storage server having a preset file size;
The writing unit is specifically configured to write the file fragments into the absolute offset of the storage file corresponding to the target storage server according to the storage file name and the storage path based on the SFTP session channel, where the absolute offset is determined according to the actual offset of the storage file and the preset file size.
In one embodiment, the file-slicing transmission request is an HTTP PUT request, and the apparatus further comprises:
the second indication module is configured to send second indication information to a request end initiating a file-slice transmission request when the information abstract value of the file slice is inconsistent with the preset information abstract value, wherein the second indication information is used for indicating the request end to resend the file-slice transmission request corresponding to the file slice, and the resend slice transmission request carries a slice update file corresponding to the file slice.
It should be noted that, the above device provided in the present application can correspondingly implement all the method steps implemented by the server in the method embodiment, and can achieve the same technical effects, and detailed descriptions of the same parts and beneficial effects as those of the method embodiment in the present embodiment are omitted.
The embodiment of the application correspondingly further provides a server, as shown in fig. 7, including: a memory 71 and a processor 72;
the processor 72 executes computer-executable instructions stored in the memory 71 to cause the server to perform a file-slicing transmission method.
It should be noted that, the server provided in the present application can correspondingly implement all the method steps implemented in the method embodiment, and can achieve the same technical effects, and specific details of the same parts and beneficial effects as those of the method embodiment in the present embodiment are not repeated here.
The embodiment of the application correspondingly provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and the computer execution instructions are used for realizing the file fragment transmission method when being executed by a processor.
Embodiments of the present application accordingly also provide a computer program product comprising computer program code which, when run on a computer, causes the computer to perform a database updating method.
It should be noted that, the computer medium and the program product provided in the present application can correspondingly implement all the method steps implemented in the method embodiment, and can achieve the same technical effects, and detailed descriptions of the same parts and beneficial effects as those of the method embodiment in the present embodiment are omitted herein.
The embodiment of the application correspondingly provides a chip which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for calling and running the computer program from the memory and executing a file fragment transmission method.
It should be noted that, the chip provided in the present application can correspondingly implement all the method steps implemented in the method embodiment, and can achieve the same technical effects, and the same parts and beneficial effects as those of the method embodiment in the present embodiment are not described in detail herein.
Those of ordinary skill in the art will appreciate that all or some of the steps, systems, functional modules/units in the apparatus, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media).
The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.
Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
In the description of the embodiments of the present application, the term "and/or" merely represents an association relationship describing an association object, which means that three relationships may exist, for example, a and/or B may represent: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" means any combination of any one or at least two of the plurality, e.g., including at least one of A, B, may mean any one or more elements selected from the set consisting of A, B and C communication. Furthermore, the term "plurality" means two or more, unless specifically stated otherwise.
In the description of embodiments of the present application, the terms "first," "second," "third," "fourth," and the like (if any) are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims (14)
1. The file slice transmission method is characterized by comprising the following steps of:
responding to a file fragment transmission request of a target file, and establishing a secure file transmission protocol SFTP session channel between the target storage server;
checking whether the information abstract value of the file fragment is consistent with a preset information abstract value or not after receiving the file fragment corresponding to one target file;
transmitting the file fragments to the target storage server based on the SFTP session channel when the information abstract value of the file fragments is consistent with a preset information abstract value;
and responding to the completion of the file slicing transmission of the target file, when the total value of the first information abstracts of all the file slices is matched with the total value of the second information abstracts of the target file, combining all the file slices in the target storage server to obtain the target file, and closing the SFTP session channel.
2. The method of claim 1, further comprising, prior to transmitting the file fragments to a target storage server based on the SFTP session channel:
judging whether a storage file corresponding to the target file is created in the target storage server or not;
The transmitting the file fragments to a target storage server based on the SFTP session channel comprises the following steps:
when a storage file corresponding to the target file is created in the target storage server, acquiring a storage file name and a storage path corresponding to the storage file;
and based on the SFTP session channel, writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths.
3. The method according to claim 2, further comprising, after determining whether a storage file corresponding to the target file has been created in the target storage server:
when a storage file corresponding to the target file is not created in the target storage server, reading file information of the target file, wherein the file information comprises basic file information and/or file data information, and determining a storage space required by the target file based on the file information;
and sending a first indication message to the target storage server, wherein the first indication message carries storage space information required by the target file, so that the target storage server creates a storage file based on the first indication message, and determines a storage file name and a storage path corresponding to the storage file.
4. The method of claim 3, further comprising, prior to transmitting the file fragments to a target storage server based on the SFTP session channel:
and locking the current file creation permission based on the global lock control, and synchronizing corresponding permission locking information with the target storage server, so that the target storage server creates a storage file based on the indication message when creating the storage file.
5. The method as recited in claim 4, further comprising:
and unlocking the current file creation authority and locking the fragment writing authority of the storage file after the target storage server finishes creating the storage file, so that when the file fragments are written into the storage file corresponding to the target storage server, the file fragments are written into the storage file once.
6. The method of any of claims 2-5, wherein the storage file is a file created by the target storage server having a preset file size;
the writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths based on the SFTP session channels comprises the following steps:
Based on the SFTP session channel, writing the file fragments into absolute offset of a storage file corresponding to the target storage server according to the storage file names and the storage paths, wherein the absolute offset is determined according to the actual offset of the storage file and the preset file size.
7. The method of claim 1, wherein the file-slicing transmission request is an HTTP PUT request, the method further comprising:
when the information abstract value of the file fragments is inconsistent with the preset information abstract value, second indicating information is sent to a request end which initiates the file fragment transmission request, the second indicating information is used for indicating the request end to resend the file fragment transmission request of the corresponding file fragments, and the resend fragment transmission request carries fragment update files corresponding to the file fragments.
8. A document slice transfer apparatus, comprising:
the channel establishment module is used for responding to a file fragment transmission request of the target file and establishing an SFTP session channel with the target storage server;
the verification module is used for verifying whether the information abstract value of the file fragment is consistent with a preset information abstract value or not when receiving the file fragment corresponding to one target file;
A transmission module configured to transmit the file fragments to a target storage server based on the SFTP session channel when the information summary value of the file fragments is consistent with a preset information summary value;
and the closing module is used for responding to the completion of the transmission of all the file fragments of the target file, combining all the file fragments in the target storage server to obtain the target file when the total value of the first information abstract of all the file fragments is matched with the total value of the second information abstract of the target file, and closing the SFTP session channel.
9. The apparatus as recited in claim 8, further comprising:
a judging module configured to judge whether a storage file corresponding to the target file has been created in the target storage server;
the transmission module includes:
an obtaining unit, configured to obtain a storage file name and a storage path corresponding to a storage file when the storage file corresponding to the target file has been created in the target storage server;
and the writing unit is used for writing the file fragments into the storage files corresponding to the target storage server according to the storage file names and the storage paths based on the SFTP session channels.
10. The apparatus as recited in claim 9, further comprising:
the reading module is used for reading file information of the target file when the storage file corresponding to the target file is not created in the target storage server, wherein the file information comprises basic file information and/or file data information, and determining the storage space required by the target file based on the file information;
the first indication module is configured to send a first indication message to the target storage server, where the first indication message carries storage space information required by the target file, so that the target storage server creates a storage file based on the first indication message, and determines a storage file name and a storage path corresponding to the storage file.
11. The apparatus according to claim 9 or 10, wherein the storage file is a file created by the target storage server and having a preset file size;
the writing unit is specifically configured to write the file fragments into the absolute offset of the storage file corresponding to the target storage server according to the storage file name and the storage path based on the SFTP session channel, where the absolute offset is determined according to the actual offset of the storage file and the preset file size.
12. A server, comprising: a memory and a processor;
the memory stores computer-executable instructions;
the processor executing computer-executable instructions stored in the memory, causing the server to perform the file-slicing transmission method of any one of claims 1-7.
13. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to implement the file fragment transmission method of any one of claims 1-7.
14. A computer program product, characterized in that the computer program product comprises computer program code which, when run on a computer, causes the computer to perform the file-slicing transmission method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310582978.2A CN116389515A (en) | 2023-05-22 | 2023-05-22 | File fragment transmission method, device, server, medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310582978.2A CN116389515A (en) | 2023-05-22 | 2023-05-22 | File fragment transmission method, device, server, medium and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116389515A true CN116389515A (en) | 2023-07-04 |
Family
ID=86965985
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310582978.2A Pending CN116389515A (en) | 2023-05-22 | 2023-05-22 | File fragment transmission method, device, server, medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116389515A (en) |
-
2023
- 2023-05-22 CN CN202310582978.2A patent/CN116389515A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10237333B2 (en) | Network transfer of large files in unstable network environments | |
| CN106250270B (en) | A kind of data back up method under cloud computing platform | |
| CN110716895B (en) | Target data archiving method, device, computer equipment and medium | |
| CN108234539B (en) | File uploading, downloading and transmitting method and device | |
| US20090262724A1 (en) | Proxy server, communication system, communication method and program | |
| CN108512821B (en) | Data transmission method, device and system, network gate and transaction data storage method | |
| CN109391673A (en) | A kind of method, system and the terminal device of management update file | |
| US9560115B2 (en) | Reusable secure file transfer for multiple systems | |
| US9769203B2 (en) | Methods, systems, and apparatus for mitigating network-based attacks | |
| CN113542402B (en) | File transmission method, device, system, electronic equipment and storage medium | |
| CN112383612B (en) | File transmission method, device, equipment and readable storage medium | |
| CN113273163A (en) | File uploading method, file downloading method and file management device | |
| WO2017039785A2 (en) | Apparatus and method for secure file transfer | |
| CN112948340A (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| CN102687472A (en) | Processing devices and methods for transmitting and receiving data | |
| US8924439B2 (en) | System for and method of partial file hashing | |
| CN112131041A (en) | Method, apparatus and computer program product for managing data placement | |
| CN116389515A (en) | File fragment transmission method, device, server, medium and program product | |
| CN109660496A (en) | Acquisition, resource checksum and the dissemination method of check information, device and electronic equipment | |
| US20190236331A1 (en) | Method, apparatus, and storage medium for data verification | |
| US20120066293A1 (en) | Data relay system, relay device, and computer readable medium | |
| CN112688905B (en) | Data transmission method, device, client, server and storage medium | |
| CN114157435A (en) | Document processing method and device based on block chain technology and client | |
| WO2021247402A1 (en) | Unidirectional gateway mediated multiplexing of concurrent data message streams | |
| CN113900990A (en) | File fragment storage method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |