CN116599953B - File uploading method, device, system, equipment and readable storage medium - Google Patents
File uploading method, device, system, equipment and readable storage medium Download PDFInfo
- Publication number
- CN116599953B CN116599953B CN202310861948.5A CN202310861948A CN116599953B CN 116599953 B CN116599953 B CN 116599953B CN 202310861948 A CN202310861948 A CN 202310861948A CN 116599953 B CN116599953 B CN 116599953B
- Authority
- CN
- China
- Prior art keywords
- file
- fragment
- storage server
- fragments
- file fragments
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 239000012634 fragment Substances 0.000 claims abstract description 350
- 230000005540 biological transmission Effects 0.000 claims abstract description 88
- 230000004044 response Effects 0.000 claims description 59
- 238000004458 analytical method Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012795 verification Methods 0.000 claims description 9
- 230000002194 synthesizing effect Effects 0.000 claims description 3
- 238000013467 fragmentation Methods 0.000 claims 5
- 238000006062 fragmentation reaction Methods 0.000 claims 5
- 230000000977 initiatory effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 12
- 230000007246 mechanism Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000012790 confirmation Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/123—Applying verification of the received information received data contents, e.g. message integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a device, a system, equipment and a readable storage medium for uploading files, wherein the method comprises the following steps: receiving file fragments and fragment data information sent by a client; the user datagram protocol is used for respectively sending each file fragment and fragment data information to a storage server side; and establishing connection with a storage server based on a transmission control protocol and carrying out integrity check on the file fragments. By using the scheme of the invention, the file integrity can be ensured while the file uploading transmission efficiency is improved, the resource consumption can be reduced, and the user experience is improved.
Description
Technical Field
The present invention relates to the field of computers, and more particularly, to a method, apparatus, system, device, and readable storage medium for file upload.
Background
The TCP (Transmission Control Protocol ) protocol is a byte stream based protocol that requires a connection to be established prior to communication, which connection needs to be completed by a "three-way handshake" mechanism. In addition, the TCP protocol also provides a confirmation mechanism, a retransmission mechanism and an integrity check mechanism to ensure the high efficiency and accuracy of message transmission. Although the TCP protocol guarantees the reliability of messages in the transmission process through a series of mechanisms, it sacrifices the data transmission rate and increases the resource consumption of the system, thus imposing a certain burden on the system.
The UDP (User Datagram Protocol ) protocol is an unreliable data transmission protocol based on datagrams, and no connection needs to be established before communication is performed, so that a higher transmission rate can be obtained without consuming excessive system resources when the UDP protocol performs data transmission, which has the disadvantages that the data can not be successfully received by a receiver, packet loss is easy to occur, and the reliability of data transmission cannot be guaranteed.
In the aspect of a transmission protocol of a private cloud platform related file uploading scene, a TCP protocol is adopted as a data transmission protocol, but because the TCP protocol needs to establish connection in advance and ensure transmission reliability, a large file such as a transmission mirror image file has low transmission speed rate and consumes large system resources.
Disclosure of Invention
Accordingly, an object of the embodiments of the present invention is to provide a method, apparatus, system, device and readable storage medium for uploading files, which can improve the transmission efficiency of file uploading, and simultaneously ensure the integrity of the files, reduce the consumption of resources, and improve the user experience.
Based on the above objects, an aspect of the embodiments of the present invention provides a method for uploading a file, including executing the following steps at a server:
Receiving file fragments and fragment data information sent by a client;
the user datagram protocol is used for respectively sending each file fragment and fragment data information to a storage server side;
and establishing connection with a storage server based on a transmission control protocol and carrying out integrity check on the file fragments.
According to one embodiment of the present invention, further comprising:
and in response to the existence of the lost file fragment, transmitting the lost file fragment to a storage server side by using a second transmission protocol.
According to one embodiment of the present invention, the shard data information includes a number of file shards and a file shard ID.
According to one embodiment of the present invention, the step of transmitting each file fragment and fragment data information to the storage server side using the first transmission protocol includes:
the server side uses a user datagram protocol to send the current file fragments to the storage server side;
and responding to the response that the server receives the returned response sent by the storage server within the threshold time, and sending the next file fragment to the storage server by the server.
According to one embodiment of the present invention, further comprising:
and in response to the threshold time, the server side sends the next file fragment to the storage server side.
According to one embodiment of the present invention, further comprising:
the server side judges whether the file fragments are all sent to the storage server side or not, and starts integrity check based on the judging result.
According to one embodiment of the present invention, the step of performing integrity check on the file fragments comprises:
and judging whether the file fragments are lost or not according to the file fragment ID (Identity document, unique code) and the number of the file fragments.
According to one embodiment of the present invention, the step of determining whether there is a file fragment loss according to the file fragment ID and the number of file fragments includes:
inquiring a file fragment list on a storage server side by adopting a transmission control protocol;
judging whether the number of the file fragments in the file fragment list is the same as the number contained in the fragment data information;
and determining that the file fragments are lost and acquiring the IDs of the lost file fragments in response to the fact that the number of the file fragments in the file fragment list is different from the number contained in the fragment data information.
According to one embodiment of the present invention, the step of transmitting the lost file fragment to the storage server side using the second transport protocol in response to the existence of the lost file fragment comprises:
the server side sends the ID of the lost file fragment to the client side and acquires the lost file fragment from the client side to be sent to the storage server side through a second transmission protocol.
According to one embodiment of the present invention, further comprising:
and responding to the file fragments received by the storage server and retransmitted by the server, synthesizing each file fragment, storing the synthesized file, and transmitting response information to the server.
According to one embodiment of the present invention, further comprising:
and in response to the absence of the lost file fragments, the storage server synthesizes each file fragment, stores the synthesized file, and sends response information to the server.
According to one embodiment of the present invention, the step of receiving the file fragment and fragment data information sent by the client includes:
the client divides the file into a plurality of file fragments based on the size and the requirement of the file to be uploaded, and sends the ID of each file fragment and the number of the file fragments to the server.
According to one embodiment of the present invention, the step of transmitting each file fragment and fragment data information to the storage server side using the first transmission protocol includes:
and analyzing the received current file fragments by the storage server side, and starting retransmission of the file fragments based on the file fragment analysis result.
According to one embodiment of the present invention, the step of the storage server end analyzing the received current file fragment includes:
The method comprises the steps that a storage server side is responded to successfully analyze received file fragments, and a positive acknowledgement message is sent to the server side;
and responding to failure of the storage server side to analyze the received file fragments, and sending a negative acknowledgement message to the server side.
According to one embodiment of the present invention, the step of transmitting each file fragment and fragment data information to the storage server side using the first transmission protocol includes:
and starting retransmission of the file fragments based on the file fragment analysis result or the waiting time.
According to one embodiment of the present invention, the step of starting retransmission of the file fragments based on the file fragment parsing result or the waiting time includes:
and in response to the failure of the storage server side to analyze the received file fragments, sending a retransmission request and a negative acknowledgement message to the server side, wherein the retransmission request comprises the information of the file fragments with the failure analysis.
According to one embodiment of the present invention, the step of starting retransmission of the file fragments based on the file fragment parsing result or the waiting time includes:
maintaining a retransmission timer and a retransmission counter at a server;
responding to the data sent by the server to the storage server, and starting a retransmission timer;
Reading the value of a retransmission counter in response to the retransmission timer ending without receiving a positive acknowledgement message sent by the storage server side;
and in response to the value of the retransmission counter being smaller than the set threshold, retransmitting the data to the storage server side, restarting the retransmission timer, and adding 1 to the value of the retransmission counter.
According to one embodiment of the present invention, the step of starting retransmission of the file fragments based on the file fragment parsing result or the waiting time further comprises:
and in response to the value of the retransmission counter being greater than or equal to the set threshold, no more data is retransmitted to the storage server side and the retransmission counter is cleared.
In another aspect of the embodiment of the present invention, there is also provided an apparatus for uploading a file, including:
the receiving module is configured to receive file fragments and fragment data information sent by the client;
the transmission module is configured to respectively send each file fragment and fragment data information to the storage server side by using a first transmission protocol;
and the verification module is configured to establish connection with the storage server based on the second transmission protocol and carry out integrity verification on the file fragments.
According to one embodiment of the present invention, the server side further comprises a retransmission module configured to request the client side to send the lost file fragments in response to the existence of the lost file fragments, and send the lost file fragments to the storage server side using a second transmission protocol.
In another aspect of the embodiment of the present invention, there is also provided a system for uploading a file, including:
a server;
the storage server is connected with the server through a first transmission protocol and/or a first transmission protocol,
wherein, the server side is used for: receiving file fragments and fragment data information sent by a client; respectively transmitting each file fragment and fragment data information to a storage server side by using a first transmission protocol; and establishing connection with a storage server based on a second transmission protocol and carrying out integrity check on the file fragments.
In another aspect of the embodiments of the present invention, there is also provided a computer apparatus including:
at least one processor; and
and a memory storing computer instructions executable on the processor, the instructions when executed by the processor performing the steps of any of the methods described above.
In another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of any of the methods described above.
The invention has the following beneficial technical effects: according to the file uploading method provided by the embodiment of the invention, the file fragments and the fragment data information sent by the client are received; respectively transmitting each file fragment and fragment data information to a storage server side by using a first transmission protocol; based on the technical scheme that the second transmission protocol is connected with the storage server end and the integrity verification is carried out on the file fragments, the file integrity can be ensured while the transmission efficiency of file uploading is improved, the consumption of resources can be reduced, and the user experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a method of file upload according to one embodiment of the present invention;
FIG. 2 is a schematic diagram of file transfer according to one embodiment of the present invention;
FIG. 3 is a schematic diagram of an integrity check in accordance with one embodiment of the present invention;
FIG. 4 is a diagram illustrating call relationships of ends according to one embodiment of the invention;
FIG. 5 is a diagram of a request retransmission mechanism according to one embodiment of the present invention;
fig. 6 is a schematic diagram of an active retransmission mechanism according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of an apparatus for file upload according to one embodiment of the present invention;
FIG. 8 is a schematic diagram of a system for file upload according to one embodiment of the present invention;
FIG. 9 is a schematic diagram of a computer device according to one embodiment of the invention;
Fig. 10 is a schematic diagram of a computer-readable storage medium according to one embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
With the above object in view, in a first aspect, an embodiment of a method for uploading a file is provided. Fig. 1 shows a schematic flow chart of the method.
As shown in fig. 1, the method may include performing the following steps at the server:
s1, receiving file fragments and fragment data information sent by a client. The method comprises the steps of slicing files required to be uploaded by a user at a client, slicing the complete files into a plurality of file slices according to the sizes and demands of the files, and then sending each file slice and slice data information to the client, wherein the slice data information comprises the number of the file slices and the ID of each file slice, and other data information can be uploaded at the same time according to the demands.
S2, respectively transmitting each file fragment and fragment data information to a storage server side by using a first transmission protocol. In the embodiment of the present invention, the first transmission protocol includes a UDP protocol, after receiving the file fragments and the fragment data information, the server transmits the file fragments and the fragment data information to the storage server for storage by using the UDP protocol, and because the UPD protocol has a risk of data loss, there may be a risk of losing a request during sending data, so in the embodiment of the present invention, a timeout period is set, if no response message is received within the timeout period, the transmission flow of the file fragments is temporarily not same as that of continuing sending other file fragments, as shown in fig. 2. The first time of file transmission is carried out by adopting the UDP protocol, the file fragments can be rapidly sent to the storage server, the file transmission efficiency is improved, and the resource consumption is reduced.
And S3, establishing connection with a storage server based on a second transmission protocol and carrying out integrity check on the file fragments. As shown in fig. 3, in the embodiment of the present invention, the second transmission protocol includes a TCP protocol, after sending the file fragments and the fragment data information, the server needs to determine whether to send all the file fragments, if all the file fragments have been sent, then enter a file integrity checking stage, the server establishes a connection with the storage server based on the TCP protocol, queries a file fragment list of the file on the storage server by using the TCP protocol, determines whether the file fragments are lost, if the file fragments are lost, the server needs to resend the lost file fragments to the storage server, and when resending, uses a TCP protocol to send reliably, so as to ensure that sending is successful, and finally, the storage server performs merging operation on the file fragments, merges the file fragments into a source file, stores the source file, and uploads returned response information to the client if merging is successful, and ends uploading the file. In the process of file slicing transmission, the calling relationship among the client, the server and the storage server is shown in fig. 4. The TCP protocol is adopted for retransmitting the file, so that the file can be safely and accurately transmitted to the storage server, the integrity of the file can be ensured, and the user experience is improved.
According to the invention, file uploading is performed in a mode of combining a TCP protocol and a UDP protocol, file transmission is realized by combining the high-efficiency characteristic of the UDP protocol with the reliability characteristic of the TCP protocol, the UDP protocol is adopted in a file transmission stage, the transmission efficiency is improved, the TCP protocol is adopted in a file integrity checking stage, the file integrity is ensured, and the purposes of improving the file transmission speed, reducing the resource consumption and improving the user experience are achieved.
By using the technical scheme of the invention, the file integrity can be ensured while the file uploading transmission efficiency is improved, the resource consumption can be reduced, and the user experience is improved.
In a preferred embodiment of the present invention, further comprising:
and in response to the existence of the lost file fragment, transmitting the lost file fragment to a storage server side by using a second transmission protocol. In the embodiment of the invention, the second transmission protocol comprises a TCP protocol, after the file fragments and the fragment data information are sent, the server needs to determine whether all the file fragments are sent, if so, the server enters a file integrity checking stage, establishes connection with the storage server based on the TCP protocol, queries a file fragment list of the file on the storage server by adopting the TCP protocol, judges whether the file fragments are lost, if so, the server needs to resend the lost file fragments to the storage server, and when resending, the file fragments are reliably sent by adopting a TCP protocol mode, thereby ensuring the success of sending.
In a preferred embodiment of the invention, the first transmission protocol comprises a user datagram protocol and the second transmission protocol comprises a transmission control protocol. The UDP (User Datagram Protocol ) protocol is a transport protocol that the Internet protocol suite supports without a connection, and provides a way for applications to send encapsulated IP packets without establishing a connection. The TCP (Transmission Control Protocol ) protocol is a connection-oriented, reliable, byte-stream based transport layer communication protocol intended to accommodate a hierarchical protocol hierarchy supporting multiple network applications, with reliable communication services being provided by means of TCP between pairs of processes in host computers connected to different but interconnected computer communication networks.
In a preferred embodiment of the present invention, the step of transmitting each file fragment and fragment data information to the storage server side using the first transmission protocol includes:
the server side uses a user datagram protocol to send the current file fragments to the storage server side;
and responding to the response that the server receives the returned response sent by the storage server within the threshold time, and sending the next file fragment to the storage server by the server. In the embodiment of the present invention, the first transmission protocol includes a UDP protocol, and since the UPD protocol has a risk of data loss, there may be a risk of request loss during data transmission, a threshold time is set in the embodiment of the present invention, if a return response message is received within the threshold time, the next file slice is continuously transmitted, for example, the threshold time is set to 2 minutes, after the file slice is transmitted, the return response is received at 1 minute, the next file slice is continuously transmitted, and so on.
In a preferred embodiment of the present invention, further comprising:
and in response to the threshold time, the server side sends the next file fragment to the storage server side. If the return response message is not received within the threshold time, the file fragments may be lost, or the return response is lost, and the next file fragment is directly sent without waiting, so that the transmitted file fragment is not blocked.
In a preferred embodiment of the present invention, further comprising:
the server side judges whether the file fragments are all sent to the storage server side or not, and starts integrity check based on the judging result. The server checks whether the file fragment transmitted to the storage server is the last file fragment transmitted from the front end, if so, the server determines that all the file fragments are transmitted to the storage server and starts the integrity check.
In a preferred embodiment of the present invention, the step of integrity checking the file fragments comprises:
and judging whether the file fragments are lost or not according to the file fragment ID and the number of the file fragments. When the file fragments are transmitted, the fragment data information is uploaded, the fragment data information comprises the file fragment ID and the total number of fragments, whether the file fragments are lost or not can be judged based on the file fragment ID and the total number of fragments, and the judgment can be carried out at a storage server side.
In a preferred embodiment of the present invention, the step of determining whether there is a file fragment loss according to the file fragment ID and the number of file fragments includes:
inquiring a file fragment list on a storage server side by adopting a transmission control protocol;
judging whether the number of the file fragments in the file fragment list is the same as the number contained in the fragment data information;
and determining that the file fragments are lost and acquiring the IDs of the lost file fragments in response to the fact that the number of the file fragments in the file fragment list is different from the number contained in the fragment data information. The server establishes connection with the storage server based on TCP protocol, adopts TCP protocol to inquire the file fragment list of the file on the storage server, compares the number of the file fragments in the file fragment list with the number of fragments contained in the fragment data information, if the number is the same, determines that no file fragments are lost, if the number is different, determines that the file fragments are lost, acquires the ID of the lost file fragments, and needs the server to resend the lost file fragments to the storage server, and adopts the TCP protocol to carry out reliable sending during resending to ensure successful sending.
In a preferred embodiment of the present invention, the step of transmitting the lost file fragment to the storage server side using the second transport protocol in response to the existence of the lost file fragment comprises:
the server side sends the ID of the lost file fragment to the client side and acquires the lost file fragment from the client side to be sent to the storage server side through a second transmission protocol. In one embodiment, after obtaining the ID of the lost file fragment, the server sends the ID of the file fragment to the client, the client resends the file fragment corresponding to the ID to the server, and the server reliably sends the file fragment in a TCP protocol manner, so as to ensure that the sending is successful. In yet another embodiment, the server temporarily stores the file fragments sent to the storage server, and after obtaining the IDs of the lost file fragments, the server uses a TCP protocol to reliably send the file fragments corresponding to the IDs, so as to ensure that the sending is successful.
In a preferred embodiment of the present invention, further comprising:
and responding to the file fragments received by the storage server and retransmitted by the server, synthesizing each file fragment, storing the synthesized file, and transmitting response information to the server. After resending the file fragments, the storage server side performs merging operation on the file fragments, merges the file fragments into a source file and stores the source file, and uploads response information to the client side after successful merging, and the uploading of the file is finished.
In a preferred embodiment of the present invention, further comprising:
and in response to the absence of the lost file fragments, the storage server synthesizes each file fragment, stores the synthesized file, and sends response information to the server. If the lost file fragments do not exist, the storage server side directly performs merging operation on the file fragments, merges the file fragments into a source file and stores the source file, uploads response information to the client side if the merging is successful, and the uploading of the file is finished
In a preferred embodiment of the present invention, the step of receiving the file fragment and fragment data information sent by the client includes:
the client divides the file into a plurality of file fragments based on the size and the requirement of the file to be uploaded, and sends the ID of each file fragment and the number of the file fragments to the server.
In a preferred embodiment of the present invention, the step of transmitting each file fragment and fragment data information to the storage server side using the first transmission protocol includes:
and analyzing the received current file fragments by the storage server side, and starting retransmission of the file fragments based on the file fragment analysis result. The retransmission mechanism and the confirmation mechanism are added in the UDP protocol used in the invention, so that the file transmission speed of the UDP protocol can be ensured, and the reliability of file transmission is improved.
In a preferred embodiment of the present invention, the step of parsing the received current file fragment at the storage server includes:
the method comprises the steps that a storage server side is responded to successfully analyze received file fragments, and a positive acknowledgement message is sent to the server side;
and responding to failure of the storage server side to analyze the received file fragments, and sending a negative acknowledgement message to the server side. In the process of data transmission, as the communication link of the network is often interfered by external factors, the data may be lost or damaged in the process of transmission, thereby reducing the communication quality. The acknowledgement messages of the UDP protocol are mainly classified into two types, namely a positive acknowledgement message and a negative acknowledgement message, and when the receiving party (storage server side) receives the messages, the message is subjected to analysis and verification, if the analysis is successful, the receiving party sends the positive acknowledgement message to the sending party (server side), and if the analysis is wrong, the receiving party sends the negative acknowledgement message to the sending party.
In a preferred embodiment of the present invention, the step of transmitting each file fragment and fragment data information to the storage server side using the first transmission protocol includes:
And starting retransmission of the file fragments based on the file fragment analysis result or the waiting time. The retransmission mechanism is mainly used for processing three cases, namely, 1, the receiver can not normally receive data finally due to network congestion or other network problems, namely, no response message is sent beyond a threshold time, 2, the data received by the receiver cannot be normally analyzed, which means that the data is lost or changed in the transmission process, 3, the data is accurately received by the receiver, but the positive acknowledgement message returned by the receiver is lost in the transmission process, namely, no response message is sent beyond the threshold time. According to the three conditions, the invention designs two retransmission mechanisms, one is a retransmission requesting mechanism, namely, a receiver sends a negative acknowledgement message and reports errors, a sender is requested to retransmit data, and the other is an active retransmission, namely, the sender does not receive a response message of the receiver within a certain time, so that the data is retransmitted.
In a preferred embodiment of the present invention, the step of starting retransmission of the file fragments based on the file fragment parsing result or the waiting time includes:
and in response to the failure of the storage server side to analyze the received file fragments, sending a retransmission request and a negative acknowledgement message to the server side, wherein the retransmission request comprises the information of the file fragments with the failure analysis. As shown in fig. 5, when the receiving side receives data, the receiving side analyzes the data, if the data analysis fails, a retransmission request and a negative acknowledgement message are sent to the sending side, the sending side retransmits the lost data, then analyzes again, if the analysis fails, a retransmission request and a negative acknowledgement message are sent to the sending side, and so on. If the data analysis is successful, adding the ID of the file fragment into the message queue maintained by the receiver, when the last data is received by the receiver, checking the ID of the file fragment in the message queue by the receiver, and if all the IDs of the file fragments are received, completing the transmission.
In a preferred embodiment of the present invention, the step of starting retransmission of the file fragments based on the file fragment parsing result or the waiting time includes:
maintaining a retransmission timer and a retransmission counter at a server;
responding to the data sent by the server to the storage server, and starting a retransmission timer;
reading the value of a retransmission counter in response to the retransmission timer ending without receiving a positive acknowledgement message sent by the storage server side;
and in response to the value of the retransmission counter being smaller than the set threshold, retransmitting the data to the storage server side, restarting the retransmission timer, and adding 1 to the value of the retransmission counter.
In a preferred embodiment of the present invention, the step of starting retransmission of the file fragments based on the file fragment parsing result or the waiting time further comprises:
and in response to the value of the retransmission counter being greater than or equal to the set threshold, no more data is retransmitted to the storage server side and the retransmission counter is cleared. As shown in fig. 6, a timeout retransmission timer and a retransmission counter are maintained at the server, after the server sends a file fragment, the timeout retransmission timer is started, if the sender does not receive a response message within a threshold time and the value of the retransmission counter is smaller than a specified value, the server will retransmit the file fragment, and restart the retransmission timer, and increase the value of the retransmission counter by 1. If the server fails to receive the response message within the threshold time, the steps are continuously repeated until the value of the retransmission counter reaches the preset value, or the server receives the response message.
The invention uses the mode of combining the TCP protocol and the UDP protocol to upload the file, because the private cloud is deployed in the intranet environment, the network environment is better, the UDP protocol can be adopted to improve the transmission efficiency on the premise of ensuring certain reliability, but the problem of packet loss still exists.
It should be noted that, it will be understood by those skilled in the art that all or part of the procedures in implementing the methods of the above embodiments may be implemented by a computer program to instruct related hardware, and the above program may be stored in a computer readable storage medium, and the program may include the procedures of the embodiments of the above methods when executed. Wherein the storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like. The computer program embodiments described above may achieve the same or similar effects as any of the method embodiments described above.
Furthermore, the method disclosed according to the embodiment of the present invention may also be implemented as a computer program executed by a CPU, which may be stored in a computer-readable storage medium. When executed by a CPU, performs the functions defined above in the methods disclosed in the embodiments of the present invention.
In view of the above object, a second aspect of the embodiments of the present invention provides an apparatus for uploading a file, as shown in fig. 7, an apparatus 200 includes:
the receiving module is configured to receive file fragments and fragment data information sent by the client;
the transmission module is configured to respectively send each file fragment and fragment data information to the storage server side by using a first transmission protocol;
and the verification module is configured to establish connection with the storage server based on the second transmission protocol and carry out integrity verification on the file fragments.
According to one embodiment of the present invention, the server side further comprises a retransmission module configured to request the client side to send the lost file fragments in response to the existence of the lost file fragments, and send the lost file fragments to the storage server side using a second transmission protocol.
In view of the above object, a third aspect of the embodiments of the present invention provides a system for uploading a file, and fig. 8 is a schematic diagram of an embodiment of the system for uploading a file provided by the present invention. As shown in fig. 8, an embodiment of the present invention includes:
a server;
the storage server is connected with the server through a first transmission protocol and/or a first transmission protocol,
wherein, the server side is used for: receiving file fragments and fragment data information sent by a client; respectively transmitting each file fragment and fragment data information to a storage server side by using a first transmission protocol; and establishing connection with a storage server based on a second transmission protocol and carrying out integrity check on the file fragments.
Based on the above object, a fourth aspect of the embodiments of the present invention proposes a computer device. Fig. 9 is a schematic diagram of an embodiment of a computer device provided by the present invention. As shown in fig. 9, an embodiment of the present invention includes the following means: at least one processor 21; and a memory 22, the memory 22 storing computer instructions 23 executable on the processor, which when executed by the processor implement any of the above methods.
In view of the above object, a fifth aspect of the embodiments of the present invention proposes a computer-readable storage medium. Fig. 10 is a schematic diagram showing an embodiment of a computer-readable storage medium provided by the present invention. As shown in fig. 10, the computer-readable storage medium 31 stores a computer program 32 that, when executed by a processor, performs any one of the methods described above.
Furthermore, the method disclosed according to the embodiment of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. The above-described functions defined in the methods disclosed in the embodiments of the present invention are performed when the computer program is executed by a processor.
Furthermore, the above-described method steps and system units may also be implemented using a controller and a computer-readable storage medium storing a computer program for causing the controller to implement the above-described steps or unit functions.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one location to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general purpose or special purpose computer or general purpose or special purpose processor. Further, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.
Claims (14)
1. The file uploading method is characterized by comprising the following steps performed at a server:
receiving file fragmentation and fragmentation data information sent by a client, wherein the fragmentation data information comprises the number of file fragments and the ID of the file fragments;
the method comprises the steps that user datagram protocol is used for respectively sending each file fragment and fragment data information to a storage server side, the server side uses the user datagram protocol for sending the current file fragment to the storage server side, the server side sends the next file fragment to the storage server side in response to the fact that the server side receives a return response sent by the storage server side within a threshold time, and the server side does not continue to wait for directly sending the next file fragment to the storage server side under the condition that no return response message is received in response to the threshold time passing;
Establishing connection with a storage server based on a transmission control protocol and carrying out integrity check on the file fragments, wherein the step of carrying out the integrity check on the file fragments comprises judging whether the file fragments are lost according to the file fragment IDs and the file fragment numbers, wherein the step of judging whether the file fragments are lost according to the file fragment IDs and the file fragment numbers comprises inquiring a file fragment list on the storage server by adopting the transmission control protocol, judging whether the number of the file fragments in the file fragment list is the same as the number of the file fragments contained in fragment data information, and determining that the file fragments are lost and obtaining the lost file fragment IDs in response to the fact that the number of the file fragments in the file fragment list is different from the number of the file fragments contained in fragment data information;
the step of sending each file fragment and fragment data information to a storage server by using a user datagram protocol comprises the following steps: and starting the retransmission of the file fragments based on the file fragment analysis result or the waiting time, wherein the step of starting the retransmission of the file fragments based on the file fragment analysis result or the waiting time comprises the following steps: maintaining a retransmission timer and a retransmission counter at the server, responding to the server to send data to the storage server, starting the retransmission timer, responding to the end of the retransmission timer, and not receiving a positive acknowledgement message sent by the storage server, reading the value of the retransmission counter, responding to the value of the retransmission counter being smaller than a set threshold, retransmitting the data to the storage server, restarting the retransmission timer, adding 1 to the value of the retransmission counter, and responding to the value of the retransmission counter being larger than or equal to the set threshold, and not retransmitting the data to the storage server and resetting the retransmission counter.
2. The method as recited in claim 1, further comprising:
in response to the existence of the lost file fragment, the lost file fragment is sent to the storage server side using a transmission control protocol.
3. The method as recited in claim 1, further comprising:
and the server side judges whether the file fragments are all sent to the storage server side or not, and starts the integrity check based on the judging result.
4. The method of claim 2, wherein the step of transmitting the lost file fragments to the storage server using the transmission control protocol in response to the existence of the lost file fragments comprises:
the server side sends the lost file fragment ID to the client side and acquires the lost file fragment from the client side to be sent to the storage server side through a transmission control protocol.
5. The method as recited in claim 2, further comprising:
and responding to the file fragments received by the storage server and retransmitted by the server, synthesizing each file fragment, storing the synthesized file, and transmitting response information to the server.
6. The method as recited in claim 2, further comprising:
And in response to the absence of the lost file fragments, the storage server synthesizes each file fragment, stores the synthesized file, and sends response information to the server.
7. The method of claim 1, wherein the step of receiving the file fragmentation and fragmentation data information sent by the client comprises:
the client-side divides the file into a plurality of file fragments based on the size and the requirement of the file to be uploaded, and sends each file fragment ID and the number of the file fragments to the server-side.
8. The method of claim 1, wherein the step of using the user datagram protocol to send each of the file fragments and fragment data information to the storage server side comprises:
and analyzing the received current file fragments by the storage server, and starting retransmission of the file fragments based on the file fragment analysis result.
9. The method of claim 8, wherein the parsing, by the storage server side, of the received current file fragment comprises:
the storage server side is responded to successfully analyzing the received file fragments, and a positive acknowledgement message is sent to the server side;
And responding to failure of the storage server to analyze the received file fragments, and sending a negative acknowledgement message to the server.
10. The method of claim 1, wherein the step of initiating retransmission of the file fragments based on the file fragment resolution results or the latency comprises:
and responding to the failure of the analysis of the received file fragments by the storage server, and sending a retransmission request and a negative acknowledgement message to the server, wherein the retransmission request comprises the information of the file fragments with the failure analysis.
11. An apparatus for uploading a file, the apparatus comprising:
the receiving module is configured to receive file fragments and fragment data information sent by the client;
the transmission module is configured to send each file fragment and fragment data information to the storage server side respectively by using a user datagram protocol, the transmission module is further configured to send the current file fragment to the storage server side by using the user datagram protocol, the server side sends the next file fragment to the storage server side in response to the server side receiving a return response sent by the storage server side within a threshold time, and the server side does not continue to wait for directly sending the next file fragment to the storage server side under the condition that no return response message is received in response to the threshold time elapse, wherein the step of sending each file fragment and fragment data information to the storage server side by using the user datagram protocol respectively comprises: and starting the retransmission of the file fragments based on the file fragment analysis result or the waiting time, wherein the step of starting the retransmission of the file fragments based on the file fragment analysis result or the waiting time comprises the following steps: maintaining a retransmission timer and a retransmission counter at the server, responding to the server to send data to the storage server, starting the retransmission timer, responding to the end of the retransmission timer without receiving a positive acknowledgement message sent by the storage server, reading the value of the retransmission counter, responding to the value of the retransmission counter being smaller than a set threshold, retransmitting the data to the storage server and restarting the retransmission timer, adding 1 to the value of the retransmission counter, and responding to the value of the retransmission counter being larger than or equal to the set threshold, not retransmitting the data to the storage server and resetting the retransmission counter;
The verification module is configured to establish connection with the storage server side based on a transmission control protocol and carry out integrity verification on file fragments, wherein the step of carrying out integrity verification on the file fragments comprises judging whether file fragments are lost according to file fragment IDs and file fragment numbers, the step of judging whether file fragments are lost according to the file fragment IDs and the file fragment numbers comprises inquiring a file fragment list on the storage server side by adopting the transmission control protocol, judging whether the number of the file fragments in the file fragment list is the same as the number of the file fragments contained in fragment data information, and determining that the file fragments are lost and obtaining the lost file fragment IDs in response to the fact that the number of the file fragments in the file fragment list is different from the number of the file fragments contained in fragment data information.
12. A system for uploading a file, comprising:
a server;
the storage server is connected with the server through a user datagram protocol and a transmission control protocol,
wherein, the service end is used for: receiving file fragments and fragment data information sent by a client; the user datagram protocol is used for respectively sending each file fragment and fragment data information to a storage server side; establishing connection with a storage server based on the transmission control protocol and carrying out integrity check on the file fragments, wherein the step of carrying out integrity check on the file fragments comprises judging whether file fragments are lost according to file fragment IDs and file fragment numbers, the step of judging whether file fragments are lost according to the file fragment IDs and the file fragment numbers comprises the steps of inquiring a file fragment list on the storage server by adopting the transmission control protocol, judging whether the number of the file fragments in the file fragment list is the same as the number of the file fragments contained in fragment data information, responding to the fact that the number of the file fragments in the file fragment list is different from the number of the file fragments contained in fragment data information, determining that the file fragments are lost and acquiring the lost file fragment IDs, respectively sending each file fragment and each file fragment data information to the storage server by using a user datagram protocol, sending the current file fragments to the storage server by using the server, responding to the fact that the server receives the file fragments at a lower end and directly receiving the file fragments after the server is not received at the lower end, responding to the server, and continuing to store the response that the file fragments are not received at the lower end, and waiting for the server, and waiting for the response time after the server is not received;
The step of sending each file fragment and fragment data information to a storage server by using a user datagram protocol comprises the following steps: and starting the retransmission of the file fragments based on the file fragment analysis result or the waiting time, wherein the step of starting the retransmission of the file fragments based on the file fragment analysis result or the waiting time comprises the following steps: maintaining a retransmission timer and a retransmission counter at the server, responding to the server to send data to the storage server, starting the retransmission timer, responding to the end of the retransmission timer, and not receiving a positive acknowledgement message sent by the storage server, reading the value of the retransmission counter, responding to the value of the retransmission counter being smaller than a set threshold, retransmitting the data to the storage server, restarting the retransmission timer, adding 1 to the value of the retransmission counter, and responding to the value of the retransmission counter being larger than or equal to the set threshold, and not retransmitting the data to the storage server and resetting the retransmission counter.
13. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method of any one of claims 1-10.
14. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310861948.5A CN116599953B (en) | 2023-07-13 | 2023-07-13 | File uploading method, device, system, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310861948.5A CN116599953B (en) | 2023-07-13 | 2023-07-13 | File uploading method, device, system, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116599953A CN116599953A (en) | 2023-08-15 |
| CN116599953B true CN116599953B (en) | 2024-02-02 |
Family
ID=87599383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310861948.5A Active CN116599953B (en) | 2023-07-13 | 2023-07-13 | File uploading method, device, system, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116599953B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117749789B (en) * | 2024-02-07 | 2024-05-03 | 成都倍特数字能源科技有限公司 | Transmission method based on UDP protocol |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101803303A (en) * | 2007-08-31 | 2010-08-11 | 瑞典福拓信息系统有限公司 | Method and system for managing transmission of fragmented data packets |
| CN102938791A (en) * | 2012-11-22 | 2013-02-20 | 上海文广互动电视有限公司 | Information-transmission-rate-adaptive-control file transferring method for 3D (Three-Dimensional) television content |
| CN102957669A (en) * | 2011-08-23 | 2013-03-06 | 中兴通讯股份有限公司 | Version updating method and version updating system for embedded equipment |
| CN105580334A (en) * | 2014-06-30 | 2016-05-11 | 北京新媒传信科技有限公司 | Data transmission method, terminal and server |
| CN114124940A (en) * | 2021-11-30 | 2022-03-01 | 上海御渡半导体科技有限公司 | Data customization transmission method based on UDP protocol |
| CN114629891A (en) * | 2021-12-17 | 2022-06-14 | 亚信科技（中国）有限公司 | File transmission method and device, electronic equipment and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4047480B1 (en) * | 2021-02-19 | 2024-04-24 | Tata Consultancy Services Limited | A low-latency fpga framework based on reliable udp and tcp re-assembly middleware |
-
2023
- 2023-07-13 CN CN202310861948.5A patent/CN116599953B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101803303A (en) * | 2007-08-31 | 2010-08-11 | 瑞典福拓信息系统有限公司 | Method and system for managing transmission of fragmented data packets |
| CN102957669A (en) * | 2011-08-23 | 2013-03-06 | 中兴通讯股份有限公司 | Version updating method and version updating system for embedded equipment |
| CN102938791A (en) * | 2012-11-22 | 2013-02-20 | 上海文广互动电视有限公司 | Information-transmission-rate-adaptive-control file transferring method for 3D (Three-Dimensional) television content |
| CN105580334A (en) * | 2014-06-30 | 2016-05-11 | 北京新媒传信科技有限公司 | Data transmission method, terminal and server |
| CN114124940A (en) * | 2021-11-30 | 2022-03-01 | 上海御渡半导体科技有限公司 | Data customization transmission method based on UDP protocol |
| CN114629891A (en) * | 2021-12-17 | 2022-06-14 | 亚信科技（中国）有限公司 | File transmission method and device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116599953A (en) | 2023-08-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9832681B2 (en) | Higher layer compression with lower layer signaling | |
| JP6023368B1 (en) | Interactive real-time media transfer protocol | |
| CN106330414B (en) | Message transmission method and device | |
| US7921215B2 (en) | Method and apparatus for optimizing and prioritizing the creation of a large number of VPN tunnels | |
| CN110995697A (en) | Big data transmission method and system | |
| US20040052234A1 (en) | Method and system for dispatching multiple TCP packets from communication systems | |
| US20040006643A1 (en) | TCP proxy providing application layer modifications | |
| KR101533870B1 (en) | Apparatus and method for retransmitting message in message transmission system | |
| KR20070015405A (en) | Session initiation protocol retransmission method | |
| CN116599953B (en) | File uploading method, device, system, equipment and readable storage medium | |
| CN110943808B (en) | Data transmission method and device, electronic equipment and storage medium | |
| US8607114B2 (en) | Communication device and communication method | |
| CN107645357B (en) | Recovery method and device for incomplete transmitted file | |
| CN115883680A (en) | UDP (user Datagram protocol) data transmission method, system and equipment based on ARQ (automatic repeat request) | |
| WO2017107148A1 (en) | Method of transmitting data and network equipment | |
| CN113824540B (en) | Multicast retransmission method, device, server and storage medium | |
| CN111543079B (en) | Radio Link Control (RLC) Acknowledged Mode (AM) data reception | |
| US8762449B2 (en) | Method of downloading large size data to a large number of networked client machines from a single server | |
| WO2020010511A1 (en) | Data transmission method and base station | |
| JP4229807B2 (en) | Data transfer method, TCP proxy device, and network system using the same | |
| CN106302361A (en) | A kind of method and apparatus preventing network attack | |
| CN117040692A (en) | Method and device for transmitting service data, electronic equipment and storage medium | |
| WO2018077195A1 (en) | Method and apparatus for data retransmission, and mac device | |
| WO2017067224A1 (en) | Packet processing method and apparatus | |
| JP2017034627A (en) | System and method for communication control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |