WO2021036228A1 - Procédé et appareil de téléversement de fichiers, terminal, serveur, système et moyen de stockage - Google Patents

Procédé et appareil de téléversement de fichiers, terminal, serveur, système et moyen de stockage Download PDF

Info

Publication number
WO2021036228A1
WO2021036228A1 PCT/CN2020/078999 CN2020078999W WO2021036228A1 WO 2021036228 A1 WO2021036228 A1 WO 2021036228A1 CN 2020078999 W CN2020078999 W CN 2020078999W WO 2021036228 A1 WO2021036228 A1 WO 2021036228A1
Authority
WO
WIPO (PCT)
Prior art keywords
file
upload
server
uploaded
service node
Prior art date
Application number
PCT/CN2020/078999
Other languages
English (en)
Chinese (zh)
Inventor
卢启伟
陈鹏宇
Original Assignee
深圳市鹰硕技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市鹰硕技术有限公司 filed Critical 深圳市鹰硕技术有限公司
Publication of WO2021036228A1 publication Critical patent/WO2021036228A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1044—Group management mechanisms 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • H04L67/1078—Resource delivery mechanisms
    • H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments

Definitions

  • the present invention relates to the technical field of file uploading, in particular to a file uploading method, device, terminal, server, system and storage medium.
  • Network storage such as various applications such as network disks and cloud disks
  • various applications are in separate formations, and different file upload methods are used to store files in their respective servers. This method is not conducive to expansion and management, and also causes a large amount of duplicate data.
  • the existence of these duplicate data takes up a lot of network bandwidth during the upload process, and on the other hand causes unnecessary pressure on the network storage server.
  • the purpose of the present invention is to provide a file upload method, device, terminal, server, system and storage medium to solve the above-mentioned background art problems.
  • a method for uploading files is provided.
  • the method is applied to an application side and includes:
  • the file to be uploaded is divided into fragments to obtain several fragments, all the fragments are uploaded to the file upload service node, and after all the fragments are successfully uploaded, the unified storage returned by the file upload service node is received ID;
  • the unified storage ID and the file parameter information of the uploaded file are associated and stored.
  • the calculation of the hash value of the file to be uploaded is specifically:
  • the file parameter information includes file name, file size and file type.
  • the MIME type information of the file to be uploaded is obtained, and the MIME type information is sent to the server.
  • the channel acquisition request carries a task category, and the task category includes a normal task and a VIP task.
  • a server for uploading files.
  • the method is applied to a server and includes:
  • the unified storage ID of the successfully matched file is obtained and returned to the application;
  • the method further includes:
  • the method further includes:
  • the unified storage ID, file hash value, and file storage address are associated and stored.
  • the server includes a normal upload service cluster and a VIP upload service cluster
  • the channel acquisition request carries a task category
  • the task category includes a normal task and a VIP task
  • the method further include:
  • the address of an idle file upload service node in the normal upload service cluster is obtained and returned to the application side;
  • the address of an idle file upload service node in the VIP upload service cluster is obtained and returned to the application side.
  • a file uploading device including:
  • the channel acquisition module is used to initiate a channel acquisition request to the server and receive the file upload service node address returned by the server;
  • the second transmission detection module is used to calculate the hash value of the file to be uploaded, and initiate a second transmission request to the file upload service node.
  • the second transmission request carries the hash value of the file to be uploaded; if the second transmission is successful, the receiving The unified storage ID returned by the file upload service node;
  • the slice upload module is used to slice the file to be uploaded when the second upload fails to obtain several slices, upload all slices to the file upload service node, and receive the file upload after all slices are uploaded successfully The unified storage ID returned by the service node.
  • the information storage module is used for associative storage of the unified storage ID and the file ID of the uploaded file.
  • the file uploading device is configured to perform operations for implementing any one of the above-mentioned file uploading methods applied to the application side.
  • a terminal including:
  • One or more processors are One or more processors;
  • Memory used to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any one of the above-mentioned file upload methods applied to the application side.
  • a file uploading device including:
  • the channel allocation module is used to receive the channel acquisition request initiated by the application side, allocate an idle file upload service node according to the load condition of each upload service node, and return the file upload service node address to the application side;
  • the second transmission processing module is used to receive the second transmission request initiated by the application side, and match the hash value carried in the second transmission request with the hash value of the file stored in the server; if the matching is successful, the matching success is obtained The unified storage ID of the file is returned to the application;
  • the upload processing module is used to receive the fragmented data sent by the application when the hash value matching fails, assemble each fragmented data into a complete file, generate a unified storage ID, and return the unified storage ID to the application end.
  • the file uploading device is used to perform operations for implementing any one of the above-mentioned file uploading methods applied to the server.
  • a server including:
  • One or more processors are One or more processors;
  • Memory used to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any one of the above-mentioned file upload methods applied to the server.
  • a file uploading system includes one or more of the above-mentioned terminals and one or more of the above-mentioned servers.
  • the server interacts.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned file upload methods applied to the application side, or any of the above One of the file upload methods applied to the server.
  • the application terminal establishes communication with the file upload service node after acquiring the channel from the server terminal.
  • the uploaded file will be checked for second transmission first. If the second transmission is successful, the unified storage ID of the file to be uploaded on the server will be received; if the second transmission is unsuccessful, then the slice upload will be performed, and the server will combine all the fragments into a complete file.
  • the application associates the unified storage ID of each uploaded file with the corresponding file parameter information; the server transfers the complete file to the object storage service, The object storage service returns the file storage address to the server, and the server associates and saves key information such as the unified storage ID, file hash value, and file storage address.
  • the server does not store files in substance, only the key information required for uploading or downloading files, which simplifies the server's file uploading and downloading efficiency. It also avoids the problem of network bandwidth pressure and storage pressure caused by a large amount of repeated data.
  • FIG. 1 is a schematic diagram of an implementation environment of a file upload method in an embodiment of the present invention
  • Figure 2 is a flowchart of a file upload method in an embodiment of the present invention
  • Figure 3 is a flowchart of a file upload method in another embodiment of the present invention.
  • Figure 4 is a structural diagram of a file uploading device in an embodiment of the present invention.
  • Figure 5 is a structural diagram of a terminal in an embodiment of the present invention.
  • Fig. 6 is a structural diagram of a file uploading device in another embodiment of the present invention.
  • Fig. 1 is a schematic diagram of an implementation environment of a file upload method in an embodiment of the present invention.
  • the implementation environment includes a terminal 100 and a server 200.
  • the terminal 100 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, or other electronic devices that can run application programs.
  • the application terminal is running on the terminal 100, and is preset with application programs that are connected with the server 200 to realize the file upload function, such as a network disk, a cloud disk, and instant messaging.
  • the user initiates a file upload instruction on the application to upload the file, and the application uploads the file to be uploaded in seconds or slices to the server 200, obtains the unified storage ID of the uploaded file, and combines the unified storage ID with File parameter information is associated and stored.
  • the server 200 handles the file upload and slice upload services, generates a unified storage ID for the uploaded file, and transfers the complete file to the object storage service, and maintains a table of key file information required for the second upload and download.
  • File upload and slice upload services initiated by third-party applications.
  • FIG. 2 it is a flowchart of a file uploading method in an embodiment of the present invention.
  • the file upload method is executed by the application, and the method includes the following steps:
  • Step S201 Initiate a channel acquisition request to the server, and receive the file upload service node address returned by the server.
  • the application before initiating a channel acquisition request to the server, the application sends the application ID and secret key to the server, and the server authenticates them. If the authentication is passed, an access token is assigned to Application side. The application side carries the access token in the channel acquisition request initiated.
  • the server allocates an idle channel to it according to the channel acquisition request, and returns the address of the file upload service node where the channel is located to the application.
  • the channel acquisition request carries a task category, and the task category includes a normal task and a VIP task.
  • the server is also used to assign different types of file upload service nodes to the application according to the task category.
  • Step S202 Calculate the hash value of the file to be uploaded, and initiate a second transfer request to the file upload service node, where the second transfer request carries the hash value of the file to be uploaded.
  • the hash value of the file to be uploaded is calculated to quickly match the file stored on the server to determine whether the same file exists.
  • the original file is generally very large, and it is unrealistic to compare the entire original file with each file on the server.
  • the hash value of the file to be uploaded is generally calculated, and the hash value of the file to be uploaded is compared with the hash value of the file stored on the server to determine whether the same file exists on the server.
  • calculating the hash value of the file to be uploaded is specifically:
  • the threshold is 64KB
  • the hash value of the entire file is calculated. If the file is larger than 64KB, 64KB from the beginning of the file and 64KB from the end of the file are obtained. Merge them together, and then calculate the hash value of this merged file.
  • step S203 if the second transfer is successful, the unified storage ID returned by the file upload service node is received.
  • step S204 if the second upload fails, the file to be uploaded is divided into fragments to obtain several fragments, all the fragments are uploaded to the file upload service node, and after all the fragments are uploaded successfully, the file upload service node returns The unified storage ID.
  • the application side slices the uploaded file according to the preset fragment size to obtain several fragments.
  • the application side uploads each segment to the file upload service node in sequence according to the sharding order.
  • the application side uploads the segment with the file name, segment serial number, total number of segments, file hash value, and segment data. Stream and other information, where the fragmented data stream is a binary data stream.
  • the application sends a complete upload message to the file upload service node.
  • the complete upload message carries the file name, file hash value, file size, file MIME (Multipurpose Internet Mail Extensions, and more). Use Internet mail extension) type information, etc.
  • Multipurpose Internet Mail Extension it is an Internet standard, first applied to e-mail systems in 1992, but later also applied to browsers.
  • the server will tell the browser the type of multimedia data they send, and the notification means is to explain the MIME type of the multimedia data, so that the browser knows which of the received information is an MP3 file, which is a Shockwave file, and so on. Therefore, the application side can obtain the MIME type information of the file to be uploaded by calling the browser API (Application Programming Interface).
  • Each MIME type is composed of two parts. The front is a large category of data, such as audio, image, etc., and the specific type is defined in the back. For example, the MIME type of Hypertext Markup Language text is text/html, and the MIME type of PNG images is image/png.
  • the file upload service node assembles a complete file according to the fragment data, the fragment sequence number, and the total number of fragments sent by the application. If the file upload service node finds that one of the fragments fails to upload when merging files, the file upload service node notifies the application to patch the missing fragments, and the application obtains the corresponding fragments and sends them to the file upload service node .
  • Step S205 Associate and store the unified storage ID and the file parameter information of the uploaded file.
  • the application After the application receives the unified storage ID, it will associate the unified storage ID with the file parameter information of the uploaded file.
  • the file parameter information includes basic file information, such as file name, file size and file type, MIME type, and so on.
  • the file parameter information may also include business information related to the file, such as user ID to which the file belongs, directory structure information of the file, and file classification information, where the directory structure information of the file may include the current node ID and the parent node ID,
  • File classification information mainly relates to the specific classification of resources. If it is a learning resource, it can be classified by subject, grade, etc., if it is a film, it can be classified by comedy, romantic drama, domestic drama, American drama, etc. In actual applications, the application terminal sets the file parameter information according to its own needs, and the present invention does not limit it.
  • the application side After the application side obtains the channel from the server side, it establishes communication with the file upload service node. First, the uploaded file will be checked for second transmission first. If the second transmission is successful, the unified storage ID of the file to be uploaded on the server will be received; if the second transmission is unsuccessful, then the slice upload will be performed, and the server will combine all the fragments into a complete file.
  • the application associates the unified storage ID of each uploaded file with the corresponding file parameter information; the server transfers the complete file to the object storage service, The object storage service returns the file storage address to the server, and the server associates and saves key information such as the unified storage ID, file hash value, and file storage address.
  • the server does not store files in substance, only the key information required for uploading or downloading files, which simplifies the server's file uploading and downloading efficiency. It also avoids the problem of network bandwidth pressure and storage pressure caused by a large amount of repeated data.
  • FIG. 3 it is a flowchart of a file uploading method in another embodiment of the present invention.
  • the file upload method is executed by the server, and the method includes the following steps:
  • Step S301 Receive a channel acquisition request initiated by the application end, allocate an idle file upload service node according to the load condition of each upload service node, and return the file upload service node address to the application end.
  • the server before receiving the channel acquisition request sent by the third-party application, the server receives the authentication request sent by the third-party application, the authentication request carries the application ID and the key, and the server authenticates it, If the authentication is passed, an access token is assigned to the application side.
  • the application side carries the access token in the channel acquisition request initiated.
  • the server includes one or more servers, each server corresponds to a file upload service node, and each file upload service node includes multiple channels.
  • the server receives the channel acquisition request initiated by the application, it generates the corresponding channel request task and adds it to the queue, and the scheduling service processes each channel request task in the queue one by one according to the queuing order.
  • the scheduling service processes each channel request task in the queue one by one according to the queuing order.
  • one or more idle channels located on the same file upload service node are allocated for each channel acquisition request, and the file upload service to which the idle channel belongs The node is returned to the application side.
  • each file upload service node in the server is divided into two types, one is a normal upload service cluster, and the other is a VIP upload service cluster.
  • the server judges that the channel acquisition request carries the task category. If the task category in the channel acquisition request is a normal task, it obtains the address of an idle file upload service node in the ordinary upload service cluster and returns it to the application; If the task category is a VIP task, the address of an idle file upload service node in the VIP upload service cluster is obtained and returned to the application side.
  • Step S302 Receive a second transfer request initiated by the application end, and match the hash value carried in the second transfer request with the hash value of the file stored on the server end.
  • the server saves a file information record table.
  • the file information record table is used to record key information of each uploaded file.
  • the key information includes at least a unified storage ID, a file hash value, and a file storage address. Including file size, file type, MIME type, file upload time, etc.
  • step S303 if the matching is successful, the unified storage ID of the successfully matched file is obtained and returned to the application side.
  • the server will return the unified storage ID of the successfully matched file to the application.
  • the application will associate the unified storage ID with the file parameter information of the file to be uploaded. The file is not recorded.
  • Step S304 If the matching fails, receive the fragmented data sent by the application, assemble each fragmented data into a complete file, generate a unified storage ID, and return the unified storage ID to the application.
  • the file upload service node assembles each fragment into a complete file according to the fragment sequence number, total number of fragments, file hash value, and fragment data stream sent by the application. If the file upload service node finds that one of the fragments fails to upload when merging files, the file upload service node notifies the application to patch the missing fragments, and the application obtains the corresponding fragments and sends them to the file upload service node .
  • the file upload service node receives a file upload completion message sent by the application terminal, and the upload completion message carries file name, file hash value, file size, MIME type information, and the like.
  • the file upload service node determines the file type information of the uploaded file according to the MIME type information. For example, the MIME type is text/html, the file type is html, the MIME type is application/msword, and the file type is doc.
  • the file upload service node dumps the complete file into the object storage service. After the object storage service is successfully stored, the file storage address is returned to the file upload service node.
  • the server sets a dump flag for each file. Before receiving the file storage address returned by the object storage service, the dump flag is set to 0; after receiving the file storage address returned by the object storage service, the dump flag is set to 0; The dump flag is set to 1.
  • the file upload service node associates the unified storage ID, the file hash value, and the file storage address for storage. Specifically, the file upload service node generates and saves a file information record for the uploaded file.
  • the file information record includes at least a unified storage ID, a file hash value, and a file storage address, and may also include file size, dump flag, MIME Type, file type, file upload time, etc.
  • the application side After the application side obtains the channel from the server side, it establishes communication with the file upload service node. First, perform the second transfer detection for the uploaded file. If the second transfer is successful, the unified storage ID of the file to be uploaded on the server will be received; if the second transfer is unsuccessful, then perform the slice upload, and the server will combine all the slices into a complete file Then generate a unified storage ID and return the unified storage ID to the application; the application associates the unified storage ID of each uploaded file with the corresponding file parameter information; the server transfers the complete file to the object storage service, The object storage service returns the file storage address to the server, and the server associates and saves key information such as the unified storage ID, file hash value, and file storage address.
  • the file upload method of the present invention By adopting the file upload method of the present invention, all file access services that are not file downloads can be processed in the own system of each third-party application.
  • the file download service When the file download service is processed, the file can be downloaded only by transmitting the unified storage ID to the server.
  • the server does not store files in a substantial way, but only stores the key information needed to upload or download files, which simplifies the server’s file upload and download efficiency. It also avoids the problem of network bandwidth pressure and storage pressure caused by a large amount of repeated data.
  • Fig. 4 is a structural diagram of a file uploading device in an embodiment of the present invention.
  • the file upload device is deployed on the application side, and includes a channel acquisition module 41, a second transmission detection module 42, a slice upload module 43, and an information storage module 44, wherein:
  • the channel acquisition module 41 is configured to initiate a channel acquisition request to the server, and receive the file upload service node address returned by the server;
  • the second transmission detection module 42 is configured to calculate the hash value of the file to be uploaded, and initiate a second transmission request to the file upload service node.
  • the second transmission request carries the hash value of the file to be uploaded; if the second transmission is successful, then receive The unified storage ID returned by the file upload service node;
  • the slice upload module 43 is used to divide the file to be uploaded into fragments when the second upload fails to obtain several fragments, upload all the fragments to the file upload service node, and receive the file after all the fragments are uploaded successfully Upload the unified storage ID returned by the service node.
  • the information storage module 44 is used for associating and storing the unified storage ID and the file ID of the uploaded file.
  • the application before initiating a channel acquisition request to the server, the application sends the application ID and secret key to the server, and the server authenticates them. If the authentication is passed, an access token is assigned to Application side. The application side carries the access token in the channel acquisition request initiated.
  • the server allocates an idle channel to it according to the channel acquisition request, and returns the address of the file upload service node where the channel is located to the application.
  • the channel acquisition request carries a task category, and the task category includes a normal task and a VIP task.
  • the server is also used to assign different types of file upload service nodes to the application according to the task category.
  • the hash value of the file to be uploaded is calculated to quickly match the file stored on the server to determine whether the same file exists.
  • the original file is generally very large, and it is unrealistic to compare the entire original file with each file on the server.
  • the hash value of the file to be uploaded is generally calculated, and the hash value of the file to be uploaded is compared with the hash value of the file stored on the server to determine whether the same file exists on the server.
  • calculating the hash value of the file to be uploaded is specifically:
  • the threshold is 64KB
  • the hash value of the entire file is calculated. If the file is larger than 64KB, 64KB from the beginning of the file and 64KB from the end of the file are obtained. Merge them together, and then calculate the hash value of this merged file.
  • the application side slices the uploaded file according to the preset fragment size to obtain several fragments.
  • the application side uploads each segment to the file upload service node in sequence according to the sharding order.
  • the application side uploads the segment with the file name, segment serial number, total number of segments, file hash value, and segment data. Stream and other information, where the fragmented data stream is a binary data stream.
  • the application sends a complete upload message to the file upload service node.
  • the complete upload message carries the file name, file hash value, file size, file MIME (Multipurpose Internet Mail Extensions, and more). Use Internet mail extension) type information, etc.
  • Multipurpose Internet Mail Extension it is an Internet standard, first applied to e-mail systems in 1992, but later also applied to browsers.
  • the server will tell the browser the type of multimedia data they send, and the notification means is to explain the MIME type of the multimedia data, so that the browser knows which of the received information is an MP3 file, which is a Shockwave file, and so on. Therefore, the application side can obtain the MIME type information of the file to be uploaded by calling the browser API (Application Programming Interface).
  • Each MIME type is composed of two parts. The front is a large category of data, such as audio, image, etc., and the specific type is defined in the back. For example, the MIME type of Hypertext Markup Language text is text/html, and the MIME type of PNG images is image/png.
  • the file upload service node assembles a complete file according to the fragment data, the fragment sequence number, and the total number of fragments sent by the application. If the file upload service node finds that one of the fragments fails to upload when merging files, the file upload service node notifies the application to patch the missing fragments, and the application obtains the corresponding fragments and sends them to the file upload service node .
  • the application After the application receives the unified storage ID, it will associate the unified storage ID with the file parameter information of the uploaded file.
  • the file uploading device is used to perform operations for implementing the file uploading method described in the above embodiment in FIG. 2.
  • the application side After the application side obtains the channel from the server side, it establishes communication with the file upload service node. First, the uploaded file will be checked for second transmission first. If the second transmission is successful, the unified storage ID of the file to be uploaded on the server will be received; if the second transmission is unsuccessful, then the slice upload will be performed, and the server will combine all the fragments into a complete file.
  • the application associates the unified storage ID of each uploaded file with the corresponding file parameter information; the server transfers the complete file to the object storage service, The object storage service returns the file storage address to the server, and the server associates and saves key information such as the unified storage ID, file hash value, and file storage address.
  • the server does not store files in substance, only the key information required for uploading or downloading files, which simplifies the server's file uploading and downloading efficiency. It also avoids the problem of network bandwidth pressure and storage pressure caused by a large amount of repeated data.
  • FIG. 5 is a structural diagram of a terminal in an embodiment of the present invention.
  • the terminal 500 shown in FIG. 5 is only an example, and should not bring any limitation to the function and application scope of the embodiment of the present invention.
  • the terminal 500 is represented in the form of a general-purpose computing device.
  • the components of the terminal 500 may include, but are not limited to: one or more processors or processing units 51, a system memory 52, and a bus 55 connecting different system components (including the system memory 52 and the processing unit 51).
  • the bus 55 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure among multiple bus structures.
  • these architectures include, but are not limited to, industry standard architecture (ISA) bus, microchannel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and peripheral component interconnection ( PCI) bus.
  • ISA industry standard architecture
  • MAC microchannel
  • the terminal 500 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the terminal, including volatile and non-volatile media, removable and non-removable media.
  • the system memory 52 may include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) 521 and/or cache memory 522.
  • the terminal 500 may further include other removable/non-removable, volatile/non-volatile computer system storage media.
  • the storage system 523 may be used to read and write non-removable, non-volatile magnetic media (not shown in FIG. 5, usually referred to as a "hard drive").
  • a disk drive for reading and writing to a removable non-volatile disk such as a "floppy disk”
  • a removable non-volatile optical disk such as CD-ROM, DVD-ROM
  • other optical media read and write optical disc drives.
  • each drive can be connected to the bus 55 through one or more data medium interfaces.
  • the memory 52 may include at least one program product having a set (e.g., at least one) program modules, which are configured to perform the functions of the various embodiments of the present invention.
  • a program/utility tool 56 having a set of (at least one) program module 58 may be stored in, for example, the memory 52.
  • Such program module 58 includes but is not limited to an operating system, one or more application programs, other program modules, and program data Each of these examples or some combination may include the implementation of a network environment.
  • the program module 58 generally executes the functions and/or methods in the described embodiments of the present invention.
  • the terminal 500 may also communicate with one or more external devices 700 (such as keyboards, pointing devices, displays 600, cameras, etc.), and may also communicate with one or more devices that enable users to interact with the terminal 500, and/or communicate with Any device (such as a network card, modem, etc.) that enables the terminal 500 to communicate with one or more other computing devices. This communication can be performed through an input/output (I/O) interface 53.
  • the terminal 500 may also communicate with one or more networks (for example, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 54. As shown in the figure, the network adapter 54 communicates with other modules of the terminal 500 through the bus 55.
  • LAN local area network
  • WAN wide area network
  • public network such as the Internet
  • terminal 500 can be used in conjunction with the terminal 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.
  • the processing unit 51 executes various functional applications and data processing by running programs stored in the system memory 52 to implement the file upload method described in Embodiment 2 of the present invention.
  • the file to be uploaded is first detected by second transmission, if If the second transfer is successful, the unified storage ID returned by the server will be received; if the second transfer fails, the file to be uploaded will be fragmented to obtain several fragments, and all fragments will be uploaded to the file upload service node. After all the fragments are uploaded successfully, Receive the unified storage ID returned by the file upload service node; store the unified storage ID and the file parameter information of the uploaded file in association.
  • Fig. 6 is a structural diagram of a file uploading device in another embodiment of the present invention.
  • the file uploading device is deployed on the server, and includes a channel allocation module 61, a second transmission processing module 62, and an upload processing module 63, wherein:
  • the channel allocation module 61 is configured to receive a channel acquisition request initiated by the application side, allocate an idle file upload service node according to the load condition of each upload service node, and return the file upload service node address to the application side;
  • the second transmission processing module 62 is configured to receive the second transmission request initiated by the application side, and match the hash value carried in the second transmission request with the hash value of the file stored in the server; if the match is successful, obtain the match The unified storage ID of the successful file is returned to the application;
  • the upload processing module 63 is used to receive the fragmented data sent by the application when the hash value matching fails, assemble each fragmented data into a complete file, generate a unified storage ID, and return the unified storage ID to Application side.
  • the server before receiving the channel acquisition request sent by the third-party application, the server receives the authentication request sent by the third-party application, the authentication request carries the application ID and the key, and the server authenticates it, If the authentication is passed, an access token is assigned to the application side.
  • the application side carries the access token in the channel acquisition request initiated.
  • the server includes one or more servers, each server corresponds to a file upload service node, and each file upload service node includes multiple channels.
  • the server receives the channel acquisition request initiated by the application, it generates the corresponding channel request task and adds it to the queue, and the scheduling service processes each channel request task in the queue one by one according to the queuing order.
  • the scheduling service processes each channel request task in the queue one by one according to the queuing order.
  • one or more idle channels located on the same file upload service node are allocated for each channel acquisition request, and the file upload service to which the idle channel belongs The node is returned to the application side.
  • each file upload service node in the server is divided into two types, one is a normal upload service cluster, and the other is a VIP upload service cluster.
  • the server judges that the channel acquisition request carries the task category. If the task category in the channel acquisition request is a normal task, it obtains the address of an idle file upload service node in the ordinary upload service cluster and returns it to the application; If the task category is a VIP task, the address of an idle file upload service node in the VIP upload service cluster is obtained and returned to the application side.
  • the server saves a file information record table.
  • the file information record table is used to record key information of each uploaded file.
  • the key information includes at least a unified storage ID, a file hash value, and a file storage address. Including file size, file type, MIME type, file upload time, etc.
  • the server will return the unified storage ID of the successfully matched file to the application.
  • the application will associate the unified storage ID with the file parameter information of the file to be uploaded. The file is not recorded.
  • the file upload service node assembles each fragment into a complete file according to the fragment sequence number, total number of fragments, file hash value, and fragment data stream sent by the application. If the file upload service node finds that one of the fragments fails to upload when merging files, the file upload service node notifies the application to patch the missing fragments, and the application obtains the corresponding fragments and sends them to the file upload service node .
  • the file upload service node receives a file upload completion message sent by the application terminal, and the upload completion message carries file name, file hash value, file size, MIME type information, and the like.
  • the file upload service node determines the file type information of the uploaded file according to the MIME type information. For example, the MIME type is text/html, the file type is html, the MIME type is application/msword, and the file type is doc.
  • the file upload service node dumps the complete file into the object storage service. After the object storage service is successfully stored, the file storage address is returned to the file upload service node.
  • the server sets a dump flag for each file. Before receiving the file storage address returned by the object storage service, the dump flag is set to 0; after receiving the file storage address returned by the object storage service, the dump flag is set to 0; The dump flag is set to 1.
  • the file upload service node associates the unified storage ID, the file hash value, and the file storage address for storage. Specifically, the file upload service node generates and saves a file information record for the uploaded file.
  • the file information record includes at least a unified storage ID, a file hash value, and a file storage address, and may also include file size, dump flag, MIME Type, file type, file upload time, etc.
  • the file uploading device is used to perform operations for implementing the file uploading method described in the above-mentioned embodiment of FIG. 3.
  • the application side After the application side obtains the channel from the server side, it establishes communication with the file upload service node. First, the uploaded file will be checked for second transmission first. If the second transmission is successful, the unified storage ID of the file to be uploaded on the server will be received; if the second transmission is unsuccessful, then the slice upload will be performed, and the server will combine all the fragments into a complete file.
  • the application associates the unified storage ID of each uploaded file with the corresponding file parameter information; the server transfers the complete file to the object storage service, The object storage service returns the file storage address to the server, and the server associates and saves key information such as the unified storage ID, file hash value, and file storage address.
  • the server does not store files in a substantial way, but only stores the key information needed to upload or download files, which simplifies the server’s file upload and download efficiency. It also avoids the problem of network bandwidth pressure and storage pressure caused by a large amount of repeated data.
  • a server is provided, and the structure diagram of the server is shown in FIG. 5.
  • the server includes a memory and one or more processors; wherein the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, one or more processing
  • the device implements the file upload method described in Figure 3, such as: receiving the channel acquisition request initiated by the application, assigning an idle file upload service node to the application according to the load of each upload service node; receiving the second upload request initiated by the application , Match the hash value carried in the second pass request with the hash value of the file stored in the server; if the match is successful, the unified storage ID of the successfully matched file is obtained and returned to the application; if the match fails, it is received
  • the fragmented data sent by the application terminal assembles each fragmented data into a complete file, generates a unified storage ID, and returns the unified storage ID to the application terminal.
  • a file upload system is provided, and the structure diagram is shown in FIG. 1.
  • the system includes one or more terminals as described above and one or more servers as described above, and the terminals interact with the server through applications running on them.
  • a computer-readable storage medium stores a computer program, and when the computer program is executed by the storage device, it implements any of the above-mentioned embodiments (for example, FIG. 2 or Figure 3 embodiment) the file upload method.
  • the file upload method in the embodiment of FIG. 2 corresponds to a computer-readable storage medium
  • the file upload method in the embodiment of FIG. 3 corresponds to another computer-readable storage medium.
  • the two kinds of computer-readable storage media may adopt any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable signal medium includes, but is not limited to, conventional systems, devices, or periods of electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or a combination of any of the above.
  • Computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber , Portable compact disk read-only memory (CD_ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by an instruction execution system, apparatus, or device, or a combination thereof.
  • the computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
  • the computer program code used to perform the operations of the present invention can be written in one or more programming languages or a combination thereof.
  • the programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional procedural programming languages. Programming language-such as "C" language or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to pass Internet connection).
  • LAN local area network
  • WAN wide area network
  • the file upload device described above can be implemented as a general-purpose processor, programmable logic controller (PLC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable Gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof.
  • PLC programmable logic controller
  • DSP digital signal processor
  • ASIC application-specific integrated circuit
  • FPGA field programmable Gate array
  • the program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments.
  • the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Theoretical Computer Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

La présente invention divulgue un procédé et un dispositif de téléchargement de fichiers, un terminal, un serveur, un système et un support de stockage, le procédé comprenant : le lancement d'une demande d'acquisition de canal à partir d'un serveur et la réception de l'adresse d'un nœud de service de téléchargement de fichiers renvoyée par le serveur ; le calcul d'une valeur de hachage d'un fichier à télécharger et le lancement d'une demande de téléchargement de secondes à partir du nœud de service de téléchargement de fichiers ; si le téléchargement de secondes est réussi, alors la réception d'un ID de stockage unifié renvoyé par le nœud de service de téléchargement de fichiers ; si le téléchargement secondaire échoue, la division du fichier à télécharger en fragments pour obtenir une pluralité de fragments, le téléchargement de tous les fragments vers le nœud de service de téléchargement de fichiers, et la réception d'un identifiant de stockage unifié renvoyé par le nœud de service de téléchargement de fichiers ; et la mise en œuvre du stockage associé de l'identifiant de stockage unifié et des informations sur les paramètres du fichier téléchargé. La présente invention assure l'indépendance de service d'applications tierces, augmente l'efficacité du téléversement et du téléchargement de fichiers, simplifie la gestion de fichiers par le serveur, et empêche l'apparition de problèmes de pression de bande passante réseau et de problèmes de pression de stockage produits par de grandes quantités de données répétées.
PCT/CN2020/078999 2019-08-23 2020-03-12 Procédé et appareil de téléversement de fichiers, terminal, serveur, système et moyen de stockage WO2021036228A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910781359.XA CN110401725A (zh) 2019-08-23 2019-08-23 文件上传方法、装置、终端、服务器、系统及存储介质
CN201910781359.X 2019-08-23

Publications (1)

Publication Number Publication Date
WO2021036228A1 true WO2021036228A1 (fr) 2021-03-04

Family

ID=68329104

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/078999 WO2021036228A1 (fr) 2019-08-23 2020-03-12 Procédé et appareil de téléversement de fichiers, terminal, serveur, système et moyen de stockage

Country Status (2)

Country Link
CN (1) CN110401725A (fr)
WO (1) WO2021036228A1 (fr)

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110401725A (zh) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 文件上传方法、装置、终端、服务器、系统及存储介质
CN111078414B (zh) * 2019-12-17 2024-03-01 北京思维造物信息科技股份有限公司 请求响应方法、装置、设备和存储介质
CN111083143B (zh) * 2019-12-17 2022-10-21 北京思维造物信息科技股份有限公司 请求响应方法、装置、设备和存储介质
CN111193785B (zh) * 2019-12-20 2023-05-02 北京淇瑀信息科技有限公司 一种文件切割传输方法、装置和电子设备
CN111131450B (zh) * 2019-12-23 2022-06-21 北京奇艺世纪科技有限公司 文件上传方法、重组方法及装置
CN111049927B (zh) * 2019-12-24 2021-03-02 珠海大横琴科技发展有限公司 一种文件存储方法、装置、电子设备和存储介质
CN111147332B (zh) * 2019-12-29 2022-04-29 北京浪潮数据技术有限公司 存储系统云备份的上传进度确定方法、系统及相关装置
CN111211966B (zh) * 2019-12-30 2022-10-18 航天信息股份有限公司 一种用于存储聊天工具中传输文件的方法及系统
CN111209254A (zh) * 2020-01-10 2020-05-29 Oppo（重庆）智能科技有限公司 文件指纹获取方法、装置、电子设备以及存储介质
CN111414334B (zh) * 2020-02-21 2024-04-05 平安科技（深圳）有限公司 基于云技术的文件分片上传方法、装置、设备及存储介质
CN111367865A (zh) * 2020-03-10 2020-07-03 山东汇贸电子口岸有限公司 一种基于aws s3的文件分片上传的实现方法
CN112422604B (zh) * 2020-06-10 2023-02-17 上海哔哩哔哩科技有限公司 文件上传方法、装置、系统及计算机设备
CN112199327A (zh) * 2020-08-24 2021-01-08 杭州雷数科技有限公司 用于处理文件的服务方法、系统、电子设备和存储介质
CN112235390A (zh) * 2020-10-12 2021-01-15 北京金和网络股份有限公司 文件服务方法及装置
CN112422682A (zh) * 2020-11-18 2021-02-26 青岛海尔科技有限公司 数据发送方法及装置、存储介质、电子装置
CN112738157A (zh) * 2020-12-11 2021-04-30 苏州浪潮智能科技有限公司 一种文件上传方法、系统及设备
CN112650729B (zh) * 2020-12-17 2023-06-13 深圳点宽网络科技有限公司 一种分布式文件系统的权限管理方法、系统以及存储介质
CN113726838B (zh) * 2021-06-17 2023-09-19 武汉理工数字传播工程有限公司 文件传输方法、装置、设备及存储介质
CN114172890B (zh) * 2021-11-03 2024-02-27 阿里巴巴（中国）有限公司 文件秒传处理方法、装置、存储介质及电子设备
CN116032913A (zh) * 2022-10-12 2023-04-28 浪潮云信息技术股份公司 基于分布式对象存储的文件上传方法及系统
CN116248670B (zh) * 2023-03-24 2023-11-07 安芯网盾(北京)科技有限公司 一种文件传输控制方法及系统

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102868765A (zh) * 2012-10-09 2013-01-09 乐视网信息技术（北京）股份有限公司 文件上传方法和系统
CN103095824A (zh) * 2013-01-09 2013-05-08 广东一一五科技有限公司 文件上传控制方法及系统
US20130238749A1 (en) * 2004-11-15 2013-09-12 Logmein, Inc. Gateway-assisted file transfer
CN103731451A (zh) * 2012-10-12 2014-04-16 腾讯科技（深圳）有限公司 一种文件上传的方法及系统
CN104067259A (zh) * 2012-04-16 2014-09-24 惠普发展公司，有限责任合伙企业 基于哈希值比较的文件上传
CN104539739A (zh) * 2015-01-26 2015-04-22 浙江大学 一种文件上传的系统、方法及装置
US20170286710A1 (en) * 2016-03-30 2017-10-05 PhazrIO Inc. Distributed cloud storage
CN107483585A (zh) * 2017-08-18 2017-12-15 西安电子科技大学 云环境中支持安全去重的高效数据完整性审计系统及方法
CN107566463A (zh) * 2017-08-21 2018-01-09 北京航空航天大学 一种提高存储可用性的多云存储管理系统
KR20190029843A (ko) * 2017-09-12 2019-03-21 주식회사 비욘드테크 비트토렌트 기반의 네트워크 품질 관리 시스템 및 이를 이용한 네트워크 서비스 품질 향상 방법
CN110401725A (zh) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 文件上传方法、装置、终端、服务器、系统及存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105827700A (zh) * 2016-03-15 2016-08-03 北京金山安全软件有限公司 一种动态文件的传输方法、装置及电子设备

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130238749A1 (en) * 2004-11-15 2013-09-12 Logmein, Inc. Gateway-assisted file transfer
CN104067259A (zh) * 2012-04-16 2014-09-24 惠普发展公司，有限责任合伙企业 基于哈希值比较的文件上传
CN102868765A (zh) * 2012-10-09 2013-01-09 乐视网信息技术（北京）股份有限公司 文件上传方法和系统
CN103731451A (zh) * 2012-10-12 2014-04-16 腾讯科技（深圳）有限公司 一种文件上传的方法及系统
CN103095824A (zh) * 2013-01-09 2013-05-08 广东一一五科技有限公司 文件上传控制方法及系统
CN104539739A (zh) * 2015-01-26 2015-04-22 浙江大学 一种文件上传的系统、方法及装置
US20170286710A1 (en) * 2016-03-30 2017-10-05 PhazrIO Inc. Distributed cloud storage
CN107483585A (zh) * 2017-08-18 2017-12-15 西安电子科技大学 云环境中支持安全去重的高效数据完整性审计系统及方法
CN107566463A (zh) * 2017-08-21 2018-01-09 北京航空航天大学 一种提高存储可用性的多云存储管理系统
KR20190029843A (ko) * 2017-09-12 2019-03-21 주식회사 비욘드테크 비트토렌트 기반의 네트워크 품질 관리 시스템 및 이를 이용한 네트워크 서비스 품질 향상 방법
CN110401725A (zh) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 文件上传方法、装置、终端、服务器、系统及存储介质

Also Published As

Publication number Publication date
CN110401725A (zh) 2019-11-01

Similar Documents

Publication Publication Date Title
WO2021036228A1 (fr) Procédé et appareil de téléversement de fichiers, terminal, serveur, système et moyen de stockage
WO2021164178A1 (fr) Procédé et appareil de téléversement de fragment de fichier basé sur la technologie en nuage, et dispositif et support de stockage
WO2018059222A1 (fr) Procédé et appareil de téléchargement en amont de tranche de fichier, et système de stockage en nuage
KR101035302B1 (ko) 클라우드 시스템 및 클라우드 시스템에서 파일 압축 및 전송 방법
WO2018161881A1 (fr) Procédé de traitement de données structurées, support de stockage de données et appareil informatique
CN109756568A (zh) 文件的处理方法、设备及计算机可读存储介质
US10917484B2 (en) Identifying and managing redundant digital content transfers
CN111935227A (zh) 通过浏览器上传文件的方法、浏览器和电子设备
CN110572422A (zh) 数据下载方法和装置
WO2021088671A1 (fr) Procédé d'appel d'une fonction de terminal, dispositif et support de stockage informatique
WO2021012795A1 (fr) Procédé et appareil de planification de nœud de réseau, dispositif électronique et support d'informations
US20140189055A1 (en) Migration of usage sessions between devices
US9948694B2 (en) Addressing application program interface format modifications to ensure client compatibility
WO2020177731A1 (fr) Procédé de communication en temps réel entre applications hébergées
US11811894B2 (en) Reduction of data transmissions based on end-user context
WO2019242279A1 (fr) Procédé et dispositif de traitement de message
US20230063599A1 (en) Edge computing network, data transmission method and apparatus, device and storage medium
CN113596194B (zh) 一种用于dns流量分类标定的方法和dns服务器
CN112073395B (zh) 一种文件分发方法和装置
CN112016280B (zh) 一种文件编辑方法、装置及计算机可读介质
US20080107111A1 (en) Method and system for controlling flow in an asymmetric communication channel
WO2021042253A1 (fr) Procédé et appareil de collecte de fichiers faisant appel à un disque en nuage, terminal, serveur et système
CN113781154A (zh) 一种信息回滚方法、系统、电子设备及存储介质
CN113760693A (zh) 用于微服务系统的本地调试的方法和装置
US20150127836A1 (en) Method for operating cloud server device, method for operating client device, and client device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20858379

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20858379

Country of ref document: EP

Kind code of ref document: A1