WO2017215646A1 - 数据传输方法和装置 - Google Patents

数据传输方法和装置 Download PDF

Info

Publication number
WO2017215646A1
WO2017215646A1 PCT/CN2017/088563 CN2017088563W WO2017215646A1 WO 2017215646 A1 WO2017215646 A1 WO 2017215646A1 CN 2017088563 W CN2017088563 W CN 2017088563W WO 2017215646 A1 WO2017215646 A1 WO 2017215646A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
file
transmission
transmitted
transmission protocol
Prior art date
Application number
PCT/CN2017/088563
Other languages
English (en)
French (fr)
Inventor
张德龙
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017215646A1 publication Critical patent/WO2017215646A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/40—Network security protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources

Definitions

  • the present disclosure relates to the field of Internet technologies, and in particular, to a data transmission method and apparatus.
  • the existing network disk server and the network disk client transmit data through a single transmission protocol, such as the HTTP hypertext transmission protocol, and a single transmission protocol cannot efficiently transmit each type of data, and the existing network disk cannot Data is transmitted using different transmission protocols depending on the type of data to be transmitted.
  • the embodiment of the invention provides a data transmission method and device, which aims to solve the problem that a single transmission protocol is used when transmitting data, and the transmission efficiency is not high.
  • An embodiment of the present invention provides a data transmission method, where the data transmission method includes:
  • the data type includes file data and a control message
  • the step of acquiring the data type of the data to be transmitted includes:
  • Determining, according to the attribute of the data to be transmitted, the data type of the data to be transmitted is file data or a control message.
  • the step of transmitting the data to be transmitted by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol includes:
  • the data to be transmitted includes the file data and the control message, splitting the file data and the control message in the data to be transmitted;
  • the data transmission method before the step of using the acquired transmission interface corresponding to the obtained transmission protocol and the obtained transmission protocol to transmit the data to be transmitted, the data transmission method also includes:
  • the step of transmitting the data to be transmitted by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol is performed.
  • the step of using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol to transmit the data to be transmitted includes:
  • An embodiment of the present invention further provides a data transmission apparatus, where the data transmission apparatus includes:
  • An acquiring module configured to acquire a data type of data to be transmitted, and obtain a transmission protocol of the data to be transmitted according to the data type
  • a transmission module configured to transmit the to-be-transmitted data by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the data type includes file data and a control message
  • the obtaining module includes:
  • An obtaining unit configured to acquire an attribute of the data to be transmitted
  • a determining unit configured to determine, according to an attribute of the data to be transmitted, a data type of the data to be transmitted as a text Piece of data or control message.
  • the transmission module includes:
  • a splitting unit configured to split file data and control messages in the data to be transmitted
  • a transmission unit configured to transmit the file data obtained by splitting using the transmission interface corresponding to the file data and the acquired transmission protocol, and simultaneously adopting a transmission interface corresponding to the control message and acquiring the transmission protocol The resulting control message is divided.
  • the data transmission device further includes:
  • An identifier module configured to: when the data to be transmitted is file data, calculate fingerprint information of the file data, and transmit the fingerprint information to a server end;
  • a receiving module configured to receive the check result information returned by the server end based on the fingerprint information
  • a determining module configured to determine, according to the check result, whether the file data exists in the server
  • the transmission module is further configured to: if the file data does not exist on the server end, transmit the file data by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the transmission module further includes:
  • a determining unit configured to determine whether a capacity value occupied by the file data is greater than a preset threshold
  • a fragmentation unit configured to divide the file data into a plurality of file slice data if a capacity value occupied by the file data is greater than a preset threshold
  • the transmission unit is further configured to transmit, by using the obtained transmission interface corresponding to the transmission protocol, and the acquired transmission protocol, the file piece data obtained by the splitting.
  • the data transmission method and device provided by the embodiment of the present invention divides the data to be transmitted into file data and control messages when transmitting data, and can transmit the file data and control messages through different transmission protocols, and adopts different data types. Different suitable transmission protocols for transmission can effectively utilize the advantages of various transmission protocols and improve transmission efficiency.
  • FIG. 1 is a schematic flow chart of a first embodiment of a data transmission method according to the present invention.
  • FIG. 2 is a schematic diagram of a refinement process of acquiring data types of data to be transmitted in FIG. 1;
  • FIG. 3 is a schematic diagram of a refinement flow of transmitting data to be transmitted in FIG. 1;
  • FIG. 4 is a schematic flowchart of a second embodiment of a data transmission method according to the present invention.
  • FIG. 5 is a schematic flowchart diagram of a third embodiment of a data transmission method according to the present invention.
  • FIG. 6 is a schematic diagram of functional modules of a first embodiment of a data transmission apparatus according to the present invention.
  • FIG. 7 is a schematic diagram of a refinement function module of the acquisition module in FIG. 6;
  • FIG. 8 is a schematic diagram of a refinement function module of the transmission module in FIG. 6;
  • FIG. 9 is a schematic diagram of functional modules of a second embodiment of a data transmission apparatus according to the present invention.
  • FIG. 10 is a schematic diagram of functional modules of a third embodiment of a data transmission apparatus according to the present invention.
  • an embodiment of the present invention provides a data transmission method.
  • FIG. 1 is a schematic flowchart diagram of a first embodiment of a data transmission method according to the present invention.
  • the data transmission method includes the following steps:
  • Step S10 Obtain a data type of data to be transmitted
  • the network disk client and the network disk server end need to perform data interaction, and the interaction data between the network disk client and the server end can be divided into multiple types, before the data is transmitted, It is necessary to judge and confirm the data type of the data to be transmitted, and then use different and more suitable transmission protocols for different data types for transmission.
  • the data type includes file data and a control message.
  • the step S10 includes:
  • Step S11 acquiring an attribute of the data to be transmitted
  • the data type of the data to be transmitted is determined by acquiring an attribute name of the data to be transmitted.
  • Step S12 Determine, according to the attribute of the data to be transmitted, that the data type of the data to be transmitted is file data or a control message.
  • the interaction data between the client and the server is divided into two types, one is file data, that is, the file data transmitted by the client and the server, and the other is control message. All interaction data outside the file data are control messages, including user rights information, operation instructions, and the number of file data. According to.
  • Step S20 Acquire a transmission protocol of the data to be transmitted according to the data type.
  • step S30 the data to be transmitted is transmitted by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the program running on the client and the server may transmit different types of data to be transmitted through different transmission protocols, and the program includes transmitting multiple types of data using different transmission protocols.
  • Program segments, such as control messages can be transmitted by using a data transfer block of the HTTP Hypertext Transfer Protocol, which can be transferred by using a data transfer block of the FTP File Transfer Protocol.
  • the data to be transmitted can include both file data and control messages.
  • two types of data to be transmitted using different transmission protocols can be simultaneously transmitted using an unused transmission interface.
  • step S30 includes the following steps:
  • Step S31 when the data to be transmitted includes the file data and the control message, splitting the file data and the control message in the data to be transmitted;
  • the data to be transmitted includes the control message and the file data at the same time, for the same type.
  • the data to be transmitted is transmitted by using a consistent transmission protocol, and the data to be transmitted is split before transmission to facilitate transmission of the same type of data to be transmitted.
  • Step S32 using the transmission interface corresponding to the file data and the obtained transmission protocol to transmit the split file data, and simultaneously adopting the transmission interface corresponding to the control message and the obtained transmission protocol transmission splitting.
  • the control message is
  • the network disk client or the server transmits data
  • Encoding when receiving the data, performing the decoding operation according to the transmission protocol, so before the data transmission, the acquired data is performed according to a transmission protocol used by the data transmission program segment. coding.
  • different data transmission program segments may be divided into two types: a control message transmission program segment and a file data transmission program segment, and the control message is transmitted through the control message transmission program segment, through the text.
  • the data transmission program segment transmits the file data
  • the control message transmission program segment and the file data transmission program segment can be set to use a plurality of different transmission protocols, and the control message transmission program segment can use different transmission protocols and files.
  • Data transfer blocks can also use different transport protocols.
  • different data transmission program segments may be referred to as different transmission interfaces.
  • the data interaction transmission interface is divided into a message transmission interface and a data transmission interface.
  • the message transmission interface can use the HTTP protocol to perform control information interaction.
  • the data transmission interface can use the FTP protocol to perform file data interaction, or can also set a second data transmission interface to perform file data interaction using the HTTP protocol.
  • the data transmission method proposed in this embodiment divides the data to be transmitted into file data and control messages when transmitting data, and can transmit the file data and control messages through different transmission protocols, and adopt different suitable data types for different data types.
  • the transmission protocol is transmitted, and the advantages of various transmission protocols can be effectively utilized to improve transmission efficiency.
  • the data transmission method further includes the steps of:
  • Step S40 when the data to be transmitted is file data, calculate fingerprint information of the file data, and transmit the fingerprint information to a server end;
  • the identifier is calculated based on the file content, and different fingerprint information is generated for different file data, and the fingerprint information may be passed through a content digest algorithm (MessageDigestAlgorithm5, MD5).
  • a secure hash algorithm (SecureHashAlgorithm1, SHA-1) and any reliable file content digest algorithm is calculated.
  • Step S50 receiving the check result information returned by the server end based on the fingerprint information
  • the server may establish an index or a database based on the file fingerprint information sent by the client, and when receiving the file fingerprint information uploaded by the client, search for the fingerprint in the index or the database.
  • the information if the fingerprint information exists, that is, the file exists in the server, the message that the file has been uploaded and the storage path information of the file on the server side are sent to the client, and the client does not upload the file. For the file, only the relevant directory is updated on the client, and the storage path of the file is established correspondingly.
  • the fingerprint information of the file stored on the server may be obtained when the client uploads the file, or may be obtained by calculation on the server after the file is uploaded to the server.
  • Step S60 determining, according to the check result, whether the file data exists in the server
  • the server may have stored the file.
  • the server may search for the owner information of the fingerprint information, and determine whether the owner of the fingerprint information and the currently uploaded fingerprint information belong to
  • the main agreement is that when the agreement is consistent, the user can upload a duplicate file, and directly write the information of the file to be uploaded in the file information list, and then point the path of the file to the path of the same file that already exists;
  • the owner information is inconsistent, it can be assumed that multiple users upload the same file.
  • the duplicated file is moved to the public data area on the server side, and the file information list is modified to point the path of the file to the file path corresponding to the common data area, and at the same time
  • the file slice in the public data area increases the number of instances. Whenever an identical file is added, the number of instances is increased. Whenever the same file is deleted, the number of instances is reduced. When 0, the file needs to be deleted in the public data area.
  • step S30 is performed.
  • the data transmission method of the present embodiment calculates the fingerprint information of the file before the file to be uploaded, uploads the fingerprint information to the server for checking, and sends the fingerprint information when the server side exists. Controlling the message to the client, terminating the uploading process of the file, avoiding repeatedly uploading the same file, saving network resources and storage resources, and improving resource utilization.
  • step S30 further includes the following steps:
  • Step S33 determining whether the capacity value occupied by the file data is greater than a preset threshold
  • the network resources are limited, if the file is too large, it takes a lot of time to upload, and it will occupy a large bandwidth for a long time, which affects the network experience of the client user.
  • the uploading time is too long, if the network fails or the network status is not good during the uploading process, it may be difficult to complete the file uploading, resulting in file upload failure, and need to restart the upload again.
  • the small file can be directly uploaded without being fragmented. Therefore, before uploading, you can determine whether the size of the file to be uploaded is greater than the preset threshold.
  • the preset threshold can be set to 20MB. For the file to be uploaded whose file size is larger than the preset threshold, the fragment upload is performed.
  • step S30 If the capacity value of the file data is less than or equal to a preset threshold, perform the step S30;
  • Step S34 if the capacity value occupied by the file data is greater than a preset threshold, the file data is divided into multiple File data;
  • the file when the user selects a file for uploading, the file may be fragmented.
  • the fragmentation mode of the file may be fragmented according to the size of the file to be uploaded, such as the file.
  • the size of the slice is usually related to the file size.
  • the file after the slice can be 2MB, 1MB, 512KB, and the minimum is usually 32KB.
  • the file to be uploaded is very large, for example, more than 10 GB, in order to avoid too many fragments, the total number of fragments is usually fixed, and the fragment size is not fixed, that is, the uploaded file is fragmented according to the number of fragments.
  • Step S35 The obtained file segment data obtained by the splitting is transmitted by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the file may be subjected to a check operation, that is, in the embodiment, after the file is sliced, before the file data is uploaded.
  • Step S40-S60 in the second embodiment of the data transmission method of the present invention when the file data does not exist on the server side, the step S35 is performed, and the split file data is transmitted to the server. .
  • each independent piece is used as a small file to calculate fingerprint information of the file content, so that each piece corresponds to a set of fingerprint information, and this is The group fingerprint information is used as fingerprint information of the file.
  • the server may continue to upload the file fragment of the file to be uploaded according to the recorded retransmission information of the file to be uploaded that is not uploaded.
  • the uploading program of the client can support the file of the resume transfer, that is, when the uploading of the file to be uploaded is partially interrupted, before the uploading process of the client restarts, the server will query the server for how much content has been uploaded, and the server will record the unuploaded.
  • the data transmission method is performed according to the size of the planned upload file, and the file segment segmented according to the file is uploaded to reduce the occupation of network resources and improve the transmission efficiency.
  • the invention further provides a data transmission device.
  • FIG. 6 is a schematic diagram of functional modules of a first embodiment of a data transmission apparatus according to the present invention.
  • the data transmission device includes:
  • the obtaining module 10 is configured to acquire a data type of data to be transmitted and obtain a transmission protocol of the data to be transmitted according to the data type;
  • the data transmission device is a client of the network disk.
  • the network disk client and the network disk server end need to perform data interaction, and the interaction data between the network disk client and the server end can be divided.
  • the data type of the data to be transmitted needs to be judged and confirmed, and different data types are transmitted using different more suitable transmission protocols.
  • the data type includes file data and a control message.
  • the obtaining module 10 includes:
  • the obtaining unit 11 is configured to acquire an attribute of the data to be transmitted
  • the program segment determines the data type of the data to be transmitted by acquiring an attribute name of the data to be transmitted.
  • the determining unit 12 is configured to determine, according to an attribute of the data to be transmitted, a data type of the data to be transmitted as file data or a control message.
  • the interaction data between the client and the server is divided into two types, one is file data, that is, the file data transmitted by the client and the server, and the other is control message. All interaction data outside the file data is control messages, including user rights information, operation instructions and metadata of file data.
  • the transmission module 20 is configured to transmit the to-be-transmitted data by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the program running on the client and the server may transmit different types of data to be transmitted through different transmission protocols, and the program includes transmitting multiple types of data using different transmission protocols.
  • Program segments, such as control messages can be transmitted by using a data transfer block of the HTTP Hypertext Transfer Protocol, which can be transferred by using a data transfer block of the FTP File Transfer Protocol.
  • the data to be transmitted can include both file data and control messages.
  • two types of data to be transmitted using different transmission protocols can be simultaneously transmitted using an unused transmission interface.
  • the transmission module 20 when the data to be transmitted includes file data and a control message, the transmission module 20 includes:
  • a splitting unit 21 configured to split file data and a control message in the data to be transmitted;
  • the data to be transmitted and the data include the control message and the file data at the same time.
  • Type of data to be transmitted, transmitted using a consistent transmission protocol, to be processed before transmission The transmission data is divided to facilitate transmission of the same type of data to be transmitted.
  • the transmission unit 22 is configured to transmit the file data obtained by splitting using the transmission interface corresponding to the file data and the obtained transmission protocol, and simultaneously transmit the transmission interface corresponding to the control message and the acquired transmission protocol. Splitting the resulting control message.
  • the network disk client or the server transmits data
  • Encoding when receiving the data, performing the decoding operation according to the transmission protocol, so before the data transmission, the acquired data is performed according to a transmission protocol used by the data transmission program segment. coding.
  • different data transmission program segments can be divided into two types: a control message transmission program segment and a file data transmission program segment, and the control message transmission block transmits the control message through the file data transmission.
  • the program segment transmits the file data
  • the control message transmission program segment and the file data transmission program segment can be set to use a plurality of different transmission protocols, and the control message transmission program segment can use different transmission protocols, and the file data transmission program Segments can also use different transport protocols.
  • different data transmission program segments may be referred to as different transmission interfaces.
  • the data interaction transmission interface is divided into a message transmission interface and a data transmission interface.
  • the message transmission interface can use the HTTP protocol to perform control information interaction.
  • the data transmission interface can use the FTP protocol to perform file data interaction, or can also set a second data transmission interface to perform file data interaction using the HTTP protocol.
  • the data transmission device provided in this embodiment divides the data to be transmitted into file data and control messages when transmitting data, and can transmit the file data and control messages through different transmission protocols, and adopt different suitable data types for different data types.
  • the transmission protocol is transmitted, and the advantages of various transmission protocols can be effectively utilized to improve transmission efficiency.
  • a second embodiment of the transmission device the data transmission device further includes:
  • the identifier module 30 is configured to calculate a fingerprint letter of the file data when the data to be transmitted is file data. Transmitting the fingerprint information to the server;
  • the identifier is calculated based on the file content, and different fingerprint information is generated for different file data, and the fingerprint information may be passed through a content digest algorithm (MessageDigestAlgorithm5, MD5).
  • a secure hash algorithm (SecureHashAlgorithm1, SHA-1) and any reliable file content digest algorithm is calculated.
  • the receiving module 40 is configured to receive the check result information returned by the server end based on the fingerprint information
  • the server may establish an index or a database based on the file fingerprint information sent by the client, and when receiving the file fingerprint information uploaded by the client, search for the fingerprint in the index or the database.
  • the information if the fingerprint information exists, that is, the file exists in the server, the message that the file has been uploaded and the storage path information of the file on the server side are sent to the client, and the client does not upload the file. For the file, only the relevant directory is updated on the client, and the storage path of the file is established correspondingly.
  • the fingerprint information of the file stored on the server may be obtained when the client uploads the file, or may be obtained by calculation on the server after the file is uploaded to the server.
  • the determining module 50 is configured to determine, according to the check result, whether the file data exists in the server;
  • the server may have stored the file.
  • the server may search for the owner information of the fingerprint information, and determine whether the owner of the fingerprint information and the currently uploaded fingerprint information belong to
  • the main agreement is that when the agreement is consistent, the user can upload a duplicate file, and directly write the information of the file to be uploaded in the file information list, and then point the path of the file to the path of the same file that already exists;
  • the owner information is inconsistent, it can be assumed that multiple users upload the same file.
  • the duplicated file is moved to the public data area on the server side, and the file information list is modified to point the path of the file to the file path corresponding to the common data area, and at the same time
  • the file slice in the public data area increases the number of instances. Whenever an identical file is added, the number of instances is increased. Whenever the same file is deleted, the number of instances is reduced. When 0, the file needs to be deleted in the public data area. .
  • the transmission module 20 is further configured to: if the file data does not exist on the server end, transmit the file data by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the data transmission device of the present embodiment calculates the fingerprint information of the file before the file to be uploaded, uploads the fingerprint information to the server for checking, and sends the fingerprint information when the server side exists. Control the message to the client, terminate the upload process of the file, avoid repeatedly uploading the same file, saving Network resources and storage resources to improve resource utilization.
  • the transmission module further includes:
  • the determining unit 23 is configured to determine whether the capacity value occupied by the file data is greater than a preset threshold
  • the network resources are limited, if the file is too large, it takes a lot of time to upload, and it will occupy a large bandwidth for a long time, which affects the network experience of the client user.
  • the uploading time is too long, if the network fails or the network status is not good during the uploading process, it may be difficult to complete the file uploading, resulting in file upload failure, and need to restart the upload again.
  • the small file can be directly uploaded without being fragmented. Therefore, before uploading, you can determine whether the size of the file to be uploaded is greater than the preset threshold.
  • the preset threshold can be set to 20MB. For the file to be uploaded whose file size is larger than the preset threshold, the fragment upload is performed.
  • the fragmentation unit 24 is configured to divide the file data into a plurality of file slice data if a capacity value occupied by the file data is greater than a preset threshold;
  • the file when the user selects a file for uploading, the file may be fragmented.
  • the fragmentation mode of the file may be fragmented according to the size of the file to be uploaded, such as the file.
  • the size of the slice is usually related to the file size.
  • the file after the slice can be 2MB, 1MB, 512KB, and the minimum is usually 32KB.
  • the file to be uploaded is very large, for example, more than 10 GB, in order to avoid too many fragments, the total number of fragments is usually fixed, and the fragment size is not fixed, that is, the uploaded file is fragmented according to the number of fragments.
  • the transmission unit 22 is further configured to transmit the file piece data obtained by the splitting by using the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol.
  • the file may be subjected to a check operation, that is, in the embodiment, the data transmission device may further include the identifier in the second embodiment.
  • the module 30, the receiving module 40, and the determining module 50, the transmitting unit 22 is further configured to: when the file data does not exist on the server end, use the obtained transmission interface corresponding to the transmission protocol and the obtained transmission protocol to transmit the The obtained file piece data is split.
  • each independent piece is used as a small file to calculate fingerprint information of the file content, so that each piece corresponds to a set of fingerprint information, and this is Group fingerprint information The fingerprint information for the file.
  • the server may continue to upload the file fragment of the file to be uploaded according to the recorded retransmission information of the file to be uploaded that is not uploaded.
  • the uploading program of the client can support the file of the resume transfer, that is, when the uploading of the file to be uploaded is partially interrupted, before the uploading process of the client restarts, the server will query the server for how much content has been uploaded, and the server will record the unuploaded.
  • the file when the file is uploaded, the file is uploaded according to the size of the planned upload file, and the file segment segmented according to the file is uploaded to reduce the occupation of network resources and improve the transmission efficiency.
  • the technical solution of the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic).
  • the disc, the optical disc includes a number of instructions for causing a terminal device (which may be a mobile phone, a computer, a cloud server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
  • the technical solutions provided by the embodiments of the present invention can be applied to the field of Internet technologies.
  • the data to be transmitted is divided into file data and control messages, and the file data and control messages may be transmitted through different transmission protocols, for different data.
  • Types are transmitted using different suitable transmission protocols, which can effectively utilize the advantages of various transmission protocols and improve transmission efficiency.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本公开涉及一种数据传输方法，所述数据传输方法包括：获取待传输数据的数据类型；根据所述数据类型获取所述待传输数据的传输协议；采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。本公开还提供一种数据传输装置。在传输数据时，将待传输数据划分为文件数据和控制消息，通过不同的传输协议传输所述文件数据和控制消息，可以有效地利用各种传输协议的优点，提高传输效率。

Description

数据传输方法和装置 技术领域
本公开涉及互联网技术领域，尤其涉及一种数据传输方法和装置。
背景技术
随着互联网技术与云计算技术的不断发展，很多互联网服务提供商都推出了云存储服务。同时，政企单位也在逐步提供私有云存储服务。通过这些在线的云存储服务，用户可以实现文件的存储、访问、备份、共享等功能。
当前存在多种有效的传输协议，而这些传输协议应对不同的应用场景会有不同的优势存在。然而现有的网盘服务器端与网盘客户端多通过单一传输协议进行数据传输，如HTTP超文本传输协议，单一的传输协议无法高效的传输每一种类型的数据，现有的网盘无法根据待传输数据的类型采用不同的传输协议传输数据。
发明内容
本发明实施例提供一种数据传输方法和装置，旨在解决传输数据时使用单一传输协议，传输效率不高的问题。
本发明实施例提供一种数据传输方法，所述数据传输方法包括：
获取待传输数据的数据类型；
根据所述数据类型获取所述待传输数据的传输协议；
采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。
可选地，所述数据类型包括文件数据以及控制消息，所述获取待传输数据的数据类型的步骤包括：
获取所述待传输数据的属性；
根据所述待传输数据的属性确定所述待传输数据的数据类型为文件数据或者控制消息。
可选地，所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤包括：
在所述待传输数据包括文件数据和控制消息时，拆分所述待传输数据中的文件数据和控制消息；
采用所述文件数据对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件数据，同时采用所述控制消息对应的传输接口以及获取的所述传输协议传输拆分得到的所述控制消息。
可选地，在所述待传输数据为文件数据时，所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤之前，所述数据传输方法还包括：
计算所述文件数据的指纹信息，并将所述指纹信息传输到服务器端；
接收所述服务器端基于所述指纹信息返回的查重结果信息；
基于所述查重结果判断所述服务端是否已存在所述文件数据；
若所述服务器端不存在所述文件数据，则执行所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤。
可选地，在所述待传输数据为文件数据时，所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤包括：
判断所述文件数据所占的容量值是否大于预设阈值；
若所述文件数据所占的容量值大于预设阈值，将所述文件数据分为多个文件片数据；
采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述拆分得到的所述文件片数据。
本发明实施例还提供一种数据传输装置，所述数据传输装置包括：
获取模块，用于获取待传输数据的数据类型以及根据所述数据类型获取所述待传输数据的传输协议；
传输模块，用于采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。
可选地，所述数据类型包括文件数据以及控制消息，所述获取模块包括：
获取单元，用于获取所述待传输数据的属性；
确定单元，用于根据所述待传输数据的属性确定所述待传输数据的数据类型为文 件数据或者控制消息。
可选地，在所述待传输数据包括文件数据和控制消息时，所述传输模块包括：
拆分单元，用于拆分所述待传输数据中的文件数据和控制消息；
传输单元，用于采用所述文件数据对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件数据，同时采用所述控制消息对应的传输接口以及获取的所述传输协议传输拆分得到的所述控制消息。
可选地，所述数据传输装置还包括：
标识模块，用于所述待传输数据为文件数据时，计算所述文件数据的指纹信息，将所述指纹信息传输到服务器端；
接收模块，用于接收所述服务器端基于所述指纹信息返回的查重结果信息；
判断模块，用于基于所述查重结果判断所述服务端是否已存在所述文件数据；
所述传输模块，还用于若所述服务器端不存在所述文件数据，则采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述文件数据。
可选地，在所述待传输数据为文件数据时，所述传输模块还还包括：
判断单元，用于判断所述文件数据所占的容量值是否大于预设阈值；
分片单元，用于若所述文件数据所占的容量值大于预设阈值，将所述文件数据分为多个文件片数据；
所述传输单元，还用于采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件片数据。
本发明实施例提供的数据传输方法和装置，在传输数据时，将待传输数据划分为文件数据和控制消息，可通过不同的传输协议传输所述文件数据和控制消息，针对不同的数据类型采用不同的合适的传输协议进行传输，可以有效地利用各种传输协议的优点，提高传输效率。
附图说明
图1为本发明数据传输方法第一实施例的流程示意图；
图2为图1中获取待传输数据的数据类型的细化流程示意图；
图3为图1中传输待传输数据的细化流程示意图；
图4为本发明数据传输方法第二实施例的流程示意图；
图5为本发明数据传输方法第三实施例的流程示意图；
图6为本发明数据传输装置第一实施例的功能模块示意图；
图7为图6中获取模块的细化功能模块示意图；
图8为图6中传输模块的细化功能模块示意图；
图9为本发明数据传输装置第二实施例的功能模块示意图；
图10为本发明数据传输装置第三实施例的功能模块示意图。
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
基于上述问题，本发明实施例提供一种数据传输方法。
参照图1，图1为本发明数据传输方法的第一实施例的流程示意图。
在本实施例中，所述数据传输方法包括步骤：
步骤S10，获取待传输数据的数据类型；
在本实施例中，网盘系统进行数据传输时网盘客户端与网盘服务器端需要进行数据交互，网盘客户端与服务器端的交互数据可分为多种类型，在将数据进行传输之前，需要对待传输数据的数据类型进行判断确认，进而对不同的数据类型采用不同的更加适合的传输协议进行传输。
所述数据类型包括文件数据以及控制消息，参照图2，所述步骤S10包括：
步骤S11，获取所述待传输数据的属性；
在本实施例中，通过获取待传输数据的属性名称，确定所述待传输数据的数据类型。
步骤S12，根据所述待传输数据的属性确定所述待传输数据的数据类型为文件数据或者控制消息。
在本实施例中，将网盘客户端与服务器端的交互数据分为两种类型，一种为文件数据，即网盘客户端与服务器端传输的文件数据本身，另一种为控制消息，除文件数据外的所有交互数据均为控制消息，包括用户权限信息，操作指令和文件数据的元数 据等。
步骤S20，根据所述数据类型获取所述待传输数据的传输协议；
步骤S30，采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。
在本实施例中，触发文件传输操作时，客户端和服务器端运行的程序可将不同类型的待传输数据通过不同的传输协议传输，所述程序包括使用不同传输协议传输不同类型数据的多个程序段，如控制消息可通过使用HTTP超文本传输协议的数据传输程序段传输，文件数据可通过使用FTP文件传输协议的数据传输程序段传输。
可以理解的是，待传输数据可同时包括文件数据和控制消息，此时对于使用不同传输协议的两种类型的待传输数据，可以采用不用的传输接口同时进行传输。
参照图3，所述步骤S30包括步骤：
步骤S31，在所述待传输数据包括文件数据和控制消息时，拆分所述待传输数据中的文件数据和控制消息；
可以理解的是，在进行数据传输时，不仅要传输文件数据，还要传输用户信息，文件数据的元数据等控制消息，此时所述待传输数据同时包括控制消息和文件数据，对于同一类型的待传输数据，使用一致的传输协议进行传输，传输前要对所述待传输数据进行拆分，方便传输同一类型的待传输数据。
步骤S32，采用所述文件数据对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件数据，同时采用所述控制消息对应的传输接口以及获取的所述传输协议传输拆分得到的所述控制消息。
可以理解的是，使用不同传输协议的待传输数据可以同时进行传输，不同的传输协议占用的端口不同，不会对传输造成影响，产生冲突。
在本实施例中，网盘客户端或服务器端传输数据时，需要根据数据类型选择不同的数据传输程序段，在发送数据时，将所述数据按照选用的数据传输程序段使用的传输协议进行编码，在接收到所述数据时，再将所述数据按照所述传输协议进行解码操作，所以在进行数据传输前，将获取到的所述数据按照所述数据传输程序段使用的传输协议进行编码。
在本实施例中，不同的数据传输程序段可以分为控制消息传输程序段和文件数据传输程序段两种类型，通过所述控制消息传输程序段传输所述控制消息，通过所述文 件数据传输程序段传输所述文件数据，所述控制消息传输程序段和所述文件数据传输程序段均可设置使用多个不同的传输协议，控制消息传输程序段可以使用不同的传输协议，文件数据传输程序段也可使用不同的传输协议。
例如，在本实施例中，可以将不同的数据传输程序段称为不同的传输接口，进行文件上传时，将数据交互的传输接口分为消息传输接口和数据传输接口。消息传输接口可使用HTTP协议进行控制信息的交互，数据传输接口可使用FTP协议进行文件数据的交互，或者还可设置第二数据传输接口，使用HTTP协议进行文件数据的交互。
本实施例提出的数据传输方法，在传输数据时，将待传输数据划分为文件数据和控制消息，可通过不同的传输协议传输所述文件数据和控制消息，针对不同的数据类型采用不同的合适的传输协议进行传输，可以有效地利用各种传输协议的优点，提高传输效率。
可以理解是，不同的用户可能上传相同的文件到云端，同一用户也可能上传相同的文件到云端，进一步地，参照图4，基于第一实施例，提出本发明数据传输方法的第二实施例，所述步骤S30之前，所述数据传输方法还包括步骤：
步骤S40，在所述待传输数据为文件数据时，计算所述文件数据的指纹信息，将所述指纹信息传输到服务器端；
在本实施例中，触发文件上传操作时，对所述文件基于文件内容计算标识，针对不同的文件数据会产生不同的所述指纹信息，所述指纹信息可以通过内容摘要算法(MessageDigestAlgorithm5，MD5)，安全散列算法(SecureHashAlgorithm1，SHA-1)等任一可靠的文件内容摘要算法进行计算得到。
步骤S50，接收所述服务器端基于所述指纹信息返回的查重结果信息；
在本实施例中，所述服务器端可以基于客户端发送的所述文件指纹信息建立索引或数据库，当接收到客户端上传的文件指纹信息时，在所述索引或数据库查找是否存在所述指纹信息，若存在所述指纹信息，即服务器已存在所述文件时，则向所述客户端发送所述文件已经上传完成的消息和所述文件在服务器端的存储路径信息，客户端不再上传所述文件，仅在客户端更新相关目录，并对应建立所述文件的存储路径。
在本实施例中，存储在服务器端的文件的所述指纹信息可以在客户端上传所述文件时获取，也可以在所述文件上传到服务器端后，在服务器端通过计算得到。
步骤S60，基于所述查重结果判断所述服务端是否已存在所述文件数据；
可以理解的是，在用户上传文件时，服务器可能已存储所述文件，此时，服务器可以查找所述指纹信息的属主信息，并判断该指纹信息的属主是否与当前上传的指纹信息属主一致，当一致时可以认定该用户上传了一个重复的文件，并在文件信息列表中直接写入该待上传文件的信息，然后将该文件的路径指向已存在的相同文件的路径；但当属主信息不一致时，可以认定多个用户上传同样的文件片。
当多个用户上传相同的文件片时，在服务器端将该重复上传的文件片移至公共数据区，并修改文件信息列表将文件片的路径指向该公共数据区对应的文件片路径，同时对公共数据区中的文件片增加实例个数的参数，每当增加一个相同文件时，将该实例个数增加；而每当删除一个相同的文件是，将该实例个数减少，当实例个数为0时，需要在公共数据区将该文件删除。
若所述服务器端不存在所述文件数据，则执行所述步骤S30。
本实施例提出的数据传输方法，在待上传文件传输之前，计算所述文件的指纹信息，将所述指纹信息上传到服务器端进行查重，在所述服务器端存在所述指纹信息时，发送控制消息到客户端，终止所述文件的上传进程，避免重复上传相同的文件，节省了网络资源和存储资源，提高资源利用率。
进一步地，基于本发明数据传输方法的第一实施例，提出本发明数据传输方法的第三实施例，参照图5，在所述待传输数据为文件数据时，所述步骤S30还包括步骤：
步骤S33，判断所述文件数据所占的容量值是否大于预设阈值；
由于网络资源是有限的，如果文件过大，则需要花费非常多的时间来上传，并且还会长时间占用较大的带宽，这会影响客户端用户的网络体验。此外，由于上传时间过长，那么在上传过程中如果网络出现故障或者网络状况不佳，则可能难以完成文件上传，导致文件上传失败，需要重新重头开始上传。对于较大的文件可以进行分片上传，而有些小文件可以直接完整上传，不进行分片，所以在上传之前，可以判断待上传文件的大小是否大于预设阈值，预设阈值可以设置为20MB，对于文件大小大于预设阈值的待上传文件才进行分片上传。
若所述文件数据所占的容量值小于或等于预设阈值，则执行所述步骤S30；
步骤S34，若所述文件数据所占的容量值大于预设阈值，将所述文件数据分为多 个文件片数据；
可以理解的是，用户在选择文件进行上传时，可能需要对所述文件进行分片，其中文件的分片方式可以采用现有的按照待上传文件的大小对文件进行分片，比如所述文件大小为T，按照文件分片大小P，将文件划分为N＝T/P片。其中，分片大小通常和文件大小有关，例如分片后的文件可以是2MB，1MB，512KB，最小通常是32KB。当待上传文件非常大时，比如超过10GB，为了避免分片太多，通常总的分片数目就固定，分片大小就不固定，即按照分片数目对待上传文件进行分片。
步骤S35，采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述拆分得到的所述文件片数据。
进一步地，在将待传输的较大文件进行分片上传之前，可以对所述文件进行查重操作，即在本实施例中，在将所述文件分片之后，上传所述文件片数据之前，执行本发明数据传输方法的第二实施例中所述步骤S40-S60，在服务器端不存在所述文件数据时，执行所述步骤S35，传输所述拆分得到的文件片数据到服务器端。
在本实施例中，若所述文件被分为多个文件片，则每个独立的分片作为一个小文件计算文件内容的指纹信息，这样每个分片对应一组指纹信息，把这一组指纹信息作为所述文件的指纹信息。
可以理解的是，当待上传的所述文件上传完成一部分时，所述服务器端可以根据记录的未上传完毕的所述待上传文件的续传信息，继续上传所述待上传文件的文件分片。客户端的上传程序可以支持断点续传文件，即待上传文件上传完成一部分中断时，在客户端的上传程序重新启动上传之前，会向服务器端查询已经上传了多少内容，服务器端会记录未上传完毕的所述待上传文件的续传信息，然后，客户端的上传组件根据此续传信息，继续上传所述待上传文件的文件分片。
本实施例提出的数据传输方法，在进行文件上传时，根据计划上传文件的大小对其进行分片，将根据所述文件切分出的文件片上传，降低网络资源的占用，提高传输效率。
本发明进一步提供一种数据传输装置。
参照图6，图6为本发明数据传输装置第一实施例的功能模块示意图。
在本实施例中，所述数据传输装置包括：
获取模块10，用于获取待传输数据的数据类型以及根据所述数据类型获取所述待传输数据的传输协议；
在本实施例中，所述数据传输装置为网盘的客户端，网盘系统进行数据传输时网盘客户端与网盘服务器端需要进行数据交互，网盘客户端与服务器端的交互数据可分为多种类型，在将数据进行传输之前，需要对待传输数据的数据类型进行判断确认，进而对不同的数据类型采用不同的更加适合的传输协议进行传输。
所述数据类型包括文件数据以及控制消息，参照图7，所述获取模块10包括：
获取单元11，用于获取所述待传输数据的属性；
在本实施例中，程序段通过获取待传输数据的属性名称，确定所述待传输数据的数据类型。
确定单元12，用于根据所述待传输数据的属性确定所述待传输数据的数据类型为文件数据或者控制消息。
在本实施例中，将网盘客户端与服务器端的交互数据分为两种类型，一种为文件数据，即网盘客户端与服务器端传输的文件数据本身，另一种为控制消息，除文件数据外的所有交互数据均为控制消息，包括用户权限信息，操作指令和文件数据的元数据。
传输模块20，用于采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。
在本实施例中，触发文件传输操作时，客户端和服务器端运行的程序可将不同类型的待传输数据通过不同的传输协议传输，所述程序包括使用不同传输协议传输不同类型数据的多个程序段，如控制消息可通过使用HTTP超文本传输协议的数据传输程序段传输，文件数据可通过使用FTP文件传输协议的数据传输程序段传输。
可以理解的是，待传输数据可同时包括文件数据和控制消息，此时对于使用不同传输协议的两种类型的待传输数据，可以采用不用的传输接口同时进行传输。
参照图8，在待传输数据包括文件数据以及控制消息时，所述传输模块20包括：
拆分单元21，用于拆分所述待传输数据中的文件数据和控制消息；
可以理解的是，在进行数据传输时，不仅要传输文件数据，还要传输用户信息，文件数据的元数据等控制消息，此时所述待传输输数据同时包括控制消息和文件数据，对于同一类型的待传输数据，使用一致的传输协议进行传输，传输前要对所述待 传输数据进行划分，方便传输同一类型的待传输数据。
传输单元22，用于采用所述文件数据对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件数据，同时采用所述控制消息对应的传输接口以及获取的所述传输协议传输拆分得到的所述控制消息。
可以理解的是，使用不同传输协议的待传输数据可以同时进行传输，不同的传输协议占用的端口不同，不会对传输造成影响，产生冲突。
在本实施例中，网盘客户端或服务器端传输数据时，需要根据数据类型选择不同的数据传输程序段，在发送数据时，将所述数据按照选用的数据传输程序段使用的传输协议进行编码，在接收到所述数据时，再将所述数据按照所述传输协议进行解码操作，所以在进行数据传输前，将获取到的所述数据按照所述数据传输程序段使用的传输协议进行编码。
在本实施例中，不同的数据传输程序段可以分为控制消息传输程序段和文件数据传输程序段两种类型，通过所述控制消息传输程序段传输所述控制消息，通过所述文件数据传输程序段传输所述文件数据，所述控制消息传输程序段和所述文件数据传输程序段均可设置使用多个不同的传输协议，控制消息传输程序段可以使用不同的传输协议，文件数据传输程序段也可使用不同的传输协议。
例如，在本实施例中，可以将不同的数据传输程序段称为不同的传输接口，进行文件上传时，将数据交互的传输接口分为消息传输接口和数据传输接口。消息传输接口可使用HTTP协议进行控制信息的交互，数据传输接口可使用FTP协议进行文件数据的交互，或者还可设置第二数据传输接口，使用HTTP协议进行文件数据的交互。
本实施例提出的数据传输装置，在传输数据时，将待传输数据划分为文件数据和控制消息，可通过不同的传输协议传输所述文件数据和控制消息，针对不同的数据类型采用不同的合适的传输协议进行传输，可以有效地利用各种传输协议的优点，提高传输效率。
可以理解是，不同的用户可能上传相同的文件到云端，同一用户也可能上传相同的文件到云端，为避免重复上传相同的文件，进一步地，参照图9，基于第一实施例提出本发明数据传输装置第二实施例，所述数据传输装置还包括：
标识模块30，用于所述待传输数据为文件数据时，计算所述文件数据的指纹信 息，将所述指纹信息传输到服务器端；
在本实施例中，触发文件上传操作时，对所述文件基于文件内容计算标识，针对不同的文件数据会产生不同的所述指纹信息，所述指纹信息可以通过内容摘要算法(MessageDigestAlgorithm5，MD5)，安全散列算法(SecureHashAlgorithm1，SHA-1)等任一可靠的文件内容摘要算法进行计算得到。
接收模块40，用于接收所述服务器端基于所述指纹信息返回的查重结果信息；
在本实施例中，所述服务器端可以基于客户端发送的所述文件指纹信息建立索引或数据库，当接收到客户端上传的文件指纹信息时，在所述索引或数据库查找是否存在所述指纹信息，若存在所述指纹信息，即服务器已存在所述文件时，则向所述客户端发送所述文件已经上传完成的消息和所述文件在服务器端的存储路径信息，客户端不再上传所述文件，仅在客户端更新相关目录，并对应建立所述文件的存储路径。
在本实施例中，存储在服务器端的文件的所述指纹信息可以在客户端上传所述文件时获取，也可以在所述文件上传到服务器端后，在服务器端通过计算得到。
判断模块50，用于基于所述查重结果判断所述服务端是否已存在所述文件数据；
可以理解的是，在用户上传文件时，服务器可能已存储所述文件，此时，服务器可以查找所述指纹信息的属主信息，并判断该指纹信息的属主是否与当前上传的指纹信息属主一致，当一致时可以认定该用户上传了一个重复的文件，并在文件信息列表中直接写入该待上传文件的信息，然后将该文件的路径指向已存在的相同文件的路径；但当属主信息不一致时，可以认定多个用户上传同样的文件片。
当多个用户上传相同的文件片时，在服务器端将该重复上传的文件片移至公共数据区，并修改文件信息列表将文件片的路径指向该公共数据区对应的文件片路径，同时对公共数据区中的文件片增加实例个数的参数，每当增加一个相同文件时，将该实例个数增加；而每当删除一个相同的文件是，将该实例个数减少，当实例个数为0时，需要在公共数据区将该文件删除。。
所述传输模块20，还用于若所述服务器端不存在所述文件数据，则采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述文件数据。
本实施例提出的数据传输装置，在待上传文件传输之前，计算所述文件的指纹信息，将所述指纹信息上传到服务器端进行查重，在所述服务器端存在所述指纹信息时，发送控制消息到客户端，终止所述文件的上传进程，避免重复上传相同的文件，节省 了网络资源和存储资源，提高资源利用率。
进一步地，参照图10，基于本发明第一实施例，提出本发明数据传输装置的第三实施例，在所述待传输数据为文件数据时，所述传输模块还包括：
判断单元23，用于判断所述文件数据所占的容量值是否大于预设阈值；
由于网络资源是有限的，如果文件过大，则需要花费非常多的时间来上传，并且还会长时间占用较大的带宽，这会影响客户端用户的网络体验。此外，由于上传时间过长，那么在上传过程中如果网络出现故障或者网络状况不佳，则可能难以完成文件上传，导致文件上传失败，需要重新重头开始上传。对于较大的文件可以进行分片上传，而有些小文件可以直接完整上传，不进行分片，所以在上传之前，可以判断待上传文件的大小是否大于预设阈值，预设阈值可以设置为20MB，对于文件大小大于预设阈值的待上传文件才进行分片上传。
分片单元24，用于若所述文件数据所占的容量值大于预设阈值，将所述文件数据分为多个文件片数据；
可以理解的是，用户在选择文件进行上传时，可能需要对所述文件进行分片，其中文件的分片方式可以采用现有的按照待上传文件的大小对文件进行分片，比如所述文件大小为T，按照文件分片大小P，将文件划分为N＝T/P片。其中，分片大小通常和文件大小有关，例如分片后的文件可以是2MB，1MB，512KB，最小通常是32KB。当待上传文件非常大时，比如超过10GB，为了避免分片太多，通常总的分片数目就固定，分片大小就不固定，即按照分片数目对待上传文件进行分片。
所述传输单元22，还用于采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述拆分得到的所述文件片数据。
进一步地，在将待传输的较大文件进行分片上传之前，可以对所述文件进行查重操作，即在本实施例中，所述数据传输装置还可包括第二实施例中所述标识模块30、接收模块40和判断模块50，所述传输单元22还用于服务器端不存在所述文件数据时，采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述拆分得到的所述文件片数据。
在本实施例中，若所述文件被分为多个文件片，则每个独立的分片作为一个小文件计算文件内容的指纹信息，这样每个分片对应一组指纹信息，将这一组指纹信息作 为所述文件的指纹信息。
可以理解的是，当待上传的所述文件上传完成一部分时，所述服务器端可以根据记录的未上传完毕的所述待上传文件的续传信息，继续上传所述待上传文件的文件分片。客户端的上传程序可以支持断点续传文件，即待上传文件上传完成一部分中断时，在客户端的上传程序重新启动上传之前，会向服务器端查询已经上传了多少内容，服务器端会记录未上传完毕的所述待上传文件的续传信息，然后，客户端的上传组件根据此续传信息，继续上传所述待上传文件的文件分片。
本实施例提出的数据传输装置，在进行文件上传时，根据计划上传文件的大小对其进行分片，将根据所述文件切分出的文件片上传，降低网络资源的占用，提高传输效率。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，云端服务器，空调器，或者网络设备等)执行本发明各个实施例所述的方法。
以上仅为本发明的优选实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关 的技术领域，均同理包括在本发明的专利保护范围内。
工业实用性
本发明实施例提供的技术方案可以应用于互联网技术领域。在本发明实施例提供的数据传输方法和装置中，在传输数据时，将待传输数据划分为文件数据和控制消息，可通过不同的传输协议传输所述文件数据和控制消息，针对不同的数据类型采用不同的合适的传输协议进行传输，可以有效地利用各种传输协议的优点，提高传输效率。

Claims (11)

  1. 一种数据传输方法，其中，所述数据传输方法包括：
    获取待传输数据的数据类型；
    根据所述数据类型获取所述待传输数据的传输协议；
    采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。
  2. 如权利要求1所述的数据传输方法，其中，所述数据类型包括文件数据以及控制消息，所述获取待传输数据的数据类型的步骤包括：
    获取所述待传输数据的属性；
    根据所述待传输数据的属性确定所述待传输数据的数据类型为文件数据或者控制消息。
  3. 如权利要求2所述的数据传输方法，其中，所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤包括：
    在所述待传输数据包括文件数据和控制消息时，拆分所述待传输数据中的文件数据和控制消息；
    采用所述文件数据对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件数据，同时采用所述控制消息对应的传输接口以及获取的所述传输协议传输拆分得到的所述控制消息。
  4. 如权利要求2所述的数据传输方法，其中，在所述待传输数据为文件数据时，所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤之前，所述数据传输方法还包括：
    计算所述文件数据的指纹信息，并将所述指纹信息传输到服务器端；
    接收所述服务器端基于所述指纹信息返回的查重结果信息；
    基于所述查重结果判断所述服务端是否已存在所述文件数据；
    若所述服务器端不存在所述文件数据，则执行所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据的步骤。
  5. 如权利要求2所述的数据传输方法，其中，在所述待传输数据为文件数据时，所述采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待 传输数据的步骤包括：
    判断所述文件数据所占的容量值是否大于预设阈值；
    若所述文件数据所占的容量值大于预设阈值，将所述文件数据分为多个文件片数据；
    采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述拆分得到的所述文件片数据。
  6. 一种数据传输装置，其中，所述数据传输装置包括：
    获取模块，设置为获取待传输数据的数据类型以及根据所述数据类型获取所述待传输数据的传输协议；
    传输模块，设置为采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述待传输数据。
  7. 如权利要求6所述的数据传输装置，其中，所述数据类型包括文件数据以及控制消息，所述获取模块包括：
    获取单元，设置为获取所述待传输数据的属性；
    确定单元，设置为根据所述待传输数据的属性确定所述待传输数据的数据类型为文件数据或者控制消息。
  8. 如权利要求7所述的数据传输装置，其中，在所述待传输数据包括文件数据和控制消息时，所述传输模块包括：
    拆分单元，设置为拆分所述待传输数据中的文件数据和控制消息；
    传输单元，设置为采用所述文件数据对应的传输接口以及获取的所述传输协议传输拆分得到的所述文件数据，同时采用所述控制消息对应的传输接口以及获取的所述传输协议传输拆分得到的所述控制消息。
  9. 如权利要求7所述的数据传输装置，其中，所述数据传输装置还包括：
    标识模块，设置为所述待传输数据为文件数据时，计算所述文件数据的指纹信息，将所述指纹信息传输到服务器端；
    接收模块，设置为接收所述服务器端基于所述指纹信息返回的查重结果信息；
    判断模块，设置为基于所述查重结果判断所述服务端是否已存在所述文件数据；
    所述传输模块，还设置为若所述服务器端不存在所述文件数据，则采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述文件数据。
  10. 如权利要求7所述的数据传输装置，其中，在所述待传输数据为文件数据时，所述传输模块还包括：
    判断单元，设置为判断所述文件数据所占的容量值是否大于预设阈值；
    分片单元，设置为若所述文件数据所占的容量值大于预设阈值，将所述文件数据分为多个文件片数据；
    所述传输单元，还设置为采用获取的所述传输协议对应的传输接口以及获取的所述传输协议传输所述拆分得到的所述文件片数据。
  11. 一种计算机存储介质，其中存储有计算机可执行指令，所述计算机可执行指令用于执行根据权利要求1至5中任一项所述的数据传输方法。
PCT/CN2017/088563 2016-06-17 2017-06-16 数据传输方法和装置 WO2017215646A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610437902.0A CN107517239A (zh) 2016-06-17 2016-06-17 数据传输方法和装置
CN201610437902.0 2016-06-17

Publications (1)

Publication Number Publication Date
WO2017215646A1 true WO2017215646A1 (zh) 2017-12-21

Family

ID=60663007

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/088563 WO2017215646A1 (zh) 2016-06-17 2017-06-16 数据传输方法和装置

Country Status (2)

Country Link
CN (1) CN107517239A (zh)
WO (1) WO2017215646A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114095493A (zh) * 2021-11-17 2022-02-25 兰州乐智教育科技有限责任公司 一种基于断点重传的文件上传方法和系统

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109450989B (zh) * 2018-10-19 2021-09-07 武汉斗鱼网络科技有限公司 一种数据传输的方法、装置、设备及介质
CN110392063B (zh) * 2019-08-27 2022-04-01 深圳市威屏科技有限公司 电子白板数据同步方法、装置、设备及介质
CN110880992A (zh) * 2019-11-29 2020-03-13 中盈优创资讯科技有限公司 网络遥测telemetry协议数据的采集方法及装置
CN112953983A (zh) * 2019-12-10 2021-06-11 中盈优创资讯科技有限公司 Sftp传输方法及装置
CN114759965B (zh) * 2020-12-29 2024-08-09 中国科学院空天信息创新研究院 数据包传输方法、装置、存储介质
CN112689022B (zh) * 2020-12-31 2023-01-06 上海宏力达信息技术股份有限公司 一种数据自动续传方法
CN113987570A (zh) * 2021-10-20 2022-01-28 国网江苏省电力有限公司镇江供电分公司 一种网盘系统
CN115134349B (zh) * 2022-06-06 2024-02-23 蚂蚁区块链科技(上海)有限公司 一种执行传输任务的方法、装置、介质及设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101026464A (zh) * 2006-02-25 2007-08-29 宇龙计算机通信科技(深圳)有限公司 无线终端的无线升级方法和无线升级系统
US20100121744A1 (en) * 2008-11-07 2010-05-13 At&T Intellectual Property I, L.P. Usage data monitoring and communication between multiple devices
CN101895514A (zh) * 2009-05-20 2010-11-24 灰熊(北京)科技有限公司 一种可分散服务器压力的客户端数据信息加载方法
CN102131111A (zh) * 2011-01-25 2011-07-20 天津大学 一种立体视频传输系统
CN104683419A (zh) * 2013-12-03 2015-06-03 航天信息股份有限公司 一种基于android系统和m2m的通信方法和系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101026464A (zh) * 2006-02-25 2007-08-29 宇龙计算机通信科技(深圳)有限公司 无线终端的无线升级方法和无线升级系统
US20100121744A1 (en) * 2008-11-07 2010-05-13 At&T Intellectual Property I, L.P. Usage data monitoring and communication between multiple devices
CN101895514A (zh) * 2009-05-20 2010-11-24 灰熊(北京)科技有限公司 一种可分散服务器压力的客户端数据信息加载方法
CN102131111A (zh) * 2011-01-25 2011-07-20 天津大学 一种立体视频传输系统
CN104683419A (zh) * 2013-12-03 2015-06-03 航天信息股份有限公司 一种基于android系统和m2m的通信方法和系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114095493A (zh) * 2021-11-17 2022-02-25 兰州乐智教育科技有限责任公司 一种基于断点重传的文件上传方法和系统

Also Published As

Publication number Publication date
CN107517239A (zh) 2017-12-26

Similar Documents

Publication Publication Date Title
WO2017215646A1 (zh) 数据传输方法和装置
WO2021164178A1 (zh) 基于云技术的文件分片上传方法、装置、设备及存储介质
US10983961B2 (en) De-duplicating distributed file system using cloud-based object store
US20160057201A1 (en) File Uploading Method, Client, and Application Server in Cloud Storage, and Cloud Storage System
CN112866310B (zh) Cdn回源的校验方法和校验服务器、cdn集群
CN107483627B (zh) 一种文件分发、下载方法、分发服务器、客户端及系统
US8874520B2 (en) Processes and methods for client-side fingerprint caching to improve deduplication system backup performance
US10579595B2 (en) Method and device for calling a distributed file system
WO2019051628A1 (zh) 一种文件分发、下载方法、分发服务器、客户端及系统
WO2017054543A1 (zh) 一种云盘资源访问方法和装置
US20140310241A1 (en) Virtual file system for automated data replication and review
US10862672B2 (en) Witness blocks in blockchain applications
CN110413595B (zh) 一种应用于分布式数据库的数据迁移方法和相关装置
WO2020001488A1 (zh) 文件下载方法及装置、客户端和计算机可读存储介质
US10558581B1 (en) Systems and techniques for data recovery in a keymapless data storage system
US9390101B1 (en) Social deduplication using trust networks
US10025811B2 (en) Method and apparatus for deduplicating encrypted data
WO2014056428A1 (zh) 一种文件上传的方法及系统
WO2016155635A1 (zh) 一种数据处理方法和设备
CN112565393B (zh) 文件上传方法、下载方法、装置、计算机设备和存储介质
CN103227818A (zh) 终端、服务器、文件传输方法、文件存储管理系统和方法
CN110990335B (zh) 日志归档方法、装置、设备及计算机可读存储介质
US9178860B2 (en) Out-of-path, content-addressed writes with untrusted clients
WO2017174013A1 (zh) 数据存储管理方法、装置及数据存储系统
CN104023070B (zh) 基于云存储的文件压缩方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17812757

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17812757

Country of ref document: EP

Kind code of ref document: A1