WO2019205897A1 - 一种数据传输方法及相关设备 - Google Patents

一种数据传输方法及相关设备 Download PDF

Info

Publication number
WO2019205897A1
WO2019205897A1 PCT/CN2019/080823 CN2019080823W WO2019205897A1 WO 2019205897 A1 WO2019205897 A1 WO 2019205897A1 CN 2019080823 W CN2019080823 W CN 2019080823W WO 2019205897 A1 WO2019205897 A1 WO 2019205897A1
Authority
WO
WIPO (PCT)
Prior art keywords
data block
data
index table
dimensional
target data
Prior art date
Application number
PCT/CN2019/080823
Other languages
English (en)
French (fr)
Inventor
范勇
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP19792099.4A priority Critical patent/EP3780438B1/en
Publication of WO2019205897A1 publication Critical patent/WO2019205897A1/zh
Priority to US17/073,438 priority patent/US11476977B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
    • H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
    • H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/1607—Details of the supervisory signal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
    • H04L1/0056—Systems characterized by the type of code used
    • H04L1/0061—Error detection codes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1806—Go-back-N protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04—Protocols for data compression, e.g. ROHC
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
    • H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC

Definitions

  • the present application relates to the field of communications, and in particular, to a data transmission method and related devices.
  • Data transmission is a basic communication service, and most computer services need to utilize data transmission services. Due to the low file transfer efficiency of the traditional file transfer protocol (FTP) server in the wide area network, the file transfer rate of the existing service system in the wide area network (about 300 KB/s) and the local file transfer rate of the service system. (6MB/s or so) The gap is tens of times. The file transfer rate of the WAN can no longer meet the growing business. The WAN data transmission has become a bottleneck that hinders the growth of the business system, so the performance of the WAN file transfer rate is improved. The data transmission applications of the WAN are faced with the problem of low bandwidth and unstable connection and excessive load when transferring small files.
  • FTP file transfer protocol
  • the number of connections is generally reduced by the channel multiplexing mechanism to achieve efficient batch small file transfer.
  • the data congestion problem existing when downloading and downloading more than 1G files is exemplary. As shown in FIG. 1 , a schematic diagram of data congestion phenomenon for downloading files of 1 G or more is generated.
  • a linear file blocking solution is adopted to avoid congestion, and a Hash Tree method is used to quickly identify whether the file content is accurate and ensure the accuracy of the file content.
  • the security is adopted.
  • the Secure Hash algorithm (SHA) divides a large file data with a data size of 6.5 MB into multiple small file data, wherein the hash value is 256 bits, including 6 1MB size and 1 0.5. Small file data in MB size.
  • the size of the hash tree (the depth of the hash tree) is positively related to the size of the file
  • the larger the file the larger the size of the hash tree, and the longer it takes to traverse the hash tree to find the file block.
  • the traversal speed is slow, the search efficiency is low, and when the transmission has a problem, the retransmission takes a long time, and the data repair efficiency is low.
  • the embodiment of the present application provides a data transmission method and related device, which is used to improve the directory traversal speed and improve the efficiency of data search and data repair.
  • a first aspect of the present application provides a data transmission method, including: a first device generates a two-dimensional index table according to a preset multi-thread sequence traversal algorithm and a size of target data to be transmitted, where the two-dimensional index table is used to indicate a storage location of each data in the target data, wherein the target data may include one or more directories, or a complex multi-layer directory, or a single file, or multiple files; the first device performs data on the target data according to the two-dimensional index table.
  • the two-dimensional data block pool includes a plurality of data blocks, each of which corresponds to a coordinate value of the two-dimensional index table, and the coordinate values of each data block in the two-dimensional index table are Different;
  • the first device indexes to the plurality of data blocks in the two-dimensional data block pool through the two-dimensional index table and sends the data blocks to the second device.
  • the first device reassembles the target data that needs to be transmitted to obtain a two-dimensional data block pool, and the first device can quickly locate each data block according to the two-dimensional data block pool, and send the two-dimensional data block pool to the second device.
  • the time taken for data traversal is reduced, the efficiency of data search is improved, and the data transmission efficiency is improved.
  • the first device in a first implementation manner of the first aspect of the embodiments of the present application, the first device generates a two-dimensional index table according to the preset multi-thread sequence traversal algorithm and the size of the target data to be transmitted.
  • the method includes: traversing, by the first device, the target data that needs to be transmitted according to a preset multi-threaded sequence traversal algorithm, to obtain a data directory, where the data directory includes at least one layer of the directory; the first device according to the size of the target data and at least one layer of the directory Determining the abscissa and the ordinate of the two-dimensional index table, the ordinate of the two-dimensional index table is used to indicate the number of groups of the target data, and the abscissa of the two-dimensional index table is used to indicate the number of data blocks corresponding to each group; A device generates a two-dimensional index table, the ordinate of the two-dimensional index table is used to indicate the number of groups, and the abscissa of the two-dimensional
  • the size of each data block is negatively correlated with the network delay; the number of threads is positively correlated with the network bandwidth.
  • the size of the data block and the number of threads are limited.
  • the relationship between the data block and the network delay, and the relationship between the number of threads and the network bandwidth can be quickly set, and the size of the data block and the number of threads can be quickly set, thereby improving the data transmission efficiency.
  • the method further includes: the first device generates a check code of each data block, and the check code is used to verify the correctness of each data block. Sex. The process of generating a check code is added to ensure the accuracy of the transmitted data block.
  • the data block includes header information and content information
  • the header information includes at least a coordinate value and a check code of the data block.
  • the coordinate value is used to indicate the location of the data block in the two-dimensional data block pool
  • the content information includes the data content indicated by the header domain information, and the data content is included in the target data.
  • the structure of the data block is limited, and the implementation manner of the embodiment of the present application is added.
  • the method includes: if the first data block fails to be transmitted, the first device receives the retransmission request sent by the second device, where the The first request includes a first coordinate value of the first data block; the first device determines a position of the first data block in the two-dimensional data block pool according to the first coordinate value; and the first device reacquires the first data block according to the location of the first data block. a data block; the first device sends the re-acquired first data block to the second device. The process of repairing the data block that failed to be transmitted is added, and only the data block that failed to be transmitted is retransmitted, which reduces the time for data repair and improves the data repair efficiency.
  • the header data of the target data block is The first indication information is further included, and the first indication information is used to indicate the location of each target data block in the large file, and constitutes a large file after the transmission is completed.
  • the first indication information indicating the composition of the large file is added, so that after receiving the data block, the second device restores the plurality of data blocks into a large file according to the first indication information.
  • the header domain information of the target data block is further included, where the second indication information is used to indicate the location of each small file included in the target data block, and is decomposed into individual small files after the transmission is completed.
  • the second indication information indicating the composition of the small file is added, so that after receiving the data block, the second device restores the data block into a plurality of small files according to the second indication information.
  • each data block has the same size.
  • the size of each data block is limited, and the obtained two-dimensional data block pool has a more reasonable structure, which is convenient for data search and improves data search efficiency.
  • the data content is one or more object files; or the data content is part of the content of the target file.
  • the header field information further includes a target path, a source file name, a target file name, a file offset address, and a data block size.
  • the content included in the header field information of the data block is limited, so that the application is more complete in the steps.
  • the second aspect of the present application provides a data transmission method, including: receiving, by a second device, a plurality of data blocks sent by the first device, each data block includes one coordinate value; and the second device generates a corresponding data according to the plurality of coordinate values.
  • a two-dimensional index table the two-dimensional index table is used to indicate a storage location of the plurality of data blocks; and the second device sorts each of the plurality of data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the data content contained in each data block in the two-dimensional data block pool constitutes the required target data.
  • the second device generates a two-dimensional index table according to the coordinate values included in the received plurality of data blocks, and generates a two-dimensional index table according to each coordinate value, and recombines the target data that needs to be transmitted to obtain a two-dimensional data block pool, and the second device can
  • the dimension data block pool quickly locates each data block and searches for the data blocks in the two-dimensional data block pool, which reduces the length of time for data traversal, improves the efficiency of data search, and improves data transmission efficiency.
  • the generating, by the second device, the corresponding two-dimensional index table according to the coordinate value includes: the second device according to the second one of the coordinate values The value determines the ordinate of the two-dimensional index table, the ordinate of the two-dimensional index table is used to indicate the number of groups of the target data; the second device determines the abscissa of the two-dimensional index table according to the first value in the coordinate value, the second The abscissa of the dimension index table is used to indicate the number of data blocks corresponding to each group; the second device generates a corresponding two-dimensional index table according to the abscissa and the ordinate.
  • the process of generating the two-dimensional index table by the first device is refined, and a specific implementation manner is provided, and the implementation manner of the application is added.
  • the size of each data block is negatively correlated with the network delay; the number of threads is positively correlated with the network bandwidth.
  • the size of the data block and the number of threads are limited.
  • the relationship between the data block and the network delay, and the relationship between the number of threads and the network bandwidth can be quickly set, and the size of the data block and the number of threads can be quickly set, thereby improving the data transmission efficiency.
  • the second device after the second device receives the multiple data blocks sent by the first device, the second device generates a corresponding image according to the multiple coordinate values. Before the two-dimensional index table, the method further includes: the second device performs verification according to the check code carried in each data block, and determines the correctness of each data block. The process of generating a check code is added to ensure the accuracy of the transmitted data block.
  • the data block includes header field information and content information, where the header field information includes at least a coordinate value of the data block and a check code.
  • the coordinate value is used to indicate the location of the data block in the two-dimensional data block pool;
  • the content information includes the data content indicated by the header domain information, and the data content is included in the target data.
  • the structure of the data block is limited, and the implementation manner of the embodiment of the present application is added.
  • the method includes: if the first data block fails to be transmitted, the second device determines a first coordinate value of the first data block; The second device sends a retransmission request to the first device, where the retransmission request includes the first coordinate value, and the second device receives the first data block that is resent by the first device.
  • the process of repairing the data block that failed to be transmitted is added, and only the data block that failed to be transmitted is retransmitted, which reduces the time for data repair and improves the data repair efficiency.
  • the target data when the target data includes a large file, and the large file is composed of multiple target data blocks, the header data of the target data block is included.
  • the first indication information is further included, and the first indication information is used to indicate the location of each target data block in the large file, and constitutes a large file after the transmission is completed.
  • the first indication information indicating the composition of the large file is added, so that after receiving the data block, the second device restores the plurality of data blocks into a large file according to the first indication information.
  • the header domain information of the target data block is further included, where the second indication information is used to indicate the location of each small file included in the target data block, and is decomposed into individual small files after the transmission is completed.
  • the second indication information indicating the composition of the small file is added, so that after receiving the data block, the second device restores the data block into a plurality of small files according to the second indication information.
  • each data block has the same size.
  • the size of each data block is limited, and the obtained two-dimensional data block pool has a more reasonable structure, which is convenient for data search and improves data search efficiency.
  • the data content is one or more object files; or the data content is part of the content of the target file.
  • the header domain information further includes a target path, a source file name, a target file name, a file offset address, and a data block size.
  • the content included in the header field information of the data block is limited, so that the application is more complete in the steps.
  • the third aspect of the present application provides a device, where the device is a first device, including: a first generating unit, configured to generate a two-dimensional index according to a preset multi-thread sequence traversal algorithm and a size of target data to be transmitted.
  • the two-dimensional index table is used to indicate a storage location of each data in the target data, wherein the target data may include one or more directories, or a complex multi-layer directory, or a single file, or multiple files; a unit, configured to perform data recombination on the target data according to the two-dimensional index table, to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, each data block corresponding to a coordinate value of the two-dimensional index table; a first sending unit, configured to index to the plurality of data blocks by using the two-dimensional index table and send the same to the second device.
  • the first device reassembles the target data that needs to be transmitted to obtain a two-dimensional data block pool, and the first device can quickly locate each data block according to the two-dimensional data block pool, and send the two-dimensional data block pool to the second device.
  • the time taken for data traversal is reduced, the efficiency of data search is improved, and the data transmission efficiency is improved.
  • the first generating unit includes: a traversing module, configured to transmit according to a preset multi-threaded sequence traversal algorithm The target data is traversed to obtain a data directory, the data directory includes at least one layer of a directory; and a determining module configured to determine an abscissa and an ordinate of the two-dimensional index table according to the size of the target data and the at least one layer directory, the two-dimensional The ordinate of the index table is used to indicate the number of packets of the target data, the abscissa of the two-dimensional index table is used to indicate the number of data blocks corresponding to each group, and the generating module is configured to generate a two-dimensional index table.
  • the process of generating the two-dimensional index table by the first device is refined, and a specific implementation manner is provided, and the implementation manner of the application is added.
  • the size of each data block is negatively correlated with the network delay; the number of threads is positively correlated with the network bandwidth.
  • the size of the data block and the number of threads are limited.
  • the relationship between the data block and the network delay, and the relationship between the number of threads and the network bandwidth can be quickly set, and the size of the data block and the number of threads can be quickly set, thereby improving the data transmission efficiency.
  • the device further includes: a second generating unit, configured to generate a check code of each data block, the check The code is used to verify the correctness of each data block. The process of generating a check code is added to ensure the accuracy of the transmitted data block.
  • the data block includes header information and content information
  • the header information includes at least a coordinate value and a check code of the data block.
  • the coordinate value is used to indicate the position of the data block in the two-dimensional data block pool
  • the content information includes the data content indicated by the header field information, and the data content is included in the target data.
  • the structure of the data block is limited, and the implementation manner of the embodiment of the present application is added.
  • the method includes: receiving, if the first data block fails to transmit, retransmitting the second device to send a request, the retransmission request includes a first coordinate value of the first data block, and a determining unit, configured to determine, according to the first coordinate value, the first data block in the two-dimensional data block pool a location unit, configured to re-acquire the first data block according to the location of the first data block, and a second sending unit, configured to send the re-acquired first data block to the second device.
  • the process of repairing data that fails to transmit is added, and only the data block that failed to be transmitted is retransmitted, which reduces the time for data repair and improves the efficiency of data repair.
  • the target data when the target data includes a large file, and the large file is composed of multiple target data blocks, the header data of the target data block is included.
  • the first indication information is further included, and the first indication information is used to indicate the location of each target data block in the large file, and constitutes a large file after the transmission is completed.
  • the first indication information indicating the composition of the large file is added, so that after receiving the data block, the second device restores the plurality of data blocks into a large file according to the first indication information.
  • the header domain information of the target data block is further included, where the second indication information is used to indicate the location of each small file included in the target data block, and is decomposed into individual small files after the transmission is completed.
  • the second indication information indicating the composition of the small file is added, so that after receiving the data block, the second device restores the data block into a plurality of small files according to the second indication information.
  • each data block has the same size.
  • the size of each data block is limited, and the obtained two-dimensional data block pool has a more reasonable structure, which is convenient for data search and improves data search efficiency.
  • the data content is one or more object files; or the data content is part of the content of the target file.
  • the header domain information further includes a target path, a source file name, a target file name, a file offset address, and a data block size.
  • the content included in the header field information of the data block is limited, so that the application is more complete in the steps.
  • the fourth aspect of the present application provides a device, where the device is a second device, including: a first receiving unit, configured to receive multiple data blocks sent by the first device, each data block includes a coordinate value; a unit, configured to generate a corresponding two-dimensional index table according to the plurality of coordinate values, where the two-dimensional index table is used to indicate a storage location of the plurality of data blocks, and a sorting unit is configured to use the two-dimensional index table to multiple data Each data block in the block is sorted to obtain a two-dimensional data block pool, and the two-dimensional data block pool is target data to be transmitted.
  • the second device generates a two-dimensional index table according to the coordinate values included in the received plurality of data blocks, and generates a two-dimensional index table according to each coordinate value, and recombines the target data that needs to be transmitted to obtain a two-dimensional data block pool, and the second device can
  • the dimension data block pool quickly locates each data block and searches for the data blocks in the two-dimensional data block pool, which reduces the length of time for data traversal, improves the efficiency of data search, and improves data transmission efficiency.
  • the generating unit includes: a first determining module, configured to determine, according to the second value in the coordinate value, the two-dimensional index table The ordinate, the ordinate of the two-dimensional index table is used to indicate the number of groups of the target data; the second determining module is configured to determine the abscissa of the two-dimensional index table according to the first value of the coordinate values, the second The abscissa of the dimension index table is used to indicate the number of data blocks corresponding to each group; and the generating module is configured to generate a corresponding two-dimensional index table according to the abscissa and the ordinate.
  • the process of generating the two-dimensional index table by the first device is refined, and a specific implementation manner is provided, and the implementation manner of the application is added.
  • the size of each data block is negatively correlated with the network delay; the number of threads is positively correlated with the network bandwidth.
  • the size of the data block and the number of threads are limited.
  • the relationship between the data block and the network delay, and the relationship between the number of threads and the network bandwidth can be quickly set, and the size of the data block and the number of threads can be quickly set, thereby improving the data transmission efficiency.
  • the device further includes: a check unit, configured to perform verification according to the check code carried in each data block. To determine the correctness of each data block. The process of generating a check code is added to ensure the accuracy of the transmitted data block.
  • the data block includes header field information and content information
  • the header field information includes at least a coordinate value and a check code of the data block.
  • the coordinate value is used to indicate the location of the data block in the two-dimensional data block pool
  • the content information includes the data content indicated by the header domain information, and the data content is included in the target data.
  • the structure of the data block is limited, and the implementation manner of the embodiment of the present application is added.
  • the method includes: acquiring, if the first data block fails to transmit, determining, by using the first data block, the first a coordinate unit, configured to send a retransmission request to the first device, where the retransmission request includes the first coordinate value, and a second receiving unit, configured to receive, by the first device, a retransmission A block of data.
  • the process of repairing data that fails to transmit is added, and only the data block that failed to be transmitted is retransmitted, which reduces the time for data repair and improves the efficiency of data repair.
  • the target data when the target data includes a large file, and the large file is composed of multiple target data blocks, the header data of the target data block is included.
  • the first indication information is further included, and the first indication information is used to indicate the location of each target data block in the large file, and constitutes a large file after the transmission is completed.
  • the first indication information indicating the composition of the large file is added, so that after receiving the data block, the second device restores the plurality of data blocks into a large file according to the first indication information.
  • the header domain information of the target data block is further included, where the second indication information is used to indicate the location of each small file included in the target data block, and is decomposed into individual small files after the transmission is completed.
  • the second indication information indicating the composition of the small file is added, so that after receiving the data block, the second device restores the data block into a plurality of small files according to the second indication information.
  • each data block has the same size.
  • the size of each data block is limited, and the obtained two-dimensional data block pool has a more reasonable structure, which is convenient for data search and improves data search efficiency.
  • the data content is one or more object files; or the data content is part of the content of the target file.
  • the header domain information further includes a target path, a source file name, a target file name, a file offset address, and a data block size.
  • the content included in the header field information of the data block is limited, so that the application is more complete in the steps.
  • a fifth aspect of the present application provides a computer readable storage medium having stored therein instructions that, when executed on a computer, cause the computer to perform the methods described in the above aspects.
  • a sixth aspect of the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the methods described in the various aspects above.
  • Figure 1 is a schematic diagram showing the relationship between download rate and file size during data downloading
  • FIG. 2 is a schematic diagram of a hash tree obtained by the SHA-256 algorithm
  • FIG. 3 is a schematic diagram of a network architecture applied to an embodiment of the present application.
  • 4A is a schematic diagram of an embodiment of a data transmission method in an embodiment of the present application.
  • 4B is a schematic diagram of data traversal using a multi-threaded sequence traversal algorithm in the embodiment of the present application.
  • 4C is a schematic diagram of a two-dimensional index table generated in the embodiment of the present application.
  • 4D is a schematic diagram of relationship between data blocks and files in a two-dimensional data block pool according to an embodiment of the present application
  • FIG. 5 is a schematic diagram of another embodiment of a data transmission method according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of an embodiment of a first device in an embodiment of the present application.
  • FIG. 7 is a schematic diagram of another embodiment of a first device in an embodiment of the present application.
  • FIG. 8 is a schematic diagram of an embodiment of a second device according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of another embodiment of a second device according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of another embodiment of a device in an embodiment of the present application.
  • FIG. 11 is a schematic diagram of another embodiment of a device in an embodiment of the present application.
  • the embodiment of the present application provides a data transmission method and related device, which is used to improve the directory traversal speed and improve the efficiency of data search and data repair.
  • first or “second” and the like referred to in the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or order.
  • the words “including” or “comprising”, and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited Those steps or units that are clearly listed may include other steps or units that are not explicitly listed or inherent to such processes, methods, products, or devices.
  • the transmission bandwidth of 2G/s is more than 2MB/s under the condition that the transmission bandwidth of the WAN is 20MB/s.
  • the actual transmission bandwidth utilization is over 80%.
  • the solution of linear file partitioning is adopted in the existing scheme, and the accuracy of the file content is ensured by using a hash tree, but the file size increases in the scheme. The longer the traversal takes, the more hardware resources are consumed.
  • the application may be in a network framework as shown in FIG. 3, in which the first device is used as the source device and the second device is used as the target device, and the first device generates a two-dimensional index table by indexing the two-dimensional index table. Each coordinate corresponds to the data and is loaded, and the loaded file data is transmitted to the second device, thereby implementing the full process processing of the transmission file from the source device to the target device, reducing the length of time required for traversing, and improving transmission efficiency.
  • the application is based on the standard HTTP1 protocol of the RFC2616, and can be extended to application scenarios such as cloud storage, network disk, and mobile transmission. It can be understood that the application itself is based on the reliable transmission of the large amount of data across the area of the cloud or the server cluster.
  • the first device and the second device involved in the present application may be a single device or multiple devices. For the convenience of description, only a single device is taken as an example for description.
  • an embodiment of the data transmission method in the embodiment of the present application includes:
  • the first device generates a two-dimensional index table according to the preset multi-thread sequence traversal algorithm and the size of the target data to be transmitted.
  • the first device generates a two-dimensional index table according to the preset multi-thread sequence traversal algorithm and the size of the target data to be transmitted, and the two-dimensional index table is used to indicate the storage location of each data in the target data.
  • the first device first determines the size of the target data that needs to be transmitted to the second device, and then divides the target data into multiple standard-sized data blocks according to the size of the target data, where each data block has the same size;
  • Each data block in the target data is traversed by a preset multi-threaded sequence traversal algorithm to obtain a data directory, the data directory includes at least one layer of the directory; the two-dimensional index is determined according to the size of the target data and the at least one layer of the directory.
  • the abscissa and the ordinate of the table wherein the ordinate of the two-dimensional index table is used to indicate the number of groups, and the abscissa of the two-dimensional index table is used to indicate the number of data blocks corresponding to each group; and a two-dimensional index table is generated.
  • the first device configures one or more threads for each layer directory, that is, one thread is allocated for each group, and each thread is used to find a data block of the corresponding group; and the concurrent is determined according to the number of groups and the preset thread pool. The number of threads.
  • the preset multi-threaded layer traversal algorithm is an improvement on the existing sequence traversal algorithm. Multiple threads are introduced, and multiple data blocks can be traversed at the same time to improve ergodic efficiency.
  • the target data can include one or more directories, or multiple complex directories.
  • the data directory may include a 4-layer directory (shown by a dashed box), which is a first layer directory, a second layer directory, a third layer directory, and a fourth layer directory, each layer.
  • the directory corresponds to one or more data blocks, each of which contains specific data content.
  • Configure threads for each layer of directories shown in the solid box, where the first level of the directory is configured with one thread, the second level of the directory is configured with six threads, the third level of the directory is configured with six threads, and the fourth level of the directory is configured with three threads.
  • Thread each thread is used to find the data content of the corresponding block of the data block; according to the size of the target data and the preset thread pool, determine the number of concurrent threads used to find the data block in the packet, the size of the preset thread pool Determining the maximum number of threads; generating a two-dimensional index table, identifying the abscissa and ordinate of the two-dimensional index table, for example, as shown in FIG. 4C, the ordinate of the generated two-dimensional index table corresponds to nine groups, The letters ABCDEFGHI are identified. The number of data blocks corresponding to each group is 9, which is identified by the number 123456789. The abscissa and the ordinate can also be identified by other means.
  • the abscissa is identified by a letter and the ordinate is identified by a number. This is not limited here. It can be understood that the same layer directory may include multiple sub-directories, which is a tree structure (not shown in FIG. 4B), which is not limited herein.
  • the first device performs data recombination on the target data according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the first device performs data recombination on the target data according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the two-dimensional data block pool includes a plurality of data blocks, and each of the data blocks respectively corresponds to a coordinate value of the two-dimensional index table.
  • the first device divides the data that needs to be transmitted into one data block according to the established two-dimensional index table, and reorganizes and sorts each data block to obtain a two-dimensional data block pool corresponding to the two-dimensional index table.
  • Each data block in the two-dimensional data block pool corresponds to one coordinate value in the two-dimensional index table.
  • the coordinate value of the first data block is (1, A).
  • the corresponding data size can be 100MB.
  • Each data block is composed of two parts: header field information and content information.
  • the header field information includes at least a coordinate value of the data block and a check code, where the coordinate value is used to indicate that the data block is in the two-dimensional data block pool. The location, the check code is used to verify the correctness of the data block; the content information includes the data content indicated by the header field information, and the data content is included in the target data.
  • the header field information may also include related information such as a target path, a source file name, a target file name, a file offset address, and a data block size.
  • the number of concurrent threads can be increased to improve the transmission efficiency.
  • the number of concurrent threads is positively related to the network bandwidth. The larger the network bandwidth is. The more threads are concurrent, and the number of concurrent threads is also related to the size of the preset thread pool. The larger the thread pool, the more concurrent threads.
  • the data content of multiple data blocks can form a large file together, or a data block can form a standard file, or a plurality of small files can form a data block, wherein the standard file is A file with the same data block size, for example, a file of 1 MB size; a large file is a file larger than a set data block size, for example, a file of 1 G size; a small file is a file smaller than a set data block size, For example, a file of 100KB size.
  • 26 data blocks constitute file 1 (411), 1 data block constitutes file 2 (412), 1 data block constitutes file 3 (413), and 1 data block includes file 4 (414).
  • document 5 (415).
  • the first device indexes the multiple data blocks through the two-dimensional index table and sends the data to the second device.
  • the first device indexes to the plurality of data blocks through the two-dimensional index table and transmits to the second device.
  • the first device indexes each data block in the two-dimensional data block pool according to the two-dimensional index table, and performs compression separately, and then sends the compressed data packet corresponding to each data block to the second device through the network.
  • the first device may obtain a corresponding data request, where the data request is used to instruct the first device to send the target data to the second device, where the target
  • the data may include one or more files, which are not described here.
  • the second device generates a corresponding two-dimensional index table according to the plurality of coordinate values.
  • the second device acquires coordinate values included in each data block according to the received data block, wherein each of the data blocks includes different coordinate values, and the obtained plurality of coordinate values are sorted according to a preset rule to obtain Corresponding two-dimensional index table.
  • the second device needs to decompress the received data packet to obtain one or more data blocks.
  • the second device sorts each of the plurality of data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the second device sorts each of the one or more data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the second device quickly locates the data block by using two-dimensional coordinates (x, y) in the two-dimensional index table, and the header field information of each data block carries a coordinate value, and the content information of each data block is It is possible to load files of different sizes, or offsets of specified positions of large files, to facilitate content verification after data transmission.
  • the block size algorithm and the number of concurrent threads are key elements.
  • the second device automatically combines the data block with other associated data blocks into a large file after receiving the data block.
  • the second device is configured according to the second device.
  • the first indication information carried in the header field information of the data block determines the location of the data block in the large file, and combines the data block and other associated data blocks into a large file for separate storage.
  • the second device automatically splits the data block into a plurality of small files after receiving the data block.
  • the second device is configured according to the header field information of the data block.
  • the carried second indication information determines the location of each small file in the data block, and stores a plurality of small files included in the data block separately.
  • the first device reassembles the target data that needs to be transmitted to obtain a two-dimensional data block pool, and the first device can quickly locate each data block according to the two-dimensional data block pool, and the two-dimensional data block pool Sending to the second device reduces the length of time for data traversal, improves the efficiency of data search, and improves data transmission efficiency.
  • another embodiment of the data transmission method in this embodiment of the present application includes:
  • the first device generates a two-dimensional index table according to the preset multi-thread sequence traversal algorithm and the size of the target data to be transmitted.
  • the first device generates a two-dimensional index table according to the preset multi-thread sequence traversal algorithm and the size of the target data to be transmitted, and the two-dimensional index table is used to indicate the storage location of each data in the target data.
  • the first device first determines the size of the target data that needs to be transmitted to the second device, and then divides the target data into multiple standard-sized data blocks according to the size of the target data, where each data block has the same size;
  • Each data block in the target data is traversed by a preset multi-threaded sequence traversal algorithm to obtain a data directory, the data directory includes at least one layer of the directory;
  • the two-dimensional index is determined according to the size of the target data and the at least one layer of the directory
  • the abscissa and the ordinate of the table wherein the ordinate of the two-dimensional index table is used to indicate the number of groups, and the abscissa of the two-dimensional index table is used to indicate the number of data blocks corresponding to each group; and a two-dimensional index table is generated.
  • the first device configures one or more threads for each layer directory, that is, one thread is allocated for each group, and each thread is used to find a data block of the corresponding group; and the concurrent is determined according to the number of groups and the preset thread pool. The number of threads.
  • the first device performs data recombination on the target data according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the first device performs data recombination on the target data according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the two-dimensional data block pool includes a plurality of data blocks, and each of the data blocks respectively corresponds to a coordinate value of the two-dimensional index table.
  • the first device divides the data that needs to be transmitted into one data block according to the established two-dimensional index table, and reorganizes and sorts each data block to obtain a two-dimensional data block pool corresponding to the two-dimensional index table.
  • Each data block in the two-dimensional data block pool corresponds to one coordinate value in the two-dimensional index table.
  • Steps 501 to 502 are similar to the foregoing steps 401 to 402, and details are not described herein again.
  • the first device generates a check code of each data block.
  • the first device generates a check code for each data block, and the check code is used to verify the correctness of each data block.
  • the generated check code may be a check code generated by using a message digest algorithm (MD5), and the check code is used to verify the consistency of the data content.
  • MD5 message digest algorithm
  • the check code may also be generated by other existing algorithms, which is not limited herein.
  • the first device indexes the multiple data blocks through the two-dimensional index table and sends the data to the second device.
  • the first device indexes to the plurality of data blocks through the two-dimensional index table and transmits to the second device.
  • the first device indexes each data block in the two-dimensional data block pool according to the two-dimensional index table, and performs compression separately, and then sends the compressed data packet corresponding to each data block to the second device through the network.
  • the first device may obtain a corresponding data request, where the data request is used to instruct the first device to send the target data to the second device, where the target
  • the data may include one or more files, which are not described here.
  • the second device performs verification according to the check code carried in each data block, and determines the correctness of each data block.
  • the second device performs verification according to the check code carried in each data block to determine the correctness of each data block.
  • step 507 is performed. If the received one or more data blocks are incorrect or the transmission fails, step 506 is performed.
  • the second device needs to perform a decompression operation on the received compressed data packet to obtain a plurality of data blocks.
  • the second device acquires a retransmission data block sent by the first device.
  • the second device acquires a retransmitted data block that is retransmitted by the first device, where the retransmitted data block is a data block in which a transmission failure or an error occurs and needs to be retransmitted.
  • the second device sends a retransmission request to the first device, where the retransmission request carries a target coordinate value indicating a location of the target data block, and the first device determines, according to the target coordinate value, the target data block in the two-dimensional data block pool.
  • the first device re-acquires the target data block according to the location of the target data block; the first device sends the re-acquired target data block to the second device according to the received retransmission request, and the second device performs the data block again. Verification, if the verification is not correct, continue to retransmit; if the verification is correct, continue with the next steps.
  • the second device generates a corresponding two-dimensional index table according to the plurality of coordinate values.
  • the second device acquires coordinate values included in each data block according to the received data block, wherein each of the data blocks includes different coordinate values, and the obtained plurality of coordinate values are sorted according to a preset rule to obtain Corresponding two-dimensional index table.
  • the second device needs to decompress the received data packet to obtain one or more data blocks.
  • the second device sorts each of the plurality of data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the second device sorts each of the plurality of data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool.
  • the second device quickly locates the data block by using two-dimensional coordinates (x, y) in the two-dimensional index table, and the header field information of each data block carries a coordinate value, and the content information of each data block is It is possible to load files of different sizes, or offsets of specified positions of large files, to facilitate content verification after data transmission.
  • the block size algorithm and the number of concurrent threads are key elements.
  • the second device automatically combines the data block with other associated data blocks into a large file after receiving the data block.
  • the second device is configured according to the second device.
  • the first indication information carried in the header field information of the data block determines the location of the data block in the large file, and combines the data block and other associated data blocks into a large file for separate storage.
  • the second device automatically splits the data block into a plurality of small files after receiving the data block.
  • the second device is configured according to the header field information of the data block.
  • the carried second indication information determines the location of each small file in the data block, and stores a plurality of small files included in the data block separately.
  • Steps 507 to 508 are similar to the above steps 404 to 405, and details are not described herein again.
  • the first device reassembles the target data that needs to be transmitted to obtain a two-dimensional data block pool, and sends the two-dimensional data block pool to the second device, and reverts the data block in which the transmission failure or error occurs.
  • the first device can quickly locate each data block, reducing the length of data traversal time, reducing the time spent on data block retransmission, and improving the efficiency of data repair.
  • an embodiment of the first device in the embodiment of the present application includes:
  • a first generating unit 601 configured to generate a two-dimensional index table according to a preset multi-threaded sequence traversal algorithm and a size of target data that needs to be transmitted, where the two-dimensional index table is used to indicate storage of each data in the target data. position;
  • the reorganization unit 602 is configured to perform data recombination on the target data according to the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, and each data block corresponds to the data block a coordinate value of the two-dimensional index table;
  • the first sending unit 603 is configured to index to the plurality of data blocks by using the two-dimensional index table and send the same to the second device.
  • the first device reassembles the target data that needs to be transmitted to obtain a two-dimensional data block pool, and sends the two-dimensional data block pool to the second device, so that the second device can quickly locate each data.
  • Blocks reduce the length of data traversal and improve the efficiency of data lookup.
  • another embodiment of the first device in this embodiment of the present application includes:
  • a first generating unit 701 configured to generate a two-dimensional index table according to a preset multi-threaded sequence traversal algorithm and a size of target data that needs to be transmitted, where the two-dimensional index table is used to indicate storage of each data in the target data. position;
  • the reorganizing unit 702 is configured to perform data recombination on the target data according to the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes multiple data blocks, and each data block corresponds to the data block a coordinate value of the two-dimensional index table;
  • the first sending unit 703 is configured to index to the plurality of data blocks by using the two-dimensional index table and send the same to the second device.
  • the first generating unit 701 includes:
  • the traversing module 7011 is configured to traverse the target data to be transmitted according to the preset multi-threaded sequence traversal algorithm to obtain a data directory, where the data directory includes at least one layer of the directory;
  • a determining module 7012 configured to determine an abscissa and an ordinate of the two-dimensional index table according to the size of the target data and the at least one layer directory, where the ordinate of the two-dimensional index table is used to indicate the number of packets of the target data, The abscissa of the two-dimensional index table is used to indicate the number of data blocks corresponding to each group;
  • the generating module 7013 is configured to generate a two-dimensional index table.
  • the size of each data block is inversely related to network latency; the number of threads is positively correlated with network bandwidth.
  • the first device further includes:
  • the second generating unit 704 is configured to generate a check code of each data block, where the check code is used to check the correctness of each data block.
  • the data block includes header domain information and content information
  • the header field information includes at least a coordinate value of the data block and the check code, where the coordinate value is used to indicate a location of the data block in the two-dimensional data block pool;
  • the content information includes data content indicated by the header domain information, and the data content is included in the target data.
  • the first device further includes:
  • the receiving unit 705 is configured to receive a retransmission request sent by the second device if the first data block fails to be transmitted, where the retransmission request includes a first coordinate value of the first data block;
  • a determining unit 706, configured to determine, according to the first coordinate value, a location of the first data block in the two-dimensional data block pool;
  • the obtaining unit 707 is configured to reacquire the first data block according to the location of the first data block;
  • the second sending unit 708 is configured to send the re-acquired first data block to the second device.
  • the header information of the target data block further includes first indication information, where The first indication information is used to indicate the location of each target data block in the large file, and constitutes the large file after the transmission is completed.
  • the header information of the target data block further includes second indication information.
  • the second indication information is used to indicate the location of each small file included in the target data block, and is decomposed into individual small files after the transmission is completed.
  • an embodiment of the second device in the embodiment of the present application includes:
  • the first receiving unit 801 is configured to receive a plurality of data blocks sent by the first device, where each data block includes a coordinate value;
  • a generating unit 802 configured to generate a corresponding two-dimensional index table according to the plurality of coordinate values, where the two-dimensional index table is used to indicate a storage location of the plurality of data blocks;
  • the sorting unit 803 is configured to sort each of the plurality of data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool is target data to be transmitted.
  • the second device generates a two-dimensional index table according to the received data block, and sorts the received data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool, and the two-dimensional data block pool
  • the data block corresponding to different coordinate values is included, which reduces the data traversal duration of the second device and improves the efficiency of data search.
  • another embodiment of the first device in the embodiment of the present application includes:
  • the first receiving unit 901 is configured to receive a plurality of data blocks sent by the first device, where each data block includes a coordinate value;
  • the generating unit 902 is configured to generate a corresponding two-dimensional index table according to the plurality of coordinate values, where the two-dimensional index table is used to indicate a storage location of the plurality of data blocks;
  • the sorting unit 903 is configured to sort each of the plurality of data blocks according to the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool is target data to be transmitted.
  • the generating unit 902 includes:
  • the first determining module 9021 is configured to determine an ordinate of the two-dimensional index table according to the second value of the coordinate values, where the ordinate of the two-dimensional index table is used to indicate the number of packets of the target data;
  • a second determining module 9022 configured to determine an abscissa of the two-dimensional index table according to the first value of the coordinate values, where the abscissa of the two-dimensional index table is used to indicate the number of data blocks corresponding to each group;
  • the generating module 9023 is configured to generate a corresponding two-dimensional index table according to the abscissa and the ordinate.
  • the size of each data block is negatively correlated with the network delay
  • the number of threads is positively related to the network bandwidth.
  • the second device further includes:
  • the checking unit 904 is configured to perform verification according to the check code carried in each data block, and determine the correctness of each data block.
  • the data block includes header domain information and content information
  • the header field information includes at least a coordinate value of the data block and the check code, where the coordinate value is used to indicate a location of the data block in the two-dimensional data block pool;
  • the content information includes data content indicated by the header domain information, and the data content is included in the target data.
  • the second device further includes:
  • the obtaining unit 905 is configured to determine a first coordinate value of the first data block if the first data block fails to be transmitted;
  • the sending unit 906 is configured to send a retransmission request to the first device, where the retransmission request includes the first coordinate value;
  • the second receiving unit 907 is configured to receive the first data block that is retransmitted by the first device.
  • FIG. 6 to FIG. 9 respectively describe the first device and the second device in the embodiment of the present application in detail from the perspective of the modular functional entity.
  • the device in the embodiment of the present application is described in detail below from the perspective of hardware processing.
  • FIG. 10 is a schematic structural diagram of a device according to an embodiment of the present disclosure.
  • the device may be a server.
  • the device 1000 may have a large difference due to different configurations or performances, and may include one or more processors (central processing units). , CPU) 1001 (eg, one or more processors) and memory 1009, one or more storage media 1008 that store application 1009 or data 1006 (eg, one or one storage device in Shanghai).
  • the memory 1009 and the storage medium 1008 may be short-term storage or persistent storage.
  • the program stored on the storage medium 1008 may include one or more modules (not shown), each of which may include a series of instruction operations in the device.
  • the processor 1001 can be configured to communicate with the storage medium 1008 on which a series of instruction operations in the storage medium 1008 are performed.
  • Device 1000 may also include one or more power sources 1002, one or more wired or wireless network interfaces 1003, one or more input and output interfaces 1004, and/or one or more operating systems 1005, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that the device structure illustrated in FIG. 10 does not constitute a limitation to the device, and may include more or less components than those illustrated, or a combination of certain components, or different component arrangements.
  • operating systems 1005 such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc.
  • the processor 1001 is a control center of the device and can be processed according to the set data transmission method.
  • the processor 1001 connects various portions of the entire device using various interfaces and lines, performs various functions of the device by running or executing software programs and/or modules stored in the memory 1009, and recalling data stored in the memory 1009. Process data to achieve data transfer.
  • the memory 1009 can be used to store software programs and modules, and the processor 1001 executes various functional applications and data processing of the device 1000 by running software programs and modules stored in the memory 1009.
  • the memory 1009 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function (such as generating a two-dimensional data block pool, etc.), and the storage data area may be stored according to the device. Use the created data (such as generating a two-dimensional index table, etc.).
  • the memory 1009 may include a high speed random access memory, and may also include a nonvolatile memory such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
  • the program of the data transmission method and the received data stream provided in the embodiment of the present application are stored in a memory, and the processor 1001 is called from the memory 1009 when it is needed.
  • FIG. 11 may be a terminal device:
  • FIG. 11 is a block diagram showing a partial structure of a device provided by an embodiment of the present application.
  • the terminal device includes: a radio frequency (RF) circuit 1110, a memory 1120, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WIFI) module 1170, and Processor 1180 and other components.
  • RF radio frequency
  • WIFI wireless fidelity
  • the processor 1180 is a control center of the terminal device.
  • the two-dimensional index table may be generated according to a preset multi-threaded sequence traversal algorithm, and the target data is reconstructed according to the two-dimensional index table to obtain a two-dimensional index.
  • a block pool that sends each block of data in the two-dimensional extent pool to other devices connected to it.
  • the RF circuit 1110 is connected to the processor 1180 via a bus, and is responsible for transmitting data to or receiving data from the Internet. It can also be used for receiving and transmitting signals during the process of transmitting and receiving information, for example, sending a two-dimensional data block pool to the second device. In addition, after receiving the retransmission request sent by the second device, the first device sends a retransmission request to the processor 1180 for processing.
  • the RF circuit 1110 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like. In addition, the RF circuit 1110 can also communicate with the network and other devices via wireless communication.
  • the above wireless communication may use any communication standard or protocol, including but not limited to global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (code division) Multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), e-mail, short messaging service (SMS), and the like.
  • GSM global system of mobile communication
  • GPRS general packet radio service
  • CDMA code division multiple access
  • WCDMA wideband code division multiple access
  • LTE long term evolution
  • SMS short messaging service
  • the memory 1120 can be used to store software programs and modules, and the processor 1180 performs various functional applications and data processing of the terminal devices by running software programs and modules stored in the memory 1120.
  • the memory 1120 can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, an application required by at least one function (for example, generating a two-dimensional index table according to a preset multi-thread sequence traversal algorithm). Etc., etc.; the storage data area can store data created according to the use of the terminal device (such as a two-dimensional data block pool, etc.).
  • the memory 1120 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
  • the disclosed system, apparatus, and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • a computer readable storage medium A number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Power Engineering (AREA)
  • Information Transfer Between Computers (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请公开了一种数据传输方法及相关设备，用于提高目录遍历速度，提高了数据查找和数据修复的效率。方法包括：第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，二维索引表用于指示目标数据中各个数据的存储位置；第一设备根据二维索引表对目标数据进行数据重组，得到二维数据块池，二维数据块池包括多个数据块，每个数据块分别对应二维索引表的一个坐标值；第一设备通过所述二维索引表索引到多个数据块并发送至第二设备。

Description

一种数据传输方法及相关设备
本申请要求于2018年04月23日提交中国国家知识产权局、申请号为201810369971.1、发明名称为“一种数据传输方法及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及通信领域，尤其涉及一种数据传输方法及相关设备。
背景技术
数据传输是一种基础通信服务，大部分计算机服务都需要利用数据传输服务。由于传统的文件传输协议(file transfer protocol，FTP)服务器在广域网中的文件传输效率低，现有的业务系统在广域网中的文件传输速率(300KB/s左右)与业务系统在本地的文件传输速率(6MB/s左右)差距数十倍，广域网的文件传输速率已经不能满足不断增长的业务，广域网数据传输已经成为阻碍业务系统增长的瓶颈，因此对广域网文件传输速率进行性能改进。广域网的数据传输应用都面临着低带宽和连接不稳定、传输批量小文件时存在的负载过重的问题。为了解决该问题，一般通过通道复用机制减少连接次数达到高效批量小文件传输。但是，对于大文件的传输，上、下载超过1G文件时存在的数据拥塞问题，示例性的，如图1所示，为下载1G以上文件产生数据拥塞现象的示意图。
现有方案中，采用线性文件分块解决方案避免拥塞，并采用哈希树(Hash Tree)方式快速识别文件内容是否准确，保证文件内容的准确性，例如，如图2所示，采用安全哈希算法(secure Hash algorithm，SHA)将数据大小为6.5MB的1个大文件数据划分为多个小文件数据，其中，哈希值的大小为256位，具体包括6个1MB大小和1个0.5MB大小的小文件数据。
但是，因为哈希树的规模(哈希树的深度)与文件的大小正相关，文件越大，哈希树的规模越大，需要遍历哈希树找到识别文件块花费的时间越长，目录遍历速度慢，查找效率低，并且当传输出现问题，进行重传的时间长，数据修复效率低。
发明内容
本申请实施例提供了一种数据传输方法及相关设备，用于提高目录遍历速度，提高了数据查找和数据修复的效率。
本申请第一方面提供了一种数据传输方法，包括：第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，该二维索引表用于指示目标数据中各个数据的存储位置，其中，目标数据可以包括一个或多个目录，或者复杂多层目录，或者单个文件，或者多个文件；第一设备根据该二维索引表对目标数据进行数据重组，得到二维数据块池，该二维数据块池包括多个数据块，每个数据块分别对应二维索引表的一个坐标值，每一个数据块在二维索引表中的坐标值都不同；第一设备通过二维 索引表索引到二维数据块池中的多个数据块并发送至第二设备。第一设备将需要进行传输的目标数据重组得到二维数据块池，第一设备能够根据二维数据块池快速定位到每一个数据块，并将该二维数据块池发送至第二设备，减少了数据遍历花费的时长，提高了数据查找的效率，进而提高了数据传输效率。
在一种可能的设计中，在本申请实施例第一方面的第一种实现方式中，第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表包括：第一设备根据预置的多线程层序遍历算法对需要传输的目标数据进行遍历，得到数据目录，该数据目录包括至少一层目录；第一设备根据目标数据的大小和至少一层目录确定二维索引表的横坐标和纵坐标，该二维索引表的纵坐标用于指示目标数据的分组数量，该二维索引表的横坐标用于指示每个分组对应的数据块数量；第一设备生成二维索引表，该二维索引表的纵坐标用于指示分组数量，该二维索引表的横坐标用于指示每个分组对应的数据块数量。对第一设备生成二维索引表的过程进行了细化，提供了具体的实现方式，增加了本申请的实现方式。
在一种可能的设计中，在本申请实施例第一方面的第二种实现方式中，每个数据块的大小与网络延迟负相关；线程数量与网络带宽正相关。对数据块的大小和线程数量进行了限定，明确数据块与网络延迟的关系，以及线程数量与网络带宽的关系，可以快速的设置数据块的大小和线程数量，提高了数据传输效率。
在一种可能的设计中，在本申请实施例第一方面的第三种实现方式中，在第一设备根据二维索引表对目标数据进行数据重组之后，在第一设备通过所述二维索引表索引到所述多个数据块并发送至第二设备之前，所述方法还包括：第一设备生成每个数据块的校验码，校验码用于校验每个数据块的正确性。增加了生成校验码的过程，确保了传输的数据块的准确性。
在一种可能的设计中，在本申请实施例第一方面的第四种实现方式中，数据块包括头域信息和内容信息；头域信息中至少包括数据块的坐标值和校验码，该坐标值用于指示数据块在二维数据块池中的位置；内容信息中包括头域信息指示的数据内容，数据内容包含于目标数据。对数据块的结构进行了限定，增加了本申请实施例的实现方式。
在一种可能的设计中，在本申请实施例第一方面的第五种实现方式中，包括：若第一数据块传输失败，则第一设备接收第二设备发送的重传请求，该重传请求中包括第一数据块的第一坐标值；第一设备根据第一坐标值确定第一数据块在二维数据块池中的位置；第一设备根据第一数据块的位置重新获取第一数据块；第一设备向第二设备发送重新获取的第一数据块。增加了对传输失败的数据块进行修复的过程，只针对传输失败的数据块进行重传，减少了进行数据修复的时间，提高了数据修复效率。
在一种可能的设计中，在本申请实施例第一方面的第六种实现方式中，当目标数据包含大文件，大文件由多个目标数据块组成时，目标数据块的头域信息中还包括第一指示信息，第一指示信息用于指示各个目标数据块在大文件中的位置，并在完成传输后组成大文件。增加了指示大文件组成的第一指示信息，以使得第二设备在接收到数据块后，根据第一指示信息将多个数据块还原成大文件。
在一种可能的设计中，在本申请实施例第一方面的第七种实现方式中，当目标数据包含多个小文件，多个小文件组成目标数据块时，目标数据块的头域信息中还包括第二 指示信息，第二指示信息用于指示目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。增加了指示小文件组成的第二指示信息，以使得第二设备在接收到数据块后，根据第二指示信息将数据块还原成多个小文件。
在一种可能的设计中，在本申请实施例第一方面的第八种实现方式中，每个数据块的大小相同。对每个数据块的大小进行限定，得到的二维数据块池的结构更合理，便于进行数据查找，提高数据查找效率。
在一种可能的设计中，在本申请实施例第一方面的第九种实现方式中，数据内容为一个或多个目标文件；或，数据内容为目标文件的部分内容。对每个数据块中包含的数据内容进行限定，可以将多个小文件一起进行传输，提高了数据传输的效率。
在一种可能的设计中，在本申请实施例第一方面的第十种实现方式中，头域信息中还包括目标路径、源文件名、目标文件名、文件偏移地址和数据块大小。对数据块的头域信息中包括的内容进行了限定，使本申请在步骤上更完善。
本申请第二方面提供了一种数据传输方法，包括：第二设备接收第一设备发送的多个数据块，每个数据块中包含一个坐标值；第二设备根据多个坐标值生成对应的二维索引表，该二维索引表用于指示多个数据块的存储位置；第二设备根据二维索引表对多个数据块中的每个数据块进行排序，得到二维数据块池，该二维数据块池中各个数据块中包含的数据内容组成了需要的目标数据。第二设备根据接收到的多个数据块中包含的坐标值，并根据各个坐标值生成二维索引表，并将需要进行传输的目标数据重组得到二维数据块池，第二设备能够根据二维数据块池快速定位到每一个数据块，并对该二维数据块池中的数据块查找，减少了数据遍历花费的时长，提高了数据查找的效率，进而提高了数据传输效率。
在一种可能的设计中，在本申请实施例第二方面的第一种实现方式中，第二设备根据坐标值生成对应的二维索引表包括：第二设备根据坐标值中的第二个值确定二维索引表的纵坐标，该二维索引表的纵坐标用于指示目标数据的分组数量；第二设备根据坐标值中的第一个值确定二维索引表的横坐标，该二维索引表的横坐标用于指示每个分组对应的数据块数量；第二设备根据横坐标和纵坐标生成对应的二维索引表。对第一设备生成二维索引表的过程进行了细化，提供了具体的实现方式，增加了本申请的实现方式。
在一种可能的设计中，在本申请实施例第二方面的第二种实现方式中，每个数据块的大小与网络延迟负相关；线程数量与网络带宽正相关。对数据块的大小和线程数量进行了限定，明确数据块与网络延迟的关系，以及线程数量与网络带宽的关系，可以快速的设置数据块的大小和线程数量，提高了数据传输效率。
在一种可能的设计中，在本申请实施例第二方面的第三种实现方式中，第二设备接收第一设备发送的多个数据块之后，第二设备根据多个坐标值生成对应的二维索引表之前，所述方法还包括：第二设备根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。增加了生成校验码的过程，确保了传输的数据块的准确性。
在一种可能的设计中，在本申请实施例第二方面的第四种实现方式中，数据块包括头域信息和内容信息；头域信息中至少包括数据块的坐标值和校验码，该坐标值用于指示数据块在二维数据块池中的位置；内容信息中包括头域信息指示的数据内容，该数据内容包含于目标数据。对数据块的结构进行了限定，增加了本申请实施例的实现方式。
在一种可能的设计中，在本申请实施例第二方面的第五种实现方式中，包括：若第一数据块传输失败，则第二设备确定第一数据块的第一坐标值；第二设备向第一设备发送重传请求，该重传请求中包括所述第一坐标值；第二设备接收所述第一设备重新发送的第一数据块。增加了对传输失败的数据块进行修复的过程，只针对传输失败的数据块进行重传，减少了进行数据修复的时间，提高了数据修复效率。
在一种可能的设计中，在本申请实施例第二方面的第六种实现方式中，当目标数据包含大文件，大文件由多个目标数据块组成时，目标数据块的头域信息中还包括第一指示信息，第一指示信息用于指示各个目标数据块在大文件中的位置，并在完成传输后组成大文件。增加了指示大文件组成的第一指示信息，以使得第二设备在接收到数据块后，根据第一指示信息将多个数据块还原成大文件。
在一种可能的设计中，在本申请实施例第二方面的第七种实现方式中，当目标数据包含多个小文件，多个小文件组成目标数据块时，目标数据块的头域信息中还包括第二指示信息，第二指示信息用于指示目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。增加了指示小文件组成的第二指示信息，以使得第二设备在接收到数据块后，根据第二指示信息将数据块还原成多个小文件。
在一种可能的设计中，在本申请实施例第二方面的第八种实现方式中，每个数据块的大小相同。对每个数据块的大小进行限定，得到的二维数据块池的结构更合理，便于进行数据查找，提高数据查找效率。
在一种可能的设计中，在本申请实施例第二方面的第九种实现方式中，数据内容为一个或多个目标文件；或，数据内容为目标文件的部分内容。对每个数据块中包含的数据内容进行限定，可以将多个小文件一起进行传输，提高了数据传输的效率。
在一种可能的设计中，在本申请实施例第二方面的第十种实现方式中，头域信息中还包括目标路径、源文件名、目标文件名、文件偏移地址和数据块大小。对数据块的头域信息中包括的内容进行了限定，使本申请在步骤上更完善。
本申请第三方面提供了一种设备，所述设备为第一设备，包括：第一生成单元，用于根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，所述二维索引表用于指示所述目标数据中各个数据的存储位置，其中，目标数据可以包括一个或多个目录，或者复杂多层目录，或者单个文件，或者多个文件；重组单元，用于根据二维索引表对目标数据进行数据重组，得到二维数据块池，该二维数据块池包括多个数据块，每个数据块分别对应二维索引表的一个坐标值；第一发送单元，用于通过所述二维索引表索引到所述多个数据块并发送至第二设备。第一设备将需要进行传输的目标数据重组得到二维数据块池，第一设备能够根据二维数据块池快速定位到每一个数据块，并将该二维数据块池发送至第二设备，减少了数据遍历花费的时长，提高了数据查找的效率，进而提高了数据传输效率。
在一种可能的设计中，在本申请实施例第三方面的第一种实现方式中，所述第一生成单元包括：遍历模块，用于根据预置的多线程层序遍历算法对需要传输的目标数据进行遍历，得到数据目录，所述数据目录包括至少一层目录；确定模块，用于根据目标数据的大小和至少一层目录确定二维索引表的横坐标和纵坐标，该二维索引表的纵坐标用于指示目标数据的分组数量，该二维索引表的横坐标用于指示每个分组对应的数据块数 量；生成模块，用于生成二维索引表。对第一设备生成二维索引表的过程进行了细化，提供了具体的实现方式，增加了本申请的实现方式。
在一种可能的设计中，在本申请实施例第三方面的第二种实现方式中，每个数据块的大小与网络延迟负相关；线程数量与网络带宽正相关。对数据块的大小和线程数量进行了限定，明确数据块与网络延迟的关系，以及线程数量与网络带宽的关系，可以快速的设置数据块的大小和线程数量，提高了数据传输效率。
在一种可能的设计中，在本申请实施例第三方面的第三种实现方式中，所述设备还包括：第二生成单元，用于生成每个数据块的校验码，该校验码用于校验每个数据块的正确性。增加了生成校验码的过程，确保了传输的数据块的准确性。
在一种可能的设计中，在本申请实施例第三方面的第四种实现方式中，数据块包括头域信息和内容信息；头域信息中至少包括数据块的坐标值和校验码，坐标值用于指示数据块在二维数据块池中的位置；内容信息中包括头域信息指示的数据内容，数据内容包含于目标数据。对数据块的结构进行了限定，增加了本申请实施例的实现方式。
在一种可能的设计中，在本申请实施例第三方面的第五种实现方式中，包括：接收单元，若第一数据块传输失败，则用于接收所述第二设备发送的重传请求，所述重传请求中包括所述第一数据块的第一坐标值；确定单元，用于根据所述第一坐标值确定所述第一数据块在所述二维数据块池中的位置；获取单元，用于根据所述第一数据块的位置重新获取第一数据块；第二发送单元，用于向所述第二设备发送重新获取的第一数据块。增加了对传输失败的数据进行修复的过程，只针对传输失败的数据块进行重传，减少了进行数据修复的时间，提高了数据修复效率。
在一种可能的设计中，在本申请实施例第三方面的第六种实现方式中，当目标数据包含大文件，大文件由多个目标数据块组成时，目标数据块的头域信息中还包括第一指示信息，第一指示信息用于指示各个目标数据块在大文件中的位置，并在完成传输后组成大文件。增加了指示大文件组成的第一指示信息，以使得第二设备在接收到数据块后，根据第一指示信息将多个数据块还原成大文件。
在一种可能的设计中，在本申请实施例第三方面的第七种实现方式中，当目标数据包含多个小文件，多个小文件组成目标数据块时，目标数据块的头域信息中还包括第二指示信息，第二指示信息用于指示目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。增加了指示小文件组成的第二指示信息，以使得第二设备在接收到数据块后，根据第二指示信息将数据块还原成多个小文件。
在一种可能的设计中，在本申请实施例第三方面的第八种实现方式中，每个数据块的大小相同。对每个数据块的大小进行限定，得到的二维数据块池的结构更合理，便于进行数据查找，提高数据查找效率。
在一种可能的设计中，在本申请实施例第三方面的第九种实现方式中，数据内容为一个或多个目标文件；或，数据内容为目标文件的部分内容。对每个数据块中包含的数据内容进行限定，可以将多个小文件一起进行传输，提高了数据传输的效率。
在一种可能的设计中，在本申请实施例第三方面的第十种实现方式中，头域信息中还包括目标路径、源文件名、目标文件名、文件偏移地址和数据块大小。对数据块的头域信息中包括的内容进行了限定，使本申请在步骤上更完善。
本申请第四方面提供了一种设备，所述设备为第二设备，包括：第一接收单元，用于接收第一设备发送的多个数据块，每个数据块中包含一个坐标值；生成单元，用于根据多个坐标值生成对应的二维索引表，所述二维索引表用于指示所述多个数据块的存储位置；排序单元，用于根据二维索引表对多个数据块中的每个数据块进行排序，得到二维数据块池，所述二维数据块池为需要传输的目标数据。第二设备根据接收到的多个数据块中包含的坐标值，并根据各个坐标值生成二维索引表，并将需要进行传输的目标数据重组得到二维数据块池，第二设备能够根据二维数据块池快速定位到每一个数据块，并对该二维数据块池中的数据块查找，减少了数据遍历花费的时长，提高了数据查找的效率，进而提高了数据传输效率。
在一种可能的设计中，在本申请实施例第四方面的第一种实现方式中，生成单元包括：第一确定模块，用于根据坐标值中的第二个值确定二维索引表的纵坐标，该二维索引表的纵坐标用于指示目标数据的分组数量；第二确定模块，用于根据所述坐标值中的第一个值确定二维索引表的横坐标，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；生成模块，用于根据横坐标和纵坐标生成对应的二维索引表。对第一设备生成二维索引表的过程进行了细化，提供了具体的实现方式，增加了本申请的实现方式。
在一种可能的设计中，在本申请实施例第四方面的第二种实现方式中，每个数据块的大小与网络延迟负相关；线程数量与网络带宽正相关。对数据块的大小和线程数量进行了限定，明确数据块与网络延迟的关系，以及线程数量与网络带宽的关系，可以快速的设置数据块的大小和线程数量，提高了数据传输效率。
在一种可能的设计中，在本申请实施例第四方面的第三种实现方式中，所述设备还包括：校验单元，用于根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。增加了生成校验码的过程，确保了传输的数据块的准确性。
在一种可能的设计中，在本申请实施例第四方面的第四种实现方式中，数据块包括头域信息和内容信息；头域信息中至少包括数据块的坐标值和校验码，该坐标值用于指示数据块在二维数据块池中的位置；内容信息中包括头域信息指示的数据内容，该数据内容包含于目标数据。对数据块的结构进行了限定，增加了本申请实施例的实现方式。
在一种可能的设计中，在本申请实施例第四方面的第五种实现方式中，包括：获取单元，若第一数据块传输失败，则用于确定所述第一数据块的第一坐标值；发送单元，用于向所述第一设备发送重传请求，所述重传请求中包括所述第一坐标值；第二接收单元，用于接收所述第一设备重新发送的第一数据块。增加了对传输失败的数据进行修复的过程，只针对传输失败的数据块进行重传，减少了进行数据修复的时间，提高了数据修复效率。
在一种可能的设计中，在本申请实施例第四方面的第六种实现方式中，当目标数据包含大文件，大文件由多个目标数据块组成时，目标数据块的头域信息中还包括第一指示信息，第一指示信息用于指示各个目标数据块在大文件中的位置，并在完成传输后组成大文件。增加了指示大文件组成的第一指示信息，以使得第二设备在接收到数据块后，根据第一指示信息将多个数据块还原成大文件。
在一种可能的设计中，在本申请实施例第五方面的第七种实现方式中，当目标数据包含多个小文件，多个小文件组成目标数据块时，目标数据块的头域信息中还包括第二 指示信息，第二指示信息用于指示目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。增加了指示小文件组成的第二指示信息，以使得第二设备在接收到数据块后，根据第二指示信息将数据块还原成多个小文件。
在一种可能的设计中，在本申请实施例第四方面的第八种实现方式中，每个数据块的大小相同。对每个数据块的大小进行限定，得到的二维数据块池的结构更合理，便于进行数据查找，提高数据查找效率。
在一种可能的设计中，在本申请实施例第四方面的第九种实现方式中，数据内容为一个或多个目标文件；或，数据内容为目标文件的部分内容。对每个数据块中包含的数据内容进行限定，可以将多个小文件一起进行传输，提高了数据传输的效率。
在一种可能的设计中，在本申请实施例第四方面的第十种实现方式中，头域信息中还包括目标路径、源文件名、目标文件名、文件偏移地址和数据块大小。对数据块的头域信息中包括的内容进行了限定，使本申请在步骤上更完善。
本申请的第五方面提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
本申请的第六方面提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
附图说明
图1为数据下载过程中下载速率和文件大小的关系示意图；
图2为通过SHA-256算法得到的哈希树的一个示意图；
图3为本申请实施例应用的网络架构的示意图；
图4A为本申请实施例中数据传输方法的一个实施例示意图；
图4B为本申请实施例中采用多线程层序遍历算法进行数据遍历的一个示意图；
图4C为本申请实施例中生成的二维索引表的一个示意图；
图4D为本申请实施例中二维数据块池中数据块和文件的关系示意图；
图5为本申请实施例中数据传输方法的另一个实施例示意图；
图6为本申请实施例中第一设备的一个实施例示意图；
图7为本申请实施例中第一设备的另一个实施例示意图；
图8为本申请实施例中第二设备的一个实施例示意图；
图9为本申请实施例中第二设备的另一个实施例示意图；
图10为本申请实施例中设备的另一个实施例示意图；
图11为本申请实施例中设备的另一个实施例示意图。
具体实施方式
本申请实施例提供了一种数据传输方法及相关设备，用于提高目录遍历速度，提高了数据查找和数据修复的效率。
为了使本技术领域的人员更好地理解本申请方案，下面将结合本申请实施例中的附图，对本申请实施例进行描述。
本申请文件中提及的“第一”或“第二”等是用于区别类似的对象，而不必用于描 述特定的顺序或先后次序。此外，本申请文件中提及的“包括”或“具有”及其任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
目前已成功在广域网传输带宽为20MB/s条件下，传输2G大文件在2MB/s以上。例如，在A地到B地的广域网传输过程中，实际传输带宽利用率达80％以上。为了确保广域网对大文件的传输过程的准确性，现有方案中采用线性文件分块的解决方案，采用哈希树的方式保证文件内容的准确性，但是该方案中随着文件规模的增大，进行遍历所需的时长越长，耗费的硬件资源越多。
本申请可以应在如图3所示的网络框架中，在该网络框架中，第一设备作为源设备，第二设备作为目标设备，第一设备生成二维索引表，通过索引二维索引表中各个坐标对应数据并加载，将加载的文件数据传输至第二设备，从而实现传输文件从源设备到目标设备的全流程处理，减少遍历所需的时长，提高传输效率。
本申请基于RFC2616采用标准的HTTP1.1协议，还可以扩展到云存储、网盘、移动传输等应用场景，具体此处不做限定。可以理解的是，本身申请是基于云或服务器集群的跨区域的大数据量的可靠传输，本申请中的涉及的第一设备和第二设备可以是单独的一个设备，也可以是多个设备构成的设备集群，为了便于描述，本申请中仅以单个设备为例进行说明。
为便于理解，下面对本申请实施例的具体流程进行描述，请参阅图4A，本申请实施例中数据传输方法的一个实施例包括：
401、第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表。
第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，二维索引表用于指示目标数据中各个数据的存储位置。
具体的，第一设备先确定需要传输到第二设备的目标数据的大小，再根据目标数据的大小将目标数据划分为多个标准大小的数据块，其中，每个数据块的大小都相同；通过预置的多线程层序遍历算法，将目标数据中的每个数据块进行遍历，得到数据目录，该数据目录包括至少一层目录；根据目标数据的大小和至少一层目录确定二维索引表的横坐标和纵坐标，其中，该二维索引表的纵坐标用于指示分组数量，该二维索引表的横坐标用于指示每个分组对应的数据块数量；生成二维索引表。其中，第一设备为每一层目录配置一个或多个线程，即为每个分组对应分配一个线程，每个线程用于查找对应分组的数据块；根据分组数量及预置的线程池确定并发线程的数量。
需要说明的是，预置的多线程层序遍历算法是对现有的层序遍历算法的改进，引入多个线程，可以同时对多个数据块进行遍历，提高遍历效率。目标数据可以包括一个或多个目录，或者复杂多次目录。
举例说明，如图4B所示，该数据目录中可以包括4层目录(虚线框所示)，分别为第一层目录、第二层目录、第三层目录、第四层目录，每一层目录都对应有一个或多个数据块，每一个数据块中包含有具体的数据内容。为每一层目录配置线程(实线框所示)，其中，第一层目录配置一个线程，第二层目录配置六个线程，第三层目录配置六个线程， 第四层目录配置三个线程，每一个线程用于查找对应分组的数据块的数据内容；根据目标数据的大小及预置的线程池，确定用于查找分组中数据块的并发线程的数量，预置的线程池的大小决定最大的线程数量；生成二维索引表，对二维索引表的横坐标和纵坐标进行标识，例如，如图4C所示，生成的二维索引表的纵坐标对应有9个分组，用字母ABCDEFGHI进行标识，每一个分组对应的数据块数量为9，用数字123456789进行标识，横坐标和纵坐标还可以是其他方式进行标识，例如，横坐标用字母标识，纵坐标用数字进行标识，具体此处不做限定。可以理解的是，同一层目录可能包含多个子目录，为树状结构(图4B中未示出)，具体此处不做限定。
402、第一设备根据二维索引表对目标数据进行数据重组，得到二维数据块池。
第一设备根据二维索引表对目标数据进行数据重组，得到二维数据块池，该二维数据块池包括多个数据块，每个数据块分别对应该二维索引表的一个坐标值。
具体的，第一设备根据建立的二维索引表对需要进行传输的数据划分成一个个的数据块，并对各个数据块进行重组排序，得到对应与二维索引表的二维数据块池，二维数据块池中的每一个数据块都对应二维索引表中的一个坐标值，例如，在图4C所示的二维索引表中，第一个数据块的坐标值为(1，A)，对应的数据大小可以为100MB。
每一个数据块由头域信息和内容信息两部分构成；其中，头域信息中至少包括该数据块的坐标值和校验码，该坐标值用于指示该数据块在二维数据块池中的位置，校验码用于校验该数据块的正确性；内容信息中包括头域信息指示的数据内容，该数据内容包含于目标数据。该头域信息中还可以包括目标路径、源文件名、目标文件名、文件偏移地址和数据块大小等相关信息。
需要说明的是，为了避免网络传输发生拥堵，在对每个数据块进行划分时需要参考当前的网络延迟，每个数据块的大小与网络延迟负相关，具体的，如表1所示，网络延迟越大，则数据块的大小越小，以使得传输速率更快，减少传输拥堵情况，例如，网络延迟和数据块的大小可以满足关系式：数据块大小＝200/网络延迟，还可以是其他对应关系，具体此处不做限定。
表1：
网络延迟(ms) 400 200 100 50 20
数据块大小(MB) 0.5 1 2 4 10
为了提高网络的整体传输效率，在不发生拥堵的前提下，为尽量提高单次的传输速率，可以提高并发线程的数量来提高传输效率，并发的线程数量与网络带宽正相关，网络带宽越大，则并发的线程数量越多，并且并发的线程数量也与预置的线程池的大小相关，线程池越大，则并发的线程数量越多。
可以理解的是，可以是多个数据块的数据内容共同构成一个大文件，还可以是一个数据块构成一个标准文件，还可以是多个小文件构成一个数据块，其中，标准文件是与设定的数据块大小相同的文件，例如，1MB大小的文件；大文件是大于设定的数据块大小的文件，例如，1G大小的文件；小文件是指小于设定的数据块大小的文件，例如，100KB大小的文件。如图4D所示，26个数据块构成文件1(411)，1个数据块构成文件2(412)，1个数据块构成文件3(413)，1个数据块中包括文件4(414)和文件5(415)。
403、第一设备通过二维索引表索引到多个数据块并发送至第二设备。
第一设备通过二维索引表索引到多个数据块并发送至第二设备。
具体的，第一设备根据二维索引表索引到二维数据块池中的每一个数据块，并分别进行压缩，再通过网络将每一个数据块对应的压缩数据包发送至第二设备。
需要说明的是，在第一设备向第二设备发送数据包之前，第一设备可以获取到对应的数据请求，该数据请求用于指示第一设备将目标数据发送至第二设备，其中，目标数据可以包括一个或多个文件，具体此处不再赘述。
404、第二设备根据多个坐标值生成对应的二维索引表。
第二设备根据接收到的数据块获取每一个数据块中包含的坐标值，其中，每一个数据块中包含的坐标值都不同，将获取到的多个坐标值按照预置规则进行排序，得到对应的二维索引表。
需要说明的是，在生成二维索引表之前，第二设备需要对接收到的数据包进行解压，得到一个或多个数据块。
405、第二设备根据二维索引表对多个数据块中的每个数据块进行排序，得到二维数据块池。
第二设备根据二维索引表对一个或多个数据块中的每个数据块进行排序，得到二维数据块池。
具体的，第二设备通过二维索引表中的二维坐标(x，y)快速定位该数据块，每一个数据块的头域信息中携带有一个坐标值，每一个数据块的内容信息中可以装载不同大小的文件，或者大文件的指定位置偏移量的内容，方便数据传输后的内容校验。其中，数据块大小算法和并发线程数设置是关键要素。
需要说明的是，若一个数据块为大文件的一部分，则第二设备在接收到该数据块后会自动将该数据块与其他关联的数据块组合成大文件，具体的，第二设备根据该数据块的头域信息中携带的第一指示信息确定该数据块在大文件中的位置，将该数据块和其他关联的数据块组合成大文件单独存储。
若一个数据块中包括多个小文件，则第二设备在接收到该数据块后会自动将该数据块分解成多个小文件，具体的，第二设备根据该数据块的头域信息中携带的第二指示信息确定该数据块中各个小文件的位置，将该数据块中包含的多个小文件单独存储。
本申请实施例中，第一设备将需要进行传输的目标数据重组得到二维数据块池，第一设备能够根据二维数据块池快速定位到每一个数据块，并将该二维数据块池发送至第二设备，减少了数据遍历花费的时长，提高了数据查找的效率，进而提高了数据传输效率。
请参阅图5，本申请实施例中数据传输方法的另一实施例包括：
501、第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表。
第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，二维索引表用于指示目标数据中各个数据的存储位置。
具体的，第一设备先确定需要传输到第二设备的目标数据的大小，再根据目标数据的大小将目标数据划分为多个标准大小的数据块，其中，每个数据块的大小都相同；通过预置的多线程层序遍历算法，将目标数据中的每个数据块进行遍历，得到数据目录， 该数据目录包括至少一层目录；根据目标数据的大小和至少一层目录确定二维索引表的横坐标和纵坐标，其中，该二维索引表的纵坐标用于指示分组数量，该二维索引表的横坐标用于指示每个分组对应的数据块数量；生成二维索引表。其中，第一设备为每一层目录配置一个或多个线程，即为每个分组对应分配一个线程，每个线程用于查找对应分组的数据块；根据分组数量及预置的线程池确定并发线程的数量。
502、第一设备根据二维索引表对目标数据进行数据重组，得到二维数据块池。
第一设备根据二维索引表对目标数据进行数据重组，得到二维数据块池，该二维数据块池包括多个数据块，每个数据块分别对应该二维索引表的一个坐标值。
具体的，第一设备根据建立的二维索引表对需要进行传输的数据划分成一个个的数据块，并对各个数据块进行重组排序，得到对应与二维索引表的二维数据块池，二维数据块池中的每一个数据块都对应二维索引表中的一个坐标值。
步骤501至步骤502与上述步骤401至402类似，具体此处不再赘述。
503、第一设备生成每个数据块的校验码。
第一设备生成每个数据块的校验码，校验码用于校验每个数据块的正确性。
具体的，本申请实施例中，生成的校验码可以为利用消息摘要算法(message digest algorithm 5，MD5)生成的校验码，该校验码用于对数据内容的一致性进行校验，校验码还可以是由其他现有算法生成，具体此处不做限定。
504、第一设备通过二维索引表索引到多个数据块并发送至第二设备。
第一设备通过二维索引表索引到多个数据块并发送至第二设备。
具体的，第一设备根据二维索引表索引到二维数据块池中的每一个数据块，并分别进行压缩，再通过网络将每一个数据块对应的压缩数据包发送至第二设备。
需要说明的是，在第一设备向第二设备发送数据包之前，第一设备可以获取到对应的数据请求，该数据请求用于指示第一设备将目标数据发送至第二设备，其中，目标数据可以包括一个或多个文件，具体此处不再赘述。
505、第二设备根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。
第二设备根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。当接收到的数据块都是正确的，则执行步骤507，若接收到的一个或多个数据块不正确或传输失败，则执行步骤506。
需要说明的是，第二设备在进行数据校验之前，还需要对接收到的压缩数据包进行解压操作，以得到多个数据块。
506、第二设备获取第一设备发送的重传数据块。
第二设备获取第一设备重传的重传数据块，该重传数据块为发生传输失败或错误并需要重传的数据块。
具体的，第二设备向第一设备发送重传请求，该重传请求中携带有指示目标数据块位置的目标坐标值，第一设备根据目标坐标值确定目标数据块在二维数据块池中的位置；第一设备根据目标数据块的位置重新获取目标数据块；第一设备根据接到的重传请求将重新获取的目标数据块发送至第二设备，第二设备再对该数据块进行校验，若校验不正确，则继续重传；若检验正确，则继续执行后续步骤。
需要说明的是，在现有技术中，若传输过程发生中断或异常，即某一个数据发生传输错误或失败，则从异常的地方开始续传或从头开始进行传输，耗费时间，即数据修复的时间长，修复效率低；本申请实施例中，若某一个或多个数据块发生传输错误或失败，则只需将发生传输错误或失败的数据块进行重传，不需要对其他数据块进行重传，耗费的时间短，数据修复效率高。
507、第二设备根据多个坐标值生成对应的二维索引表。
第二设备根据接收到的数据块获取每一个数据块中包含的坐标值，其中，每一个数据块中包含的坐标值都不同，将获取到的多个坐标值按照预置规则进行排序，得到对应的二维索引表。
需要说明的是，在生成二维索引表之前，第二设备需要对接收到的数据包进行解压，得到一个或多个数据块。
508、第二设备根据二维索引表对多个数据块中的每个数据块进行排序，得到二维数据块池。
第二设备根据二维索引表对多个数据块中的每个数据块进行排序，得到二维数据块池。
具体的，第二设备通过二维索引表中的二维坐标(x，y)快速定位该数据块，每一个数据块的头域信息中携带有一个坐标值，每一个数据块的内容信息中可以装载不同大小的文件，或者大文件的指定位置偏移量的内容，方便数据传输后的内容校验。其中，数据块大小算法和并发线程数设置是关键要素。
需要说明的是，若一个数据块为大文件的一部分，则第二设备在接收到该数据块后会自动将该数据块与其他关联的数据块组合成大文件，具体的，第二设备根据该数据块的头域信息中携带的第一指示信息确定该数据块在大文件中的位置，将该数据块和其他关联的数据块组合成大文件单独存储。
若一个数据块中包括多个小文件，则第二设备在接收到该数据块后会自动将该数据块分解成多个小文件，具体的，第二设备根据该数据块的头域信息中携带的第二指示信息确定该数据块中各个小文件的位置，将该数据块中包含的多个小文件单独存储。
步骤507至步骤508与上述步骤404至405类似，具体此处不再赘述。
本申请实施例中，第一设备将需要进行传输的目标数据重组得到二维数据块池，并将该二维数据块池发送至第二设备，并将发生传输失败或错误的数据块冲重新发送至第二设备，第一设备能够快速定位到每一个数据块，在减少数据遍历时长的同时，减少了数据块重传花费的时长，提高了数据修复的效率。
上面对本申请实施例中数据传输方法进行了描述，下面对本申请实施例中设备进行描述，请参阅图6，本申请实施例中第一设备的一个实施例包括：
第一生成单元601，用于根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，所述二维索引表用于指示所述目标数据中各个数据的存储位置；
重组单元602，用于根据所述二维索引表对所述目标数据进行数据重组，得到二维数据块池，所述二维数据块池包括多个数据块，每个数据块分别对应所述二维索引表的一个坐标值；
第一发送单元603，用于通过所述二维索引表索引到所述多个数据块并发送至第二 设备。
本申请实施例中，第一设备将需要进行传输的目标数据重组得到二维数据块池，并将该二维数据块池发送至第二设备，以使得第二设备能够快速定位到每一个数据块，减少了数据遍历时长，提高了数据查找的效率。
请参阅图7，本申请实施例中第一设备的另一个实施例包括：
第一生成单元701，用于根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，所述二维索引表用于指示所述目标数据中各个数据的存储位置；
重组单元702，用于根据所述二维索引表对所述目标数据进行数据重组，得到二维数据块池，所述二维数据块池包括多个数据块，每个数据块分别对应所述二维索引表的一个坐标值；
第一发送单元703，用于通过所述二维索引表索引到所述多个数据块并发送至第二设备。
在一种可行的实施方式中，第一生成单元701包括：
遍历模块7011，用于根据预置的多线程层序遍历算法对需要传输的目标数据进行遍历，得到数据目录，所述数据目录包括至少一层目录；
确定模块7012，用于根据所述目标数据的大小和所述至少一层目录确定二维索引表的横坐标和纵坐标，所述二维索引表的纵坐标用于指示目标数据的分组数量，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；
生成模块7013，用于生成二维索引表。
在一种可行的实施方式中，每个数据块的大小与网络延迟负相关；线程数量与网络带宽正相关。
在一种可行的实施方式中，第一设备还包括：
第二生成单元704，用于生成每个数据块的校验码，所述校验码用于校验每个数据块的正确性。
在一种可行的实施方式中，数据块包括头域信息和内容信息；
所述头域信息中至少包括所述数据块的坐标值和所述校验码，所述坐标值用于指示所述数据块在所述二维数据块池中的位置；
所述内容信息中包括所述头域信息指示的数据内容，所述数据内容包含于所述目标数据。
在一种可行的实施方式中，所第一设备还包括：
接收单元705，若第一数据块传输失败，则用于接收所述第二设备发送的重传请求，所述重传请求中包括所述第一数据块的第一坐标值；
确定单元706，用于根据所述第一坐标值确定所述第一数据块在所述二维数据块池中的位置；
获取单元707，用于根据所述第一数据块的位置重新获取第一数据块；
第二发送单元708，用于向所述第二设备发送重新获取的第一数据块。
在一种可行的实施方式中，当所述目标数据包含大文件，所述大文件由多个目标数据块组成时，所述目标数据块的头域信息中还包括第一指示信息，所述第一指示信息用于指示各个目标数据块在所述大文件中的位置，并在完成传输后组成所述大文件。
在一种可行的实施方式中，当所述目标数据包含多个小文件，所述多个小文件组成目标数据块时，所述目标数据块的头域信息中还包括第二指示信息，所述第二指示信息用于指示所述目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。
请参阅图8，本申请实施例中第二设备的一个实施例包括：
第一接收单元801，用于接收第一设备发送的多个数据块，每个数据块中包含一个坐标值；
生成单元802，用于根据多个坐标值生成对应的二维索引表，所述二维索引表用于指示所述多个数据块的存储位置；
排序单元803，用于根据所述二维索引表对所述多个数据块中的每个数据块进行排序，得到二维数据块池，所述二维数据块池为需要传输的目标数据。
本申请实施例中，第二设备根据接收到的数据块生成二维索引表，并根据该二维索引表对接收到的数据块进行排序，得到二维数据块池，该二维数据块池包括对应不同坐标值的各个数据块，减少了第二设备的数据遍历时长，提高了数据查找的效率。
请参阅图9，本申请实施例中第一设备的另一个实施例包括：
第一接收单元901，用于接收第一设备发送的多个数据块，每个数据块中包含一个坐标值；
生成单元902，用于根据多个坐标值生成对应的二维索引表，所述二维索引表用于指示所述多个数据块的存储位置；
排序单元903，用于根据所述二维索引表对所述多个数据块中的每个数据块进行排序，得到二维数据块池，所述二维数据块池为需要传输的目标数据。
在一种可行的实施方式中，生成单元902包括：
第一确定模块9021，用于根据所述坐标值中的第二个值确定二维索引表的纵坐标，所述二维索引表的纵坐标用于指示目标数据的分组数量；
第二确定模块9022，用于根据所述坐标值中的第一个值确定二维索引表的横坐标，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；
生成模块9023，用于根据所述横坐标和所述纵坐标生成对应的二维索引表。
在一种可行的实施方式中，每个数据块的大小与网络延迟负相关；
所述线程数量与网络带宽正相关。
在一种可行的实施方式中，所述第二设备还包括：
校验单元904，用于根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。
在一种可行的实施方式中，所述数据块包括头域信息和内容信息；
所述头域信息中至少包括所述数据块的坐标值和所述校验码，所述坐标值用于指示所述数据块在所述二维数据块池中的位置；
所述内容信息中包括所述头域信息指示的数据内容，所述数据内容包含于所述目标数据。
在一种可行的实施方式中，所述第二设备还包括：
获取单元905，若第一数据块传输失败，则用于确定所述第一数据块的第一坐标值；
发送单元906，用于向所述第一设备发送重传请求，所述重传请求中包括所述第一坐标值；
第二接收单元907，用于接收所述第一设备重新发送的第一数据块。
上面图6至图9从模块化功能实体的角度分别对本申请实施例中第一设备和第二设备进行详细描述，下面从硬件处理的角度对本申请实施例中设备进行详细描述。
图10是本申请实施例提供的一种设备的结构示意图，该设备可以是服务器，该设备1000可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上处理器(central processing units，CPU)1001(例如，一个或一个以上处理器)和存储器1009，一个或一个以上存储应用程序1009或数据1006的存储介质1008(例如一个或一个以上海量存储设备)。其中，存储器1009和存储介质1008可以是短暂存储或持久存储。存储在存储介质1008的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对设备中的一系列指令操作。更进一步地，处理器1001可以设置为与存储介质1008通信，在设备1000上执行存储介质1008中的一系列指令操作。
设备1000还可以包括一个或一个以上电源1002，一个或一个以上有线或无线网络接口1003，一个或一个以上输入输出接口1004，和/或，一个或一个以上操作系统1005，例如Windows Serve，Mac OS X，Unix，Linux，FreeBSD等等。本领域技术人员可以理解，图10中示出的设备结构并不构成对设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
下面结合图10对设备的各个构成部件进行具体的介绍：
处理器1001是设备的控制中心，可以按照设置的数据传输方法进行处理。处理器1001利用各种接口和线路连接整个设备的各个部分，通过运行或执行存储在存储器1009内的软件程序和/或模块，以及调用存储在存储器1009内的数据，执行设备的各种功能和处理数据，从而实现数据的传输。
存储器1009可用于存储软件程序以及模块，处理器1001通过运行存储在存储器1009的软件程序以及模块，从而执行设备1000的各种功能应用以及数据处理。存储器1009可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如生成二维数据块池等)等；存储数据区可存储根据设备的使用所创建的数据(比如生成二维索引表等)等。此外，存储器1009可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。在本申请实施例中提供的数据传输方法的程序和接收到的数据流存储在存储器中，当需要使用时，处理器1001从存储器1009中调用。
下面结合图11对设备的各个构成部件进行具体的介绍，该设备可以是终端设备：
图11示出的是与本申请实施例提供的设备的部分结构的框图。参考图11，所述终端设备包括：射频(radio frequency，RF)电路1110、存储器1120、输入单元1130、显示单元1140、传感器1150、音频电路1160、无线保真(wireless fidelity，WIFI)模块1170和处理器1180等部件。本领域技术人员可以理解，图11中示出的终端设备结构并不构成对所述终端设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
所述处理器1180是终端设备的控制中心，在本申请实施例中，可以根据预置的多 线程层序遍历算法生成二维索引表，并根据二维索引表对目标数据进行重组得到二维数据块池，并将该二维数据块池中的各个数据块发送至与其连接的其他设备。
RF电路1110通过总线与所述处理器1180连接，负责向互联网发送数据或者从互联网接收数据，还可用于收发信息过程中，信号的接收和发送，例如，向第二设备发送二维数据块池中的各个数据块；另外，第一设备在接收到第二设备发送的重传请求后，将重传请求发送给所述处理器1180处理。通常，所述RF电路1110包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(low noise amplifier，LNA)、双工器等。此外，所述RF电路1110还可以通过无线通信与网络和其他设备通信。上述无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯系统(global system of mobile communication，GSM)、通用分组无线服务(general packet radio service，GPRS)、码分多址(code division multiple access，CDMA)、宽带码分多址(wideband code division multiple access，WCDMA)、长期演进(long term evolution，LTE)、电子邮件、短消息服务(short messaging service，SMS)等。
存储器1120可用于存储软件程序以及模块，所述处理器1180通过运行存储在所述存储器1120的软件程序以及模块，从而执行终端设备的各种功能应用以及数据处理。所述存储器1120可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如根据预置的多线程层序遍历算法生成二维索引表等)等；存储数据区可存储根据终端设备的使用所创建的数据(比如二维数据块池等)等。此外，所述存储器1120可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计 算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (34)

  1. 一种数据传输方法，其特征在于，包括：
    第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，所述二维索引表用于指示所述目标数据中各个数据的存储位置；
    所述第一设备根据所述二维索引表对所述目标数据进行数据重组，得到二维数据块池，所述二维数据块池包括多个数据块，每个数据块分别对应所述二维索引表的一个坐标值；
    所述第一设备通过所述二维索引表索引到所述多个数据块并发送至第二设备。
  2. 根据权利要求1所述的方法，其特征在于，所述第一设备根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表包括：
    所述第一设备根据预置的多线程层序遍历算法对需要传输的目标数据进行遍历，得到数据目录，所述数据目录包括至少一层目录；
    所述第一设备根据所述目标数据的大小和所述至少一层目录确定二维索引表的横坐标和纵坐标，所述二维索引表的纵坐标用于指示目标数据的分组数量，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；
    所述第一设备生成二维索引表。
  3. 根据权利要求2所述的方法，其特征在于，
    每个数据块的大小与网络延迟负相关；
    所述线程数量与网络带宽正相关。
  4. 根据权利要求1-3任一所述的方法，其特征在于，所述第一设备根据所述二维索引表对所述目标数据进行数据重组之后，所述第一设备通过所述二维索引表索引到所述多个数据块并发送至第二设备之前，所述方法还包括：
    所述第一设备生成每个数据块的校验码，所述校验码用于校验每个数据块的正确性。
  5. 根据权利要求4所述的方法，其特征在于，
    所述数据块包括头域信息和内容信息；
    所述头域信息中至少包括所述数据块的坐标值和所述校验码，所述坐标值用于指示所述数据块在所述二维数据块池中的位置；
    所述内容信息中包括所述头域信息指示的数据内容，所述数据内容包含于所述目标数据。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    若第一数据块传输失败，则所述第一设备接收所述第二设备发送的重传请求，所述重传请求中包括所述第一数据块的第一坐标值；
    所述第一设备根据所述第一坐标值确定所述第一数据块在所述二维数据块池中的位置；
    所述第一设备根据所述第一数据块的位置重新获取第一数据块；
    所述第一设备向所述第二设备发送重新获取的第一数据块。
  7. 根据权利要求5所述的方法，其特征在于，
    当所述目标数据包含大文件，所述大文件由多个目标数据块组成时，所述目标数据 块的头域信息中还包括第一指示信息，所述第一指示信息用于指示各个目标数据块在所述大文件中的位置，并在完成传输后组成所述大文件。
  8. 根据权利要求5所述的方法，其特征在于，
    当所述目标数据包含多个小文件，所述多个小文件组成目标数据块时，所述目标数据块的头域信息中还包括第二指示信息，所述第二指示信息用于指示所述目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。
  9. 一种数据传输方法，其特征在于，包括：
    第二设备接收第一设备发送的多个数据块，每个数据块中包含一个坐标值；
    所述第二设备根据多个坐标值生成对应的二维索引表，所述二维索引表用于指示所述多个数据块的存储位置；
    所述第二设备根据所述二维索引表对所述多个数据块中的每个数据块进行排序，得到二维数据块池，所述二维数据块池为需要传输的目标数据。
  10. 根据权利要求9所述的方法，其特征在于，所述第二设备根据多个坐标值生成对应的二维索引表包括：
    所述第二设备根据所述坐标值中的第二个值确定二维索引表的纵坐标，所述二维索引表的纵坐标用于指示目标数据的分组数量；
    所述第二设备根据所述坐标值中的第一个值确定二维索引表的横坐标，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；
    所述第二设备根据所述横坐标和所述纵坐标生成对应的二维索引表。
  11. 根据权利要求10所述的方法，其特征在于，
    每个数据块的大小与网络延迟负相关；
    所述线程数量与网络带宽正相关。
  12. 根据权利要求9-11任一所述的方法，其特征在于，所述第二设备接收第一设备发送的多个数据块之后，所述第二设备根据多个坐标值生成对应的二维索引表之前，所述方法还包括：
    所述第二设备根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。
  13. 根据权利要求12所述的方法，其特征在于，
    所述数据块包括头域信息和内容信息；
    所述头域信息中至少包括所述数据块的坐标值和所述校验码，所述坐标值用于指示所述数据块在所述二维数据块池中的位置；
    所述内容信息中包括所述头域信息指示的数据内容，所述数据内容包含于所述目标数据。
  14. 根据权利要求13所述的方法，其特征在于，所述方法还包括：
    若第一数据块传输失败，则所述第二设备确定所述第一数据块的第一坐标值；
    所述第二设备向所述第一设备发送重传请求，所述重传请求中包括所述第一坐标值；
    所述第二设备接收所述第一设备重新发送的第一数据块。
  15. 根据权利要求13所述的方法，其特征在于，
    当所述目标数据包含大文件，所述大文件由多个目标数据块组成时，所述目标数据块的头域信息中还包括第一指示信息，所述第一指示信息用于指示各个目标数据块在所述大文件中的位置，并在完成传输后组成所述大文件。
  16. 根据权利要求13所述的方法，其特征在于，
    当所述目标数据包含多个小文件，所述多个小文件组成目标数据块时，所述目标数据块的头域信息中还包括第二指示信息，所述第二指示信息用于指示所述目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。
  17. 一种设备，其特征在于，所述设备为第一设备，包括：
    第一生成单元，用于根据预置的多线程层序遍历算法和需要传输的目标数据的大小生成二维索引表，所述二维索引表用于指示所述目标数据中各个数据的存储位置；
    重组单元，用于根据所述二维索引表对所述目标数据进行数据重组，得到二维数据块池，所述二维数据块池包括多个数据块，每个数据块分别对应所述二维索引表的一个坐标值；
    第一发送单元，用于通过所述二维索引表索引到所述多个数据块并发送至第二设备。
  18. 根据权利要求17所述的设备，其特征在于，所述第一生成单元包括：
    遍历模块，用于根据预置的多线程层序遍历算法对需要传输的目标数据进行遍历，得到数据目录，所述数据目录包括至少一层目录；
    确定模块，用于根据所述目标数据的大小和所述至少一层目录确定二维索引表的横坐标和纵坐标，所述二维索引表的纵坐标用于指示目标数据的分组数量，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；
    生成模块，用于生成二维索引表。
  19. 根据权利要求18所述的设备，其特征在于，
    每个数据块的大小与网络延迟负相关；
    所述线程数量与网络带宽正相关。
  20. 根据权利要求17-19任一所述的设备，其特征在于，所述设备还包括：
    第二生成单元，用于生成每个数据块的校验码，所述校验码用于校验每个数据块的正确性。
  21. 根据权利要求20所述的设备，其特征在于，
    所述数据块包括头域信息和内容信息；
    所述头域信息中至少包括所述数据块的坐标值和所述校验码，所述坐标值用于指示所述数据块在所述二维数据块池中的位置；
    所述内容信息中包括所述头域信息指示的数据内容，所述数据内容包含于所述目标数据。
  22. 根据权利要求21所述的设备，其特征在于，所述设备还包括：
    接收单元，若第一数据块传输失败，则用于接收所述第二设备发送的重传请求，所述重传请求中包括所述第一数据块的第一坐标值；
    确定单元，用于根据所述第一坐标值确定所述第一数据块在所述二维数据块池中的位置；
    获取单元，用于根据所述第一数据块的位置重新获取第一数据块；
    第二发送单元，用于向所述第二设备发送重新获取的第一数据块。
  23. 根据权利要求21所述的设备，其特征在于，
    当所述目标数据包含大文件，所述大文件由多个目标数据块组成时，所述目标数据块的头域信息中还包括第一指示信息，所述第一指示信息用于指示各个目标数据块在所述大文件中的位置，并在完成传输后组成所述大文件。
  24. 根据权利要求21所述的设备，其特征在于，
    当所述目标数据包含多个小文件，所述多个小文件组成目标数据块时，所述目标数据块的头域信息中还包括第二指示信息，所述第二指示信息用于指示所述目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。
  25. 一种设备，其特征在于，所述设备为第二设备，包括：
    第一接收单元，用于接收第一设备发送的多个数据块，每个数据块中包含一个坐标值；
    生成单元，用于根据多个坐标值生成对应的二维索引表，所述二维索引表用于指示所述多个数据块的存储位置；
    排序单元，用于根据所述二维索引表对所述多个数据块中的每个数据块进行排序，得到二维数据块池，所述二维数据块池为需要传输的目标数据。
  26. 根据权利要求25所述的设备，其特征在于，所述生成单元包括：
    第一确定模块，用于根据所述坐标值中的第二个值确定二维索引表的纵坐标，所述二维索引表的纵坐标用于指示目标数据的分组数量；
    第二确定模块，用于根据所述坐标值中的第一个值确定二维索引表的横坐标，所述二维索引表的横坐标用于指示每个分组对应的数据块数量；
    生成模块，用于根据所述横坐标和所述纵坐标生成对应的二维索引表。
  27. 根据权利要求26所述的设备，其特征在于，
    每个数据块的大小与网络延迟负相关；
    所述线程数量与网络带宽正相关。
  28. 根据权利要求25-27任一所述的设备，其特征在于，所述设备还包括：
    校验单元，用于根据每个数据块中携带的校验码进行校验，确定每个数据块的正确性。
  29. 根据权利要求28所述的设备，其特征在于，
    所述数据块包括头域信息和内容信息；
    所述头域信息中至少包括所述数据块的坐标值和所述校验码，所述坐标值用于指示所述数据块在所述二维数据块池中的位置；
    所述内容信息中包括所述头域信息指示的数据内容，所述数据内容包含于所述目标数据。
  30. 根据权利要求29所述的设备，其特征在于，所述设备还包括：
    获取单元，若第一数据块传输失败，则用于确定所述第一数据块的第一坐标值；
    发送单元，用于向所述第一设备发送重传请求，所述重传请求中包括所述第一坐标值；
    第二接收单元，用于接收所述第一设备重新发送的第一数据块。
  31. 根据权利要求29所述的设备，其特征在于，
    当所述目标数据包含大文件，所述大文件由多个目标数据块组成时，所述目标数据块的头域信息中还包括第一指示信息，所述第一指示信息用于指示各个目标数据块在所述大文件中的位置，并在完成传输后组成所述大文件。
  32. 根据权利要求29所述的设备，其特征在于，
    当所述目标数据包含多个小文件，所述多个小文件组成目标数据块时，所述目标数据块的头域信息中还包括第二指示信息，所述第二指示信息用于指示所述目标数据块中包含的各个小文件的位置，并在完成传输后分解成各个小文件。
  33. 一种计算机可读存储介质，包括指令，当其在计算机上运行时，使得计算机执行如权利要求1-14任意一项所述的方法。
  34. 一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行如权利要求1-14任意一项所述的方法。
PCT/CN2019/080823 2018-04-23 2019-04-01 一种数据传输方法及相关设备 WO2019205897A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19792099.4A EP3780438B1 (en) 2018-04-23 2019-04-01 Data transmission method and related device
US17/073,438 US11476977B2 (en) 2018-04-23 2020-10-19 Data transmission method and related device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810369971.1 2018-04-23
CN201810369971.1A CN108809514B (zh) 2018-04-23 2018-04-23 一种数据传输方法及相关设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/073,438 Continuation US11476977B2 (en) 2018-04-23 2020-10-19 Data transmission method and related device

Publications (1)

Publication Number Publication Date
WO2019205897A1 true WO2019205897A1 (zh) 2019-10-31

Family

ID=64093166

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/080823 WO2019205897A1 (zh) 2018-04-23 2019-04-01 一种数据传输方法及相关设备

Country Status (4)

Country Link
US (1) US11476977B2 (zh)
EP (1) EP3780438B1 (zh)
CN (1) CN108809514B (zh)
WO (1) WO2019205897A1 (zh)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108809514B (zh) 2018-04-23 2021-01-12 华为技术有限公司 一种数据传输方法及相关设备
CN111200479B (zh) * 2018-11-19 2022-04-12 福建天泉教育科技有限公司 传输数据的校验方法、存储介质
CN109887456A (zh) * 2019-01-17 2019-06-14 硅谷数模半导体（北京）有限公司 数据压缩方法及装置
CN116057925A (zh) * 2019-11-20 2023-05-02 深圳市欢太科技有限公司 一种数据传输方法、数据下载方法及终端
CN113160187B (zh) * 2021-04-27 2022-02-15 圣名科技(广州)有限责任公司 一种设备的故障检测方法及装置
CN117785499A (zh) * 2022-09-22 2024-03-29 华为技术有限公司 聚合通信中的数据传输方法以及系统
CN115576899B (zh) * 2022-12-09 2023-03-21 深圳市木浪云科技有限公司 构建索引的方法和装置以及文件查找方法和装置
CN117688593B (zh) * 2024-02-02 2024-04-30 新汽有限公司 一种网络大数据的管理系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104753626A (zh) * 2013-12-25 2015-07-01 华为技术有限公司 一种数据压缩方法、设备及系统
CN105677904A (zh) * 2016-02-04 2016-06-15 杭州数梦工场科技有限公司 基于分布式文件系统的小文件存储方法及装置
CN105933412A (zh) * 2016-04-20 2016-09-07 北京云宏信达信息科技有限公司 文件接收方法和装置及文件传输系统
WO2018037495A1 (ja) * 2016-08-24 2018-03-01 株式会社日立製作所 転送装置及びネットワークシステム
CN108809514A (zh) * 2018-04-23 2018-11-13 华为技术有限公司 一种数据传输方法及相关设备

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030126201A1 (en) * 2001-11-30 2003-07-03 Khoi Hoang Efficient storage of data files received in a non-sequential manner
CA2415018C (en) * 2002-12-23 2006-09-19 Ibm Canada Limited - Ibm Canada Limitee Adaptive parallel data clustering when loading a data structure containing data clustered along one or more dimensions
US7212681B1 (en) * 2003-01-15 2007-05-01 Cisco Technology, Inc. Extension of two-dimensional variable length coding for image compression
AU2005322044A1 (en) * 2004-12-24 2006-07-06 Aspera, Inc. Bulk data transfer
US7636767B2 (en) * 2005-11-29 2009-12-22 Cisco Technology, Inc. Method and apparatus for reducing network traffic over low bandwidth links
US8077059B2 (en) * 2006-07-21 2011-12-13 Eric John Davies Database adapter for relational datasets
US8725986B1 (en) * 2008-04-18 2014-05-13 Netapp, Inc. System and method for volume block number to disk block number mapping
CN101782921B (zh) * 2009-12-28 2013-01-16 北京握奇数据系统有限公司 一种目录创建、查询和删除方法及装置
CN101945207B (zh) * 2010-09-17 2013-01-16 福建新大陆通信科技股份有限公司 一种基于机顶盒升级数据的二维索引表快速缓冲方法
US8707141B1 (en) * 2011-08-02 2014-04-22 Cisco Technology, Inc. Joint optimization of packetization and error correction for video communication
CN103019895B (zh) * 2012-12-28 2015-01-28 华为技术有限公司 文件存储方法及装置
CN103116555B (zh) * 2013-03-05 2014-03-05 中国人民解放军国防科学技术大学 基于多体并行缓存结构的数据访问方法
CN104243425B (zh) * 2013-06-19 2018-09-04 深圳市腾讯计算机系统有限公司 一种在内容分发网络中进行内容管理的方法、装置及系统
CN105591915A (zh) * 2014-10-22 2016-05-18 中兴通讯股份有限公司 一种路由表的维护方法和装置
CN105302550B (zh) * 2015-10-12 2019-03-26 江苏中威科技软件系统有限公司 将页面转为版式数据流文件的方法及系统
CN106528830B (zh) * 2016-11-16 2019-05-10 华为技术有限公司 一种恢复文件索引目录的方法和装置
CN110784528B (zh) * 2019-10-22 2023-04-18 北京天融信网络安全技术有限公司 一种文件下载方法、装置和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104753626A (zh) * 2013-12-25 2015-07-01 华为技术有限公司 一种数据压缩方法、设备及系统
CN105677904A (zh) * 2016-02-04 2016-06-15 杭州数梦工场科技有限公司 基于分布式文件系统的小文件存储方法及装置
CN105933412A (zh) * 2016-04-20 2016-09-07 北京云宏信达信息科技有限公司 文件接收方法和装置及文件传输系统
WO2018037495A1 (ja) * 2016-08-24 2018-03-01 株式会社日立製作所 転送装置及びネットワークシステム
CN108809514A (zh) * 2018-04-23 2018-11-13 华为技术有限公司 一种数据传输方法及相关设备

Also Published As

Publication number Publication date
EP3780438A4 (en) 2021-04-21
EP3780438B1 (en) 2023-10-25
US20210037098A1 (en) 2021-02-04
EP3780438A1 (en) 2021-02-17
CN108809514A (zh) 2018-11-13
CN108809514B (zh) 2021-01-12
US11476977B2 (en) 2022-10-18

Similar Documents

Publication Publication Date Title
WO2019205897A1 (zh) 一种数据传输方法及相关设备
US11716409B2 (en) Packet transmission method and apparatus
CN113934660B (zh) 加速网络分组处理
US9158806B2 (en) Integrity checking and selective deduplication based on network parameters
US8954392B2 (en) Efficient de-duping using deep packet inspection
US7643505B1 (en) Method and system for real time compression and decompression
US9703796B2 (en) Shared dictionary between devices
CN110928483B (zh) 数据存储、数据获取方法及设备
US8560552B2 (en) Method for lossless data reduction of redundant patterns
US10142257B2 (en) Dynamic scaling of redundancy elimination middleboxes
US11316804B2 (en) Forwarding entry update method and apparatus in a memory
US20220004524A1 (en) Chunking method and apparatus
US20230006937A1 (en) Packet flow identification with reduced decode operations
EP3720068A1 (en) Message processing method and device
CN112995053A (zh) 发送报文的方法及装置
CN101526959B (zh) 一种数据存储方法和装置
CN107203559B (zh) 一种划分数据条带的方法和装置
CN110912969B (zh) 一种文件高速传输源节点、目的节点装置及系统
CN106209666B (zh) 一种基于负载均衡器的链路复用方法及系统
CN112395296A (zh) 一种大数据归档方法、装置、设备及存储介质
CN111506658B (zh) 数据处理方法、装置、第一设备及存储介质
CN113316770B (zh) 数据修复方法和装置
WO2024016975A1 (zh) 报文转发方法、装置、设备及芯片系统
WO2023130953A1 (zh) 数据查找方法及装置、网络设备
CN118139132A (zh) 一种报文处理方法、报文处理装置、网络设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19792099

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019792099

Country of ref document: EP

Effective date: 20201026