US11476977B2 - Data transmission method and related device - Google Patents

Data transmission method and related device Download PDF

Info

Publication number
US11476977B2
US11476977B2 US17/073,438 US202017073438A US11476977B2 US 11476977 B2 US11476977 B2 US 11476977B2 US 202017073438 A US202017073438 A US 202017073438A US 11476977 B2 US11476977 B2 US 11476977B2
Authority
US
United States
Prior art keywords
data
data block
index table
dimensional
dimensional index
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US17/073,438
Other languages
English (en)
Other versions
US20210037098A1 (en
Inventor
Yong Fan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Assigned to HUAWEI TECHNOLOGIES CO., LTD. reassignment HUAWEI TECHNOLOGIES CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: FAN, YONG
Publication of US20210037098A1 publication Critical patent/US20210037098A1/en
Application granted granted Critical
Publication of US11476977B2 publication Critical patent/US11476977B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
    • H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
    • H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/1607—Details of the supervisory signal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1806—Go-back-N protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04—Protocols for data compression, e.g. ROHC
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
    • H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
    • H04L1/0056—Systems characterized by the type of code used
    • H04L1/0061—Error detection codes

Definitions

  • This application relates to the communications field, and in particular, to a data transmission method and a related device.
  • Data transmission is a basic communication service, and most computer services require the data transmission service.
  • File transfer efficiency of a conventional file transfer protocol (FTP) server in a wide area network is low, a difference between a file transfer rate (approximately 300 KB/s) of an existing service system in the wide area network and a local file transfer rate (approximately 6 MB/s) of the service system is dozens of times, the file transfer rate in the wide area network cannot meet requirements for increasing services, and data transmission in the wide area network has become a bottleneck that hinders growth of a service system. Therefore, performance of the file transfer rate in the wide area network needs to be improved.
  • FTP file transfer protocol
  • FIG. 1 is a schematic diagram in which a data congestion phenomenon occurs when a file whose size is greater than 1 G is downloaded.
  • a linear file blocking solution is used to avoid congestion, and a hash tree manner is used to quickly identify whether file content is accurate, so as to ensure accuracy of the file content.
  • a secure hash algorithm SHA
  • a hash value has 256 bits, and further includes six pieces of small file data each with a size of 1 MB and one piece of small file data with a size of 0.5 MB.
  • a size of a hash tree (a depth of the hash tree) is positively correlated to a size of a file, and a larger file indicates a larger size of the hash tree
  • a time for which the hash tree needs to be traversed to find a file block for identification is long, a directory traversal speed is slow, and search efficiency is low.
  • a transmission problem occurs, a time for retransmission is long, and data repair efficiency is low.
  • Various embodiments provide a data transmission method and a related device, to improve a directory traversal speed and improve data search and data repair efficiency.
  • a first aspect of this application provides a data transmission method, including: generating, by a first device, a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted, where the two-dimensional index table is used to indicate a storage location of each piece of data in the target data, and the target data may include one or more directories, a complex multi-level directory, a single file, or a plurality of files; performing, by the first device, data reconstruction on the target data based on the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, each data block corresponds to one pair of coordinates in the two-dimensional index table, and each data block has a different pair of coordinates in the two-dimensional index table; and obtaining, by the first device, the plurality of data blocks in the two-dimensional data block pool through indexing by using the two-dimensional index table, and sending the plurality of data blocks to a second device
  • the first device reconstructs the target data that needs to be transmitted, to obtain the two-dimensional data block pool.
  • the first device can quickly determine each data block through locating based on the two-dimensional data block pool, and send the two-dimensional data block pool to the second device, such that data traversal duration is reduced, to improve data search efficiency and improve data transmission efficiency.
  • the generating, by a first device, a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted includes: traversing, by the first device based on the preset multi-thread sequence traversal algorithm, the target data that needs to be transmitted, to obtain a data directory, where the data directory includes at least one level of directory; determining, by the first device, a horizontal coordinate and a vertical coordinate of the two-dimensional index table based on the size of the target data and the at least one level of directory, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data, and the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group; and generating, by the first device, the two-dimensional index table, where the vertical coordinate of the two-dimensional index table is used to indicate the quantity of groups, and the horizontal coordinate of the two
  • a size of each data block is negatively correlated to a network delay; and a quantity of threads is positively correlated to a network bandwidth.
  • the size of the data block and the quantity of threads are limited, and a relationship between the data block and the network delay and a relationship between the quantity of threads and the network bandwidth are specified. Therefore, the size of the data block and the quantity of threads can be quickly set, to improve data transmission efficiency.
  • the method further includes: generating, by the first device, a check code of each data block, where the check code is used to check correctness of each data block. A process of generating the check code is added, to ensure accuracy of a to-be-transmitted data block.
  • the data block includes header field information and content information;
  • the header field information includes at least coordinates and a check code of the data block, and the coordinates are used to indicate a location of the data block in the two-dimensional data block pool;
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • a structure of the data block is limited, and an implementation of the embodiment of this application is added.
  • the method includes: if a first data block fails to be transmitted, receiving, by the first device, a retransmission request sent by the second device, where the retransmission request includes first coordinates of the first data block; determining, by the first device, a location of the first data block in the two-dimensional data block pool based on the first coordinates; re-obtaining, by the first device, the first data block based on the location of the first data block; and sending, by the first device, the re-obtained first data block to the second device.
  • a process of repairing a data block that fails to be transmitted is added, and only the data block that fails to be transmitted is retransmitted, to reduce a data repair time, and improve data repair efficiency.
  • header field information of the target data blocks further includes first indication information, where the first indication information is used to indicate a location of each target data block in the large file, and the large file is formed after transmission is completed.
  • the first indication information indicating that the large file is formed is added, such that after receiving the data block, the second device restores the plurality of data blocks to the large file based on the first indication information.
  • header field information of the target data block further includes second indication information, where the second indication information is used to indicate a location of each small file included in the target data block, and the small files are obtained through division after transmission is completed.
  • the second indication information indicating that the small file is formed is added, such that after receiving the data block, the second device restores the data block to the plurality of small files based on the second indication information.
  • all data blocks have a same size.
  • a size of each data block is limited, such that the obtained two-dimensional data block pool has a more proper structure, to facilitate a data search and improve data search efficiency.
  • the data content is one or more target files; or the data content is a part of content of a target file.
  • Data content included in each data block is limited, and the plurality of small files may be transmitted together, to improve data transmission efficiency.
  • the header field information further includes a target path, a source file name, a target file name, a file offset address, and a data block size. Content included in the header field information of the data block is limited, such that steps in this application are more complete.
  • a second aspect of this application provides a data transmission method, including: receiving, by a second device, a plurality of data blocks sent by a first device, where each data block includes one pair of coordinates; generating, by the second device, a corresponding two-dimensional index table based on a plurality of pairs of coordinates, where the two-dimensional index table is used to indicate storage locations of the plurality of data blocks; and ranking, by the second device, each of the plurality of data blocks based on the two-dimensional index table, to obtain a two-dimensional data block pool, where data content included in data blocks in the two-dimensional data block pool forms target data that needs to be transmitted.
  • the second device generates the two-dimensional index table based on the coordinates included in the plurality of received data blocks, and reconstructs the target data that needs to be transmitted, to obtain the two-dimensional data block pool.
  • the second device can quickly determine each data block through locating based on the two-dimensional data block pool, and search the data block in the two-dimensional data block pool, such that data traversal duration is reduced, to improve data search efficiency and improve data transmission efficiency.
  • the generating, by the second device, a corresponding two-dimensional index table based on coordinates includes: determining, by the second device, a vertical coordinate of the two-dimensional index table based on a second value in the coordinates, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data; determining, by the second device, a horizontal coordinate of the two-dimensional index table based on a first value in the coordinates, where the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group; and generating, by the second device, the corresponding two-dimensional index table based on the horizontal coordinate and the vertical coordinate.
  • a process in which the second device generates the two-dimensional index table is refined, a specific implementation is provided, and an implementation of this application is added.
  • a size of each data block is negatively correlated to a network delay; and a quantity of threads is positively correlated to a network bandwidth.
  • the size of the data block and the quantity of threads are limited, and a relationship between the data block and the network delay and a relationship between the quantity of threads and the network bandwidth are specified. Therefore, the size of the data block and the quantity of threads can be quickly set, to improve data transmission efficiency.
  • the method further includes: performing, by the second device, a check based on a check code carried in each data block, to determine correctness of each data block. A process of generating the check code is added, to ensure accuracy of a to-be-transmitted data block.
  • the data block includes header field information and content information;
  • the header field information includes at least coordinates and a check code of the data block, and the coordinates are used to indicate a location of the data block in the two-dimensional data block pool;
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • a structure of the data block is limited, and an implementation of the embodiment of this application is added.
  • the method includes: if a first data block fails to be transmitted, determining, by the second device, first coordinates of the first data block; sending, by the second device, a retransmission request to the first device, where the retransmission request includes the first coordinates; and receiving, by the second device, the first data block resent by the first device.
  • a process of repairing a data block that fails to be transmitted is added, and only the data block that fails to be transmitted is retransmitted, to reduce a data repair time, and improve data repair efficiency.
  • header field information of the target data blocks further includes first indication information, where the first indication information is used to indicate a location of each target data block in the large file, and the large file is formed after transmission is completed.
  • the first indication information indicating that the large file is formed is added, such that after receiving the data block, the second device restores the plurality of data blocks to the large file based on the first indication information.
  • header field information of the target data block further includes second indication information, and the second indication information is used to indicate a location of each small file included in the target data block, and the small files are obtained through division after transmission is completed.
  • the second indication information indicating that the small file is formed is added, such that after receiving the data block, the second device restores the data block to the plurality of small files based on the second indication information.
  • all data blocks have a same size.
  • a size of each data block is limited, such that the obtained two-dimensional data block pool has a more proper structure, to facilitate a data search and improve data search efficiency.
  • the data content is one or more target files; or the data content is a part of content of a target file.
  • Data content included in each data block is limited, and the plurality of small files may be transmitted together, to improve data transmission efficiency.
  • the header field information further includes a target path, a source file name, a target file name, a file offset address, and a data block size. Content included in the header field information of the data block is limited, such that steps in this application are more complete.
  • a third aspect of this application provides a device, where the device is a first device and includes: a first generation unit, configured to generate a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted, where the two-dimensional index table is used to indicate a storage location of each piece of data in the target data, and the target data may include one or more directories, a complex multi-level directory, a single file, or a plurality of files; a reconstruction unit, configured to perform data reconstruction on the target data based on the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, and each data block corresponds to one pair of coordinates in the two-dimensional index table; and a first sending unit, configured to: obtain the plurality of data blocks through indexing by using the two-dimensional index table, and send the plurality of data blocks to a second device.
  • a first generation unit configured to generate a two-dimensional index
  • the first device reconstructs the target data that needs to be transmitted, to obtain the two-dimensional data block pool.
  • the first device can quickly determine each data block through locating based on the two-dimensional data block pool, and send the two-dimensional data block pool to the second device, such that data traversal duration is reduced, to improve data search efficiency and improve data transmission efficiency.
  • the first generation unit includes: a traversal module, configured to traverse, based on the preset multi-thread sequence traversal algorithm, the target data that needs to be transmitted, to obtain a data directory, where the data directory includes at least one level of directory; a determining module, configured to determine a horizontal coordinate and a vertical coordinate of the two-dimensional index table based on the size of the target data and the at least one level of directory, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data, and the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group; and a generation module, configured to generate the two-dimensional index table.
  • a traversal module configured to traverse, based on the preset multi-thread sequence traversal algorithm, the target data that needs to be transmitted, to obtain a data directory, where the data directory includes at least one level of directory
  • a determining module configured to determine a horizontal coordinate and a vertical coordinate of the two-dimensional index table based on
  • a size of each data block is negatively correlated to a network delay; and a quantity of threads is positively correlated to a network bandwidth.
  • the size of the data block and the quantity of threads are limited, and a relationship between the data block and the network delay and a relationship between the quantity of threads and the network bandwidth are specified. Therefore, the size of the data block and the quantity of threads can be quickly set, to improve data transmission efficiency.
  • the device further includes: a second generation unit, configured to generate a check code of each data block, where the check code is used to check correctness of each data block. A process of generating the check code is added, to ensure accuracy of a to-be-transmitted data block.
  • the data block includes header field information and content information;
  • the header field information includes at least coordinates and a check code of the data block, and the coordinates are used to indicate a location of the data block in the two-dimensional data block pool;
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • a structure of the data block is limited, and an implementation of the embodiment of this application is added.
  • the method includes: a receiving unit, configured to: if a first data block fails to be transmitted, receive a retransmission request sent by the second device, where the retransmission request includes first coordinates of the first data block; a determining unit, configured to determine a location of the first data block in the two-dimensional data block pool based on the first coordinates; an obtaining unit, configured to re-obtain the first data block based on the location of the first data block; and a second sending unit, configured to send the re-obtained first data block to the second device.
  • a process of repairing a data block that fails to be transmitted is added, and only the data block that fails to be transmitted is retransmitted, to reduce a data repair time, and improve data repair efficiency.
  • header field information of the target data blocks further includes first indication information, where the first indication information is used to indicate a location of each target data block in the large file, and the large file is formed after transmission is completed.
  • the first indication information indicating that the large file is formed is added, such that after receiving the data block, the second device restores the plurality of data blocks to the large file based on the first indication information.
  • header field information of the target data block further includes second indication information, and the second indication information is used to indicate a location of each small file included in the target data block, and the small files are obtained through division after transmission is completed.
  • the second indication information indicating that the small file is formed is added, such that after receiving the data block, the second device restores the data block to the plurality of small files based on the second indication information.
  • all data blocks have a same size.
  • a size of each data block is limited, such that the obtained two-dimensional data block pool has a more proper structure, to facilitate a data search and improve data search efficiency.
  • the data content is one or more target files; or the data content is a part of content of a target file.
  • Data content included in each data block is limited, and the plurality of small files may be transmitted together, to improve data transmission efficiency.
  • the header field information further includes a target path, a source file name, a target file name, a file offset address, and a data block size. Content included in the header field information of the data block is limited, such that steps in this application are more complete.
  • a fourth aspect of this application provides a device, where the device is a second device and includes: a first receiving unit, configured to receive a plurality of data blocks sent by a first device, where each data block includes one pair of coordinates; a generation unit, configured to generate a corresponding two-dimensional index table based on a plurality of pairs of coordinates, where the two-dimensional index table is used to indicate storage locations of the plurality of data blocks; and a ranking unit, configured to rank each of the plurality of data blocks based on the two-dimensional index table, to obtain a two-dimensional data block pool, where the two-dimensional data block pool is target data that needs to be transmitted.
  • the second device generates the two-dimensional index table based on the coordinates included in the plurality of received data blocks, and reconstructs the target data that needs to be transmitted, to obtain the two-dimensional data block pool.
  • the second device can quickly determine each data block through locating based on the two-dimensional data block pool, and search the data block in the two-dimensional data block pool, such that data traversal duration is reduced, to improve data search efficiency and improve data transmission efficiency.
  • the generation unit includes: a first determining module, configured to determine a vertical coordinate of the two-dimensional index table based on a second value in the coordinates, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data; a second determining module, configured to determine a horizontal coordinate of the two-dimensional index table based on a first value in the coordinates, where the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group; and a generation module, configured to generate the corresponding two-dimensional index table based on the horizontal coordinate and the vertical coordinate.
  • a first determining module configured to determine a vertical coordinate of the two-dimensional index table based on a second value in the coordinates, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data
  • a second determining module configured to determine a horizontal coordinate of the two-dimensional index table based on a first value in the coordinates, where the horizontal coordinate of the two-dimensional index table
  • a size of each data block is negatively correlated to a network delay; and a quantity of threads is positively correlated to a network bandwidth.
  • the size of the data block and the quantity of threads are limited, and a relationship between the data block and the network delay and a relationship between the quantity of threads and the network bandwidth are specified. Therefore, the size of the data block and the quantity of threads can be quickly set, to improve data transmission efficiency.
  • the device further includes: a check unit, is configured to perform a check based on a check code carried in each data block, to determine correctness of each data block. A process of generating the check code is added, to ensure accuracy of a to-be-transmitted data block.
  • the data block includes header field information and content information;
  • the header field information includes at least coordinates and a check code of the data block, and the coordinates are used to indicate a location of the data block in the two-dimensional data block pool;
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • a structure of the data block is limited, and an implementation of the embodiment of this application is added.
  • the method includes: an obtaining unit, configured to: if a first data block fails to be transmitted, determine first coordinates of the first data block; a sending unit, configured to send a retransmission request to the first device, where the retransmission request includes the first coordinates; and a second receiving unit, configured to receive the first data block resent by the first device.
  • a process of repairing a data block that fails to be transmitted is added, and only the data block that fails to be transmitted is retransmitted, to reduce a data repair time, and improve data repair efficiency.
  • header field information of the target data blocks further includes first indication information, where the first indication information is used to indicate a location of each target data block in the large file, and the large file is formed after transmission is completed.
  • the first indication information indicating that the large file is formed is added, such that after receiving the data block, the second device restores the plurality of data blocks to the large file based on the first indication information.
  • header field information of the target data block further includes second indication information, and the second indication information is used to indicate a location of each small file included in the target data block, and the small files are obtained through division after transmission is completed.
  • the second indication information indicating that the small file is formed is added, such that after receiving the data block, the second device restores the data block to the plurality of small files based on the second indication information.
  • all data blocks have a same size.
  • a size of each data block is limited, such that the obtained two-dimensional data block pool has a more proper structure, to facilitate a data search and improve data search efficiency.
  • the data content is one or more target files; or the data content is a part of content of a target file.
  • Data content included in each data block is limited, and the plurality of small files may be transmitted together, to improve data transmission efficiency.
  • the header field information further includes a target path, a source file name, a target file name, a file offset address, and a data block size. Content included in the header field information of the data block is limited, such that steps in this application are more complete.
  • a fifth aspect of this application provides a computer readable storage medium.
  • the computer readable storage medium stores an instruction, and when the instruction is run on a computer, the computer is enabled to perform the method in the foregoing aspects.
  • a sixth aspect of this application provides a computer program product including an instruction.
  • the computer program product When the computer program product is run on a computer, the computer is enabled to perform the method in the foregoing aspects.
  • FIG. 1 is a schematic diagram of a relationship between a download rate and a file size in a data download process
  • FIG. 2 is a schematic diagram of a hash tree obtained by using an SHA-256 algorithm
  • FIG. 3 is a schematic diagram of a network architecture to which an embodiment of this application is applied;
  • FIG. 4A is a schematic diagram of an embodiment of a data transmission method according to an embodiment of this application.
  • FIG. 4B is a schematic diagram of performing data traversal by using a multi-thread sequence traversal algorithm according to an embodiment of this application;
  • FIG. 4C is a schematic diagram of a generated two-dimensional index table according to an embodiment of this application.
  • FIG. 4D is a schematic diagram of a relationship between a data block and a file in a two-dimensional data block pool according to an embodiment of this application;
  • FIG. 5 is a schematic diagram of another embodiment of a data transmission method according to an embodiment of this application.
  • FIG. 6 is a schematic diagram of an embodiment of a first device according to an embodiment of this application.
  • FIG. 7 is a schematic diagram of another embodiment of a first device according to an embodiment of this application.
  • FIG. 8 is a schematic diagram of an embodiment of a second device according to an embodiment of this application.
  • FIG. 9 is a schematic diagram of another embodiment of a second device according to an embodiment of this application.
  • FIG. 10 is a schematic diagram of another embodiment of a device according to an embodiment of this application.
  • FIG. 11 is a schematic diagram of another embodiment of a device according to an embodiment of this application.
  • Embodiments provide a data transmission method and a related device, to improve a directory traversal speed and improve data search and data repair efficiency.
  • the terms such as “first”, “second”, and the like are intended to distinguish between similar objects but do not necessarily indicate a specific order or sequence.
  • the terms “include”, “have” and any other variants herein mean to cover the non-exclusive inclusion, for example, a process, a method, a system, a product, or a device that includes a list of steps or units is not necessarily limited to those units, but may include other units not expressly listed or inherent to such a process, a method, a product, or a device.
  • a first device is used as a source device, and a second device is used as a target device.
  • the first device generates a two-dimensional index table, indexes and loads data corresponding to coordinates in the two-dimensional index table, and transmits the loaded file data to the second device, such that an end-to-end process of transmitting a file from the source device to the target device is implemented, to reduce duration required for traversal and improve transmission efficiency.
  • a first device and a second device in this example each may be an independent device, or may be a device cluster including a plurality of devices. For ease of description, only a single device is used as an example for description in this application.
  • an embodiment of a data transmission method in this embodiment includes the following steps.
  • a first device generates a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted.
  • the first device generates the two-dimensional index table based on the preset multi-thread sequence traversal algorithm and the size of the target data that needs to be transmitted.
  • the two-dimensional index table is used to indicate a storage location of each piece of data in the target data.
  • the first device first determines the size of the target data that needs to be transmitted to a second device, and then divides the target data into a plurality of data blocks with standard sizes based on the size of the target data, where sizes of all the data blocks are the same.
  • the first device traverses all data blocks in the target data by using the preset multi-thread sequence traversal algorithm, to obtain a data directory, where the data directory includes at least one level of directory.
  • the first device determines a horizontal coordinate and a vertical coordinate of the two-dimensional index table based on the size of the target data and the at least one level of directory, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups, and the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group.
  • the first device generates the two-dimensional index table.
  • the first device configures one or more threads for each level of directory, in other words, correspondingly allocates a thread to each group, where each thread is used to search a data block in a corresponding group; and determines a quantity of concurrent threads based on the quantity of groups and a preset thread pool.
  • the preset multi-thread sequence traversal algorithm is an improvement to an existing sequence traversal algorithm.
  • the plurality of threads are introduced, such that a plurality of data blocks can be simultaneously traversed, to improve traversal efficiency.
  • the target data may include one or more directories or a complex multi-level directory.
  • the data directory may include four levels of directories (shown in a dashed-line box): a first-level directory, a second-level directory, a third-level directory, and a fourth-level directory.
  • Each level of directory corresponds to one or more data blocks, and each data block includes some data content.
  • a thread (shown in a solid-line box) is configured for each level of directory. One thread is configured for the first-level directory, six threads are configured for the second-level directory, six threads are configured for the third-level directory, and three threads are configured for the fourth-level directory.
  • Each thread is used to search the data block in the corresponding group for data content.
  • the quantity of concurrent threads used to search the data block in the group is determined based on the size of the target data and the preset thread pool, where a largest quantity of threads is determined by a size of the preset thread pool.
  • the two-dimensional index table is generated, to identify the horizontal coordinate and the vertical coordinate of the two-dimensional index table.
  • the vertical coordinate of the generated two-dimensional index table corresponds to nine groups that are identified by using letters A to I, and each group corresponds to nine data blocks that are identified by using numbers 1 to 9.
  • the horizontal coordinate and the vertical coordinate may alternatively be identified in another manner.
  • the horizontal coordinate is identified by using a letter
  • the vertical coordinate is identified by using a number.
  • one level of directory may include a plurality of subdirectories that are in a tree structure (not shown in FIG. 4B ). This is not further limited herein.
  • the first device performs data reconstruction on the target data based on the two-dimensional index table, to obtain a two-dimensional data block pool.
  • the first device performs data reconstruction on the target data based on the two-dimensional index table, to obtain the two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, and each data block corresponds to one pair of coordinates in the two-dimensional index table.
  • the first device divides, based on the generated two-dimensional index table, data that needs to be transmitted into data blocks, and reconstructs and sorts the data blocks to obtain the two-dimensional data block pool corresponding to the two-dimensional index table.
  • Each data block in the two-dimensional data block pool corresponds to the coordinates in the two-dimensional index table.
  • coordinates of a first data block are (1, A), and correspondingly, a data size may be 100 MB.
  • Each data block includes header field information and content information.
  • the header field information includes at least coordinates and a check code of the data block, the coordinates are used to indicate a location of the data block in the two-dimensional data block pool, and the check code is used to check correctness of the data block.
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • the header field information may further include related information such as a target path, a source file name, a target file name, a file offset address, and a data block size.
  • a current network delay needs to be referenced when each data block is divided.
  • a size of each data block is negatively correlated to a network delay.
  • a larger network delay indicates a smaller size of a data block, such that a transmission rate is higher, and a quantity of transmission congestion situation is reduced.
  • the quantity of concurrent threads may be increased to improve transmission efficiency.
  • the quantity of concurrent threads is positively correlated to the network bandwidth.
  • a larger network bandwidth indicates a larger quantity of concurrent threads.
  • the quantity of concurrent threads is also related to the size of the preset thread pool.
  • a larger thread pool indicates a larger quantity of concurrent threads.
  • data content of a plurality of data blocks may jointly form a large file, one data block may form a standard file, or a plurality of small files may form a data block.
  • the standard file is a file whose size is the same as that of a specified data block, for example, a 1 MB file.
  • the large file is a file whose size is greater than that of the specified data block, for example, a 1 GB file.
  • the small file is a file whose size is less than that of the specified data block, for example, a 100 KB file. As shown in FIG.
  • 26 data blocks form a file 1 ( 411 ), one data block forms a file 2 ( 412 ), one data block forms a file 3 ( 413 ), and one data block includes a file 4 ( 414 ) and a file 5 ( 415 ).
  • the first device obtains the plurality of data blocks through indexing by using the two-dimensional index table, and sends the plurality of data blocks to the second device.
  • the first device obtains the plurality of data blocks through indexing by using the two-dimensional index table, and sends the plurality of data blocks to the second device.
  • the first device obtains each data block in the two-dimensional data block pool through indexing based on the two-dimensional index table, compresses each data block, and then sends, to the second device by using a network, a compressed data packet corresponding to each data block.
  • the first device may obtain a corresponding data request.
  • the data request is used to instruct the first device to send the target data to the second device.
  • the target data may include one or more files. Details are not described herein.
  • the second device generates a corresponding two-dimensional index table based on a plurality of pairs of coordinates.
  • the second device obtains, based on the received data blocks, coordinates included in all the data blocks, where the coordinates included in all the data blocks are different; and sorts the plurality of pairs of obtained coordinates according to a preset rule, to obtain the corresponding two-dimensional index table.
  • the second device needs to decompress the received data packets to obtain one or more data blocks.
  • the second device ranks each of the plurality of data blocks based on the two-dimensional index table, to obtain a two-dimensional data block pool.
  • the second device ranks each of the one or more data blocks based on the two-dimensional index table to obtain the two-dimensional data block pool.
  • the second device quickly locates the data block by using two-dimensional coordinates (x,y) in the two-dimensional index table.
  • Header field information of each data block carries one pair of coordinates, and files with different sizes or content of a specified location offset of a large file may be loaded onto content information of each data block, to facilitate a content check after data transmission. Setting a data block size algorithm and the quantity of concurrent threads is a key factor.
  • the second device after receiving the data block, the second device automatically combines the data block with another associated data block to form the large file. Further, the second device determines a location of the data block in the large file based on first indication information carried in header field information of the data block, and combines the data block with the another associated data block to form the large file for separate storage.
  • the second device after receiving the data block, the second device automatically divides the data block into the plurality of small files. Further, the second device determines a location of each small file in the data block based on second indication information carried in header field information of the data block, and separately stores the plurality of small files included in the data block.
  • the first device reconstructs the target data that needs to be transmitted, to obtain the two-dimensional data block pool.
  • the first device can quickly determine each data block through locating based on the two-dimensional data block pool, and send the two-dimensional data block pool to the second device, such that data traversal duration is reduced, to improve data search efficiency and improve data transmission efficiency.
  • FIG. 5 another embodiment of a data transmission method in an embodiment of this application includes the following steps.
  • a first device generates a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted.
  • the first device generates the two-dimensional index table based on the preset multi-thread sequence traversal algorithm and the size of the target data that needs to be transmitted, where the two-dimensional index table is used to indicate a storage location of each piece of data in the target data.
  • the first device first determines the size of the target data that needs to be transmitted to a second device, and then divides the target data into a plurality of data blocks with standard sizes based on the size of the target data, where sizes of all the data blocks are the same.
  • the first device traverses all data blocks in the target data by using the preset multi-thread sequence traversal algorithm, to obtain a data directory, where the data directory includes at least one level of directory.
  • the first device determines a horizontal coordinate and a vertical coordinate of the two-dimensional index table based on the size of the target data and the at least one level of directory, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups, and the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group.
  • the first device generates the two-dimensional index table.
  • the first device configures one or more threads for each level of directory, in other words, correspondingly allocates a thread to each group, where each thread is used to search a data block in a corresponding group; and determines a quantity of concurrent threads based on the quantity of groups and a preset thread pool.
  • the first device performs data reconstruction on the target data based on the two-dimensional index table, to obtain a two-dimensional data block pool.
  • the first device performs data reconstruction on the target data based on the two-dimensional index table to obtain the two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, and each data block corresponds to one pair of coordinates in the two-dimensional index table.
  • the first device divides, based on the generated two-dimensional index table, data that needs to be transmitted into data blocks, and reconstructs and sorts the data blocks to obtain the two-dimensional data block pool corresponding to the two-dimensional index table.
  • Each data block in the two-dimensional data block pool corresponds to the coordinates in the two-dimensional index table.
  • Step 501 and step 502 are similar to step 401 and step 402 . Details are not described herein again.
  • the first device generates a check code of each data block.
  • the first device generates the check code of each data block, where the check code is used to check correctness of each data block.
  • the generated check code may be a check code generated by using a message-digest algorithm 5 (MD5), where the check code is used to perform a check to determine consistency of data content.
  • MD5 message-digest algorithm 5
  • the check code may alternatively be generated by using another existing algorithm. This is not further limited herein.
  • the first device obtains a plurality of data blocks through indexing by using the two-dimensional index table, and sends the plurality of data blocks to the second device.
  • the first device obtains the plurality of data blocks through indexing by using the two-dimensional index table, and sends the plurality of data blocks to the second device.
  • the first device obtains each data block in the two-dimensional data block pool through indexing based on the two-dimensional index table, compresses each data block, and then sends, to the second device by using a network, a compressed data packet corresponding to each data block.
  • the first device may obtain a corresponding data request.
  • the data request is used to instruct the first device to send the target data to the second device.
  • the target data may include one or more files. Details are not described herein.
  • the second device performs a check based on the check code carried in each data block, to determine correctness of each data block.
  • the second device performs the check based on the check code carried in each data block, to determine correctness of each data block.
  • step 507 is performed. If one or more of the received data blocks is/are incorrect or fails/fail to be transmitted, step 506 is performed.
  • the second device before performing a data check, the second device further needs to decompress the received compressed data packet, to obtain the plurality of data blocks.
  • the second device obtains a retransmitted data block sent by the first device.
  • the second device obtains the retransmitted data block retransmitted by the first device, where the retransmitted data block is a data block that fails to be transmitted or is incorrectly transmitted and that needs to be retransmitted.
  • the second device sends a retransmission request to the first device, where the retransmission request carries target coordinates indicating a location of a target data block.
  • the first device determines the location of the target data block in the two-dimensional data block pool based on the target coordinates.
  • the first device re-obtains the target data block based on the location of the target data block.
  • the first device sends the re-obtained target data block to the second device based on the received retransmission request, and then the second device checks the data block. If the check fails, retransmission continues to be performed; or if the check succeeds, subsequent steps continue to be performed.
  • a transmission process is interrupted or abnormal, in other words, a piece of data is incorrectly transmitted or fails to be transmitted, transmission is resumed from an abnormal place or is started from the beginning, and consequently a time is consumed, to be specific, a data repair time is long, and repair efficiency is low.
  • one or more data blocks are incorrectly transmitted or fail to be transmitted, only the data block that is incorrectly transmitted or fails to be transmitted needs to be retransmitted, and another data block does not need to be retransmitted, such that a short time is consumed, and data repair efficiency is high.
  • the second device generates a corresponding two-dimensional index table based on a plurality of pairs of coordinates.
  • the second device obtains, based on the received data blocks, coordinates included in all the data blocks, where the coordinates included in all the data blocks are different; and sorts the plurality of pairs of obtained coordinates according to a preset rule, to obtain the corresponding two-dimensional index table.
  • the second device needs to decompress the received data packets to obtain one or more data blocks.
  • the second device ranks each of the plurality of data blocks based on the two-dimensional index table, to obtain a two-dimensional data block pool.
  • the second device ranks each of the plurality of data blocks based on the two-dimensional index table to obtain the two-dimensional data block pool.
  • the second device quickly locates the data block by using two-dimensional coordinates (x,y) in the two-dimensional index table.
  • Header field information of each data block carries one pair of coordinates, and files with different sizes or content of a specified location offset of a large file may be loaded onto content information of each data block, to facilitate a content check after data transmission. Setting a data block size algorithm and the quantity of concurrent threads is a key factor.
  • the second device after receiving the data block, the second device automatically combines the data block with another associated data block to form the large file. Further, the second device determines a location of the data block in the large file based on first indication information carried in header field information of the data block, and combines the data block with the another associated data block to form the large file for separate storage.
  • the second device after receiving the data block, the second device automatically divides the data block into the plurality of small files. Further, the second device determines a location of each small file in the data block based on second indication information carried in header field information of the data block, and separately stores the plurality of small files included in the data block.
  • Step 507 and step 508 are similar to step 404 and step 405 . Details are not described herein again.
  • the first device reconstructs the target data that needs to be transmitted to obtain the two-dimensional data block pool, and sends the two-dimensional data block pool to the second device; and resends, to the second device, the data block that is incorrectly transmitted or fails to be transmitted.
  • the first device can quickly determine each data block through locating, such that while data traversal duration is reduced, data block retransmission duration is reduced, and data repair efficiency is improved.
  • an embodiment of a first device in an embodiment of this application includes:
  • a first generation unit 601 configured to generate a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted, where the two-dimensional index table is used to indicate a storage location of each piece of data in the target data;
  • a reconstruction unit 602 configured to perform data reconstruction on the target data based on the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, and each data block corresponds to one pair of coordinates in the two-dimensional index table;
  • a first sending unit 603 configured to: obtain the plurality of data blocks through indexing by using the two-dimensional index table, and send the plurality of data blocks to a second device.
  • the first device reconstructs the target data that needs to be transmitted to obtain the two-dimensional data block pool, and sends the two-dimensional data block pool to the second device, such that the second device can quickly determine each data block through locating, to reduce data traversal duration and improve data search efficiency.
  • FIG. 7 another embodiment of a first device in an embodiment of this application includes:
  • a first generation unit 701 configured to generate a two-dimensional index table based on a preset multi-thread sequence traversal algorithm and a size of target data that needs to be transmitted, where the two-dimensional index table is used to indicate a storage location of each piece of data in the target data;
  • a reconstruction unit 702 configured to perform data reconstruction on the target data based on the two-dimensional index table to obtain a two-dimensional data block pool, where the two-dimensional data block pool includes a plurality of data blocks, and each data block corresponds to one pair of coordinates in the two-dimensional index table;
  • a first sending unit 703 configured to: obtain the plurality of data blocks through indexing by using the two-dimensional index table, and send the plurality of data blocks to a second device.
  • the first generation unit 701 includes:
  • a traversal module 7011 configured to traverse, based on the preset multi-thread sequence traversal algorithm, the target data that needs to be transmitted, to obtain a data directory, where the data directory includes at least one level of directory;
  • a determining module 7012 configured to determine a horizontal coordinate and a vertical coordinate of the two-dimensional index table based on the size of the target data and the at least one level of directory, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data, and the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group; and
  • a generation module 7013 configured to generate the two-dimensional index table.
  • a size of each data block is negatively correlated to a network delay; and a quantity of threads is positively correlated to a network bandwidth.
  • the first device further includes:
  • a second generation unit 704 configured to generate a check code of each data block, where the check code is used to check correctness of each data block.
  • the data block includes header field information and content information.
  • the header field information includes at least coordinates and the check code of the data block, and the coordinates are used to indicate a location of the data block in the two-dimensional data block pool.
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • the first device further includes:
  • a receiving unit 705 configured to: if a first data block fails to be transmitted, receive a retransmission request sent by the second device, where the retransmission request includes first coordinates of the first data block;
  • a determining unit 706 configured to determine a location of the first data block in the two-dimensional data block pool based on the first coordinates
  • an obtaining unit 707 configured to re-obtain the first data block based on the location of the first data block
  • a second sending unit 708 configured to send the re-obtained first data block to the second device.
  • header field information of the target data block further includes first indication information, where the first indication information is used to indicate a location of each target data block in the large file, and the large file is formed after transmission is completed.
  • header field information of the target data block further includes second indication information, where the second indication information is used to indicate a location of each small file included in the target data block, and the small files are obtained through division after transmission is completed.
  • an embodiment of a second device in an embodiment of this application includes:
  • a first receiving unit 801 configured to receive a plurality of data blocks sent by a first device, where each data block includes one pair of coordinates;
  • a generation unit 802 configured to generate a corresponding two-dimensional index table based on a plurality of pairs of coordinates, where the two-dimensional index table is used to indicate storage locations of the plurality of data blocks;
  • a ranking unit 803 configured to rank each of the plurality of data blocks based on the two-dimensional index table, to obtain a two-dimensional data block pool, where the two-dimensional data block pool is target data that needs to be transmitted.
  • the second device generates the two-dimensional index table based on the received data block, and ranks the received data block based on the two-dimensional index table, to obtain the two-dimensional data block pool.
  • the two-dimensional data block pool includes data blocks corresponding to different coordinates, such that data traversal duration of the second device is reduced, and data search efficiency is improved.
  • another embodiment of a second device in an embodiment includes:
  • a first receiving unit 901 configured to receive a plurality of data blocks sent by a first device, where each data block includes one pair of coordinates;
  • a generation unit 902 configured to generate a corresponding two-dimensional index table based on a plurality of pairs of coordinates, where the two-dimensional index table is used to indicate storage locations of the plurality of data blocks;
  • a ranking unit 903 configured to rank each of the plurality of data blocks based on the two-dimensional index table, to obtain a two-dimensional data block pool, where the two-dimensional data block pool is target data that needs to be transmitted.
  • the generation unit 902 includes:
  • a first determining module 9021 configured to determine a vertical coordinate of the two-dimensional index table based on a second value in the coordinates, where the vertical coordinate of the two-dimensional index table is used to indicate a quantity of groups of the target data;
  • a second determining module 9022 configured to determine a horizontal coordinate of the two-dimensional index table based on a first value in the coordinates, where the horizontal coordinate of the two-dimensional index table is used to indicate a quantity of data blocks corresponding to each group;
  • a generation module 9023 configured to generate the corresponding two-dimensional index table based on the horizontal coordinate and the vertical coordinate.
  • a size of each data block is negatively correlated to a network delay.
  • the quantity of threads is positively correlated to a network bandwidth.
  • the second device further includes:
  • a check unit 904 configured to perform a check based on a check code carried in each data block, to determine correctness of each data block.
  • the data block includes header field information and content information.
  • the header field information includes at least coordinates and the check code of the data block, and the coordinates are used to indicate a location of the data block in the two-dimensional data block pool.
  • the content information includes data content indicated by the header field information, and the data content is included in the target data.
  • the second device further includes:
  • an obtaining unit 905 configured to: if a first data block fails to be transmitted, determine first coordinates of the first data block;
  • a sending unit 906 configured to send a retransmission request to the first device, where the retransmission request includes the first coordinates;
  • a second receiving unit 907 configured to receive the first data block resent by the first device.
  • the first device and the second device in the embodiments of this application are respectively described in detail above from a perspective of a modular function entity in FIG. 6 to FIG. 9 , and devices in embodiments of this application are described in detail below from a perspective of hardware processing.
  • FIG. 10 is a schematic structural diagram of a device 1000 according to an embodiment of this application.
  • the device may be a server.
  • the device 1000 may differ greatly due to different configurations or performance, and may include one or more processors (CPU) 1001 (for example, one or more processors), a memory 1009 , and one or more storage media 1008 (for example, one or more mass storage devices) that store an application program 1007 or data 1006 .
  • the memory 1009 and the storage medium 1008 may be transient storages or persistent storages.
  • the program stored in the storage medium 1008 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the device.
  • the processor 1001 may be configured to: communicate with the storage medium 1008 , and perform, on the device 1000 , the series of instruction operations in the storage medium 1008 .
  • the device 1000 may further include one or more power supplies 1002 , one or more wired or wireless network interfaces 1003 , one or more input/output interfaces 1004 , and/or one or more operating systems 1005 , for example, Windows Serve, Mac OS X, Unix, Linux, and FreeBSD.
  • Windows Serve Windows Serve
  • Mac OS X Unix
  • Linux FreeBSD
  • FIG. 10 a structure of the device shown in FIG. 10 does not constitute a limitation on the device, and the device may include more or fewer components than those shown in the figure, combine some components, or have different component arrangements.
  • the processor 1001 is a control center of the device, and may perform processing according to a specified data transmission method.
  • the processor 1001 is coupled to all parts of the entire device by using various interfaces and lines, and performs various functions of the device and data processing by running or executing a software program and/or a module stored in the memory 1009 and invoking data stored in the memory 1009 , to implement data transmission.
  • the memory 1009 may be configured to store the software program and the module.
  • the processor 1001 executes various function applications of the device 1000 and data processing by running the software program and the module stored in the memory 1009 .
  • the memory 1009 may mainly include a program storage area and a data storage area.
  • the program storage area may store an operating system, an application program required by at least one function (for example, generation of a two-dimensional data block pool), and the like.
  • the data storage area may store data (for example, generation of a two-dimensional index table) created based on use of the device, and the like.
  • the memory 1009 may include a high-speed random access memory, and may further include a nonvolatile memory, for example, at least one magnetic disk storage device, a flash memory device, or another volatile solid-state storage device.
  • a program of the data transmission method provided in the embodiment of this application and a received data flow are stored in the memory.
  • the processor 1001 invokes the program and the received data flow from the memory 1009 .
  • the device may be a terminal device.
  • FIG. 11 is a block diagram of a partial structure of a device according to an embodiment of this application.
  • the terminal device includes components such as a radio frequency (RF) circuit 1110 , a memory 1120 , an input unit 1130 , a display unit 1140 , a sensor 1150 , an audio circuit 1160 , a wireless fidelity (WIFI) module 1170 , and a processor 1180 .
  • RF radio frequency
  • WIFI wireless fidelity
  • FIG. 11 a structure of the terminal device shown in FIG. 11 does not constitute a limitation on the terminal device, and the terminal device may include more or fewer components than those shown in the figure, combine some components, or have different component arrangements.
  • the processor 1180 is a control center of the terminal device.
  • the processor 1180 may generate a two-dimensional index table based on a preset multi-thread sequence traversal algorithm, reconstruct target data based on the two-dimensional index table to obtain a two-dimensional data block pool, and send each data block in the two-dimensional data block pool to another device coupled to the processor 1180 .
  • the RF circuit 1110 is coupled to the processor 1180 by using a bus, and is responsible for sending data to the Internet or receiving data from the Internet.
  • the RF circuit 1110 may be further configured to: receive and send signals in an information receiving/sending process, for example, send each data block in the two-dimensional data block pool to a second device.
  • a first device after receiving a retransmission request sent by the second device, a first device sends the retransmission request to the processor 1180 for processing.
  • the RF circuit 1110 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.
  • LNA low noise amplifier
  • the RF circuit 1110 may further communicate with a network and another device through wireless communication.
  • Any communications standard or protocol may be used for the wireless communication, including but not limited to a global system for mobile communications (GSM), a general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), an e-mail, a short message service (SMS), and the like.
  • GSM global system for mobile communications
  • GPRS general packet radio service
  • CDMA code division multiple access
  • WCDMA wideband code division multiple access
  • LTE long term evolution
  • SMS short message service
  • the memory 1120 may be configured to store a software program and a module.
  • the processor 1180 executes various function applications of the terminal device and data processing by running the software program and the module stored in the memory 1120 .
  • the memory 1120 may mainly include a program storage area and a data storage area.
  • the program storage area may store an operating system, an application program required by at least one function (for example, generation of a two-dimensional index table based on a preset multi-thread sequence traversal algorithm), and the like.
  • the data storage area may store data (for example, a two-dimensional data block pool) created based on use of the terminal device, and the like.
  • the memory 1120 may include a high-speed random access memory, and may further include a nonvolatile memory, for example, at least one magnetic disk storage device, a flash memory device, or another volatile solid-state storage device.
  • the disclosed system, apparatus, and method may be implemented in another manner.
  • the apparatus embodiment described above is merely an example.
  • division into the units is merely logical function division and may be other division in an actual application.
  • a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed.
  • the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces.
  • the indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.
  • the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, and may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.
  • function units in the embodiments of this application may be integrated into one processing unit, each of the units may exist alone physically, or two or more units are integrated into one unit.
  • the foregoing integrated unit may be implemented in a form of hardware, or may be implemented in a form of a software function unit.
  • the integrated unit When the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, the integrated unit may be stored in a computer readable storage medium.
  • the computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to perform all or some of the steps of the methods described in the embodiments of this application.
  • the foregoing storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Power Engineering (AREA)
  • Information Transfer Between Computers (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
US17/073,438 2018-04-23 2020-10-19 Data transmission method and related device Active 2039-04-12 US11476977B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
CN201810369971.1 2018-04-23
CN201810369971.1A CN108809514B (zh) 2018-04-23 2018-04-23 一种数据传输方法及相关设备
PCT/CN2019/080823 WO2019205897A1 (zh) 2018-04-23 2019-04-01 一种数据传输方法及相关设备

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/080823 Continuation WO2019205897A1 (zh) 2018-04-23 2019-04-01 一种数据传输方法及相关设备

Publications (2)

Publication Number Publication Date
US20210037098A1 US20210037098A1 (en) 2021-02-04
US11476977B2 true US11476977B2 (en) 2022-10-18

Family

ID=64093166

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/073,438 Active 2039-04-12 US11476977B2 (en) 2018-04-23 2020-10-19 Data transmission method and related device

Country Status (4)

Country Link
US (1) US11476977B2 (de)
EP (1) EP3780438B1 (de)
CN (1) CN108809514B (de)
WO (1) WO2019205897A1 (de)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108809514B (zh) 2018-04-23 2021-01-12 华为技术有限公司 一种数据传输方法及相关设备
CN111200479B (zh) * 2018-11-19 2022-04-12 福建天泉教育科技有限公司 传输数据的校验方法、存储介质
CN109887456A (zh) * 2019-01-17 2019-06-14 硅谷数模半导体（北京）有限公司 数据压缩方法及装置
CN113160187B (zh) * 2021-04-27 2022-02-15 圣名科技(广州)有限责任公司 一种设备的故障检测方法及装置
CN117785499A (zh) * 2022-09-22 2024-03-29 华为技术有限公司 聚合通信中的数据传输方法以及系统
CN115576899B (zh) * 2022-12-09 2023-03-21 深圳市木浪云科技有限公司 构建索引的方法和装置以及文件查找方法和装置
CN117688593B (zh) * 2024-02-02 2024-04-30 新汽有限公司 一种网络大数据的管理系统

Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030126201A1 (en) * 2001-11-30 2003-07-03 Khoi Hoang Efficient storage of data files received in a non-sequential manner
US20040123039A1 (en) * 2002-12-23 2004-06-24 International Business Machines Corporation System and method for adatipvely loading input data into a multi-dimensional clustering table
WO2006071866A2 (en) 2004-12-24 2006-07-06 Aspera, Inc. Bulk data transfer
US7203372B1 (en) * 2003-01-15 2007-04-10 Cisco Technology, Inc. Extension of two-dimensional variable length coding for image compression
US20070124415A1 (en) * 2005-11-29 2007-05-31 Etai Lev-Ran Method and apparatus for reducing network traffic over low bandwidth links
US20080021914A1 (en) * 2006-07-21 2008-01-24 Eric John Davies Database adapter for relational datasets
CN101782921A (zh) 2009-12-28 2010-07-21 北京握奇数据系统有限公司 一种目录创建、查询和删除方法及装置
CN101945207A (zh) 2010-09-17 2011-01-12 福建新大陆通信科技股份有限公司 一种基于机顶盒升级数据的二维索引表快速缓冲方法
CN103019895A (zh) 2012-12-28 2013-04-03 华为技术有限公司 文件存储方法及装置
CN103116555A (zh) 2013-03-05 2013-05-22 中国人民解放军国防科学技术大学 基于多体并行缓存结构的数据访问方法
US8707141B1 (en) * 2011-08-02 2014-04-22 Cisco Technology, Inc. Joint optimization of packetization and error correction for video communication
WO2014201942A1 (en) 2013-06-19 2014-12-24 Tencent Technology (Shenzhen) Company Limited Method, server and system for managing content in content delivery network
CN104753626A (zh) 2013-12-25 2015-07-01 华为技术有限公司 一种数据压缩方法、设备及系统
CN105302550A (zh) 2015-10-12 2016-02-03 江苏中威科技软件系统有限公司 将页面转为版式数据流文件的方法及系统
US9280457B2 (en) * 2008-04-18 2016-03-08 Netapp, Inc. System and method for volume block number to disk block number mapping
WO2016061925A1 (zh) 2014-10-22 2016-04-28 中兴通讯股份有限公司 一种路由表的维护方法、装置及存储介质
CN105677904A (zh) 2016-02-04 2016-06-15 杭州数梦工场科技有限公司 基于分布式文件系统的小文件存储方法及装置
CN105933412A (zh) 2016-04-20 2016-09-07 北京云宏信达信息科技有限公司 文件接收方法和装置及文件传输系统
CN106528830A (zh) 2016-11-16 2017-03-22 华为技术有限公司 一种恢复文件索引目录的方法和装置
WO2018037495A1 (ja) 2016-08-24 2018-03-01 株式会社日立製作所 転送装置及びネットワークシステム
CN108809514A (zh) 2018-04-23 2018-11-13 华为技术有限公司 一种数据传输方法及相关设备
CN110784528A (zh) 2019-10-22 2020-02-11 北京天融信网络安全技术有限公司 一种文件下载方法、装置和存储介质

Patent Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030126201A1 (en) * 2001-11-30 2003-07-03 Khoi Hoang Efficient storage of data files received in a non-sequential manner
US20040123039A1 (en) * 2002-12-23 2004-06-24 International Business Machines Corporation System and method for adatipvely loading input data into a multi-dimensional clustering table
US7203372B1 (en) * 2003-01-15 2007-04-10 Cisco Technology, Inc. Extension of two-dimensional variable length coding for image compression
WO2006071866A2 (en) 2004-12-24 2006-07-06 Aspera, Inc. Bulk data transfer
US20070124415A1 (en) * 2005-11-29 2007-05-31 Etai Lev-Ran Method and apparatus for reducing network traffic over low bandwidth links
US20080021914A1 (en) * 2006-07-21 2008-01-24 Eric John Davies Database adapter for relational datasets
US9280457B2 (en) * 2008-04-18 2016-03-08 Netapp, Inc. System and method for volume block number to disk block number mapping
CN101782921A (zh) 2009-12-28 2010-07-21 北京握奇数据系统有限公司 一种目录创建、查询和删除方法及装置
CN101945207A (zh) 2010-09-17 2011-01-12 福建新大陆通信科技股份有限公司 一种基于机顶盒升级数据的二维索引表快速缓冲方法
US8707141B1 (en) * 2011-08-02 2014-04-22 Cisco Technology, Inc. Joint optimization of packetization and error correction for video communication
CN103019895A (zh) 2012-12-28 2013-04-03 华为技术有限公司 文件存储方法及装置
CN103116555A (zh) 2013-03-05 2013-05-22 中国人民解放军国防科学技术大学 基于多体并行缓存结构的数据访问方法
WO2014201942A1 (en) 2013-06-19 2014-12-24 Tencent Technology (Shenzhen) Company Limited Method, server and system for managing content in content delivery network
CN104753626A (zh) 2013-12-25 2015-07-01 华为技术有限公司 一种数据压缩方法、设备及系统
WO2016061925A1 (zh) 2014-10-22 2016-04-28 中兴通讯股份有限公司 一种路由表的维护方法、装置及存储介质
CN105302550A (zh) 2015-10-12 2016-02-03 江苏中威科技软件系统有限公司 将页面转为版式数据流文件的方法及系统
CN105677904A (zh) 2016-02-04 2016-06-15 杭州数梦工场科技有限公司 基于分布式文件系统的小文件存储方法及装置
CN105933412A (zh) 2016-04-20 2016-09-07 北京云宏信达信息科技有限公司 文件接收方法和装置及文件传输系统
WO2018037495A1 (ja) 2016-08-24 2018-03-01 株式会社日立製作所 転送装置及びネットワークシステム
CN106528830A (zh) 2016-11-16 2017-03-22 华为技术有限公司 一种恢复文件索引目录的方法和装置
CN108809514A (zh) 2018-04-23 2018-11-13 华为技术有限公司 一种数据传输方法及相关设备
CN110784528A (zh) 2019-10-22 2020-02-11 北京天融信网络安全技术有限公司 一种文件下载方法、装置和存储介质

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Lkhagvasuren et al, "Multi-level Byte Index Chunking Mechanism for File Synchronization", International Journal of Software Engineering and Its Applications, vol. 8, No. 3, pp. 339-350. (Year: 2014). *
R. Fielding et al,"Hypertext Transfer Protocol—HTTP/1.1",Network Working Group, RFC 2616, Jun. 1999, total 114 pages.
Xia et al, "Similarity and Locality Based Indexing for High Performance Data Deduplication", IEEE transactions on computers, vol. 64, No. 4, pp. 1162-1176. (Year: 2015). *

Also Published As

Publication number Publication date
CN108809514B (zh) 2021-01-12
WO2019205897A1 (zh) 2019-10-31
EP3780438A4 (de) 2021-04-21
US20210037098A1 (en) 2021-02-04
CN108809514A (zh) 2018-11-13
EP3780438A1 (de) 2021-02-17
EP3780438B1 (de) 2023-10-25

Similar Documents

Publication Publication Date Title
US11476977B2 (en) Data transmission method and related device
US11012892B2 (en) Resource obtaining method, apparatus, and system
US10333835B2 (en) Packet transmission method, apparatus, and system
CN110489059B (zh) 数据集群存储的方法、装置及计算机设备
BR112019018902A2 (pt) método de comunicação sem fios, e dispositivo terminal
US11032827B2 (en) Route selection method and apparatus
US11102322B2 (en) Data processing method and apparatus, server, and controller
CN110659151A (zh) 数据校验方法及装置，存储介质
CN110727738B (zh) 基于数据分片的全局路由系统、电子设备及存储介质
CN113746692A (zh) 网络流量统计的方法、电子设备及存储介质
CN103326925B (zh) 一种消息推送方法及装置
US20140156809A1 (en) Data downloading method, terminal, server, and system
CN108243056B (zh) 一种获取异常信息的方法及装置
CN111049750B (zh) 一种报文转发方法、系统及设备
CN111294285B (zh) 一种网络数据的分发方法及负载均衡器
CN112165505A (zh) 去中心化的数据处理方法、电子装置和存储介质
US20180027525A1 (en) Resource Allocation Method and Device and Information Feedback Method and Device
CN112995053A (zh) 发送报文的方法及装置
CN106209666B (zh) 一种基于负载均衡器的链路复用方法及系统
CN110636005B (zh) 知识中心网络的知识路由方法及装置
CN110912969B (zh) 一种文件高速传输源节点、目的节点装置及系统
CN116028196A (zh) 数据处理方法、装置及存储介质
CN113438274A (zh) 数据传输方法、装置、计算机设备及可读存储介质
US20210377197A1 (en) A mail transmission method, server and system
CN109002296A (zh) 一种软件下载方法、装置及基站

Legal Events

Date Code Title Description
AS Assignment

Owner name: HUAWEI TECHNOLOGIES CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:FAN, YONG;REEL/FRAME:054088/0211

Effective date: 20201019

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

CC Certificate of correction