CN113992658A

CN113992658A - Data transmission method and related device, equipment, system and storage medium

Info

Publication number: CN113992658A
Application number: CN202111265967.9A
Authority: CN
Inventors: 张致江; 张明; 黄开元; 王芝斌; 夏静霆; 殷奎
Original assignee: iFlytek Co Ltd
Current assignee: iFlytek Co Ltd
Priority date: 2021-10-28
Filing date: 2021-10-28
Publication date: 2022-01-28
Anticipated expiration: 2041-10-28
Also published as: CN113992658B

Abstract

The application discloses a data transmission method, and a related device, equipment, system and storage medium, wherein the data transmission method comprises the following steps: acquiring a cluster addressing request related to a target file; the cluster addressing request is obtained by preprocessing a first request message of the user equipment by the gateway equipment, and the first request message comprises a file name and a file operation of the target file; responding to the cluster addressing request, and inquiring a metadata center based on file operation to obtain a target cluster; and file operations are executed in the target cluster; based on the file name and the cluster address of the target cluster, generating an access protocol message matched with the cluster type of the target cluster, and sending the access protocol message to the gateway equipment; and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file. By the scheme, the user can access without perception, and meanwhile, the data transmission efficiency is improved and the cost is reduced.

Description

Data transmission method and related device, equipment, system and storage medium

Technical Field

The present application relates to the field of computer communications technologies, and in particular, to a data transmission method, and a related apparatus, device, system, and storage medium.

Background

In the field of back-end storage, development of the method is often gradual, clusters such as swift and the like bring more convenience to usability and storage support of small files. However, as the service data grows, problems such as incomplete data download, inconvenience in expansion/capacity expansion, and the like are gradually revealed.

Thanks to the continuous development of the storage technology, by deploying clusters such as ceph and the like, the user service requirements can be solved by utilizing the strong consistency of the clusters, and the problems can be alleviated. However, in the practical application process, there often exists a certain obstacle to data intercommunication between clusters, and especially, it is still difficult to implement data intercommunication between heterogeneous clusters such as the swift cluster and the ceph cluster. The prior art is generally realized by integrally migrating data of a certain cluster (such as a swift cluster) to another cluster (such as a ceph cluster), and this way cannot achieve user imperceptible access on one hand, so user experience is often poor, and on the other hand, the problems of low efficiency and high cost exist. In view of the above, how to implement user unaware access, and at the same time, improve data transmission efficiency and reduce cost.

Disclosure of Invention

The technical problem mainly solved by the application is to provide a data transmission method, a related device, equipment, a system and a storage medium, which can realize the non-perception access of a user, and simultaneously improve the data transmission efficiency and reduce the cost.

In order to solve the above technical problem, a first aspect of the present application provides a data transmission method, including: acquiring a cluster addressing request related to a target file; the cluster addressing request is obtained by preprocessing a first request message of user equipment by gateway equipment, and the first request message comprises a file name and file operation of a target file; responding to the cluster addressing request, and inquiring a metadata center based on file operation to obtain a target cluster; wherein, the file operation is executed in the target cluster; based on the file name and the cluster address of the target cluster, generating an access protocol message matched with the cluster type of the target cluster, and sending the access protocol message to the gateway equipment; and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file.

In order to solve the above technical problem, a second aspect of the present application provides a data transmission method, including: receiving query operation of forwarding equipment; the query operation is sent by the forwarding device in response to a cluster addressing request from the gateway device, the cluster addressing request is obtained by preprocessing a first request message of the user device by the gateway device and is related to a target file, and the first request message comprises a file name and a file operation of the target file; responding to the query operation, and querying to obtain a target cluster based on the file operation; the file operation is executed in the target cluster, the forwarding device generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway device, and the gateway device generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish the transmission of the target file.

In order to solve the above technical problem, a third aspect of the present application provides a data transmission method, including: receiving a first request message of user equipment; the first request message comprises a file name and a file operation of the target file; preprocessing the first request message to obtain a cluster addressing request, and sending the cluster addressing request to forwarding equipment; receiving an access protocol message generated by the forwarding device based on the cluster addressing request; the forwarding equipment responds to the cluster addressing request, queries a metadata center based on file operation to obtain a target cluster, executes a file in the target cluster, and generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster; and generating a second request message based on the access protocol message, and sending the second request message to the target cluster to finish transmitting the target file.

In order to solve the above technical problem, a fourth aspect of the present application provides a data transmission method, including: the method comprises the steps that gateway equipment receives a first request message sent by user equipment; the first request message comprises a file name and a file operation of the target file; the gateway equipment preprocesses the first request message to obtain a cluster addressing request and sends the cluster addressing request to the forwarding equipment; the forwarding equipment responds to the cluster addressing request, queries a metadata center based on file operation, and obtains a target cluster; wherein, the file operation is executed in the target cluster; the forwarding equipment generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway equipment; and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file.

In order to solve the above technical problem, a fifth aspect of the present application provides a data transmission apparatus, including: the system comprises a request acquisition module, a cluster query module and a generation and forwarding module, wherein the request acquisition module is used for acquiring a cluster addressing request related to a target file; the cluster addressing request is obtained by preprocessing a first request message of user equipment by gateway equipment, and the first request message comprises a file name and file operation of a target file; the cluster query module is used for responding to a cluster addressing request and querying a metadata center based on file operation to obtain a target cluster; wherein, the file operation is executed in the target cluster; the generation forwarding module is used for generating an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster and sending the access protocol message to the gateway equipment; and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file.

In order to solve the above technical problem, a sixth aspect of the present application provides a data transmission apparatus, including: the operation receiving module is used for receiving the query operation of the forwarding equipment; the query operation is sent by the forwarding device in response to a cluster addressing request from the gateway device, the cluster addressing request is obtained by preprocessing a first request message of the user device by the gateway device and is related to a target file, and the first request message comprises a file name and a file operation of the target file; the target query module is used for responding to query operation and querying to obtain a target cluster based on file operation; the file operation is executed in the target cluster, the forwarding device generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway device, and the gateway device generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish the transmission of the target file.

In order to solve the above technical problem, a seventh aspect of the present application provides a data transmission apparatus, including: the system comprises a first receiving module, an addressing request module, a second receiving module and a file transmission module, wherein the first receiving module is used for receiving a first request message of user equipment; the first request message comprises a file name and a file operation of the target file; the addressing request module is used for preprocessing the first request message to obtain a cluster addressing request and sending the cluster addressing request to the forwarding equipment; a second receiving module, configured to receive an access protocol message generated by the forwarding device based on the cluster addressing request; the forwarding equipment responds to the cluster addressing request, queries a metadata center based on file operation to obtain a target cluster, executes a file in the target cluster, and generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster; and the file transmission module is used for generating a second request message based on the access protocol message and sending the second request message to the target cluster so as to finish transmitting the target file.

In order to solve the above technical problem, an eighth aspect of the present application provides a forwarding device, including a memory, a communication circuit and a processor, wherein the memory and the communication circuit are coupled to the processor, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the data transmission method in the first aspect.

In order to solve the above technical problem, a ninth aspect of the present application provides a metadata center, including a memory, a communication circuit, and a processor, where the memory and the communication circuit are coupled to the processor, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the data transmission method in the second aspect.

In order to solve the above technical problem, a tenth aspect of the present application provides a gateway device, including a memory, a communication circuit and a processor, wherein the memory and the communication circuit are coupled to the processor, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the data transmission method in the third aspect.

In order to solve the above technical problem, an eleventh aspect of the present application provides a data transmission system, including a user device, a gateway device, a metadata center, a plurality of clusters, and a forwarding device, where the user device is in communication connection with the gateway device, the gateway device is in communication connection with the forwarding device and the plurality of clusters, respectively, and the forwarding device is in communication connection with the metadata center; the gateway device receives a first request message sent by user equipment, the first request message comprises a file name and a file operation of a target file, the first request message is preprocessed to obtain a cluster addressing request, the cluster addressing request is sent to the forwarding device, the forwarding device responds to the cluster addressing request, a metadata center is inquired based on the file operation to obtain a target cluster, the file operation is executed in the target cluster, an access protocol message matched with the cluster type of the target cluster is generated based on the file name and the cluster address of the target cluster, the access protocol message is sent to the gateway device, the gateway device further generates a second request message based on the access protocol message, and the second request message is sent to the target cluster to finish target file transmission.

In order to solve the above technical problem, a twelfth aspect of the present application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being for implementing the data transmission method in the first aspect, or implementing the data transmission method in the second aspect, or implementing the data transmission method in the third aspect.

In the above scheme, a cluster addressing request related to a target file is obtained, where the cluster addressing request is obtained by preprocessing a first request message of a user device by a gateway device, where the first request message includes a file name and a file operation of the target file, and on this basis, a metadata center is queried based on the file operation in response to the cluster addressing request, so as to obtain a target cluster, where the file operation is executed in the target cluster, thereby generating an access protocol message matched with a cluster type of the target cluster based on the file name and a cluster address of the target cluster, and sending the access protocol message to the gateway device, so that the gateway device generates a second request message based on the access protocol message, and sends the second request message to the target cluster, so as to complete transmission of the target file, on one hand, since overall data migration is not required, data transmission efficiency can be improved and cost can be reduced, on the other hand, the target cluster to be subjected to file operation can be obtained by inquiring the metadata center, the access protocol message matched with the cluster type of the target cluster is generated, and the gateway device generates the second request message adaptive to the target cluster according to the access protocol message, so that the user equipment is supported to access any cluster without perception. Therefore, the user can access without perception, and meanwhile, the data transmission efficiency is improved and the cost is reduced.

Drawings

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a data transmission method according to the present application;

FIG. 2 is a block diagram of an embodiment of the data transmission system of the present application;

FIG. 3 is a framework diagram of an embodiment of clusters;

FIG. 4 is a process diagram of one embodiment of a write operation in the case where the target file is a complete file;

FIG. 5 is a process diagram of one embodiment of a write operation in the case where the target file is a fragmented file;

FIG. 6 is a process diagram of another embodiment of a write operation in the case where the target file is a fragmented file;

FIG. 7 is a schematic flow chart diagram illustrating another embodiment of a data transmission method according to the present application;

FIG. 8 is a schematic flow chart diagram illustrating a data transmission method according to another embodiment of the present application;

FIG. 9 is a schematic flow chart diagram illustrating a data transmission method according to another embodiment of the present application;

FIG. 10 is a block diagram of an embodiment of a data transmission apparatus according to the present application;

FIG. 11 is a block diagram of another embodiment of a data transfer device according to the present application;

FIG. 12 is a block diagram of a data transfer device according to another embodiment of the present application;

FIG. 13 is a block diagram of an embodiment of a forwarding device of the present application;

FIG. 14 is a block diagram of a metadata center according to an embodiment of the present application;

FIG. 15 is a block diagram of a gateway device according to an embodiment of the present application;

FIG. 16 is a block diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The following describes in detail the embodiments of the present application with reference to the drawings attached hereto.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a data transmission method according to an embodiment of the present application. Specifically, the method may include the steps of:

step S11: a cluster addressing request associated with a target file is obtained.

In the embodiment of the present disclosure, the cluster addressing request is obtained by preprocessing, by the gateway device, the first request message of the user equipment, where the first request message includes a file name and a file operation of the target file.

In one implementation scenario, please refer to fig. 2 in combination, and fig. 2 is a schematic diagram of a framework of an embodiment of a data transmission system according to the present application. As shown in fig. 2, the data transmission system includes a user device, a gateway device, a forwarding device, a plurality of clusters, and a metadata center, where the user device is in communication connection with the gateway device, the gateway device is in communication connection with the forwarding device and the plurality of clusters, respectively, the forwarding device is in communication connection with the metadata center, and the forwarding device is configured to execute the steps of the data transmission method in this embodiment. In addition, the specific framework of the forwarding device may refer to the related description in any of the following embodiments of the forwarding device, which is not described herein again. In addition, the execution operations of other related devices may refer to the following related disclosure embodiments, which are not repeated herein.

In a specific implementation scenario, the relationship between any two of the clusters may be any one of a heterogeneous cluster and a homogeneous cluster. Illustratively, referring to fig. 3 in conjunction, fig. 3 is a block diagram of an embodiment of several clusters. As shown in fig. 3, the plurality of clusters may include 6 clusters, which may be numbered by numbers 1 to 6 for convenience of description, and are denoted by clu1 (i.e., cluster No. 1), clu2 (i.e., cluster No. 2), clu3 (i.e., cluster No. 3), clu4 (i.e., cluster No. 4), clu5 (i.e., cluster No. 5), and clu6 (i.e., cluster No. 6), where clu1 and clu2 are swift clusters, that is, clu1 and clu2 may be considered as homogeneous clusters, and clu3 to clu6 are ceph clusters, that is, any two clusters from clu3 to clu6 may be considered as homogeneous clusters, but any one of clu1 and clu3 to clu6 is a heterogeneous cluster, and similarly, any one of clu2 and clu3 to clu6 is also a heterogeneous cluster. Other cases may be analogized, and no one example is given here. It should be noted that, for specific meanings of the swift cluster, the ceph cluster, the isomorphic cluster and the heterogeneous cluster, reference may be made to relevant technical details, which are not described herein again.

In a specific implementation scenario, the forwarding device and the gateway device are independent of each other, that is, they are decoupled from each other, that is, the forwarding device does not depend on the gateway device, the gateway device does not depend on the forwarding device, and both devices can operate independently. Although the two servers are independent of each other, the two servers may be integrated into the same server, or the two servers may be integrated into different servers, which is not limited herein.

In a specific implementation scenario, the user device may include, but is not limited to, a desktop computer, a laptop computer, a tablet computer, a mobile phone, and the like.

In a specific implementation scenario, as shown in fig. 2, the metadata center may store location meta information, and the location meta information is used to indicate a file storage location, which may specifically refer to the following related description, and is not repeated here. In addition, in order to improve the data storage efficiency of the metadata center, as shown in fig. 2, the metadata center may be further configured with a database, so that the location meta-information may be conveniently and quickly queried through the metadata center.

In an implementation scenario, the gateway device may parse the first request information to obtain the file name and the file operation, where the file operation may specifically be any one of a write operation and a read operation. Furthermore, a message format of the first request information may be obtained through parsing, where the message format may specifically include but is not limited to: http (Hyper-Text Transfer Protocol) request header, request body, etc., which are not limited herein.

In an implementation scenario, the gateway device may encapsulate a request header of the first request message and a part of request metadata (e.g., whether cross-cluster forwarding is required, a domain name type, etc.), to obtain a new http request body, which is sent to the forwarding device as a cluster addressing request, and after the forwarding device obtains the cluster addressing request, the operations such as cluster addressing, data transmission, etc. may be performed through the steps in the embodiment of the data transmission method disclosed in this application.

Step S12: and responding to the cluster addressing request, and inquiring the metadata center based on file operation to obtain a target cluster.

In the embodiment of the disclosure, the file operation is executed in the target cluster. Illustratively, the file operation is a write operation, and the target file needs to be written to the target cluster, or the file operation is a read operation, and the target file needs to be read in the target cluster.

In one implementation scenario, the metadata center may store several first mapping tables and several second mapping tables, each first mapping table including mappings between a first storage segment and at least one second storage segment, each second mapping table including mappings between at least one candidate cluster of second storage segments. In the above manner, the target cluster and the target storage segment can be determined by performing double mapping through the first mapping table and the second mapping table, so as to implement the file operation on the target file, and the double mapping uses the storage segment as the minimum granularity.

In a specific implementation scenario, for the convenience of mapping, the first mapping table may further define a default mapping fragment, which represents a default mapping of the first storage fragment to the second storage fragment, that is, if there is no special case, the first storage fragments are all mapped to the default mapping fragment defined by the first mapping table. Similarly, the second mapping table may also define a default mapping cluster, which represents a candidate cluster for default mapping of the second storage segment, that is, if there is no special case, the second storage segments are mapped to the default mapping cluster defined by the second mapping table.

In a specific implementation scenario, for example, a plurality of clusters include a swift cluster and a ceph cluster, a container in the swift cluster is used as a storage fragment, and a bucket in the ceph cluster is used as a storage fragment, for convenience of description, in this case, the first mapping table may be referred to as CBMap (i.e., container/bucket mapping table) for short, and two attributes may be defined: an all map (AllMap) for characterizing all second storage segments (including but not limited to: contiiner, bucket) having a mapping relationship with the first storage segment, and a default map (DefaultMap) for characterizing the second storage segments (contiiner or bucket) of the default map of the second storage segments. Similarly, for convenience of description, the second mapping table may be referred to as an ACL (Access Control List), a cluster List (CV) that each second storage segment (including but not limited to: container, bucket) can Access, and may define two attributes: an all cluster (AllCV) and a default cluster (DefaultCV), wherein the all cluster is used for characterizing all candidate clusters which have mapping relation with the second storage segment, and the default cluster is used for characterizing the candidate clusters which have default mapping relation with the second storage segment.

In a specific implementation scenario, a current storage segment (e.g., container, bucket, etc.) in a current cluster where a target file is located may be obtained, and the first mapping table and the second mapping table are sequentially queried to obtain a target storage segment having a mapping relationship with the current storage segment and a target cluster having a mapping relationship with the target storage segment. Specifically, the first mapping table may be queried to obtain the target storage segment based on the second storage segment having a mapping relationship with the current storage segment. Illustratively, the second storage segment of the default mapping defined by the first mapping table may be the target storage segment. Further, a second mapping table may be queried to obtain a target cluster based on the candidate cluster having a mapping relationship with the target storage segment. Illustratively, a candidate cluster of a default mapping defined by the second mapping table may be the target cluster. According to the method, the current storage segment where the target file is located is obtained, the first mapping table and the second mapping table are sequentially inquired for double mapping, the target storage segment and the target cluster for subsequently executing file operation are obtained, and the inquiry convenience is improved.

In a specific implementation scenario, please refer to fig. 3 in combination, taking an example that the user equipment uses a contiainer name of swiftuater-continer 1, account is swiftuater-account 1, and the file name of the target file is swiftuater-object 1, that is, the first storage segment may be regarded as/v 1/swiftuater-account 1/swiftuater-continer 1, as shown in fig. 3, the second storage segment with which a mapping relationship exists includes: v1/swift-account 1/swift-container 1 (i.e., itself), ceph-bucket1, ceph-bucket2, and define a default mapping to ceph-bucket1, then the first mapping table may be expressed as { all-map [/v 1/swift-account 1 [/W ]

swift-contianer 1, ceph-bucket1, ceph-bucket2, default-map: ceph-bucket1 }. On this basis, based on the first mapping table, the target storage segment may be determined to be ceph-bucket 1. With continued reference to FIG. 3, candidate clusters that have a mapping relationship with ceph-bucket1 include: clu3 (i.e., Cluster No. 3) and clu4 (i.e., Cluster No. 4), and defines a default mapping to clu3, the second mapping table may be denoted as all-cv: [3,4], default-cv:3 ]. On this basis, based on the second mapping table, the target cluster may be determined to be clu3 (i.e., cluster No. 3). Other cases may be analogized, and no one example is given here.

In an implementation scenario, the metadata center may also store several pieces of file location information, where each piece of file location information includes a third storage segment where the stored file is located and a cluster to which the third storage segment belongs. It should be noted that, the specific storage manner of the file location information may refer to the following related description, which is not repeated herein. By the mode, the metadata center stores the file position information of the stored file, so that the stored file can be read directly according to the file position information in the follow-up process, and the file reading efficiency is improved.

In a specific implementation scenario, the position information of a plurality of files stored in a metadata center can be queried to obtain a target storage segment where a target file is located and a target cluster where the target storage segment belongs; or, as mentioned above, the metadata center may also store a plurality of first mapping tables and a plurality of second mapping tables, and may also query a plurality of file location information, a plurality of first mapping tables, and a plurality of second mapping tables stored in the metadata center to obtain the target storage segment and the target cluster, which is not limited herein.

In a specific implementation scenario, the file location information may be characterized by a URL (Uniform Resource Locator). Still taking the foregoing target file swift-object 1 as an example, the file location information may be recorded as/ceph-bucket 1/swift-object 1, and since the candidate cluster mapped by default by ceph-bucket1 may be known as clu3 (i.e. cluster No. 3) through the second mapping table { all-cv: [3,4], default-cv:3} of the second storage segment ceph-bucket1 in the metadata center, the target storage segment (i.e. ceph-bucket1) and the target cluster (i.e. clu3) where the target file is located are determined through the file location information stored in the metadata center and the second mapping table; or, the file location information may also be recorded as/clu 3/ceph-bucket 1/swift-object 1, so that the target storage segment (i.e., ceph-bucket1) and the target cluster (i.e., clu3) where the target file is located may be directly determined.

In a specific implementation scenario, after querying a plurality of file location information stored in the metadata center, if a target storage segment and a target cluster are not queried yet, double mapping may be performed based on the first mapping table and the second mapping table to perform file lookup in a plurality of clusters in the data transmission system, and after finding the target storage segment and the target cluster where the file location information is located, the file location information is recorded, and the file location information is stored in the metadata center.

Step S13: and generating an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sending the access protocol message to the gateway equipment.

In one implementation scenario, as described above, since the target storage segment (e.g., any one of container and bucket) is already determined and the target cluster is also already determined, it may be determined whether protocol conversion is required according to the cluster type of the target cluster (e.g., swift cluster or ceph cluster). For example, if the original request is the swift protocol and the target cluster is the ceph cluster, the protocol conversion is needed, whereas if the original request is the swift protocol and the target cluster is still the swift cluster, the protocol conversion is not needed.

In a specific implementation scenario, if protocol conversion is required, a protocol access message matching the cluster type is generated according to the target storage segment and the file name of the target file, for example, when the original request is a swift protocol and the target cluster is a ceph cluster, a protocol access message (i.e., s3 protocol request message) may be generated based on the target storage segment (i.e., bucket name) of the target ceph cluster and the file name (i.e., object name) of the target file, and the protocol access message is returned to the gateway device.

In a specific implementation scenario, if protocol conversion is not required, a protocol access message matching the cluster type may also be generated according to the target storage segment and the file name of the target file, for example, when the original request is a swift protocol and the target cluster is still a swift cluster, a protocol access message (i.e., the foregoing http request packet) may be generated based on the target storage segment (i.e., container name) of the target swift cluster and the file name (i.e., object name) of the target file, and the protocol access message is returned to the gateway device.

In the embodiment of the disclosure, the gateway device generates a second request message based on the access protocol message, and sends the second request message to the target cluster to complete the transmission of the target file.

In an implementation scenario, taking the access protocol message as an s3 protocol request message as an example, the gateway device may generate a second request message (e.g., a new http request message) according to the s3 protocol request message, forward the second request message to the target cluster, and return a processing result (e.g., feedback of successful writing, downloading of the target file, etc.) of the target cluster to the user equipment.

In an implementation scenario, taking the access protocol message as an http request message as an example, the gateway device may add a cluster address of the target cluster to the original first request message according to the http request message to obtain a second request message (e.g., a new http request message), forward the second request message to the target cluster, and return a processing result of the target cluster (e.g., feedback of successful writing, downloading of the target file, etc.) to the user equipment.

In some disclosed embodiments, please refer to fig. 4 in combination, fig. 4 is a schematic process diagram of an embodiment of a write operation in a case where a target file is a complete file. Note that, in the case where the target file is a complete file (i.e., a normal object), this indicates that the user writes a certain file at a time. As shown in fig. 4 and described in the related description of the foregoing disclosure, the target cluster and the target storage segment thereof may be determined by performing double mapping through the first mapping table (e.g., the foregoing CBMap) and the second mapping table (e.g., the foregoing CV), and on this basis, an access protocol message may be generated and cooperate with the gateway device and the target cluster to complete the write operation of the target file. After that, in order to facilitate subsequent reading operation on the target file, first file location information of the target file may be generated based on the target cluster and the target storage segment, and as described above, the writing operation is performed on the target storage segment in the target cluster, and based on this, the first file location information may be stored in the metadata center, so as to search for the target file based on the first file location information in the subsequent process, and the specific search process may refer to the related description in the foregoing disclosed embodiment, which is not described herein again. According to the mode, under the condition that the target file is a complete file, the first file position information of the target file is generated based on the target cluster and the target storage segment thereof, the writing operation is executed on the target storage segment in the target cluster, the first file position information is stored in the metadata center based on the first file position information, the first file position information is used for searching the target file, the follow-up reading operation on the target file can be greatly facilitated, and the reading efficiency is improved.

In one implementation scenario, as described in the foregoing disclosure, the first location information may also be represented in a key structure. Taking the target cluster as a swift cluster as an example, the first location information may be represented in the form of: the/v 1/{ account }/{ container }/{ object }, which may be expressed as: v 1/swift-account 1/swift-contianer 1/swift-object 1. Other cases may be analogized, and no one example is given here.

In one implementation scenario, similar to the foregoing description, in the case that the target cluster is a ceph cluster, the first location information may also be represented in a key structure, and the representation may be: /{ bucket }/{ object }, which may be expressed, for example, as: the/ceph-bucket 1/swift-object 1. Other cases may be analogized, and no one example is given here.

In some disclosed embodiments, please refer to fig. 5 in combination, where fig. 5 is a schematic process diagram of an embodiment of a write operation when a target file is a fragmented file. When the target file is a sharded file (i.e., manifest object), the user cuts a certain file into a plurality of continuous shards, and writes the shards sequentially. As shown in fig. 5 and the foregoing disclosure, when performing a write operation on a first file, double mapping may be performed through a first mapping table (e.g., the foregoing CBMap) and a second mapping table (e.g., the foregoing CV), so as to determine a target cluster and a target storage segment thereof, and on this basis, an access protocol message may be generated and cooperate with a gateway device and the target cluster to complete the write operation on the first file. After that, second file location information of the first file may be generated based on the target cluster and the target storage segment thereof, and the write operation is performed on the target storage segment in the target cluster, and all the fragmented files are written into the target storage segment in the target cluster, based on which, the second file location information may be stored in the metadata center, and the second file location information is used to search the target cluster and the target storage segment thereof when the write operation is performed on the non-first file, and of course, after the write operation of the target file is completed, the second file location information may also query the target cluster and the target storage segment in which the second file location information is located when the read operation is performed on the target file, and the specific process may refer to the relevant description in the foregoing disclosed embodiment, which is not described herein again. According to the mode, after the addressing operation is performed on the first file, the second file position information of the first file is generated based on the target cluster and the target storage segment of the first file, and the second file position information is stored in the metadata center, so that the writing operation of the subsequent fragmented file can be greatly facilitated, the subsequent reading operation on the target file can be facilitated, and the reading efficiency can be improved.

In an implementation scenario, the second location information may also be represented in a form of a key structure, which may specifically refer to the related description in the foregoing disclosed embodiment, and is not described herein again.

In an implementation scenario, in a case that the target file includes several fragmented files, the first request message for the fragmented file may include a fragment sequence number of the fragmented file and a fragment length of the target file. For example, in a case that the target file includes 4 fragmented files, the first request message for the first fragmented file (i.e., the first fragmented file) may include a fragment sequence number (e.g., 1) of the fragmented file and a fragment length (e.g., 4) of the target file, and further, the first request message for the second fragmented file may include a fragment sequence number (e.g., 2) of the fragmented file and a fragment length (e.g., 4) of the target file, and so on, which is not to be exemplified herein.

In an implementation scenario, please refer to fig. 6 in combination, where fig. 6 is a schematic process diagram of another embodiment of the write operation when the target file is a fragmented file. Specifically, fig. 6 is a process diagram of an embodiment of a write operation in the case of a non-first file. As shown in fig. 6, for a non-first file, a target storage fragment of a target file may be obtained based on second location information of the first file, and a candidate cluster of a default mapping of the target storage fragment is obtained as a target cluster in combination with a mapping table, on this basis, an access protocol message may be generated and may cooperate with a gateway device and the target cluster to complete a write operation of the non-first file. Further, after that, similarly to the first file, second file location information of the non-first file may be generated based on the target cluster and the target storage segment thereof, and the second file location information of the non-first file is stored in the metadata center.

Referring to fig. 7, fig. 7 is a schematic flowchart illustrating a data transmission method according to another embodiment of the present application. Specifically, the method may include the steps of:

step S71: an inquiry operation of a forwarding device is received.

In this embodiment of the present disclosure, the query operation is sent by the forwarding device in response to a cluster addressing request from the gateway device, where the cluster addressing request is obtained by preprocessing a first request message of the user equipment by the gateway device and is related to the target file, and the first request message includes a file name and a file operation of the target file, and for the related operation of the forwarding device, reference may be made to related description in the foregoing embodiment, and details are not described here again. It should be noted that, in the present embodiment, the data transmission method is executed by the metadata center in the data transmission system shown in fig. 2, and the embodiment of the present disclosure mainly describes differences from the foregoing embodiment, and the same or similar points may refer to the related description in the foregoing embodiment, and are not described again here.

Step S72: and responding to the query operation, and querying to obtain the target cluster based on the file operation.

In the embodiment of the disclosure, a file operation is executed in a target cluster, a forwarding device generates an access protocol message matched with a cluster type of the target cluster based on a file name and a cluster address of the target cluster, and sends the access protocol message to a gateway device, and the gateway device generates a second request message based on the access protocol message and sends the second request message to the target cluster to complete transmission of a target file.

In one implementation scenario, the file operation is a write operation, the metadata center stores a plurality of first mapping tables and a plurality of second mapping tables, each first mapping table includes a mapping relationship between a first storage segment and at least one second storage segment, and each second mapping table includes a mapping relationship between a second storage segment and at least one candidate cluster. For specific meanings of the first mapping table and the second mapping table, and related operations of the metadata center when the file operation is a write operation, reference may be specifically made to related descriptions in the foregoing disclosed embodiments, and details are not described here.

In one implementation scenario, the file operation is a read operation, the metadata center stores a plurality of file location information, and each file location information includes a third storage segment where the stored file is located and a cluster to which the third storage segment belongs. For the above file location information and the related operations of the metadata center when the file operation is a write operation, reference may be specifically made to the related descriptions in the foregoing disclosed embodiments, and details are not described here again.

According to the scheme, on one hand, the data transmission efficiency can be improved and the cost can be reduced because the whole data migration is not needed, on the other hand, the target cluster to be subjected to the file operation can be obtained by inquiring the metadata center, the access protocol message matched with the cluster type of the target cluster is generated, and the gateway device generates the second request message adaptive to the target cluster according to the access protocol message, so that the user equipment is supported to access any cluster without perception. Therefore, the user can access without perception, and meanwhile, the data transmission efficiency is improved and the cost is reduced.

Referring to fig. 8, fig. 8 is a schematic flowchart illustrating a data transmission method according to another embodiment of the present application. Specifically, the method may include the steps of:

step S81: a first request message of a user equipment is received.

In this embodiment of the disclosure, the first request message includes a file name and a file operation of the target file, which may specifically refer to relevant descriptions in the foregoing embodiment of the disclosure, and details are not described here again. It should be noted that, in the present embodiment, the data transmission method is executed by the gateway device in the data transmission system shown in fig. 2, and the embodiment of the present disclosure mainly describes differences from the foregoing embodiment, and the same or similar parts may refer to the related description in the foregoing embodiment, and are not described again here.

Step S82: and preprocessing the first request message to obtain a cluster addressing request, and sending the cluster addressing request to the forwarding equipment.

Reference may be made to the related description in the foregoing embodiments, which are not repeated herein.

Step S83: receiving an access protocol message generated by the forwarding device based on the cluster addressing request.

In this embodiment of the disclosure, the forwarding device responds to the cluster addressing request, queries the metadata center based on the file operation to obtain the target cluster, executes the file in the target cluster, and generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster.

Step S84: and generating a second request message based on the access protocol message, and sending the second request message to the target cluster to finish transmitting the target file.

Referring to fig. 9, fig. 9 is a schematic flowchart illustrating a data transmission method according to another embodiment of the present application. Specifically, the method may include the steps of:

step S91: the gateway device receives a first request message sent by the user device.

In this embodiment of the disclosure, the first request message includes a file name and a file operation of the target file, which may specifically refer to relevant descriptions in the foregoing embodiment of the disclosure, and details are not described here again.

Step S92: the gateway device preprocesses the first request message to obtain a cluster addressing request, and sends the cluster addressing request to the forwarding device.

Step S93: and the forwarding equipment responds to the cluster addressing request and queries the metadata center based on file operation to obtain a target cluster.

In the embodiment of the present disclosure, the file operation is executed in the target cluster, which may specifically refer to the related description in the foregoing embodiment, and is not described herein again.

Step S94: and the forwarding equipment generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway equipment.

Step S95: and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file.

Referring to fig. 10, fig. 10 is a schematic diagram of a frame of an embodiment of a data transmission device 100 according to the present application. The data transmission apparatus 100 includes: the system comprises a request acquisition module 101, a cluster query module 102 and a generation and forwarding module 103, wherein the request acquisition module 101 is used for acquiring a cluster addressing request related to a target file; the cluster addressing request is obtained by preprocessing a first request message of user equipment by gateway equipment, and the first request message comprises a file name and file operation of a target file; the cluster query module 102 is configured to respond to a cluster addressing request, query a metadata center based on a file operation, and obtain a target cluster; wherein, the file operation is executed in the target cluster; the generation forwarding module 103 is configured to generate an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and send the access protocol message to the gateway device; and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file.

In some disclosed embodiments, the file operation is a write operation, the metadata center stores a number of first mapping tables and a number of second mapping tables, each first mapping table includes a mapping relationship between a first storage segment and at least one second storage segment, and each second mapping table includes a mapping relationship between a second storage segment and at least one candidate cluster.

Therefore, the target cluster and the target storage segment can be determined by double mapping through the first mapping table and the second mapping table, so that the file operation can be performed on the target file, and the double mapping takes the storage segment as the minimum granularity.

In some disclosed embodiments, the cluster querying module 102 includes a segment obtaining sub-module, configured to obtain a current storage segment in a current cluster where the target file is located; the cluster querying module 102 includes a mapping querying submodule, configured to query the first mapping table and the second mapping table in sequence, and obtain a target storage segment having a mapping relationship with a current storage segment and a target cluster having a mapping relationship with the target storage segment.

Therefore, the current storage segment where the target file is located is obtained, and the first mapping table and the second mapping table are sequentially inquired for double mapping, so that the target storage segment and the target cluster for subsequently executing file operation are obtained, and the inquiry convenience is improved.

In some disclosed embodiments, the target file is a complete file, and the data transmission apparatus 100 further includes a first generating module, configured to generate first file location information of the target file based on the target cluster and the target storage segment thereof; wherein the write operation is performed on a target storage segment in the target cluster; the data transmission device 100 further includes a first storage module, configured to store the first file location information in a metadata center; the first file position information is used for searching the target file.

Therefore, under the condition that the target file is a complete file, the first file position information of the target file is generated based on the target cluster and the target storage segment thereof, the writing operation is executed on the target storage segment in the target cluster, the first file position information is stored in the metadata center based on the first file position information, the first file position information is used for searching the target file, the subsequent reading operation on the target file can be greatly facilitated, and the reading efficiency is improved.

In some disclosed embodiments, the target file includes a plurality of fragmented files arranged in sequence, and the plurality of fragmented files perform write operations in sequence, and the data transmission apparatus 100 further includes a second generating module, configured to generate second file location information of the first fragmented file based on the target cluster and the target storage fragment thereof; the write operation is executed on a target storage segment in a target cluster, and all the fragment files are written into the target storage segment in the target cluster; the data transmission apparatus 100 further includes a second storage module, configured to store the second file location information in the metadata center; and the second file position information is used for searching the target cluster and the target storage segment thereof when the write operation is executed on the non-first file.

Therefore, after the addressing operation is performed on the first file, the second file position information of the first file is generated based on the target cluster and the target storage segment of the first file, and the second file position information is stored in the metadata center, so that the writing operation of the subsequent fragmented file can be greatly facilitated, the subsequent reading operation of the target file can be facilitated, and the reading efficiency can be improved.

In some disclosed embodiments, the file operation is a read operation, and the metadata center stores a plurality of file location information, each file location information including a third storage segment where the stored file is located and a cluster to which the third storage segment belongs.

Therefore, the metadata center stores the file position information of the stored file, so that the stored file can be read directly according to the file position information in the follow-up process, and the file reading efficiency is improved.

In some disclosed embodiments, the cluster querying module 102 includes a location querying sub-module, configured to query location information of a plurality of files to obtain a target storage segment where a target file is located and a target cluster to which the target storage segment belongs.

Therefore, the target storage segment and the target cluster are obtained by inquiring the position information of the plurality of files, the inquiring speed can be increased, and the file reading efficiency can be further improved.

In some disclosed embodiments, the relationship between the current cluster where the target file is located and the target cluster is any one of a heterogeneous cluster and a homogeneous cluster; and/or the data transmission method is executed by the forwarding device, and the forwarding device and the gateway device are independent.

Therefore, the relationship between the current cluster where the target file is located and the target cluster is any one of the heterogeneous cluster and the homogeneous cluster, that is, the non-perception data transmission can be realized between the heterogeneous clusters and between the homogeneous clusters, which is beneficial to further expanding the application range of data transmission on the basis of improving the user experience; the data transmission method is executed by the forwarding device, the forwarding device and the gateway device are mutually independent, cluster query and message splicing can be separated from each other, dependence between the gateway device and the forwarding device is decoupled, and the gateway device is beneficial to being unaware of data transmission between heterogeneous clusters.

Referring to fig. 11, fig. 11 is a schematic diagram of a data transmission device 110 according to an embodiment of the present application. The data transmission device 110 includes an operation receiving module 111 and a target query module 112, wherein the operation receiving module 111 is used for receiving query operations of the forwarding equipment; the query operation is sent by the forwarding device in response to a cluster addressing request from the gateway device, the cluster addressing request is obtained by preprocessing a first request message of the user device by the gateway device and is related to a target file, and the first request message comprises a file name and a file operation of the target file; the target query module 112 is configured to query to obtain a target cluster based on a file operation in response to a query operation; the file operation is executed in the target cluster, the forwarding device generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway device, and the gateway device generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish the transmission of the target file.

In some disclosed embodiments, the file operation is a write operation, the metadata center stores a number of first mapping tables and a number of second mapping tables, each first mapping table includes a mapping relationship between a first storage segment and at least one second storage segment, each second mapping table includes a mapping relationship between a second storage segment and at least one candidate cluster; and/or the file operation is a read operation, the metadata center stores a plurality of file position information, and each file position information comprises a third storage segment where the stored file is located and a cluster to which the third storage segment belongs.

Therefore, the target cluster and the target storage segment can be determined by double mapping through the first mapping table and the second mapping table, so that the file operation can be carried out on the target file, and the double mapping takes the storage segment as the minimum granularity; the metadata center stores the file position information of the stored file, so that the stored file can be read directly according to the file position information in the follow-up process, and the file reading efficiency is improved.

Referring to fig. 12, fig. 12 is a schematic diagram of a data transmission device 120 according to an embodiment of the present application. The data transmission device 120 includes a first receiving module 121, an addressing request module 122, a second receiving module 123 and a file transmission module 124, where the first receiving module 121 is configured to receive a first request message of a user equipment; the first request message comprises a file name and a file operation of the target file; the addressing request module 122 is configured to preprocess the first request message to obtain a cluster addressing request, and send the cluster addressing request to the forwarding device; the second receiving module 123 is configured to receive an access protocol message generated by the forwarding device based on the cluster addressing request; the forwarding equipment responds to the cluster addressing request, queries a metadata center based on file operation to obtain a target cluster, executes a file in the target cluster, and generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster; the file transfer module 124 is configured to generate a second request message based on the access protocol message, and send the second request message to the target cluster to complete the transfer of the target file.

Referring to fig. 13, fig. 13 is a schematic diagram of a framework of an embodiment of a forwarding device 130 according to the present application. The forwarding device 130 includes a memory 131, a processor 132 and a communication circuit 133, the memory 131 and the communication circuit 133 are coupled to the processor 132, the memory 131 stores program instructions, and the processor 132 is configured to execute the program instructions to implement the steps in the above-described data transmission method embodiment. Specifically, forwarding device 130 may include, but is not limited to: desktop computers, notebook computers, servers, mobile phones, tablet computers, and the like, without limitation.

Specifically, the processor 132 is configured to control itself, the memory 131 and the communication circuit 133 to implement the steps in any of the above-described embodiments of the data transmission method. Processor 132 may also be referred to as a CPU (Central Processing Unit). The processor 132 may be an integrated circuit chip having signal processing capabilities. The Processor 132 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 132 may be commonly implemented by integrated circuit chips.

Referring to fig. 14, fig. 14 is a block diagram illustrating a metadata center 140 according to an embodiment of the present invention. Metadata center 140 includes a memory 141, a processor 142, and a communication circuit 143, memory 141 and communication circuit 143 are coupled to processor 142, memory 141 has stored therein program instructions, and processor 142 is configured to execute the program instructions to implement the steps in the above-described data transmission method embodiments. In particular, metadata center 140 may include, but is not limited to: desktop computers, servers, etc., without limitation thereto.

Specifically, the processor 142 is configured to control itself, the memory 141 and the communication circuit 133 to implement the steps in any of the above-described embodiments of the data transmission method. Processor 142 may also be referred to as a CPU (Central Processing Unit). The processor 142 may be an integrated circuit chip having signal processing capabilities. The Processor 142 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 142 may be commonly implemented by integrated circuit chips.

Referring to fig. 15, fig. 15 is a schematic diagram of a framework of an embodiment of a gateway device 150 according to the present application. Gateway device 150 includes a memory 151, a processor 152, and a communication circuit 153, where memory 151 and communication circuit 153 are coupled to processor 152, and program instructions are stored in memory 151, and processor 152 is configured to execute the program instructions to implement the steps in any of the above-described embodiments of data transmission methods. Specifically, gateway device 150 may include, but is not limited to: desktop computers, servers, etc., without limitation thereto.

Specifically, the processor 152 is configured to control itself, the memory 151 and the communication circuit 153 to implement the steps in the above-described data transmission method embodiment. Processor 152 may also be referred to as a CPU (Central Processing Unit). The processor 152 may be an integrated circuit chip having signal processing capabilities. The Processor 152 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 152 may be commonly implemented by an integrated circuit chip.

Referring to fig. 2, as shown in fig. 2, the data transmission system includes a user device, a gateway device, a metadata center, a plurality of clusters, and a forwarding device, where the user device is communicatively connected to the gateway device, the gateway device is communicatively connected to the forwarding device and the clusters, respectively, the forwarding device is communicatively connected to the metadata center, the gateway device receives a first request message sent by the user device, where the first request message includes a file name and a file operation of a target file, pre-processes the first request message to obtain a cluster addressing request, and sends the cluster addressing request to the forwarding device, the forwarding device queries the metadata center based on the file operation in response to the cluster addressing request to obtain the target cluster, and the file operation is executed in the target cluster, and generates an access protocol message matching a cluster type of the target cluster based on the file name and a cluster address of the target cluster, and sending the access protocol message to the gateway device, and the gateway device also generates a second request message based on the access protocol message and sends the second request message to the target cluster to complete the transmission of the target file. The steps executed by each device side such as the forwarding device, the metadata center, the gateway device, and the like, may refer to the relevant description in each data transmission method embodiment, and are not described herein again.

Referring to fig. 16, fig. 16 is a block diagram illustrating an embodiment of a computer readable storage medium 160 according to the present application. The computer readable storage medium 160 stores program instructions 161 capable of being executed by a processor, the program instructions 161 for implementing the steps in any of the data transmission method embodiments described above.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

The foregoing description of the various embodiments is intended to highlight various differences between the embodiments, and the same or similar parts may be referred to each other, and for brevity, will not be described again herein.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. A method of data transmission, comprising:

acquiring a cluster addressing request related to a target file; the cluster addressing request is obtained by preprocessing a first request message of user equipment by gateway equipment, and the first request message comprises a file name and a file operation of the target file;

responding to the cluster addressing request, and inquiring a metadata center based on the file operation to obtain a target cluster; wherein the file operation is executed at the target cluster;

generating an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sending the access protocol message to the gateway equipment;

and the gateway equipment generates a second request message based on the access protocol message and sends the second request message to the target cluster to finish transmitting the target file.

2. The method of claim 1, wherein the file operation is a write operation, wherein the metadata center stores a plurality of first mapping tables and a plurality of second mapping tables, wherein each first mapping table comprises a mapping relationship between a first storage segment and at least one second storage segment, and wherein each second mapping table comprises a mapping relationship between the second storage segment and at least one candidate cluster.

3. The method of claim 2, wherein querying a metadata center based on the file operation to obtain a target cluster comprises:

acquiring a current storage segment of the current cluster of the target file;

and sequentially querying the first mapping table and the second mapping table to obtain a target storage segment having a mapping relation with the current storage segment and a target cluster having a mapping relation with the target storage segment.

4. The method of claim 2, wherein the target file is a complete file, and after the querying the metadata center based on the file operation to obtain the target cluster, the method further comprises:

generating first file position information of the target file based on the target cluster and the target storage segment thereof; wherein the write operation is performed on a target storage segment in the target cluster;

storing the first file position information in the metadata center; and the first file position information is used for searching the target file.

5. The method of claim 2, wherein the target file comprises a plurality of fragmented files arranged in sequence, and the write operation is performed on the fragmented files in sequence, and after the querying the metadata center based on the file operation is performed on a first fragmented file to obtain a target cluster, the method further comprises:

generating second file position information of the first file based on the target cluster and the target storage segment thereof; the write operation is executed on a target storage segment in the target cluster, and all the fragmentation files are written into the target storage segment in the target cluster;

storing the second file position information in the metadata center; and the second file position information is used for searching the target cluster and the target storage segment thereof when the write operation is executed on the non-first file.

6. The method of claim 1, wherein the file operation is a read operation, and the metadata center stores a plurality of file location information, and each file location information includes a third storage segment where a stored file is located and a cluster to which the third storage segment belongs.

7. The method of claim 6, wherein querying a metadata center based on the file operation to obtain a target cluster comprises:

and inquiring the position information of the plurality of files to obtain a target storage segment where the target file is located and a target cluster where the target storage segment belongs.

8. The method according to claim 1, wherein the relationship between the current cluster where the target file is located and the target cluster is any one of a heterogeneous cluster and a homogeneous cluster;

and/or the data transmission method is executed by a forwarding device, and the forwarding device and the gateway device are independent from each other.

9. A method of data transmission, comprising:

receiving query operation of forwarding equipment; the query operation is sent by a forwarding device in response to a cluster addressing request from a gateway device, where the cluster addressing request is obtained by preprocessing a first request message of a user device by the gateway device and is related to a target file, and the first request message includes a file name and a file operation of the target file;

responding to the query operation, and querying to obtain a target cluster based on the file operation;

the file operation is executed in the target cluster, the forwarding device generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway device, and the gateway device generates a second request message based on the access protocol message, and sends the second request message to the target cluster to complete the transmission of the target file.

10. The method of claim 9,

the file operation is a write operation, the metadata center stores a plurality of first mapping tables and a plurality of second mapping tables, each first mapping table comprises a mapping relation between a first storage segment and at least one second storage segment, and each second mapping table comprises a mapping relation between the second storage segment and at least one candidate cluster;

and/or the file operation is a read operation, the metadata center stores a plurality of file position information, and each file position information comprises a third storage segment where a stored file is located and a cluster to which the third storage segment belongs.

11. A method of data transmission, comprising:

receiving a first request message of user equipment; wherein the first request message comprises a file name and a file operation of a target file;

preprocessing the first request message to obtain a cluster addressing request, and sending the cluster addressing request to forwarding equipment;

receiving an access protocol message generated by the forwarding device based on the cluster addressing request; the forwarding equipment responds to the cluster addressing request, queries a metadata center based on the file operation to obtain a target cluster, executes the file in the target cluster, and generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster;

and generating a second request message based on the access protocol message, and sending the second request message to the target cluster to finish transmitting the target file.

12. A method of data transmission, comprising: the method comprises the steps that gateway equipment receives a first request message sent by user equipment; wherein the first request message comprises a file name and a file operation of a target file;

the gateway device preprocesses the first request message to obtain a cluster addressing request, and sends the cluster addressing request to a forwarding device;

the forwarding equipment responds to the cluster addressing request, queries a metadata center based on the file operation, and obtains a target cluster; wherein the file operation is executed at the target cluster;

the forwarding device generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and sends the access protocol message to the gateway device;

13. A data transmission apparatus, comprising:

the request acquisition module is used for acquiring a cluster addressing request related to a target file; the cluster addressing request is obtained by preprocessing a first request message of user equipment by gateway equipment, and the first request message comprises a file name and a file operation of the target file;

the cluster query module is used for responding to the cluster addressing request and querying a metadata center based on the file operation to obtain a target cluster; wherein the file operation is executed at the target cluster;

a generation forwarding module, configured to generate an access protocol message matching the cluster type of the target cluster based on the file name and the cluster address of the target cluster, and send the access protocol message to the gateway device;

14. A data transmission apparatus, comprising:

the operation receiving module is used for receiving the query operation of the forwarding equipment; the query operation is sent by a forwarding device in response to a cluster addressing request from a gateway device, where the cluster addressing request is obtained by preprocessing a first request message of a user device by the gateway device and is related to a target file, and the first request message includes a file name and a file operation of the target file;

the target query module is used for responding to the query operation and querying to obtain a target cluster based on the file operation;

15. A data transmission apparatus, comprising:

a first receiving module, configured to receive a first request message of a user equipment; wherein the first request message comprises a file name and a file operation of a target file;

the addressing request module is used for preprocessing the first request message to obtain a cluster addressing request and sending the cluster addressing request to the forwarding equipment;

a second receiving module, configured to receive an access protocol message generated by the forwarding device based on the cluster addressing request; the forwarding equipment responds to the cluster addressing request, queries a metadata center based on the file operation to obtain a target cluster, executes the file in the target cluster, and generates an access protocol message matched with the cluster type of the target cluster based on the file name and the cluster address of the target cluster;

and the file transmission module is used for generating a second request message based on the access protocol message and sending the second request message to the target cluster so as to finish transmitting the target file.

16. A forwarding device comprising a memory, a communication circuit, and a processor, the memory and the communication circuit coupled to the processor, the memory having stored therein program instructions, the processor being configured to execute the program instructions to implement the data transmission method of any one of claims 1 to 8.

17. A metadata center comprising a memory, a communication circuit, and a processor, the memory and the communication circuit coupled to the processor, the memory having stored therein program instructions for execution by the processor to implement the data transfer method of any of claims 9 to 10.

18. A gateway device comprising a memory, a communication circuit, and a processor, the memory and the communication circuit coupled to the processor, the memory having stored therein program instructions, the processor configured to execute the program instructions to implement the data transmission method of claim 11.

19. A data transmission system is characterized by comprising user equipment, gateway equipment, a metadata center, a plurality of clusters and forwarding equipment, wherein the user equipment is in communication connection with the gateway equipment;

wherein the gateway device receives a first request message sent by a user device, the first request message includes a file name and a file operation of a target file, preprocesses the first request message to obtain a cluster addressing request, and sends the cluster addressing request to a forwarding device, the forwarding device responds to the cluster addressing request, queries a metadata center based on the file operation to obtain a target cluster, the file operation is executed in the target cluster, generates an access protocol message matched with a cluster type of the target cluster based on the file name and a cluster address of the target cluster, and sends the access protocol message to the gateway device, and the gateway device further generates a second request message based on the access protocol message and sends the second request message to the target cluster, to complete the transfer of the target file.

20. A computer-readable storage medium, characterized in that program instructions executable by a processor for implementing the data transmission method according to any one of claims 1 to 12 are stored.