WO2023134392A1

WO2023134392A1 - Method and system for reading data from cloud storage device and writing data thereinto

Info

Publication number: WO2023134392A1
Application number: PCT/CN2022/139611
Authority: WO
Inventors: 陈智殷; 丁敬文; 白平昌
Original assignee: 华为云计算技术有限公司
Priority date: 2022-01-14
Filing date: 2022-12-16
Publication date: 2023-07-20
Also published as: CN116483252A

Abstract

A method and system for reading data from a cloud storage device and writing data thereinto. The method comprises: a cloud gateway receiving a data reading request, which is sent by a client, and identification information of an application plug-in serving end, or receiving a data writing request, which is sent by the client, and the identification information of the application plug-in serving end; the cloud gateway sending, according to the identification information of the application plug-in serving end, the received data reading request or data writing request to the corresponding application plug-in serving end; and the application plug-in serving end processing the data reading request or the data writing request, so as to realize a data reading operation or a data writing operation on a cloud storage device. By means of the method, the utilization rate of resources can be increased, and the usage experience of a user can be improved.

Description

Data reading and writing method and system for cloud storage device

This application claims the priority of the Chinese patent application with the application number 2022100398329 and the application name "Data reading and writing method and system of cloud storage device" submitted to the China Patent Office on January 14, 2022, and the entire content of the application is passed References are incorporated in this application.

technical field

The present application relates to the field of storage technology, and in particular to a data reading and writing method and system for a cloud storage device.

Background technique

In the cloud storage system, before storing the data in the storage system, or after reading the data from the storage system, it is usually necessary to perform some general processing operations on the data. Large, multimedia data usually needs to be compressed before being stored in the storage node; another example, in a scenario where data is stored in the form of objects in the storage system, when part of the data is needed, It is necessary to read the entire object and find the required data through a specific query statement.

Usually, these common processing operations are downgraded to the storage side and completed by the storage side, but this affects other business functions on the storage side, such as storage services, making the utilization of resources on the storage side unreasonable.

Contents of the invention

This application discloses a data reading and writing method and system for cloud storage devices. According to the method described in this application, the cloud gateway sends the received data reading request or data writing request sent by the client to the corresponding application The plug-in server uses the plug-in server to process the data read request or data write request to realize the data read operation or data write operation of the cloud storage device. Implementing this application can improve resource utilization and improve user experience .

In the first aspect, the present application provides a data reading method of a cloud storage device. The cloud gateway, the application plug-in server and the cloud storage device are set in at least one cloud data center, and the client establishes a connection with the cloud gateway. The methods described include:

The cloud gateway receives the data read request sent by the client and the identification information of the application plug-in server, wherein the data read request carries the identification information of the cloud storage device and the data to be read in the The reading location information of the cloud storage device, the cloud storage device stores the data to be read, and the application plug-in server is used for near data processing;

The cloud gateway forwards the data read request to the application plug-in server according to the identification information of the application plug-in server;

The application plug-in server reads the data to be read from the cloud storage device according to the identification information of the cloud storage device carried in the data read request and the read location information, and Get data and carry out near-data processing, and send the processed data to be read to the cloud gateway;

The cloud gateway sends the processed data to be read to the client.

It can be seen that the general processing operation is set as an application plug-in in the cloud data center, and the application plug-in server performs the general processing operation, and returns the processed result to each client, compared with the general processing performed by each client Operation avoids the waste of resources, improves resource utilization, and improves user experience; the cloud gateway is set in at least one cloud data center where the application plug-in server and cloud storage device are located, and the cloud gateway serves according to the application plug-in The terminal’s identification information assigns the data reading request to the corresponding application plug-in server, and the application plug-in server performs near-data processing on the data to be read, and returns the processed data to be read to the client through the cloud gateway to realize The data reading operation enables each device to display its own capabilities and make full use of its strengths.

Based on the first aspect, in a possible implementation manner, the near-data processing includes any one of data compression, data decompression, data encryption, data decryption, data analysis, and data processing.

Based on the first aspect, in a possible implementation manner, the cloud gateway receives the data reading request sent by the client and the identification information of the application plug-in server, including:

The cloud gateway receives the HTTP message sent by the client, the HTTP message includes the data read request and the identification information of the application plug-in server;

The cloud gateway forwards the data read request to the application plug-in server according to the identification information of the application plug-in server, including:

The cloud gateway forwards the data read request in the HTTP message to the application plug-in server according to the identification information of the application plug-in server in the HTTP message; or

The cloud gateway forwards the HTTP message to the application plug-in server according to the identification information of the application plug-in server.

It can be understood that the data reading request and the identification information of the application plug-in server may exist in the form of a message. When the data reading request and the identification information of the application plug-in server exist in the form of a message, the cloud gateway may The identification information of the server sends the data message or the data reading request in the message to the application plug-in server.

Based on the first aspect, in a possible implementation manner, the identification information of the application plug-in server is a network address of the application plug-in server.

It can be understood that the identification information of the application plug-in server can be the network address of the application plug-in server, and the cloud gateway directly sends the data reading request to the application plug-in server according to the network address, which is convenient and fast.

Based on the first aspect, in a possible implementation manner, the cloud gateway records the correspondence between the network address of the application plug-in server and the identification information of the application plug-in server, and the cloud gateway records the corresponding relationship between the network address of the application plug-in server and the The identification information forwards the data read request to the application plug-in server, including:

The cloud gateway obtains the network address of the application plug-in server from the corresponding relationship according to the identification information of the application plug-in server, and the cloud gateway forwards the data read request to the Described application plug-in server.

Based on the first aspect, in a possible implementation manner, the cloud gateway, the application plug-in server, and the cloud storage device are set in one cloud data center, or in multiple cloud data centers connected to each other, In the case that the cloud gateway, the application plug-in server and the cloud storage device are arranged in multiple cloud data centers connected to each other, the cloud gateway, the application plug-in server and the cloud storage device One or any combination of them is set in one cloud data center of the plurality of cloud data centers.

Based on the first aspect, in a possible implementation manner, the client is set under the cloud, or set in the at least one cloud data center.

Based on the first aspect, in a possible implementation manner, the identification information of the application plug-in server is selected by the client from identification information of multiple application plug-in servers.

In a second aspect, the present application provides a method for writing data into a cloud storage device. The cloud gateway, the application plug-in server, and the cloud storage device are set in at least one cloud data center, and the client establishes a connection with the cloud gateway. Methods include:

The cloud gateway receives the data write request sent by the client and the identification information of the application plug-in server, wherein the data write request carries the identification information of the cloud storage device to be accessed, the data to be written, the The writing location information of the data to be written in the cloud storage device, the application plug-in server is used for near data processing;

The cloud gateway forwards the data writing request to the application plug-in server according to the identification information of the application plug-in server;

The application plug-in server performs near data processing on the data to be written carried in the data write request, and according to the identification information of the cloud storage device carried in the data write request and the write location The information writes the processed data to be written to the cloud storage device.

It can be seen that the general processing operation is set as an application plug-in in the cloud data center, and the application plug-in server performs the general processing operation, and returns the processed result to each client, compared with the general processing performed by each client Operation avoids the waste of resources, improves resource utilization, and improves user experience; the cloud gateway is set in at least one cloud data center where the application plug-in server and cloud storage device are located, and the cloud gateway serves according to the application plug-in The terminal’s identification information assigns the data reading request to the corresponding application plug-in server, and the application plug-in server performs near-data processing on the data to be read, and returns the processed data to be read to the client through the cloud gateway to realize The write operation of data enables each device to display its own capabilities and give full play to its strengths.

Based on the second aspect, in a possible implementation manner, the near-data processing includes any one of data compression, data decompression, data encryption, data decryption, data analysis, and data calculation.

Based on the second aspect, in a possible implementation manner, the cloud gateway receiving the data writing request sent by the client and the identification information of the application plug-in server includes: the cloud gateway receiving the request sent by the client An HTTP message, the HTTP message including the data write request and the identification information of the application plug-in server;

The cloud gateway forwarding the data write request to the application plug-in server according to the identification information of the application plug-in server includes: forwarding the data write request by the cloud gateway according to the identification information of the application plug-in server in the HTTP message The data writing request in the HTTP message is sent to the application plug-in server; or the cloud gateway forwards the HTTP message to the application plug-in server according to the identification information of the application plug-in server.

It can be understood that the data writing request and the identification information of the application plug-in server can exist in the form of a message. When the data writing request and the identification information of the application plug-in server exist in the form of a message, the cloud gateway can The identification information of the server sends the message or the data writing request in the message to the application plug-in server.

Based on the second aspect, in a possible implementation manner, the identification information of the application plug-in server is a network address of the application plug-in server.

It can be understood that the identification information of the application plug-in server can be the network address of the application plug-in server, and the cloud gateway directly sends the data writing request to the application plug-in server according to the network address, which is convenient and fast.

Based on the second aspect, in a possible implementation manner, the cloud gateway records the correspondence between the network address of the application plug-in server and the identification information of the application plug-in server, and the cloud gateway records the corresponding relationship between the network address of the application plug-in server and the The identification information forwards the data writing request to the application plug-in server, including:

The cloud gateway obtains the network address of the application plug-in server from the corresponding relationship according to the identification information of the application plug-in server, and the cloud gateway forwards the data write request to the Described application plug-in server.

Based on the second aspect, in a possible implementation manner, the cloud gateway, the application plug-in server, and the cloud storage device are set in one cloud data center, or in multiple cloud data centers connected to each other, In the case that the cloud gateway, the application plug-in server and the cloud storage device are arranged in multiple cloud data centers connected to each other, the cloud gateway, the application plug-in server and the cloud storage device One or any combination of them is set in one cloud data center of the plurality of cloud data centers.

Based on the second aspect, in a possible implementation manner, the client is set under the cloud, or set in the at least one cloud data center.

Based on the second aspect, in a possible implementation manner, the identification information of the application plug-in server is selected by the client from identification information of multiple application plug-in servers.

In a third aspect, the present application provides a data reading system for a cloud storage device, including a cloud gateway, an application plug-in server, and a cloud storage device, the cloud gateway, the application plug-in server, and the cloud storage device to perform the method described in the first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, the present application provides a data writing system for a cloud storage device, including a cloud gateway, an application plug-in server, and a cloud storage device. The cloud gateway, the application plug-in server, and the cloud storage device use to perform the method described in the second aspect or any possible implementation manner of the second aspect.

In a fifth aspect, the present application provides a readable storage medium, including program instructions. When the program instructions are run on a computer, the computer executes the above-mentioned first aspect or any one of the possible implementations of the first aspect. described method.

In a sixth aspect, the present application provides yet another readable storage medium, including program instructions. When the program instructions are run on a computer, the computer executes the above-mentioned second aspect or any possible implementation of the second aspect. method described.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can also obtain other drawings based on these drawings on the premise of not paying creative work.

FIG. 1 is a schematic diagram of a system architecture provided by the present application;

FIG. 2 is a schematic diagram of another system architecture provided by the present application;

FIG. 3 is a schematic flowchart of a data reading method of a cloud storage device provided by the present application;

FIG. 4 is an example diagram of a data reading method of a cloud storage device provided by the present application;

FIG. 5 is a schematic flowchart of a data writing method of a cloud storage device provided by the present application;

FIG. 6 is an example diagram of a data writing method of a cloud storage device provided by the present application;

FIG. 7 is a schematic structural diagram of a cloud gateway provided by the present application;

FIG. 8 is a schematic structural diagram of another cloud gateway provided by the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

For ease of understanding, before introducing the system architecture and method embodiments of the present application, first introduce the technical terms involved in the present application.

Near data processing (near data processing, NDP), in the separation of storage and calculation or data-centric system architecture, the calculation of part of the data is placed on the storage side to complete, thereby reducing the invalid data transmitted between the storage side and the calculation side, Improve bandwidth utilization and reduce costs. For example, in one scenario, the data collection is stored in the form of objects. When some of the data is needed, the entire object needs to be read out, and the required data is determined through a specific query statement. This query operation can be stored in the side implementation, this example is only for illustration, and does not constitute a limitation to the solution of this application.

This application provides a system, see Figure 1, Figure 1 is a schematic diagram of a system architecture provided by this application, the system involves at least one client 110, a switch 120 and a storage node cluster 130, the storage node cluster 130 includes at least one storage node .

The client 110 may be a mobile phone terminal, a desktop computer, a notebook, a physical server, a virtual machine, etc. of a user.

The switch 120 may be an Ethernet switch, an InfiniBand switch, an Ethernet switch (RDMA over converged ethernet, RoCE) based on a remote direct memory access (remote direct memory access, RDMA) technology, and the like. The client 110 can communicate with the storage node through the switch 120 . Optionally, the switch 120 is an optional device, and the client 110 can also communicate with the storage node through the network.

The client 110 may send an access request to the storage node through the switch 120, and the access request may be a data read request or a data write request. The storage node receives the access request sent by the client 110 , performs a read operation or a write operation according to the access request, and returns an access request response message to the client 110 .

In the system architecture shown in Figure 1, each client usually needs to perform some common processing operations on data, such as data compression processing, data decompression processing, data encryption, data decryption, data analysis, data processing, image processing, etc. In order to improve resource utilization and provide better services for users, general business-related processing operations can be used as application plug-ins, and different processing operations can be deployed as different application plug-ins in the cloud data center where the storage node is located. , each application plug-in and each storage node jointly provide services for users, wherein the application plug-in can be, for example, a data compression plug-in, a data decompression plug-in, a data encryption plug-in, a data decryption plug-in, a data analysis plug-in, a data processing plug-in, an image processing plug-in, etc. .

Referring to FIG. 2 , FIG. 2 is a schematic diagram of another system architecture provided by the present application. The system shown in Figure 2 is based on the system shown in Figure 1, and multiple application plug-ins and cloud gateways are deployed in the cloud data center where at least one storage node is located. The cloud gateway can be, for example, an application programming interface (application programming interface). interface, API) gateway.

Wherein, the multiple application plug-ins include application plug-in 1, application plug-in 2, ... application plug-in n, application plug-in 1, application plug-in 2, ... application plug-in n are different application plug-ins, and different application plug-ins may have different functions, for example, The application plug-in 1 can be a data compression plug-in, and the data compression plug-in is used to compress data. The application plug-in 2 can be an image processing plug-in, and the image processing plug-in is used to process images. The image processing can include image enhancement, noise removal/filtering, One or more of image restoration, etc. Optionally, some application plug-ins in different application plug-ins may also have the same function but different plug-in versions, for example, application plug-in 1 and application plug-in 2 may both be data compression plug-ins with data compression function, wherein application plug-in 1 The method A is used to compress the data, and the application plug-in 2 is used to compress the data by the method B, then it is called that the application plug-in 1 and the application plug-in 2 have the same function but different plug-in versions.

The device where the storage node is located is called a cloud storage device, and the application plug-in can be deployed on the cloud storage device where the storage node is located, or can be deployed on an independent device. That is, one application plug-in can be deployed on one cloud storage device, or multiple application plug-ins can be deployed on one cloud storage device at the same time. In other words, some cloud storage devices deploy an application plug-in, and some cloud Multiple application plug-ins are deployed on storage devices, and some cloud storage devices have no application plug-ins deployed. This application does not specifically limit the deployment location and deployment quantity of application plug-ins.

In this application, regardless of whether the application plug-in is deployed on an independent device or on a cloud storage device, the device where the application plug-in is located is called the application plug-in server, and the storage node for processing read or write operations is located A device called a cloud storage device. If the application plug-in is deployed on the cloud storage device, the application plug-in uses the computing resources on the cloud storage device, such as processor and memory, when computing and processing the data. In this case, the application plug-in server is cloud storage device; if the application plug-in is deployed on an independent device, the application plug-in uses the computing resources on the independent device, such as processor, memory, etc., when computing and processing data. In this case, the application plug-in server refers to standalone device.

For example, a cloud storage device may be an object storage (object storage service, OBS) device (also called an OBS bucket), or an elastic volume service (EVS) device, or an elastic file service (scalable file service). , SFS) equipment, network attached storage (network attached storage, NAS) equipment, open system direct-attached storage (direct-attached storage, DAS) equipment and any other storage equipment in the cloud data center.

In Figure 2, clients can be mobile phones, desktop computers, notebooks, etc. used by tenants in the cloud data center, or virtual machines, containers or bare metal servers purchased by tenants in the cloud data center in the cloud data center (bare metal server, BMS) and so on.

It can be understood that each application plug-in server, cloud gateway, and cloud storage device among multiple application plug-in servers can be set in the same cloud data center, or in multiple cloud data centers connected to each other. In the case of multiple cloud data centers connected to each other, any one or any combination of each application plug-in server, cloud gateway, and storage server in multiple application plug-in servers can be set in multiple cloud data centers in a cloud data center.

The system architecture diagram shown in FIG. 2 is only for example, and does not constitute a limitation to the present application. Practical applications may include more or fewer devices than those shown in Figure 2, and this application does not specifically limit the number of application plug-ins, the deployment locations of application plug-ins, the number of application plug-in servers, and the number of cloud storage devices.

Based on the system shown in Figure 2, when the client sends an access request, it first passes through the cloud gateway, and then the cloud gateway sends it to the cloud storage device or the application plug-in server to implement data read or write operations. A data reading method of a cloud storage device and a data writing method of a cloud storage device provided by the present application are respectively described below in conjunction with the system structure shown in FIG. Then introduce the data writing method of the cloud storage device. Referring to FIG. 3 , FIG. 3 is a schematic flowchart of a data reading method of a cloud storage device provided by the present application, and the method includes but is not limited to the following description.

S201. The client sends a data read request and identification information of the application plug-in server to the cloud gateway.

The data read request carries the identification information of the cloud storage device and the read location information of the data to be read in the cloud storage device. The identification information of the cloud storage device indicates which cloud storage device the data to be read is located in, and the read location The information indicates the specific location information of the data to be read in the cloud storage device. Optionally, the identification information of the cloud storage device and the read location information of the data to be read in the cloud storage device can be represented by a link, for example, the data read request can be curl-m 30-X GET”http://xxxxxxxx ", where curl-m 30-X GET is the read command, and http://xxxxxxxx is the identification information of the cloud storage device and the read location information of the data to be read on the cloud storage device. The identification information of the cloud storage device and the reading position information of the data to be read in the cloud storage device can also be expressed in other forms, for example, the identification information of the cloud storage device can be expressed by a network address, etc. The identification information and the read position information of the data to be read in the cloud storage device are not specifically limited.

If the data to be read requires near-data processing, the client needs to send a data read request and the identification information of the application plug-in server to the cloud gateway. The application plug-in server is used for near-data processing, and the identification information of the application plug-in server is used for Indicates which application plug-in server is to perform near-data processing on the data to be read. The identification information of the application plug-in server may be the name of the application plug-in corresponding to the server and the version number of the application plug-in. For example, the identification information of the application plug-in may be "data decompression plug-in-v1", where "data decompression plug-in" is the name of the application plug-in, and "v1" is the version number of the application plug-in; the identification information of the application plug-in server can also be the identity document (identity document, ID) of the application plug-in corresponding to the server or the identity ID of the application plug-in server For example, the identification ID of the application plug-in may be application plug-in 1; the identification information of the application plug-in server may also be the network address of the application plug-in server, and so on. Optionally, the identification information of the application plug-in server may exist in the form of a key-value pair (key-value, KV).

Generally, the data reading request and the identification information of the application plug-in server are sent by the client at the same time. For example, the data reading request and the identification information of the application plug-in server may be,

curl -m 30 -X GET "http://xxxxxxxx" -H "Host:koomap.cn-north-4.yyyyyy.com"–kv

Wherein, curl-m 30-X GET is a data read command, "http://xxxxxxxx" is the identification information of the cloud storage device and the reading location information of the data to be read in the cloud storage device, H"Host: koomap.cn-north-4.yyyyyy.com"–kv is the identification information of the application plug-in server, and yyyyyy in the identification information of the application plug-in server can be the supplier information.

Optionally, the client stores the identification information of each application plug-in server in the cloud data center, and the identification information of the required application plug-in server is selected by the client from the identification information of each application plug-in server. The method is not limited.

Optionally, the data read request and the identification information of the application plug-in server may exist in the form of a message, for example, the message may be an HTTP message or an HTTPS message, wherein the identification information of the application plug-in server is located in the message In the header, this application does not limit the specific format of the identification information of the application plug-in server.

Optionally, if the data to be read does not require near data processing, the client only needs to send a data read request to the cloud gateway.

S202. The cloud gateway forwards the data reading request to the application plug-in server according to the identification information of the application plug-in server.

Optionally, if the identification information of the application plug-in server is the name of the application plug-in and the version number of the application plug-in, or the identity ID of the application plug-in or the identity ID of the application plug-in server, the cloud gateway records the information of the application plug-in server. The corresponding relationship between the identification information and the network address of the application plug-in server, and the cloud gateway can maintain the corresponding relationship in real time. The cloud gateway determines the network address of the application plug-in server from the corresponding relationship according to the identification information of the application plug-in server, and sends the data read request to the application plug-in server according to the network address of the application plug-in server.

Optionally, if the data read request and the identification information of the application plug-in server exist in the form of a message, the cloud gateway determines the network address of the application plug-in server according to the identification information of the application plug-in server in the header of the message , to send the data read request or the entire message to the application plug-in server.

S203. The application plug-in server reads the data to be read from the cloud storage device according to the identification information and read location information of the cloud storage device carried in the data read request, and performs near-data processing on the data to be read.

S204. The application plug-in server sends the processed data to be read to the cloud gateway.

S205. The cloud gateway sends the processed data to be read to the client.

In order to facilitate the understanding of steps S203 to S205, the present application provides an example diagram of a data reading method of a cloud storage device, as shown in FIG. 4 .

In Figure 4, when near-data processing is required, that is, the application plug-in service is required, the cloud gateway receives the data reading request sent by the client and the identification information of the application plug-in server, and the cloud gateway, according to the identification information of the application plug-in server, Send the data read request to the corresponding application plug-in server. After receiving the data read request, the application plug-in server requests the cloud storage device to be read according to the identification information and read location information of the cloud storage device in the read request. To fetch data, the cloud storage device sends the data to be read to the application plug-in server. After the application plug-in server receives the data to be read, it calculates and processes the data to be read, and then sends the processed data to be read to the cloud The gateway, and the cloud gateway sends the processed data to be read to the client.

Optionally, when near-data processing is not required, that is, the application plug-in service is not required, and the client sends only a data read request and does not include the identification information of the application plug-in server, the cloud gateway receives the data read request After the request, according to the identification information and reading location information of the cloud storage device in the data read request, request the data to be read from the cloud storage device, and the cloud storage device executes the read operation, and sends the data to be processed to the cloud gateway, the cloud The gateway then sends the data to be read to the client.

It can be seen that the application plug-in server is deployed in the cloud data center, and the application plug-in server is used to implement general processing operations without the need for each client to perform processing operations separately, which improves resource utilization and user experience; this implementation In this example, the cloud gateway is used to determine whether to send the data read request to the application plug-in server or to the cloud storage device according to whether the identification information of the application plug-in server is included. The application plug-in server is used for near data processing, and the cloud storage device uses In order to perform read operations, each device can display its own capabilities and give full play to its strengths.

FIG. 5 is a schematic flowchart of a method for writing data to a cloud storage device provided by the present application, and the method includes but is not limited to the following description.

S301. The client sends a data writing request and the identification information of the application plug-in server to the cloud gateway.

The write request carries the identification information of the cloud storage device that needs to be accessed, the data to be written and the write location information of the data to be written in the cloud storage device, and the identification information of the cloud storage device indicates that the data to be written should be written to In which cloud storage device, the write location information indicates the specific location information where the data to be written is written into the cloud storage device. The identification information and writing location information of the cloud storage device can be expressed in the form of a link, or in other forms. For example, the identification information of the cloud storage device can be expressed by a network address. The expression form of the write location information of the input data in the cloud storage device is not specifically limited.

If the data to be written requires near data processing, the client needs to send a data write request and the identification information of the application plug-in server to the cloud gateway. The application plug-in server is used for near data processing, and the identification information of the application plug-in server is used for Indicates which application plug-in server is to perform near-data processing on the data to be read. Usually, the data writing request and the identification information of the application plug-in server are sent simultaneously by the client. For specific content such as the identification information of the application plug-in server and the identification information of the cloud storage device, please refer to the description of the relevant content in step S201 in the method embodiment in FIG.

Optionally, the identification information of the application plug-in server is selected by the client from identification information of multiple application plug-in servers.

Optionally, the data write request and the identification information of the application plug-in server may exist in the form of a message, for example, the message may be an HTTP message or an HTTPS message, wherein the identification information of the application plug-in server is located in the message In the header, this application does not limit the specific format of the identification information of the application plug-in server.

Optionally, if the data to be written does not require near data processing, the client only needs to send a data write request to the cloud gateway.

S302. The cloud gateway forwards the data writing request to the application plug-in server according to the identification information of the application plug-in server.

Optionally, if the identification information of the application plug-in server is the name of the application plug-in and the version number of the application plug-in, or the identity ID of the application plug-in or the identity ID of the application plug-in server, the cloud gateway records the information of the application plug-in server. The corresponding relationship between the identification information and the network address of the application plug-in server, and the cloud gateway can maintain the corresponding relationship in real time. The cloud gateway determines the network address of the application plug-in server from the corresponding relationship according to the identification information of the application plug-in server, and sends the data writing request to the application plug-in server according to the network address of the application plug-in server.

Optionally, if the data write request and the identification information of the application plug-in server exist in the form of a message, the cloud gateway determines the network address of the application plug-in server according to the identification information of the application plug-in server in the header of the message , to send the data write request or the entire message to the application plug-in server.

S303. The application plug-in server performs near-data processing on the data to be written carried in the data write request, and writes the processed data to be written according to the identification information and write location information of the cloud storage device carried in the data write request. into the cloud storage device.

In order to facilitate understanding of step S303, the present application provides an example diagram of a data writing method of a cloud storage device, as shown in FIG. 6 .

In Figure 6, when near-data processing is required, that is, the application plug-in service is required, the client sends a data write request and the identification information of the application plug-in server to the cloud gateway, and the cloud gateway receives the data write request sent by the client and the identification information of the application plug-in server, according to the identification information of the application plug-in server, the data writing request is sent to the corresponding application plug-in server, and the application plug-in server calculates and processes the data to be written in the data writing request, and obtains The processed data to be written, and the processed data to be written are sent to the cloud storage device, and the cloud storage device performs a write operation on the processed data to be written, and writes the processed data to be written.

Optionally, after writing the processed data to be written into the cloud storage device, the cloud storage device returns a response message to the application plug-in server, and after receiving the response message, the application plug-in server sends the data write to the cloud gateway The request responds, and the cloud gateway sends the data write request response to the client.

Optionally, when near-data processing is not required, that is, the application plug-in service is not required, the client sends a data write request to the cloud gateway, and the cloud gateway writes the data according to the identification information of the cloud storage device in the data write request The write request is sent to the cloud storage device, and the cloud storage device performs a write operation on the data to be written, writes the data to be written according to the write location information, and then returns a data write request response to the cloud gateway, and the cloud gateway then writes the data The request response is sent to the client.

It can be seen that the application plug-in server is deployed in the cloud data center, and the application plug-in server is used to implement general processing operations without the need for each client to perform processing operations separately, which improves resource utilization and user experience; this implementation In this example, the cloud gateway is used to determine whether to send the data writing request to the application plug-in server or to the cloud storage device according to whether the identification information of the application plug-in server is included. The application plug-in server is used for near data processing, and the cloud storage device uses In order to perform write operations, each device can display its own capabilities and give full play to its strengths.

The present application provides a structure of a cloud gateway 700, see FIG. 7, which is a schematic structural diagram of the cloud gateway 700 provided by the present application. The cloud gateway 700 includes:

The receiving unit 701 is configured to receive the data reading request sent by the client and the identification information of the application plug-in server, wherein the data reading request carries the identification information of the cloud storage device and the reading position of the data to be read in the cloud storage device information;

A sending unit 702, configured to forward the data reading request to the application plug-in server according to the identification information of the application plug-in server;

The receiving unit 701 is also used to receive the processed data to be read sent by the application plug-in server, wherein the processed data to be read is obtained by the application plug-in server according to the identification information of the cloud storage device in the data read request and The reading location information is obtained after reading the data to be read from the cloud storage device and performing near-data processing on the data to be read;

The sending unit 702 is also configured to send the processed data to be read to the client.

In a possible implementation, the receiving unit 701 is also configured to receive the data write request sent by the client and the identification information of the application plug-in server, wherein the data write request carries the identification information of the cloud storage device to be accessed, the Write data and write location information of data to be written in the cloud storage device;

The sending unit 702 is further configured to forward the data writing request to the application plug-in server according to the identification information of the application plug-in server.

In a possible implementation, the receiving unit 701 is also configured to receive the data write request response sent by the application plug-in server, and the data write request response is that the application plug-in server closes the data to be written in the data write request. Issued after data processing and writing the processed data to be written to the cloud storage device according to the identification information and writing location information of the cloud storage device;

The sending unit 702 is also configured to send the data writing request response to the client.

In a possible implementation manner, near-data processing includes any one of data compression, data decompression, data encryption, data decryption, data analysis, and data processing.

In a possible implementation, the receiving unit 701 is also configured to receive an HTTP message sent by the client, the HTTP message includes a data read request and the identification information of the application plug-in server, or the HTTP message includes a data write request and The identification information of the application plug-in server; the sending unit 702 is also used to forward the data read request or data write request in the HTTP message to the application plug-in server according to the identification information of the application plug-in server in the HTTP message; or, according to The identification information of the application plug-in server forwards the HTTP message to the application plug-in server.

In a possible implementation manner, the identification information of the application plug-in server is a network address of the application plug-in server.

In a possible implementation, the cloud gateway 700 further includes a recording unit 703, and the recording unit 703 is used to record the corresponding relationship between the network address of the application plug-in server and the identification information of the application plug-in server; the sending unit 702 is also used to, according to the application plug-in service The terminal identification information obtains the network address of the application plug-in server from the corresponding relationship, and forwards the data read request or data write request to the application plug-in server according to the network address of the application plug-in server.

In a possible implementation, the cloud gateway 700, the application plug-in server, and the cloud storage device are set in a cloud data center, or are set in multiple cloud data centers connected to each other, and the cloud gateway 700, the application plug-in server, and When the cloud storage device is set in multiple cloud data centers connected to each other, one or any combination of the cloud gateway 700, the application plug-in server and the cloud storage device is set in one cloud data center of the multiple cloud data centers .

In a possible implementation manner, the client is set under the cloud, or in at least one cloud data center.

In a possible implementation manner, the identification information of the application plug-in server is selected by the client from identification information of multiple application plug-in servers.

In a possible implementation, the receiving unit 701 is also configured to receive a data read request sent by the client, where the data read request carries the identification information of the cloud storage device and the read location of the data to be read in the cloud storage device information, the sending unit 702 is also used to request the data to be read from the cloud storage device according to the data read request, the receiving unit 701 is also used to receive the data to be read returned by the cloud storage device, and the sending unit 702 is also used to send the data to be read Data is sent to the client.

In a possible implementation, the receiving unit 701 is also configured to receive a data write request sent by the client, where the data write request carries the identification information of the cloud storage device to be accessed, the data to be written, the data to be written In the write position information of the cloud storage device, the sending unit 702 is also used to send the data writing request to the cloud storage device, and the receiving unit 701 is also used to receive the data writing request response sent by the cloud storage device, and the sending unit 702 also uses Used to send a data write request response to the client.

Each functional unit in FIG. 7 is used to implement the method in the method embodiment in FIG. 3 or FIG. 5 . For details, refer to the description of the method embodiment in FIG. 3 or 5 . For the sake of brevity, details are not repeated here.

Referring to FIG. 8, FIG. 8 is another structural schematic diagram of a cloud gateway 800 provided by the present application. The cloud gateway 800 can be implemented by a general bus architecture. The cloud gateway 800 includes at least one processor 801, a communication bus 802, and a memory 803 and at least one communication interface 804.

The processor 801 may be a general-purpose CPU, NP, microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, application-specific integrated circuit (application-specific integrated circuit, ASIC), programmable logic device (programmable logic device, PLD) or a combination thereof. Wherein, the PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL) or any combination thereof.

Communication bus 802 is used to transfer information between the various components. The communication bus 802 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG. 8 , but it does not mean that there is only one bus or one type of bus.

The memory 803 may be a read-only memory (read-only memory, ROM) or other types of static storage devices capable of storing static information and instructions, or may be a random access memory (random access memory, RAM) or capable of storing information and instructions Other types of dynamic storage devices can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage , optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, can also be used to carry or store program codes in the form of instructions or data structures and Any other medium capable of being accessed by a computer, but not limited to. The memory 803 may exist independently and be connected to the processor 801 through the communication bus 802; the memory 803 may also be integrated with the processor 801.

The communication interface 804 is used to communicate with other devices or communication networks. The communication interface 804 may include a wired communication interface, and may also include a wireless communication interface. Wherein, the wired communication interface may be, for example, an Ethernet interface, and the Ethernet interface may be an optical interface, an electrical interface or a combination thereof. The wireless communication interface may be a wireless local area network (wireless local area networks, WLAN) interface, a cellular network communication interface, or a combination thereof.

In a possible implementation, as an embodiment, the processor 801 may include one or more CPUs, for example, CPU0 and CPU1 shown in FIG. 8 .

In a possible implementation, as an embodiment, the cloud gateway 800 may include multiple processors, such as a processor 801 and a processor 805 as shown in FIG. 8 . Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data such as computer program instructions.

Each functional unit in the cloud gateway 700 is implemented by software in the cloud gateway 800 . In other words, the functional units included in the cloud gateway 700 are generated after the processor in the cloud gateway 800 reads the program code stored in the memory. In some embodiments, the memory 803 is used to store the program code 810 of the solution of the present application, and the processor 801 is used to execute the program code 810 stored in the memory 803 . That is, the cloud gateway 800 can implement the method provided by the method embodiment in FIG. 3 or FIG. 5 through the processor 801 and the program code 810 in the memory 803 .

The processor 801, the communication interface 804, etc. in the cloud gateway 800 can implement the functions and/or various steps and methods implemented by the devices in the above method embodiments, and for the sake of brevity, details are not repeated here. The receiving unit 701 and the sending unit 702 in the cloud gateway 700 can be located in the communication interface 804 in the cloud gateway 800; the recording unit 703 can be located in the processor 801 or the processor 805 in the cloud gateway 800, and the corresponding relationship recorded by the recording unit 703 It may be stored in the memory 803 in the cloud gateway 800 .

In some possible embodiments, the above cloud gateway can be implemented as a virtualization device.

For example, the virtualization device can be a virtual machine (virtual machine, VM) running a program for sending messages, and the virtual machine is deployed on the hardware device. For example, the hardware device can be a physical server, and the hardware device can also be cloud storage. equipment. A virtual machine refers to a complete computer system that is simulated by software and has complete hardware system functions and runs in a completely isolated environment. The virtual machine can be configured as a network device. For example, the cloud gateway can be implemented based on a common physical server combined with a network functions virtualization (network functions virtualization, NFV) technology. The cloud gateway is a virtual host, virtual router or virtual switch. Those skilled in the art can virtualize a cloud gateway with the above-mentioned functions on a common physical server by combining NFV technology by reading this application, and details will not be repeated here.

The virtualization appliance may be a container, which is an entity used to provide an isolated virtualization environment. For example, the container can be a docker container, and the container can be configured as a cloud gateway. For example, a cloud gateway can be created through the corresponding image. For example, a container proxy-container instance can be created for the proxy-container through the image of the container proxy-container that provides proxy services, and the container instance proxy-container can be provided as a cloud gateway. When implemented with container technology, the cloud gateway can use the kernel of the physical machine to run, and multiple cloud gateways can share the operating system of the physical machine. Different cloud gateways can be isolated through container technology. A containerized cloud gateway can run in a virtualized environment, for example, a virtual machine, and a containerized cloud gateway can also run directly on a physical machine.

For example, a virtualization device can be a Pod. A Pod is Kubernetes (Kubernetes is a container orchestration engine open sourced by Google, referred to as K8s for short). It is the basic unit for deploying, managing, and orchestrating containerized applications. A Pod can include one or more containers. Each container in the same Pod is usually deployed on the same host, so each container in the same Pod can communicate through the host, and can share the storage resources and network resources of the host. The Pod can be configured as a cloud gateway. For example, specifically, a container as a service (container as a service, CaaS, which is a container-based PaaS service) can be instructed to create a Pod, and the Pod can be provided as a cloud gateway.

Of course, the cloud gateway can also be other virtualization devices, and they will not be listed here.

In some possible embodiments, the above cloud gateway may also be implemented by a general processor. For example, the general-purpose processor may be in the form of a chip. Specifically, the general-purpose processor implementing the cloud gateway includes a processing circuit, an input interface and an output interface connected and communicated with the processing circuit internally, and the processing circuit is used to execute the steps of generating information or messages in the above-mentioned various method embodiments through the input interface The processing circuit is configured to execute the receiving step in each of the above method embodiments through an input interface, and the processing circuit is configured to execute the sending step in each of the above method embodiments through an output interface. Optionally, the general processor may further include a storage medium, and the processing circuit is configured to execute the storage steps in the foregoing method embodiments through the storage medium. The storage medium may store instructions executed by the processing circuit, and the processing circuit is configured to execute the instructions stored in the storage medium to perform the various method embodiments described above.

An embodiment of the present application provides a system, the system includes a cloud gateway, an application plug-in server and a cloud storage device, wherein the cloud gateway can be the cloud gateway 700 or cloud gateway 800, and the cloud gateway, the application plug-in server and the cloud storage device can be used The data reading method of the cloud storage device described in the method embodiment shown in FIG. 3 can also be used to realize the data writing method of the cloud storage device described in the method embodiment shown in FIG. 5 .

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the above-mentioned embodiments, all or part may be implemented by software, hardware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product may comprise code. When the computer program product is read and executed by a computer, some or all steps of the methods described in the above method embodiments can be realized. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium, or a semiconductor medium.

The steps in the method of the embodiment of the present application can be adjusted in order, merged or deleted according to actual needs; the units in the device of the embodiment of the present application can be divided, combined or deleted according to actual needs.

The embodiments of the present application have been introduced in detail above, and specific examples have been used in this paper to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; meanwhile, for Those skilled in the art will have changes in specific implementation methods and application scopes based on the ideas of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

A data reading method of a cloud storage device, characterized in that the cloud gateway, the application plug-in server and the cloud storage device are arranged in at least one cloud data center, and the client establishes a connection with the cloud gateway, the method comprising:

The cloud gateway receives the data read request sent by the client and the identification information of the application plug-in server, wherein the data read request carries the identification information of the cloud storage device and the data to be read in the The reading location information of the cloud storage device, the cloud storage device stores the data to be read, and the application plug-in server is used for near data processing;

The cloud gateway forwards the data read request to the application plug-in server according to the identification information of the application plug-in server;

The application plug-in server reads the data to be read from the cloud storage device according to the identification information of the cloud storage device carried in the data read request and the read location information, and Get data and carry out near-data processing, and send the processed data to be read to the cloud gateway;

The cloud gateway sends the processed data to be read to the client.
The method according to claim 1, wherein the near-data processing includes any one of data compression, data decompression, data encryption, data decryption, data analysis, and data processing.
The method according to claim 1 or 2, characterized in that,

The cloud gateway receives the data reading request sent by the client and the identification information of the application plug-in server, including:

The cloud gateway receives the HTTP message sent by the client, the HTTP message includes the data read request and the identification information of the application plug-in server;

The cloud gateway forwards the data read request to the application plug-in server according to the identification information of the application plug-in server, including:

The cloud gateway forwards the data read request in the HTTP message to the application plug-in server according to the identification information of the application plug-in server in the HTTP message; or

The cloud gateway forwards the HTTP message to the application plug-in server according to the identification information of the application plug-in server.
The method according to any one of claims 1 to 3, wherein the identification information of the application plug-in server is the network address of the application plug-in server.
The method according to any one of claims 1 to 3, wherein the cloud gateway records the corresponding relationship between the network address of the application plug-in server and the identification information of the application plug-in server, and the cloud gateway records according to The identification information of the application plug-in server forwards the data read request to the application plug-in server, including:

The cloud gateway obtains the network address of the application plug-in server from the corresponding relationship according to the identification information of the application plug-in server, and the cloud gateway forwards the data read request to the Described application plug-in server.
The method according to any one of claims 1 to 5, wherein the cloud gateway, the application plug-in server, and the cloud storage device are set in a cloud data center, or set in multiple interconnected In a cloud data center, when the cloud gateway, the application plug-in server and the cloud storage device are arranged in multiple cloud data centers connected to each other, the cloud gateway, the application plug-in server And one or any combination of the cloud storage devices is set in one cloud data center of the plurality of cloud data centers.
The method according to any one of claims 1 to 6, wherein the client is set under the cloud, or in the at least one cloud data center.
The method according to any one of claims 1 to 7, wherein the identification information of the application plug-in server is selected by the client from identification information of multiple application plug-in servers.
A method for writing data into a cloud storage device, characterized in that the cloud gateway, the application plug-in server and the cloud storage device are set in at least one cloud data center, and the client establishes a connection with the cloud gateway, the method comprising:

The cloud gateway receives the data write request sent by the client and the identification information of the application plug-in server, wherein the data write request carries the identification information of the cloud storage device to be accessed, the data to be written, the The writing position information of the data to be written in the cloud storage device, the application plug-in server is used for near data processing;

The cloud gateway forwards the data writing request to the application plug-in server according to the identification information of the application plug-in server;

The application plug-in server performs near data processing on the data to be written carried in the data write request, and according to the identification information of the cloud storage device carried in the data write request and the write location The information writes the processed data to be written to the cloud storage device.
The method according to claim 9, wherein the near-data processing includes any one of data compression, data decompression, data encryption, data decryption, data analysis, and data calculation.
The method according to claim 9 or 10, characterized in that,

The cloud gateway receiving the data writing request sent by the client and the identification information of the application plug-in server includes: the cloud gateway receiving the HTTP message sent by the client, the HTTP message including the A data writing request and identification information of the application plug-in server;

The cloud gateway forwarding the data write request to the application plug-in server according to the identification information of the application plug-in server includes: forwarding the data write request by the cloud gateway according to the identification information of the application plug-in server in the HTTP message The data writing request in the HTTP message is sent to the application plug-in server; or the cloud gateway forwards the HTTP message to the application plug-in server according to the identification information of the application plug-in server.
The method according to any one of claims 9 to 11, wherein the identification information of the application plug-in server is the network address of the application plug-in server.
The method according to any one of claims 9 to 11, wherein the cloud gateway records the corresponding relationship between the network address of the application plug-in server and the identification information of the application plug-in server, and the cloud gateway records according to The identification information of the application plug-in server forwards the data writing request to the application plug-in server, including:

The cloud gateway obtains the network address of the application plug-in server from the corresponding relationship according to the identification information of the application plug-in server, and the cloud gateway forwards the data write request to the Described application plug-in server.
The method according to any one of claims 9 to 13, characterized in that, the cloud gateway, the application plug-in server and the cloud storage device are set in a cloud data center, or in multiple interconnected In a cloud data center, when the cloud gateway, the application plug-in server and the cloud storage device are arranged in multiple cloud data centers connected to each other, the cloud gateway, the application plug-in server And one or any combination of the cloud storage devices is set in one cloud data center of the plurality of cloud data centers.
The method according to any one of claims 9 to 14, wherein the client is set under the cloud, or in the at least one cloud data center.
The method according to any one of claims 9 to 15, wherein the identification information of the application plug-in server is selected by the client from identification information of multiple application plug-in servers.
A data reading system for a cloud storage device, characterized in that it includes a cloud gateway, an application plug-in server and a cloud storage device, the cloud gateway, the application plug-in server and the cloud storage device are used to execute The method described in any one of 1 to 8 is required.
A data writing system for a cloud storage device, characterized in that it includes a cloud gateway, an application plug-in server and a cloud storage device, and the cloud gateway, the application plug-in server and the cloud storage device are used to execute The method described in any one of 9 to 16 is required.