CN103167026B

CN103167026B - A kind of cloud store environmental data processing method, system and equipment

Info

Publication number: CN103167026B
Application number: CN201310048783.6A
Authority: CN
Inventors: 肖立群; 孙晓斌; 李琳; 张迎华; 李焕
Original assignee: Digital Star Technology Development (beijing) Co Ltd
Current assignee: Digital Star Technology Development (beijing) Co Ltd
Priority date: 2013-02-06
Filing date: 2013-02-06
Publication date: 2016-05-18
Anticipated expiration: 2033-02-06
Also published as: CN103167026A

Abstract

The invention discloses a kind of cloud store environmental data processing method, when meta data server receives the data read request of client or data write request, the load retrieving according to described data read request or data write request is no more than to the storage server of predetermined threshold value, be defined as responding the storage server of described data read request or data write request, upgrade the metadata of the storage server of local described response data read request or data write request, and notify the information of the storage server of data read request described in described client end response or data write request. The present invention also discloses a kind of cloud store environmental data treatment system and equipment, use technical scheme of the present invention, reduced enforcement hardware cost, improved the reliability of data storages.

Description

Cloud storage environment data processing method, system and equipment

Technical Field

The invention relates to a data storage technology, in particular to a cloud storage environment data processing method, system and device.

Background

Currently, in digital cinema following the DCI (digital cinema initiatives) technical specification, independent servers request multimedia data from a central storage server and play the multimedia data during operation, and since all the multimedia data are stored in the central storage server and the central server needs to respond to all the independent server reading requests, the central server needs to have high-speed data processing performance, so that the hardware cost is high during implementation; meanwhile, all multimedia data are stored in the central storage server, and once the central storage server is down, all data may be lost, so that how to ensure the reliability of data storage becomes a problem to be solved urgently.

Disclosure of Invention

In view of the above, the present invention provides a method, a system, and a device for processing data in a cloud storage environment, which can reduce the cost of hardware implementation and improve the reliability of data storage.

In order to achieve the purpose, the technical scheme of the invention is realized as follows:

the invention discloses a cloud storage environment data processing method, which comprises the following steps:

when a metadata server receives a data reading request or a data writing request of a client, determining a storage server with a load not exceeding a preset threshold value, which is retrieved according to the data reading request or the data writing request, as a storage server responding to the data reading request or the data writing request;

and the metadata server updates the metadata of the local storage server responding to the data reading request or the data writing request and informs the client of the information of the storage server responding to the data reading request or the data writing request.

Preferably, when the metadata server receives a data read request or a data write request from a client, determining a storage server with a load not exceeding a preset threshold, which is retrieved according to the data read request or the data write request, as a storage server responding to the data read request or the data write request, including:

when the metadata server receives a data reading request of the client, retrieving a storage server where the data reading request is read according to local metadata, and determining a storage server with a load not exceeding a preset threshold value in the retrieved storage server as a storage server responding to the data reading request;

and when the metadata server receives a data writing request of the client, searching a storage server meeting the storage space requirement of the data writing request according to local metadata, and determining the storage server with the load not exceeding a preset threshold value in the searched storage server as the storage server responding to the data writing request.

Preferably, the updating the metadata of the local storage server responding to the data read request or the data write request by the metadata server includes:

the metadata server updates the load information in the metadata of the local storage server responding to the data read request; or updating data identification information, load information and residual storage space information in the local metadata of the storage server responding to the data writing request.

Preferably, when the metadata server receives a data write request from a client, the method further includes:

and the metadata server determines the retrieved storage server as a storage server for backing up the data written by the client into the storage server responding to the data writing request according to the local metadata and the storage server with the data writing request retrieval load not exceeding a set threshold, updates data identification information, load information and residual storage space information in the metadata of the backup storage server, and sends a backup instruction for sending the data written by the client to the backup storage server to respond to the data writing request.

Preferably, the method further comprises:

the metadata server synchronizes local metadata to a slave metadata server, and the slave metadata server determines that the metadata server is unavailable when the slave metadata server does not receive periodic heartbeat information of the metadata server within preset time, and processes a data read request or a data write request of the client instead of the metadata server.

Preferably, the method further comprises:

when the metadata server analyzes that the frequency of reading the same data by the client through a data reading request exceeds a preset maximum value according to local metadata, a storage server meeting the requirement of the data storage space is retrieved according to the local metadata, a storage server with the load not exceeding a preset threshold value in the retrieved storage server is determined as a storage server for backing up the data, a backup instruction for sending the data to the backup storage server is sent to the storage server where the data is located, and data identification information and residual storage space information in the metadata of the backup storage server are updated;

when the metadata server analyzes that the frequency of the same data read by the client through a data reading request is smaller than a preset minimum value according to local metadata, it is determined that one or more storage servers in which the data are located delete the data, a backup deletion instruction for deleting the data is sent to the one or more storage servers, and data identification information and residual storage space information in the metadata of the one or more storage servers are updated.

Preferably, the method further comprises:

when the metadata server detects that the cloud storage environment is newly added to a storage server, the frequency of reading the same data by the client through a data reading request is sequenced according to local metadata, a backup instruction for sending the data to the newly added storage server is sent to the storage server where a preset number of data with the highest reading frequency are located, and data identification information and residual storage space information in the metadata of the newly added storage server are updated;

when the metadata server detects that the number of the storage servers in the cloud storage environment is reduced, retrieving data without backup according to local metadata, determining the storage servers which meet the storage space requirement of the data and have loads not exceeding a set threshold value in the storage servers without the backup data as the storage servers for backing up the data, sending a backup instruction for sending the data to the backup storage servers to the storage servers storing the data, and updating data identification information and residual storage space information of the metadata of the backup storage servers.

The invention also discloses a metadata server, which comprises: the device comprises a first receiving unit, a first storage unit, a first retrieval determining unit, a first updating unit and a first sending unit; wherein,

the first receiving unit is used for receiving a data reading request or a data writing request of a client;

the first storage unit is used for storing metadata;

the first retrieval determining unit is configured to determine, when the first receiving unit receives a data read request or a data write request, a storage server whose load retrieved according to the data read request or the data write request does not exceed a set threshold as a storage server that responds to the data read request or the data write request;

the first updating unit is configured to update the metadata of the storage server responding to the data read request or the data write request determined by the first retrieval determining unit of the first storage unit;

the first sending unit is configured to send, to the client, information notifying the storage server that responds to the data read request or the data write request and is determined by the first retrieval determining unit.

Preferably, the first retrieval determining unit is further configured to, when the first receiving unit receives a data read request from the client, retrieve, according to metadata of the first storage unit, a storage server where data is read by the data read request, and determine, as a storage server responding to the data read request, a storage server whose load does not exceed a set threshold in the retrieved storage servers; when the first receiving unit receives a data writing request of the client, according to the metadata of the first storage unit, a storage server meeting the storage space requirement of the data writing request is searched, and a storage server with the load not exceeding a set threshold value in the searched storage server is determined as a storage server responding to the data writing request.

Preferably, the first updating unit is further configured to update load information in metadata of the storage server of the first storage unit responding to the data read request; or updating load information and remaining storage space information in the metadata of the storage server of the first storage unit responding to the data write request.

Preferably, the first retrieval determining unit is further configured to, when the first receiving unit receives a data write request from the client, retrieve, according to the metadata of the first storage unit, a storage server whose load does not exceed a set threshold, and determine the retrieved storage server as a storage server that backs up data written by the client into a storage server that responds to the data write request;

the first updating unit is configured to update data identification information, load information, and remaining storage space information of metadata in the first storage unit when the first retrieval determining unit determines the backup storage server, where the metadata is metadata of the backup storage server determined by the first retrieval determining unit;

the first sending unit is further configured to send, to the storage server responding to the data write request, a backup instruction for sending the data written by the client to the backup storage server when the first retrieval determining unit determines that the backup storage server is located.

Preferably, the first storage unit is further configured to generate metadata synchronization information;

the first sending unit is further configured to send the metadata synchronization information generated by the first storage unit to a slave metadata server; sending periodic heartbeat information to the slave metadata server.

Preferably, the metadata server further comprises:

the first analysis unit is used for analyzing the frequency of the client side for reading the same data through the data reading request according to the metadata of the first storage unit;

the first retrieval determining unit is further configured to, when the first analyzing unit analyzes that the frequency of reading the same data by the client through a data reading request exceeds a preset maximum value, retrieve a storage server meeting the requirement of the data storage space according to the metadata of the first storage unit, and determine a storage server with a load not exceeding a preset threshold value in the retrieved storage servers as a storage server for backing up the data; when the first analysis unit analyzes that the frequency of reading the same data by the client through the data reading request is less than a preset minimum value, determining one or more storage servers where the data are located to delete the data;

the first sending unit is further configured to send a backup instruction for sending the data to the backup storage server determined by the first retrieval determining unit to the storage server where the data is located; sending a backup deletion instruction for deleting the data to the one or more storage servers for deleting the data determined by the first retrieval determining unit;

the first updating unit is further configured to update data identification information and remaining storage space information in the metadata of the backup storage server determined by the first retrieval determining unit in the first storage unit; and updating the data identification information and the residual storage space information in the metadata of the one or more storage servers of the first storage unit, which delete the data determined by the first retrieval unit.

Preferably, the metadata server further comprises:

the device comprises a first detection unit, a second detection unit and a control unit, wherein the first detection unit is used for detecting the number of storage servers in the cloud storage environment;

the first retrieval determining unit is further configured to, when the first detecting unit detects that the cloud storage environment newly joins the storage server, rank, according to the metadata of the first storage unit, frequencies at which the client reads the same data through a data read request; when the first detection unit detects that the number of storage servers in the cloud storage environment is reduced, retrieving data which are not backed up according to metadata of the first storage unit, and determining the storage servers which meet the storage space requirement of the data and have loads not exceeding a set threshold value in the storage servers which do not back up the data as the storage servers which back up the data;

the first sending unit is further configured to send a backup instruction for sending the data to the newly added storage server to a storage server where a preset number of data with the highest reading frequency are located according to the sorting result of the first retrieval determining unit; sending a backup instruction for sending the data without backup to the backup storage server determined by the first retrieval determining unit;

the first updating unit is further configured to update data identification information and remaining storage space information of metadata of the first storage unit when the first sending unit sends a backup instruction to the newly added storage server, where the metadata is metadata of the newly added storage server; when the first sending unit sends a backup instruction to the backup storage server, updating data identification information and residual storage space information of metadata of the first storage unit, wherein the metadata is metadata of the backup storage server.

The invention also discloses a storage server, which comprises: a second receiving unit and a second transmitting unit; wherein,

and the second receiving unit is used for receiving a backup instruction which is sent by the metadata server and used for sending the data written by the client to the backup storage server.

And the second sending unit is used for sending the data written by the client to a backup storage server specified by the backup instruction according to the backup instruction received by the second receiving unit.

Preferably, the second receiving unit is further configured to receive a backup instruction sent by the metadata server and used for sending the data, of which the reading frequency exceeds a preset maximum value through the data reading request, of the client to the backup storage server; and receiving a backup deletion instruction sent by the metadata server for deleting the data of which the reading frequency of the client through the data reading request is less than the preset minimum value.

Preferably, the second receiving unit is further configured to receive a backup instruction sent by the metadata server and used for sending a preset number of data with the highest reading frequency of the client through the data reading request to the backup storage server; and receiving a backup instruction which is sent by the metadata server and sends data without backup to the backup storage server.

The present invention also discloses a dependent metadata server, which includes: the third receiving unit, the third storage unit, the third retrieval determining unit, the third updating unit and the third sending unit; wherein,

the third receiving unit is configured to receive periodic heartbeat information sent by the metadata server; receiving a data reading request or a data writing request of a client;

the third storage unit is used for storing metadata;

the third retrieval determining unit is configured to determine that the metadata server is unavailable when the third receiving unit does not receive heartbeat information within a preset time, retrieve a storage server with a load not exceeding a set threshold according to a data read request or a data write request received by the third receiving unit, and determine the retrieved storage server as a storage server responding to the data read request or the data write request;

the third updating unit is configured to update the metadata of the storage server responding to the data read request or the data write request determined by the third retrieval determining unit of the third storage unit;

the third sending unit is configured to send, to the client, information notifying the storage server that responds to the data read request or the data write request and is determined by the third retrieval determining unit.

The invention also discloses a cloud storage environment data processing system, which comprises: the system comprises a client, at least two storage servers and a metadata server; wherein,

the metadata server is used for determining a storage server with a load not exceeding a preset threshold value retrieved according to a data read request or a data write request as a storage server responding to the data read request or the data write request when receiving the data read request or the data write request of a client; and updating the metadata of the local storage server responding to the data reading request or the data writing request, and informing the client of the information of the storage server responding to the data reading request or the data writing request.

Preferably, the metadata server is further configured to, when receiving a data read request from the client, retrieve a storage server where the data read request is located according to the local metadata, and determine a storage server with a load not exceeding a preset threshold in the retrieved storage servers as a storage server responding to the data read request; and when a data writing request of the client is received, searching a storage server meeting the storage space requirement of the data writing request according to local metadata, and determining the storage server with the load not exceeding a preset threshold value in the searched storage server as the storage server responding to the data writing request.

Preferably, the metadata server is further configured to update load information in metadata of the local storage server responding to the data read request; or updating data identification information, load information and residual storage space information in the local metadata of the storage server responding to the data writing request.

Preferably, the metadata server is further configured to, when receiving a data write request from the client, determine, according to a storage server whose local metadata retrieval load does not exceed a set threshold, the retrieved storage server as a storage server that backs up data written by the storage server responding to the data write request from the client, update data identification information, load information, and remaining storage space information in metadata of the backup storage server, and send a backup instruction to the storage server responding to the data write request, where the backup instruction is to send data written by the client to the backup storage server.

Preferably, the system further comprises: the slave metadata server is used for receiving the metadata synchronized to the local by the metadata server; and when the heartbeat information of the metadata server is not received within preset time, determining that the metadata server is unavailable, and processing a data reading request or a data writing request of the client instead of the metadata server.

Preferably, the metadata server is further configured to, when it is analyzed according to local metadata that the frequency of reading the same data by the client through a data read request exceeds a preset maximum value, retrieve a storage server meeting the requirement of the data storage space according to the local metadata, determine a storage server with a load not exceeding a preset threshold value in the retrieved storage server as a storage server for backing up the data, send a backup instruction for sending the data to the backup storage server to the storage server where the data is located, and update data identification information and remaining storage space information in metadata of the backup storage server; when the frequency of reading the same data by a client through a data reading request is smaller than a preset minimum value according to local metadata analysis, determining one or more storage servers where the data are located to delete the data, sending a backup deletion instruction for deleting the data to the one or more storage servers, and updating data identification information and residual storage space information in the metadata of the one or more storage servers.

Preferably, the metadata server is further configured to, when detecting that the cloud storage environment newly joins the storage server, sequence, according to local metadata, a frequency at which the client reads the same data through a data read request, send, to the storage server where a preset number of data with the highest read frequency are located, a backup instruction for sending the data to the newly joined storage server, and update data identification information and remaining storage space information in the metadata of the newly joined storage server; when the fact that the number of the storage servers in the cloud storage environment is reduced is detected, retrieving data without backup according to local metadata, determining the storage servers which meet the storage space requirement of the data and have loads not exceeding a set threshold value in the storage servers without the backup data as the storage servers for backing up the data, sending a backup instruction for sending the data to the backup storage servers to the storage servers storing the data, and updating data identification information and residual storage space information of the metadata of the backup storage servers.

In the technical scheme provided by the invention, the storage servers form a cloud storage environment to store data, the hardware cost of the storage servers is low, and the deployment quantity of the storage servers can be adjusted according to different storage data capacities required by different application scenes, so that the hardware cost is effectively controlled;

furthermore, the metadata server backs up the metadata in the slave metadata server, so that when the metadata server is unavailable, the slave metadata server can respond to a data read request or a data write request of the client instead of the metadata server, and the reliability of data processing is improved;

furthermore, the metadata server adjusts the number of storage servers for storing the data according to the frequency of reading the same data through the data reading request by the client, so that the response speed of the data reading request of the client is improved;

further, when the metadata server is newly added to the storage server in the cloud storage environment, determining data stored by the newly added storage server according to the data reading frequency, so that the load of the storage server is balanced; when the number of the storage servers in the cloud storage environment is reduced, data which are not backed up are retrieved and backed up, and the reliability of data storage is improved.

Drawings

FIG. 1 is a schematic flow chart of a data processing method for a cloud storage environment according to the present invention;

FIG. 2 is a schematic diagram of a data processing system of a cloud storage environment according to the present invention;

fig. 3 is a schematic flow chart illustrating an implementation of a cloud storage environment data processing method according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

Fig. 1 is a schematic flow chart of an implementation of the cloud storage environment data processing method of the present invention, as shown in fig. 1, including:

step 101: when a metadata server receives a data reading request or a data writing request of a client, determining a storage server with a load not exceeding a preset threshold value, which is retrieved according to the data reading request or the data writing request, as a storage server responding to the data reading request or the data writing request;

the load is specifically a data read request or a data write request and response time responded by the storage server, and when the load of the storage server reaches a preset threshold, it means that the input/output of the storage server reaches the throughput performance upper limit of the storage server.

The obtaining of the storage server load information is the category of the prior art.

The data reading request of the client carries identification information of data to be written and the storage space requirement of the data to be written.

The metadata of the storage server comprises identification information of data stored by the storage server, load information of the storage server and residual storage space information. Correspondingly, when the metadata server determines the retrieved storage server as a storage server responding to the data reading request, the load information in the metadata of the storage server needs to be updated; when the metadata server determines the retrieved storage server as a storage server responding to the data writing request, the metadata server needs to add identification information of data written by the client to the metadata of the storage server, and update load information and remaining storage space information of the storage server. In this way, when the metadata server subsequently receives a data read request or a data write request, the storage server responding to the subsequent data read request or the data write request may be retrieved and determined according to the data identification information, the load information, and the remaining storage space information in the current storage metadata of the storage server, so as to ensure that the response of the subsequent data read request or the data write request is not delayed.

Preferably, when the metadata server receives a data read request or a data write request from a client, determining a storage server whose retrieval load does not exceed a preset threshold according to the data read request or the data write request as a storage server responding to the data read request or the data write request, including:

When the metadata server retrieves the storage servers meeting the retrieval conditions, the storage server with the minimum load in the storage servers meeting the retrieval conditions is preferentially selected to be determined as the storage server responding to the data writing request or the data writing request, so as to balance the load of the storage servers.

The number of the storage servers which are determined to respond to the data reading request can be one or more, and the number of the storage servers which are determined to respond to the data reading request can be preset to be a maximum value and a minimum value. Specifically, the minimum value of the number of the storage servers responding to the data reading request is preset, so that the situation that the client cannot receive complete data in time due to low data receiving rate can be avoided; the maximum value of the number of the storage servers responding to the data reading request is preset, so that the situation that the subsequent data reading request or data writing request of the client cannot be responded in time due to the fact that excessive storage servers respond to the data reading request can be avoided.

In order to backup data to be written, which is sent by a client to a storage server, at another storage server, preferably, when the metadata server receives a data writing request of the client, the method further includes:

Preferably, the method further comprises:

The metadata server periodically sends heartbeat information to the slave metadata server to identify that the metadata server is in a normal working state, and when the slave metadata server does not receive the heartbeat information within preset time, the slave metadata server indicates that the metadata server is in an unavailable state.

Step 102: and the metadata server updates the metadata corresponding to the local storage server responding to the data reading request or the data writing request and informs the client of the information of the storage server responding to the data reading request or the data writing request.

The information of the storage server is information for identifying the storage server, and in a cloud storage environment formed by the storage server, the information of the storage server includes an Internet Protocol (IP) address of the storage server.

Preferably, the method further comprises:

and the client sends a data reading request or data to be written to a storage server responding to the data reading request or the data writing request, and receives data read by the data reading request or data writing success information returned by the storage server.

Preferably, the method further comprises:

when the metadata server analyzes that the frequency of the same data read by a client through a data reading request is smaller than a preset minimum value according to local metadata, it is determined that one or more storage servers in which the data are located delete the data, a backup deletion instruction for deleting the data is sent to the one or more storage servers, and data identification information and residual storage space information in the metadata of the one or more storage servers are updated.

The load information in the metadata is specifically a data read request or a data write request and response time responded by the storage server, so that the frequency of the client sending the same data read request to read the same data can be analyzed according to the load information of all the storage servers in the cloud storage environment.

When the frequency of reading the same data through the data reading request by the client exceeds a preset maximum value, the throughput performance of the existing storage servers storing the data cannot guarantee that the data reading request of the client for the data can be responded in time, and the response speed of the data reading request for the data can be improved through the processing of increasing the number of the storage servers storing the data; when the frequency of reading the same data through the data reading request by the client is smaller than the preset minimum value, the client represents that the number of the existing storage servers for storing the data is large, and the storage space of the storage servers can be saved through the operation of deleting the data in the storage servers.

Preferably, the method further comprises:

The method has the advantages that through retrieving data without backup and carrying out backup processing, the possibility that the data is lost due to the fact that the data is not backed up when the storage server is unavailable due to faults is avoided; when the metadata server searches for a storage server that satisfies the storage space requirement of the data and whose load does not exceed a set threshold among the storage servers that do not back up the data, it is preferable that the metadata server selects a storage server with the smallest load among the storage servers that satisfy the search condition and determines the storage server as the storage server that backs up the data, so that the load of the storage servers can be balanced.

Fig. 2 is a schematic diagram of a composition structure of a data processing system in a cloud storage environment according to the present invention, and as shown in fig. 2, the system includes: a metadata server 21, at least two storage servers 22, and a client 23; wherein,

the metadata server 21 is configured to, when receiving a data read request or a data write request from a client 23, determine a storage server 22 with a load not exceeding a preset threshold, which is retrieved according to the data read request or the data write request, as the storage server 22 responding to the data read request or the data write request; updating the local metadata of the storage server 22 responding to the data read request or the data write request, and informing the client 23 of the information of the storage server 22 responding to the data read request or the data write request.

Preferably, the metadata server 21 is further configured to, when receiving a data read request from the client 23, retrieve the storage server 22 where the data read request reads data according to the local metadata, and determine a storage server 22 with a load not exceeding a preset threshold in the retrieved storage server 22 as the storage server 22 responding to the data read request; when a data write request of the client 23 is received, the storage servers 22 meeting the storage space requirement of the data write request are retrieved according to the local metadata, and the storage servers 22 with loads not exceeding a preset threshold value in the retrieved storage servers 22 are determined as the storage servers 22 responding to the data write request.

Preferably, the metadata server 21 is further configured to update load information in the metadata of the local storage server 22 responding to the data read request; or update the data identification information, the load information and the remaining storage space information in the local metadata of the storage server 22 responding to the data write request.

Preferably, the metadata server 21 is further configured to, when receiving a data write request from the client 23, determine, according to the storage server 22 whose local metadata retrieval load does not exceed a set threshold, the retrieved storage server 22 as the storage server 22 that backs up data written by the client 23 into the storage server 22 responding to the data write request, update the data identification information, the load information, and the remaining storage space information in the metadata of the backup storage server 22, and send a backup instruction to the storage server 22 responding to the data write request, where the backup instruction is to send the data written by the client 23 to the backup storage server 22.

Preferably, the system further comprises:

a slave metadata server 24 for receiving metadata synchronized to local by the metadata server 21; and when the heartbeat information of the metadata server 21 is not received within a preset time, determining that the metadata server 21 is unavailable, and processing a data read request or a data write request of the client 23 instead of the metadata server 21.

Preferably, the metadata server 21 is further configured to, when it is analyzed according to the local metadata that the frequency of reading the same data by the client 23 through a data reading request exceeds a preset maximum value, retrieve, according to the local metadata, a storage server 22 that meets the requirement of the data storage space, determine, as the storage server 22 that backs up the data, the storage server 22 whose load does not exceed a preset threshold in the retrieved storage server 22, send, to the storage server 22 where the data is located, a backup instruction for sending the data to the backup storage server 22, and update data identification information and remaining storage space information in the metadata of the backup storage server 22; when the frequency of reading the same data by the client 23 through a data reading request is smaller than a preset minimum value according to the local metadata analysis, determining that one or more storage servers 22 in which the data is located delete the data, sending a backup deletion instruction for deleting the data to the one or more storage servers 22, and updating data identification information and remaining storage space information in the metadata of the one or more storage servers 22.

Preferably, the metadata server 21 is further configured to, when detecting that the cloud storage environment newly joins the storage server 22, sequence, according to local metadata, frequencies of the client 23 reading the same data through a data reading request, send, to the storage server 22 where a preset number of data with the highest reading frequency are located, a backup instruction for sending the data to the newly joined storage server 22, and update data identification information and remaining storage space information in the metadata of the newly joined storage server 22; when the reduction of the number of the storage servers 22 in the cloud storage environment is detected, retrieving data without backup according to local metadata, determining the storage servers 22 which meet the storage space requirement of the data and have a load not exceeding a set threshold value in the storage servers 22 without the backup of the data as the storage servers 22 for backing up the data, sending a backup instruction for sending the data to the backup storage servers 22 to the storage servers 22 for storing the data, and updating the data identification information and the remaining storage space information of the metadata of the backup storage servers 22.

Preferably, the metadata server 21 includes: a first receiving unit 211, a first storing unit 212, a first retrieval determining unit 213, a first updating unit 214, and a first transmitting unit 215; wherein,

the first receiving unit 211 is configured to receive a data read request or a data write request of the client 23;

the first storage unit 212 is used for storing metadata;

the first retrieval determining unit 213 is configured to, when the first receiving unit 211 receives a data read request or a data write request, determine, as a storage server 22 responding to the data read request or the data write request, a storage server 22 whose load retrieved according to the data read request or the data write request does not exceed a set threshold;

the first updating unit 214 is configured to update the metadata of the storage server 22 responding to the data read request or the data write request determined by the first retrieval determining unit 213 of the first storage unit 212;

the first sending unit 215 is configured to send, to the client 23, information notifying the storage server 22 responding to the data read request or the data write request determined by the first retrieval determining unit 213.

Preferably, the first retrieving and determining unit 213 is further configured to, when the first receiving unit 211 receives a data reading request from the client 23, retrieve, according to the metadata of the first storing unit 212, a storage server 22 where data is read by the data reading request, and determine, as a storage server 22 responding to the data reading request, a storage server 22 whose load does not exceed a set threshold in the retrieved storage servers 22; when the first receiving unit 211 receives a data writing request from the client 23, the storage server 22 meeting the storage space requirement of the data writing request is retrieved according to the metadata of the first storage unit 212, and the storage server 22 with a load not exceeding a set threshold in the retrieved storage server 22 is determined as the storage server 22 responding to the data writing request.

Preferably, the first updating unit 214 is further configured to update load information in the metadata of the storage server 22 of the first storage unit 212 responding to the data read request; or updating the load information and the remaining storage space information in the metadata of the storage server 22 of the first storage unit 212 responding to the data write request.

Preferably, the first retrieval determining unit 213 is further configured to, when the first receiving unit 211 receives a data writing request from the client 23, retrieve, according to the metadata of the first storage unit 212, a storage server 22 with a load not exceeding a set threshold, and determine the retrieved storage server 22 as a storage server 22 backing up data written by the client 23 into the storage server 22 responding to the data writing request;

the first updating unit 214 is configured to update the data identification information, the load information, and the remaining storage space information of the metadata in the first storage unit 212 when the first retrieval determining unit 213 determines the backup storage server 22, where the metadata is the metadata of the backup storage server 22 determined by the first retrieval determining unit 213;

the first sending unit 215 is further configured to send, to the storage server 22 responding to the data writing request, a backup instruction for sending the data written by the client 23 to the backup storage server 22 when the first retrieval determining unit 213 determines that the backup storage server 22 is used.

Preferably, the first storage unit 212 is further configured to generate metadata synchronization information;

the first sending unit 215 is further configured to send the metadata synchronization information generated by the first storage unit 212 to the metadata server 21; the periodic heartbeat information is sent to the metadata server 21.

Preferably, the metadata server 21 further includes:

a first analyzing unit 216, configured to analyze, according to the metadata of the first storage unit 212, a frequency that the client 23 reads the same data through a data read request;

the first retrieval determining unit 213 is further configured to, when the first analyzing unit 216 analyzes that the frequency of reading the same data by the client 23 through a data read request exceeds a preset maximum value, retrieve the storage server 22 meeting the requirement of the data storage space according to the metadata of the first storage unit 212, and determine the storage server 22 with a load not exceeding a preset threshold value in the retrieved storage servers 22 as the storage server 22 for backing up the data; when the first analysis unit 216 analyzes that the frequency of reading the same data by the client 23 through a data read request is less than a preset minimum value, determining that one or more of the storage servers 22 in which the data is located delete the data;

the first sending unit 215 is further configured to send a backup instruction for sending the data to the backup storage server 22 determined by the first retrieval determining unit 213 to the storage server 22 where the data is located; sending a backup deletion instruction to delete the data to the one or more storage servers 22 that delete the data determined by the first retrieval determining unit 213;

the first updating unit 214 is further configured to update the data identification information and the remaining storage space information in the metadata of the backup storage server 22 in the first storage unit 212, which is determined by the first retrieval determining unit 213; the data identification information and the remaining storage space information in the metadata of one or more storage servers 22 of the first storage unit 212 in which the data is deleted by the first retrieval unit are updated.

Preferably, the metadata server 21 further includes:

a first detecting unit 217, configured to detect the number of storage servers 22 in the cloud storage environment;

the first retrieval determining unit 213 is further configured to, when the first detecting unit 217 detects that the cloud storage environment newly joins the storage server 22, sort, according to the metadata of the first storage unit 212, the frequency at which the client 23 reads the same data through a data read request; when the first detection unit 217 detects that the number of the storage servers 22 in the cloud storage environment is reduced, retrieving data which is not backed up according to the metadata of the first storage unit 212, and determining the storage server 22 which meets the storage space requirement of the data and has a load not exceeding a set threshold value in the storage servers 22 which do not back up the data as the storage server 22 which backs up the data;

the first sending unit 215 is further configured to send, according to the sorting result of the first retrieval determining unit 213, a backup instruction for sending the data to the newly added storage server 22 to the storage server 22 where the preset number of data with the highest reading frequency is located; sending a backup instruction for sending the data without backup to the backup storage server 22 determined by the first retrieval determining unit 213;

the first updating unit 214 is further configured to update the data identification information and the remaining storage space information of the metadata of the first storage unit 212 when the first sending unit 215 sends the backup instruction to the newly joining storage server 22, where the metadata is the metadata of the newly joining storage server 22; when the first sending unit 215 sends a backup instruction to the backup storage server 22, the data identification information and the remaining storage space information of the metadata of the first storage unit 212 are updated, where the metadata is the metadata of the backup storage server 22.

Preferably, the storage server 22 includes: a second receiving unit 221 and a second transmitting unit 222; wherein,

the second receiving unit 221 is configured to receive a backup instruction sent by the metadata server 21 to send the data written by the client 23 to the backup storage server 22.

The second sending unit 222 is configured to send, according to the backup instruction received by the second receiving unit 221, the data written by the client 23 to the backup storage server 22 specified by the backup instruction.

Preferably, the second receiving unit 221 is further configured to receive a backup instruction sent by the metadata server 21 and used for sending the data, of which the reading frequency exceeds a preset maximum value through the data reading request, of the client 23 to the backup storage server 22; receiving a backup deletion instruction sent by the metadata server 21 to delete the data with the reading frequency of the client 23 being less than the preset minimum value through the data reading request.

Preferably, the second receiving unit 221 is further configured to receive a backup instruction sent by the metadata server 21 and used for sending a preset number of data with the highest reading frequency of the client 23 through the data reading request to the backup storage server 22; receiving a backup instruction sent by the metadata server 21 to send data without backup to the backup storage server 22.

Preferably, the slave metadata server 24 includes: a third receiving unit 241, a third storing unit 242, a third retrieval determining unit 243, a third updating unit 244, and a third transmitting unit 245; wherein,

the third receiving unit 241 is configured to receive periodic heartbeat information sent by the metadata server 21; receiving a data read request or a data write request of the client 23;

the third storage unit 242 is configured to store metadata;

the third retrieval determining unit 243 is configured to determine that the metadata server 21 is unavailable when the third receiving unit 241 does not receive heartbeat information within a preset time, retrieve a storage server 22 with a load not exceeding a set threshold according to a data read request or a data write request received by the third receiving unit 241, and determine the retrieved storage server 22 as a storage server 22 responding to the data read request or the data write request;

the third updating unit 244 is configured to update the metadata of the third storage unit 242, where the metadata is the metadata of the storage server 22 that is determined by the third retrieval determining unit 243 and responds to the data read request or the data write request;

the third sending unit 245 is configured to send, to the client 23, information notifying the storage server 22 responding to the data read request or the data write request determined by the third retrieval determining unit 243.

Fig. 3 is a schematic flow chart illustrating an implementation process of a cloud storage environment data processing method according to an embodiment of the present invention, as shown in fig. 3, including:

step 301: the client sends a data writing request to the metadata server;

step 302: the metadata server retrieves and determines that the storage server 1 responds to the data writing request according to local metadata, the storage server 2 backs up data written into the storage server 1 by the client, and the metadata of the storage server 1 and the storage server 2 are updated;

the metadata server retrieves a storage server 1 and a storage server 2 which meet the storage space requirement of the data writing request and have a load not exceeding a preset threshold value according to the data writing request and local metadata, determines that the storage server 1 is a storage server responding to the data writing request, and updates data identification information, load information and residual storage space information in the metadata of the storage server 1; and determining that the storage server 2 is a storage server for backing up data written into the storage server 1 by the client, and updating data identification information and residual storage space information in the metadata of the storage server 2.

Step 303: the metadata server sends metadata synchronization information to the subordinate metadata servers;

wherein, the metadata synchronization information carries the metadata of the storage server 1 and the storage server 2 updated in step 302.

Step 304: the metadata server returns notification information of the storage server 1 responding to the data writing request to the client;

step 305: the metadata server sends a backup instruction to the storage server 1;

the backup instruction is an instruction for sending data written by the client into the storage server 1 to the storage server 2.

It should be noted that the execution order of steps 303, 304, and 305 may be changed arbitrarily.

Step 306: the client sends data to be written to the storage server 1;

before sending the data to be written to the storage server 1, the client may also send a data writing request to the storage server 1, and after receiving the confirmation information returned by the storage server 1, send the data to be written to the storage server 1.

Step 307: the storage server 1 sends data written by the client to the storage server 2;

after writing the data sent by the client to the local storage server 1 in step 306, according to the backup instruction received in step 305, sending the data written by the client in step 306 to the storage server 2, and backing up the data by the storage server 2.

Step 308: the storage server 2 returns data writing success information to the client;

step 309: a client sends a data reading request to a metadata server;

step 310: the metadata server retrieves and determines that the storage server 1 and the storage server 2 respond to the data reading request according to local metadata;

the metadata server retrieves that the storage servers where the data read by the data write request are located are a storage server 1 and a storage server 2 according to the data write request and the local metadata, the loads of the storage server 1 and the storage server 2 do not exceed a preset threshold, and the minimum value of the number of the preset storage servers responding to the data read request is 2, and then the storage server 1 and the storage server 2 are determined to respond to the data read request.

Step 311: the metadata server sends notification information of response of the storage server 1 and the storage server 2 to the data reading request to the client;

step 312: the client sends a data reading request to the storage server 1 and the storage server 2;

step 313: the storage server 1 and the storage server 2 send data read by the data read request to the client.

In the processing step, the slave metadata server further receives heartbeat information periodically sent by the metadata server, and when the slave metadata server initiates that the heartbeat information of the metadata server is not received within a preset time, the slave metadata server replaces the metadata server to interact with the client, the storage server 1 and the storage server 2 so as to respond to a data read request or a data write request of the client, and the specific interaction processing is the same as the interaction of the metadata server, the storage server and the client in steps 301 to 313, and is not described again.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.

Claims

1. A cloud storage environment data processing method is characterized by comprising the following steps:

the metadata server updates the metadata of the local storage server responding to the data reading request or the data writing request and informs the client of the information of the storage server responding to the data reading request or the data writing request;

the metadata of the storage server comprises identification information of data stored by the storage server, load information of the storage server and residual storage space information;

the load of the storage server comprises a data read request or a data write request responded by the storage server and response time;

2. The method according to claim 1, wherein when the metadata server receives a data read request or a data write request from a client, determining a storage server with a load not exceeding a preset threshold value, which is retrieved according to the data read request or the data write request, as a storage server responding to the data read request or the data write request, comprises:

3. The method of claim 1, wherein the metadata server updates the metadata of the local storage server responding to the data read request or the data write request, and comprises:

4. The method according to claim 1 or 2, wherein when the metadata server receives a data write request from a client, the method further comprises:

5. The method of claim 1, further comprising:

6. The method of claim 1, 2, 3 or 5, further comprising:

7. A metadata server, characterized in that the metadata server comprises: the device comprises a first receiving unit, a first storage unit, a first retrieval determining unit, a first updating unit and a first sending unit; wherein,

the first storage unit is used for storing metadata;

the first sending unit is configured to send, to the client, information notifying the storage server that responds to the data read request or the data write request and is determined by the first retrieval determining unit;

8. The metadata server of claim 7,

the first retrieval determining unit is further configured to, when the first receiving unit receives a data read request from the client, retrieve, according to the metadata of the first storage unit, a storage server where data read by the data read request is located, and determine, as a storage server responding to the data read request, a storage server whose load does not exceed a set threshold in the retrieved storage servers; when the first receiving unit receives a data writing request of the client, according to the metadata of the first storage unit, a storage server meeting the storage space requirement of the data writing request is searched, and a storage server with the load not exceeding a set threshold value in the searched storage server is determined as a storage server responding to the data writing request.

9. The metadata server of claim 7,

the first updating unit is further configured to update load information in metadata of the storage server of the first storage unit responding to the data read request; or updating load information and remaining storage space information in the metadata of the storage server of the first storage unit responding to the data write request.

10. The metadata server according to claim 7 or 8,

the first retrieval determining unit is further configured to, when the first receiving unit receives a data write request from the client, retrieve, according to the metadata of the first storage unit, a storage server whose load does not exceed a set threshold, and determine the retrieved storage server as a storage server that backs up data written by the client into the storage server that responds to the data write request;

11. The metadata server of claim 7,

the first storage unit is also used for generating metadata synchronization information;

12. The metadata server according to claim 7, 8, 9 or 11, wherein the metadata server further comprises:

13. A storage server, the storage server comprising: a second receiving unit and a second transmitting unit; wherein,

the second receiving unit is used for receiving a backup instruction which is sent by the metadata server and used for sending the data written by the client to the backup storage server;

the second sending unit is configured to send, according to the backup instruction received by the second receiving unit, data written by the client to a backup storage server specified by the backup instruction;

the second receiving unit is further configured to receive a backup instruction sent by the metadata server and used for sending the data with the reading frequency exceeding a preset maximum value through the data reading request from the client to the backup storage server; and receiving a backup deletion instruction sent by the metadata server for deleting the data of which the reading frequency of the client through the data reading request is less than the preset minimum value.

14. The storage server of claim 13,

the second receiving unit is further configured to receive a backup instruction sent by the metadata server and used for sending a preset amount of data with the highest reading frequency of the client through the data reading request to the backup storage server; and receiving a backup instruction which is sent by the metadata server and sends data without backup to the backup storage server.

15. A cloud storage environment data processing system, the system comprising: the system comprises a client, at least two storage servers and a metadata server; wherein,

the metadata server is used for determining a storage server with a load not exceeding a preset threshold value retrieved according to a data read request or a data write request as a storage server responding to the data read request or the data write request when receiving the data read request or the data write request of a client; updating the metadata of the local storage server responding to the data reading request or the data writing request, and informing the client of the information of the storage server responding to the data reading request or the data writing request;

the metadata server is further configured to, when it is analyzed according to local metadata that the frequency of reading the same data by the client through a data read request exceeds a preset maximum value, retrieve a storage server meeting the data storage space requirement according to the local metadata, determine a storage server with a load not exceeding a preset threshold value among the retrieved storage servers as a storage server for backing up the data, send a backup instruction for sending the data to the backup storage server to the storage server where the data is located, and update data identification information and remaining storage space information in the metadata of the backup storage server; when the frequency of reading the same data by a client through a data reading request is smaller than a preset minimum value according to local metadata analysis, determining one or more storage servers where the data are located to delete the data, sending a backup deletion instruction for deleting the data to the one or more storage servers, and updating data identification information and residual storage space information in the metadata of the one or more storage servers.

16. The system of claim 15,

the metadata server is further configured to, when receiving a data read request from the client, retrieve a storage server where the data read request read data is located according to local metadata, and determine a storage server with a load not exceeding a preset threshold value in the retrieved storage server as a storage server responding to the data read request; and when a data writing request of the client is received, searching a storage server meeting the storage space requirement of the data writing request according to local metadata, and determining the storage server with the load not exceeding a preset threshold value in the searched storage server as the storage server responding to the data writing request.

17. The system of claim 15,

the metadata server is further used for updating the load information in the metadata of the local storage server responding to the data read request; or updating data identification information, load information and residual storage space information in the local metadata of the storage server responding to the data writing request.

18. The system of claim 15,

and the metadata server is further used for determining the retrieved storage server as a storage server for backing up the data of the storage server written by the client and responding to the data writing request according to the storage server with the local metadata retrieval load not exceeding a set threshold value when receiving the data writing request of the client, updating data identification information, load information and residual storage space information in the metadata of the backup storage server, and sending a backup instruction for sending the data written by the client to the backup storage server to respond to the data writing request.

19. The system of claim 15, further comprising:

the slave metadata server is used for receiving the metadata synchronized to the local by the metadata server; and when the heartbeat information of the metadata server is not received within preset time, determining that the metadata server is unavailable, and processing a data reading request or a data writing request of the client instead of the metadata server.

20. The system according to any one of claims 15 to 19,

the metadata server is further configured to, when detecting that the cloud storage environment newly joins a storage server, sequence, according to local metadata, a frequency at which the client reads the same data through a data read request, send, to a storage server where a preset number of data with the highest read frequency are located, a backup instruction for sending the data to the newly joined storage server, and update data identification information and remaining storage space information in the metadata of the newly joined storage server; when the fact that the number of the storage servers in the cloud storage environment is reduced is detected, retrieving data without backup according to local metadata, determining the storage servers which meet the storage space requirement of the data and have loads not exceeding a set threshold value in the storage servers without the backup data as the storage servers for backing up the data, sending a backup instruction for sending the data to the backup storage servers to the storage servers storing the data, and updating data identification information and residual storage space information of the metadata of the backup storage servers.