CN114237986A - Cluster availability mode control method, device, equipment and storage medium - Google Patents

Cluster availability mode control method, device, equipment and storage medium Download PDF

Info

Publication number
CN114237986A
CN114237986A CN202111613066.4A CN202111613066A CN114237986A CN 114237986 A CN114237986 A CN 114237986A CN 202111613066 A CN202111613066 A CN 202111613066A CN 114237986 A CN114237986 A CN 114237986A
Authority
CN
China
Prior art keywords
mode
target
availability
cluster
read
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111613066.4A
Other languages
Chinese (zh)
Inventor
侯永进
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Telecom Corp Ltd
Original Assignee
China Telecom Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Telecom Corp Ltd filed Critical China Telecom Corp Ltd
Priority to CN202111613066.4A priority Critical patent/CN114237986A/en
Publication of CN114237986A publication Critical patent/CN114237986A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • G06F11/1464Management of the backup or restore process for networked environments

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a method, a device, equipment and a storage medium for controlling cluster availability modes, wherein the method comprises the following steps: determining target user data in a target cluster; identifying a first mode corresponding to a target cluster by using read-write state information of target user data in a first historical period; predicting a second mode corresponding to the target cluster by using the read-write state information of the target user data in a second historical period; judging whether the first mode and the second mode are the same to obtain a judgment result; determining an availability mode to be adapted of the target cluster as a target mode based on the judgment result; and if the target mode is different from the current availability mode of the target cluster, converting the availability mode of the target cluster based on the target mode. By the scheme, the availability of user data can be guaranteed, and the utilization rate of remote storage resources can be improved.

Description

Cluster availability mode control method, device, equipment and storage medium
Technical Field
The invention relates to the technical field of cloud computing data storage, in particular to a cluster availability mode control method, a cluster availability mode control device, cluster availability mode control equipment and a storage medium.
Background
In order to ensure the availability of data and enable a user to obtain complete and reliable data at any time according to requirements, a storage node for storing backup data needs to be added into a storage node list of a cluster in advance, and then disaster recovery is performed by means of a scheduling algorithm of a system, namely, the user data is backed up by using the storage node for storing the backup data.
Meanwhile, in order to combat regional and large-scale extreme natural disasters, at least one mirror image cluster needs to be built in a data center in a different place to realize disaster recovery. However, this method has high requirements on the network and infrastructure, and is costly, and moreover, the remote backup of the user data in the cluster with a low access amount also occupies a large amount of storage resources, resulting in waste of energy.
Therefore, how to reduce the occupation of remote storage resources while ensuring the availability of data is an urgent problem to be solved.
Disclosure of Invention
Embodiments of the present invention provide a method, an apparatus, a device, and a storage medium for controlling a cluster availability mode, so as to improve a utilization rate of a remote storage resource while ensuring availability of user data, and the specific technical solution is as follows:
in a first aspect, an embodiment of the present invention provides a cluster availability mode control method, which is applied to an electronic device, and the method includes:
determining target user data in a target cluster; wherein the target cluster is a cluster to be controlled with respect to an availability mode; the categories of the availability patterns include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
identifying a first mode corresponding to the target cluster by using the read-write state information of the target user data in a first historical period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
predicting a second mode corresponding to the target cluster by using the read-write state information of the target user data in a second historical period; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
judging whether the first mode and the second mode are the same or not to obtain a judgment result;
determining an availability mode to be adapted of the target cluster as a target mode based on the judgment result;
and if the target mode is different from the current availability mode of the target cluster, converting the availability mode of the target cluster based on the target mode.
Optionally, the electronic device is in communication with a master control node in the target cluster;
the converting the availability mode of the target user data based on the target mode comprises:
and sending a mode conversion instruction which is specific to the target cluster and carries the mode information of the target mode to the main control node, so that the main control node converts the availability mode of the target cluster into the target mode after receiving the mode conversion instruction.
Optionally, the determining, based on the determination result, an availability mode to be adapted for the target cluster, as a target mode, includes:
if the judgment result shows that the first mode and the second mode are different, determining the current availability mode of the target cluster as the availability mode to be adapted of the target cluster to obtain the target mode;
and if the judgment result shows that the first mode and the second mode are the same, determining the second mode as the availability mode to be adapted of the target cluster to obtain the target mode.
Optionally, the identifying a first mode corresponding to the target cluster by using the read-write state information of the target user data in a first history period includes:
determining the read-write state information of the target user data in each target sub-period; wherein, each target sub-period is each sub-period included in the first history period;
for each target sub-period, calculating a judgment index value related to the read-write state in the target sub-period based on the read-write state information of the target user data in the target sub-period;
if the ratio of the number of the judgment index values meeting the first condition in the determined judgment index values is greater than a preset ratio, determining that the first mode corresponding to the target cluster is a disaster recovery mode which represents that remote backup is not performed; wherein the first condition is less than a first preset threshold;
if the ratio of the number of judgment index values meeting the second condition in the determined judgment index values is larger than a preset ratio, determining that the first mode corresponding to the target cluster is a disaster recovery mode representing remote full data backup; the second condition is that the second condition is greater than a second preset threshold value, and the second preset threshold value is greater than the first preset threshold value;
otherwise, determining that the first mode corresponding to the target cluster is: and characterizing a disaster recovery mode of remote incremental backup.
Optionally, the read-write status information includes: information representing the frequency of occurrence of reading and writing, and/or the amount of data read and written;
for each target sub-period, calculating a judgment index value related to the read-write state in the target sub-period based on the read-write state information of the target user data in the target sub-period, including:
and aiming at each target sub-period, calculating the average value of the read-write frequency and/or the average value of the read-write data quantity in the target sub-period based on the read-write state information of the target user data in the target sub-period.
Optionally, the predicting a second mode corresponding to the target cluster by using the read-write state information of the target user data in a second history period includes:
inputting the read-write state information of the target user data in a second historical period into a pre-trained mode prediction model to obtain a mode prediction result, wherein the mode prediction result is used as a second mode corresponding to the target cluster;
the mode prediction model is a model obtained by training based on sample read-write state information and corresponding label values, and the sample read-write state information is as follows: and reading and writing state information of the sample user data in a sample time interval, wherein the label value is the predicted availability mode in the specified time interval after the sample time interval.
Optionally, the initial availability mode of the target cluster is a mode set according to a default disaster recovery rule of the system;
the default disaster recovery rule of the system comprises the following steps:
if the availability grade of the target cluster is a first grade, the initial availability mode is a disaster recovery mode representing remote full data backup;
if the availability grade of the target cluster is a second grade, the initial availability mode is a disaster recovery mode representing remote incremental backup;
if the availability grade of the target cluster is a third grade, the initial availability mode is a disaster recovery mode which represents that remote backup is not carried out;
wherein the first level is higher than the second level, which is higher than a third level.
In a second aspect, an embodiment of the present invention provides a device for controlling a cluster availability mode, where the device is applied to an electronic device, and the device includes:
the first determining module is used for determining target user data in a target cluster; wherein the target cluster is a cluster to be controlled with respect to an availability mode, and the types of the availability mode include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
the identification module is used for identifying a first mode corresponding to the target cluster by utilizing the read-write state information of the target user data in a first historical period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
the prediction module is used for predicting a second mode corresponding to the target cluster by utilizing the read-write state information of the target user data in a second historical period; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
the judging module is used for judging whether the first mode and the second mode are the same or not to obtain a judging result;
a second determining module, configured to determine, based on the determination result, an availability mode to be adapted for the target cluster as a target mode;
and the conversion module is used for converting the availability mode of the target cluster based on the target mode if the target mode is different from the current availability mode of the target cluster.
Optionally, the conversion module is specifically configured to:
if the target mode is different from the current availability mode of the target cluster, sending a mode conversion instruction carrying mode information of the target mode to the main control node, so that the main control node converts the availability mode of the target cluster into the target mode after receiving the mode conversion instruction.
Optionally, the second determining module includes:
a first determining sub-module, configured to determine, if the determination result indicates that the first mode and the second mode are different, a current availability mode of the target cluster as an availability mode to be adapted to the target cluster, so as to obtain a target mode;
and the second judgment submodule is used for determining the second mode as the availability mode to be adapted to the target cluster to obtain the target mode if the judgment result shows that the first mode and the second mode are the same.
Optionally, the identification module includes:
the determining submodule is used for determining the read-write state information of the target user data in each target sub-period; wherein, each target sub-period is each sub-period included in the first history period;
the calculation sub-module is used for calculating a judgment index value related to the read-write state in each target sub-period based on the read-write state information of the target user data in the target sub-period aiming at each target sub-period;
the first setting submodule is used for determining that a first mode corresponding to the target cluster is a disaster recovery mode which represents that remote backup is not carried out if the proportion of the number of judgment index values meeting a first condition in the determined judgment index values is greater than a preset proportion; wherein the first condition is less than a first preset threshold;
the second setting submodule is used for determining that the first mode corresponding to the target cluster is a disaster recovery mode representing remote full data backup if the proportion of the number of judgment index values meeting the second condition in the determined judgment index values is greater than the preset proportion; the second condition is that the second condition is greater than a second preset threshold value, and the second preset threshold value is greater than the first preset threshold value;
a third setting submodule, configured to determine, in addition to the above, that the first mode corresponding to the target cluster is: and characterizing a disaster recovery mode of remote incremental backup.
Optionally, the read-write status information includes: information representing the frequency of occurrence of reading and writing, and/or the amount of data read and written;
the calculation submodule is specifically configured to:
and aiming at each target sub-period, calculating the average value of the read-write frequency and/or the average value of the read-write data quantity in the target sub-period based on the read-write state information of the target user data in the target sub-period.
Optionally, the prediction module includes:
the analysis submodule is used for inputting the read-write state information of the target user data in a second historical period into a pre-trained mode prediction model to obtain a mode prediction result which is used as a second mode corresponding to the target cluster;
the mode prediction model is a model obtained by training based on sample read-write state information and corresponding label values, and the sample read-write state information is as follows: and reading and writing state information of the sample user data in a sample time interval, wherein the label value is the predicted availability mode in the specified time interval after the sample time interval.
Optionally, the initial availability mode of the target cluster is a mode set according to a default disaster recovery rule of the system;
the default disaster recovery rule of the system comprises the following steps:
if the availability grade of the target cluster is a first grade, the initial availability mode is a disaster recovery mode representing remote full data backup;
if the availability grade of the target cluster is a second grade, the initial availability mode is a disaster recovery mode representing remote incremental backup;
if the availability grade of the target cluster is a third grade, the initial availability mode is a disaster recovery mode which represents that remote backup is not carried out;
wherein the first level is higher than the second level, which is higher than a third level.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the steps of the cluster availability mode control method when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the above cluster availability mode control method.
Embodiments of the present invention further provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the above cluster availability mode control method.
The embodiment of the invention has the following beneficial effects:
according to the cluster availability mode control method provided by the embodiment of the invention, a first mode is obtained according to the read-write state information in a first historical time period of target user data in a target cluster, and a second mode is obtained by performing trend prediction according to the read-write state information in a second historical time period; and determining a target mode by judging whether the first mode and the second mode are the same, wherein the types of the availability modes comprise a disaster recovery mode for representing remote full data backup, a disaster recovery mode for representing remote incremental backup and a disaster recovery mode for representing no remote backup. Therefore, the scheme can flexibly switch the availability mode of the target cluster, thereby saving storage resources; and judging whether a first mode obtained by reading and writing state information in a first historical time period is the same as a second mode obtained by performing trend prediction according to the state information in a second historical time period, and then determining the target mode to ensure the credibility of the target mode. Therefore, the scheme can ensure the availability of the user data and improve the utilization rate of the remote storage resources.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by referring to these drawings.
Fig. 1 is a flowchart of a cluster availability mode control method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an overall architecture of a data storage system according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an initialization availability mode setting module according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating an algorithm of a recent data read/write status determination module according to an embodiment of the present invention;
fig. 5 is a flowchart illustrating prediction performed by the historical long-period data read-write habit analysis module according to the embodiment of the present invention;
fig. 6 is a flowchart of a decision making process performed by the storage availability security posture determining and deciding module according to an embodiment of the present invention;
FIG. 7 is a flowchart of an embodiment of an availability security decision making execution module;
FIG. 8 is a functional diagram of an inter-pool communication module according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a cluster availability mode control apparatus according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments given herein by one of ordinary skill in the art, are within the scope of the invention.
In the cloud computing PAAS (Platform as a Server), data availability means that a user can obtain complete and reliable data from a distributed storage cluster at any time according to needs. The distributed storage cluster is generally located in a resource pool of the same area, such as a city. The clusters can realize distributed storage of large-scale unstructured data such as pictures, voice, videos and the like, and are used for supporting user service application. In order to guarantee the availability of data, disaster recovery backup needs to be performed on user data in a cluster. In the related art, storage nodes for storing backup data are added to a storage node list of a cluster in advance, and then user data is backed up by using the storage nodes for storing the backup data. In the implementation mode, the disaster tolerance range of one cluster is limited to the storage nodes covered by the cluster, and one cluster is generally located in one region, although the cost is low, if regional and large-range extreme natural disasters occur, the single-cluster disaster recovery mode is greatly examined, and at the moment, the cloud service is exposed to risks of data loss and service interruption caused by serious disasters.
In addition, if disaster recovery is realized by building mirror image clusters in a data center in different places (two clusters are generally built), the requirements on networks and infrastructures are high, the cost is high, user data in the clusters with low access volumes are backed up in different places, a large amount of storage resources are occupied, and a large amount of energy is consumed by equipment in the clusters in different places during operation.
In order to solve the above problem, and reduce the occupation of storage resources in different places while ensuring the availability of data, embodiments of the present invention provide a method, an apparatus, a device, and a storage medium for controlling a cluster availability mode.
First, a method for controlling a cluster availability mode according to an embodiment of the present invention is described below.
The cluster availability mode control method provided by the embodiment of the invention can be applied to electronic equipment. For example, in one implementation, the electronic device may be a device that is independent of and in communication with a master control node in the cluster. In the implementation mode, the electronic equipment is independent of the main control node and is physically separated from the cluster storage layer, and machine resources of the storage cluster cannot be occupied. In a specific application, the electronic device may be a server or a terminal device, for example: smart phones, tablets, desktop computers, all of which are reasonable.
For example, in another implementation, the electronic device may also be a master control node in a storage cluster. In this implementation, the master control node can also implement intelligent control over the mode of the user data on the premise of having a management function for each storage node in the storage cluster.
Specifically, the execution subject of the cluster availability mode control method may be a cluster availability mode control device. For example, when the cluster availability mode control method is applied to a main control node, the cluster availability mode control device may be a computer program running in the main control node, and the computer program may be used to implement switching control of the availability mode. When the cluster availability mode control method is applied to a terminal device independent from a main control node, the cluster availability mode control device may be functional software running in the terminal device, for example: cluster availability pattern control software. When the cluster availability mode control method is applied to a server independent from the main control node, the cluster availability mode control means may be a computer program running in the server, and the computer program may be used to implement mode switching control. It can be understood that, no matter the cluster availability mode control method is applied to the terminal device or the server which is independent from the master control node, the cluster availability mode control device is separately deployed in a device, and is physically separated from the storage layer of the cluster, that is, separated from the storage nodes in the storage cluster, so that the machine resources of the storage cluster are not occupied.
In order to implement the cluster availability mode control method of the present invention, firstly, it is necessary to ensure that the existing cluster can normally operate, mount the client and the storage path of the distributed storage system on the user side, and ensure that the cluster can normally provide services for the user. And setting relevant basic parameters and relevant resource configurations in a storage layer of the cluster.
And the key of the master node of the distributed storage cluster is ensured to be sent to the slave nodes, so that the slave distributed storage cluster master node can normally access each slave node to realize data synchronization.
The cloud service operator needs to divide a certain storage space in another resource pool, which is located far enough away from the area where the resource pool is deployed in the distributed storage cluster, for example, another city with a distance of one kilometer away, and has stable and reliable network conditions, for backup of user data at a higher guarantee level. The storage space only needs to provide necessary configuration information to the system, and no additional operation is needed. The system can be reused by different storage clusters and can be adjusted according to the situation of storage resource surplus sources and the operation situation of companies.
The cluster availability mode control method provided by the embodiment of the invention can comprise the following steps:
determining target user data in a target cluster; wherein the target cluster is a cluster to be controlled with respect to an availability mode, and the types of the availability mode include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
identifying a first mode corresponding to the target cluster by using the read-write state information of the target user data in a first historical period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
predicting a second mode corresponding to the target cluster by using the read-write state information of the target user data in a second historical period; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
judging whether the first mode and the second mode are the same or not to obtain a judgment result;
determining an availability mode to be adapted of the target cluster as a target mode based on the judgment result;
and if the target mode is different from the current availability mode of the target cluster, converting the availability mode of the target cluster based on the target mode.
In the scheme provided by the embodiment of the invention, a first mode is obtained according to the read-write state information in a first historical time period of target user data in a target cluster, and a second mode is obtained by performing trend prediction according to the read-write state information in a second historical time period; and determining a target mode by judging whether the first mode and the second mode are the same, wherein the types of the availability modes comprise a disaster recovery mode for representing remote full data backup, a disaster recovery mode for representing remote incremental backup and a disaster recovery mode for representing no remote backup. Therefore, the scheme can flexibly switch the availability mode of the target cluster, thereby saving storage resources; and judging whether a first mode obtained by reading and writing state information in a first historical time period is the same as a second mode obtained by performing trend prediction according to the state information in a second historical time period, and then determining the target mode to ensure the credibility of the target mode. Therefore, the scheme can ensure the availability of the user data and improve the utilization rate of the remote storage resources.
The following describes a cluster availability mode control method provided by the embodiment of the present invention with reference to the accompanying drawings.
As shown in fig. 1, the cluster availability mode control method provided in the embodiment of the present invention is applied to an electronic device; the method may comprise steps S101-S106:
s101, determining target user data in a target cluster; the target cluster is a cluster to be controlled about an availability mode, and the target user data is all user data contained in the target cluster; the categories of the availability patterns include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
for example, the target user data may be all user data included in the target cluster; of course, the target user data may also be part of the data in the target cluster, such as: a specified type of user data, data stored for a specified period of time, and the like.
The embodiment of the invention provides three availability modes for configuring a target cluster:
availability mode 1: representing a disaster recovery mode of remote full data backup; that is, in this mode, the backup is performed on the full amount of the target user data in the target cluster in the storage space in the resource pool in the different place.
Availability mode 2: representing a disaster recovery mode of remote incremental backup; that is, in this mode, only the incremental data of the target cluster is backed up in different places;
availability mode 3: and characterizing a disaster recovery mode without remote backup.
In addition, the initial availability guarantee mode of the target cluster may adopt a default configuration mode, for example: characterizing a disaster recovery mode without remote backup; and operation and maintenance personnel can set a required mode according to the requirements of users. In addition, for the initial availability guarantee mode, the setting weight of the operation and maintenance personnel may be higher than the weight of the default configuration, and if the operation and maintenance personnel do not actively set, the operation and maintenance personnel use the availability guarantee mode of the default configuration as the initial mode of the cluster.
For example, in an implementation manner, the initial availability mode of the target cluster is a mode set according to a default disaster recovery rule of the system;
the default disaster recovery rule of the system comprises the following steps:
if the availability grade of the target cluster is a first grade, the initial availability mode is a disaster recovery mode representing remote full data backup;
if the availability grade of the target cluster is a second grade, the initial availability mode is a disaster recovery mode representing remote incremental backup;
if the availability grade of the target cluster is a third grade, the initial availability mode is a disaster recovery mode which represents that remote backup is not carried out;
wherein the first level is higher than the second level, which is higher than a third level.
Operation and maintenance personnel can comprehensively judge the safety level requirement required by data storage according to the customer requirement and the actual generation and application scene of data. It will be appreciated that the above-mentioned "first level", "second level" and "third level" are merely used to distinguish the different availability levels by nomenclature, and that the specific nomenclature adopted by "first level", "second level" and "third level" may in turn be "high level", "medium level" and "general" in a specific application, although not limited thereto.
In addition, in the cluster availability mode control method in this embodiment, the availability mode control for the target cluster may be performed periodically, that is, when entering each period, S101 to S106 may be performed.
The target cluster may be a newly established cluster in a local cluster storage layer, or a cluster that has been established before the cluster availability mode control method of the present invention is used, and the performance of the target cluster is required to meet the requirement of disaster recovery level for implementing the cluster availability mode control method of the present invention.
S102, identifying a first mode corresponding to the target cluster by using read-write state information of the target user data in a first historical period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
the read-write state information includes information representing the frequency of occurrence of reading and writing and/or the data amount of reading and writing, and the read-write state information may be the frequency of reading and writing in each unit time length and/or the data amount of reading and writing, for example, the daily read-write frequency. The first historical time period may be a recent historical time period starting from the current time point, for example: the first month, the first march, the first may, and so on. And, the first mode may be: and identifying an availability mode matched with the target cluster data in a first historical time period as a first mode by analyzing the read-write state information of the target user data in the target cluster in the first historical time period.
It can be understood that, because the read-write state information of the target user data in the target cluster in the first history time period represents the read-write state of the target user data in the first history time period, the availability mode adapted to the target cluster in the first history time period can be identified by analyzing the read-write state information of the target user data in the target cluster in the first history time period.
For example, if the read-write frequency in the first historical time period is higher, it may be considered that the importance degree of the target user data in the target cluster is higher, then the availability mode adapted to the target cluster may be a disaster recovery mode representing remote full data backup, so as to ensure the availability of the target user data in the target cluster; if the reading and writing frequency in the first historical time period is lower, the availability mode matched with the target cluster can be a disaster recovery mode which represents that remote backup is not carried out so as to meet the requirement of saving remote storage resources; if the read-write frequency in the first historical time period is moderate, the availability mode adapted to the target cluster can be a disaster recovery mode representing remote incremental backup, and in general, the access frequency of incremental data in the future is relatively high, so that the target cluster with low availability requirement can be only subjected to incremental backup in a resource pool in a remote place, and the availability can be guaranteed.
It should be noted that the read-write state information of the target user data in the first history period may be obtained from history log information of the target user data, and a specific obtaining manner is not limited in this embodiment of the present invention.
S103, predicting a second mode corresponding to the target cluster by using the read-write state information of the target user data in a second historical period; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
the second history period may be a long history period starting from the current time point, for example: the previous year, the previous two years, etc. The second mode may be: and predicting an availability mode matched with the target cluster in a specified time period after the current time by analyzing the read-write state information of the target user data in the target cluster in a second historical time period, wherein the availability mode is used as the second mode. Illustratively, the specified period may be a period adjacent to the current time, such as: one month, two months, etc. after the current time.
It can be understood that, because the read-write state information of the target user data in the target cluster in the second historical time period represents the read-write state of the target user data in the second historical time period, by analyzing the read-write state information of the target user data in the second historical time period, the rule of the read-write state change of the target user data in the second historical time period can be mined, and therefore, the availability mode adapted to the target cluster in the specified time period after the current time is predicted based on the rule of the read-write state change.
Optionally, in an implementation manner, predicting a second mode corresponding to the target cluster by using read-write state information of the target user data in a second history period may include:
inputting the read-write state information of the target user data in a second historical period into a pre-trained mode prediction model to obtain a mode prediction result, wherein the mode prediction result is used as a second mode corresponding to the target cluster;
the mode prediction model is a model obtained by training based on sample read-write state information and corresponding label values, and the sample read-write state information is as follows: and reading and writing state information of the sample user data in a sample time interval, wherein the label value is the predicted availability mode in the specified time interval after the sample time interval.
For example, in one implementation, the pattern prediction model may be a machine learning model, such as: the deep neural network model, at this time, the read-write state information in the sample period may be: the reading and writing frequency and/or the reading and writing data quantity of the sample user data in the previous N-1 year-N year, wherein N is any positive integer; the label value of the sample data is a true value of the availability mode corresponding to the sample user data in a month after the sample time period. The training process of the model may be: and inputting the read-write state information in the sample time period into an initial deep neural network model to obtain the output result of the predicted availability mode in a month after the sample time period, and adjusting the parameters of the deep neural network model by minimizing the difference between the output result and the label value to obtain a frequency prediction model meeting the requirement. The specific representation form of the sample read-write state information input to the model may be a vector form, but is not limited to this.
In another implementation manner, the mode prediction model may also be a mathematical model for implementing availability mode prediction, and likewise, the read-write state information in the sample period may be: the reading and writing frequency and/or the reading and writing data volume of the sample user data in the previous N-1 year-N year are/is determined, wherein N is any positive integer; the label value of the sample data is the true value of the availability pattern of the sample user data within one month after the sample period. The training process of the model may be: and establishing a mathematical expression between the output and the input of the model, wherein the input of the model is read-write state information in a sample time period, the output of the model is the output result of an availability mode matched with the sample user data in one month after the sample time period, whether the model is accurate is verified by comparing the output result of the model with a tag value, and parameters in the mathematical expression are modified based on the error between the output result of the model and the tag value, so that the required mathematical model is trained.
S104, judging whether the first mode and the second mode are the same to obtain a judgment result;
it can be understood that, if the first mode obtained through the above steps is the same as the second mode, it indicates that the recent read-write status of the target user data in the target cluster conforms to the rule of the long-term read-write status, and at this time, the predicted second mode is considered to be reliable. If the first mode is different from the second mode, the recent read-write frequency of the target user data is obviously different from the long-term read-write frequency rule, and the predicted second mode is considered to be unreliable.
And the first mode and the second mode are the same or different according to the judgment result.
S105, determining the availability mode to be adapted of the target cluster as a target mode based on the judgment result;
in this embodiment, the target mode is an availability mode to be set by the target cluster, and the target mode may be a first mode, or a predicted second mode, or a current availability mode of the target cluster.
For example, in a specific implementation manner, determining a target mode to be adapted to the target cluster based on the determination result may include steps a1-a 2:
if the judgment result shows that the first mode and the second mode are different, determining the current availability mode of the target cluster as the availability mode to be adapted of the target cluster to obtain the target mode;
and if the judgment result shows that the first mode is the same as the second mode, determining the second mode as the availability mode to be adapted of the target cluster to obtain the target mode.
It can be understood that, as the determination result in step S104 represents whether the rule of the recent read-write state and the long-term read-write state of the target user data conforms to, that is, whether the second mode is reliable, the second mode or the current availability mode of the target cluster may be taken as the target mode according to the determination result.
S106, if the target mode is different from the current availability mode of the target cluster, based on the target mode, the availability mode of the target cluster is converted.
Because the target user data in the target cluster can change in read-write state with time, there may be a situation that the read-write frequency is high in a certain period of time and the read-write frequency is low in another period of time. If the target user data is backed up in a fixed availability mode in a different place, a large amount of storage space of a different-place resource pool may be occupied by user data with a low access amount, and a large amount of energy is consumed when equipment in a different-place cluster operates. Therefore, after the target mode is determined in step S105, if the target mode is different from the current availability mode of the target cluster, the availability mode of the target cluster may be converted into the target mode, so as to improve the utilization rate of the remote storage resource.
Optionally, in another implementation manner, if the electronic device to which the cluster availability mode control method is applied is a device that is independent from the main control node, the converting, based on the target mode, the availability mode of the target cluster may include:
and sending a mode conversion instruction which is specific to the target cluster and carries the mode information of the target mode to the main control node, so that the main control node converts the availability mode of the target cluster into the target mode after receiving the mode conversion instruction.
Optionally, in another implementation manner, if the electronic device to which the cluster availability mode control method is applied is a master control node, then, at this time, based on the target mode, performing conversion processing on the availability mode of the target cluster may include:
converting the availability mode of the target cluster to the target mode. In addition, for convenience of understanding, the following takes different target modes as examples, and a specific implementation manner of converting the availability mode of the target cluster into the target mode is described.
For example, if the current availability mode of the target cluster is a disaster recovery mode that characterizes that no remote backup is performed, and the target mode is a disaster recovery mode that characterizes that remote full data backup is performed, the process of converting the availability mode of the target cluster into the target mode may be: the method comprises the steps of applying for dividing necessary storage space in a resource pool of different places, and backing up all user data in a target cluster to different places in a data transmission mode while keeping the storage performance of the original target cluster.
Illustratively, if the current availability mode of the target cluster is a disaster recovery mode representing remote incremental backup, and the target mode is a disaster recovery mode representing remote full data backup, the historical data can be gradually supplemented by using the copy data of the original target cluster, and finally, a full backup is constructed in the remote storage space.
For example, if the current availability mode of the target cluster is a disaster recovery mode representing remote full data backup, and the target mode is a disaster recovery mode representing no remote backup, the target user data of the target cluster that has been backed up in the remote storage space may be deleted, and the part of the storage space may be released.
Of course, the above-illustrated specific conversion process of the availability mode is only an example, and the embodiment of the present invention does not limit the mode conversion process.
In this embodiment, a first mode is obtained according to read-write state information of target user data in a target cluster in a first historical time period, and a second mode is obtained by performing trend prediction according to the read-write state information in a second historical time period; and determining a target mode by judging whether the first mode and the second mode are the same, wherein the types of the availability modes comprise a disaster recovery mode for representing remote full data backup, a disaster recovery mode for representing remote incremental backup and a disaster recovery mode for representing no remote backup. Therefore, the scheme can flexibly switch the availability mode of the target cluster, thereby saving storage resources; and judging whether a first mode obtained by reading and writing state information in a first historical time period is the same as a second mode obtained by performing trend prediction according to the state information in a second historical time period, and then determining the target mode to ensure the credibility of the target mode. Therefore, the scheme can ensure the availability of the user data and improve the utilization rate of the remote storage resources.
Optionally, in an implementation manner, identifying a first availability pattern corresponding to the target cluster by using the read-write status information of the target user data in the first history period may include steps B1-B5:
b1, determining the read-write state information of the target user data in each target sub-period; wherein, each target sub-period is each sub-period included in the first history period;
for example, if the duration of the first history period is three months, the first history period may be divided into three target sub-periods, each of which has a duration of one month.
B2, for each target sub-period, based on the read-write state information of the target user data in the target sub-period, calculating the evaluation index value about the read-write state in the target sub-period;
the evaluation index value of the read-write state may be an index value for evaluating the read-write state of the target user data in the target sub-period. If the read-write state information includes information representing the frequency of occurrence of reading and writing, the evaluation index value may be an average value, a median, or the like of the read-write frequency data in the target sub-period. In addition, the embodiment of the present invention does not limit the specific form of the evaluation index value.
For example, in a specific implementation manner, if the target sub-period is one month, the evaluation index value in the target sub-period may be an average value of the read-write frequency of each day in one month. For example, in another specific implementation manner, if the target sub-period is one month, the evaluation index value in the target sub-period may be a median of the read-write frequency of each day in the one month.
B3, if the proportion of the number of the judgment index values meeting the first condition in the determined judgment index values is larger than a preset proportion, determining that the first mode corresponding to the target cluster is a disaster recovery mode which represents that remote backup is not performed; wherein the first condition is less than a first preset threshold;
in this implementation, the first preset threshold may be set to a lower value related to the read-write status of the target user data. It can be understood that, after calculating the evaluation index values related to the read-write status in each target sub-period through step B2, there are a plurality of evaluation index values, and if the number of evaluation index values smaller than the first preset threshold occupies a proportion of the total number, which is greater than the preset proportion, the read-write frequency of the target user data in the first history period is low, or the read-write data volume is small, the importance degree of the target user data in the target cluster is considered to be not high, for example, the preset proportion may be set to be 50%, that is, the number of evaluation index values meeting the first condition is greater than half of the number of evaluation index values; of course, the preset ratio can be other ratios, such as: 55%, 60%, etc. At this time, it is determined that the first mode corresponding to the target cluster is a disaster recovery mode representing that remote backup is not performed, so as to save remote storage space.
B4, if the number of judgment index values meeting the second condition in the determined judgment index values is larger than the target number, determining that the first mode corresponding to the target cluster is a disaster recovery mode representing remote full data backup; the second condition is that the second condition is greater than a second preset threshold value, and the second preset threshold value is greater than the first preset threshold value;
in this implementation, the second preset threshold may be set to a higher value related to the read-write status of the target user data. It can be understood that, after the evaluation index values regarding the read-write state in each target sub-period are calculated in step B2, if the evaluation index values more than the preset ratio are greater than the second preset threshold value in the plurality of index values, the read-write frequency of the target user data in the first history period is higher, or the read-write data volume is larger, the importance degree of the target user data in the target cluster is considered to be high, and at this time, the first mode corresponding to the target cluster is determined to be the disaster recovery mode representing the remote full data backup, so as to ensure the availability of the target user data.
B5, otherwise, determining that the first mode corresponding to the target cluster is: and characterizing a disaster recovery mode of remote incremental backup.
That is, if the read-write frequency of the target user data or the read-write data volume in the first historical time period is in a moderate state, the first mode is determined to be the disaster recovery mode representing the remote incremental backup. It can be understood that, in general, the access frequency of the incremental data in the future is relatively high, and therefore, the read-write frequency or the read-write data volume of the target user data in the first historical period of the target cluster is relatively moderate, and the incremental backup can be performed only in the resource pool in the different place, that is, the availability can be guaranteed.
Therefore, through the implementation mode, the first mode matched with the target cluster in the first historical time period can be quickly and effectively determined through the read-write state information of the target user data in the first historical time period.
In order to better illustrate the contents of the embodiments of the present invention, the contents of the embodiments of the present invention are described below with reference to fig. 2 to 8.
FIG. 2 illustrates an overall architecture diagram of a data storage system including a distributed storage data availability intelligence assurance system. As shown in fig. 2, the client at the application side is mounted with the distributed object storage system through the gateway of the distributed storage cluster, so as to ensure that the distributed storage cluster can normally provide services for the user. In the storage layer, relevant basic parameters and relevant resource configurations are set, and to ensure that a secret key of the distributed storage cluster master node is sent to the slave nodes, the slave distributed storage cluster master node can normally access each slave node to achieve data synchronization.
The data storage system comprises a distributed storage cluster layer, an intelligent storage data availability guarantee system and a remote backup storage resource pool, wherein a cluster 1-a cluster n in the distributed storage cluster layer are storage layers for storing user data. The distributed intelligent security system for the availability of the storage data is a system for realizing the cluster availability mode control method in the embodiment and comprises a read-write state information acquisition module and an intelligent data availability analysis control system. Specifically, the distributed storage data availability intelligent security system is an electronic device for implementing the cluster availability mode control method according to the embodiment of the present invention.
The remote backup storage resource pool is a storage space which is far enough outside an area where the distributed storage cluster deployment resource pool is located in advance and is divided from another resource pool with stable and reliable network conditions, namely temporary storage 1-n in the distributed storage cluster is prepared for temporarily backing up user data with higher security level. The backup distributed storage cluster is automatically newly built in a storage space divided by a remote resource pool under the control of a message instruction transmitted by the inter-pool communication module by the backup storage application control module and is used for storing corresponding backup data, so that when an original cluster faces a major regional disaster, a support service application can be started immediately, and no perception is provided at a user side.
The backup storage application control module is used for receiving a message instruction transmitted by an inter-pool communication module of the data availability intelligent analysis control system, executing commands and operations in a backup distributed storage cluster divided by a remote backup storage resource pool, returning operation results to the inter-pool communication module, and simultaneously feeding back the change information of the availability mode of the cluster to a user or operation and maintenance personnel in a preset mode, for example, in a mode of sending the change information to a client. And the remote backup storage resource pool is a backup storage resource which is applied by the cloud service operator and the target resource pool operator in communication and coordination.
The read-write state information acquisition module can be deployed on a distributed storage cluster layer and is used for acquiring the read-write state information of user data in each cluster, and transmitting the acquired read-write state information to the data availability intelligent analysis control system through the communication interface for analysis by the recent data read-write state judgment module and the historical long-period data read-write habit analysis module. In addition, in the distributed storage cluster layer, a cluster availability configuration module may be further deployed in each cluster master node server, and is used to record the current availability mode of the distributed storage cluster where the cluster is located, and record the mode conversion condition of the cluster.
The data availability intelligent analysis control system is independently deployed on one server, is separated from a distributed storage cluster layer on a physical layer, does not occupy machine resources of the distributed storage cluster, and the server deployed in actual operation can select the machine resources according to actual conditions. The physical machine and the virtual machine can be used for deploying the data availability intelligent analysis control system, and the performance of the physical machine is better when the physical machine is deployed. The method has the function of combining historical long-period read-write state data and recent read-write state data of user data of each cluster in a distributed storage cluster layer to carry out modeling and prediction of the historical long-period read-write state of the user and judgment of the recent read-write state of the user, so that a decision on an optimal data availability mode of user data storage of each storage cluster in the distributed storage cluster layer is made. And the availability security decision execution module converts the decision information of the availability storage security situation judgment and decision module into a specific and executable availability security decision execution scheme. And finally, generating a specific execution instruction through the storage decision control module, transmitting the specific execution instruction to the distributed storage cluster layer through the communication interface to execute corresponding operation, communicating with a remote standby storage resource pool through the inter-pool communication module, and transmitting request information such as application or release of a storage space.
In the distributed storage cluster layer, the initialization setting of the availability mode of each cluster is carried out through an initialization availability mode setting module. As shown in fig. 3, the initial availability mode may adopt a mode of system default configuration, such as: and characterizing a disaster recovery mode without remote backup, and setting a required availability mode by operation and maintenance personnel according to user requirements. The availability modes that can be selected and configured by the operation and maintenance personnel according to the user requirements may include availability mode1, availability mode2, and availability mode 3, which are respectively: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup.
And the setting weight of the operation and maintenance personnel is higher than the weight of the default configuration of the system. Operation and maintenance personnel comprehensively judge the availability grade requirement required by data storage according to the customer requirement and the actual generation and application scene of data, and the availability grade in the system is set into three grades: high level availability, medium level availability, and general availability. Wherein, the initial availability mode corresponding to the "advanced availability" is: an availability mode1, namely a disaster recovery mode for representing remote full data backup; the "intermediate availability" corresponds to an initial availability pattern of: an availability mode2, namely a disaster recovery mode for representing remote incremental backup; the "general availability" corresponds to an initial availability pattern of: availability mode 3, i.e. disaster recovery mode featuring no off-site backup. If the operation and maintenance personnel do not actively set the availability mode, the system takes the availability mode configured by the default of the system as the initial availability mode of the cluster. And then, intelligently deciding a proper availability mode by the distributed storage data availability intelligent guarantee system, and automatically converting the availability mode.
The recent data read-write state distinguishing module in fig. 2 is configured to store historical read-write state data, which includes recent read-write frequency and read-write data amount of the user data, from the user data of each cluster acquired by the read-write state information acquisition module. The time range is set to be almost three months by default, and can also be set according to actual business requirements. For example, fig. 4 shows a decision flow chart of a specific algorithm of the recent data read-write status decision module, as shown in fig. 4, including the following steps:
(1) acquiring the read-write frequency and the read-write data volume of the previous three months of user data storage, namely the read-write state information in the first historical time period in the previous text, and taking the average value of the read-write frequency and the data volume of each month in the three months, namely the judgment index value in the previous text, as input, namely X { (F1, D1), (F2, D2), (F3, D3) };
(2) determining whether two or more data bits in the input X are greater than the pattern threshold ML 2: (FL2, DL2), that is, the second preset threshold value in the foregoing, wherein for each data bit, the read-write frequency may be set to be greater than FL2, and the read-write data amount is greater than DL2, one of the two conditions is satisfied, that is, the data bit is considered to be greater than ML2, or when both conditions are satisfied, the data bit is considered to be greater than ML2, which may be specifically set according to an actual application scenario. If yes, the output Mode1 is M1, and the availability Mode1, i.e., the first Mode in the above, is availability Mode 1; if not, executing the step (3);
(3) determining whether two or more data bits in the input X are less than the pattern threshold ML 1: (FL1, DL1), i.e. the first preset threshold value, similarly, for each data bit, the read-write frequency may be set to be smaller than FL1, and the read-write data amount may be set to be smaller than DL2, and one of the two conditions is satisfied, i.e. the data bit is considered to be smaller than ML2, or both conditions are satisfied simultaneously, and the data bit is considered to be smaller than ML 2. If yes, the output Mode1 is M3, that is, the availability Mode1 is availability Mode 3;
(4) otherwise, two or more data bits in the input X are not less than the Mode threshold ML1 and not greater than the Mode threshold ML2, and the output Mode1 is equal to M2, i.e., the availability Mode1 is availability Mode 2.
Fig. 5 shows a schematic flow chart of prediction performed by the historical long-period data read-write habit analysis module. As shown in fig. 5, the read-write state information acquisition module acquires the previous year read-write frequency of user data in a cluster, that is, the read-write state information in the second history period in the foregoing, as input data; inputting the input data into a historical long-period data reading and writing habit analysis module to obtain a reading and writing frequency prediction result one month after the current time (F4, D4); then, based on the prediction result, the prediction result Mode2 of the next-month availability pattern, i.e., the second pattern above, is output.
FIG. 6 illustrates a flow diagram for a storage availability security posture determination and decision module to make a decision. As shown in fig. 6, the core algorithm logic of the storage availability security posture judgment and decision module is as follows:
inputting: the output value Mode1 of the recent state judgment module and the output value Mode2 of the long-term trend prediction module;
the core algorithm formula is as follows: rmode=Mode1☉Mode2;
And (3) outputting: moder.
Performing an exclusive nor operation on the Mode1 and the Mode2, if the exclusive nor value is 0, indicating that the latest three-month read-write state reflected by the output value Mode1 of the recent state judgment module is obviously different from the read-write rule according to the user data in the past year, in this case, continuously keeping the original Mode, and judging until the next judgment period, wherein the output ModeR is 0;
if R ismodeThe value of the short-term data read-write state judgment module is 1, which indicates that the short-term data read-write state judgment module is consistent with the availability mode judgment result of the historical long-period data read-write habit analysis module for one month in the future. Reflecting that the read-write state of the user data storage basically accords with the read-write rule in the last year in the last three months, and the availability mode of the next month at the momentThe judgment is more reliable. The output at this time is: the Mode r 1 is Mode2, and has a value of one of three availability modes M1, M2, and M3.
Fig. 7 illustrates a flow diagram of the execution of the availability security decision execution module. As shown in fig. 7, the internally executed algorithm logic of the availability security decision execution module is:
when the output of the storage availability security situation judgment and decision module is received: when the ModeR is equal to 0, the data storage availability mode of the current user is kept unchanged;
when the output of the storage availability security situation judgment and decision module is received: when the ModeR is equal to a certain value of M1, M2, M3, the availability pattern of the current cluster is compared. Let the availability mode of the current cluster be M0.
When M0 is ModeR, the availability mode is not changed;
when M0 ≠ Moder, the original availability mode is changed to the availability mode corresponding to the current value of Moder.
The functional schematic diagram of the inter-pool communication module is shown in fig. 8:
and the inter-pool communication module is used for performing inter-system communication and data transmission with the target disaster recovery resource pool through the gateway interface module. And when receiving a user data storage availability mode change instruction of the availability safety decision execution module, the inter-pool communication module sends a message instruction to the target disaster backup resource pool to confirm the use condition of the standby storage resource space, and if the opposite side meets the basic condition for realizing the availability mode at the corresponding level, the inter-pool communication module calls a pre-prepared automatic distributed storage cluster deployment instruction set to build a distributed storage cluster with the same configuration in the target disaster backup resource pool.
Because the availability mode in the distributed storage data availability intelligent guarantee system is divided into three levels: general availability, medium level availability, high level availability. The disaster recovery from the strange place can be carried out in two availability modes of medium-level availability and high-level availability.
After a distributed storage cluster with the same configuration is built in a target disaster backup resource pool in a different place, a communication module between pools sends a prefabricated test instruction to verify whether the performance of the newly built cluster meets the requirement of a corresponding disaster backup level. And controlling the local cluster to copy and transmit user data to a backup distributed storage cluster built by the target disaster backup resource pool according to a rule corresponding to the availability mode level. Specifically, the standby storage application control module receives a message instruction transmitted by an inter-pool communication module of the data availability intelligent analysis control system, and executes a command and an operation in a standby distributed storage cluster divided by a standby storage resource pool at a different place.
In order to more clearly understand the content of the embodiment of the present invention, the following describes specific implementation steps of the cluster availability mode control method according to a specific example.
Step 1: ensuring that the conventional distributed storage cluster can normally operate, completing the mounting of a storage path between a client and a distributed storage system at a user side, and setting relevant basic parameters and relevant resource allocation in a storage layer of the cluster; and ensuring that the secret key of the distributed storage cluster main node is sent to the slave nodes, and the slave distributed storage cluster main node can normally access each slave node. Meanwhile, a certain storage space is divided in a resource pool in different places for temporary backup of user data with higher security level.
Step 2: initializing and setting the availability mode of the user data through an initialization storage availability mode setting module; the default availability mode is general availability, i.e. availability mode 3 is adopted in the cluster availability configuration module. The required initial availability mode can be set by operation and maintenance personnel according to the requirements of users. Here, the set weight of the operation and maintenance personnel is higher than the weight of the default configuration.
The guarantee mode for the availability of the distributed storage system in the distributed storage data availability intelligent guarantee system is divided into three levels: general availability, medium level availability, high level availability.
If the operation and maintenance personnel do not actively set, the system takes the availability mode 3 corresponding to the general availability as the initial data availability mode of the cluster according to the default availability mode. And then, intelligently deciding a proper availability mode of each cluster by using a distributed data storage availability intelligent guarantee system, and automatically carrying out mode conversion according to an intelligent analysis decision scheme.
And step 3: and the storage read-write state information acquisition module acquires the read-write state information of the user data in the cluster, and transmits the read-write state information to the data availability intelligent analysis control system through the communication interface, so that the recent data read-write state judgment module and the historical long-period data read-write habit analysis module can perform judgment, mathematical modeling and prediction.
And 4, step 4: the data availability intelligent analysis control system performs modeling and prediction on historical long-period read-write states of users and analyzes the recent read-write states of the user data by combining historical long-period read-write state data and recent read-write state data of the user data of each cluster in a distributed storage cluster layer, so that a decision on an availability mode of each cluster is made.
And 5: and converting the decision information of the storage security situation judgment and decision module into a specific and executable scheme through the security decision execution module.
Step 6: and generating a specific execution instruction according to the scheme through the storage decision control module, and transmitting the specific execution instruction to the corresponding cluster through the communication interface so as to execute corresponding operation.
And 7: and the safety decision execution module is communicated with a remote backup storage resource pool through the drive inter-pool communication module to transmit request information such as application or release of the storage space.
And 8: the backup storage application control module receives a message instruction transmitted by an inter-pool communication module of the data availability intelligent analysis control system; and backing up or releasing user data in the spare distributed storage clusters divided in the remote spare storage resource pool, and returning an operation result to the inter-pool communication module.
In this embodiment, a first mode is obtained according to read-write state information of target user data in a target cluster in a first historical time period, and a second mode is obtained by performing trend prediction according to the read-write state information in a second historical time period; and determining a target mode by judging whether the first mode and the second mode are the same, wherein the types of the availability modes comprise a disaster recovery mode for representing remote full data backup, a disaster recovery mode for representing remote incremental backup and a disaster recovery mode for representing no remote backup. Therefore, the scheme can flexibly switch the availability mode of the target cluster, thereby saving storage resources; and judging whether a first mode obtained by reading and writing state information in a first historical time period is the same as a second mode obtained by performing trend prediction according to the state information in a second historical time period, and then determining the target mode to ensure the credibility of the target mode. Therefore, the scheme can ensure the availability of the user data and improve the utilization rate of the remote storage resources.
Corresponding to the embodiment of the method, the embodiment of the invention also provides a cluster availability mode control device, which is applied to electronic equipment, wherein the electronic equipment is communicated with the main control node in the storage cluster; as shown in fig. 9, the apparatus includes:
a first determining module 910, configured to determine target user data in a target cluster; wherein the target cluster is a cluster to be controlled with respect to an availability mode, and the types of the availability mode include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
an identifying module 920, configured to identify a first mode corresponding to the target cluster by using read-write state information of the target user data in a first history period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
a predicting module 930, configured to predict, by using the read-write state information of the target user data in a second history period, a second mode corresponding to the target cluster; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
a determining module 940, configured to determine whether the first mode and the second mode are the same, so as to obtain a determination result;
a second determining module 950, configured to determine, based on the determination result, an availability mode to be adapted for the target cluster as a target mode;
the switching module 960, if the target mode is different from the current availability mode of the target cluster, switches the availability mode of the target cluster based on the target mode.
Optionally, the conversion module is specifically configured to:
if the target mode is different from the current availability mode of the target cluster, sending a mode conversion instruction carrying mode information of the target mode to the main control node, so that the main control node converts the availability mode of the target cluster into the target mode after receiving the mode conversion instruction.
Optionally, the second determining module includes:
a first determining sub-module, configured to determine, if the determination result indicates that the first mode and the second mode are different, a current availability mode of the target cluster as an availability mode to be adapted to the target cluster, so as to obtain a target mode;
and the second judgment submodule is used for determining the second mode as the availability mode to be adapted to the target cluster to obtain the target mode if the judgment result shows that the first mode and the second mode are the same.
Optionally, the identification module includes:
the determining submodule is used for determining the read-write state information of the target user data in each target sub-period; wherein, each target sub-period is each sub-period included in the first history period;
the calculation sub-module is used for calculating a judgment index value related to the read-write state in each target sub-period based on the read-write state information of the target user data in the target sub-period aiming at each target sub-period;
the first setting submodule is used for determining that a first mode corresponding to the target cluster is a disaster recovery mode which represents that remote backup is not carried out if the proportion of the number of judgment index values meeting a first condition in the determined judgment index values is greater than a preset proportion; wherein the first condition is less than a first preset threshold
The second setting submodule is used for determining that the first mode corresponding to the target cluster is a disaster recovery mode representing remote full data backup if the proportion of the number of judgment index values meeting the second condition in the determined judgment index values is greater than the preset proportion; the second condition is that the second condition is greater than a second preset threshold value, and the second preset threshold value is greater than the first preset threshold value;
a third setting submodule, configured to determine, in addition to the above, that the first mode corresponding to the target cluster is: and characterizing a disaster recovery mode of remote incremental backup.
Optionally, the read-write status information includes: information representing the frequency of occurrence of reading and writing, and/or the amount of data read and written;
the calculation submodule is specifically configured to:
and aiming at each target sub-period, calculating the average value of the read-write frequency and/or the average value of the read-write data quantity in the target sub-period based on the read-write state information of the target user data in the target sub-period.
Optionally, the prediction module includes:
the analysis submodule is used for inputting the read-write state information of the target user data in a second historical period into a pre-trained mode prediction model to obtain a mode prediction result which is used as a second mode corresponding to the target cluster;
the mode prediction model is a model obtained by training based on sample read-write state information and corresponding label values, and the sample read-write state information is as follows: and reading and writing state information of the sample user data in a sample time interval, wherein the label value is the predicted availability mode in the specified time interval after the sample time interval.
Optionally, the initial availability mode of the target cluster is a mode set according to a default disaster recovery rule of the system;
the default disaster recovery rule of the system comprises the following steps:
if the availability grade of the target cluster is a first grade, the initial availability mode is a disaster recovery mode representing remote full data backup;
if the availability grade of the target cluster is a second grade, the initial availability mode is a disaster recovery mode representing remote incremental backup;
if the availability grade of the target cluster is a third grade, the initial availability is a disaster recovery mode which represents that remote backup is not carried out;
wherein the first level is higher than the second level, which is higher than a third level.
The embodiment of the present invention further provides an electronic device, as shown in fig. 10, which includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 complete mutual communication through the communication bus 1004,
a memory 1003 for storing a computer program;
the processor 1001 is configured to implement the steps of the cluster availability mode control method when executing the program stored in the memory 1003.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Alternatively, the memory may be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In a further embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored, which, when being executed by a processor, implements the steps of the above-mentioned cluster availability mode control method.
In a further embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of the cluster availability pattern control method in the above-described embodiment.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (16)

1. A cluster availability mode control method is applied to an electronic device, and comprises the following steps:
determining target user data in a target cluster; wherein the target cluster is a cluster to be controlled with respect to an availability mode; the categories of the availability patterns include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
identifying a first mode corresponding to the target cluster by using the read-write state information of the target user data in a first historical period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
predicting a second mode corresponding to the target cluster by using the read-write state information of the target user data in a second historical period; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
judging whether the first mode and the second mode are the same or not to obtain a judgment result;
determining an availability mode to be adapted of the target cluster as a target mode based on the judgment result;
and if the target mode is different from the current availability mode of the target cluster, converting the availability mode of the target cluster based on the target mode.
2. The method of claim 1, wherein the electronic device is in communication with a master control node in the target cluster;
the converting the availability mode of the target user data based on the target mode comprises:
and sending a mode conversion instruction which is specific to the target cluster and carries the mode information of the target mode to the main control node, so that the main control node converts the availability mode of the target cluster into the target mode after receiving the mode conversion instruction.
3. The method according to claim 1, wherein the determining, based on the determination result, an availability mode to be adapted by the target cluster as a target mode comprises:
if the judgment result shows that the first mode and the second mode are different, determining the current availability mode of the target cluster as the availability mode to be adapted of the target cluster to obtain the target mode;
and if the judgment result shows that the first mode and the second mode are the same, determining the second mode as the availability mode to be adapted of the target cluster to obtain the target mode.
4. The method according to any one of claims 1 to 3, wherein the identifying a first pattern corresponding to the target cluster by using the read-write status information of the target user data in a first history period comprises:
determining the read-write state information of the target user data in each target sub-period; wherein, each target sub-period is each sub-period included in the first history period;
for each target sub-period, calculating a judgment index value related to the read-write state in the target sub-period based on the read-write state information of the target user data in the target sub-period;
if the ratio of the number of the judgment index values meeting the first condition in the determined judgment index values is greater than a preset ratio, determining that the first mode corresponding to the target cluster is a disaster recovery mode which represents that remote backup is not performed; wherein the first condition is less than a first preset threshold;
if the ratio of the number of judgment index values meeting the second condition in the determined judgment index values is larger than a preset ratio, determining that the first mode corresponding to the target cluster is a disaster recovery mode representing remote full data backup; the second condition is that the second condition is greater than a second preset threshold value, and the second preset threshold value is greater than the first preset threshold value;
otherwise, determining that the first mode corresponding to the target cluster is: and characterizing a disaster recovery mode of remote incremental backup.
5. The method of claim 4, wherein the read/write status information comprises: information representing the frequency of occurrence of reading and writing, and/or the amount of data read and written;
for each target sub-period, calculating a judgment index value related to the read-write state in the target sub-period based on the read-write state information of the target user data in the target sub-period, including:
and aiming at each target sub-period, calculating the average value of the read-write frequency and/or the average value of the read-write data quantity in the target sub-period based on the read-write state information of the target user data in the target sub-period.
6. The method according to any one of claims 1 to 3, wherein the predicting the second pattern corresponding to the target cluster by using the read-write status information of the target user data in the second history period comprises:
inputting the read-write state information of the target user data in a second historical period into a pre-trained mode prediction model to obtain a mode prediction result, wherein the mode prediction result is used as a second mode corresponding to the target cluster;
the mode prediction model is a model obtained by training based on sample read-write state information and corresponding label values, and the sample read-write state information is as follows: and reading and writing state information of the sample user data in a sample time interval, wherein the label value is the predicted availability mode in the specified time interval after the sample time interval.
7. The method according to any one of claims 1 to 3, wherein the initial availability mode of the target cluster is a mode set according to a default disaster recovery rule of the system;
the default disaster recovery rule of the system comprises the following steps:
if the availability grade of the target cluster is a first grade, the initial availability mode is a disaster recovery mode representing remote full data backup;
if the availability grade of the target cluster is a second grade, the initial availability mode is a disaster recovery mode representing remote incremental backup;
if the availability grade of the target cluster is a third grade, the initial availability mode is a disaster recovery mode which represents that remote backup is not carried out;
wherein the first level is higher than the second level, which is higher than a third level.
8. A cluster availability mode control device applied to an electronic device, the device comprising:
the first determining module is used for determining target user data in a target cluster; wherein the target cluster is a cluster to be controlled with respect to an availability mode, and the types of the availability mode include: the method comprises the steps of representing a disaster recovery mode of remote full data backup, representing a disaster recovery mode of remote incremental backup and representing a disaster recovery mode without remote backup;
the identification module is used for identifying a first mode corresponding to the target cluster by utilizing the read-write state information of the target user data in a first historical period; wherein the first pattern is an availability pattern adapted to the target cluster within the first history period;
the prediction module is used for predicting a second mode corresponding to the target cluster by utilizing the read-write state information of the target user data in a second historical period; wherein the second mode is an availability mode that is adapted to the target cluster within a specified period of time after a current time;
the judging module is used for judging whether the first mode and the second mode are the same or not to obtain a judging result;
a second determining module, configured to determine, based on the determination result, an availability mode to be adapted for the target cluster as a target mode;
and the conversion module is used for converting the availability mode of the target cluster based on the target mode if the target mode is different from the current availability mode of the target cluster.
9. The apparatus of claim 8, wherein the conversion module is specifically configured to:
if the target mode is different from the current availability mode of the target cluster, sending a mode conversion instruction carrying mode information of the target mode to the main control node, so that the main control node converts the availability mode of the target cluster into the target mode after receiving the mode conversion instruction.
10. The apparatus of claim 8, wherein the second determining module comprises:
a first determining sub-module, configured to determine, if the determination result indicates that the first mode and the second mode are different, a current availability mode of the target cluster as an availability mode to be adapted to the target cluster, so as to obtain a target mode;
and the second judgment submodule is used for determining the second mode as the availability mode to be adapted to the target cluster to obtain the target mode if the judgment result shows that the first mode and the second mode are the same.
11. The apparatus according to any one of claims 8-10, wherein the identification module comprises:
the determining submodule is used for determining the read-write state information of the target user data in each target sub-period; wherein, each target sub-period is each sub-period included in the first history period;
the calculation sub-module is used for calculating a judgment index value related to the read-write state in each target sub-period based on the read-write state information of the target user data in the target sub-period aiming at each target sub-period;
the first setting submodule is used for determining that a first mode corresponding to the target cluster is a disaster recovery mode which represents that remote backup is not carried out if the proportion of the number of judgment index values meeting a first condition in the determined judgment index values is greater than a preset proportion; wherein the first condition is less than a first preset threshold;
the second setting submodule is used for determining that the first mode corresponding to the target cluster is a disaster recovery mode representing remote full data backup if the proportion of the number of judgment index values meeting the second condition in the determined judgment index values is greater than the preset proportion; the second condition is that the second condition is greater than a second preset threshold value, and the second preset threshold value is greater than the first preset threshold value;
a third setting submodule, configured to determine, in addition to the above, that the first mode corresponding to the target cluster is: and characterizing a disaster recovery mode of remote incremental backup.
12. The apparatus of claim 11, wherein the read/write status information comprises: information representing the frequency of occurrence of reading and writing, and/or the amount of data read and written;
the calculation submodule is specifically configured to:
and aiming at each target sub-period, calculating the average value of the read-write frequency and/or the average value of the read-write data quantity in the target sub-period based on the read-write state information of the target user data in the target sub-period.
13. The apparatus according to any one of claims 8-10, wherein the prediction module comprises:
the analysis submodule is used for inputting the read-write state information of the target user data in a second historical period into a pre-trained mode prediction model to obtain a mode prediction result which is used as a second mode corresponding to the target cluster;
the mode prediction model is a model obtained by training based on sample read-write state information and corresponding label values, and the sample read-write state information is as follows: and reading and writing state information of the sample user data in a sample time interval, wherein the label value is the predicted availability mode in the specified time interval after the sample time interval.
14. The apparatus according to any one of claims 8-10, wherein the initial availability mode of the target cluster is a mode set according to a default disaster recovery rule of the system;
the default disaster recovery rule of the system comprises the following steps:
if the availability grade of the target cluster is a first grade, the initial availability mode is a disaster recovery mode representing remote full data backup;
if the availability grade of the target cluster is a second grade, the initial availability mode is a disaster recovery mode representing remote incremental backup;
if the availability grade of the target cluster is a third grade, the initial availability mode is a disaster recovery mode which represents that remote backup is not carried out;
wherein the first level is higher than the second level, which is higher than a third level.
15. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.
16. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
CN202111613066.4A 2021-12-27 2021-12-27 Cluster availability mode control method, device, equipment and storage medium Pending CN114237986A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111613066.4A CN114237986A (en) 2021-12-27 2021-12-27 Cluster availability mode control method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111613066.4A CN114237986A (en) 2021-12-27 2021-12-27 Cluster availability mode control method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114237986A true CN114237986A (en) 2022-03-25

Family

ID=80763505

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111613066.4A Pending CN114237986A (en) 2021-12-27 2021-12-27 Cluster availability mode control method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114237986A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115906170A (en) * 2022-12-02 2023-04-04 杨磊 Safety protection method and AI system applied to storage cluster

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115906170A (en) * 2022-12-02 2023-04-04 杨磊 Safety protection method and AI system applied to storage cluster
CN115906170B (en) * 2022-12-02 2023-12-15 北京金安道大数据科技有限公司 Security protection method and AI system applied to storage cluster

Similar Documents

Publication Publication Date Title
US9912752B1 (en) Retention-based data management in a network-based data store
US8244671B2 (en) Replica placement and repair strategies in multinode storage systems
US20190158422A1 (en) Analyzing resource placement fragmentation for capacity planning
US20100274762A1 (en) Dynamic placement of replica data
US11579933B2 (en) Method for establishing system resource prediction and resource management model through multi-layer correlations
CN106708653B (en) Mixed tax big data security protection method based on erasure code and multiple copies
US10754720B2 (en) Health check diagnostics of resources by instantiating workloads in disaggregated data centers
US20100241616A1 (en) Perpetual archival of data
US10243819B1 (en) Template generation based on analysis
US11782770B2 (en) Resource allocation based on a contextual scenario
US10148521B2 (en) Capacity planning in a backbone network
CN112199193A (en) Resource scheduling method and device, electronic equipment and storage medium
CN115033340A (en) Host selection method and related device
CN114237986A (en) Cluster availability mode control method, device, equipment and storage medium
CN114598665A (en) Resource scheduling method and device, computer readable storage medium and electronic equipment
CN112363815B (en) Redis cluster processing method and device, electronic equipment and computer readable storage medium
US10594620B1 (en) Bit vector analysis for resource placement in a distributed system
CN110347546A (en) Monitor task dynamic adjusting method, device, medium and electronic equipment
CN114546425A (en) Model deployment method and device, electronic equipment and storage medium
CN114217825A (en) Method, apparatus, machine-readable storage medium and processor for decision engine
CN114443293A (en) Deployment system and method for big data platform
CN114398006A (en) Distributed storage mode control method, device, equipment and storage medium
CN111327663A (en) Bastion machine distribution method and equipment
CN116668271B (en) Container migration method, device, equipment and medium of container cluster management system
CN113282245B (en) Method for auditing supply number and host platform

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination