CN114265563B - Object storage method and device based on cloud computing and storage medium - Google Patents
Object storage method and device based on cloud computing and storage medium Download PDFInfo
- Publication number
- CN114265563B CN114265563B CN202111667982.6A CN202111667982A CN114265563B CN 114265563 B CN114265563 B CN 114265563B CN 202111667982 A CN202111667982 A CN 202111667982A CN 114265563 B CN114265563 B CN 114265563B
- Authority
- CN
- China
- Prior art keywords
- storage
- storage module
- target
- standby
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
Embodiments of the present application relate to the technical field of data storage, and some embodiments provide an object storage method, an object storage device, and a storage medium based on cloud computing. The method comprises the following steps: acquiring at least one object and requirement information from the received object storage request, wherein the requirement information comprises the number of nodes and the number of persistent volumes; acquiring target nodes matched with the number of the nodes and acquiring target persistent volumes matched with the number of the persistent volumes; mounting a storage module corresponding to the target persistent volume to a target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type; and storing the object to the storage module of the storage category and the storage module of the standby category simultaneously. According to the method, when a certain storage module is damaged or the access path of a certain target persistent volume is in a problem, the standby storage object or the normal target persistent volume can be obtained to realize storage access, and therefore high availability of the object storage service is guaranteed.
Description
Technical Field
The embodiment of the application relates to the technical field of data storage, in particular to an object storage method and device based on cloud computing and a storage medium.
Background
In the Cloud Native (Cloud Native) environment, it has become standard to deploy clusters using techniques like kubernets and manage different applications based on the deployed clusters. Various applications deployed on a cluster typically need to perform a variety of tasks, such as training tasks, prediction tasks, or classification tasks, etc., which typically require access to the storage service of the cluster to perform storage of data or objects (objects).
At present, object storage can be usually realized based on a kubernets cluster, a computing unit (Pod) corresponding to a storage request can be started through the kubernets cluster when the storage request is received, a Persistent Volume (PV) is mounted in a Pod (Docker) of the Pod, and the Pod can be directly stored through a file Interface of a Portable Operating System Interface (POSIX) to realize object storage.
Disclosure of Invention
However, for the reason of the access path, the prior art can only access the object to the storage service of the cluster through the file interface of POSIX by mounting the object to the persistent volume in the Pod of the Pod to realize the storage of the object, but the mounted persistent volume is a direct access path, and if the access path is problematic, the availability of the object storage service is reduced.
Therefore, in the prior art, the low availability of the object storage service is a very annoying problem.
To this end, there is a strong need for an improved cloud computing-based object storage method, apparatus and storage medium to ensure high availability of the object storage service.
In this context, embodiments of the present application are intended to provide a cloud computing-based object storage method, apparatus, and storage medium.
In a first aspect of the present application, there is provided a cloud computing-based object storage method, including:
acquiring at least one object and requirement information from a received object storage request, wherein the requirement information comprises the number of nodes and the number of persistent volumes;
acquiring target nodes matched with the number of the nodes, and acquiring target persistent volumes matched with the number of the persistent volumes;
mounting a storage module corresponding to the target persistent volume to the target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type;
and simultaneously storing the object to the storage module of the storage category and the storage module of the standby category.
In a second aspect of the present application, there is provided a cloud computing-based object storage apparatus comprising:
an obtaining unit, configured to obtain at least one object and requirement information from a received object storage request, where the requirement information includes the number of nodes and the number of persistent volumes;
the obtaining unit is further configured to obtain target nodes that match the number of the nodes, and obtain target persistent volumes that match the number of the persistent volumes;
the mounting unit is used for mounting the storage module corresponding to the target persistent volume to the target node through the target persistent volume; the storage module comprises a storage module and a standby module, wherein the storage module is of a storage type and a standby type;
and the storage unit is used for simultaneously storing the object to the storage module of the storage type and the storage module of the standby type.
In a third aspect of the present application, a computer-readable storage medium is provided, comprising instructions which, when run on a computer, cause the computer to perform the method according to the first aspect.
In a fourth aspect of the present application, there is provided a computing device comprising a processor and a memory:
the memory is used for storing program codes;
the processor is configured to perform the method of the first aspect according to instructions in the program code.
According to the object storage method, device and storage medium based on cloud computing in the embodiments of the present application, a required number of target nodes can be obtained according to demand information in a received object storage request, and a plurality of storage modules are mounted to the target nodes through different target persistent volumes, where a storage module has a storage module for storing a storage category of an original object and a storage module for storing a backup category of a copied object, so that the received object can store the object to different storage modules through different target persistent volumes.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present application will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present application are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:
fig. 1 is a schematic application scenario diagram of an object storage system based on cloud computing according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an object storage method based on cloud computing according to an embodiment of the present application;
fig. 3 is a schematic flowchart illustrating an implementation of object storage by a tenant according to an embodiment of the present application;
FIG. 4 is a diagram illustrating a data block and a spare block according to an embodiment of the present application;
fig. 5 is a schematic flowchart of an object storage method based on cloud computing according to another embodiment of the present application;
FIG. 6 is a block diagram of a cloud computing-based object storage apparatus according to an embodiment of the present disclosure;
FIG. 7 is a block diagram of a cloud computing-based object storage apparatus according to another embodiment of the present application;
fig. 8 is a schematic structural diagram of an object storage apparatus based on cloud computing according to an embodiment of the present application;
FIG. 9 is a diagram schematically illustrating a structure of a computer-readable storage medium according to an embodiment of the present application;
fig. 10 schematically shows a structural diagram of a computing device according to an embodiment of the present application.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
The principles and spirit of the present application will be described with reference to a number of exemplary embodiments. It is understood that these examples are given solely to enable those skilled in the art to better understand and to practice the present application, and are not intended to limit the scope of the present application in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As will be appreciated by one skilled in the art, embodiments of the present application may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to the embodiment of the application, an object storage method and device based on cloud computing and a storage medium are provided.
In this context, it is to be understood that the terms referred to:
cloud computing (cloud computing) refers to a delivery and use mode of an IT infrastructure, and refers to obtaining required resources in an on-demand and easily-extensible manner through a network; the generalized cloud computing refers to a delivery and use mode of a service, and refers to obtaining a required service in an on-demand and easily-extensible manner through a network. Such services may be IT and software, internet related, or other services. Cloud computing is a product of development and fusion of traditional computers and network technologies, such as grid computing (gridding), distributed computing (DistributedComputing), parallel computing (ParallelComputing), utility computing (UtilityComputing), network storage (network storage technologies), virtualization (Virtualization), load balancing (LoadBalance), and the like.
With the development of diversification of internet, real-time data stream and connecting equipment and the promotion of demands of search service, social network, mobile commerce, open collaboration and the like, cloud computing is rapidly developed. Different from the prior parallel distributed computing, the generation of cloud computing can promote the revolutionary change of the whole internet mode and the enterprise management mode in concept.
Machine Learning (ML) is a multi-domain cross subject, and relates to multi-domain subjects such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The method specially studies how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and the like.
Kubernets is a portable, extensible, open source platform for managing containerized workloads and services that facilitates declarative configuration and automation. Kubernets possess a large and rapidly growing ecosystem. Services, support and tools of kubernets are widely available.
Moreover, any number of elements in the drawings are by way of example and not by way of limitation, and any nomenclature is used solely for differentiation and not by way of limitation.
In the embodiment of the application, at least one object and requirement information are obtained from a received object storage request, wherein the requirement information comprises the number of nodes and the number of persistent volumes; acquiring target nodes matched with the number of the nodes and acquiring target persistent volumes matched with the number of the persistent volumes; mounting a storage module corresponding to the target persistent volume to a target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type; and storing the object to the storage module of the storage category and the storage module of the standby category at the same time.
By way of example, with reference to fig. 1 in conjunction, there is shown a schematic diagram of a cloud computing-based object storage system provided by an embodiment of the present application. The cloud computing-based object storage system may include a machine learning platform 10, a cloud computing-based object storage 20, and a storage module 30.
The machine learning platform 10 is configured to send an object storage request including, but not limited to, at least one object and requirement information to the cloud computing based object storage 20, where the number of the machine learning platforms 10 may be one or more.
The cloud computing-based object storage device 20 is configured to obtain at least one object and requirement information from the received object storage request, where the requirement information includes the number of nodes and the number of persistent volumes; acquiring target nodes matched with the number of the nodes and acquiring target persistent volumes matched with the number of the persistent volumes; mounting the storage module 30 corresponding to the target persistent volume to the target node through the target persistent volume; the storage module 30 is classified into a storage class and a standby class; and storing the object to the storage modules 30 of the storage class and the storage modules 30 of the standby class at the same time, wherein the number of the target nodes is the same as the number of the nodes required in the demand information, and the number of the storage modules 30 in one target node can be one or more.
Optionally, communication between the machine learning platform 10 and the cloud computing-based object storage 20 is via a network.
The technical solution of the present application will be described in detail with reference to several embodiments.
Exemplary method
Fig. 2 is a schematic flowchart of an embodiment of a cloud computing-based object storage method, where the embodiment of the present application is applied to the cloud computing-based object storage system shown in fig. 1, where the cloud computing-based object storage system includes a machine learning platform, a cloud computing-based object storage device, and a storage module, and the embodiment of the present application only takes the cloud computing-based object storage device as an example to describe a processing procedure of object storage. Rather, embodiments of the present application may be applied to any scenario where applicable. Referring to fig. 2, the method includes the following steps:
step S201, at least one object and requirement information are acquired from the received object storage request.
In this embodiment of the present application, the requirement information includes the number of nodes and the number of persistent volumes.
Specifically, the sent object storage request may be sent by a training task, a classification task, a prediction task, and other tasks of the model in the machine learning scene, or may be sent by various applications (applications), which is not limited in this embodiment of the present Application. The object storage request may be used to apply for storage of an object to a cloud-computing-based object storage apparatus, where the cloud-computing-based object storage apparatus may be kuberents, distributed storage services, distributed storage systems, ceph, longhorn, or the like, and the embodiments of the present application are not limited thereto, the object storage request may include at least one object and requirement information, the number of the objects may be one or more, the requirement information may be used to indicate a storage space that should be allocated by the cloud-computing-based object storage apparatus, that is, the requirement information includes the number of nodes and the number of persistent volumes, a node may be a computing unit of the cloud-computing-based object storage apparatus, and may obtain one or more computing units according to the number of nodes in the requirement information, and a persistent volume may mount a storage module to the node, so that the object may directly store the object to the storage module through the storage module mounted on the node, and the number of persistent volumes in the requirement information may be the number of storage modules that need to be mounted.
Step S202, obtaining target nodes matched with the number of the nodes, and obtaining target persistent volumes matched with the number of the persistent volumes.
In the embodiment of the present application, the number of target persistent volumes corresponding to each target node is the same, for example, the number of target persistent volumes corresponding to each target node is two.
Step S203, mount the storage module corresponding to the target persistent volume to the target node through the target persistent volume.
In the embodiment of the application, the storage module is classified into a storage class and a standby class. A target persistent volume may correspond to a storage module, and the storage module may be a memory, a disk, or the like, which is not limited in this embodiment. A target persistent volume may correspond to a target node, and thus, a target persistent volume may mount a corresponding storage module to the target node corresponding to the target persistent volume,
and the storage module of the storage category can store the original object, and the storage module of the standby category can store the copied object or store the standby object after the original object is coded.
Step S204, storing the object to the storage module of the storage type and the storage module of the standby type at the same time.
In the embodiment of the present application, one storage module may be further divided into a plurality of data blocks, the data block in the storage module of the storage category may be used to store an original object, and the data block in the storage module of the backup category may be used to store a backup object or a copy object, so that the data block in the storage module of the backup category may also be used as a backup block.
As an optional implementation manner, the manner in which the object is simultaneously stored in the storage module of the storage category and the storage module of the standby category in step S204 may specifically include the following steps:
storing the object to a storage module of the storage category;
coding the object according to preset protection parameters to obtain a standby object;
and storing the standby object to the storage module of the standby category.
By implementing the implementation mode, the object can be stored in the storage module of the storage type, the object and the preset protection parameter are coded to obtain the standby object, and the standby object is stored in the storage module of the standby type, so that when the storage module fails, the lost object in the failed storage module can be recovered according to the objects stored in other normal storage modules and/or the standby object, and the reliability of object storage is improved.
In this embodiment of the application, the preset protection parameter may be an Erasure Code (EC), and the original object and the preset protection parameter are encoded to obtain an encoded standby object. The erasure codes can be determined according to the number of target nodes or the total number of the storage modules, so that the maximum erasure code set is obtained, and the storage objects are guaranteed to be written into one erasure code set.
Optionally, after storing the standby object to the storage module of the standby category, the method may further include the following steps:
performing health monitoring on the target node to obtain a monitoring result;
and when the monitoring result indicates that alarm information exists in the target node, maintaining the target node according to the alarm information.
By implementing the implementation mode, the health monitoring can be carried out on each target node, and when the problem in the target node is monitored, the problem in the target node can be timely maintained according to the alarm information, so that the problem of the target node is timely solved.
In the embodiment of the application, each target node can be allocated with a health monitoring daemon, and the health monitoring daemon can maintain a replicac allocation mode of the cloud computing-based object storage device; the erasure code can be maintained and calculated, and the correct storage of the object is ensured; the remaining space of the storage module and the call instruction of the storage module can be monitored, and the health state of the storage module can be known in time. When a problem exists in the target node, alarm information can be generated according to the monitored problem, and maintenance is performed according to the problem indicated in the alarm information.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating an implementation of object storage by a Tenant (Tenant) according to an embodiment of the present disclosure; the object storage device based on cloud computing can be provided with an independent Tenant for processing a request sent by a machine learning model, one Tenant can have a preset object storage interface for receiving the object storage request sent by the machine learning model, object1 and object2 in fig. 3 represent two different objects needing to be stored, tenant is used for realizing storage of object1 and object2, node1, node2, node3 and Node4 respectively represent 4 acquired target nodes, tenant demamon 1, tenant demamon 2, tenant demamon 3 and Tenant demamon 4 represent daemon processes for health monitoring of each target Node, a, b, c, d, e, f, 30g and h represent 8 storage modules which are mounted into the target nodes through target persistent volumes, and each storage module comprises n data blocks, for example, storage module a comprises a1, a2, 8230an, storage modules are the same as storage modules in storage modules, and storage modules in the storage modules are no longer repeated. In fig. 3, object1 can be stored in data block a1, data block c1, data block e1 and data block g1 through Tenant demamon 2, while spare objects can be stored in data block b1, data block d1, data block f1 and data block h1, object2 can be stored in data block a2, data block c2, data block e2 and data block g2 through Tenant demamon 3, while spare objects can be stored in data block b2, data block d2, data block f2 and data block h 2.
In an embodiment of the present application, the number of the storage modules in the storage category is greater than or equal to the number of the storage modules in the standby category, and the maintaining manner of the target node according to the alarm information may include the following steps:
when the alarm information indicates that the storage module is damaged, acquiring the damage number of the damaged storage module;
and if the damaged number is less than or equal to the number of the storage modules of the standby category, repairing the damaged storage modules according to the undamaged storage modules to obtain all repaired storage modules.
By implementing the implementation mode, whether the object stored in the damaged storage module can be recovered can be determined according to the size relationship between the number of the damaged storage modules and the number of the storage modules in the standby category, so that the object can be timely recovered when the damaged object is absent, and the safety of the object is ensured.
In the embodiment of the application, only when the number of the damaged storage modules is less than or equal to the number of the storage modules in the standby category, the damaged storage modules can be recovered, the remaining original objects and the remaining standby objects in the normal storage modules can be acquired, and the damaged objects are recovered according to the acquired remaining original objects and the remaining standby objects in an erasure code decoding manner to obtain all the recovered storage modules.
Referring to fig. 4, fig. 4 is a schematic diagram of a data block and a spare block according to an embodiment of the present application; from fig. 3, it can be seen that the original object1 is stored in the data block a1, the data block c1, the data block e1 and the data block g1, and therefore, the data block a1, the data block c1, the data block e1 and the data block g1 can be regarded as the original data block, and the spare object is stored in the data block b1, the data block d1, the data block f1 and the data block h1, and therefore, the data block b1, the data block d1, the data block f1 and the data block h1 can be regarded as the spare block.
In another embodiment of the present application, in order to improve the efficiency of object storage, the received object storage request may be classified in advance, and as shown in fig. 5, before the step S201, the following steps S501 to S504 may be further included:
step S501, receiving the object storage request through a preset object storage interface.
In this embodiment of the Application, one preset object Storage Interface corresponds to one machine learning model, an object Storage device based on cloud computing may be provided with a single tenant to process a request sent by one machine learning model, one tenant may have a preset object Storage Interface to receive an object Storage request sent by the machine learning model, and the object Storage Interface may be an Application Programming Interface (API) of an Application program of a Simple Storage Service (S3), and the like. The machine learning model can be a model for realizing training tasks, classification tasks, prediction tasks and the like in a machine learning scene.
Step S502, if the object storage request is a primary request, executing step S201 to step S204;
step S503, if the object storage request is a re-request, acquiring an acquired target node corresponding to the preset object storage interface.
In the embodiment of the application, the acquired target node is a node acquired during the initial request; when an object storage request is received, the cloud computing-based object storage device can detect whether a target node corresponding to a machine learning model which sends the object storage request is acquired, and if the target node is not detected, the object storage request can be considered as a primary request; if a target node is detected, the object storage request may be considered a re-request.
Step S504, storing the object in the object storage request to the storage module of the acquired target node.
In the embodiment of the application, the object in the object storage request can be stored in the storage module of the storage category, and the object and the standby object obtained by encoding the preset protection parameter can be stored in the storage module of the standby category.
By implementing the steps S501 to S504, a preset object storage interface may correspond to a machine learning model, and the machine learning model only sends requests such as object storage to the corresponding object storage interface each time, so that objects to be stored may be directly stored in a previously allocated storage module when object storage requests are sent for subsequent times, and a target node and a mount storage module do not need to be reallocated to the object storage requests again, thereby avoiding waste of computing resources.
In another embodiment of the present application, the step S501 of receiving the object storage request through a preset object storage interface may further include the following steps:
performing security verification on the received access request information through a preset security key to obtain a verification result; wherein the access request information is sent through the machine learning model;
and if the verification result indicates safety, establishing communication connection between the object storage interface and the machine learning model.
By implementing the implementation mode, the security verification can be performed on the received access request information through the security key, so that the request received by the object storage interface is sent by the machine learning model corresponding to the request, and the security of the object operation in the storage module is ensured.
In the embodiment of the application, the object is prevented from sending errors in the request sending process, so that the object in the object storage request can be encrypted and checked, when the machine learning model needs to establish communication connection with the object storage interface, the machine learning model sends access request information to the object storage interface, the access request information received through the object storage interface is combined into a character string, the character string is encrypted and verified through a preset safety key, a token value is obtained, and the token value is sent to the machine learning model; and then, when a request sent by the machine learning model is received, decrypting the request through the token value in the request to compare whether the access request information is consistent with the access request information sent by the machine learning model for the first time, and if so, considering that the verification is passed.
Optionally, the object storage method based on cloud computing may further include the following steps:
acquiring an object identifier from an object reading request received by the preset object storage interface;
reading a target object matched with the object identifier from the acquired target node according to the obtained object identifier;
and sending the target object to a machine learning model corresponding to the preset object storage interface.
When the implementation of the implementation manner is carried out, the target object matched with the object identifier can be read from the acquired target node according to the object identifier in the object reading request when the received object reading request is received, and the target object is sent to the machine learning model corresponding to the object storage interface, so that the high availability of object reading is realized.
In the embodiment of the application, the object identifier may be an identifier of an object that needs to be read from the storage module, and the corresponding target object may be accurately found from the storage module through the object identifier, read, and send to the machine learning model corresponding to the preset object storage interface.
Referring to fig. 6 and 7 together, fig. 6 is a schematic diagram of a framework of a cloud-based object storage device according to an embodiment of the present application; the cloud computing-based object storage apparatus in fig. 6 is constructed based on a kuberents cluster, and first, an object storage service Operator (controller) is developed on the kuberenets cluster, where the Operator is used to accept an object storage request for creating different object storage services Tenant, and after receiving the object storage request, the Operator applies for the kuberents cluster and the storage services according to the number of nodes (nodes) required in the object storage request and the number of persistent volumes required in one Node. One Tenant corresponds to a set of high-availability object storage service and corresponds to a plurality of nodes and persistent volumes under the nodes, and the Tenant simultaneously provides a set of uniform S3 API access points so that an Application (Application) sends a request to the Tenant through the S3 API access points.
FIG. 7 is a block diagram of a cloud computing-based object storage apparatus according to another embodiment of the present application; the cloud computing-based object storage device in fig. 7 is constructed based on a distributed storage service, and the distributed storage service provides two storage modes, one is a data storage service of a conventional PV, for example, a training task may be stored through a POSIX file interface implemented by the conventional PV, and the other is a high-availability object storage mode provided by the present application, for example, a prediction task may store an object in the prediction task through the high-availability object storage method of the present application.
According to the method and the device, when a certain storage module is damaged or an access path of a certain target persistent volume is in a problem, the standby storage object or the normal target persistent volume can be obtained to realize storage access, and therefore high availability of the object storage service is guaranteed. In addition, the reliability of object storage can be improved. In addition, the problem of the target node can be solved in time. In addition, the security of the object can be guaranteed. In addition, the method and the device can avoid the waste of computing resources. In addition, the method and the device can also ensure the safety of object operation in the storage module. In addition, the method and the device can achieve high availability of object reading.
Exemplary devices
Having described the method of the exemplary embodiment of the present application, next, a cloud computing-based object storage apparatus of the exemplary embodiment of the present application will be described with reference to fig. 8, the apparatus including:
an obtaining unit 801, configured to obtain at least one object and requirement information from the received object storage request, where the requirement information includes the number of nodes and the number of persistent volumes.
The obtaining unit 801 is further configured to obtain a target node matching the number of nodes, and obtain a target persistent volume matching the number of persistent volumes.
A mounting unit 802, configured to mount, to the target node, the storage module corresponding to the target persistent volume through the target persistent volume acquired by the acquiring unit 801; the storage module is classified into a storage class and a standby class.
A storage unit 803, configured to store the object acquired by the acquisition unit 801 in the storage module of the storage category and the storage module of the standby category mounted by the mounting unit 802 at the same time.
As an optional implementation manner, the way for the storage unit 803 to store the object to the storage module of the storage category and the storage module of the standby category at the same time is specifically:
storing the object to a storage module of the storage category;
coding the object according to preset protection parameters to obtain a standby object;
and storing the standby object to the storage module of the standby category.
By implementing the implementation mode, the object can be stored in the storage module of the storage type, the object and the preset protection parameter are coded to obtain the standby object, and the standby object is stored in the storage module of the standby type, so that when the storage module fails, the lost object in the failed storage module can be recovered according to the objects stored in other normal storage modules and/or the standby object, and the reliability of object storage is improved.
As an optional implementation, the storage unit 803 is further configured to:
after the standby object is stored in the standby type storage module, health monitoring is carried out on the target node to obtain a monitoring result;
and when the monitoring result indicates that alarm information exists in the target node, maintaining the target node according to the alarm information.
By implementing the implementation mode, the health monitoring can be carried out on each target node, and when the problem in the target node is monitored, the problem in the target node can be timely maintained according to the alarm information, so that the problem of the target node is timely solved.
As an optional implementation manner, the number of the storage modules in the storage category is greater than or equal to the number of the storage modules in the standby category, and a specific manner of the storage unit 803 maintaining the target node according to the alarm information is as follows:
when the alarm information indicates that the storage module is damaged, acquiring the damage number of the damaged storage module;
and if the damaged number is less than or equal to the number of the storage modules of the standby category, repairing the damaged storage modules according to the undamaged storage modules to obtain all repaired storage modules.
By implementing the implementation mode, whether the object stored in the damaged storage module can be recovered or not can be determined according to the size relationship between the number of the damaged storage modules and the number of the storage modules in the standby category, so that the object can be recovered in time when the damaged object is lost, and the safety of the object is ensured.
As an optional implementation, the obtaining unit 801 is further configured to:
before acquiring at least one object and demand information from a received object storage request, receiving the object storage request through a preset object storage interface; wherein one preset object storage interface corresponds to one machine learning model;
if the object storage request is a primary request, executing the step of acquiring at least one object and demand information from the received object storage request;
if the object storage request is a secondary request, acquiring an acquired target node corresponding to the preset object storage interface, wherein the acquired target node is the node acquired in the primary request;
and storing the object in the object storage request into the storage module of the acquired target node.
By implementing the implementation mode, one preset object storage interface corresponds to one machine learning model, and the machine learning model only sends the object storage and other requests to the corresponding object storage interface each time, so that the objects to be stored can be directly stored in the previously allocated storage module when the object storage requests are sent for a plurality of times in the following, and the target nodes and the mount storage modules do not need to be reallocated to the object storage requests, thereby avoiding the waste of computing resources.
As an optional implementation, the obtaining unit 801 is further configured to:
before receiving the object storage request through a preset object storage interface, performing security verification on the received access request information through a preset security key to obtain a verification result; wherein the access request information is sent through the machine learning model;
and if the verification result indicates safety, establishing communication connection between the object storage interface and the machine learning model.
By implementing the implementation mode, the security verification can be performed on the received access request information through the security key, so that the request received by the object storage interface is sent by the machine learning model corresponding to the request, and the security of the object operation in the storage module is ensured.
As an optional implementation, the obtaining unit 801 is further configured to:
acquiring an object identifier from an object reading request received by the preset object storage interface;
reading a target object matched with the object identifier from the acquired target node according to the obtained object identifier;
and sending the target object to a machine learning model corresponding to the preset object storage interface.
When the implementation of the implementation manner is carried out, the target object matched with the object identifier can be read from the acquired target node according to the object identifier in the object reading request when the received object reading request is received, and the target object is sent to the machine learning model corresponding to the object storage interface, so that the high availability of object reading is realized.
Exemplary Medium
Having introduced the apparatus of the exemplary embodiment of the present application, the computer-readable storage medium of the exemplary embodiment of the present application is described with reference to fig. 9, please refer to fig. 9, which illustrates a computer-readable storage medium, which is an optical disc 900, on which a computer program (i.e., a program product) is stored, where the computer program, when executed by a processor, implements the steps described in the above method embodiments, for example, obtains at least one object and requirement information from a received object storage request, where the requirement information includes the number of nodes and the number of persistent volumes; acquiring target nodes matched with the number of the nodes and acquiring target persistent volumes matched with the number of the persistent volumes; mounting a storage module corresponding to the target persistent volume to a target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type; storing the object to a storage module of a storage type and a storage module of a standby type at the same time; the specific implementation of each step is not repeated here.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, a phase change memory (PRAM), a Static Random Access Memory (SRAM), a Dynamic Random Access Memory (DRAM), other types of Random Access Memories (RAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a flash memory, or other optical and magnetic storage media, which are not described in detail herein.
Exemplary computing device
Having described the methods, media and apparatus of the exemplary embodiments of the present application, a computing device for cloud computing-based object storage of the exemplary embodiments of the present application will now be described with reference to fig. 10.
FIG. 10 illustrates a block diagram of an exemplary computing device 100, which computing device 100 may be a computer system or server, suitable for use in implementing embodiments of the present application. The computing device 100 shown in fig. 10 is only an example and should not impose any limitations on the functionality or scope of use of embodiments of the present application.
As shown in fig. 10, components of computing device 100 may include, but are not limited to: one or more processors or processing units 1001, a system memory 1002, and a bus 1003 that couples the various system components (including the system memory 1002 and the processing unit 1001).
The system memory 1002 can include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 10021 and/or cache memory 10022. The computing device 100 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, ROM10023 can be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 10, and typically referred to as a "hard drive"). Although not shown in FIG. 10, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 1003 by one or more data media interfaces. Included in system memory 1002 may be at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the application.
Programs/utilities 10025, having a set (at least one) of program modules 10024, can be stored, for example, in system memory 1002, and such program modules 10024 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment. Program modules 10024 generally perform the functions and/or methodologies of the embodiments described herein.
The processing unit 1001 executes various functional applications and data processing by running a program stored in the system memory 1002, for example, acquires at least one object and requirement information including the number of nodes and the number of persistent volumes from a received object storage request; acquiring target nodes matched with the number of the nodes and acquiring target persistent volumes matched with the number of the persistent volumes; mounting a storage module corresponding to the target persistent volume to a target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type; and storing the object to the storage module of the storage category and the storage module of the standby category at the same time. The specific implementation of each step is not repeated here. It should be noted that although several units/modules or sub-units/sub-modules of the cloud computing based object storage are mentioned in the above detailed description, such partitioning is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module, according to embodiments of the application. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
In the description of the present application, it is noted that the terms "first", "second", and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units into only one type of logical function may be implemented in other ways, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the spirit and scope of the exemplary embodiments of the present application, and are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
While the spirit and principles of the application have been described with reference to several particular embodiments, it is to be understood that the application is not limited to the particular embodiments disclosed, nor is the division of aspects, which is for convenience only as the features in such aspects may not be combined to benefit from the present disclosure. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A cloud computing-based object storage method, comprising:
acquiring at least one object and requirement information from a received object storage request, wherein the requirement information comprises the number of nodes and the number of persistent volumes;
acquiring target nodes matched with the number of the nodes, and acquiring target persistent volumes matched with the number of the persistent volumes;
mounting a storage module corresponding to the target persistent volume to the target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type;
and storing the object to the storage module of the storage category and the storage module of the standby category simultaneously.
2. The cloud-computing-based object storage method of claim 1, the storing the object to the storage class of storage modules and the standby class of storage modules simultaneously, comprising:
storing the object to a storage module of the storage category;
coding the object according to preset protection parameters to obtain a standby object;
and storing the standby object to the storage module of the standby category.
3. The cloud-computing-based object storage method of claim 2, after storing the backup object to the backup class of storage modules, the method further comprising:
performing health monitoring on the target node to obtain a monitoring result;
and when the monitoring result indicates that alarm information exists in the target node, maintaining the target node according to the alarm information.
4. The cloud-computing-based object storage method of claim 3, the number of storage modules of the storage category being greater than or equal to the number of storage modules of the backup category, the maintaining the target node according to the alert information comprising:
when the alarm information indicates that the storage module is damaged, acquiring the damage number of the damaged storage module;
and if the damaged number is less than or equal to the number of the storage modules of the standby category, repairing the damaged storage modules according to the undamaged storage modules to obtain all repaired storage modules.
5. The cloud-computing-based object storage method according to any one of claims 1 to 4, before the obtaining of the at least one object and the requirement information from the received object storage request, the method further comprising:
receiving the object storage request through a preset object storage interface; wherein one preset object storage interface corresponds to one machine learning model;
if the object storage request is a primary request, executing the steps from the beginning of acquiring at least one object and demand information from the received object storage request to the end of storing the object to the storage module of the storage category and the storage module of the standby category at the same time;
if the object storage request is a secondary request, acquiring an acquired target node corresponding to the preset object storage interface, wherein the acquired target node is the node acquired in the primary request;
and storing the object in the object storage request into the storage module of the acquired target node.
6. The cloud-computing-based object storage method according to claim 5, before receiving the object storage request through a preset object storage interface, the method further comprising:
performing security verification on the received access request information through a preset security key to obtain a verification result; wherein the access request information is sent through the machine learning model;
and if the verification result indicates safety, establishing communication connection between the object storage interface and the machine learning model.
7. The cloud-computing-based object storage method of claim 5, the method further comprising:
acquiring an object identifier from an object reading request received by the preset object storage interface;
reading a target object matched with the object identifier from the acquired target node according to the obtained object identifier;
and sending the target object to a machine learning model corresponding to the preset object storage interface.
8. A cloud computing-based object storage apparatus, comprising:
an obtaining unit, configured to obtain at least one object and requirement information from a received object storage request, where the requirement information includes the number of nodes and the number of persistent volumes;
the obtaining unit is further configured to obtain target nodes that match the number of the nodes, and obtain target persistent volumes that match the number of the persistent volumes;
the mounting unit is used for mounting the storage module corresponding to the target persistent volume to the target node through the target persistent volume; the storage module comprises a storage module, a storage module and a standby module, wherein the storage module is of a storage type and a standby type;
and the storage unit is used for simultaneously storing the object to the storage module of the storage type and the storage module of the standby type.
9. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1-7.
10. A computing device, the computing device comprising a processor and a memory:
the memory is used for storing program codes;
the processor is configured to perform the method of any one of claims 1 to 7 according to instructions in the program code.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111667982.6A CN114265563B (en) | 2021-12-31 | 2021-12-31 | Object storage method and device based on cloud computing and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111667982.6A CN114265563B (en) | 2021-12-31 | 2021-12-31 | Object storage method and device based on cloud computing and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114265563A CN114265563A (en) | 2022-04-01 |
| CN114265563B true CN114265563B (en) | 2023-03-28 |
Family
ID=80832062
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111667982.6A Active CN114265563B (en) | 2021-12-31 | 2021-12-31 | Object storage method and device based on cloud computing and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114265563B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704200A (en) * | 2021-11-01 | 2021-11-26 | 北京国科环宇科技股份有限公司 | Data storage method, device, equipment and storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10360099B2 (en) * | 2017-04-28 | 2019-07-23 | Netapp Inc. | Object format resilient to remote object store errors |
| US10642783B2 (en) * | 2018-01-12 | 2020-05-05 | Vmware, Inc. | System and method of using in-memory replicated object to support file services wherein file server converts request to block I/O command of file handle, replicating said block I/O command across plural distributed storage module and performing said block I/O command by local storage module |
| US20210109822A1 (en) * | 2019-10-11 | 2021-04-15 | EMC IP Holding Company LLC | Systems and methods for backup and restore of container-based persistent volumes |
| US11467775B2 (en) * | 2019-10-15 | 2022-10-11 | Hewlett Packard Enterprise Development Lp | Virtual persistent volumes for containerized applications |
| CN111506391B (en) * | 2020-03-31 | 2022-08-30 | 新华三大数据技术有限公司 | Container deployment method and device |
| US11687267B2 (en) * | 2020-04-14 | 2023-06-27 | Hewlett Packard Enterprise Development Lp | Containerized application manifests and virtual persistent volumes |
| CN111966305B (en) * | 2020-10-22 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Persistent volume allocation method and device, computer equipment and storage medium |
| CN113672175A (en) * | 2021-08-09 | 2021-11-19 | 浙江大华技术股份有限公司 | Distributed object storage method, device and equipment and computer storage medium |
-
2021
- 2021-12-31 CN CN202111667982.6A patent/CN114265563B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704200A (en) * | 2021-11-01 | 2021-11-26 | 北京国科环宇科技股份有限公司 | Data storage method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114265563A (en) | 2022-04-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11657146B2 (en) | Compressibility metric-based detection of a ransomware threat to a storage system | |
| US11720692B2 (en) | Hardware token based management of recovery datasets for a storage system | |
| US11615185B2 (en) | Multi-layer security threat detection for a storage system | |
| US11687418B2 (en) | Automatic generation of recovery plans specific to individual storage elements | |
| US11500788B2 (en) | Logical address based authorization of operations with respect to a storage system | |
| US20210216666A1 (en) | Inter-I/O Relationship Based Detection of a Security Threat to a Storage System | |
| US20220164109A1 (en) | Data replication in a storage system | |
| US11704202B2 (en) | Recovering from system faults for replicated datasets | |
| US11893023B2 (en) | Deterministic searching using compressed indexes | |
| EP4058919A1 (en) | Creating a replica of a storage system | |
| US11886308B2 (en) | Dual class of service for unified file and object messaging | |
| US10929053B2 (en) | Safe destructive actions on drives | |
| CN111104201A (en) | System migration method and device, electronic equipment and storage medium | |
| US20210383010A1 (en) | Measurement Interval Anomaly Detection-based Generation of Snapshots | |
| US20210232323A1 (en) | Managing voltage threshold shifts | |
| US11645162B2 (en) | Recovery point determination for data restoration in a storage system | |
| US20220244858A1 (en) | Storage Element Attribute-based Determination of a Data Protection Policy for Use Within a Storage System | |
| US20210382995A1 (en) | Host Anomaly-Based Generation of Snapshots | |
| US11675898B2 (en) | Recovery dataset management for security threat monitoring | |
| CN114265563B (en) | Object storage method and device based on cloud computing and storage medium | |
| US20210109682A1 (en) | Efficient data compression by grouping similar data within a data segment | |
| US11657155B2 (en) | Snapshot delta metric based determination of a possible ransomware attack against data maintained by a storage system | |
| US20210382992A1 (en) | Remote Analysis of Potentially Corrupt Data Written to a Storage System | |
| US10740088B2 (en) | Countermeasure verification assistance system and method | |
| US11941116B2 (en) | Ransomware-based data protection parameter modification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |