CN111857592A - Data storage method and device based on object storage system and electronic equipment - Google Patents
Data storage method and device based on object storage system and electronic equipment Download PDFInfo
- Publication number
- CN111857592A CN111857592A CN202010705650.1A CN202010705650A CN111857592A CN 111857592 A CN111857592 A CN 111857592A CN 202010705650 A CN202010705650 A CN 202010705650A CN 111857592 A CN111857592 A CN 111857592A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- pool
- active
- policy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000000694 effects Effects 0.000 claims description 41
- 238000012163 sequencing technique Methods 0.000 claims description 13
- 238000002955 isolation Methods 0.000 claims description 6
- 238000003672 processing method Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 230000002085 persistent effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004378 air conditioning Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Abstract
The invention discloses a data storage method and device based on an object storage system and electronic equipment. Wherein, the method comprises the following steps: acquiring the number of hard disks on an object storage node; creating storage pools with different data security types based on the number of hard disks; creating an object storage strategy based on the storage pool, wherein the storage strategy is used for indicating the read-write rule of the object data; the storing of the object data is performed based on the object storage policy. The invention solves the technical problem of data read-write performance reduction caused by capacity expansion of the data pool in the related technology.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a data storage method and device based on an object storage system and electronic equipment.
Background
In the related technology, a storage pool of an object storage system can be divided into an index pool and an independent persistent data pool, object data with different sizes are written in through a client, wherein a metadata part in the object data can be stored in the index pool, a data part can be written in the independent persistent data pool through processes of caching, merging and the like, when the client reads an object, the client accesses the metadata first, directly reads the data in the corresponding data pool according to a data storage position fed back by the metadata, and then returns the data to a user.
When the independent persistent data pool is to be fully written, in order to avoid that data cannot be continuously written, service continuity can be ensured by adding a new hard disk or a new storage node in the existing persistent data pool. However, the current method of adding a hard disk or a storage node has a plurality of disadvantages:
the first drawback is that the addition of a hard disk or a storage node leads to an increase in the utilization rate of the hard disk due to a new round of automatic rebalancing of the written data, occupies more cluster resources to process data migration, increases IO delay, affects the performance of front-end service IO, and causes a decrease in read-write performance;
the second drawback is that new and old data are stored in the same storage pool, so that cold and hot data cannot be distinguished more effectively, and the data read-write performance is also reduced.
The third drawback is that random allocation is adopted in the object data storage process, so that when data are written into a plurality of data pools, some data pools have more written data, some data pools have less written data, and the difference of data storage capacity is large, thereby causing the problem of unbalanced space allocation.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a data storage method and device based on an object storage system and electronic equipment, and aims to at least solve the technical problem of data read-write performance reduction caused by capacity expansion of a data pool in the related art.
According to an aspect of the embodiments of the present invention, there is provided a data storage method based on an object storage system, where the object storage system includes a plurality of object storage nodes, the data storage method includes: acquiring the number of hard disks on an object storage node; creating storage pools with different data security types based on the number of the hard disks; creating an object storage policy based on the storage pool, wherein the storage policy is used for indicating read-write rules of object data; performing storage of object data based on the object storage policy.
Optionally, the data security type includes: multiple copy types and erasure code types.
Optionally, when the object storage policy is created, an object index pool and a cache pool are determined, where the object index pool is used for storing object metadata, and the cache pool is used for caching object data.
Optionally, the object storage policy includes: the data processing method comprises an active data pool and an inactive data pool, wherein the active data pool is used for indicating a storage pool with a reading and writing frequency higher than a preset frequency threshold, and the inactive data pool is used for indicating a storage pool with a reading and writing frequency lower than an access frequency equal to or lower than the preset frequency threshold.
Optionally, the step of performing storage of object data based on the object storage policy includes: and creating a bucket based on the object storage policy, wherein the bucket is used for writing object data to realize the storage of the object data.
Optionally, after the object data is written into the active data pool in the bucket, the object index information is updated, wherein the object index information is used for recording the writing position of the object data.
Optionally, after performing storage of the object data based on the object storage policy, the data storage method further includes: setting a plurality of attribute parameters of each activity data pool in the activity pool group, wherein the attribute parameters at least comprise: writing weight and writing times; calculating an object write coefficient for each active data pool based on the plurality of attribute parameters; sequencing all the object writing coefficients to obtain a sequencing result; and based on the sorting result, taking the active data pool with the minimum object writing coefficient as the current writing data pool.
Optionally, after setting a plurality of attribute parameters of the activity data pool, the data storage method further includes: and if the object storage strategy is updated or the object storage gateway is restarted, resetting the attribute parameters.
Optionally, after performing storage of the object data based on the object storage policy, the data storage method further includes: judging the available capacity of each activity data pool; if the available capacity is lower than a preset capacity threshold value, setting the active data pool as an inactive data pool, and sending a capacity alarm signal; after the capacity alarm signal is sent out, a new storage pool is created to obtain a new storage pool; adding the new storage pool into an active pool group where an active data pool sending out a capacity alarm signal is located to serve as a new active data pool; after determining a new active data pool, resetting the attribute parameters of each of the active data pools based on the currently available space of each of the active data pools in the active pool group.
Optionally, after performing storage of the object data based on the object storage policy, the data storage method further includes: detecting the available space of each activity data pool in the activity pool group according to a preset time interval period; updating the available capacity ratio of each of the active data pools based on the available space of each of the active data pools; updating a write weight value for each of the active data pools based on the available capacity ratio.
Optionally, after performing storage of the object data based on the object storage policy, the data storage method further includes: receiving fault information, wherein the fault information is used for indicating that a hard disk in an active data pool has a fault; determining a fault activity data pool and a fault hard disk based on the fault information; and replacing the fault hard disk in the fault active data pool based on the resource mutual isolation rule between the active data pools.
According to another aspect of the embodiments of the present invention, there is also provided a data storage apparatus based on an object storage system, where the object storage system includes a plurality of object storage nodes, the data storage apparatus includes: the acquisition unit is used for acquiring the number of hard disks on the object storage node; a first creating unit, configured to create storage pools with different data security types based on the number of hard disks; a second creating unit, configured to create an object storage policy based on the storage pool, where the storage policy is used to indicate a read-write rule of object data; and the execution unit is used for executing the storage of the object data based on the object storage strategy.
Optionally, the data security type includes: multiple copy types and erasure code types.
Optionally, when the object storage policy is created, an object index pool and a cache pool are determined, where the object index pool is used for storing object metadata, and the cache pool is used for caching object data.
Optionally, the object storage policy includes: the data processing method comprises an active data pool and an inactive data pool, wherein the active data pool is used for indicating a storage pool with a reading and writing frequency higher than a preset frequency threshold, and the inactive data pool is used for indicating a storage pool with a reading and writing frequency lower than an access frequency equal to or lower than the preset frequency threshold.
Optionally, the execution unit includes: the storage system comprises a first creating module and a second creating module, wherein the first creating module is used for creating a bucket based on the object storage policy, and the bucket is used for writing object data to realize the storage of the object data.
Optionally, after the object data is written into the active data pool in the bucket, the object index information is updated, wherein the object index information is used for recording the writing position of the object data.
Optionally, the data storage device further comprises: a first setting module for setting a plurality of attribute parameters of each of the active data pools in the active pool group after performing storage of object data based on the object storage policy, wherein the attribute parameters include at least: writing weight and writing times; a first calculation module for calculating an object write coefficient for each active data pool based on the plurality of attribute parameters; the first sequencing module is used for sequencing all the object write-in coefficients to obtain a sequencing result; and the first determining module is used for taking the active data pool with the minimum object writing coefficient as the current writing data pool based on the sorting result.
Optionally, the data storage device further comprises: and the first updating module is used for resetting the attribute parameters if the object storage strategy is updated or the object storage gateway is restarted after the plurality of attribute parameters of the active data pool are set.
Optionally, the data storage device further comprises: a first judgment module for judging an available capacity of each active data pool after performing storage of object data based on the object storage policy; the second setting module is used for setting the active data pool as an inactive data pool and sending out a capacity alarm signal when the available capacity is lower than a preset capacity threshold; the first newly-built module is used for newly building a storage pool after sending the capacity alarm signal to obtain a new storage pool; the second determining module is used for adding the new storage pool into an active pool group where an active data pool sending out a capacity alarm signal is located to serve as a new active data pool; and the third setting module is used for resetting the attribute parameters of each activity data pool based on the current available space of each activity data pool in the activity pool group after determining a new activity data pool.
Optionally, the data storage device further comprises: the detection module is used for detecting the available space of each activity data pool in the activity pool group according to a preset time interval period after the object data storage is executed based on the object storage strategy; a second updating module for updating the available capacity ratio of each of the active data pools based on the available space of each of the active data pools; a third updating module for updating the write weight value of each of the active data pools based on the available capacity ratio.
Optionally, the data storage device further comprises: a receiving module, configured to receive failure information after performing storage of object data based on the object storage policy, where the failure information is used to indicate that a hard disk in an active data pool fails; the third determining module is used for determining a fault activity data pool and a fault hard disk based on the fault information; and the replacing module is used for replacing the fault hard disk in the fault active data pool based on the resource mutual isolation rule between the active data pools.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-described object storage system-based data storage methods via execution of the executable instructions.
In the embodiment of the invention, the number of hard disks on an object storage node is acquired; creating storage pools with different data security types based on the number of hard disks; creating an object storage strategy based on the storage pool, wherein the storage strategy is used for indicating the read-write rule of the object data; the storing of the object data is performed based on the object storage policy. In this embodiment, the storage pool may be directly expanded without adding a hard disk and a service node, so that additional data rebalancing is not required for the storage pool, system bandwidth resources are not occupied, front-end service performance is not affected, and data read-write performance during the expansion process of the data pool is improved, thereby solving the technical problem of data read-write performance reduction caused by the expansion of the data pool in the related art.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow chart of an alternative method for storing data based on an object storage system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an alternative object-based storage system data storage device, according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
To facilitate understanding of the invention by those skilled in the art, some terms or nouns referred to in the embodiments of the invention are explained below:
an object storage system: the Storage system is constructed based on Object Storage devices (Object Storage devices), wherein objects are basic units of data Storage, one Object is actually a combination of a data part and a metadata part of a file, and each Object Storage Device can be intelligent and automatically manages data distribution of the objects, and provides a flattened data structure and a concurrent data access mode.
An index pool: storing metadata information of an object, for example, when the size of the object is smaller than 1MB, storing a data portion and a metadata portion of the object, when the size of the object is larger than 1MB, only storing the metadata information of the object, and an index pool is generally created by using a medium for a hard disk of an SSD, which can improve the read-write performance of the index and the small object.
A data pool: data portions in the storage object, for example, data portions of objects larger than 1MB and objects larger than 4MB in size.
Storage strategy: user data storage rules are defined, including an index pool, a cache pool, and an active data pool and an inactive data pool.
An active data pool: the data pool with high access frequency can provide read, write and delete services of the object data, and all newly written object data are stored in the active pool.
The movable pool group: the system is composed of a plurality of activity pools, and the number of the activity pools in the group is not limited.
An inactive pool: and the data pool with low access frequency or no access is provided, and only the reading and deleting services of the object data are provided. The inactive pool may be converted to an active pool, and the active pool may also be converted to an inactive pool.
The embodiments of the invention described below may be applied in an object storage system/object storage cluster. The invention is illustrated below with reference to various examples.
Embodiment one data storage method based on object storage system
In accordance with an embodiment of the present invention, there is provided an object storage system based data storage method embodiment, it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
According to an aspect of the embodiments of the present invention, a data storage method based on an object storage system is provided, where the object storage system includes a plurality of object storage nodes.
Fig. 1 is a flowchart of an alternative data storage method based on an object storage system according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, acquiring the number of hard disks on an object storage node;
step S104, establishing storage pools with different data security types based on the number of hard disks;
step S106, based on the storage pool, an object storage strategy is created, wherein the storage strategy is used for indicating the read-write rule of the object data;
step S108, the storage of the object data is executed based on the object storage strategy.
Through the steps, the number of hard disks on the object storage node can be acquired; creating storage pools with different data security types based on the number of hard disks; creating an object storage strategy based on the storage pool, wherein the storage strategy is used for indicating the read-write rule of the object data; the storing of the object data is performed based on the object storage policy. In this embodiment, the storage pool may be directly expanded without adding a hard disk and a service node, so that additional data rebalancing is not required for the storage pool, system bandwidth resources are not occupied, front-end service performance is not affected, and data read-write performance during the expansion process of the data pool is improved, thereby solving the technical problem of data read-write performance reduction caused by the expansion of the data pool in the related art.
The present invention will be described below with reference to the above steps.
In the embodiment of the invention, firstly, the storage resource needs to be created according to the capacity expansion requirement.
Step S102, acquiring the number of hard disks on the object storage node.
The object storage nodes may be understood as object storage servers, each of which may comprise at least one hard disk.
And step S104, creating storage pools with different data security types based on the number of the hard disks.
On the object storage system, a storage pool with different data security types is created according to the number of hard disks on the object storage node, and optionally, the data security types include: multiple copy types and erasure code types. Here, the multiple copy type refers to a storage pool having more than one copy (M > ═ 2), and data is stored in a plurality of copies in the entire storage pool. If the number of hard disks in the storage pool is larger, the high aggregation performance of the storage pool is exerted, the number of times of reading and writing of each disk in unit time is reduced, and the probability of multi-disk faults occurring in the storage pool at the same time is relatively reduced. The multiple copies and the erasure codes can effectively ensure that data is lost due to disk failure, and the multiple hard disks can reduce the risk of failure of the disks due to frequent reading and writing. In the embodiment of the invention, the number of the data disks in a single storage pool is controlled within 500, and the method can be flexibly adjusted.
And step S106, creating an object storage strategy based on the storage pool, wherein the storage strategy is used for indicating the read-write rule of the object data.
Optionally, when creating the object storage policy, an object index pool and a cache pool are determined, where the object index pool is used to store object metadata, and the cache pool is used to cache object data.
After the storage pool is created, a new object storage strategy can be created, and an object index pool and a cache pool are specified, wherein the object index pool and the cache pool can be the same storage pool or different storage pools.
Alternatively, the object storage policy includes: the data processing system comprises an active data pool and an inactive data pool, wherein the active data pool is used for indicating a storage pool with a reading and writing frequency higher than a preset frequency threshold, and the inactive data pool is used for indicating a storage pool with a reading and writing frequency lower than an access frequency equal to or lower than the preset frequency threshold.
After the object index pool and the cache pool are determined, an active data pool and an inactive data pool are selected, and a storage strategy can comprise a plurality of active data pools and inactive data pools.
After the object storage strategy is successfully created, a new bucket is created according to the object storage strategy to execute the storage of the object data, and in the creation process, the object storage strategy can be loaded and the validity of the object storage strategy can be verified.
Step S108, the storage of the object data is executed based on the object storage strategy.
Optionally, the step of performing storage of the object data based on the object storage policy includes: and creating a bucket based on the object storage policy, wherein the bucket is used for writing the object data to realize the storage of the object data.
Alternatively, after the object data is written into the active data pool in the bucket, the object index information is updated, wherein the object index information is used for recording the writing position of the object data.
And the client writes the object into the bucket created based on the object storage policy through the object storage gateway.
In an embodiment of the present invention, after performing storage of object data based on the object storage policy, the data storage method further includes: setting a plurality of attribute parameters of each activity data pool in the activity pool group, wherein the attribute parameters at least comprise: writing weight and writing times; calculating an object write coefficient for each active data pool based on the plurality of attribute parameters; sequencing all the object writing coefficients to obtain a sequencing result; and based on the sorting result, taking the active data pool with the minimum object writing coefficient as the current writing data pool.
In the created object storage policy, setting at least two attribute parameters for the storage pool that has been set as the active data pool, including: the write weight and the number of write times, for example, set the total number of write weights to 10000, the number of write times is counted from 0, when the storage policy configures only one active pool, the write weight of the active pool is 10000, and all the object data are written into the same active data pool.
If multiple active data pools are configured in the object storage policy, the write weight assigned to each active data pool is proportionally assigned according to the actual available space capacity. And selecting the active data pool with the minimum r value as the currently written write data pool by calculating the coefficient r (r ═ number/weight) of each active storage pool. For example, when the object data comes once and the active data pool needs to be selected, the current active data pool includes: a: a, 1000, weight; b: b, 2000, weight; c: c, 4000 weight; d: d, 1500 weight; after the r values of the active data pools are calculated, the active data pool with the minimum r value can be selected as a write data pool, and the object index information is updated.
Optionally, after setting a plurality of attribute parameters of the active data pool, the data storage method further includes: and if the object storage strategy is updated or the object storage gateway is restarted, resetting the attribute parameters. That is, after the object storage policy is updated or the object storage gateway is restarted, the number value is cleared and the r value is recalculated.
And judging that each object data written into the storage bucket by the client is written into one of the active data pools according to the calculated r value, then updating the index information of the object, and recording the position of the active pool in which the object is written.
The active data pool can provide reading service for the object, the written object data is read through the client, the data part of the object is read according to the position of the object data recorded in the index pool in the active data pool, and the data part is returned to the client.
In an embodiment of the present invention, after performing storage of object data based on the object storage policy, the data storage method further includes: judging the available capacity of each activity data pool; if the available capacity is lower than the preset capacity threshold value, setting the active data pool as an inactive data pool, and sending a capacity alarm signal; after sending out the capacity alarm signal, newly building a storage pool to obtain a new storage pool; adding the new storage pool into an active pool group where an active data pool sending out a capacity alarm signal is located to serve as a new active data pool; after determining the new active data pool, the attribute parameters for each active data pool in the active pool group are reset based on the currently available space for each active data pool.
When an alarm occurs in the available capacity of one active data pool, the active data pool can be set as an inactive data pool, and a new weight of the inactive pool is assigned, at this time, the active data pool cannot accept the writing of new data, but can provide data reading and deleting services together with other active data pools.
For an active data pool with a certain capacity or some capacity being full, the operation of capacity expansion of the whole pool can be carried out. The newly created pool is added directly to the active pool and set as the new active pool, with the number starting from 0. At the moment, the writing weight value is recalculated according to the available space of all the active data pools in the current active pool group, and after new object data are written, the active data pools are judged to be written into according to the latest r value.
By expanding the capacity according to the level of the storage pool, the phenomenon that bandwidth resources are occupied by IO due to rebalancing of originally written data is avoided, the continuity of IO is guaranteed, meanwhile, the performance advantages of the storage pools are utilized to the maximum degree, the data distribution is more balanced, the utilization rate of the disks is more balanced, and the imperceptibility of upper-layer application is well guaranteed.
Optionally, after performing storage of the object data based on the object storage policy, the data storage method further includes: detecting the available space of each activity data pool in the activity pool group according to a preset time interval period; updating the available capacity ratio of each activity data pool based on the available space of each activity data pool; the write weight value for each active data pool is updated based on the available capacity ratio.
In the aspect of capacity updating of an active data pool, the number of objects stored in the active data pool is guaranteed to be evenly distributed as much as possible for a newly-built active pool group, but the space occupation situation of the active data pool is different due to different sizes of each object data, so that the available capacity ratio of the active data pool needs to be updated according to a preset time interval period (for example, one day is used as a time interval period), capacity balanced distribution is guaranteed as much as possible, if a new active data pool is not added to the active pool group, the available space of each existing active pool is updated by adopting an initial updating time interval (for example, 1 time per day is defaulted), and then the writing weight value is updated.
As an alternative embodiment of the present invention, after performing storage of object data based on an object storage policy, the data storage method further includes: receiving fault information, wherein the fault information is used for indicating that a hard disk in an active data pool has a fault; determining a fault activity data pool and a fault hard disk based on the fault information; and replacing the fault hard disk in the fault active data pool based on the resource mutual isolation rule between the active data pools.
In the aspect of fault processing of the active pool, a plurality of active data pools exist in one active pool, when one or more hard disks in the active data pools have faults, the storage pool state to which the hard disks belong can be abnormal, the fault hard disks can be directly replaced on the premise of not stopping front-end service and not influencing normal reading and writing of other active data pools, the internal part of the active pool with new hard disks can automatically rebalance data, the uniform distribution of each disk data in the current active data pool is ensured, the fault hard disks in other abnormal active data pools can also be directly replaced, resources among different active data pools are mutually isolated, and the recovery under abnormal conditions does not influence other active data pools which normally work.
Through the embodiment, the active data pool which can be written in according to the self-adaptive distribution of the weight of the storage pool can ensure the maximum utilization of the space of the storage pool, and the performance of a plurality of active data pools is aggregated, the efficiency of data writing is improved, when capacity expansion is carried out, the capacity expansion of the whole storage pool or the whole group of the active pool is realized, extra data rebalancing is not needed, the system bandwidth resource is not occupied, the front-end service performance is not influenced, the arc isolation of the resources of the active data pools is realized, and the normal work of other active pools is not influenced by the recovery under the abnormal condition.
Second embodiment data storage device based on object storage system
Fig. 2 is a schematic diagram of an alternative data storage apparatus based on an object storage system, where the object storage system includes a plurality of object storage nodes, and as shown in fig. 2, the data storage apparatus includes: an obtaining unit 21, a first creating unit 23, a second creating unit 25, an executing unit 27, wherein,
an obtaining unit 21, configured to obtain the number of hard disks on the object storage node;
a first creating unit 23, configured to create storage pools of different data security types based on the number of hard disks;
a second creating unit 25, configured to create an object storage policy based on the storage pool, where the storage policy is used to indicate a read-write rule of the object data;
an execution unit 27 for executing the storage of the object data based on the object storage policy.
The data storage apparatus based on the object storage system may acquire the number of hard disks on the object storage node through the acquisition unit 21, then create a storage pool of different data security types based on the number of hard disks through the first creation unit 23, create an object storage policy based on the storage pool through the second creation unit 25, wherein the storage policy is used for indicating a read-write rule of the object data, and perform storage of the object data based on the object storage policy through the execution unit 27. In this embodiment, the storage pool may be directly expanded without adding a hard disk and a service node, so that additional data rebalancing is not required for the storage pool, system bandwidth resources are not occupied, front-end service performance is not affected, and data read-write performance during the expansion process of the data pool is improved, thereby solving the technical problem of data read-write performance reduction caused by the expansion of the data pool in the related art.
Optionally, the data security types include: multiple copy types and erasure code types.
Optionally, when the object storage policy is created, an object index pool and a cache pool are determined, where the object index pool is used for storing object metadata, and the cache pool is used for caching object data.
In the embodiment of the present invention, the object storage policy includes: the data processing system comprises an active data pool and an inactive data pool, wherein the active data pool is used for indicating a storage pool with a reading and writing frequency higher than a preset frequency threshold, and the inactive data pool is used for indicating a storage pool with a reading and writing frequency lower than an access frequency equal to or lower than the preset frequency threshold.
Optionally, the execution unit includes: the device comprises a first creating module and a second creating module, wherein the first creating module is used for creating a bucket based on an object storage policy, and the bucket is used for writing object data to realize the storage of the object data.
Alternatively, after the object data is written into the active data pool in the bucket, the object index information is updated, wherein the object index information is used for recording the writing position of the object data.
As an alternative embodiment of the present invention, the data storage device further includes: a first setting module for setting a plurality of attribute parameters of each of the active data pools in the active pool group after performing storage of the object data based on the object storage policy, wherein the attribute parameters include at least: writing weight and writing times; the first calculation module is used for calculating an object writing coefficient of each activity data pool based on a plurality of attribute parameters; the first sequencing module is used for sequencing all the object write-in coefficients to obtain a sequencing result; and the first determining module is used for taking the active data pool with the minimum object writing coefficient as the current writing data pool based on the sequencing result.
Optionally, the data storage device further includes: and the first updating module is used for resetting the attribute parameters if the object storage strategy is updated or the object storage gateway is restarted after the plurality of attribute parameters of the active data pool are set.
Alternatively, the data storage device further includes: a first judgment module for judging an available capacity of each active data pool after performing storage of the object data based on the object storage policy; the second setting module is used for setting the active data pool as the inactive data pool and sending out a capacity alarm signal when the available capacity is lower than a preset capacity threshold; the first newly-built module is used for newly building a storage pool after sending out a capacity alarm signal to obtain a new storage pool; the second determining module is used for adding the new storage pool into an active pool group where an active data pool sending out a capacity alarm signal is located to serve as a new active data pool; and the third setting module is used for resetting the attribute parameters of each activity data pool based on the current available space of each activity data pool in the activity pool group after determining the new activity data pool.
Optionally, the data storage device further includes: the detection module is used for detecting the available space of each activity data pool in the activity pool group according to a preset time interval period after the storage of the object data is executed based on the object storage strategy; a second updating module for updating the available capacity ratio of each activity data pool based on the available space of each activity data pool; and the third updating module is used for updating the writing weight value of each activity data pool based on the available capacity ratio.
Alternatively, the data storage device further includes: the system comprises a receiving module, a storage module and a processing module, wherein the receiving module is used for receiving fault information after the storage of object data is executed based on an object storage strategy, and the fault information is used for indicating that a hard disk in an active data pool is in fault; the third determining module is used for determining a fault activity data pool and a fault hard disk based on the fault information; and the replacing module is used for replacing the fault hard disk in the fault active data pool based on the resource mutual isolation rule between the active data pools.
The object-based storage system data storage device may further include a processor and a memory, the obtaining unit 21, the first creating unit 23, the second creating unit 25, the executing unit 27, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor comprises a kernel, and the kernel calls a corresponding program unit from the memory. The kernel can be set to be one or more, and the operating parameters of the air conditioning equipment are adjusted by adjusting the kernel parameters so as to be adjusted to be parameters suitable for the environment where the user is located.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including: a processor; and a memory for storing executable instructions for the processor; wherein the processor is configured to perform any one of the above object based storage system data storage methods via execution of executable instructions.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: acquiring the number of hard disks on an object storage node; creating storage pools with different data security types based on the number of hard disks; creating an object storage strategy based on the storage pool, wherein the storage strategy is used for indicating the read-write rule of the object data; the storing of the object data is performed based on the object storage policy.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (13)
1. A data storage method based on an object storage system is characterized in that the object storage system comprises a plurality of object storage nodes, and the data storage method comprises the following steps:
acquiring the number of hard disks on an object storage node;
creating storage pools with different data security types based on the number of the hard disks;
creating an object storage policy based on the storage pool, wherein the storage policy is used for indicating read-write rules of object data;
performing storage of object data based on the object storage policy.
2. The data storage method of claim 1, wherein the data security type comprises: multiple copy types and erasure code types.
3. The data storage method of claim 1, wherein in creating the object storage policy, an object index pool and a cache pool are determined, wherein the object index pool is used for storing object metadata, and the cache pool is used for caching object data.
4. The data storage method of claim 1, wherein the object storage policy comprises: the data processing method comprises an active data pool and an inactive data pool, wherein the active data pool is used for indicating a storage pool with a reading and writing frequency higher than a preset frequency threshold, and the inactive data pool is used for indicating a storage pool with a reading and writing frequency lower than an access frequency equal to or lower than the preset frequency threshold.
5. The data storage method of claim 4, wherein the step of performing storage of object data based on the object storage policy comprises:
and creating a bucket based on the object storage policy, wherein the bucket is used for writing object data to realize the storage of the object data.
6. The data storage method according to claim 5, wherein after object data is written to the active data pool in the bucket, object index information is updated, wherein the object index information is used for recording the writing position of the object data.
7. The data storage method of claim 4, wherein after performing storage of object data based on the object storage policy, the data storage method further comprises:
Setting a plurality of attribute parameters of each activity data pool in the activity pool group, wherein the attribute parameters at least comprise: writing weight and writing times;
calculating an object write coefficient for each active data pool based on the plurality of attribute parameters;
sequencing all the object writing coefficients to obtain a sequencing result;
and based on the sorting result, taking the active data pool with the minimum object writing coefficient as the current writing data pool.
8. The data storage method of claim 7, wherein after setting the plurality of attribute parameters of the active data pool, the data storage method further comprises:
and if the object storage strategy is updated or the object storage gateway is restarted, resetting the attribute parameters.
9. The data storage method of claim 4, wherein after performing storage of object data based on the object storage policy, the data storage method further comprises:
judging the available capacity of each activity data pool;
if the available capacity is lower than a preset capacity threshold value, setting the active data pool as an inactive data pool, and sending a capacity alarm signal;
After the capacity alarm signal is sent out, a new storage pool is created to obtain a new storage pool;
adding the new storage pool into an active pool group where an active data pool sending out a capacity alarm signal is located to serve as a new active data pool;
after determining a new active data pool, resetting the attribute parameters of each of the active data pools based on the currently available space of each of the active data pools in the active pool group.
10. The data storage method of claim 4, wherein after performing storage of object data based on the object storage policy, the data storage method further comprises:
detecting the available space of each activity data pool in the activity pool group according to a preset time interval period;
updating the available capacity ratio of each of the active data pools based on the available space of each of the active data pools;
updating a write weight value for each of the active data pools based on the available capacity ratio.
11. The data storage method of claim 4, wherein after performing storage of object data based on the object storage policy, the data storage method further comprises:
Receiving fault information, wherein the fault information is used for indicating that a hard disk in an active data pool has a fault;
determining a fault activity data pool and a fault hard disk based on the fault information;
and replacing the fault hard disk in the fault active data pool based on the resource mutual isolation rule between the active data pools.
12. A data storage device based on an object storage system, wherein the object storage system comprises a plurality of object storage nodes, the data storage device comprising:
the acquisition unit is used for acquiring the number of hard disks on the object storage node;
a first creating unit, configured to create storage pools with different data security types based on the number of hard disks;
a second creating unit, configured to create an object storage policy based on the storage pool, where the storage policy is used to indicate a read-write rule of object data;
and the execution unit is used for executing the storage of the object data based on the object storage strategy.
13. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the object storage system based data storage method of any one of claims 1 to 11 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010705650.1A CN111857592A (en) | 2020-07-21 | 2020-07-21 | Data storage method and device based on object storage system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010705650.1A CN111857592A (en) | 2020-07-21 | 2020-07-21 | Data storage method and device based on object storage system and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111857592A true CN111857592A (en) | 2020-10-30 |
Family
ID=73000793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010705650.1A Pending CN111857592A (en) | 2020-07-21 | 2020-07-21 | Data storage method and device based on object storage system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111857592A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342272A (en) * | 2021-06-07 | 2021-09-03 | 深圳数联天下智能科技有限公司 | Sitting posture data storage method, sitting posture data display method, intelligent cushion and system |
| CN113687920A (en) * | 2021-08-30 | 2021-11-23 | 济南浪潮数据技术有限公司 | Object strategy operation method, device and equipment of distributed system |
| CN114449196A (en) * | 2021-12-21 | 2022-05-06 | 天翼云科技有限公司 | Cloud video monitoring data storage method and device and electronic equipment |
| CN114465957A (en) * | 2021-12-29 | 2022-05-10 | 天翼云科技有限公司 | Data writing method and device |
| CN116136846A (en) * | 2023-04-19 | 2023-05-19 | 苏州浪潮智能科技有限公司 | Unstructured object data management method, device and storage system |
| WO2024012592A1 (en) * | 2022-07-15 | 2024-01-18 | 锐捷网络股份有限公司 | Adaptive data disk capacity management method and apparatus, electronic device, and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104484130A (en) * | 2014-12-04 | 2015-04-01 | 北京同有飞骥科技股份有限公司 | Construction method of horizontal expansion storage system |
| CN109542342A (en) * | 2018-11-09 | 2019-03-29 | 锐捷网络股份有限公司 | Metadata management and data reconstruction method, equipment and storage medium |
| CN109669631A (en) * | 2018-12-10 | 2019-04-23 | 浪潮(北京)电子信息产业有限公司 | A kind of storage pool planing method, device, equipment and computer readable storage medium |
| CN110222013A (en) * | 2019-06-11 | 2019-09-10 | 深信服科技股份有限公司 | A kind of method, system, equipment and storage medium that cluster storage capacity determines |
| CN111274259A (en) * | 2020-02-16 | 2020-06-12 | 西安奥卡云数据科技有限公司 | Data updating method for storage nodes in distributed storage system |
-
2020
- 2020-07-21 CN CN202010705650.1A patent/CN111857592A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104484130A (en) * | 2014-12-04 | 2015-04-01 | 北京同有飞骥科技股份有限公司 | Construction method of horizontal expansion storage system |
| CN109542342A (en) * | 2018-11-09 | 2019-03-29 | 锐捷网络股份有限公司 | Metadata management and data reconstruction method, equipment and storage medium |
| CN109669631A (en) * | 2018-12-10 | 2019-04-23 | 浪潮(北京)电子信息产业有限公司 | A kind of storage pool planing method, device, equipment and computer readable storage medium |
| CN110222013A (en) * | 2019-06-11 | 2019-09-10 | 深信服科技股份有限公司 | A kind of method, system, equipment and storage medium that cluster storage capacity determines |
| CN111274259A (en) * | 2020-02-16 | 2020-06-12 | 西安奥卡云数据科技有限公司 | Data updating method for storage nodes in distributed storage system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342272A (en) * | 2021-06-07 | 2021-09-03 | 深圳数联天下智能科技有限公司 | Sitting posture data storage method, sitting posture data display method, intelligent cushion and system |
| CN113687920A (en) * | 2021-08-30 | 2021-11-23 | 济南浪潮数据技术有限公司 | Object strategy operation method, device and equipment of distributed system |
| CN113687920B (en) * | 2021-08-30 | 2024-02-13 | 济南浪潮数据技术有限公司 | Object policy operation method, device and equipment of distributed system |
| CN114449196A (en) * | 2021-12-21 | 2022-05-06 | 天翼云科技有限公司 | Cloud video monitoring data storage method and device and electronic equipment |
| CN114449196B (en) * | 2021-12-21 | 2023-08-04 | 天翼云科技有限公司 | Cloud video monitoring data storage method and device and electronic equipment |
| CN114465957A (en) * | 2021-12-29 | 2022-05-10 | 天翼云科技有限公司 | Data writing method and device |
| CN114465957B (en) * | 2021-12-29 | 2024-03-08 | 天翼云科技有限公司 | Data writing method and device |
| WO2024012592A1 (en) * | 2022-07-15 | 2024-01-18 | 锐捷网络股份有限公司 | Adaptive data disk capacity management method and apparatus, electronic device, and storage medium |
| CN116136846A (en) * | 2023-04-19 | 2023-05-19 | 苏州浪潮智能科技有限公司 | Unstructured object data management method, device and storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111857592A (en) | Data storage method and device based on object storage system and electronic equipment | |
| US8301670B2 (en) | Managing snapshot storage pools | |
| US7055053B2 (en) | System and method for failover | |
| US8245272B2 (en) | System and method for monitoring computer system resource performance | |
| CN109428913B (en) | Storage capacity expansion method and device | |
| CN105677236B (en) | A kind of storage device and its method for storing data | |
| US11449402B2 (en) | Handling of offline storage disk | |
| CN112272217B (en) | Kafka cluster load balancing method, system, equipment and medium | |
| CN109582213A (en) | Data reconstruction method and device, data-storage system | |
| CN112764680B (en) | Disk stripe selection method and storage system | |
| CN113625945A (en) | Distributed storage slow disk processing method, system, terminal and storage medium | |
| CN106293492A (en) | A kind of memory management method and distributed file system | |
| CN106484313A (en) | Data message backup method, data back up method and device | |
| CN102981939B (en) | Disk monitoring method | |
| CN113946291A (en) | Data access method, device, storage node and readable storage medium | |
| CN114443332A (en) | Storage pool detection method and device, electronic equipment and storage medium | |
| CN113760187B (en) | Method, system, terminal and storage medium for generating deduplication IO thread | |
| CN108306780B (en) | Cloud environment-based virtual machine communication quality self-optimization system and method | |
| CN111007988B (en) | RAID internal wear balancing method, system, terminal and storage medium | |
| CN107357686A (en) | A kind of daily record delet method and device | |
| CN116700606A (en) | Data storage method, device, equipment and storage medium | |
| CN115993932A (en) | Data processing method, device, storage medium and electronic equipment | |
| CN112883124B (en) | Data processing method and device, computer equipment and storage medium | |
| US10318207B1 (en) | Apparatus and methods for inter-version replication checking | |
| CN111124275A (en) | Monitoring service optimization method and device of distributed block storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100094 101, floors 1-5, building 7, courtyard 3, fengxiu Middle Road, Haidian District, Beijing Applicant after: Beijing Xingchen Tianhe Technology Co.,Ltd. Address before: 100097 room 806-1, block B, zone 2, Jinyuan times shopping center, indigo factory, Haidian District, Beijing Applicant before: XSKY BEIJING DATA TECHNOLOGY Corp.,Ltd. |
|
| CB02 | Change of applicant information |