CN111522501B - Disk array space dividing method and device, electronic equipment and storage medium - Google Patents
Disk array space dividing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111522501B CN111522501B CN201910105236.4A CN201910105236A CN111522501B CN 111522501 B CN111522501 B CN 111522501B CN 201910105236 A CN201910105236 A CN 201910105236A CN 111522501 B CN111522501 B CN 111522501B
- Authority
- CN
- China
- Prior art keywords
- blocks
- stripe
- target
- disk array
- strip
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Abstract
The embodiment of the application provides a disk array space dividing method, a disk array space dividing device, electronic equipment and a storage medium, wherein the method comprises the following steps: when a preset storage space increasing rule is met, determining a target strip needing to be increased; acquiring attribute information of each strip in a disk array; determining a target block to be released according to the attribute information of each strip and the number of blocks required for constructing the target strip; the target blocks are released and the target stripes are composed using the target blocks. The disk array space dividing method provided by the embodiment of the application can conveniently and flexibly adjust the redundancy of the strips and the space conversion of the array according to the requirements of video application and the state change of the storage disk, meets the amplification of the number of the strips, and can improve the adjustment efficiency of the storage space in the disk array. The old space is split, and a new storage space with redundancy is constructed, so that the image data storage space with redundancy can be provided under the condition that a magnetic disk is damaged.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a method and an apparatus for partitioning a disk array space, an electronic device, and a storage medium.
Background
RAID (redundant Array of independent disks) is a fast and ultra-large capacity external storage subsystem formed by combining a large number of Inexpensive disks into a Disk group with a large capacity according to a certain rule, such as striping, blocking, or interleaving. Under the control and management of the array controller, the system realizes fast, parallel or cross access, has strong fault-tolerant capability, and improves the efficiency of the whole disk system by the addition effect generated by providing data by individual disks. With this technique, data is divided into a plurality of sectors, each of which is stored on a respective hard disk.
In the related technology, a fixed storage space is allocated to each monitoring device through a RAID technology, and a cyclic coverage mode is adopted, namely, old image data is released, new image data is recorded, and periodic storage and updating of the image data are completed. However, in the process of video storage, a part of hard disks may be damaged or monitoring equipment may be newly added, in the related art, in the face of the above situation, the storage space of the entire disk array needs to be completely readjusted, the adjustment is not flexible enough, the workload of the disk array manager is increased, and the adjustment efficiency of the storage space in the disk array is low.
Disclosure of Invention
An object of the embodiments of the present application is to provide a method and an apparatus for partitioning a disk array space, an electronic device, and a storage medium, so as to improve adjustment efficiency of a storage space in a disk array. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a disk array space partitioning method, which is applied to a disk array, where the disk array is composed of multiple physical disks, each physical disk is logically partitioned into multiple blocks, at least two of the blocks form a stripe, and the stripe includes a data block and a check block, and the method includes:
when a preset storage space increasing rule is met, determining a target strip needing to be increased;
acquiring attribute information of each strip in the disk array;
determining a target block to be released according to the attribute information of each stripe and the number of blocks required for constructing the target stripe;
and releasing the target blocks and utilizing the target blocks to form the target stripe.
Optionally, the strip is used to store data collected by the monitoring device, and the preset storage space increase rule is as follows: and adding new monitoring equipment, increasing the storage time of data acquired by the monitoring equipment or causing a physical disk in the disk array to have a fault.
Optionally, the determining a target block to be released according to the attribute information of each stripe and the number of blocks required to construct the target stripe includes:
judging whether the total number of the redundant check blocks of all the strips in the disk array is not less than the number of blocks required for constructing the target strip or not according to the attribute information of each strip;
and when the total number of the check blocks of all the stripe redundancies in the disk array is not less than the number of the blocks required for constructing the target stripe, determining the target blocks required to be released in the check blocks of each stripe.
Optionally, after determining, according to the attribute information of each stripe, whether the total number of check blocks redundant to all stripes in the disk array is not less than the number of blocks required for constructing the target stripe, the method further includes:
and when the total number of the check blocks of all the stripe redundancies in the disk array is less than the number of the blocks required for constructing the target stripe, determining the data blocks and the check blocks which need to be released in each stripe as the target blocks which need to be released.
Optionally, the strip is used for storing data collected by the monitoring device, and the method further includes:
determining a strip to be split when the storage time of the data acquired by the monitoring equipment is reduced or the number of the monitoring equipment is reduced;
releasing the data block and the check block of the stripe to be split to obtain a split block;
and taking each split block as a check block, and adding the check block into other strips of the disk array except the strip to be split.
In a second aspect, an embodiment of the present application provides a disk array space partitioning apparatus, which is applied to a disk array, where the disk array is composed of multiple physical disks, each of the physical disks is logically partitioned into multiple blocks, at least two of the blocks form a stripe, and the stripe includes a data block and a check block, and the apparatus includes:
the target strip determining module is used for determining a target strip needing to be added when a preset storage space increasing rule is met;
an attribute information obtaining module, configured to obtain attribute information of each stripe in the disk array;
a target block determining module, configured to determine a target block to be released according to the attribute information of each stripe and the number of blocks required to construct the target stripe;
and the stripe adding module is used for releasing the target blocks and utilizing the target blocks to form the target stripes.
Optionally, the strip is used to store data collected by the monitoring device, and the preset storage space increase rule is as follows: and adding new monitoring equipment, increasing the storage time of data acquired by the monitoring equipment or causing a physical disk in the disk array to have a fault.
Optionally, the target block determining module includes:
a redundancy judgment submodule, configured to judge, according to attribute information of each stripe, whether the total number of redundant check blocks of all stripes in the disk array is not less than the number of blocks required for constructing the target stripe;
and the check block selection submodule is used for determining a target block to be released in the check blocks of each stripe when the total number of the check blocks of all the stripe redundancies in the disk array is not less than the number of blocks required for constructing the target stripe.
Optionally, the target block determining module further includes:
and the data block selection submodule is used for determining the data blocks and the check blocks which need to be released in each stripe as the target blocks which need to be released when the total number of the check blocks of all the stripe redundancies in the disk array is less than the number of the blocks required for constructing the target stripes.
Optionally, the strip is used to store data collected by the monitoring device, and the apparatus further includes:
the strip determining module is used for determining strips to be split when the storage time of the data acquired by the monitoring equipment is reduced or the number of the monitoring equipment is reduced;
the stripe splitting module is used for releasing the data block and the check block of the stripe to be split to obtain a split block;
and the check block adding module is used for adding each split block serving as a check block to other strips of the disk array except the strip to be split.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement the disk array space partitioning method according to any one of the first aspect above when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method for partitioning a disk array space according to any one of the above first aspects is implemented.
According to the disk array space dividing method and device, the electronic equipment and the storage medium, when the preset storage space increasing rule is met, the target strips needing to be increased are determined; acquiring attribute information of each strip in a disk array; determining a target block to be released according to the attribute information of each strip and the number of blocks required for constructing the target strip; the target blocks are released and the target stripes are composed using the target blocks. The redundancy of the strips and the space conversion of the array can be conveniently and flexibly adjusted according to the requirements of video application and the state change of the storage disk, the amplification of the number of the strips is met, and the adjustment efficiency of the storage space in the disk array can be improved. The old space is split, and a new storage space with redundancy is constructed, so that the image data storage space with redundancy can be provided under the condition that a magnetic disk is damaged. Of course, not all advantages described above need to be achieved at the same time in the practice of any one product or method of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of a disk array space partitioning method implemented in the present application;
FIG. 2 is a schematic diagram of a disk array space partitioning apparatus implemented in the present application;
FIG. 3 is a first schematic diagram of a disk array space partitioning implemented in the present application;
FIG. 4 is a second schematic diagram of a disk array space partitioning implemented in the present application;
FIG. 5 is a third schematic diagram of a disk array space partitioning implemented in the present application;
FIG. 6 is a fourth exemplary illustration of a disk array space partitioning implemented in the present application;
FIG. 7 is another schematic diagram of a disk array space partitioning apparatus implemented in the present application;
fig. 8 is a schematic diagram of an electronic device implemented by the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
First, terms in the present application will be explained.
The video monitoring system comprises a front-end camera, a transmission cable and a video monitoring platform. The cameras can be divided into network digital cameras and analog cameras and can be used for collecting front-end video image signals. The complete video monitoring system is composed of a majority of camera shooting, transmission, control, display and record registration 5. The video camera transmits video images to the control host through a network cable or a coaxial video cable, the control host distributes video signals to each monitor and video equipment, and simultaneously, voice signals needing to be transmitted can be synchronously recorded into the video camera.
RAID 2.0: a single hard disk is divided into logical blocks according to a certain size (size), and the logical blocks (hereinafter referred to as blocks) are used for RAID instead of disks. The method can also realize various RAID levels of 0, 1, 5 and 6, and hot spare blocks are distributed on each disk, so that the method is an enhanced RAID technology, and effectively solves the problems that the capacity of a mechanical hard disk is increased more and more, the time required for reconstructing one mechanical hard disk is increased more and more, and the reconstruction window of the traditional RAID group is increased more and more, so that the risk of completely losing data due to the failure of one hard disk during reconstruction is caused.
EC: the Erasure Coding is a data protection method, which divides data into segments, expands and codes redundant data blocks, and stores the redundant data blocks in different locations, such as a disk, a storage node or other geographical locations, and a storage scheme based on the Erasure codes can easily realize distributed/cloud storage of K + M (K is original data, and M is redundant backup).
Sripe (stripe) is a method of dividing continuous data into data blocks of the same size and writing each data block to a different disk in the array. Simply stated, striping is a method of consolidating multiple disk drives into a volume, typically accomplished through a hardware controller. Striping is a technique for automatically balancing the load of I/O (Input/Output) to multiple physical disks, and is to divide a continuous piece of data into many small parts and store them on different disks. This enables multiple processes to access multiple different portions of data simultaneously without disk conflicts, and to maximize I/O parallelism when sequential access to such data is required, resulting in very good performance.
In the field of security protection, a video recording video stores special characteristics of the video recording video, and the total video recording point position, the video recording period, the required space and the like of a security protection project are determined; in most projects, monitoring equipment is installed step by step in the initial deployment stage, namely video points are added into a system step by step in batches and periods, the division of RAID array space in the related technology is determined at the beginning of establishment, a user divides storage space of RAID after considering subsequent expansion points, so that the storage space is determined to be abundant in a period of time, the storage space is wasted before the video points in the subsequent batch are added, the free storage space cannot be effectively converted into redundant space, and data loss is caused when a disk is damaged. However, if the storage space of the RAID is divided only according to the number of the current actual video point locations, and when a new video point location is subsequently added, the storage space of the entire RAID needs to be manually re-divided, which is not flexible enough to meet the demand for increasing and decreasing the video point locations. Even if the video point location is reserved in the initial stage, the situation of additionally increasing the video point location is inevitable.
In view of this, an embodiment of the present application provides a method for partitioning a disk array space, which is applied to a disk array, and referring to fig. 1, where the disk array is composed of a plurality of physical disks, each of the physical disks is logically partitioned into a plurality of blocks, at least two of the blocks form a stripe, and the stripe includes a data block and a check block, and the method includes:
s101, when a preset storage space increasing rule is met, determining a target stripe needing to be increased.
The disk array space division method in the embodiment of the application can be realized through a division system, and the division system is any system capable of realizing the disk array space division method in the embodiment of the application. For example:
the partitioning system may be an electronic device comprising: a processor, a memory, a communication interface, and a bus; the processor, the memory and the communication interface are connected through a bus and complete mutual communication; the memory stores executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the disk array space dividing method of the embodiment of the application.
The partitioning system may also be an application program, configured to execute the disk array space partitioning method according to the embodiment of the present application when running.
The partitioning system may also be a storage medium for storing executable code for executing the disk array space partitioning method according to the embodiment of the present application.
Specifically, the partitioning system may be a RAID controller.
When the partitioning system detects that a preset storage space increasing rule is met, the partitioning system determines target stripes needing to be increased, including determining the number of the target stripes needing to be increased and the number of data blocks and check blocks in each target stripe. The number of data blocks in the target stripe and the number of check blocks in the target stripe may be input by a user, may be set in advance, or may be calculated by the partitioning system according to the number of data blocks and check blocks of each stripe in the disk array. For example, the dividing system calculates the average value of the number of data blocks of each stripe in the disk array, and the average value is used as the number of data blocks in the target stripe; and calculating the average value of the number of check blocks of each strip in the disk array, and taking the integral value as the number of the check blocks in the target strip. Rounding herein is any rounding method including, but not limited to, rounding up, rounding down, and the like.
In view of the characteristics of video storage, optionally, the number of data blocks of each stripe in the disk array is set to be a preset number, and the preset number may be set according to an actual storage requirement, for example, set to be 4, 5, or 16, and the like, so that the number of data blocks in the target stripe is the preset number, and the number of check blocks in the target stripe is determined according to the specified redundancy. The specified redundancy of the target stripe may be the same as or different from the redundancy of other stripes in the disk array. The specified redundancy may be input by a user, or may be calculated by the partitioning system according to the redundancy of each stripe in the disk array, for example, the partitioning system calculates an average of the redundancies of each stripe in the disk array to obtain the specified redundancy. It can be understood that, the block in the embodiment of the present application is a minimum partition unit, and when determining the number of parity blocks in the target stripe according to the specified redundancy, if the number of parity blocks in the target stripe is not an integer, rounding needs to be performed, including but not limited to rounding up, rounding down, rounding up, and the like.
The preset storage space increasing rule can be set according to actual conditions, and the preset storage space increasing rule represents conditions that the storage space needs to be increased. Optionally, in the embodiment of the present application, the stripe is used to store data collected by the monitoring device, and the preset storage space increasing rule is as follows: and adding new monitoring equipment, increasing the storage time of data acquired by the monitoring equipment or causing a fault to occur to a physical disk in the disk array.
When a new monitoring device is added, that is, video point positions are increased, or when a physical disk in a disk array has a fault, or when the storage time of data collected by the monitoring device is increased, the storage space in the current RAID is insufficient, so that the redundancy of the RAID needs to be reduced to increase the storage space, that is, a new stripe is added, thereby satisfying the need of data storage collected by the monitoring device.
S102, acquiring the attribute information of each strip in the disk array.
The partitioning system obtains attribute information of each stripe in the disk array, where the attribute information of a stripe may include redundancy of the stripe, size of storage space in the stripe, and identification of each data block and check block in the stripe.
And S103, determining the target blocks to be released according to the attribute information of each stripe and the number of blocks required for constructing the target stripe.
The partitioning system calculates the number of target blocks to be released according to the number of blocks required for constructing the target stripe, and determines the target blocks to be released in each stripe according to the number. The case of adding a new stripe may include: after the monitoring equipment is added, a new video point location needs to be added; or a physical disk in the RAID has failed; or the time that the data collected by the monitoring device needs to be saved increases, etc. For example, when the time required for storing the data collected by the monitoring device is increased from 30 days to 60 days, the storage space is doubled, and a new stripe can be constructed by releasing the check block; when monitoring equipment is added, a new stripe can be established in a mode of releasing the check block; when the physical disk fails, a new stripe can be constructed by splitting the old stripe and releasing the check blocks. The condition of releasing the strip can be specific to the conditions of physical disk failure, change of data storage period collected by the monitoring equipment and the like. When the physical disk fails, the target blocks comprise available blocks in the target stripe, and the blocks in the target stripe which are unavailable due to the physical disk failure do not belong to the target blocks.
For example, in RAID, all physical disks in total have L available blocks, the current number of stripes is X, each stripe includes N data blocks and several check blocks, and it is assumed that a target stripes need to be added, because the time duration of data collected by the monitoring device that needs to be stored is determined, that is, the number of data blocks of the newly added target stripes is N, therefore, the newly added stripes need at least a · N blocks, but to ensure redundancy, on the premise of an average redundancy space, a stripes need to have a · N blocks
And (4) blocks. Therefore, the check blocks of the current X stripes can be selected
The block serves as a target block. The partitioning system may averagely select the parity blocks in the current X stripes as the target blocks according to an averagely selecting method (because the minimum unit of the blocks that can be selected is 1, there may be a case where the number of the target blocks selected in different stripes differs by 1).
For example, in RAID, before a physical disk fails, all the physical disks have L total available blocks, the number of stripes is X, each stripe includes N data blocks and several parity blocks, the failed physical disk includes K blocks, and the K blocks are distributed in a stripes (hereinafter referred to as a damaged stripe), where B damaged stripes are to be written with new data, and the new data needs C total stripes, where [ C-B [ ]]Individual strips had no damaging effect; the dividing system firstly releases all B damaged stripe blocks to obtain Y usable blocks as target blocks, and in order to ensure redundancy, the dividing system also needs to release on the premise of average redundant space
The check block is used as a target block, and the dividing system can select the current [ X-B ] according to an average selection method]The parity blocks are selected on average from the stripes as target blocks (since the minimum unit of the blocks that can be selected is 1, there may be a case where the number of target blocks selected in different stripes differs by 1).
Optionally, in step S103, determining a target block to be released according to the attribute information of each stripe and the number of blocks required to construct the target stripe, includes:
step one, according to the attribute information of each strip, judging whether the total number of the redundant check blocks of all strips in the disk array is not less than the number of blocks required for constructing the target strip.
The dividing system calculates the total number of check blocks of all strips in the disk array and judges the total number of the check blocks and the number of blocks required for constructing the target strip. For example, if the number of data blocks in a stripe is N and the number of parity blocks is M (typically M ≦ N), then the number of parity blocks that the stripe can redundancy is M. If there are K stripes in total, and the number of parity chunks in each stripe is M, the total number of parity chunks of all stripe redundancies in the disk array is K × M.
And secondly, when the total number of the redundant check blocks of all the stripes in the disk array is not less than the number of blocks required for constructing the target stripes, determining the target blocks to be released in the check blocks of all the stripes.
Generally, the data blocks in the stripe are used for storing the target data, and the check blocks in the stripe are used for performing redundant backup on the data in the data blocks. Therefore, when the target block is selected, the check block of the stripe can be preferentially selected, so that the influence on the target data is reduced. For example, when the total number of redundant check blocks of all stripes in the disk array is not less than the number of blocks required for constructing a target stripe, acquiring the number X of blocks required for constructing the target stripe, then traversing a stripe linked list with the highest redundancy (in the embodiment of the present application, the redundancy of a stripe refers to the number of redundant check blocks in the stripe) and counting the number Y, assuming that Y > is X, randomly selecting X stripes from the stripes with the highest redundancy, and selecting 1 check block for each stripe; if X > Y > -X/2, selecting 1 check block from each strip on the strip with the highest redundancy, randomly selecting X-Y strips and then selecting 1 check block, when X/2> Y, selecting 2 check blocks from each strip on the strip with the highest redundancy, considering the selected check blocks as released, adjusting the redundancy of the strips to the corresponding redundancy level, and traversing the next strip with the highest redundancy until the number of the selected check blocks reaches X.
In one possible embodiment, the stripes are ranked according to the importance level of the data stored in the stripes, and each stripe corresponds to a corresponding importance level. Optionally, determining a target block to be released in the parity chunks of the stripes includes: and selecting the check blocks from the stripes as target blocks in sequence according to the sequence of the importance levels of the stripes from low to high until the number of the target blocks reaches the number of blocks required for constructing the target stripes.
Of course, in addition to selecting the target block from the parity blocks, the target block to be released may be selected from the data blocks of each stripe, but after the target block is selected from the data blocks of a stripe, the data blocks in the stripe need to be complemented again by using the parity blocks, for example, one data block is selected from the stripe a as the target block, and one parity block (the parity block of the stripe a is preferentially selected) needs to be divided into the data blocks of the stripe a to ensure the storage space of the stripe a. Therefore, the target blocks needing to be released are determined in the check blocks of each stripe, and the efficiency is higher.
Optionally, after determining, according to the attribute information of each stripe, whether the total number of the redundant check blocks of all stripes in the disk array is not less than the number of blocks required for constructing the target stripe, the method further includes:
and when the total number of the check blocks of all the stripe redundancies in the disk array is less than the number of the blocks required for constructing the target stripe, determining the data blocks and the check blocks which need to be released in each stripe as the target blocks which need to be released.
When the number of redundant check blocks in each stripe is less than the number of blocks required for constructing the target stripe, it indicates that the number of redundant check blocks is insufficient, the stripe (including the data block and the check block) in the old space needs to be destroyed, and the redundant check block is released to serve as the target block to be released. Or when the storage time of the data collected by the monitoring equipment is allowed to be reduced, the storage space capable of being released can be determined according to the reduced storage time, the data block capable of being released is further determined, the check block capable of being released is determined according to the reduced storage space and the redundancy requirement, and the data block and the check block capable of being released are used as target blocks.
And S104, releasing the target blocks and utilizing the target blocks to form the target stripe.
The partitioning system first releases each target block and then assembles each target block into the above-mentioned stripe that needs to be added.
In the embodiment of the application, the redundancy of the strips and the space conversion of the array can be conveniently and flexibly adjusted according to the requirement of video application and the state change of the storage disk, the amplification of the number of the strips is met, and the adjustment efficiency of the storage space in the disk array can be improved. The old space is split, a new storage space with redundancy is built, and the data storage space collected by the monitoring equipment with redundancy can be provided under the condition that a magnetic disk is damaged.
Optionally, the strip of the embodiment of the present application is used to store data collected by the monitoring device, and the method further includes:
step one, when the number of the monitoring devices is reduced, determining a strip to be split.
And determining the strips corresponding to the reduced monitoring equipment as the strips to be split.
And step two, releasing the data block and the check block of the stripe to be split to obtain a split block.
And thirdly, taking each split block as a check block, and adding the check block into other strips of the disk array except the strip to be split.
And taking the split block as a check block, and adding the check block into other strips of the disk array except the strip to be split, so that the redundancy of the other strips is increased.
Optionally, the strip of the embodiment of the present application is used to store data collected by the monitoring device, and the method further includes:
step one, when the storage time of the data collected by the monitoring equipment is reduced, determining a strip to be split.
The data of the monitoring equipment can be stored in a periodic coverage mode, and when the storage time of the data collected by the monitoring equipment is shortened, the storage space which can be released can be determined according to the shortened storage time, so that the strip which can be released can be determined.
And step two, releasing the data block and the check block of the stripe to be split to obtain a split block.
And thirdly, taking each split block as a check block, and adding the check block into other strips of the disk array except the strip to be split.
And taking the split block as a check block, and adding the check block into other strips of the disk array except the strip to be split, so that the redundancy of the other strips is increased.
An embodiment of the present application further provides a disk array space dividing apparatus, as shown in fig. 2, including: the system comprises an array space detection module 201, an array space decision module 202 and an array space processing module 203, wherein the array space processing module 203 comprises an array redundancy processing submodule and an array old space processing submodule.
In combination with image data services, when an array is established, matching array forms, such as 4+4, 16+16 and the like, can be established according to the actual configuration of monitoring equipment of a client and a disk in the equipment, wherein the number before the addition represents the number of data blocks of each strip in the array, and the number after the addition represents the number of check blocks of each strip in the array. At this time, if the array stripe has a higher redundant space, it can deal with more disk damages without affecting the data loss, and can also satisfy the space requirement of the image data service (hereinafter, replaced by the application side) of the application side.
In the using process, if a storage space needs to be increased, an array space detection module needs to be informed, or an automatic detection function is set in advance; when the array space detection module obtains notification or detects that the space is insufficient (the space address of an application side exceeds the current maximum space), the array decides whether the array releases the strip blocks by releasing the strip blocks of the strip or destroying the strip blocks of the old space according to a set array space processing strategy and the current array strip condition; for example, the preset minimum redundancy of the disk array is M ' (M ' > = 0), if M ' is 1, when the existing stripe redundancies M are all found to be greater than M ', the stripes are released, and when the existing stripe redundancies M are all found to be equal to or less than M ', the stripes in the old space are destroyed. And deciding which strips to adjust into which form of N + M; for example, according to the required space and the size of the stripe blocks, calculating the number of the required stripe blocks, assuming that the number is X, then traversing the stripe chain table with the highest redundancy and counting out the number Y, and the required stripe blocks X, assuming that Y > is X, randomly selecting X stripes from the stripes with the highest redundancy, and selecting 1 stripe block from each stripe; if X > Y > -X/2, selecting 1 block from each strip on the strip with the highest redundancy, randomly selecting X-Y strips and then selecting 1 block, and when X/2> Y, selecting 2 blocks from each strip on the strip with the highest redundancy, and traversing the next strip with the high redundancy; after the selected stripe releases the block, it adjusts its redundancy into its corresponding redundancy level.
After the array space processing module obtains the instruction of the array space decision module, the instruction is sent to the array redundancy processing submodule or the array old space processing submodule, the strips on the strips are disassembled, the strips are recycled, a new N + M strip is formed according to a new decision strategy, and an expansion space is provided for an application party.
For example, fig. 3, it is assumed that a 4+4 array is created according to video parameters such as the storage period, the number of paths, and the code stream of image data, and it is assumed that there are stripe (stripe)0, stripe 1, and stripe 2; after the operation is carried out for a period of time, the number of the network cameras is newly increased, namely the number of paths is increased, a storage space is correspondingly required to be increased (other storage space increasing scenes such as the storage period of image data increased in the monitoring industry and the like), the application party informs the array of the need of increasing the space through the array space detection module at the moment, and the array space decision module assumes that the strategy is to release the strips of the strips according to the set strategy, and selects the strips to be released, wherein each strip releases a plurality of blocks; the information is fed to an array spatial processing module.
As shown in fig. 4, the array space processing module executes according to a policy, and the array redundancy processing sub-module releases 2 stripe blocks, here, for example, parity blocks, on each of the stripes 0, 1, 2 (of course, data blocks may also be released, but it is more efficient to preferentially release the parity blocks); after the array space processing module recovers the released strips, the array space processing module applies for building new 4+2 strips 3 again, and expands the space of the array, and after the processing, as shown in fig. 5, the whole array becomes a 4+2 array, thereby reducing the redundancy of the original array, increasing the capacity of the array, and meeting the requirement of an application party on new space, and certainly, the situation that the strips of 4+4 and 4+2 may exist in the whole array is solved. Vice versa, in some cases, the application party can reduce the traffic volume due to the requirements of balanced services and the like, the array space detection module informs the array of reducing the space, and the array space decision module selects the old space stripe needing to be recovered and the stripe needing to be added with redundancy and gives the old space stripe and the stripe to the array space processing module; and after the old space processing submodule of the array splits the old space stripe and recovers the stripe, newly-added stripes are redistributed to the stripes needing to be added with redundancy, and the space of the array is adjusted. The redundant processing sub-module and the old space processing sub-module of the array can be simultaneously set and executed so as to meet the new writing space requirement of an application side.
The design is also suitable for the expansion and reduction of the array space when the array redundancy is not influenced by the damage of the disk, and the principle is as above; when the disk damage exceeds the array redundancy, the application side continues writing, and at this time, through the interaction of the application side, the stripes in the old space are determined to be removed, and a new stripe is constructed after recovery, wherein the new stripe is a stripe with complete data, for example, a stripe without redundancy, such as a 4+0 stripe, can also be constructed as a 4+1 stripe, so as to ensure that newly recorded image data is complete, for the application side, the adjustment may be to shorten the recording period or stop recording of part of the point locations, and the corresponding adjustment may be to a policy set in the array space decision module.
In the embodiment of the application, the stripe management technology based on the image data can conveniently and flexibly adjust the redundancy of the stripes and the space transformation of the array according to the requirements of the image data and the state change of the storage disk, and meets the requirements of the amplification and the reduction of the image data. The old space is split, and a new storage space with redundancy is built, so that the image data storage space with redundancy can be provided under the condition that most of the disks are damaged.
An embodiment of the present application further provides a device for dividing a disk array space, where, referring to fig. 7, the device is applied to a disk array, the disk array is composed of a plurality of physical disks, each of the physical disks is logically divided into a plurality of blocks, at least two of the blocks form a stripe, the stripe includes a data block and a check block, and the method includes:
a target stripe determining module 701, configured to determine a target stripe that needs to be added when a preset storage space addition rule is satisfied;
an attribute information obtaining module 702, configured to obtain attribute information of each stripe in the disk array;
a target block determining module 703, configured to determine a target block to be released according to the attribute information of each stripe and the number of blocks required to construct the target stripe;
and a stripe adding module 704, configured to release the target blocks and compose the target stripes by using the target blocks.
Optionally, the strip is used to store data collected by the monitoring device, and the preset storage space increasing rule is as follows: and adding new monitoring equipment, increasing the storage time of the data acquired by the monitoring equipment or causing the physical disk in the disk array to be in fault.
Optionally, the target block determining module 701 includes:
a redundancy judgment submodule, configured to judge, according to attribute information of each stripe, whether the total number of redundant check blocks of all stripes in the disk array is not less than the number of blocks required for constructing the target stripe;
and the check block selection submodule is used for determining a target block to be released in the check blocks of each stripe when the total number of the check blocks of all the stripe redundancies in the disk array is not less than the number of blocks required for constructing the target stripe.
Optionally, the target block determining module 701 further includes:
and the data block selection submodule is used for determining the data blocks and the check blocks which need to be released in each stripe as the target blocks which need to be released when the total number of the check blocks of all the stripe redundancies in the disk array is less than the number of the blocks required for constructing the target stripes.
Optionally, the above-mentioned strip is used for storing data collected by the monitoring device, and the disk array space partitioning apparatus in an embodiment of the present application further includes:
the strip determining module is used for determining strips to be split when the storage time of the data acquired by the monitoring equipment is reduced or the number of the monitoring equipment is reduced;
the stripe splitting module is used for releasing the data block and the check block of the stripe to be split to obtain a split block;
and the check block adding module is used for adding each split block serving as a check block to other strips of the disk array except the strip to be split.
The embodiment of the application also provides an electronic device, which comprises a processor and a memory;
a memory for storing a computer program;
the processor is used for realizing the following steps when executing the program stored in the memory:
when a preset storage space increasing rule is met, determining a target strip needing to be increased;
acquiring attribute information of each stripe in the disk array;
determining a target block to be released according to the attribute information of each stripe and the number of blocks required for constructing the target stripe;
and releasing the target blocks and using the target blocks to compose the target stripe.
Optionally, as shown in fig. 8, the electronic device according to the embodiment of the present application further includes a communication interface 802 and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete communication with each other through the communication bus 804.
Optionally, when the processor 801 is used to execute a program stored in the memory 803, any of the above-described disk array space division methods can also be implemented.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any one of the disk array space partitioning methods is provided.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the embodiments of the apparatus, the electronic device, and the storage medium, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims (8)
1. A disk array space division method is applied to a disk array, the disk array is composed of a plurality of physical disks, each physical disk is logically divided into a plurality of blocks, at least two blocks form a strip, the strip comprises data blocks and check blocks, and the method comprises the following steps:
when a preset storage space increasing rule is met, determining a target strip needing to be increased, wherein the strip is used for storing data collected by monitoring equipment, and the preset storage space increasing rule is as follows: adding new monitoring equipment, increasing the storage time of data acquired by the monitoring equipment or causing a physical disk in the disk array to break down;
acquiring attribute information of each strip in the disk array;
determining a target block to be released according to the attribute information of each stripe and the number of blocks required for constructing the target stripe;
releasing the target blocks and utilizing the target blocks to form the target stripes;
determining a target block to be released according to the attribute information of each stripe and the number of blocks required for constructing the target stripe, including:
judging whether the total number of the redundant check blocks of all the strips in the disk array is not less than the number of blocks required for constructing the target strip or not according to the attribute information of each strip;
and when the total number of the check blocks of all the stripe redundancies in the disk array is not less than the number of the blocks required for constructing the target stripe, determining the target blocks required to be released in the check blocks of each stripe.
2. The method of claim 1, wherein after determining whether the total number of parity blocks redundant to all stripes in the disk array is not less than the number of blocks required to construct the target stripe according to the attribute information of each stripe, the method further comprises:
and when the total number of the check blocks of all the stripe redundancies in the disk array is less than the number of the blocks required for constructing the target stripe, determining the data blocks and the check blocks which need to be released in each stripe as the target blocks which need to be released.
3. The method of claim 1, wherein the strip is used to store data collected by a monitoring device, the method further comprising:
determining a strip to be split when the storage time of the data acquired by the monitoring equipment is reduced or the number of the monitoring equipment is reduced;
releasing the data block and the check block of the stripe to be split to obtain a split block;
and taking each split block as a check block, and adding the check block into other strips of the disk array except the strip to be split.
4. The utility model provides a disk array space partitioning device, characterized in that is applied to the disk array, the disk array comprises a plurality of physical disks, every physical disk logic divides into a plurality of blocks, at least two the block becomes a strip, the strip includes data block and check block, the device includes:
a target strip determining module, configured to determine a target strip that needs to be added when a preset storage space addition rule is met, where the strip is used to store data collected by a monitoring device, and the preset storage space addition rule is: adding new monitoring equipment, increasing the storage time of data acquired by the monitoring equipment or causing a physical disk in the disk array to break down;
an attribute information obtaining module, configured to obtain attribute information of each stripe in the disk array;
a target block determining module, configured to determine a target block to be released according to the attribute information of each stripe and the number of blocks required to construct the target stripe;
the stripe adding module is used for releasing the target blocks and utilizing the target blocks to form the target stripes;
the target block determination module includes:
a redundancy judgment submodule, configured to judge, according to attribute information of each stripe, whether the total number of redundant check blocks of all stripes in the disk array is not less than the number of blocks required for constructing the target stripe;
and the check block selection submodule is used for determining a target block to be released in the check blocks of each stripe when the total number of the check blocks of all the stripe redundancies in the disk array is not less than the number of blocks required for constructing the target stripe.
5. The apparatus of claim 4, wherein the target block determination module further comprises:
and the data block selection submodule is used for determining the data blocks and the check blocks which need to be released in each stripe as the target blocks which need to be released when the total number of the check blocks of all the stripe redundancies in the disk array is less than the number of the blocks required for constructing the target stripes.
6. The apparatus of claim 4, wherein the strip is configured to store data collected by a monitoring device, the apparatus further comprising:
the strip determining module is used for determining strips to be split when the storage time of the data acquired by the monitoring equipment is reduced or the number of the monitoring equipment is reduced;
the stripe splitting module is used for releasing the data block and the check block of the stripe to be split to obtain a split block;
and the check block adding module is used for adding each split block serving as a check block to other strips of the disk array except the strip to be split.
7. An electronic device comprising a processor and a memory;
the memory is used for storing a computer program;
the processor, when executing the program stored in the memory, implementing the method steps of any of claims 1-3.
8. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-3.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910105236.4A CN111522501B (en) | 2019-02-01 | 2019-02-01 | Disk array space dividing method and device, electronic equipment and storage medium |
| PCT/CN2019/127476 WO2020155930A1 (en) | 2019-02-01 | 2019-12-23 | Disk array space division method, apparatus, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910105236.4A CN111522501B (en) | 2019-02-01 | 2019-02-01 | Disk array space dividing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111522501A CN111522501A (en) | 2020-08-11 |
| CN111522501B true CN111522501B (en) | 2022-01-21 |
Family
ID=71840819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910105236.4A Active CN111522501B (en) | 2019-02-01 | 2019-02-01 | Disk array space dividing method and device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111522501B (en) |
| WO (1) | WO2020155930A1 (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112506430B (en) * | 2020-12-04 | 2022-07-26 | 苏州浪潮智能科技有限公司 | Method and device for keeping SSD fixed stripe size |
| CN112783689B (en) * | 2021-02-08 | 2022-09-02 | 上海交通大学 | Partial stripe write optimization method and device based on LRC coding |
| CN113867644B (en) * | 2021-09-30 | 2023-08-04 | 苏州浪潮智能科技有限公司 | Disk array optimization method, disk array optimization device, computer equipment and storage medium |
| CN114564335B (en) * | 2022-01-14 | 2024-03-29 | 中国科学技术大学 | Local repairable code redundancy conversion method based on stripe combination and storage medium |
| CN114442950B (en) * | 2022-01-21 | 2024-01-23 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system, device and computer readable storage medium |
| CN114489508B (en) * | 2022-01-26 | 2023-09-01 | 重庆紫光华山智安科技有限公司 | Data management method, device, equipment and medium |
| CN114995771B (en) * | 2022-08-02 | 2022-12-13 | 苏州浪潮智能科技有限公司 | Formatting scheduling method, device, equipment and medium for redundant array of independent disks |
| CN116149575B (en) * | 2023-04-20 | 2023-07-25 | 北京大学 | Server-oriented non-perception computing disk redundant array writing method and system |
| CN116301660B (en) * | 2023-05-10 | 2023-08-22 | 深圳创新科技术有限公司 | Video recording and writing method, device, equipment and medium based on raid |
| CN117112455B (en) * | 2023-10-24 | 2024-02-06 | 苏州元脑智能科技有限公司 | Space address conversion method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104267913A (en) * | 2014-10-20 | 2015-01-07 | 北京北亚时代科技有限公司 | Storage method and system allowing dynamic asynchronous RAID level adjustment |
| CN106970764A (en) * | 2017-02-24 | 2017-07-21 | 中国科学院计算技术研究所 | An a kind of watt record RAID based on band version writes ordered method and system |
| CN108628541A (en) * | 2017-03-22 | 2018-10-09 | 华为技术有限公司 | A kind of method, apparatus and storage system of file storage |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060106893A1 (en) * | 2004-11-02 | 2006-05-18 | Rodger Daniels | Incremental backup operations in storage networks |
| US20060155944A1 (en) * | 2005-01-13 | 2006-07-13 | Hitachi, Ltd. | System and method for data migration and shredding |
| CN103186438A (en) * | 2013-04-02 | 2013-07-03 | 浪潮电子信息产业股份有限公司 | Method of improving disk array data reconstruction efficiency |
| CN107491263B (en) * | 2016-06-12 | 2022-07-22 | 北京忆恒创源科技股份有限公司 | Data reconstruction method based on storage object |
-
2019
- 2019-02-01 CN CN201910105236.4A patent/CN111522501B/en active Active
- 2019-12-23 WO PCT/CN2019/127476 patent/WO2020155930A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104267913A (en) * | 2014-10-20 | 2015-01-07 | 北京北亚时代科技有限公司 | Storage method and system allowing dynamic asynchronous RAID level adjustment |
| CN106970764A (en) * | 2017-02-24 | 2017-07-21 | 中国科学院计算技术研究所 | An a kind of watt record RAID based on band version writes ordered method and system |
| CN108628541A (en) * | 2017-03-22 | 2018-10-09 | 华为技术有限公司 | A kind of method, apparatus and storage system of file storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111522501A (en) | 2020-08-11 |
| WO2020155930A1 (en) | 2020-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111522501B (en) | Disk array space dividing method and device, electronic equipment and storage medium | |
| US11740826B2 (en) | Policy-based hierarchical data protection in distributed storage | |
| EP3101530B1 (en) | Data distribution method, data storage method, related apparatus, and system | |
| WO2018001110A1 (en) | Method and device for reconstructing stored data based on erasure coding, and storage node | |
| CN110502365B (en) | Data storage and recovery method and device and computer equipment | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| US8464090B2 (en) | Recovery of failed disks in an array of disks | |
| US10048999B2 (en) | Method and apparatus for optimizing recovery of single-disk failure | |
| CN108153622B (en) | Fault processing method, device and equipment | |
| EP3537687B1 (en) | Access method for distributed storage system, related device and related system | |
| CN107846316B (en) | Cloud mobile phone management system and exception handling method thereof | |
| CN109582213B (en) | Data reconstruction method and device and data storage system | |
| US20200278913A1 (en) | Policy-based hierarchical data protection in distributed storage | |
| CN115994044A (en) | Database fault processing method and device based on monitoring service and distributed cluster | |
| CN109189327A (en) | The compression processing method and device of block chain data | |
| CN115981926B (en) | Method, device and equipment for improving disk array performance | |
| CN113051428B (en) | Method and device for back-up storage at front end of camera | |
| US10664190B1 (en) | Geographically dispersed data protection and replication | |
| CN102325171B (en) | Data storage method in monitoring system and system | |
| CN109426590A (en) | Method for the method for back end storing data and for restoring data | |
| CN104052799B (en) | A kind of method that High Availabitity storage is realized using resource ring | |
| CN115858250A (en) | Data recovery method, device, storage medium and electronic device | |
| CN115878381A (en) | Data recovery method and device based on SRM disc, storage medium and electronic device | |
| CN107154960B (en) | Method and apparatus for determining service availability information for distributed storage systems | |
| CN107885615B (en) | Distributed storage data recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |