CN113687798A - Method, device and equipment for controlling data reconstruction and readable medium - Google Patents
Method, device and equipment for controlling data reconstruction and readable medium Download PDFInfo
- Publication number
- CN113687798A CN113687798A CN202111250139.8A CN202111250139A CN113687798A CN 113687798 A CN113687798 A CN 113687798A CN 202111250139 A CN202111250139 A CN 202111250139A CN 113687798 A CN113687798 A CN 113687798A
- Authority
- CN
- China
- Prior art keywords
- read
- data
- data reconstruction
- cluster
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000004044 response Effects 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000002159 abnormal effect Effects 0.000 claims abstract description 18
- 238000004590 computer program Methods 0.000 claims description 11
- 238000011084 recovery Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0632—Configuration or reconfiguration of storage systems by initialisation or re-initialisation of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a device, equipment and a readable medium for controlling data reconstruction, wherein the method comprises the following steps: in response to receiving the data reconstruction instruction, calculating a data reconstruction amount and the remaining available capacity in the cluster, and comparing the data reconstruction amount with the remaining available capacity in the cluster; responding to the fact that the remaining available capacity in the cluster is larger than the data reconstruction amount, marking the OSD with the fault as an abnormal state and judging whether the read-write operation of the client exists or not; in response to the existence of the read-write operation of the client, limiting the processing speed of the read-write operation of the client and adjusting the priority of the read-write of the cluster; and reconstructing data, and recovering the processing speed of the read-write operation of the client after the data reconstruction is finished. By using the scheme of the invention, the problem that the reconstruction cannot be completed due to insufficient residual space and the data is lost can be avoided, and the problem that the reconstruction cannot be completed due to insufficient residual space caused by new data writing in the reconstruction process can be avoided.
Description
Technical Field
The present invention relates to the field of computers, and more particularly, to a method, apparatus, device, and readable medium for controlling data reconstruction.
Background
When the distributed storage cluster has faults such as disk failure, power failure and the like, or operations such as capacity expansion, capacity reduction and the like, data reconstruction is carried out. After the PG (place Group, a carrier for placing objects) completes the Peering (process in which copies in the PG achieve data consistency through PG logs), the PG in the Active state can provide services to the outside. However, if there are inconsistent objects on each copy of the PG, repair, i.e., data reconstruction, is required. Data reconstruction is a basic and very important function of distributed storage that guarantees data integrity and consistency.
However, if the cluster is very high when data reconstruction occurs, problems may occur. If the remaining available space of the cluster is smaller than the size of the data volume on the failed disk, that is, the remaining space is insufficient to complete data reconstruction, data loss may occur, if the result of a calculated hash algorithm (a hash algorithm for calculating data distribution) after reconstruction is not very balanced, reconstruction may not be completed under the condition that the available space is very limited, and then a risk of data loss may occur, if data reconstruction is performed, service reading and writing of a client also exist, and thus, a new data volume generated by a service may occupy a certain space and cause insufficient space for data reconstruction.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a device, and a readable medium for controlling data reconstruction, which can avoid the problem that data is lost due to incomplete reconstruction caused by insufficient remaining space, and can avoid the problem that the incomplete reconstruction cannot be completed due to insufficient remaining space caused by new data writing during the reconstruction process.
In view of the above object, an aspect of embodiments of the present invention provides a method of controlling data reconstruction, including the steps of:
in response to receiving the data reconstruction instruction, calculating a data reconstruction amount and the remaining available capacity in the cluster, and comparing the data reconstruction amount with the remaining available capacity in the cluster;
responding to the fact that the remaining available capacity in the cluster is larger than the data reconstruction amount, marking the OSD with the fault as an abnormal state and judging whether the read-write operation of the client exists or not;
in response to the existence of the read-write operation of the client, limiting the processing speed of the read-write operation of the client and adjusting the priority of the read-write of the cluster;
and reconstructing data, and recovering the processing speed of the read-write operation of the client after the data reconstruction is finished.
According to an embodiment of the present invention, further comprising:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
According to one embodiment of the invention, performing data reconstruction includes:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
According to one embodiment of the present invention, in response to the presence of the client read-write operation, limiting a processing speed of the client read-write operation and adjusting a priority of the cluster read-write comprises:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
In another aspect of the embodiments of the present invention, there is also provided an apparatus for controlling data reconstruction, the apparatus including:
the comparison module is configured to respond to the received data reconstruction instruction, calculate the data reconstruction amount and the residual available capacity in the cluster and compare the data reconstruction amount with the residual available capacity in the cluster;
the judging module is configured to respond that the remaining available capacity in the cluster is larger than the data reconstruction amount, mark the fault OSD as an abnormal state and judge whether the read-write operation of the client exists or not;
the adjusting module is configured to respond to the existence of the read-write operation of the client, limit the processing speed of the read-write operation of the client and adjust the priority of the read-write of the cluster;
and the reconstruction module is configured to reconstruct the data and recover the processing speed of the read-write operation of the client after the data reconstruction is finished.
According to one embodiment of the invention, the system further comprises an alarm module configured to:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
According to one embodiment of the invention, the reconstruction module is further configured to:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
According to one embodiment of the invention, the adjustment module is further configured to:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
In another aspect of an embodiment of the present invention, there is also provided a computer apparatus including:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of any of the methods described above.
In another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program, which when executed by a processor implements the steps of any one of the above-mentioned methods.
The invention has the following beneficial technical effects: in the method for controlling data reconstruction provided by the embodiment of the invention, the data reconstruction amount and the residual available capacity in the cluster are calculated by responding to the received data reconstruction instruction, and the data reconstruction amount is compared with the residual available capacity in the cluster; responding to the fact that the remaining available capacity in the cluster is larger than the data reconstruction amount, marking the OSD with the fault as an abnormal state and judging whether the read-write operation of the client exists or not; in response to the existence of the read-write operation of the client, limiting the processing speed of the read-write operation of the client and adjusting the priority of the read-write of the cluster; the technical scheme of reconstructing the data and recovering the processing speed of the read-write operation of the client after the data reconstruction is completed can avoid the problem that the reconstruction cannot be completed due to insufficient residual space, and the problem that the reconstruction cannot be completed due to insufficient residual space caused by new data writing in the reconstruction process.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a method of controlling data reconstruction in accordance with one embodiment of the present invention;
FIG. 2 is a schematic diagram of an apparatus for controlling data reconstruction according to one embodiment of the present invention;
FIG. 3 is a schematic diagram of a computer device according to one embodiment of the present invention;
fig. 4 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
In view of the above objects, a first aspect of embodiments of the present invention proposes an embodiment of a method of controlling data reconstruction. Fig. 1 shows a schematic flow diagram of the method.
As shown in fig. 1, the method may include the steps of:
s1, in response to receiving the instruction for data reconstruction, calculates the amount of data reconstruction and the remaining available capacity in the cluster, and compares the amount of data reconstruction and the remaining available capacity in the cluster.
Before starting data reconstruction, calculating the data volume to be reconstructed and the current residual available capacity of the cluster, then carrying out different processing according to the comparison result of the reconstructed data volume and the residual capacity, and if the residual capacity of the cluster is not enough to complete the reconstruction of the data volume on the fault disk, not carrying out the operation marked as out state on the fault OSD, and further not triggering the reconstruction.
S2 responding to the surplus available capacity in the cluster is larger than the data reconstruction amount, the fault OSD is marked as an abnormal state and whether the read-write operation of the client exists is judged.
If the residual capacity of the cluster is enough to complete reconstruction of the data volume on the fault disk, the OSD on the fault disk is marked as an out state, then different processing is carried out according to different conditions of whether client service reading and writing exist in the current cluster, and if the client does not have the service reading and writing, data reconstruction can be directly carried out.
S3 responding to the client read-write operation, limiting the processing speed of the client read-write operation and adjusting the priority of the cluster read-write operation.
If the client service read-write operation exists, the processing speed of the client service read-write operation is limited, the priority of cluster read-write is adjusted to be recovery priority, namely, the operation generated by data reconstruction is processed preferentially, and then the operation of the client service read-write operation is processed, so that the problem that the reconstruction cannot be completed due to insufficient space required by reconstruction caused by large data volume written by the service can be solved.
And S4, reconstructing data, and recovering the processing speed of the read-write operation of the client after the data reconstruction is completed.
When data reconstruction is carried out, the PG carries out data rebalancing through the Upmap algorithm after the mapping of the Crush algorithm, so that the quantity of the reconstructed PG is ensured to be balanced as much as possible, namely the reconstructed PG is preferentially distributed on the OSD with less quantity of PG, and the problem that the reconstruction cannot be completed due to the unbalanced PG distribution can be avoided.
By the technical scheme, the problem that reconstruction cannot be completed due to insufficient residual space and data loss is caused can be solved, and the problem that reconstruction cannot be completed due to insufficient residual space caused by new data writing in the reconstruction process can be solved.
In a preferred embodiment of the present invention, the method further comprises:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm. And if the residual capacity of the cluster is not enough to complete reconstruction of the data volume on the fault disk, the operation marked as out state is not carried out on the fault OSD, further the reconstruction is not triggered, and the alarm of the capacity insufficiency is sent to a display device of the cluster or is sent to mobile equipment of staff through a network.
In a preferred embodiment of the present invention, performing data reconstruction comprises:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
In a preferred embodiment of the present invention, in response to the existence of the client read-write operation, limiting a processing speed of the client read-write operation and adjusting a priority of the cluster read-write operation includes:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority. In some embodiments, if there is a client read-write operation, all the read-write operations of the client are suspended, and a request of the read-write operation is put into a read-write queue to wait for data reconstruction and then be recovered.
The prior art strategy does not limit and control the data reconstruction, and may bring the risk of data loss when the remaining available space of the cluster is not enough to complete the reconstruction. Before data reconstruction is carried out, the reconstruction data volume and the size of the cluster remaining available space are calculated and compared, so that the problem that data is lost due to the fact that reconstruction cannot be completed due to the fact that the remaining space is insufficient when reconstruction starts is solved, and then when the cluster remaining capacity is larger than the data reconstruction volume, the strategy of limiting the data writing volume of a client side by setting the reconstruction speed as data recovery priority is adopted, so that the problem that reconstruction cannot be completed due to the fact that the remaining space is insufficient due to new data writing in the reconstruction process is solved. Meanwhile, in the reconstruction process, the distribution of PG on each OSD is more balanced by the strategy of rebalancing through the Upmap algorithm after the Crush algorithm is mapped, and the risk that reconstruction cannot be completed is further reduced.
It should be noted that, as will be understood by those skilled in the art, all or part of the processes in the methods of the above embodiments may be implemented by instructing relevant hardware through a computer program, and the above programs may be stored in a computer-readable storage medium, and when executed, the programs may include the processes of the embodiments of the methods as described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Furthermore, the method disclosed according to an embodiment of the present invention may also be implemented as a computer program executed by a CPU, and the computer program may be stored in a computer-readable storage medium. The computer program, when executed by the CPU, performs the above-described functions defined in the method disclosed in the embodiments of the present invention.
In view of the above object, according to a second aspect of the embodiments of the present invention, there is provided an apparatus for controlling data reconstruction, as shown in fig. 2, the apparatus 200 includes:
the comparison module 201, the comparison module 201 is configured to respond to the received data reconstruction instruction, calculate the data reconstruction amount and the remaining available capacity in the cluster, and compare the data reconstruction amount and the remaining available capacity in the cluster;
the judging module 202, where the judging module 202 is configured to mark a faulty OSD as an abnormal state and judge whether there is a client read-write operation in response to that the remaining available capacity in the cluster is greater than the data reconstruction amount;
the adjusting module 203, the adjusting module 203 is configured to respond to the existence of the client read-write operation, limit the processing speed of the client read-write operation and adjust the priority of the cluster read-write;
and the reconstruction module 204, the reconstruction module 204 is configured to perform data reconstruction, and recover the processing speed of the read-write operation of the client after the data reconstruction is completed.
In a preferred embodiment of the present invention, the system further comprises an alarm module, wherein the alarm module is configured to:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
In a preferred embodiment of the present invention, the reconstruction module 204 is further configured to:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
In a preferred embodiment of the present invention, the adjustment module is further configured to:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device. Fig. 3 is a schematic diagram of an embodiment of a computer device provided by the present invention. As shown in fig. 3, an embodiment of the present invention includes the following means: at least one processor 21; and a memory 22, the memory 22 storing computer instructions 23 executable on the processor, the instructions when executed by the processor implementing the method of:
in response to receiving the data reconstruction instruction, calculating a data reconstruction amount and the remaining available capacity in the cluster, and comparing the data reconstruction amount with the remaining available capacity in the cluster;
responding to the fact that the remaining available capacity in the cluster is larger than the data reconstruction amount, marking the OSD with the fault as an abnormal state and judging whether the read-write operation of the client exists or not;
in response to the existence of the read-write operation of the client, limiting the processing speed of the read-write operation of the client and adjusting the priority of the read-write of the cluster;
and reconstructing data, and recovering the processing speed of the read-write operation of the client after the data reconstruction is finished.
In a preferred embodiment of the present invention, the method further comprises:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
In a preferred embodiment of the present invention, performing data reconstruction comprises:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
In a preferred embodiment of the present invention, in response to the existence of the client read-write operation, limiting a processing speed of the client read-write operation and adjusting a priority of the cluster read-write operation includes:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
In view of the above object, a fourth aspect of the embodiments of the present invention proposes a computer-readable storage medium. FIG. 4 is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. As shown in fig. 4, the computer-readable storage medium 31 stores a computer program 32 that, when executed by a processor, performs the method of:
in response to receiving the data reconstruction instruction, calculating a data reconstruction amount and the remaining available capacity in the cluster, and comparing the data reconstruction amount with the remaining available capacity in the cluster;
responding to the fact that the remaining available capacity in the cluster is larger than the data reconstruction amount, marking the OSD with the fault as an abnormal state and judging whether the read-write operation of the client exists or not;
in response to the existence of the read-write operation of the client, limiting the processing speed of the read-write operation of the client and adjusting the priority of the read-write of the cluster;
and reconstructing data, and recovering the processing speed of the read-write operation of the client after the data reconstruction is finished.
In a preferred embodiment of the present invention, the method further comprises:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
In a preferred embodiment of the present invention, performing data reconstruction comprises:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
In a preferred embodiment of the present invention, in response to the existence of the client read-write operation, limiting a processing speed of the client read-write operation and adjusting a priority of the cluster read-write operation includes:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
Furthermore, the methods disclosed according to embodiments of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions defined in the methods disclosed in embodiments of the invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method of controlling data reconstruction, comprising the steps of:
in response to receiving the data reconstruction instruction, calculating a data reconstruction amount and the remaining available capacity in the cluster, and comparing the data reconstruction amount with the remaining available capacity in the cluster;
responding to the fact that the remaining available capacity in the cluster is larger than the data reconstruction amount, marking the OSD with the fault as an abnormal state and judging whether the read-write operation of the client exists or not;
in response to the existence of the read-write operation of the client, limiting the processing speed of the read-write operation of the client and adjusting the priority of the read-write of the cluster;
and reconstructing data, and recovering the processing speed of the read-write operation of the client after the data reconstruction is finished.
2. The method of claim 1, further comprising:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
3. The method of claim 1, wherein performing data reconstruction comprises:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
4. The method of claim 1, wherein limiting a processing speed of client read and write operations and adjusting a priority of cluster read and write in response to the presence of client read and write operations comprises:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
5. An apparatus for controlling data reconstruction, the apparatus comprising:
a comparison module configured to calculate a data reconstruction amount and a remaining available capacity in the cluster in response to receiving an instruction for data reconstruction, and compare the data reconstruction amount and the remaining available capacity in the cluster;
the judging module is configured to respond that the residual available capacity in the cluster is larger than the data reconstruction amount, mark the fault OSD as an abnormal state and judge whether the read-write operation of the client exists or not;
the adjusting module is configured to respond to the existence of the read-write operation of the client, limit the processing speed of the read-write operation of the client and adjust the priority of the read-write of the cluster;
and the reconstruction module is configured to reconstruct data and recover the processing speed of the read-write operation of the client after the data reconstruction is finished.
6. The apparatus of claim 5, further comprising an alert module configured to:
and in response to the fact that the remaining available capacity in the cluster is smaller than the data reconstruction amount, not performing the operation of marking the fault OSD as an abnormal state and sending out a capacity shortage alarm.
7. The apparatus of claim 5, wherein the reconstruction module is further configured to:
in response to the data reconstruction, the PG is rebalanced by the Upmap algorithm after being mapped by the Crush algorithm so that the reconstructed PG is preferentially distributed to the OSDs with the smaller PG number.
8. The apparatus of claim 5, wherein the adjustment module is further configured to:
and responding to the existence of client read-write operation, and setting the priority of the data read-write queue as data recovery priority.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 4.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111250139.8A CN113687798A (en) | 2021-10-26 | 2021-10-26 | Method, device and equipment for controlling data reconstruction and readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111250139.8A CN113687798A (en) | 2021-10-26 | 2021-10-26 | Method, device and equipment for controlling data reconstruction and readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113687798A true CN113687798A (en) | 2021-11-23 |
Family
ID=78588087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111250139.8A Pending CN113687798A (en) | 2021-10-26 | 2021-10-26 | Method, device and equipment for controlling data reconstruction and readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113687798A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114564303A (en) * | 2022-01-29 | 2022-05-31 | 苏州浪潮智能科技有限公司 | Method, system, device and storage medium for adaptive reconstruction speed control |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106471461A (en) * | 2014-06-04 | 2017-03-01 | 纯存储公司 | Automatically reconfigure storage device memorizer topology |
| CN106951559A (en) * | 2017-03-31 | 2017-07-14 | 联想(北京)有限公司 | Data reconstruction method and electronic equipment in distributed file system |
| CN107133228A (en) * | 2016-02-26 | 2017-09-05 | 华为技术有限公司 | A kind of method and device of fast resampling |
| US20180285219A1 (en) * | 2017-03-28 | 2018-10-04 | Amazon Technologies, Inc. | Adaptive data recovery for clustered data devices |
| CN109144787A (en) * | 2018-09-03 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of data reconstruction method, device, equipment and readable storage medium storing program for executing |
| CN110515771A (en) * | 2019-08-23 | 2019-11-29 | 北京浪潮数据技术有限公司 | A kind of object storage device setting method, system, equipment and computer media |
| CN110908606A (en) * | 2019-11-15 | 2020-03-24 | 浪潮电子信息产业股份有限公司 | Data reconstruction method of distributed file system |
| CN110955522A (en) * | 2019-11-12 | 2020-04-03 | 华中科技大学 | Resource management method and system for coordination performance isolation and data recovery optimization |
| CN111078119A (en) * | 2018-10-18 | 2020-04-28 | 深信服科技股份有限公司 | Data reconstruction method, system, device and computer readable storage medium |
| CN111708484A (en) * | 2020-05-22 | 2020-09-25 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for controlling data brushing speed |
| CN112181736A (en) * | 2020-09-23 | 2021-01-05 | 星辰天合（北京）数据科技有限公司 | Distributed storage system and configuration method thereof |
| CN112433881A (en) * | 2020-10-29 | 2021-03-02 | 苏州浪潮智能科技有限公司 | Data recovery method and device for distributed storage system |
| CN113220235A (en) * | 2021-05-17 | 2021-08-06 | 北京青云科技股份有限公司 | Read-write request processing method, device, equipment and storage medium |
-
2021
- 2021-10-26 CN CN202111250139.8A patent/CN113687798A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106471461A (en) * | 2014-06-04 | 2017-03-01 | 纯存储公司 | Automatically reconfigure storage device memorizer topology |
| CN107133228A (en) * | 2016-02-26 | 2017-09-05 | 华为技术有限公司 | A kind of method and device of fast resampling |
| US20180285219A1 (en) * | 2017-03-28 | 2018-10-04 | Amazon Technologies, Inc. | Adaptive data recovery for clustered data devices |
| CN106951559A (en) * | 2017-03-31 | 2017-07-14 | 联想(北京)有限公司 | Data reconstruction method and electronic equipment in distributed file system |
| CN109144787A (en) * | 2018-09-03 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of data reconstruction method, device, equipment and readable storage medium storing program for executing |
| CN111078119A (en) * | 2018-10-18 | 2020-04-28 | 深信服科技股份有限公司 | Data reconstruction method, system, device and computer readable storage medium |
| CN110515771A (en) * | 2019-08-23 | 2019-11-29 | 北京浪潮数据技术有限公司 | A kind of object storage device setting method, system, equipment and computer media |
| CN110955522A (en) * | 2019-11-12 | 2020-04-03 | 华中科技大学 | Resource management method and system for coordination performance isolation and data recovery optimization |
| CN110908606A (en) * | 2019-11-15 | 2020-03-24 | 浪潮电子信息产业股份有限公司 | Data reconstruction method of distributed file system |
| CN111708484A (en) * | 2020-05-22 | 2020-09-25 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for controlling data brushing speed |
| CN112181736A (en) * | 2020-09-23 | 2021-01-05 | 星辰天合（北京）数据科技有限公司 | Distributed storage system and configuration method thereof |
| CN112433881A (en) * | 2020-10-29 | 2021-03-02 | 苏州浪潮智能科技有限公司 | Data recovery method and device for distributed storage system |
| CN113220235A (en) * | 2021-05-17 | 2021-08-06 | 北京青云科技股份有限公司 | Read-write request processing method, device, equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114564303A (en) * | 2022-01-29 | 2022-05-31 | 苏州浪潮智能科技有限公司 | Method, system, device and storage medium for adaptive reconstruction speed control |
| CN114564303B (en) * | 2022-01-29 | 2024-06-07 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium for self-adaptive reconstruction speed control |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3617886B1 (en) | Hot backup system, hot backup method, and computer device | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| EP3537687B1 (en) | Access method for distributed storage system, related device and related system | |
| CN110046029A (en) | Data processing method and device applied to multi-level buffer in cluster | |
| CN106708653B (en) | Mixed tax big data security protection method based on erasure code and multiple copies | |
| CN106557266A (en) | For the method and apparatus of RAID RAID | |
| CN111400267A (en) | Method and device for recording log | |
| CN113821168A (en) | Shared storage migration system and method, electronic equipment and storage medium | |
| CN113687798A (en) | Method, device and equipment for controlling data reconstruction and readable medium | |
| CN113282250B (en) | Method, device and equipment for cloud volume expansion and readable medium | |
| CN113590049B (en) | Storage volume cross-node cloning method, device, equipment and readable medium | |
| CN115955488B (en) | Distributed storage copy cross-machine room placement method and device based on copy redundancy | |
| CN115826876B (en) | Data writing method, system, storage hard disk, electronic device and storage medium | |
| CN115202589B (en) | Placement group member selection method, device and equipment and readable storage medium | |
| CN113625948B (en) | Method, device and equipment for filling dummy into solid state disk and readable medium | |
| CN107154960B (en) | Method and apparatus for determining service availability information for distributed storage systems | |
| CN112783688A (en) | Erasure code data recovery method and device based on available partition level | |
| CN110389863A (en) | A kind of data reconstruction method, device, equipment and computer readable storage medium | |
| CN111679777B (en) | Panorama generating method and system and object model generating method and system | |
| CN104239182A (en) | Cluster file system split-brain processing method and device | |
| CN117851132B (en) | Data recovery optimization method for distributed object storage | |
| CN105022586A (en) | Method, device, and system for data processing | |
| CN113076228B (en) | Distributed system and management method and device thereof | |
| CN114138736A (en) | Method, device, equipment and readable medium for selecting members of distributed file system PG | |
| CN114860284A (en) | Multi-center application upgrading method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211123 |
|
| RJ01 | Rejection of invention patent application after publication |