CN114237520A - Ceph cluster data balancing method and system - Google Patents
Ceph cluster data balancing method and system Download PDFInfo
- Publication number
- CN114237520A CN114237520A CN202210184861.4A CN202210184861A CN114237520A CN 114237520 A CN114237520 A CN 114237520A CN 202210184861 A CN202210184861 A CN 202210184861A CN 114237520 A CN114237520 A CN 114237520A
- Authority
- CN
- China
- Prior art keywords
- data
- slave
- group
- homing
- data blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a ceph cluster data balancing method and a system, which belong to the technical field of network communication, the data balancing method averagely divides data blocks to be balanced through a main homing group to generate a plurality of balancing lists, then the main homing group reserves one balancing list, other balancing lists are respectively sent to a slave homing group, finally the main homing group and the slave homing group correspondingly send data blocks stored by the main homing group to a newly-added data storage device according to the names of the data blocks on the balancing lists to realize balancing, the plurality of homing groups are simultaneously matched with each other to execute balancing operation in unit time, and the efficiency of data balancing is greatly improved.
Description
Technical Field
The invention belongs to the technical field of network communication, and particularly relates to a ceph cluster data balancing method and system.
Background
In the concept of ceph distributed Storage system, one hard disk corresponds to one Object-Based Storage Device (OSD), and the OSD divides the Storage space of the hard disk into one small data block, which is called Object. An OSD contains thousands of objects, and in order to manage these objects, ceph uses a Place Group (PG) to divide the objects under a PG by a stable algorithm. ceph adopts three copies by default to ensure data reliability, namely, one object exists in three OSD at the same time, the three OSD are divided into a main OSD and two slave OSD, the main OSD corresponds to the main PG, and the slave OSD corresponds to the slave PG. When a new hard disk is added to the ceph cluster, the data of the original OSD needs to be balanced into the new OSD. Since the minimum unit of ceph data equalization is PG, that is, after a new hard disk is added, a part of PG of the original OSD needs to be equalized into the new OSD. At present, only the main PG initiates data equalization, the equalization speed is low, the equalization efficiency is low, the data equalization is a time-consuming process, and the read-write request of a user end cannot be received in the equalization process, so that the request waiting time of the user end is long, and the user experience is poor.
Disclosure of Invention
Objects of the invention
In order to overcome the above disadvantages, the present invention aims to provide a ceph cluster data balancing method and system, so as to solve the technical problems that the existing ceph cluster data balancing consumes a long time, and if a client sends a write request during the balancing period, a request reply can be obtained only by waiting for a long time, and the user experience effect is poor.
(II) technical scheme
In order to achieve the above object, one aspect of the present application provides the following technical solutions:
a ceph cluster data balancing method is applied to a ceph distributed storage system and comprises the following steps:
the main homing group counts the total number of all data blocks to be balanced, averagely divides all the counted data blocks into preset equal parts after counting is completed, respectively generates a balance list based on each preset equal part, and determines the content of each balance list according to the name of the data block in the corresponding preset equal part;
wherein, the preset equal part is the sum of the number of the main homing group and the plurality of the slave homing groups;
the master homing group keeps a balance list and distributes the rest balance lists to different slave homing groups respectively;
and the main homing group and the slave homing group balance the corresponding data blocks stored in the self into the newly added data storage device according to the data block names on the self balance list.
According to the data block balancing method and device, data blocks to be balanced are divided averagely through the main homing group to generate a plurality of balancing lists, then the main homing group keeps one balancing list, other balancing lists are sent to the auxiliary homing groups respectively, and finally the data blocks stored by the main homing group and the auxiliary homing groups are correspondingly sent to the newly-added data storage device according to the names of the data blocks on the balancing lists to achieve balancing, the multiple homing groups are matched with one another to perform balancing operation at the same time in unit time, and the data balancing efficiency is greatly improved.
Further, after counting the total number of all the data blocks to be equalized and before averagely dividing all the counted data blocks into predetermined equal parts, the method further comprises the following steps: and marking the data block which is requested to be read and written by the client as an emergency data block, and marking the rest data blocks which are not requested to be read and written by the client as normal data blocks.
Generating a balance list based on each predetermined equal part further comprises: and judging whether each balance list contains the name of the emergency data block, and if so, placing the name of the emergency data block in each balance list before the name of the common data block.
Because the data blocks can not be read and written in the balancing process, the requested data blocks and the data blocks which are not requested are distinguished, the requested data blocks are marked as emergency data blocks, the data blocks which are not requested are marked as common data blocks, the names of the emergency data blocks are also placed in front of the common data blocks when a balancing list is generated, so that the emergency data blocks can be preferentially balanced, the read-write request initiated by a user side can be responded immediately after balancing is completed, the waiting time of the request of the user side is shortened, the use experience of a user is improved, the user side write-in request is received after the data blocks are balanced, and the data consistency can be ensured.
Further, after the master-home group and the slave-home group balance the corresponding data blocks stored in the master-home group and the slave-home group to the newly added data storage device according to the data block names on the self balance list, the method further comprises the following steps: deleting a predetermined number of slave-homing groups may free up storage space of the data storage device.
Further, after the master-home group and the slave-home group balance the corresponding data blocks stored in the master-home group and the slave-home group to the newly added data storage device according to the data block names on the self balance list, the method further comprises the following steps: and the main homing group checks whether all data blocks in the newly added data storage device are the same as the counted number of the data blocks to be balanced, and if the data blocks are detected to be lacked, the main homing group supplements and balances the lacked data blocks to the newly added data storage device.
After the balance is completed, the main reset group also checks whether the data blocks in the new data storage device are consistent with the data blocks counted before, and if the data blocks are detected to be lacking, the lacking data blocks are supplemented and balanced to the newly added data storage device, so that the balance integrity and accuracy are ensured.
Another aspect of the present invention provides a ceph cluster data balancing system, configured to implement the above ceph cluster data balancing method, where the balancing system includes: the data storage system comprises a main data storage device and a plurality of slave data storage devices, wherein a main homing group is arranged in the main data storage device, each slave data storage device is provided with a slave homing group, and the main homing group and the plurality of slave homing groups are mutually communicated and are internally and synchronously stored with the same data block;
wherein, the main homing group comprises:
the statistical module is used for counting the total number of all data blocks to be balanced;
the dividing module is used for averagely dividing all the counted data blocks in the main grouping into preset equal parts after the counting module completes counting;
the balance list generating module is used for respectively generating a balance list based on each preset equal part and determining the content of each balance list according to the name of the data block in the corresponding preset equal part, wherein the preset equal part is the sum of the number of the main homing group and the number of the plurality of the auxiliary homing groups;
the distribution module is used for distributing the rest balance lists to different slave homing groups respectively after reserving one balance list to the master homing group;
the first balancing module balances the corresponding data blocks stored in the main homing group into a newly added data storage device according to the data block names on the balancing list reserved by the main homing group;
the slave homing group includes:
the receiving module is used for receiving the balance list sent by the distributing module;
and the second equalizing module equalizes the corresponding data blocks stored in the slave-group into the newly-added data storage device according to the data block names on the equalizing list received from the slave-group.
Further, the main homing group also comprises: the marking module is used for marking the data block which is requested to be read and written by the client as an emergency data block and marking the rest data blocks which are not requested to be read and written by the client as common data blocks;
the balance list generating module comprises: the device comprises a judgment submodule and a position adjustment submodule, wherein the judgment submodule is used for judging whether each balance list contains the name of an emergency data block, and the position adjustment submodule is used for placing the name of the emergency data block in front of the name of the common data block when the judgment submodule judges that the name of the emergency data block exists in the balance list.
Further, still include: and the deleting module is used for deleting the preset number of slave homing groups after the first equalizing module equalizes the data blocks stored in the master homing group into the newly added data storage device and the second equalizing module equalizes the data blocks stored in the slave homing group into the newly added data storage device. Further, the master homing group further comprises:
the checking module is used for checking whether all data blocks in the newly added data storage device are the same as the counted number of the data blocks to be balanced or not after the newly added data storage device completes the balancing;
and the supplementary balancing module is used for supplementary balancing the missing data blocks into the newly added data storage device after the checking module checks that the newly added data storage device lacks a certain data block.
Drawings
Fig. 1 is a working schematic diagram of the ceph cluster data balancing method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings in conjunction with the following detailed description. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of existing structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
The invention provides a ceph cluster data balancing method, which is applied to a ceph distributed storage system, and specifically, the ceph distributed storage system comprises the following steps: a master data storage device (master OSD) and a plurality of slave data storage devices (slave OSD), wherein, a master homing group (master PG) is arranged in the master data storage device (master OSD), a slave homing group (slave PG) is arranged in each slave data storage device (slave OSD), the master homing group (master PG) and the plurality of slave homing groups (slave PGs) are mutually communicated and connected, the master homing group (master PG) and the slave homing group (slave PG) synchronously store the same data block (object), when in storage, the client initiates a write request to the data block (object) of the master PG, and after the master homing group (master PG) receives the write request, and writing data and forming data blocks (objects), and synchronously storing the written data blocks (objects) into a slave set (slave PG) by a master set (master PG) to form copy backup data blocks (objects), wherein the number and the type of the data blocks (objects) in the master set (master PG) and the slave set (slave PG) are the same. When a new storage device is added to the ceph cluster, some data blocks (objects) to be balanced need to be copied to the new storage device by the original storage device, which is a balancing process, and some data blocks (objects) do not need to be copied and balanced.
Specifically, the equalization method comprises the following steps:
the method comprises the following steps: the method comprises the steps that a main homing group (main PG) counts the total number of all data blocks (objects) to be balanced in the main homing group (main PG), after counting is completed, all the counted data blocks (objects) are averagely divided into preset equal parts, the divided equal parts need to be divided according to the total number of the main homing group (main PG) and a secondary homing group (secondary PG), finally, a balance list is respectively generated based on each preset equal part, and the content of each balance list is determined according to the name of the data block in the corresponding preset equal part;
step two: the master homing group (master PG) keeps a balance list and distributes the rest balance lists to different slave homing groups (slave PGs) respectively;
step three: the master-homing group (master PG) and the slave-homing group (slave PG) balance the corresponding data blocks (objects) stored in the master-homing group and the slave-homing group into the newly added data storage device according to the data block names on the self balance list.
Specifically, since the types and the number of the data blocks (objects) in the master home group (master PG) and the plurality of slave home groups (slave PGs) are the same, the generated balance lists can be randomly reserved by one and then the rest balance lists can be randomly distributed to different slave home groups (slave PGs).
Referring to fig. 1, describing the above steps in an embodiment, the ceph distributed storage system includes 3 storage devices, a main storage device (main OSD, not shown by default) and a slave storage device 1 (slave OSD1, not shown by default) and a slave storage device 2 (slave OSD2, not shown by default), the main storage device (main OSD, not shown by default) has a main homing group (main PG) therein, the slave storage device 1 has a slave homing group (slave PG1) therein, and the slave storage device 2 has a slave homing group (slave PG2) therein;
the method comprises the following steps: the main group (main PG) counts the total number of all data blocks (object) to be equalized in the main group (main PG), 9 data blocks (object 1, object2.. objection 9) are totally obtained, after the statistics is completed, all the data blocks are averagely divided into 3 equal parts, and 3 equalization lists (a first equalization list, a second equalization list and a third equalization list) are respectively generated, wherein the first equalization list comprises data block object1, object2 and object3, the second equalization list comprises data block object4, object5 and object6, and the third equalization list comprises data block object7, object8 and object 9;
step two: the master-homing group (master PG) keeps the first equalization list, distributes the second equalization list and the third equalization list to the slave-homing group 1 (slave PG1) and the homing group 2 (slave PG 2);
step three: the master-homing group (master PG), the slave-homing group 1 (slave PG1), and the slave-homing group 2 (slave PG2) balance the corresponding data blocks (objects) stored in themselves into the new PG of the newly added data storage device according to the data block names on the own balance list.
Specifically, since the types and the numbers of the data blocks in the master-home group (master PG) and the plurality of slave-home groups (slave PGs) are the same, the generated balance lists can be randomly reserved and then randomly distributed to different slave-home groups (slave PGs).
Specifically, in the balancing process, if a user terminal sends a read-write request to a certain data block (object), the master-homing group (master PG) cannot reply to the request of the user terminal, and needs to perform data balancing waiting, and only after the data block (object) is balanced, the master-homing group (master PG) can reply to the user terminal to the data block, so that the read-write request is synchronized to the slave-homing group and also needs to perform data balancing waiting.
Preferably, each time after counting the total number of all the data blocks (objects) to be equalized and before averagely dividing all the data blocks (objects) into predetermined equal parts, the main homing group (main PG) determines whether some data blocks (objects) are sent by the user side or not, determines which data block (object) is pointed in the read-write request if the read-write request is received, then marks the requested data block (object) as an urgent data block, and marks the data block (object) which is not requested as a normal data block.
When a balance list is generated, if an emergency data block and a common data block exist in the balance list, the name of the emergency data block is placed in front of the name of the common data block on the balance list, so that each homing group reads information from the head to the tail end during balance, the emergency data block can be balanced preferentially, and after the balance is completed, the data of a main homing group (main PG) is replied to a user side, so that the waiting time of the user side is reduced, and the use experience of the user is improved.
The newly added data storage device can be used as a new main data storage device (main OSD) and can also be used as a slave data storage device (slave OSD) of the original main data storage device.
Specifically, after the data blocks (objects) on the equalization list are equalized to the newly added data storage device by both the master-home group (master PG) and the slave-home group (slave PG), a predetermined number of slave-home groups (slave PGs) are deleted to release the storage space of the slave data storage device (slave OSD). Specifically, in the above-described embodiment, the slave-homing group 1 is deleted, and the slave-homing group 2 is retained. Specifically, since the slave-homing group 1 (slave PG1) is to be deleted, there is no need to synchronize the data of the slave-homing group 1 when reading the read-write request of the user side at the later stage.
Preferably, after the equalization is completed, the main homing group (main PG) further needs to check whether all data blocks (object) in the newly added data storage device are the same as the counted number of data blocks (object) to be equalized, and if a certain data block (object) is detected to be missing, the main homing group (main PG) complementarily equalizes the missing data block (object) in the newly added data storage device to ensure the consistency of the data.
Another aspect of the present invention provides a ceph cluster data balancing system, configured to implement the above ceph cluster data balancing method, where the balancing system includes: a master data storage device (master OSD) and a plurality of slave data storage devices (slave OSD), wherein the master data storage device (master OSD) is internally provided with a master homing group (master PG), each slave data storage device (slave OSD) is provided with a slave homing group (slave PG), and the master homing group (master PG) and the plurality of slave homing groups (slave PGs) are mutually communicated and internally store the same data block (object) synchronously;
wherein the master homing group (master PG) comprises:
the statistical module is used for counting the total number of all data blocks (objects) to be balanced;
the dividing module is used for averagely dividing all the data blocks (objects) to be balanced in the main group (main PG) into preset equal parts after the counting module completes counting;
the equalization list generation module is used for respectively generating an equalization list based on each preset equal part and determining the content of each equalization list according to the data block name in the corresponding preset equal part, wherein the preset equal part is the sum of the number of a main homing group (main PG) and a plurality of slave homing groups (slave PGs);
the distribution module is used for distributing the rest balance lists to different slave homing groups (slave PGs) respectively after reserving one balance list to the master homing group (master PG);
the first balancing module balances the corresponding data blocks stored in the main homing group into a newly added data storage device according to the data block names on the balancing list reserved by the main homing group;
the slave homing group (slave PG) includes:
the receiving module is used for receiving the balance list sent by the distributing module;
and the second equalizing module equalizes the corresponding data blocks (objects) stored in the slave-set group into the newly-added data storage device according to the data block names on the equalizing list received from the slave-set group.
Further, the main homing group (main PG) further includes: the marking module is used for marking the data block (object) which is requested to be read and written by the client as an emergency data block and marking the rest data blocks which are not requested to be read and written by the client as normal data blocks;
the balance list generating module comprises: the device comprises a judgment submodule and a position adjustment submodule, wherein the judgment submodule is used for judging whether each balance list contains the name of an emergency data block, and the position adjustment submodule is used for placing the name of the emergency data block in front of the name of the common data block when the judgment submodule judges that the name of the emergency data block exists in the balance list.
Further, still include: a deletion module for deleting a predetermined number of slave home groups (slave PGs) after the master home group (master PG) and the slave home group (slave PG) balance the data blocks (objects) stored therein into the newly added data storage.
Further, the master homing group (master PG) further includes:
the checking module is used for checking whether all the data blocks (objects) in the newly added data storage device are the same as the counted number of the data blocks (objects) to be balanced or not after the newly added data storage device completes the balancing;
and the supplementary balancing module is used for supplementing and balancing the missing data blocks (objects) to the newly added data storage device after the checking module checks that the newly added data storage device lacks certain data blocks (objects).
It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explaining the principles of the invention and are not to be construed as limiting the invention. Therefore, any modification, equivalent replacement, improvement and the like made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. Further, it is intended that the appended claims cover all such variations and modifications as fall within the scope and boundaries of the appended claims or the equivalents of such scope and boundaries.
Claims (8)
1. A ceph cluster data balancing method is applied to a ceph distributed storage system, and comprises the following steps:
the main homing group counts the total number of all data blocks to be balanced, averagely divides all the counted data blocks into preset equal parts after counting is completed, respectively generates a balance list based on each preset equal part, and determines the content of each balance list according to the name of the data block in the corresponding preset equal part;
wherein the preset equal part is the sum of the number of the main homing group and the number of the plurality of the slave homing groups;
the master homing group keeps a balance list and distributes the rest balance lists to different slave homing groups respectively;
and the main homing group and the slave homing group balance the corresponding data blocks stored in the self into the newly added data storage device according to the data block names on the self balance list.
2. The ceph cluster data equalization method of claim 1, further comprising, after counting a total number of all data blocks to be equalized and before equally dividing all counted data blocks into predetermined equal portions: marking the data block which is requested to be read and written by the client as an emergency data block, and marking the rest data blocks which are not requested to be read and written by the client as common data blocks;
the generating a balance list based on each predetermined equal part further comprises: and judging whether each balance list contains the name of the emergency data block, and if so, placing the name of the emergency data block in each balance list before the name of the common data block.
3. The ceph cluster data balancing method according to claim 1 or 2, wherein after the master home group and the slave home group balance the corresponding data blocks stored in the master home group and the slave home group to the newly added data storage device according to the data block names on the self balancing list, the method further comprises: a predetermined number of slave homing groups are deleted.
4. The ceph cluster data balancing method according to claim 1 or 2, wherein after the master home group and the slave home group balance the corresponding data blocks stored in the master home group and the slave home group to the newly added data storage device according to the data block names on the self balancing list, the method further comprises: and the main homing group checks whether all data blocks in the newly added data storage device are the same as the counted number of the data blocks to be balanced, and if the data blocks are detected to be lacked, the main homing group supplements and balances the lacked data blocks to the newly added data storage device.
5. A ceph cluster data balancing system for implementing the ceph cluster data balancing method according to any one of claims 1 to 4, the balancing system comprising: the data storage system comprises a main data storage device and a plurality of slave data storage devices, wherein a main homing group is arranged in the main data storage device, each slave data storage device is provided with a slave homing group, and the main homing group and the plurality of slave homing groups are mutually communicated and are internally and synchronously stored with the same data block;
wherein the master homing group comprises:
the statistical module is used for counting the total number of all data blocks to be balanced;
the dividing module is used for averagely dividing all the counted data blocks in the main grouping into preset equal parts after the counting module completes counting;
the equalizing list generating module is used for respectively generating an equalizing list based on each preset equal part and determining the content of each equalizing list according to the name of the data block in the corresponding preset equal part, wherein the preset equal part is the sum of the number of the main homing group and the number of the plurality of the auxiliary homing groups;
the distribution module is used for distributing the rest balance lists to different slave homing groups respectively after reserving one balance list to the master homing group;
the first balancing module balances the corresponding data blocks stored in the main homing group into a newly added data storage device according to the data block names on the balancing list reserved by the main homing group;
the slave homing group comprises:
the receiving module is used for receiving the balance list sent by the distributing module;
and the second equalizing module equalizes the corresponding data blocks stored in the slave-group into the newly-added data storage device according to the data block names on the equalizing list received from the slave-group.
6. The ceph cluster data equalization system of claim 5, further comprising within a master homing group: the marking module is used for marking the data block which is requested to be read and written by the client as an emergency data block and marking the rest data blocks which are not requested to be read and written by the client as common data blocks;
the balance list generating module comprises: the device comprises a judgment submodule and a position adjustment submodule, wherein the judgment submodule is used for judging whether each balance list contains the name of an emergency data block, and the position adjustment submodule is used for placing the name of the emergency data block in front of the name of a common data block when the judgment submodule judges that the name of the emergency data block exists in the balance list.
7. The ceph cluster data balancing system according to claim 5 or 6, further comprising: and the deleting module is used for deleting the preset number of slave homing groups after the first equalizing module equalizes the data blocks stored in the master homing group into the newly added data storage device and the second equalizing module equalizes the data blocks stored in the slave homing group into the newly added data storage device.
8. The ceph cluster data balancing system according to claim 5 or 6, wherein the master-homing group further comprises:
the checking module is used for checking whether all data blocks in the newly added data storage device are the same as the counted number of the data blocks to be balanced or not after the newly added data storage device completes the balancing;
and the supplementary balancing module is used for supplementary balancing the missing data blocks into the newly added data storage device after the checking module checks that the newly added data storage device lacks a certain data block.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210184861.4A CN114237520B (en) | 2022-02-28 | 2022-02-28 | Ceph cluster data balancing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210184861.4A CN114237520B (en) | 2022-02-28 | 2022-02-28 | Ceph cluster data balancing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114237520A true CN114237520A (en) | 2022-03-25 |
| CN114237520B CN114237520B (en) | 2022-07-08 |
Family
ID=80748267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210184861.4A Active CN114237520B (en) | 2022-02-28 | 2022-02-28 | Ceph cluster data balancing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114237520B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160378364A1 (en) * | 2015-06-23 | 2016-12-29 | Western Digital Technologies, Inc. | Data management for object based storage |
| CN106844399A (en) * | 2015-12-07 | 2017-06-13 | 中兴通讯股份有限公司 | Distributed data base system and its adaptive approach |
| CN106951559A (en) * | 2017-03-31 | 2017-07-14 | 联想(北京)有限公司 | Data reconstruction method and electronic equipment in distributed file system |
| CN106951445A (en) * | 2017-02-17 | 2017-07-14 | 深圳市中博睿存信息技术有限公司 | A kind of distributed file system and its memory node loading method |
| CN107544848A (en) * | 2017-08-30 | 2018-01-05 | 深圳云天励飞技术有限公司 | Cluster expansion method, apparatus, electronic equipment and storage medium |
| CN108509157A (en) * | 2018-04-13 | 2018-09-07 | 郑州云海信息技术有限公司 | A kind of data balancing method and device applied to distributed file system |
| CN109558437A (en) * | 2018-11-16 | 2019-04-02 | 新华三技术有限公司成都分公司 | Main OSD method of adjustment and device |
| CN109710456A (en) * | 2018-12-10 | 2019-05-03 | 新华三技术有限公司 | A kind of data reconstruction method and device |
| CN109992206A (en) * | 2019-03-27 | 2019-07-09 | 新华三技术有限公司成都分公司 | Distributed data storage method and relevant apparatus |
| CN112817535A (en) * | 2021-02-03 | 2021-05-18 | 柏科数据技术（深圳）股份有限公司 | Method and device for distributing homing groups and distributed storage system |
| CN113271323A (en) * | 2020-02-14 | 2021-08-17 | 中移(苏州)软件技术有限公司 | Cluster capacity expansion method and device and storage medium |
-
2022
- 2022-02-28 CN CN202210184861.4A patent/CN114237520B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160378364A1 (en) * | 2015-06-23 | 2016-12-29 | Western Digital Technologies, Inc. | Data management for object based storage |
| CN106844399A (en) * | 2015-12-07 | 2017-06-13 | 中兴通讯股份有限公司 | Distributed data base system and its adaptive approach |
| CN106951445A (en) * | 2017-02-17 | 2017-07-14 | 深圳市中博睿存信息技术有限公司 | A kind of distributed file system and its memory node loading method |
| CN106951559A (en) * | 2017-03-31 | 2017-07-14 | 联想(北京)有限公司 | Data reconstruction method and electronic equipment in distributed file system |
| CN107544848A (en) * | 2017-08-30 | 2018-01-05 | 深圳云天励飞技术有限公司 | Cluster expansion method, apparatus, electronic equipment and storage medium |
| CN108509157A (en) * | 2018-04-13 | 2018-09-07 | 郑州云海信息技术有限公司 | A kind of data balancing method and device applied to distributed file system |
| CN109558437A (en) * | 2018-11-16 | 2019-04-02 | 新华三技术有限公司成都分公司 | Main OSD method of adjustment and device |
| CN109710456A (en) * | 2018-12-10 | 2019-05-03 | 新华三技术有限公司 | A kind of data reconstruction method and device |
| CN109992206A (en) * | 2019-03-27 | 2019-07-09 | 新华三技术有限公司成都分公司 | Distributed data storage method and relevant apparatus |
| CN113271323A (en) * | 2020-02-14 | 2021-08-17 | 中移(苏州)软件技术有限公司 | Cluster capacity expansion method and device and storage medium |
| CN112817535A (en) * | 2021-02-03 | 2021-05-18 | 柏科数据技术（深圳）股份有限公司 | Method and device for distributing homing groups and distributed storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114237520B (en) | 2022-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| US7239605B2 (en) | Item and method for performing a cluster topology self-healing process in a distributed data system cluster | |
| US20050071393A1 (en) | Data storage subsystem | |
| CN109376197B (en) | Data synchronization method, server and computer storage medium | |
| CN106843745A (en) | Capacity expansion method and device | |
| CN111880751B (en) | Hard disk migration method, distributed storage cluster system and storage medium | |
| CN102843403A (en) | File processing method based on distributed file system, system, and client | |
| CN104216835B (en) | A kind of method and device for realizing internal memory fusion | |
| CN103761162A (en) | Data backup method of distributed file system | |
| CN111125049B (en) | RDMA and nonvolatile memory-based distributed file data block read-write method and system | |
| CN105740248B (en) | A kind of method of data synchronization, apparatus and system | |
| CN104346373A (en) | Partition log queue synchronization management method and device | |
| EP2954424A1 (en) | Method, device, and system for peer-to-peer data replication and method, device, and system for master node switching | |
| CN101834904A (en) | Method and equipment for database backup | |
| CN110531936A (en) | The crop type storage organization and method of distributed correcting and eleting codes mixing storage based on multi storage | |
| CN102957546A (en) | Method, device and system for processing configuration | |
| CN109597903A (en) | Image file processing apparatus and method, document storage system and storage medium | |
| CN109815207A (en) | Date storage method and Client Agent | |
| CN114237520B (en) | Ceph cluster data balancing method and system | |
| CN111435286A (en) | Data storage method, device and system | |
| CN112653746B (en) | Distributed storage method and system for concurrently creating object storage equipment | |
| CN111737226B (en) | Method for optimizing HBase cluster performance based on Redis cluster | |
| CN111459416B (en) | Distributed storage-based thermal migration system and migration method thereof | |
| TWI789346B (en) | Message sending method and terminal equipment | |
| CN106487702B (en) | Multicast message two-stage copying method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |