CN116627341A - Distributed block storage system and bitmap operation method, device, equipment and medium thereof - Google Patents
Distributed block storage system and bitmap operation method, device, equipment and medium thereof Download PDFInfo
- Publication number
- CN116627341A CN116627341A CN202310638122.2A CN202310638122A CN116627341A CN 116627341 A CN116627341 A CN 116627341A CN 202310638122 A CN202310638122 A CN 202310638122A CN 116627341 A CN116627341 A CN 116627341A
- Authority
- CN
- China
- Prior art keywords
- bitmap
- request
- target
- storage system
- distributed block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000012545 processing Methods 0.000 claims description 52
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 4
- 230000004048 modification Effects 0.000 description 18
- 238000012986 modification Methods 0.000 description 18
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000006978 adaptation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0665—Virtualisation aspects at area level, e.g. provisioning of virtual or logical volumes
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a bitmap operation method and device of a distributed block storage system, electronic equipment, a readable storage medium and the distributed block storage system, which are applied to the technical field of distributed storage. The method comprises the following steps: the storage server establishes a bitmap caching module for storing bitmap object data of the target volume in a memory in advance. After receiving bitmap operation requests of merging a plurality of bitmap operation tasks of the same target bitmap object from a client, the storage server judges whether bitmap object data corresponding to the target bitmap object is cached, and if so, based on the bitmap operation requests, the storage server directly performs updating operation on the corresponding bitmap object data read from the bitmap caching module. The application optimizes the bitmap modifying operation of the distributed block storage system, and can effectively improve the storage performance of the system.
Description
Technical Field
The present application relates to the field of distributed storage technologies, and in particular, to a bitmap operation method and apparatus for a distributed block storage system, an electronic device, a readable storage medium, and a distributed block storage system.
Background
In order to overcome the defect of poor performance caused by the adoption of a centralized storage server for storing all data in the traditional network storage system, the distributed network storage system is applied. The distributed network storage system adopts an expandable system structure, data to be stored are stored in a plurality of independent storage servers in a scattered mode, and the storage load is shared by the plurality of storage servers, so that the reliability, availability and access efficiency of the storage system can be improved, the expansion is easy, the practicability is higher, and the requirement of large-scale storage application can be met. In order to further improve the storage performance of the storage system, block storage, such as SAN (Storage Area Network ) technology, is applied to a distributed network storage system, where the block storage is a storage service used by mapping the bare disk space to a host entirely.
The space of each storage volume of a distributed block storage system may be discretely distributed over each machine in a cluster, and to achieve fast statistics, bitmap statistics are typically used to represent the state of each discrete data block. And the bitmap method is used for statistics of the use capacity and the object state of the storage volume, so that the accuracy of the bitmap needs to be ensured. Related art in order to ensure consistency of bitmap data, bitmap operations are all atomic operations. When bitmap modification operation is performed, an object lock is required to be added at the server side, then latest data is acquired from a target disk, the lock is released after the data is written into the disk, and during the period, the server cannot respond to a request on the object. Meanwhile, different bitmap modification operations of the same object of the client are sent independently, and the server cannot respond to processing requests of multiple bitmap operations at the same time, so that multiple requests are backlogged on the server. For application scenarios with high response speed requirements or high concurrency application scenarios, the bitmap modification operation of the related art may seriously affect the storage performance of the distributed block storage system.
In view of this, the technical problem that the bitmap modification operation affects the storage performance of the distributed block storage system is effectively solved, which needs to be solved by those skilled in the art.
Disclosure of Invention
The application provides a bitmap operation method and device of a distributed block storage system, electronic equipment, a readable storage medium and the distributed block storage system, which can effectively solve the problem of poor storage performance of the distributed block storage system caused by bitmap modification operation.
In order to solve the technical problems, the application provides the following technical scheme:
in one aspect, the present application provides a bitmap operation method of a distributed block storage system, including:
a bitmap caching module is established in a memory in advance and is used for storing bitmap object data of a target volume;
when a bitmap operation request of a client combining a plurality of bitmap operation tasks of the same target bitmap object is received, judging whether the bitmap caching module contains the target bitmap object or not;
and if the bitmap caching module contains the target bitmap object, updating the corresponding bitmap object data read from the bitmap caching module based on the bitmap operation request.
Illustratively, the storing bitmap object data of the target volume includes:
counting the heat value of each storage volume;
and selecting each target volume according to the heat value of each storage volume at the current moment and the preset hot spot identification condition, and storing bitmap object data of each target volume into the bitmap caching module.
Illustratively, the counting the heat value of each storage volume includes:
invoking a heat calculation relational expression to calculate a heat value of each storage volume; the heat calculation relational expression is as follows:
R＝W 1 ·L+W 2 ·N+W 3 ·T+W 4 ·Q；
wherein R is a heat value, W 1 Weight value W of current operation type L 2 For operating the weight value of the number N, W 3 Weight value of last operation time T, W 4 To store a weight value for the volume total capacity Q.
Illustratively, the counting the heat value of each storage volume includes:
presetting hot spot volume updating frequency;
and based on the hot-spot volume updating frequency, when the detection reaches the target time, counting the heat values of all the storage volumes again.
Illustratively, the counting the heat value of each storage volume includes:
counting the sum of the request quantity at each moment;
and if the sum of the current request quantity reaches the maximum request threshold, counting the heat values of all the storage volumes again, and resetting the sum of the current request quantity.
Illustratively, after updating the bitmap object data read from the bitmap caching module based on the bitmap operation request, the method further includes:
and after the bitmap caching module is updated, writing the memory data corresponding to the updated bitmap caching module into the corresponding disk.
Another aspect of the present application provides a bitmap operation method of a distributed block storage system, including:
judging whether a target bitmap object which belongs to a currently generated bitmap operation task has a bitmap operation request which is sent to the same storage server and is being processed;
if yes, merging a plurality of bitmap operation tasks generated by the target bitmap object after the bitmap operation request into a new bitmap operation request, and then issuing the new bitmap operation request to the target storage server, so that the target storage server contains the target bitmap object in a bitmap caching module, and updating corresponding bitmap object data read from the bitmap caching module based on the new bitmap operation request; the bitmap caching module is established in the memory in advance and is used for storing bitmap object data of the target volume.
Illustratively, the merging the bitmap operation tasks generated by the target bitmap object after the bitmap operation request into a new bitmap operation request includes:
Judging whether the request average processing time delay exists or not;
if the request average processing time delay exists, the current bitmap operation task is put into a request queue, all bitmap operation tasks of the request queue are combined, and the issuing time of the new bitmap operation request is determined according to the request average processing time delay.
Illustratively, the determining whether the bitmap operation task currently generated belongs to the target bitmap object, after the bitmap operation request is sent to the same storage server and is being processed, further includes:
calculating the current processing time length of the bitmap operation request;
and updating the average processing time delay of the request according to the processing time length of each historical bitmap operation request and the current processing time length.
Another aspect of the present application provides a bitmap operation apparatus of a distributed block storage system, including:
the object caching module is used for establishing a bitmap caching module in a memory in advance, and the bitmap caching module is used for storing bitmap object data of a target volume;
the buffer judgment module is used for judging whether the bitmap buffer module contains the target bitmap object or not when receiving bitmap operation requests of a plurality of bitmap operation tasks of the same target bitmap object combined by the client;
And the bitmap operation module is used for updating the corresponding bitmap object data read from the bitmap caching module based on the bitmap operation request if the bitmap caching module contains the target bitmap object.
Another aspect of the present application provides a bitmap operation apparatus of a distributed block storage system, including:
the request judging module is used for judging whether a bitmap operation request which is sent to the same storage server and is being processed exists in a target bitmap object to which the bitmap operation task which is generated at present belongs;
a request merging and issuing module, configured to merge a plurality of bitmap operation tasks generated by a target bitmap object after a bitmap operation request into a new bitmap operation request if a bitmap operation request which is being processed and is sent to the same storage server exists in a target bitmap object to which the bitmap operation task currently generated belongs, and issue the new bitmap operation request to the target storage server, so that the target storage server includes the target bitmap object in a bitmap caching module, and update corresponding bitmap object data read from the bitmap caching module based on the new bitmap operation request; the bitmap caching module is established in the memory in advance and is used for storing bitmap object data of the target volume.
The application also provides an electronic device comprising a processor for implementing the steps of the bitmap operation method of the distributed block storage system according to any one of the preceding claims when executing a computer program stored in a memory.
The application also provides a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the bitmap operation method of the distributed block storage system as described in any of the preceding claims.
The application finally provides a distributed block storage system, which comprises a plurality of storage servers and a plurality of clients;
each storage server for implementing the bitmap operation method of the distributed block storage system as set forth in any one of the preceding claims when executing the computer program stored in the memory;
each client, when executing a computer program stored in a memory, implements the bitmap operation method of the distributed block storage system as described in any one of the preceding claims.
The technical scheme provided by the application has the advantages that the bitmap object is cached in advance by the server side, the bitmap operation of the cached bitmap object can be directly modified for the read data in the memory, the modifying efficiency of the bitmap operation can be effectively improved, and the technical problem that the bitmap modifying operation affects the storage performance of the distributed block storage system is effectively solved; the modification operations of the client to the same bitmap object are combined and then sent to the server together, so that backlog of a plurality of requests at the server is avoided, the number of tasks received by the server is reduced, the processing speed of the requests is effectively improved, and the storage performance of the distributed block storage system is improved.
In addition, the application also provides a corresponding implementation device, electronic equipment, a readable storage medium and a distributed block storage system for the bitmap operation method of the distributed block storage system, so that the method has more practicability, and the device, the electronic equipment, the readable storage medium and the distributed block storage system have corresponding advantages.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.
Drawings
For a clearer description of the present application or of the technical solutions related thereto, the following brief description will be given of the drawings used in the description of the embodiments or of the related art, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained from these drawings without the inventive effort of a person skilled in the art.
FIG. 1 is a flow chart of a method of bitmap operation of a distributed block storage system according to the present application;
FIG. 2 is a flow chart of another method of bitmap operation for a distributed block storage system according to the present application;
FIG. 3 is a block diagram of one embodiment of a bitmap operation device of a distributed block storage system according to the present application;
FIG. 4 is a block diagram of another embodiment of a bitmap operation device of a distributed block storage system according to the present application;
FIG. 5 is a block diagram of an embodiment of an electronic device according to the present application;
FIG. 6 is a block diagram illustrating one embodiment of a distributed block storage system according to the present application.
Detailed Description
In order to better understand the aspects of the present application, the present application will be described in further detail with reference to the accompanying drawings and detailed description. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The terms "comprising" and "having" and any variations thereof in the description and claims of the application and in the foregoing drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may include other steps or elements not expressly listed. Various non-limiting embodiments of the present application are described in detail below.
Referring to fig. 1 first, fig. 1 is a flow chart of a bitmap operation method of a distributed block storage system provided by the present application, where the distributed block storage system is a storage system aggregated by a plurality of storage servers and capable of providing a unified access interface and a management interface for an application server, and the embodiment is applicable to any one of the storage servers of the distributed block storage system, in other words, the execution subject of the embodiment is the storage server of the distributed block storage system, and the embodiment may include the following:
s101: and establishing a bitmap caching module in a memory in advance.
The bitmap buffer module in the step is a part which is pre-opened in the memory space, and the number, the positions in the memory space and the capacity of the occupied memory space can be flexibly selected according to actual conditions, so that the bitmap buffer module is not limited in any way. Before the bitmap caching module is used, the bitmap caching module needs to be initialized, and the bitmap caching module in this embodiment is used for storing bitmap object data of a target volume. The target volume is a storage volume with frequent reading and writing, is a storage volume which is cached in advance according to the actual situation, and the number of the target volumes and the selection conditions can be flexibly selected according to the actual situation, so that the realization of the application is not affected.
S102: when a bitmap operation request of combining a plurality of bitmap operation tasks of the same target bitmap object by a client is received, judging whether the bitmap buffer module contains the target bitmap object or not, if yes, executing S103; if not, S104 is performed.
It can be understood that the distributed block storage system does not support high concurrency, the client sends one request to the storage server at a time, and in order to reduce the waiting time and the processing task amount of the server, in the process that the current server processes the bitmap operation request, the client can combine a plurality of bitmap operation tasks which also need to be subjected to bitmap operation into one bitmap operation request, and send the combined bitmap operation request to the storage server for processing.
S103: based on the bitmap operation request, the corresponding bitmap object data read from the bitmap caching module is updated.
S104: based on the bitmap operation request, a bitmap object lock is added first, modification operation is carried out on corresponding bitmap object data acquired from the disk, and the lock is released after the disk is to be written.
In the above step, after the storage server receives the bitmap operation request, it determines whether the bitmap object to be modified by the bitmap operation request is cached, the storage server caches the target volume meeting the condition, and establishes a copy in the memory, so that the cached bitmap object can directly read the corresponding bitmap object data from the memory, and the read bitmap data is modified according to the bitmap operation request, including but not limited to adding, deleting and modifying, that is, updating the data in the bitmap cache module, without reading the disk data. Of course, the performance degradation caused by memory occupation and the performance degradation caused by bitmap modification operation are considered, the number of target volumes is limited, and for performing bitmap modification operation on volumes without cache, or needing to add an object lock at the storage server, the latest data is obtained from the disk, and the lock is released after writing into the disk. In order to improve the data writing efficiency, after the bitmap caching module is updated, only the memory data corresponding to the updated bitmap caching module can be written into the corresponding disk.
In the technical scheme provided by the application, the bitmap object is cached in advance by the server side, the bitmap operation of the cached bitmap object can be directly modified for the read data in the memory, the modifying efficiency of the bitmap operation can be effectively improved, and the technical problem that the bitmap modifying operation affects the storage performance of the distributed block storage system is effectively solved; the modification operations of the client to the same bitmap object are combined and then sent to the server together, so that backlog of a plurality of requests at the server is avoided, the number of tasks received by the server is reduced, the processing speed of the requests is effectively improved, and the storage performance of the distributed block storage system is improved.
In the above embodiment, the selection condition of how to set the target volume is not limited, and the selection manner of the target volume provided in this embodiment may include the following:
counting the heat value of each storage volume;
and selecting each target volume according to the heat value of each storage volume at the current moment and the preset hot spot identification condition, and storing bitmap object data of each target volume into a bitmap caching module.
The preset hot spot condition is a condition for judging whether the hot spot belongs to frequent operation or not, and can be flexibly determined according to an actual application scene, for example, the preset hot spot identification condition can be the latest 10 frequent operation volumes. The heat value is a frequency used to represent how often each storage volume is operated, and may be represented in any of a number of existing heat metrics, such as the number of writes per unit time, without affecting the implementation of the present application.
As an implementation manner of the foregoing embodiment, in order to better represent the frequency of each storage volume being operated, and improve the efficiency of the subsequent bitmap modification operation, the present application further provides a calculation manner of the heat value of each storage volume, and in this embodiment, the heat calculation relational expression may be directly invoked to calculate the heat value of each storage volume; the heat calculation relation can be expressed as:
R＝W 1 ·L+W 2 ·N+W 3 ·T+W 4 ·Q；
wherein R is a heat value, W 1 Weight value W of current operation type L 2 For operating the weight value of the number N, W 3 Weight value of last operation time T, W 4 For storing volume total capacityWeight value of Q.
In this embodiment, when the bitmap object of the storage volume needs to be operated, the heat statistics is performed each time, the statistical dimensions may include, but are not limited to, operation types, operation numbers, last operation time, total capacity, and consideration of the influence of different dimensions on operation possibility, the most suitable weight value of each statistical dimension may be given to each of the statistical dimensions according to the actual situation, the weight values of different operation types are different, if the weight value of the write operation is greater than the weight value of the read operation, the weight value of the total capacity is greater than the weight value of the operation types, the heat of the bitmap object is calculated by weighting according to each of the statistical dimensions, so that the target volumes possibly performing the bitmap operation in a future period can be predicted more accurately, and the target volumes are cached in advance, thereby effectively improving the efficiency of the bitmap operation.
It can be understood that the heat of different storage volumes will change, that is, the probability that different storage volumes are accessed under different conditions is continuously changed, in order to improve the efficiency of bitmap operation, after a period of time or a certain number of requests, the present application loads the bitmap cache of a new bitmap object according to the heat, and eliminates the bitmap cache of a bitmap object with low heat, which may include the following contents:
as an exemplary embodiment, the hot spot volume update frequency may be preset; based on the hot-spot volume update frequency, when the detection reaches the target time, the heat values of all the storage volumes are counted again.
As another implementation parallel to the above embodiment, the sum of the request numbers at each time may be counted in real time; and if the sum of the current request quantity reaches the maximum request threshold, counting the heat values of all the storage volumes again, and resetting the sum of the current request quantity.
The hot-spot volume update frequency and the maximum request threshold value of the embodiment can be flexibly selected according to practical situations, and the application is not limited in any way.
As can be seen from the above, the above embodiment constructs the bitmap caching module in advance at the storage server side, and initializes the bitmap caching module on each OSD (Object Storage Device ) for caching bitmap object data of a target volume, such as the last 10 frequently operated volumes, in the memory. And when the bitmap object of each volume needs to be operated, carrying out heat statistics according to the operation type, the operation quantity, the last operation time and the total capacity, carrying out weighted calculation according to the statistics to obtain the heat of each bitmap object, and updating the bitmap object data of the target volume in the bitmap cache module in real time. And judging whether the bitmap is cached or not during the bitmap modifying operation, if so, directly acquiring and modifying the bitmap from the memory during the modifying operation, and waiting for memory data to fall after the modifying is completed. Based on this, the present application also describes how the client in the distributed block storage system performs the bitmap operation request, and the execution subject of the following embodiment is the client in the distributed block storage system, please refer to fig. 2, fig. 2 is a flow chart of another bitmap operation method of the distributed block storage system provided by the present application, which may include the following contents:
S201: and judging whether a target bitmap object which belongs to the currently generated bitmap operation task has a bitmap operation request which is sent to the same storage server and is being processed. If yes, S202 is executed. If not, S203 is performed.
S202: merging a plurality of bitmap operation tasks generated after the bitmap operation request of the target bitmap object into a new bitmap operation request, and then issuing the new bitmap operation request to the target storage server so that the target storage server contains the target bitmap object in the bitmap caching module, and updating corresponding bitmap object data read from the bitmap caching module based on the new bitmap operation request.
S203: and directly issuing the bitmap operation task as a new bitmap operation request to the target storage server.
In this embodiment, when the client modifies the bitmap operation and issues the bitmap operation, it is first determined whether other modification requests on the same bitmap object are sent to the storage server side for processing, and no matter whether the storage server starts to process the bitmap operation request, as long as the client has sent the bitmap operation request to the storage server and the storage server has not processed the bitmap operation request, the client will combine the bitmap operation tasks of the target bitmap object, and issue the bitmap operation task as a new bitmap operation request after combining the bitmap operation tasks. If the currently generated bitmap operation task belongs to the target bitmap object, the bitmap operation task is not sent to the same storage server and/or the bitmap operation request being processed, the bitmap operation task is directly sent to the corresponding storage server as a bitmap operation request to be processed, and a bitmap caching module is pre-established in the memory of the storage server and is used for storing the bitmap object data of the target volume. The processing of the bitmap operation request by the storage server may refer to the implementation manner described in any one of the foregoing embodiments, which will not be described herein.
As can be seen from the above, the present embodiment optimizes the bitmap modification operation of the distributed block storage system, so as to effectively improve the storage performance of the system.
As an exemplary implementation manner of the foregoing embodiment, in order to reduce the waiting time of the server side and improve the bitmap operation efficiency, if there is a bitmap operation request that is being processed and is sent to the same storage server in the target bitmap object to which the bitmap operation task currently generated belongs, the value of the request average processing delay may be read first, that is, whether there is a request average processing delay is determined; if the request average processing time delay exists, the current bitmap operation task is put into the request queue, all the current bitmap operation tasks of the request queue are combined, and the issuing time of the new bitmap operation request is determined according to the request average processing time delay.
In this embodiment, the average processing delay of the request is used to indicate the approximate time required by the storage server to process a bitmap operation request, and for the distributed block storage system which just starts to run or the distributed block storage system which just starts to run the technical solution of the present application, the time required by the storage server to process a bitmap operation request is not pre-calculated, and the default value of the average processing delay of the request is not pre-set, so that the bitmap operation task can be directly issued as a bitmap operation request to the corresponding storage server for processing. If the target bitmap operation object has other bitmap operation requests being processed and the average processing time delay of the requests can be read, the bitmap operation task is put into a queue. The issuing time of the new bitmap operation request can be determined by setting a timer, and when the timer is finished, the tasks in the queue are issued as a new bitmap operation request. Every time a bitmap operation task joins the request queue, all bitmap operation tasks in the request queue are converged and sent for processing when the timer expires. The server receives the bitmap operation requests, and modifies all bitmap operation tasks together to be dropped, so that the processing efficiency can be effectively increased.
In order to accurately represent the time required by the storage server to process the bitmap operation request, to improve the efficiency of the server side in processing the request, the average processing delay value of the request may be updated in real time, that is, after judging whether the bitmap operation request being processed is sent to the same storage server by the target bitmap object to which the currently generated bitmap operation task belongs, the method may further include:
calculating the current processing time length of the bitmap operation request which is currently processed; and then updating the value of the request average processing time delay stored in the client according to the processing time length and the current processing time length of each history bitmap operation request.
It should be noted that, in the present application, the steps are not strictly executed sequentially, so long as they conform to the logic sequence, the steps may be executed simultaneously, or may be executed according to a certain preset sequence, and fig. 1-2 are only schematic, and are not meant to represent only such execution sequence.
The application also provides a corresponding device for the bitmap operation method of the distributed block storage system, so that the method is more practical. Wherein the device may be described separately from the functional module and the hardware. In the following description, a bitmap operation device of a distributed block storage system according to the present application is described, where the bitmap operation device is configured to implement a bitmap operation method of a distributed block storage system according to the present application, in this embodiment, the bitmap operation device of a distributed block storage system may include or be divided into one or more program modules, where the one or more program modules are stored in a storage medium and executed by one or more processors, to implement a bitmap operation method of a distributed block storage system according to the first embodiment of the present application. Program modules in the present application are referred to as a series of computer program instruction segments capable of performing particular functions more appropriately than the program itself describing the execution of the bitmap operating means of the distributed block storage system in the storage medium. The following description will specifically describe functions of each program module of the present embodiment, and a bitmap operation device of the distributed block storage system described below and a bitmap operation method of the distributed block storage system described above may be referred to correspondingly to each other.
Referring first to fig. 3, fig. 3 is a block diagram of a bitmap operation device of a distributed block storage system according to an embodiment of the present application, where the device is built in a storage server, and may include:
the object buffer module 301 is configured to create a bitmap buffer module in the memory in advance, where the bitmap buffer module is configured to store bitmap object data of the target volume.
The buffer determining module 302 is configured to determine, when a bitmap operation request of a client combining multiple bitmap operation tasks of the same target bitmap object is received, whether the bitmap buffer module includes the target bitmap object.
And the bitmap operation module 303 is configured to update the bitmap object data read from the bitmap buffer module based on the bitmap operation request if the bitmap buffer module includes the target bitmap object.
Illustratively, in some implementations of this embodiment, the object cache module 301 may further be configured to: counting the heat value of each storage volume; and selecting each target volume according to the heat value of each storage volume at the current moment and the preset hot spot identification condition, and storing bitmap object data of each target volume into a bitmap caching module.
As an exemplary implementation of the foregoing embodiment, the foregoing object cache module 301 may further be configured to: invoking a heat calculation relational expression to calculate a heat value of each storage volume; the heat calculation relational expression is:
R＝W 1 ·L+W 2 ·N+W 3 ·T+W 4 ·Q；
wherein R is a heat value, W 1 Weight value W of current operation type L 2 For operating the weight value of the number N, W 3 Weight value of last operation time T, W 4 To store a weight value for the volume total capacity Q.
As another exemplary implementation of the above embodiment, the above object cache module 301 may further be configured to:
presetting hot spot volume updating frequency;
based on the hot-spot volume update frequency, when the detection reaches the target time, the heat values of all the storage volumes are counted again.
As yet another exemplary implementation of the foregoing embodiment, the foregoing object buffer module 301 may further be configured to:
counting the sum of the request quantity at each moment;
and if the sum of the current request quantity reaches the maximum request threshold, counting the heat values of all the storage volumes again, and resetting the sum of the current request quantity.
Illustratively, in other implementations of this embodiment, the apparatus may further include a disk drop module, for writing, after the bitmap cache module is updated, the memory data corresponding to the updated bitmap cache module into the corresponding disk.
Based on the angle of the functional modules, please refer to fig. 4, fig. 4 is a block diagram of a bitmap operation device of a distributed block storage system according to an embodiment of the present application, where the device is built in a client, and the device may include:
the request judging module 401 is configured to judge whether there is a bitmap operation request that is being processed and is sent to the same storage server, for a target bitmap object to which the bitmap operation task currently generated belongs.
A request merging and issuing module 402, configured to merge, if a target bitmap object to which a currently generated bitmap operation task belongs has a bitmap operation request that is sent to the same storage server and is being processed, a plurality of bitmap operation tasks generated after the bitmap operation request by the target bitmap object into a new bitmap operation request, and issue the new bitmap operation request to the target storage server, so that the target storage server includes the target bitmap object in the bitmap caching module, and update corresponding bitmap object data read from the bitmap caching module based on the new bitmap operation request; the bitmap caching module is established in the memory in advance and is used for storing bitmap object data of the target volume.
Illustratively, in other implementations of the present embodiment, the request merge issuing module 402 may be further configured to:
Judging whether the request average processing time delay exists or not;
if the request average processing time delay exists, the current bitmap operation task is put into the request queue, all the current bitmap operation tasks of the request queue are combined, and the issuing time of the new bitmap operation request is determined according to the request average processing time delay.
As an exemplary implementation manner of the foregoing embodiment, the foregoing apparatus may further include a delay updating module configured to: calculating the current processing time length of the bitmap operation request; and updating the average processing time delay of the request according to the processing time length and the current processing time length of each historical bitmap operation request.
The functions of each functional module of the bitmap operation device of the distributed block storage system of the present application may be specifically implemented according to the method in the above method embodiment, and the specific implementation process may refer to the related description of the above method embodiment, which is not repeated herein.
As can be seen from the above, the present embodiment optimizes the bitmap modification operation of the distributed block storage system, so as to effectively improve the storage performance of the system.
The bitmap operation device of the distributed block storage system is described from the perspective of functional modules, and further, the application also provides an electronic device, which is described from the perspective of hardware. Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 5, the electronic device comprises a memory 50 for storing a computer program; a processor 51 for implementing the steps of the bitmap operation method of the distributed block storage system as mentioned in any of the above embodiments when executing a computer program.
Processor 51 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and processor 51 may also be a controller, microcontroller, microprocessor, or other data processing chip, among others. The processor 51 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 51 may also include a main processor, which is a processor for processing data in an awake state, also called a CPU (Central Processing Unit ), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 51 may be integrated with a GPU (Graphics Processing Unit, image processor) for taking care of rendering and drawing of the content that the display screen is required to display. In some embodiments, the processor 51 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
Memory 50 may include one or more computer-readable storage media, which may be non-transitory. Memory 50 may also include high-speed random access memory as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. The memory 50 may in some embodiments be an internal storage unit of the electronic device, such as a hard disk of a server. The memory 50 may in other embodiments also be an external storage device of the electronic device, such as a plug-in hard disk provided on a server, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like. Further, the memory 50 may also include both internal storage units and external storage devices of the electronic device. The memory 50 may be used to store not only application software installed in an electronic device, but also various types of data, such as: code or the like that executes a program during the bitmap operation method of the distributed block storage system may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 50 is at least used for storing a computer program 501, which, when loaded and executed by the processor 51, is capable of implementing the relevant steps of the bitmap operation method of the distributed block storage system disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502, data 503, and the like, where the storage mode may be transient storage or permanent storage. Operating system 502 may include Windows, unix, linux, among other things. The data 503 may include, but is not limited to, data corresponding to bitmap operation results of the distributed block storage system, and the like.
In some embodiments, the electronic device may further include a display 52, an input/output interface 53, a communication interface 54, or network interface, a power supply 55, and a communication bus 56. Among other things, the display 52, input output interface 53 such as a Keyboard (Keyboard) pertain to a user interface, which may optionally also include standard wired interfaces, wireless interfaces, etc. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device and for displaying a visual user interface. Communication interface 54 may optionally include a wired interface and/or a wireless interface, such as a WI-FI interface, a bluetooth interface, etc., typically used to establish a communication connection between an electronic device and other electronic devices. The communication bus 56 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The bus may be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in fig. 5, but not only one bus or one type of bus.
Those skilled in the art will appreciate that the configuration shown in fig. 5 is not limiting of the electronic device and may include more or fewer components than shown, for example, may also include sensors 57 to perform various functions.
The functions of each functional module of the electronic device of the present application may be specifically implemented according to the method in the above method embodiment, and the specific implementation process may refer to the relevant description of the above method embodiment, which is not repeated herein.
As can be seen from the above, the present embodiment optimizes the bitmap modification operation of the distributed block storage system, so as to effectively improve the storage performance of the system.
It will be appreciated that the bitmap operation methods of the distributed block storage system of the above-described embodiments, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution contributing to the related art, or may be embodied in the form of a software product stored in a storage medium, which performs all or part of the steps of the methods of the various embodiments of the present application. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), an electrically erasable programmable ROM, registers, a hard disk, a multimedia card, a card-type Memory (e.g., SD or DX Memory, etc.), a magnetic Memory, a removable disk, a CD-ROM, a magnetic disk, or an optical disk, etc., that can store program code.
Based on this, the present application also provides a readable storage medium storing a computer program which, when executed by a processor, performs the steps of the bitmap operation method of the distributed block storage system according to any one of the embodiments above.
The present application also provides a distributed block storage system, please refer to fig. 6, which may include a plurality of storage servers 601 and a plurality of clients 602; each storage server 601 for implementing the bitmap operation method of the distributed block storage system according to any one of the above when executing the computer program stored in the memory; each client, when executing a computer program stored in a memory, implements the bitmap operation method of the distributed block storage system as described in any one of the above.
The functions of the storage server and the client of the distributed block storage system according to the embodiments of the present application may be specifically implemented according to the method in the embodiments of the method, and the specific implementation process may refer to the related description of the embodiments of the method, which is not repeated herein.
As can be seen from the above, the present embodiment optimizes the bitmap modification operation of the distributed block storage system, so as to effectively improve the storage performance of the system.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the hardware including the device and the electronic equipment disclosed in the embodiments, the description is relatively simple because the hardware includes the device and the electronic equipment corresponding to the method disclosed in the embodiments, and relevant places refer to the description of the method.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The bitmap operation method, the bitmap operation device, the electronic equipment, the readable storage medium and the distributed block storage system provided by the application are described in detail. The principles and embodiments of the present application have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the method of the present application and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the application can be made without departing from the principles of the application and these modifications and adaptations are intended to be within the scope of the application as defined in the following claims.
Claims (14)
1. A method of bitmap operation for a distributed block storage system, comprising:
a bitmap caching module is established in a memory in advance and is used for storing bitmap object data of a target volume;
when a bitmap operation request of a client combining a plurality of bitmap operation tasks of the same target bitmap object is received, judging whether the bitmap caching module contains the target bitmap object or not;
and if the bitmap caching module contains the target bitmap object, updating the corresponding bitmap object data read from the bitmap caching module based on the bitmap operation request.
2. The bitmap operation method of a distributed block storage system according to claim 1, wherein the storing bitmap object data of a target volume includes:
counting the heat value of each storage volume;
and selecting each target volume according to the heat value of each storage volume at the current moment and the preset hot spot identification condition, and storing bitmap object data of each target volume into the bitmap caching module.
3. The method of bitmap operation of a distributed block storage system according to claim 2, wherein said counting the heat value of each storage volume comprises:
Invoking a heat calculation relational expression to calculate a heat value of each storage volume; the heat calculation relational expression is as follows:
R＝W 1 ·L+W 2 ·N+W 3 ·T+W 4 ·Q；
wherein R is a heat value, W 1 Weight value W of current operation type L 2 Weights for the number of operations NValue, W 3 Weight value of last operation time T, W 4 To store a weight value for the volume total capacity Q.
4. The method of bitmap operation of a distributed block storage system according to claim 2, wherein said counting the heat value of each storage volume comprises:
presetting hot spot volume updating frequency;
and based on the hot-spot volume updating frequency, when the detection reaches the target time, counting the heat values of all the storage volumes again.
5. The method of bitmap operation of a distributed block storage system according to claim 2, wherein said counting the heat value of each storage volume comprises:
counting the sum of the request quantity at each moment;
and if the sum of the current request quantity reaches the maximum request threshold, counting the heat values of all the storage volumes again, and resetting the sum of the current request quantity.
6. The bitmap operation method of a distributed block storage system according to any one of claims 1 to 5, wherein after updating the corresponding bitmap object data read from the bitmap caching module based on the bitmap operation request, further comprising:
And after the bitmap caching module is updated, writing the memory data corresponding to the updated bitmap caching module into the corresponding disk.
7. A method of bitmap operation for a distributed block storage system, comprising:
judging whether a target bitmap object which belongs to a currently generated bitmap operation task has a bitmap operation request which is sent to the same storage server and is being processed;
if yes, merging a plurality of bitmap operation tasks generated by the target bitmap object after the bitmap operation request into a new bitmap operation request, and then issuing the new bitmap operation request to the target storage server, so that the target storage server contains the target bitmap object in a bitmap caching module, and updating corresponding bitmap object data read from the bitmap caching module based on the new bitmap operation request; the bitmap caching module is established in the memory in advance and is used for storing bitmap object data of the target volume.
8. The bitmap operation method of a distributed block storage system according to claim 7, wherein merging a plurality of bitmap operation tasks generated by the target bitmap object after the bitmap operation request into a new bitmap operation request, comprises:
Judging whether the request average processing time delay exists or not;
if the request average processing time delay exists, the current bitmap operation task is put into a request queue, all bitmap operation tasks of the request queue are combined, and the issuing time of the new bitmap operation request is determined according to the request average processing time delay.
9. The method for bitmap operation of a distributed block storage system according to claim 8, wherein said determining whether a target bitmap object to which a currently generated bitmap operation task belongs has a bitmap operation request which is sent to the same storage server and is being processed, further comprises:
calculating the current processing time length of the bitmap operation request;
and updating the average processing time delay of the request according to the processing time length of each historical bitmap operation request and the current processing time length.
10. A bitmap operation apparatus of a distributed block storage system, comprising:
the object caching module is used for establishing a bitmap caching module in a memory in advance, and the bitmap caching module is used for storing bitmap object data of a target volume;
the buffer judgment module is used for judging whether the bitmap buffer module contains the target bitmap object or not when receiving bitmap operation requests of a plurality of bitmap operation tasks of the same target bitmap object combined by the client;
And the bitmap operation module is used for updating the corresponding bitmap object data read from the bitmap caching module based on the bitmap operation request if the bitmap caching module contains the target bitmap object.
11. A bitmap operation apparatus of a distributed block storage system, comprising:
the request judging module is used for judging whether a bitmap operation request which is sent to the same storage server and is being processed exists in a target bitmap object to which the bitmap operation task which is generated at present belongs;
a request merging and issuing module, configured to merge a plurality of bitmap operation tasks generated by a target bitmap object after a bitmap operation request into a new bitmap operation request if a bitmap operation request which is being processed and is sent to the same storage server exists in a target bitmap object to which the bitmap operation task currently generated belongs, and issue the new bitmap operation request to the target storage server, so that the target storage server includes the target bitmap object in a bitmap caching module, and update corresponding bitmap object data read from the bitmap caching module based on the new bitmap operation request; the bitmap caching module is established in the memory in advance and is used for storing bitmap object data of the target volume.
12. An electronic device comprising a processor and a memory, the processor being configured to implement the steps of the bitmap operation method of the distributed block storage system according to any one of claims 1 to 6 and/or the bitmap operation method of the distributed block storage system according to any one of claims 7 to 9 when executing a computer program stored in the memory.
13. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the bitmap operation method of the distributed block storage system according to any one of claims 1 to 6 and/or the bitmap operation method of the distributed block storage system according to any one of claims 7 to 9.
14. A distributed block storage system comprising a plurality of storage servers and a plurality of clients;
each storage server for implementing the bitmap operation method of the distributed block storage system according to any one of claims 1 to 6 when executing a computer program stored in a memory;
each client, when executing a memory-stored computer program, implements a bitmap operation method of a distributed block storage system according to any one of claims 7 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310638122.2A CN116627341A (en) | 2023-05-30 | 2023-05-30 | Distributed block storage system and bitmap operation method, device, equipment and medium thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310638122.2A CN116627341A (en) | 2023-05-30 | 2023-05-30 | Distributed block storage system and bitmap operation method, device, equipment and medium thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116627341A true CN116627341A (en) | 2023-08-22 |
Family
ID=87591761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310638122.2A Pending CN116627341A (en) | 2023-05-30 | 2023-05-30 | Distributed block storage system and bitmap operation method, device, equipment and medium thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116627341A (en) |
-
2023
- 2023-05-30 CN CN202310638122.2A patent/CN116627341A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105205014B (en) | A kind of date storage method and device | |
| US10838626B2 (en) | Methods, apparatuses, and computer program products for controlling write requests in storage system | |
| US20150134912A1 (en) | Scheduler, multi-core processor system, and scheduling method | |
| US20120297216A1 (en) | Dynamically selecting active polling or timed waits | |
| CN110287038A (en) | Promote the method and system of the data-handling efficiency of Spark Streaming frame | |
| CN116662038B (en) | Industrial information detection method, device, equipment and medium based on shared memory | |
| CN113625973A (en) | Data writing method and device, electronic equipment and computer readable storage medium | |
| CN108595251B (en) | Dynamic graph updating method, device, storage engine interface and program medium | |
| US20180217875A1 (en) | Data processing system and data processing method | |
| CN116089477B (en) | Distributed training method and system | |
| CN117056054A (en) | Interrupt control method, interrupt controller, computer device, and storage medium | |
| CN116594563A (en) | Distributed storage capacity expansion method and device, electronic equipment and storage medium | |
| CN116627341A (en) | Distributed block storage system and bitmap operation method, device, equipment and medium thereof | |
| US9405470B2 (en) | Data processing system and data processing method | |
| CN108062224B (en) | Data reading and writing method and device based on file handle and computing equipment | |
| CN115840654A (en) | Message processing method, system, computing device and readable storage medium | |
| CN111966301B (en) | Migration speed control method, device and medium for distributed object storage system | |
| CN111858665B (en) | Method, system, terminal and storage medium for improving soft copy reading performance | |
| US20170075589A1 (en) | Memory and bus frequency scaling by detecting memory-latency-bound workloads | |
| KR20220142059A (en) | In-memory Decoding Cache and Its Management Scheme for Accelerating Deep Learning Batching Process | |
| CN113138718A (en) | Storage method, apparatus, system, and medium for distributed block storage system | |
| CN113918527B (en) | Scheduling method and device based on file cache and computing equipment | |
| CN116991781B (en) | Request processing device, method, chip, storage medium and electronic equipment | |
| CN112671918B (en) | Binary system-based distributed data downloading method, device, equipment and medium | |
| CN116737605B (en) | Data prefetching method, device, equipment and medium based on chip multilevel storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |