CN107894921A - A kind of realization method and system of distributed block storage volume performance statistics - Google Patents
A kind of realization method and system of distributed block storage volume performance statistics Download PDFInfo
- Publication number
- CN107894921A CN107894921A CN201711100035.2A CN201711100035A CN107894921A CN 107894921 A CN107894921 A CN 107894921A CN 201711100035 A CN201711100035 A CN 201711100035A CN 107894921 A CN107894921 A CN 107894921A
- Authority
- CN
- China
- Prior art keywords
- storage volume
- volume
- distributed
- storage
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/501—Performance criteria
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5018—Thread allocation
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of implementation method of distributed block storage volume performance statistics, comprise the following steps：In read-write IO thread/process, shared drive mapping is carried out to multiple storage volume；The performance statistic of storage volume is write to the corresponding metadata information of distributed storage cluster；The performance statistics of storage volume are checked using storage volume information inspection order.Solve multithreading, multi-process to the access problem of same volume by using System V shared drives, then deposit statistics using distributed data pond, while system resources consumption is reduced, accurate statistics go out the performance data of each storage volume.Using the program can the information such as the IOPS of each storage volume of accurate statistics, bandwidth, delay, and significantly reduce the resource consumption to internal memory, CPU.
Description
Technical field
The present invention relates to technical field of distributed memory, especially a kind of realization side of distributed block storage volume performance statistics
Method and system.
Background technology
Distributed memory system uses expansible framework, shares storage load using memory node, not only increases and is
Reliability, availability and the access efficiency of system, are also easier to extend.Therefore, made extensively in big data epoch, distributed storage
With.
In distributed block storage, volume (also commonly referred to as disk or subregion) be we use for file management
Basic container unit.In traditional SAN (Storage Area Network) storages, master control is generally 2, each other active and standby mould
Formula, storage volume Information Statistics can be solved using static variable.But in distributed storage, because memory node is cluster fortune
OK, there is provided volume in multiple nodes all may have access to, therefore use static variable can not accurate statistics storage volume performance.
The content of the invention
It is an object of the invention to provide a kind of realization method and system of distributed block storage volume performance statistics, can accurately unite
The information such as the IOPS of each storage volume, bandwidth, delay are counted, and significantly reduce the resource consumption to internal memory, CPU.
To achieve the above object, the present invention uses following technical proposals：
A kind of implementation method of distributed block storage volume performance statistics, comprises the following steps：
In read-write IO thread/process, shared drive mapping is carried out to multiple storage volume；
The performance statistic of storage volume is write to the corresponding metadata information of distributed storage cluster；
The performance statistics of storage volume are checked using storage volume information inspection order.
Further, it is described in read-write IO thread/process, multiple storage volume are carried out with shared drive mapping, including：
The Memory Mapping File of detection internal memory volume whether there is, if it does, being transferred to next step；If it does not, wound
Build Memory Mapping File corresponding to internal memory volume；
Internal memory mapping is carried out for the read-write IO of each internal memory volume, and obtains performance statistic.
Further, the name form of the Memory Mapping File of the internal memory volume is " storage Pool name "+"/"+" file label
Claim ".
Further, the acquisition performance statistical data packet includes, according to the timestamp of record, to the IOPS values and band of record
Width values are set to 0, IOPS and bandwidth value in the record time interval of 1 second.
Further, the performance statistic by storage volume writes the corresponding metadata letter of distributed storage cluster
Breath, including, on processing time interval per second, IOPS values and the volume metadata letter of bandwidth value write-in distributed storage cluster
In breath.
Further, the performance statistic by storage volume writes the corresponding metadata letter of distributed storage cluster
Breath, including, the performance statistics of storage volume are added into metadata corresponding to the storage volume inside write-in distributed storage pond and believed
In breath.
A kind of distributed block storage volume performance statistics system, including distributed storage cluster, by several distributed storages
Node is formed；
Distributed storage pond, the data organization unit being abstracted by distributed storage cluster；
Storage volume, the organizational unit of distributed storage pond memory space.
The effect provided in the content of the invention is only the effect of embodiment, rather than whole effects that invention is all, above-mentioned
A technical scheme in technical scheme has the following advantages that or beneficial effect：
The present invention proposes a kind of implementation of the volume performance statistics of distributed block storage, using linux system
System V shared drives, make multiple threads, the statistical information of multiple process whole accessible storages volume, at the same it is per second calculating
The performance statistics such as the IOPS that goes out, bandwidth, delay are stored in object corresponding to the volume data message in distributed storage pond.It is logical
Cross and solve multithreading, multi-process to the access problem of same volume using System V shared drives, then using distributed number
Statistics is deposited according to pond, while system resources consumption is reduced, accurate statistics go out the performance data of each storage volume.Use
The program can the information such as the IOPS of each storage volume of accurate statistics, bandwidth, delay, and significantly reduce and internal memory, CPU resource disappeared
Consumption.
Brief description of the drawings
Fig. 1 is distributed storage volume information statistics configuration diagram of the present invention；
Fig. 2 is that distributed storage of the present invention volume performance statistics realizes schematic diagram；
Fig. 3 is the implementation method flow chart of distributed block storage volume performance statistics of the present invention.
Embodiment
For the technical characterstic for illustrating this programme can be understood, below by embodiment, and its accompanying drawing is combined, to this hair
It is bright to be described in detail.Following disclosure provides many different embodiments or example is used for realizing the different knots of the present invention
Structure.In order to simplify disclosure of the invention, hereinafter the part and setting of specific examples are described.In addition, the present invention can be with
Repeat reference numerals and/or letter in different examples.This repetition is that for purposes of simplicity and clarity, itself is not indicated
Relation between various embodiments are discussed and/or set.It should be noted that part illustrated in the accompanying drawings is not necessarily to scale
Draw.Present invention omits the description to known assemblies and treatment technology and process to avoid being unnecessarily limiting the present invention.
As shown in figure 1, the volume information statistics framework of distributed block storage reasonable in design, in multiple memory nodes composition
Distributed storage cluster takes out several distributed storage ponds and carries out data tissue, and a distributed storage cluster can be built
Multiple distributed storage ponds, storage volume are a kind of depositor spatial organization frameworks of storage pool.
As shown in Fig. 2 using the System V shared drives of linux system, multiple threads, multiple processes are made all to visit
Ask the statistical information of storage volume, while the performance statistics such as per second the IOPS calculated, bandwidth, delay are stored in distribution
In object corresponding to the volume data message of storage pool.
In the reading and writing data process or thread of multiple storage volume, storage volume shared drive is mapped to kernel, mainly
In order to solve in same memory node, process statistical information can not be total to when multiple read-write processes read while write a storage volume
The problem of enjoying.
Internal memory mapping is the multi-process that solves the problems, such as single node, and the metadata information for writing storage cluster is to solve point
The statistical information sharing problem of memory node in cloth storage cluster.By using storage volume information inspection order, arbitrarily depositing
Storage node checks the performance statistics of storage volume.
As shown in figure 3, a kind of implementation method of distributed block storage volume performance statistics, comprises the following steps：
S1, read-write IO thread/process in, to multiple storage volume carry out shared drive mapping；
S2, the corresponding metadata information by the performance statistic write-in distributed storage cluster of storage volume；
S3, the performance statistics for checking using storage volume information inspection order storage volume.
In step S1, in read-write IO thread/process, multiple storage volume are carried out with shared drive mapping, including：
S11, the Memory Mapping File of detection internal memory volume whether there is, if it does, being transferred to next step；If do not deposit
Creating Memory Mapping File corresponding to internal memory volume；The name form of the Memory Mapping File of internal memory volume is " storage pool name
Claim "+"/"+" file label claims ".
S12, the read-write IO for each internal memory volume carry out internal memory mapping, and obtain performance statistic.Obtain performance system
Count including according to the timestamp of record, being set to 0 to the IOPS values and bandwidth value of record, in the record time interval of 1 second
IOPS and bandwidth value.
In step S2, the performance statistic of storage volume is write to the corresponding metadata information of distributed storage cluster, will
The performance statistics of storage volume are added inside write-in distributed storage pond in metadata information corresponding to the storage volume.Including,
On processing time interval per second, IOPS values and bandwidth value are write in the volume metadata information of distributed storage cluster.Deposit
Storage volume just saves a metadata information of oneself in storage pool in itself, and this modification is simply additional in metadata information
Metadata information, specific bottom-layer design is preserved by K/V key-value pair, the same equivalent to deposit database.
Although above-mentioned the embodiment of the present invention is described with reference to accompanying drawing, model not is protected to the present invention
The limitation enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme those skilled in the art are not
Need to pay various modifications or deformation that creative work can make still within protection scope of the present invention.
Claims (7)
1. a kind of implementation method of distributed block storage volume performance statistics, it is characterized in that, comprise the following steps：
In read-write IO thread/process, shared drive mapping is carried out to multiple storage volume；
The performance statistic of storage volume is write to the corresponding metadata information of distributed storage cluster；
The performance statistics of storage volume are checked using storage volume information inspection order.
2. a kind of implementation method of distributed block storage volume performance statistics as claimed in claim 1, it is characterized in that, it is described to read
Write in IO thread/process, multiple storage volume are carried out with shared drive mapping, including：
The Memory Mapping File of detection internal memory volume whether there is, if it does, being transferred to next step；Should if it does not, creating
Memory Mapping File corresponding to internal memory volume；
Internal memory mapping is carried out for the read-write IO of each internal memory volume, and obtains performance statistic.
3. a kind of implementation method of distributed block storage volume performance statistics as claimed in claim 2, it is characterized in that, the internal memory
The name form of the Memory Mapping File of volume is " storage Pool name "+"/"+" file label claims ".
4. a kind of implementation method of distributed block storage volume performance statistics as claimed in claim 2, it is characterized in that, the acquisition
Performance statistic includes, and according to the timestamp of record, the IOPS values and bandwidth value of record are set to 0, the record time interval of 1 second
Interior IOPS and bandwidth value.
5. a kind of implementation method of distributed block storage volume performance statistics as claimed in claim 4, it is characterized in that, it is described to deposit
The corresponding metadata information of the performance statistic write-in distributed storage cluster of storage volume, including, processing time interval per second
On, IOPS values and bandwidth value are write in the volume metadata information of distributed storage cluster.
6. a kind of implementation method of distributed block storage volume performance statistics as claimed in claim 1, it is characterized in that, it is described to deposit
The corresponding metadata information of the performance statistic write-in distributed storage cluster of storage volume, including, by the performance statistics of storage volume
Inside information adding write-in distributed storage pond in metadata information corresponding to the storage volume.
7. a kind of distributed block storage volume performance statistics system, it is characterized in that, including distributed storage cluster, by several distributions
Formula memory node is formed；
Distributed storage pond, the data organization unit being abstracted by distributed storage cluster；
Storage volume, the organizational unit of distributed storage pond memory space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711100035.2A CN107894921A (en) | 2017-11-09 | 2017-11-09 | A kind of realization method and system of distributed block storage volume performance statistics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711100035.2A CN107894921A (en) | 2017-11-09 | 2017-11-09 | A kind of realization method and system of distributed block storage volume performance statistics |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107894921A true CN107894921A (en) | 2018-04-10 |
Family
ID=61804887
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711100035.2A Pending CN107894921A (en) | 2017-11-09 | 2017-11-09 | A kind of realization method and system of distributed block storage volume performance statistics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107894921A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109669632A (en) * | 2018-12-10 | 2019-04-23 | 浪潮电子信息产业股份有限公司 | Metadata wiring method, device and medium based on distributed objects storage system |
| CN111949486A (en) * | 2020-08-14 | 2020-11-17 | 苏州浪潮智能科技有限公司 | Method and device for realizing real-time cluster service monitoring |
| CN112231189A (en) * | 2020-09-11 | 2021-01-15 | 苏州浪潮智能科技有限公司 | Method and device for counting sizes of universal interface layer blocks |
| CN113625954A (en) * | 2021-06-30 | 2021-11-09 | 济南浪潮数据技术有限公司 | Volume information query method, device, equipment and readable storage medium |
| CN114461388A (en) * | 2022-01-13 | 2022-05-10 | 苏州浪潮智能科技有限公司 | Block storage service metadata configuration method, device, equipment and medium |
| CN115167786A (en) * | 2022-09-06 | 2022-10-11 | 浪潮电子信息产业股份有限公司 | Data storage method, device, system, equipment and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722405A (en) * | 2012-06-06 | 2012-10-10 | 新浪网技术(中国)有限公司 | Counting method in high concurrent and multithreaded application and system |
| WO2012137265A1 (en) * | 2011-04-08 | 2012-10-11 | Hitachi, Ltd. | Computer, computer system, and data communication method |
| CN105426522A (en) * | 2015-12-07 | 2016-03-23 | 浪潮(北京)电子信息产业有限公司 | File system performance statistical method and system |
| WO2016154231A1 (en) * | 2015-03-23 | 2016-09-29 | Netapp, Inc. | Methods and systems for real-time activity tracing in a storage environment |
-
2017
- 2017-11-09 CN CN201711100035.2A patent/CN107894921A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012137265A1 (en) * | 2011-04-08 | 2012-10-11 | Hitachi, Ltd. | Computer, computer system, and data communication method |
| CN102722405A (en) * | 2012-06-06 | 2012-10-10 | 新浪网技术(中国)有限公司 | Counting method in high concurrent and multithreaded application and system |
| WO2016154231A1 (en) * | 2015-03-23 | 2016-09-29 | Netapp, Inc. | Methods and systems for real-time activity tracing in a storage environment |
| CN105426522A (en) * | 2015-12-07 | 2016-03-23 | 浪潮(北京)电子信息产业有限公司 | File system performance statistical method and system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109669632A (en) * | 2018-12-10 | 2019-04-23 | 浪潮电子信息产业股份有限公司 | Metadata wiring method, device and medium based on distributed objects storage system |
| CN111949486A (en) * | 2020-08-14 | 2020-11-17 | 苏州浪潮智能科技有限公司 | Method and device for realizing real-time cluster service monitoring |
| CN111949486B (en) * | 2020-08-14 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Method and device for realizing real-time cluster service monitoring |
| CN112231189A (en) * | 2020-09-11 | 2021-01-15 | 苏州浪潮智能科技有限公司 | Method and device for counting sizes of universal interface layer blocks |
| CN112231189B (en) * | 2020-09-11 | 2022-08-19 | 苏州浪潮智能科技有限公司 | Method and device for counting sizes of universal interface layer blocks |
| CN113625954A (en) * | 2021-06-30 | 2021-11-09 | 济南浪潮数据技术有限公司 | Volume information query method, device, equipment and readable storage medium |
| CN113625954B (en) * | 2021-06-30 | 2024-03-15 | 济南浪潮数据技术有限公司 | Method, device and equipment for inquiring volume information and readable storage medium |
| CN114461388A (en) * | 2022-01-13 | 2022-05-10 | 苏州浪潮智能科技有限公司 | Block storage service metadata configuration method, device, equipment and medium |
| CN114461388B (en) * | 2022-01-13 | 2024-01-12 | 苏州浪潮智能科技有限公司 | Block storage service metadata configuration method, device, equipment and medium |
| CN115167786A (en) * | 2022-09-06 | 2022-10-11 | 浪潮电子信息产业股份有限公司 | Data storage method, device, system, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107894921A (en) | A kind of realization method and system of distributed block storage volume performance statistics | |
| US11947829B2 (en) | Data writing method, device, storage server, and computer readable storage medium | |
| US11500586B2 (en) | Data read-write method and apparatus and circular queue | |
| KR102380670B1 (en) | Fine-grained bandwidth provisioning in a memory controller | |
| CN103246616B (en) | A kind of globally shared buffer replacing method of access frequency within long and short cycle | |
| CN105280215B (en) | Dynamic random access memory DRAM method for refreshing, equipment and system | |
| CN112463074B (en) | Data classification storage method, system, terminal and storage medium | |
| CN113220716B (en) | Device and method for improving real-time database data processing performance | |
| US20140115016A1 (en) | Systems and methods for enabling parallel processing of write transactions | |
| US20200027499A1 (en) | Configuring dynamic random access memory refreshes for systems having multiple ranks of memory | |
| CN109408411A (en) | The L1 Cache management method of GPGPU based on data access number | |
| CN109801659A (en) | The activation management of DRAM memory bank | |
| CN105302489B (en) | A kind of remote embedded accumulator system of heterogeneous polynuclear and method | |
| CN103019974A (en) | Memory access processing method and controller | |
| CN103729239A (en) | Distributed type lock algorithm of mirror-image metadata | |
| CN103106150A (en) | Method capable of achieving automatic simplifying configuration and automatic simplifying distribution for storage system | |
| CN110083306A (en) | A kind of distributed objects storage system and storage method | |
| WO2017131784A1 (en) | Hybrid concurrency control | |
| CN115408149A (en) | Time sequence storage engine memory design and distribution method and device | |
| CN102880467A (en) | Method for verifying Cache coherence protocol and multi-core processor system | |
| CN105740170A (en) | Cache dirty page flashing method and apparatus | |
| CN115840654B (en) | Message processing method, system, computing device and readable storage medium | |
| CN108701091A (en) | Chance memory for dynamic workload is finely tuned | |
| CN116243868A (en) | Task processing method, device, equipment and computer readable storage medium | |
| CN112905121B (en) | Data refreshing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180410 |