CN112306815A

CN112306815A - Method, device, equipment and medium for monitoring IO (input/output) information between OSD (on Screen display) side master and slave in Ceph

Info

Publication number: CN112306815A
Application number: CN202011281879.3A
Authority: CN
Inventors: 闵顺新
Original assignee: New H3C Big Data Technologies Co Ltd
Current assignee: New H3C Big Data Technologies Co Ltd
Priority date: 2020-11-16
Filing date: 2020-11-16
Publication date: 2021-02-02
Anticipated expiration: 2040-11-16
Also published as: CN112306815B

Abstract

The disclosure provides a method, a device, equipment and a medium for monitoring IO information between OSD (on screen display) side master and slave in Ceph, wherein the method comprises the following steps: each logic set PG of the logic set PG layer counts the statistical information between a master OSD and a slave OSD in all OSD lists mapped by the logic set PG; the object storage device OSD layer counts all statistical information; the OSD issues abnormal statistical information, and the cluster monitoring process MON of the Ceph collects and summarizes the abnormal statistical information. The method and the device realize IO information statistics between the OSD side master and slave. The method and the device for monitoring the IO and the link abnormality of the OSD are based on the IO statistical information among the OSD, and the method and the device can monitor common fault scenes such as slow IO and link abnormality among the OSD in the cluster operation process in a log printing mode. The method and the device for fault diagnosis and repair based on the IO statistical information among the OSD can provide basis for further fault diagnosis and repair.

Description

Method, device, equipment and medium for monitoring IO (input/output) information between OSD (on Screen display) side master and slave in Ceph

Technical Field

The present disclosure relates to the field of computer information monitoring technologies, and more particularly, to a method, an apparatus, a device, and a medium for monitoring IO information between a master and a slave on an OSD side in a Ceph.

Background

Ceph, a unified, distributed storage system designed for excellent performance, reliability, and scalability, has now become one of the most popular open source storage solutions.

In the Ceph memory system, the OSD process parses the received message, if it is an OP type operation. An OpRequest object is newly built, the object runs through the execution process of the whole operation until the object is destroyed, and the Ceph provides an OpTracker mechanism to track the object, record some events and help us to perform performance analysis and tuning. Through the OpTracker mechanism, the following functions can be realized:

1) tracking the execution process of the OP, and marking the occurrence of certain events at certain specific time points; when the time of one OP process exceeds a certain threshold, a warning message is printed. This information can be used for later analysis.

2) The OP that has completed is tracked and recorded. By sending a dump command to the OSD, a certain number of OP information in a certain period of time in the history can be viewed.

The prior art has the defects that the monitoring granularity is too small, each OP is mainly taken as an object, and IO information statistics between a master OSD and a slave OSD cannot be formed; monitored IO information is not well summarized and cannot be used as a basis for further repairing faults among OSD; because the monitoring granularity is too small, monitoring each OP will have a certain impact on cluster IO performance. And the switch of the OpTracker cannot be opened in the actual operation process of the cluster, and useful statistical information cannot be acquired through an OpTracker mechanism.

Disclosure of Invention

In order to solve the related problem statistics of the prior art that the delay of the current Ceph cluster service is too long, the main reasons are slow IO and link problems, and the Peer hang-up problem is 3. What this disclosure will solve is the technical problem of monitoring for slow IO and link abnormality between master and slave OSDs.

In order to achieve the technical purpose, the present disclosure provides a method for monitoring IO information between a master and a slave at an OSD side in a Ceph, including:

each logic set PG of the logic set PG layer counts the statistical information between a master OSD and a slave OSD in all OSD lists mapped by the logic set PG;

the object storage device OSD layer counts all statistical information;

the OSD issues abnormal statistical information, and the cluster monitoring process MON of the Ceph collects and summarizes the abnormal statistical information.

Further, the statistical information specifically includes:

operating conditions of all OPs passing through an IO path within a certain time period, wherein the operating conditions specifically include: the total number of returned OPs, the number of IO returned with error codes, the average delay of the IO, and the number of hung OPs.

Further, the statistics of the logic set PG on all OSD lists mapped by the logic set PG specifically includes the following statistics information between the master OSD and the slave OSD:

for each OP processed by each of the logical sets,

recording the sending time at the stage of sending the OP;

recording a reception time during a link transmission from the master OSD to the slave OSD;

and recording the total time delay of all OPs processed by the logic set PG and any slave OSD, the total number of returned IO and the number of IO returned error codes in a certain time period.

Further, the counting of all statistical information of the OSD layer of the object storage device specifically includes:

each OSD bears a certain number of PGs, and each OSD traverses all the PGs borne by the OSD at regular time intervals to obtain and summarize statistical information inside all the PGs.

Further, the OSD issuing abnormal statistical information specifically includes:

by means of printing warning information in the OSD logs, monitoring of common fault scenes with slow IO and abnormal links among OSD in the cluster operation process and statistics of abnormal information are achieved.

In order to achieve the above technical object, the present disclosure can also provide a device for monitoring IO information between a master and a slave at an OSD side in a Ceph, including:

the logic set statistical information module is used for counting statistical information between a master OSD and a slave OSD in all OSD lists mapped by the logic set PG in each logic set PG of the logic set PG layer;

the object storage device statistical information module is used for counting all statistical information of the OSD layer of the object storage device;

and the cluster monitoring statistical information processing module is used for OSD (on-screen display) issuing abnormal statistical information, and the cluster monitoring process MON of the Ceph collects and summarizes the abnormal statistical information.

Further, the statistical information specifically includes:

Further, the logic set statistical information module specifically includes:

the sending time recording submodule is used for recording the sending time at the stage of sending the OP;

the receiving time recording submodule is used for recording the receiving time in the link transmission process from the main OSD to the slave OSD;

and the statistical information recording submodule is used for recording the total time delay of all OPs, the total number of returned IOs and the number of IOs returning error codes, which are processed by the logic set PG and any slave OSD within a certain time period.

To achieve the above technical objects, the present disclosure can also provide a computer storage medium having a computer program stored thereon, where the computer program is used to implement the steps of the method for OSD side inter-master-slave IO information monitoring in Ceph described above when the computer program is executed by a processor.

In order to achieve the above technical object, the present disclosure further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method for monitoring IO information between the OSD side master and slave in the Ceph when executing the computer program.

The beneficial effect of this disclosure does:

1. the method and the device realize IO information statistics between the OSD side master and slave.

2. Based on IO statistical information among the OSD, the method and the device can realize monitoring of common fault scenes such as slow IO, abnormal link and the like among the OSD in the cluster operation process in a log printing mode.

3. Based on the IO statistical information among the OSD, the method and the device can provide basis for further fault judgment and repair.

Drawings

FIG. 1 illustrates a prior art architecture diagram of a Ceph storage architecture;

FIG. 2 shows a prior art addressing flow diagram for Ceph;

FIG. 3 shows a schematic diagram of a Ceph data operation flow;

FIG. 4 shows a process flow diagram of a typical failure scenario-master-slave OSD link exception;

fig. 5 shows a schematic diagram of an OSD IO information statistics flow;

fig. 6 shows a schematic flow diagram of embodiment 1 of the present disclosure;

fig. 7 shows a schematic structural diagram of embodiment 2 of the present disclosure;

fig. 8 shows a schematic structural diagram of embodiment 4 of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

Various structural schematics according to embodiments of the present disclosure are shown in the figures. The figures are not drawn to scale, wherein certain details are exaggerated and possibly omitted for clarity of presentation. The shapes of various regions, layers, and relative sizes and positional relationships therebetween shown in the drawings are merely exemplary, and deviations may occur in practice due to manufacturing tolerances or technical limitations, and a person skilled in the art may additionally design regions/layers having different shapes, sizes, relative positions, as actually required.

The present disclosure relates to the interpretation of terms:

RADS Reliable Automic Distributed ObjectStore: reliable, automatic, distributed object storage

OSD ObjectStore Device: object storage device

OSD CObjectStore Device Client: client for communicating with OSD

MONCeph monitor: ceph's cluster monitoring process

RBD RADOS Block Device: RADOS block device

RADOSGWRADOSGateWay: RADOS gateway

Ceph FSCeph File System: ceph file system

librados librados: libraries in Ceph to simplify access to RADS

OP Operation: operations with instruction types

PG PlacementGroup: logical collection of a set of objects

Object: storing objects

Ceph storage architecture

The Ceph storage architecture is shown in FIG. 1:

and finally calling an OSDC module to directly access the RADOS object storage system through the client of the Ceph and through the RBD, the RADOSGW, the Ceph FS or the librados. The OSDC is a module at the bottom of client comparison, and the core of the OSDC is to encapsulate operation data, calculate the address of an object, send a request and process timeout.

A RADOS cluster is mainly composed of two types of nodes. One is a plurality of OSDs responsible for performing data storage and maintenance functions, and the other is a plurality of MONs responsible for performing system status detection and maintenance. Once an application accesses the Ceph cluster to perform a write operation, data is stored in the OSD as an object. The OSD is the only component in the Ceph cluster that stores the actual user data and responds to client read requests. An OSD daemon is bound to a physical disk of the cluster. The MON component keeps track of the health of the entire cluster through a series of maps, including the map that records the state of each OSD component, i.e., OSDMap. All OSD nodes report the state to the MON node, and the MON collects the information to generate OSDMap which is shared to the whole cluster.

PG PlacementGroup: the principle of the role of logical collections of objects in Ceph addressing is detailed:

fig. 2 shows an addressing flow of Ceph, and a file to be operated by a user is first cut into Objects by size. Each object is mapped to a certain PG by a hash algorithm. The PG is then mapped to a set of OSDs by the CRUSH algorithm. The object is finally stored in the selected set of OSDs. The PG in the figure is a logical container that contains multiple objects and is mapped to multiple OSDs.

Ceph data operation flow

Taking writing data as an example, the data operation flow of Ceph is shown in fig. 3. Assume that one copy of data needs to be written to three copies of the OSD.

The OSDC will communicate directly with the Primary OSD initiating a write operation (step 1). After receiving the request, the Primary OSD initiates a write operation to the Secondary OSD and the Tertiary OSD respectively (steps 2 and 3). After the Secondary OSD and the Tertiary OSD respectively complete the write operation, confirmation information is sent to the Primary OSD (steps 4 and 5). When the Primary OSD confirms that the writing of the other two OSDs is completed, the Primary OSD also completes the data writing and confirms the completion of the data writing operation to the client (step 6). The OSDC selects three OSD to send data according to a certain rule, and the states of the selected three OSD are UP and not DOWN.

In the actual operation process, the Ceph cluster may cause too large time delay of the service layer due to the existence of faults between OSDs, and may cause service interruption seriously. The invention provides a method for monitoring IO information between OSD side master and slave in Ceph.

The first embodiment is as follows:

as shown in fig. 6:

the utility model provides a method for monitoring IO information between OSD side master slave in Ceph, which comprises the following steps:

s1: each logic set PG of the logic set PG layer counts the statistical information between a master OSD and a slave OSD in all OSD lists mapped by the logic set PG;

specifically, the statistical information specifically includes:

Specifically, the S1 specifically includes:

for each OP processed by each of the logical sets,

recording the sending time at the stage of sending the OP;

And the PG layer counts all IO information.

The PG layer counts the statistical information between the master OSD and the slave OSD in all the OSD lists mapped by the PG.

The technical scheme of the disclosure is explained in detail below by a typical fault scene, namely master-slave OSD link exception.

The IO path between the OSD side master and slave is shown in fig. 4. The master OSD sends OP out in the stage I, and is transmitted to the slave OSD through the link transmission of the stage II. Stage (c) represents that from receiving the OP from the OSD, the OP is processed until a reply message to the OP is returned. The reply message is transmitted back to the master OSD via the link at stage (r) and received by the master OSD at stage (v).

The IO information counted by the PG layer mainly includes the status of all OPs passing through the IO path in fig. 6 within a certain time period, preferably within 10s, including the number of returned OPs, the number of IO returning an error code (the returned error code indicates that the OP fails to be executed), the average delay of IO, and the number of OPs hanging up for 50 s.

Let all OSD lists to which PG maps be osd.1, osd.2, osd.3, and main OSD be osd.1. The IO statistics for the final PG layer formation are similar to table 1.

TABLE 1 PG side IO statistics

The various statistical terms are implemented as follows:

1) each PG stores the statistical information in io _ stat _ t, and uses special mutual exclusion element lock _ io _ stat to protect the statistical information, and the newly defined io _ stat _ t is only used for OP monitoring, so the granularity of mutual exclusion element is small enough.

2) IO number, failed IO number, total delay: for each OP of the PG process, the transmission time is recorded in step (r) of fig. 7, and the reception time is recorded in step (r) of fig. 7. And step two, the total time delay of all OPs between the PG processing and a certain slave OSD within 10s, the total number of returned IOs and the total number of failed IOs in the returned IOs can be obtained.

3) Number of IOs not returned over 50 seconds: for OP which is not returned after timeout, each PG is designed with two maps, namely tid _ to _ OP _ map _ count and tid _ to _ read _ map _ count, which are respectively used for storing write OP between master and slave OSD and read OP between master and slave OSD. When a stored OP gets a return response from the OSD, it is deleted from the map of the corresponding type, otherwise it is always present. Access to them also requires the acquisition of a special mutex lock _ io _ stat.

S2: the object storage device OSD layer counts all statistical information;

specifically, the counting of all statistical information of the OSD layer of the object storage device specifically includes:

And counting all IO information by the OSD layer.

Each OSD carries a certain number of PGs. The OSD will go through all PGs every 10s, obtain IO statistics inside all PGs,

1) the average time delay between the master OSD and each slave OSD within 10s is calculated.

2) And accumulating the number of returned IOs and the number of failed IOs within 10 s.

3) The number of IOs that have not been processed for more than 50 seconds are accumulated in the tid _ to _ op _ map _ count and the tid _ to _ read _ map _ count.

The final statistics are similar to table 2, assuming osd.1 is the main OSD.

TABLE 2 OSD side statistics

For an access to a field in the PG memory, a PG lock generally needs to be acquired. The thread in which the read-write flow is located also needs to be acquired frequently and occupies the PG lock for a long time. Each thread is used more frequently for a PG lock of a PG, and the granularity of the PG lock is larger, so that the acquisition latency of the PG lock may be longer. Because the PG lock corresponding to each PG needs to be acquired before the OSD traverses the PG every 10s to acquire the statistical information of each OP, the PG number is large, which may cause that the IO statistical summary time of the OSD layer is too long and exceeds the statistical interval of 10s, resulting in that the IO statistical information cannot be summarized in time. The scheme avoids the dependence on the PG lock by recording the OP being processed by the PG layer by using the mutual exclusion element with small granularity and a special data structure. The detailed statistical flow is shown in fig. 5.

S3: the OSD issues abnormal statistical information, and the cluster monitoring process MON of the Ceph collects and summarizes the abnormal statistical information.

OSD issuing abnormal IO statistical information

If the OSD layer finds an anomaly in the statistics,

1) by means of printing warning information in the OSD logs, monitoring of common fault scenes such as slow IO (input/output) and abnormal links among OSD (on screen display) in the cluster operation process is achieved.

2) Further, the OSD uploads all statistics of the period to the MON.

MON summary statistics

The MON receives statistical information from the various OSDs,

1) and printing abnormal master-slave OSD link IO statistical information.

2) Further, the fault OSD is judged according to a certain strategy, and some repairing means are tried.

Example two:

as shown in fig. 6:

the present disclosure also provides a device for monitoring IO information between OSD side master and slave in Ceph, including:

a logic set statistical information module 100, configured to count statistical information between a master OSD and a slave OSD in all OSD lists mapped by a logic set PG in each logic set PG of a logic set PG layer;

an object storage device statistical information module 200, configured to count all statistical information on the OSD layer of the object storage device;

the cluster monitoring statistical information performing module 300 is configured to issue abnormal statistical information through OSD, and the cluster monitoring process MON of Ceph collects and summarizes the abnormal statistical information.

The logic set statistical information module 100 of the present disclosure is sequentially connected to the object storage device statistical information module 200 and the cluster monitoring statistical information module 300.

Further, the statistical information specifically includes:

Further, the logic set statistical information module 100 specifically includes:

Example three:

the present disclosure can also provide a computer storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the steps of the method for OSD-side inter-master-slave IO information monitoring in Ceph described above.

The computer storage medium of the present disclosure may be implemented with a semiconductor memory, a magnetic core memory, a magnetic drum memory, or a magnetic disk memory.

Semiconductor memories are mainly used as semiconductor memory elements of computers, and there are two types, Mos and bipolar memory elements. Mos devices have high integration, simple process, but slow speed. The bipolar element has the advantages of complex process, high power consumption, low integration level and high speed. NMos and CMos were introduced to make Mos memory dominate in semiconductor memory. NMos is fast, e.g. 45ns for 1K bit sram from intel. The CMos power consumption is low, and the access time of the 4K-bit CMos static memory is 300 ns. The semiconductor memories described above are all Random Access Memories (RAMs), i.e. read and write new contents randomly during operation. And a semiconductor Read Only Memory (ROM), which can be read out randomly but cannot be written in during operation, is used to store solidified programs and data. The ROM is classified into a non-rewritable fuse type ROM, PROM, and a rewritable EPROM.

The magnetic core memory has the characteristics of low cost and high reliability, and has more than 20 years of practical use experience. Magnetic core memories were widely used as main memories before the mid 70's. The storage capacity can reach more than 10 bits, and the access time is 300ns at the fastest speed. The typical international magnetic core memory has a capacity of 4 MS-8 MB and an access cycle of 1.0-1.5 mus. After semiconductor memory is rapidly developed to replace magnetic core memory as a main memory location, magnetic core memory can still be applied as a large-capacity expansion memory.

Drum memory, an external memory for magnetic recording. Because of its fast information access speed and stable and reliable operation, it is being replaced by disk memory, but it is still used as external memory for real-time process control computers and medium and large computers. In order to meet the needs of small and micro computers, subminiature magnetic drums have emerged, which are small, lightweight, highly reliable, and convenient to use.

Magnetic disk memory, an external memory for magnetic recording. It combines the advantages of drum and tape storage, i.e. its storage capacity is larger than that of drum, its access speed is faster than that of tape storage, and it can be stored off-line, so that the magnetic disk is widely used as large-capacity external storage in various computer systems. Magnetic disks are generally classified into two main categories, hard disks and floppy disk memories.

Hard disk memories are of a wide variety. The structure is divided into a replaceable type and a fixed type. The replaceable disk is replaceable and the fixed disk is fixed. The replaceable and fixed magnetic disks have both multi-disk combinations and single-chip structures, and are divided into fixed head types and movable head types. The fixed head type magnetic disk has a small capacity, a low recording density, a high access speed, and a high cost. The movable head type magnetic disk has a high recording density (up to 1000 to 6250 bits/inch) and thus a large capacity, but has a low access speed compared with a fixed head magnetic disk. The storage capacity of a magnetic disk product can reach several hundred megabytes with a bit density of 6250 bits per inch and a track density of 475 tracks per inch. The disk set of the multiple replaceable disk memory can be replaced, so that the disk set has large off-body capacity, large capacity and high speed, can store large-capacity information data, and is widely applied to an online information retrieval system and a database management system.

Example four:

the disclosure further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the method for monitoring the IO information between the master and the slave at the OSD side in the Ceph are implemented.

Fig. 8 is a schematic diagram of an internal structure of an electronic device in one embodiment. As shown in fig. 8, the electronic device includes a processor, a storage medium, a memory, and a network interface connected through a system bus. The storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store control information sequences, and when the computer readable instructions are executed by the processor, the processor can realize a method for monitoring IO information between a master and a slave at the OSD side in Ceph. The processor of the electrical device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer readable instructions, and when the computer readable instructions are executed by the processor, the processor may be enabled to execute a method for OSD side inter-master-slave IO information monitoring in Ceph. The network interface of the computer device is used for connecting and communicating with the terminal. Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

The electronic device includes, but is not limited to, a smart phone, a computer, a tablet, a wearable smart device, an artificial smart device, a mobile power source, and the like.

The processor may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor is a control unit (control unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (for example, executing remote data reading and writing programs, etc.) stored in the memory and calling data stored in the memory.

The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connected communication between the memory and at least one processor or the like.

Fig. 8 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 8 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.

For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor through a power management device, so that functions such as charge management, discharge management, and power consumption management are implemented through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.

Further, the electronic device may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used to establish a communication connection between the electronic device and other electronic devices.

Optionally, the electronic device may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.

Further, the computer usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method for monitoring IO information between OSD side master and slave in Ceph is characterized by comprising the following steps:

the object storage device OSD layer counts all statistical information;

2. The method according to claim 1, wherein the statistical information specifically comprises:

3. The method of claim 2, wherein the logic set PG counts statistics information between the master OSD and the slave OSD in all OSD lists mapped by the logic set PG, and the statistics information specifically includes:

for each OP processed by each of the logical sets,

recording the sending time at the stage of sending the OP;

4. The method of claim 1, wherein the counting of all statistical information by the OSD layer of the object storage device specifically comprises:

5. The method of claim 1, wherein the OSD issuing abnormal statistical information specifically includes:

6. The utility model provides a device of OSD side inter-master-slave IO information monitoring in Ceph which characterized in that includes:

7. The apparatus of claim 6, wherein the statistical information specifically comprises:

8. The apparatus of claim 7, wherein the logic set statistics module specifically comprises:

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for OSD-side inter-master-slave IO information monitoring in Ceph according to any one of claims 1 to 5 when executing the computer program.

10. A computer storage medium having computer program instructions stored thereon, wherein the program instructions, when executed by a processor, are configured to implement the steps corresponding to the method for OSD side inter-master-slave IO information monitoring in Ceph according to any one of claims 1 to 5.