CN111045599A - Parameter configuration method and device of distributed storage system and computer equipment - Google Patents
Parameter configuration method and device of distributed storage system and computer equipment Download PDFInfo
- Publication number
- CN111045599A CN111045599A CN201911054078.0A CN201911054078A CN111045599A CN 111045599 A CN111045599 A CN 111045599A CN 201911054078 A CN201911054078 A CN 201911054078A CN 111045599 A CN111045599 A CN 111045599A
- Authority
- CN
- China
- Prior art keywords
- host
- target
- storage system
- osd
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000004364 calculation method Methods 0.000 claims abstract description 63
- 238000004590 computer program Methods 0.000 claims description 9
- 230000005856 abnormality Effects 0.000 claims description 4
- 230000008901 benefit Effects 0.000 abstract description 9
- 230000008569 process Effects 0.000 abstract description 9
- 238000013135 deep learning Methods 0.000 abstract 1
- 238000011084 recovery Methods 0.000 description 10
- 238000012544 monitoring process Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000001994 activation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a parameter configuration method and device of a distributed storage system and computer equipment, and relates to the technical field of computers. The method comprises the following steps: firstly, acquiring role information corresponding to each host in a cluster for deploying a distributed storage system; collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host; then inquiring a target calculation rule corresponding to a target parameter needing to be configured in the distributed storage system; and finally, configuring the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the host respectively. According to the method and the device, parameter configuration is not needed to be carried out on the distributed storage system for deep learning manually, and the learning cost can be saved. And through the automatic parameter configuration process, the distributed storage system can be quickly adjusted to the optimal state, and the better performance advantage of the distributed storage system is exerted in time.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for configuring parameters of a distributed storage system, and a computer device.
Background
For a distributed storage system (such as Ceph), the distributed storage system comprises two forms of block storage and object storage, and has the characteristics of various configurations, clear structure and the like. There are many parameters in the parameter configuration of the distributed storage system, these parameters have great influence on the use of software, in the case of extremely pursuing performance, a small change of one parameter may cause a huge fluctuation of performance, and in the default case, the default value of the software setting is a moderate configuration, which is suitable for most common hardware in the market at present.
When the performance of the distributed storage system is optimized, developers who are not deeply knowledgeable about the distributed storage system at present cannot adjust the distributed storage system to an optimal state according to various complex software and hardware scenes. One cluster is generally composed of dozens of hosts, hundreds of Object Storage Devices (OSD) are individually optimized, huge workload is generated, and under the conditions of close online time and heavy tasks, general configuration personnel can select default parameters, so that the performance of the distributed Storage system is relatively general, and the better performance advantage of the distributed Storage system cannot be exerted.
Disclosure of Invention
In view of this, the present application provides a parameter configuration method and apparatus for a distributed storage system, and a computer device, and mainly aims to solve the technical problems that when the performance of the distributed storage system is optimized at present, the accuracy and efficiency are low and it is difficult to adjust the distributed storage system to an optimal state according to various complex software and hardware scenarios in a manner of manually configuring parameters.
According to an aspect of the present application, there is provided a parameter configuration method of a distributed storage system, the method including:
acquiring role information corresponding to each host in a cluster for deploying a distributed storage system;
collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host;
inquiring a target calculation rule corresponding to a target parameter needing to be configured in the distributed storage system, wherein each parameter needing to be configured in the distributed storage system has a calculation rule which is respectively corresponding to preset calculation rules;
and configuring the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the host respectively.
According to another aspect of the present application, there is provided a parameter configuration apparatus of a distributed storage system, the apparatus including:
the acquisition module is used for acquiring role information corresponding to the hosts in the cluster for deploying the distributed storage system;
the collection module is used for collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host;
the query module is used for querying a target calculation rule corresponding to a target parameter needing to be configured in the distributed storage system, wherein each parameter needing to be configured in the distributed storage system has a calculation rule which is respectively corresponding to the preset calculation rule;
and the configuration module is used for configuring the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the host respectively.
According to yet another aspect of the present application, there is provided a non-transitory readable storage medium having stored thereon a computer program which, when executed by a processor, implements a parameter configuration method of the above-described distributed storage system.
According to still another aspect of the present application, there is provided a computer apparatus including a nonvolatile readable storage medium, a processor, and a computer program stored on the nonvolatile readable storage medium and executable on the processor, the processor implementing the parameter configuration method of the distributed storage system when executing the program.
By means of the technical scheme, the parameter configuration method and device of the distributed storage system and the computer equipment are provided. Compared with the prior art, the method and the device can automatically collect the software and hardware information corresponding to the host and the own role according to the role information corresponding to the host in the cluster for deploying the distributed storage system. And the parameters are automatically configured efficiently and accurately by combining the calculation rules corresponding to each parameter to be configured in the distributed storage system. The parameter configuration of the distributed storage system does not need to be manually carried out again, and the learning cost can be saved. And through the automatic parameter configuration process, the distributed storage system can be quickly adjusted to the optimal state, and the better performance advantage of the distributed storage system is exerted in time.
The above description is only an outline of the technical solution of the present application, and the present application can be implemented in accordance with the content of the description so as to make the technical means of the present application more clearly understood, and the detailed description of the present application will be given below so that the above and other objects, features, and advantages of the present application can be more clearly understood.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application to the disclosed embodiment. In the drawings:
fig. 1 is a schematic flowchart illustrating a parameter configuration method of a distributed storage system according to an embodiment of the present application;
fig. 2 is a schematic flowchart illustrating a parameter configuration method of another distributed storage system according to an embodiment of the present application;
fig. 3 shows a schematic flowchart of a parameter configuration apparatus of a distributed storage system according to an embodiment of the present application.
Detailed Description
The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Aiming at the technical problems that the accuracy and efficiency are low and the distributed storage system is difficult to adjust to the optimal state according to various complex software and hardware scenes by manually configuring parameters during performance optimization of the existing distributed storage system, the embodiment provides a parameter configuration method of the distributed storage system, as shown in fig. 1, the method comprises the following steps:
101. and acquiring role information corresponding to the hosts in the cluster for deploying the distributed storage system.
The role information corresponding to the host may include a role of a monitoring node, a role of an access node, a role of a storage node, and the like.
For example, after the cluster deploys the distributed storage system, basic information of the current cluster is obtained, including IP addresses of all hosts, host names, whether the hosts play roles in the cluster as monitoring nodes, storage nodes or access nodes, and the like.
The execution subject of the embodiment may be a device or equipment for automatically configuring parameters of the distributed storage system, and may be configured on the distributed storage system side, specifically for performing performance tuning on the distributed storage system, so as to exert better performance advantages of the distributed storage system.
102. And collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host.
There may be multiple hosts in a cluster and there may be differences in the functional roles each host plays in the cluster. For the distributed storage system configured in the cluster, when configuring corresponding parameters for the distributed storage system, the distributed storage system is mainly configured by combining related software and hardware information of the functions of the hosts in the cluster, and the software and hardware information is rarely referred to for use. Therefore, in this embodiment, for different roles of the host, the corresponding software and hardware information is collected, and then the parameter configuration of the distributed storage system is performed based on the software and hardware information with pertinence. By the method, certain resource calling waste can be saved, software and hardware information most relevant to parameter configuration can be quickly acquired, and the time consumed for collecting the software and hardware information is reduced.
103. And inquiring a target calculation rule corresponding to the target parameter needing to be configured in the distributed storage system.
Each parameter needing to be configured in the distributed storage system has a calculation rule which is respectively corresponding to the preset calculation rule. The calculation rules may include parameter determination methods, calculation formulas, and the like, and the calculation rules may be partially the same or completely different, and are preset according to actual requirements. For example, for each parameter that needs to be configured in the distributed storage system, the corresponding calculation rule may be edited in advance, and then these mapping relationships are stored, and then the corresponding calculation rule is directly invoked when which parameter needs to be configured.
104. And configuring target parameters according to the inquired target calculation rule and the collected software and hardware information corresponding to the host.
For example, a determination method of a target parameter is obtained from a target calculation rule, software and hardware parameters related to the determination method are obtained from collected software and hardware information, then analysis and determination are performed according to the determination method, and finally the target parameter is determined for configuration and automatically filled in a corresponding parameter configuration item in a distributed storage system.
Compared with the prior art, the parameter configuration method of the distributed storage system in the embodiment can automatically collect software and hardware information corresponding to the roles of the host and the host, and efficiently and accurately automatically configure the parameters by combining the calculation rules corresponding to each parameter to be configured in the distributed storage system. The parameter configuration of the distributed storage system does not need to be manually carried out again, and the learning cost can be saved. And through the automatic parameter configuration process, the distributed storage system can be quickly adjusted to the optimal state, and the better performance advantage of the distributed storage system is exerted in time.
Further, as a refinement and an extension of the specific implementation of the foregoing embodiment, in order to fully describe the specific implementation process in this embodiment, taking a Ceph distributed storage system as an example, another parameter configuration method of a distributed storage system is provided, as shown in fig. 2, the method includes:
201. and obtaining host identification information corresponding to the hosts from a control center of a cluster for deploying the Ceph distributed storage system.
The host identification information may include a host name, an ID number, and the like.
The distributed storage system of the Ceph is new storage software, comprises two forms of block storage and object storage, the market share of the Ceph is continuously increased, and more companies are used, and the development prospect is wide due to the characteristics of open source, various configurations, clear structures and the like. Because the disposition of Ceph depends on the hardware conditions, such as which CPU is used, how large hard disk is, whether the network is a single network card or a dual network card, the raid card condition, whether raid is done, and other hardware conditions. And the software conditions such as the deployed operating system version, the Docker use or not, the civetweb use or not and the like are also depended.
There are many parameters in the Ceph configuration, which have a great influence on the use of software, and in the case of extreme pursuit of performance, a small change in one parameter may cause a huge fluctuation in performance, and in the default case, the default value of the software setting is a moderate configuration, which is suitable for most common hardware in the market at present. However, this results in more general Ceph performance and does not provide the advantages of Ceph.
In order to solve the above problems, this embodiment proposes an improvement measure for Ceph having many parameters, and each tuning depends on various software and hardware conditions currently deployed. The deployment node executing the method of the embodiment is called a central computing node, and the node can be connected with all hosts in the cluster and collects software and hardware information of each host in the cluster. And then configuring parameters of the Ceph according to calculation so as to achieve the condition that the Ceph performance is optimal. The process shown in steps 201 to 206 may be specifically performed.
202. And identifying characters in the acquired host identification information, and acquiring role information corresponding to the host identification information by inquiring a preset list.
Wherein, different specific characters are stored in the preset list and respectively correspond to the role information which is configured in advance.
For example, the IP address and the host name of each host in the cluster are obtained from a control center (for controlling each host in the cluster) of the cluster; and then inquiring the corresponding role according to the host name. If the name contains a specific monitor character, the name corresponds to a monitoring node and is used for monitoring the Ceph cluster state; the name comprises a specific OSD character, and the specific OSD character corresponds to a storage node and is used for storing data of the Ceph cluster; the inclusion of a specific rgw character in the name corresponds to an access node that acts as a gateway for the Ceph cluster. The role information corresponding to the hosts in the cluster can be accurately acquired through the table look-up mode.
203. And collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host.
Optionally, step 203 may specifically include: if the target host is in the storage node role, collecting the size and the rotating speed of a hard disk of the target host; and if the target host is in the role of the access node, collecting whether the target host is deployed through the Docker container.
For example, different collection commands are sent to the respective hosts according to the role. For example, the storage node must collect the size and rotation speed of the hard disk, and the access node does not need to collect the index. The access node needs to collect whether it is deployed through Docker, while the storage node does not. By the method, certain resource calling waste can be saved, software and hardware information most relevant to parameter configuration can be quickly acquired, and the time consumed for collecting the software and hardware information is reduced.
Wherein, the form of the collection command can be as shown in table 1:
TABLE 1
And sending a hardware collection command corresponding to the role of each host computer in the cluster, collecting hardware information such as a CPU (central processing unit), a hard disk, a raid card and the like of the corresponding host computer, and storing the hardware information in a file. And sending a software collection command corresponding to the role of each host in the cluster, collecting the operating system version of the corresponding host, judging whether software information such as Docker is used or not, and storing the software information in a file.
204. And inquiring a target calculation rule corresponding to the target parameter needing to be configured in the Ceph distributed storage system.
Optionally, the target parameter may specifically be a system parameter of a cluster operating system or a software parameter of a single OSD. Therefore, the embodiment can realize automatic configuration of system parameters of the operating system of the Ceph cluster and automatic configuration of software parameters of each OSD in the Ceph cluster, thereby comprehensively improving the performance of the Ceph cluster.
For example, parameters that need to be configured in the Ceph distributed storage system can be divided into the above-described system parameters (global part) and software parameters (osd.x part). The global part is each system parameter required to be set by a host operating system of the ceph software and is configured aiming at the whole cluster; the OSD.X part is each software parameter which needs to be configured with the Ceph of the host corresponding to the OSD.X, and is configured according to the host corresponding to the OSD.X, the value of the X determines which host corresponds to the OSD.X part, and the software parameters of the OSD.X part are configured according to each host.
Wherein the global portion can include: "mon _ compact _ on _ start" (whether the supervisor performs log compression at startup), "mon _ OSD _ reporter _ subtree _ level" (OSD redundancy report level), "filescore _ fd _ cache _ size" (file open interface cache size), "filescore _ op _ threads" (file operation concurrency), "filescore _ queue _ max _ bytes" (file operation maximum queuing number), "filescore _ max _ sync _ interval" (file maximum synchronization interval), "OSD _ pool _ default _ flag _ num" (pool default pg number), "OSD _ pool _ default _ pgp _ num" (pool default pgp number), "OSD _ pool _ default _ size" (pool default size), "OSD _ pool _ default _ min _ size" (pool default minimum size), "OSD _ pg _ object _ context _ cache _ count" (pg context buffer number), "OSD _ map _ message _ max" (maximum mapping relationship message buffer number), "OSD _ recovery _ threads" (single OSD maximum data recovery thread), and the like.
The osd.x part may include: "devs" (device binding number), "host" (host name), "OSD journal" (OSD buffer location), "weight" (weight).
205. Analyzing the inquired target calculation rule, extracting software and hardware parameters used for calculation from the collected software and hardware information, and determining the target parameters according to a judgment mode and/or a calculation formula in the target calculation rule.
For example, the target calculation rule is analyzed, the parameters used for calculation are extracted from the collected software and hardware information of each host, and then the specific values of the system parameters or the software parameters which need to be set currently are obtained according to the judgment mode or the calculation formula in the calculation rule.
Optionally, if the target parameter is a system parameter of an OSD redundancy report level (e.g., mon _ OSD _ reporter _ subtree _ level), step 205 may specifically include: firstly, acquiring the number of hosts in a cluster and the OSD number corresponding to the hosts; then, calculating the total OSD number in the cluster according to the obtained host number and the obtained OSD number; if the number of the hosts is smaller than a preset host number threshold value and the total OSD number is larger than a preset OSD number threshold value, configuring the target parameter to be an OSD fault reporting level so as to determine that an OSD fault is abnormal when the same OSD fault is reported by the OSD with the number corresponding to at least the preset OSD number threshold value; if the number of the HOSTs is larger than the preset HOST number threshold, the target parameter is configured to be HOST HOST fault reporting level, so that when the same OSD faults are reported by the HOSTs of which the number corresponds to at least the preset HOST number threshold, the OSD faults are determined to be abnormal.
For example, if there is only one host in the collected host information, and the low-allocation type is 3 OSDs, then "mon _ OSD _ reporter _ subtree _ level" is set as OSD, and when a cluster fails to have a hard disk, the cluster will immediately report the failure information. If there are 30 HOSTs and each HOST is 10 OSD models, then "mon _ OSD _ reporter _ subtree _ level" will be set as HOST, which means that only when multiple HOSTs report hard disk errors at the same time, the monitoring HOST will set hard disk errors. This is because if the value is set to OSD when the number of hard disks is large, there may be a short period of no response of the hard disks during performance jitter, and after the performance jitter, the hard disks recover to normal, and if this process is reported as a hard disk failure, this may result in a cycle of frequent failure, recovery, and failure recovery … of the cluster.
Because of the different cluster sizes, the parameter settings may be different here. This parameter is the minimum unit for reporting faults, and when 3 different OSDs report a certain OSD fault, the system considers the OSD fault. When set to HOST, the system considers an OSD fault when 3 different HOSTs report the OSD fault. Based on the rule, if only 1 or 2 hosts are configured, the standard reported by 3 host can never be reached, and when an OSD fault exists, the fault can not be reported, so that the system is unavailable. By the method, the dynamic change can be realized according to the cluster scale, and the parameters meeting the service requirements can be accurately set.
Optionally, if the target parameter is a software parameter of the weight of the target OSD, step 205 may specifically include: and calculating target parameters by using a preset formula weight ═ a × disk _ speed/b + size/average _ size, wherein weight is the weight of the target OSD, disk _ speed is the hard disk rotating speed corresponding to the target OSD, size is the hard disk size corresponding to the target OSD, average _ size is the average value of the hard disk sizes corresponding to the OSDs in the cluster, and a and b are preset constants. For example, a may be 5, b may be 500000, and the like.
By the formula, the capacity of the size and the rotating speed of the OSD hard disk and the capacity ratio of each OSD in the cluster are considered, and the corresponding weight of the OSD can be accurately calculated. Subsequently, a more reasonable Placement Group (PG) can be assigned according to the OSD weight, so as to achieve optimal performance of the Ceph distributed storage system.
In addition to the two exemplary parameter determination methods described above, in order to richly explain the parameter determination process in the present embodiment, several examples are given to facilitate understanding.
And determining a parameter related to the "OSD _ recovery _ max _ active" (maximum number of activations for OSD recovery). When the cluster is large and the performance is good, the value is set to be a large value, so that the data recovery process is fast after the OSD fault. When the cluster is small and the performance is poor, the value is set to be a small value, so that the data recovery is slow after the fault, but the request for normal access of the cluster is not influenced.
For example, if the host hard disk speed is less than 500000 and the total number of hard disks is less than 30, then osd _ recovery _ max _ active is 1; if the host hard disk speed is greater than or equal to 500000 and less than 600000 and the total number of hard disks is greater than or equal to 30 and less than 60, then osd _ recovery _ max _ active is 3.
Regarding the parameter determination of "filescore _ fd _ cache _ size" (the cache size of the file opening interface), when the host memory is large, the value is set to be a large value, so that the success rate and the response speed of file opening can be faster; when the memory of the host is small, the value is set to be a small value so as to limit direct error reporting when files occupying more caches are opened and reduce the risk of downtime.
For example, if the memory size of the host is smaller than 64G, filescore _ fd _ cache _ size is 2; if the memory size of the target host is larger than or equal to 64G and smaller than 96G, filescore _ fd _ cache _ size is 4; if the memory size of the target host is greater than or equal to 96G and less than 128G, filescore _ fd _ cache _ size is 8.
206. And configuring according to the determined target parameters.
And performing corresponding calculation according to the collected software and hardware information of the host in the cluster according to the system parameters and the software parameters which need to be set, and performing automatic setting. After the overall configuration is completed, performance tuning is complete.
Compared with the prior art, the parameter configuration method of the distributed storage system in the embodiment can automatically collect software and hardware information corresponding to the roles of the host and the host, and efficiently and accurately automatically configure the parameters by combining the calculation rules corresponding to each parameter to be configured in the distributed storage system. The parameter configuration of the distributed storage system does not need to be manually carried out again, and the learning cost can be saved. And through the automatic parameter configuration process, the distributed storage system can be quickly adjusted to the optimal state, and the better performance advantage of the distributed storage system is exerted in time.
Further, as a specific implementation of the method shown in fig. 1 to fig. 2, an embodiment of the present application provides a parameter configuration apparatus of a distributed storage system, as shown in fig. 3, the apparatus includes: an acquisition module 31, a collection module 32, a query module 33, and a configuration module 34.
The acquiring module 31 may be configured to acquire role information corresponding to each host in a cluster in which the distributed storage system is deployed;
a collecting module 32, configured to collect software and hardware information corresponding to the host and the role of the host according to the role information corresponding to the host;
the query module 33 is configured to query a target calculation rule corresponding to a target parameter that needs to be configured in the distributed storage system, where each parameter that needs to be configured in the distributed storage system has a calculation rule that is preset and corresponding to each parameter;
the configuration module 34 is configured to configure the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the hosts, respectively.
In a specific application scenario, optionally, the target parameter may be a system parameter of a cluster operating system or a software parameter of a single object storage device OSD;
a configuration module 34, specifically configured to analyze the target calculation rule, extract software and hardware parameters used for calculation from the collected software and hardware information, and determine the target parameters according to a determination manner and/or a calculation formula in the target calculation rule; and then configuring according to the determined target parameters.
In a specific application scenario, the obtaining module 31 may be specifically configured to obtain host identifier information corresponding to each host from a control center of the cluster; and then identifying characters in the host identification information, and acquiring role information corresponding to the host identification information by inquiring a preset list, wherein different specific characters are stored in the preset list and correspond to the role information which is configured in advance respectively.
In a specific application scenario, the collecting module 32 is specifically configured to collect the size and the rotation speed of the hard disk of the target host if the target host has a storage node role; and if the target host is in the role of the access node, collecting whether the target host is deployed through a Docker container.
In a specific application scenario, the configuration module 34 is further specifically configured to, if the target parameter is a system parameter at an OSD redundancy report level, first obtain the number of hosts in the cluster and the respective OSD numbers of the hosts; then, calculating the total OSD number in the cluster according to the host number and the OSD number; if the number of the hosts is smaller than a preset host number threshold and the total OSD number is larger than a preset OSD number threshold, configuring the target parameter to be an OSD error reporting level so as to determine that an OSD fault abnormality occurs when the same OSD is reported by the number of OSD corresponding to at least the preset OSD number threshold; and if the number of the HOSTs is larger than a preset HOST number threshold, configuring the target parameter as a HOST HOST fault reporting level so as to determine that OSD fault abnormality occurs when the same OSD faults are reported by the HOSTs of which the number corresponds to at least the preset HOST number threshold.
In a specific application scenario, the configuration module 34 may be further configured to, if the target parameter is a software parameter of the weight of the target OSD, calculate the target parameter by using a preset formula weight ═ a × disk _ speed/b + size/average _ size, where weight is the weight of the target OSD, disk _ speed is a hard disk rotation speed corresponding to the target OSD, size is a hard disk size corresponding to the target OSD, average _ size is an average value of the hard disk sizes corresponding to the OSDs in the cluster, and a and b are preset constants.
In a specific application scenario, optionally, the distributed storage system may be a Ceph distributed storage system.
It should be noted that other corresponding descriptions of the functional units related to the parameter configuration apparatus of the distributed storage system provided in this embodiment may refer to the corresponding descriptions in fig. 1 to fig. 2, and are not described again here.
Based on the methods shown in fig. 1 and fig. 2, correspondingly, the embodiment of the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for configuring parameters of the distributed storage system shown in fig. 1 and fig. 2 is implemented.
Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method of the embodiments of the present application.
Based on the foregoing methods shown in fig. 1 and fig. 2 and the virtual device embodiment shown in fig. 3, to achieve the foregoing object, an embodiment of the present application further provides a computer device, which may specifically be a personal computer, a server, a network device, and the like, where the entity device includes a storage medium and a processor; a storage medium for storing a computer program; a processor for executing a computer program to implement the parameter configuration method of the distributed storage system as shown in fig. 1 and 2.
Optionally, the computer device may further include a user interface, a network interface, a camera, Radio Frequency (RF) circuitry, sensors, audio circuitry, a WI-FI module, and so forth. The user interface may include a Display screen (Display), an input unit such as a keypad (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (e.g., a bluetooth interface, WI-FI interface), etc.
It will be understood by those skilled in the art that the computer device structure provided in the present embodiment is not limited to the physical device, and may include more or less components, or combine some components, or arrange different components.
The storage medium may further include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-described physical devices, and supports the operation of the information processing program as well as other software and/or programs. The network communication module is used for realizing communication among components in the storage medium and other hardware and software in the entity device.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present application can be implemented by software plus a necessary general hardware platform, and can also be implemented by hardware. By applying the technical scheme of the application, compared with the prior art, the embodiment can automatically collect software and hardware information corresponding to the roles of the host and the host, and efficiently and accurately automatically configure the parameters by combining the calculation rules corresponding to each parameter to be configured in the distributed storage system. The parameter configuration of the distributed storage system does not need to be manually carried out again, and the learning cost can be saved. And through the automatic parameter configuration process, the distributed storage system can be quickly adjusted to the optimal state, and the better performance advantage of the distributed storage system is exerted in time.
Those skilled in the art will appreciate that the figures are merely schematic representations of one preferred implementation scenario and that the blocks or flow diagrams in the figures are not necessarily required to practice the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario may be distributed in the devices in the implementation scenario according to the description of the implementation scenario, or may be located in one or more devices different from the present implementation scenario with corresponding changes. The modules of the implementation scenario may be combined into one module, or may be further split into a plurality of sub-modules.
The above application serial numbers are for description purposes only and do not represent the superiority or inferiority of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any variations that can be made by those skilled in the art are intended to fall within the scope of the present application.
Claims (10)
1. A method for configuring parameters of a distributed storage system, comprising:
acquiring role information corresponding to each host in a cluster for deploying a distributed storage system;
collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host;
inquiring a target calculation rule corresponding to a target parameter needing to be configured in the distributed storage system, wherein each parameter needing to be configured in the distributed storage system has a calculation rule which is respectively corresponding to preset calculation rules;
and configuring the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the host respectively.
2. The method of claim 1, wherein the target parameter is a system parameter of a cluster operating system or a software parameter of a single object storage device OSD;
configuring the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the host, specifically comprising:
analyzing the target calculation rule, extracting software and hardware parameters used for calculation from the collected software and hardware information, and determining the target parameters according to a judgment mode and/or a calculation formula in the target calculation rule;
and configuring according to the determined target parameters.
3. The method of claim 2, wherein the obtaining role information corresponding to each host in the cluster in which the distributed storage system is deployed further comprises:
acquiring host identification information corresponding to the hosts from a control center of the cluster;
identifying characters in the host identification information, and acquiring role information corresponding to the host identification information by inquiring a preset list, wherein different specific characters are stored in the preset list and correspond to the role information which is configured in advance respectively.
4. The method according to claim 3, wherein the collecting software and hardware information corresponding to the host and its own role according to the role information corresponding to the host specifically comprises:
if the target host is in the storage node role, collecting the size and the rotating speed of a hard disk of the target host;
and if the target host is in the role of the access node, collecting whether the target host is deployed through a Docker container.
5. The method according to claim 2, wherein if the target parameter is a system parameter at an OSD redundancy report level, the parsing the target calculation rule, extracting software and hardware parameters used for calculation from the collected software and hardware information, and determining the target parameter according to a determination manner and/or a calculation formula in the target calculation rule specifically includes:
acquiring the number of hosts in the cluster and the OSD number corresponding to the hosts;
calculating the total OSD number in the cluster according to the host number and the OSD number;
if the number of the hosts is smaller than a preset host number threshold and the total OSD number is larger than a preset OSD number threshold, configuring the target parameter to be an OSD error reporting level so as to determine that an OSD fault abnormality occurs when the same OSD is reported by the number of OSD corresponding to at least the preset OSD number threshold;
and if the number of the HOSTs is larger than a preset HOST number threshold, configuring the target parameter as a HOST HOST fault reporting level so as to determine that OSD fault abnormality occurs when the same OSD faults are reported by the HOSTs of which the number corresponds to at least the preset HOST number threshold.
6. The method according to claim 2, wherein if the target parameter is a software parameter of a weight of a target OSD, the analyzing the target calculation rule, extracting software and hardware parameters used for calculation from the collected software and hardware information, and determining the target parameter according to a determination method and/or a calculation formula in the target calculation rule specifically includes:
and calculating the target parameter by using a preset formula weight ═ a disc _ speed/b + size/average _ size, wherein weight is the weight of the target OSD, disc _ speed is the hard disk rotating speed corresponding to the target OSD, size is the hard disk size corresponding to the target OSD, average _ size is the average value of the hard disk sizes corresponding to the OSDs in the cluster, and a and b are preset constants.
7. The method of any one of claims 1 to 6, wherein the distributed storage system is a Ceph distributed storage system.
8. A parameter configuration apparatus of a distributed storage system, comprising:
the acquisition module is used for acquiring role information corresponding to the hosts in the cluster for deploying the distributed storage system;
the collection module is used for collecting software and hardware information corresponding to the host and the own role according to the role information corresponding to the host;
the query module is used for querying a target calculation rule corresponding to a target parameter needing to be configured in the distributed storage system, wherein each parameter needing to be configured in the distributed storage system has a calculation rule which is respectively corresponding to the preset calculation rule;
and the configuration module is used for configuring the target parameters according to the target calculation rule and the collected software and hardware information corresponding to the host respectively.
9. A non-transitory readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the parameter configuration method of the distributed storage system according to any one of claims 1 to 7.
10. A computer device comprising a non-volatile readable storage medium, a processor, and a computer program stored on the non-volatile readable storage medium and executable on the processor, wherein the processor implements the parameter configuration method of the distributed storage system according to any one of claims 1 to 7 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911054078.0A CN111045599A (en) | 2019-10-31 | 2019-10-31 | Parameter configuration method and device of distributed storage system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911054078.0A CN111045599A (en) | 2019-10-31 | 2019-10-31 | Parameter configuration method and device of distributed storage system and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111045599A true CN111045599A (en) | 2020-04-21 |
Family
ID=70232814
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911054078.0A Pending CN111045599A (en) | 2019-10-31 | 2019-10-31 | Parameter configuration method and device of distributed storage system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111045599A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111654410A (en) * | 2020-04-28 | 2020-09-11 | 长沙证通云计算有限公司 | Gateway request monitoring method, device, equipment and medium |
| CN112306815A (en) * | 2020-11-16 | 2021-02-02 | 新华三大数据技术有限公司 | Method, device, equipment and medium for monitoring IO (input/output) information between OSD (on Screen display) side master and slave in Ceph |
| CN113282241A (en) * | 2021-05-26 | 2021-08-20 | 上海仪电（集团）有限公司中央研究院 | Ceph distributed storage-based hard disk weight optimization method and device |
| CN113760188A (en) * | 2021-07-30 | 2021-12-07 | 浪潮电子信息产业股份有限公司 | Parameter adjusting and selecting method, system and device for distributed storage system |
| CN113791890A (en) * | 2020-11-06 | 2021-12-14 | 北京沃东天骏信息技术有限公司 | Container distribution method and device, electronic equipment and storage medium |
| CN113992703A (en) * | 2021-09-29 | 2022-01-28 | 浪潮电子信息产业股份有限公司 | Distributed storage system parameter optimization method and related components |
| CN114564460A (en) * | 2022-02-25 | 2022-05-31 | 苏州浪潮智能科技有限公司 | Parameter tuning method, device, equipment and medium based on distributed storage system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126407A (en) * | 2016-06-22 | 2016-11-16 | 西安交通大学 | A kind of performance monitoring Operation Optimization Systerm for distributed memory system and method |
| CN107071030A (en) * | 2017-04-19 | 2017-08-18 | 广东浪潮大数据研究有限公司 | The dispositions method and system of a kind of Ceph distributed memory systems |
| CN109067877A (en) * | 2018-08-03 | 2018-12-21 | 平安科技（深圳）有限公司 | A kind of control method, server and the storage medium of cloud computing platform deployment |
| CN109194506A (en) * | 2018-08-16 | 2019-01-11 | 北京京东尚科信息技术有限公司 | Block chain network dispositions method, platform and computer storage medium |
| US20190020540A1 (en) * | 2017-07-14 | 2019-01-17 | Quanta Cloud Technology Inc. | Mechanism for hardware configuration and software deployment |
-
2019
- 2019-10-31 CN CN201911054078.0A patent/CN111045599A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126407A (en) * | 2016-06-22 | 2016-11-16 | 西安交通大学 | A kind of performance monitoring Operation Optimization Systerm for distributed memory system and method |
| CN107071030A (en) * | 2017-04-19 | 2017-08-18 | 广东浪潮大数据研究有限公司 | The dispositions method and system of a kind of Ceph distributed memory systems |
| US20190020540A1 (en) * | 2017-07-14 | 2019-01-17 | Quanta Cloud Technology Inc. | Mechanism for hardware configuration and software deployment |
| CN109067877A (en) * | 2018-08-03 | 2018-12-21 | 平安科技（深圳）有限公司 | A kind of control method, server and the storage medium of cloud computing platform deployment |
| CN109194506A (en) * | 2018-08-16 | 2019-01-11 | 北京京东尚科信息技术有限公司 | Block chain network dispositions method, platform and computer storage medium |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111654410A (en) * | 2020-04-28 | 2020-09-11 | 长沙证通云计算有限公司 | Gateway request monitoring method, device, equipment and medium |
| CN113791890A (en) * | 2020-11-06 | 2021-12-14 | 北京沃东天骏信息技术有限公司 | Container distribution method and device, electronic equipment and storage medium |
| CN112306815A (en) * | 2020-11-16 | 2021-02-02 | 新华三大数据技术有限公司 | Method, device, equipment and medium for monitoring IO (input/output) information between OSD (on Screen display) side master and slave in Ceph |
| CN112306815B (en) * | 2020-11-16 | 2023-07-25 | 新华三大数据技术有限公司 | Method, device, equipment and medium for monitoring IO information between OSD side and master slave in Ceph |
| CN113282241A (en) * | 2021-05-26 | 2021-08-20 | 上海仪电（集团）有限公司中央研究院 | Ceph distributed storage-based hard disk weight optimization method and device |
| CN113282241B (en) * | 2021-05-26 | 2024-04-09 | 上海仪电（集团）有限公司中央研究院 | Hard disk weight optimization method and device based on Ceph distributed storage |
| CN113760188A (en) * | 2021-07-30 | 2021-12-07 | 浪潮电子信息产业股份有限公司 | Parameter adjusting and selecting method, system and device for distributed storage system |
| CN113992703A (en) * | 2021-09-29 | 2022-01-28 | 浪潮电子信息产业股份有限公司 | Distributed storage system parameter optimization method and related components |
| CN113992703B (en) * | 2021-09-29 | 2024-04-05 | 浪潮电子信息产业股份有限公司 | Distributed storage system parameter tuning method and related components |
| CN114564460A (en) * | 2022-02-25 | 2022-05-31 | 苏州浪潮智能科技有限公司 | Parameter tuning method, device, equipment and medium based on distributed storage system |
| CN114564460B (en) * | 2022-02-25 | 2024-01-19 | 苏州浪潮智能科技有限公司 | Parameter tuning method, device, equipment and medium based on distributed storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111045599A (en) | Parameter configuration method and device of distributed storage system and computer equipment | |
| EP3606008B1 (en) | Method and device for realizing resource scheduling | |
| CN108182140B (en) | Determining and monitoring performance capabilities of computer resource services | |
| JP4345313B2 (en) | Operation management method of storage system based on policy | |
| US20180375726A1 (en) | Resource Configuration Method, Virtualized Network Function Manager, and Element Management System | |
| US9335932B2 (en) | Storage unit selection for virtualized storage units | |
| US20180336113A1 (en) | Monitoring applications running on containers | |
| CN109495306B (en) | Method and equipment for expanding capacity of service network | |
| US10038593B2 (en) | Method and system for recovering virtual network | |
| US10289441B1 (en) | Intelligent scale-out federated restore | |
| US20140032753A1 (en) | Computer system and node search method | |
| CN111597099B (en) | Non-invasive simulation method for monitoring running quality of application deployed on cloud platform | |
| US9858166B1 (en) | Methods, systems, and computer readable mediums for optimizing the deployment of application workloads in a converged infrastructure network environment | |
| CN107666493B (en) | Database configuration method and equipment thereof | |
| CN110719194B (en) | Network data analysis method and device | |
| US11381451B2 (en) | Methods, systems, and computer readable mediums for selecting and configuring a computing system to support a replicated application | |
| US20200012509A1 (en) | Virtualised software application performance | |
| CN111625419B (en) | Log acquisition method, system, equipment and computer readable storage medium | |
| US20140201554A1 (en) | Power source management device, power source management method, and computer-readable recording medium | |
| US9641595B2 (en) | System management apparatus, system management method, and storage medium | |
| US10503535B1 (en) | Requirements based performance monitoring for virtual machine networks | |
| US10432548B2 (en) | Workload deployment in computing networks | |
| CN111212087A (en) | Method, device, equipment and storage medium for determining login server | |
| US9928097B1 (en) | Methods, systems, and computer readable mediums for defining and updating a virtual computing system comprising distributed resource components | |
| TWI507889B (en) | Management node , method , machine 一accessible medium and system for reservation and execution image writing of native computing devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |