CN111858538B

CN111858538B - Method, device, equipment and medium for configuring BeeGFS quota by cluster

Info

Publication number: CN111858538B
Application number: CN202010564597.8A
Authority: CN
Inventors: 王金喜
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2020-06-19
Filing date: 2020-06-19
Publication date: 2022-05-24
Anticipated expiration: 2040-06-19
Also published as: CN111858538A

Abstract

The invention discloses a method for configuring BeeGFS quotas for a cluster, which comprises the following steps: configuring nodes deploying BeeGFS service ends in the cluster as mutual secrecy-free nodes, and configuring node environments deploying the nodes deploying the BeeGFS service ends; configuring NIS service for a node deploying a BeeGFS service end, and testing whether the NIS service is successfully configured; in response to the success of NIS service configuration, closing BeeGFS server-side service and BeeGFS client-side service, modifying quota configuration of the BeeGFS server-side, and then restarting the BeeGFS server-side service and the BeeGFS client-side service; creating a user in a file system of a BeeGFS server; restarting management service in BeeGFS server side service, and checking whether error log information appears in the log; and responding to the log information without error report in the log, and setting a quota for the user at the BeeGFS client. The invention also discloses a device, equipment and a medium. The method and the device can improve the success rate of configuring the BeeGFS quota, improve the efficiency and improve the availability of the file system.

Description

Method, device, equipment and medium for configuring BeeGFS quota by cluster

Technical Field

The present invention relates to the field of file service systems, and in particular, to a method, an apparatus, a device, and a medium for configuring a BeeGFS quota for a cluster.

Background

The BeeGFS is the leading parallel cluster file system, the development focus of which is performance, and which is designed to be very easy to install and manage. BeeGFS is one of the solutions if I/O intensive workloads are required. A distributed metadata architecture is implemented to improve the extended performance and flexibility of applications. The beegfr file system is more concerned with the performance of the file system than other parallel file systems.

In some prior art techniques, beegfr allows system-wide disk space allocation quotas and block file numbers to be defined on a per user or per group basis. This can be used to organize users in different access layers with different levels of restriction and can also prevent individuals from using the resources of all file systems individually. Quota enforcement allows quota limits to be defined and applied throughout the system. After this function is enabled, the BeeGFS management daemon will periodically collect quota reports from all storage targets, check if the quota limit is exceeded, and notify the rest of the system that the user is no longer allowed to use more resources. BeeGFS quota management relies on quota data provided by the underlying file system of the storage server target. Thus, the functionality of such file systems determines which quota types a BeeGFS is able to manage.

However, through multiple experiments, the success rate of the BeeGFS on the configuration quota of the high-performance cluster is not high, and the BeeGFS fails most of the time, even if the client checks that the configuration is good, the written file detection is often unsuccessful, and the usability of the BeeGFS in the cluster is influenced.

Disclosure of Invention

In view of this, an object of the embodiments of the present invention is to provide a method for improving a cluster configuration beegfr quota, which solves the problem of high failure rate of configuring the beegfr quota, and greatly improves a success rate of configuring the beegfr quota by the cluster.

Based on the above object, an aspect of the present invention provides a method for configuring a BeeGFS quota by a cluster, where the method includes:

configuring nodes deploying BeeGFS service ends in the cluster as mutual secrecy-free nodes, and configuring node environments deploying the nodes deploying the BeeGFS service ends;

configuring NIS (Network Information Services) Services for nodes deploying BeeGFS service terminals, and testing whether the NIS Services are configured successfully;

in response to the success of NIS service configuration, closing BeeGFS service end service and BeeGFS client service, modifying quota configuration of the BeeGFS service end, and then restarting the BeeGFS service end service and the BeeGFS client service;

Creating a user in a file system of a BeeGFS server;

restarting management service in BeeGFS server side service, and checking whether error log information appears in the log;

and responding to the log information without error report, and setting a quota for the user at the BeeGFS client.

In some embodiments of the method for configuring a beegfr quota by a cluster of the present invention, configuring a node environment for deploying a node of a beegfr server further includes:

and closing and disabling the firewall, the NetworkManager and selinux services of the nodes in the cluster.

In some embodiments of the method for cluster provisioning of beegfr quotas of the present invention, in response to a successful NIS service provisioning, closing the begfs server service and the begfs client service and modifying the quota provisioning of the begfs server, and then restarting the begfs server service and the begfs client service further comprises:

and in response to successful configuration, closing BeeGFS client service, closing BeeGFS server service, modifying quota configuration of the BeeGFS server, opening BeeGFS server service, and opening BeeGFS client service.

In some embodiments of the method for configuring a beegfr quota for a cluster of the present invention, in response to that no error log information occurs in the log, setting a quota for the user at the begfs client further includes:

Detecting whether the quota is effective or not by the management service at preset time intervals.

In some embodiments of the method for configuring a BeeGFS quota by a cluster of the present invention, the method further comprises:

and responding to the failure of NIS service configuration, and checking whether nodes deploying the BeeGFS service end in the cluster are mutually secret-free or not.

In some embodiments of the method for configuring a BeeGFS quota by a cluster of the present invention, in response to that no error-reporting log information occurs in the log, setting a quota for the user at the BeeGFS client further includes:

detecting whether the quota is effective;

and in response to the quota not being validated, checking whether the node environment of the node deploying the BeeGFS server is correct.

In another aspect of the embodiments of the present invention, a device for configuring a BeeGFS quota for a cluster is further provided, where the device includes:

the environment configuration module is configured to configure nodes deploying BeeGFS service ends in the cluster as mutual secrecy-free nodes and configure node environments deploying the nodes deploying the BeeGFS service ends;

the NIS service configuration module is configured to configure NIS service for the node deploying the BeeGFS service end and test whether the NIS service is configured successfully;

the quota configuration modification module is configured to respond to the successful configuration of the NIS service, close BeeGFS server-side service and BeeGFS client-side service, modify the quota configuration of the BeeGFS server-side, and then restart the BeeGFS server-side service and the BeeGFS client-side service;

The method comprises the steps that a user creating module is configured to create a user in a file system of a BeeGFS server side;

the log checking module is configured to restart management services in BeeGFS server side services and check whether error log information appears in the log;

and the quota setting module is configured to set a quota aiming at the user at the BeeGFS client in response to that no error report log information appears in the log.

In some embodiments of the apparatus for configuring a BeeGFS quota for a cluster of the present invention, the modified quota configuring module is further configured to:

In another aspect of the embodiments of the present invention, there is also provided a computer device, including:

at least one processor; and

a memory storing a computer program operable on the processor, the processor executing the program to perform the aforementioned method for configuring the BeeGFS quota by the cluster.

In another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, where the computer-readable storage medium stores a computer program, and is characterized in that the computer program, when executed by a processor, executes the foregoing method for configuring a BeeGFS quota by a cluster.

The invention has at least the following beneficial technical effects: the technical scheme of the invention greatly improves the success rate of the BeeGFS quota, improves the efficiency and improves the availability of the file system.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained according to the drawings without creative efforts.

Fig. 1 shows a schematic block diagram of an embodiment of a method of cluster provisioning a BeeGFS quota in accordance with the present invention;

fig. 2 shows a flowchart of an embodiment of a method for configuring a BeeGFS quota for a cluster according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.

It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it is understood that "first" and "second" are only used for convenience of description and should not be construed as limiting the embodiments of the present invention, and the descriptions thereof in the following embodiments are omitted.

Based on the foregoing object, a first aspect of the embodiments of the present invention provides an embodiment of a method for configuring a BeeGFS quota by a cluster. Fig. 1 is a schematic block diagram illustrating an embodiment of a method for configuring a BeeGFS quota for a cluster according to the present invention. In the embodiment shown in fig. 1, the method comprises at least the following steps:

s100, configuring nodes deploying BeeGFS service ends in the cluster to be mutually secret-free, and configuring a node environment deploying the nodes deploying the BeeGFS service ends;

s200, configuring NIS service for the node of the BeeGFS service end, and testing whether the NIS service is configured successfully;

s300, in response to the success of NIS service configuration, closing BeeGFS server-side service and BeeGFS client-side service, modifying quota configuration of the BeeGFS server-side, and then restarting the BeeGFS server-side service and the BeeGFS client-side service;

s400, creating a user in a file system of a BeeGFS server;

s500, restarting management service in BeeGFS server side service, and checking whether error log information appears in the log;

s600, responding to the fact that the log information of error report does not appear in the log, and setting quota for the user at the BeeGFS client.

In some embodiments of the present invention, according to step S100, it is first ensured that nodes deploying beegfr servers can log in to a cluster without mutual encryption, at least that nodes deploying beegfr servers are ensured to be without mutual encryption, and a node environment of the nodes deploying beegfr servers is configured.

According to step S200, the NIS services in the cluster are configured, and nodes configuring the BeeGFS service end are necessarily incorporated therein. This step is a critical step that would otherwise result in failure of the BeeGFS quota configuration. In the step of configuration, whether the node deploying the BeeGFS server is configured successfully or not needs to be checked, the checking is performed through a command of yptest \ ypwhich, and in addition, a 'ssh common user' is used for detecting whether login can be performed or not, and the following states are generally displayed:

[root@bm02～]#su ddd

Bash-4.2$

then, according to step S300, the service of the beegfr client and the service of the beegfr server are closed, and then the configuration files of the beegfr server and the beegfr client are modified, which is a basic configuration for implementing the begfs quota; then, the server side service and the client side service are started.

According to step S400, a user who uses the file system of the beegfr server is created, which is also a key step. When a common user is created, the home directory of the common user must be created in the BeeGFS file system (and the BeeGFS directory mounted by the client), otherwise, when the client configures the BeeGFS quota, no configuration information can be seen.

After the setting is finished, according to step S500, the mgmtd management service of the beegfr must be restarted, and whether the log has an error log is checked, so that the problem is timely eliminated.

According to step S600, when the error log does not show a problem, a quota setting command is executed at the client, and quota parameter setting is performed for the user and the user group.

The success rate of configuring the BeeGFS quota can be greatly improved according to the steps, and especially the steps S200, S400 and S500 are indispensable.

According to some embodiments of the method for configuring a BeeGFS quota by a cluster of the present invention, configuring a node environment of a node deploying a BeeGFS server further includes:

and closing and disabling the firewall, the network manager and the selinux services of the nodes in the cluster.

In some embodiments of the present invention, the step of configuring the node environment of the node deploying the beegfr server includes ensuring that services such as a firewall, a NetworkManager, and selinux of each node in the cluster are closed and disabled.

According to some embodiments of the method for configuring a beegfr quota by a cluster of the present invention, in response to a successful NIS service configuration, closing the begfs server service and the begfs client service and modifying the quota configuration of the begfs server, and then restarting the begfs server service and the begfs client service further includes:

In some embodiments of the present invention, a BeeGFS client service and a BeeGFS server service are closed in sequence, and then configuration files of the BeeGFS server and the client are modified, which is a basic configuration for realizing a BeeGFS quota; then, the server side service and the client side service are started. Where the order may not be chaotic or otherwise may result in unsuccessful configuration.

According to some embodiments of the method for configuring a beegfr quota by a cluster of the present invention, in response to that no error log information occurs in the log, setting a quota for the user at the begfs client further includes:

and detecting whether the quota is effective or not by the management service at preset time intervals.

In some embodiments of the present invention, after setting the quota parameter, a file is written in a home directory of a user of the begfs client for testing, and it is verified whether the quota is valid. Since the mgmtd manager defaults to performing the test once in 10 minutes after starting up, data test is written after 10 minutes.

According to some embodiments of the method for configuring a beegfr quota by a cluster of the present invention, the method further comprises:

and in response to the NIS service configuration failure, checking whether the nodes deploying the BeeGFS service end in the cluster are mutually secret-free.

In some embodiments of the present invention, fig. 2 is a flowchart illustrating an embodiment of a method for configuring a beegfr quota for a cluster according to the present invention, and as shown in fig. 2, if the NIS service configuration is unsuccessful, it needs to check whether nodes deploying a beegfr service end in the cluster are mutually secret-free again.

According to some embodiments of the method for configuring a BeeGFS quota by a cluster of the present invention, in response to no error-reporting log information appearing in the log, setting a quota for the user at the BeeGFS client further includes:

detecting whether the quota is effective;

and in response to the quota not being validated, checking whether the node environment of the node deploying the BeeGFS service end is correct.

In some embodiments of the present invention, as shown in fig. 2, if it is detected that the quota is not valid, it is required to restart checking whether the node environment of the node deploying the begfs server is correct.

On the other hand, the embodiment of the invention provides an embodiment of a device for configuring a BeeGFS quota for a cluster. The device includes:

The quota configuration modification module is configured to respond to the success of NIS service configuration, close BeeGFS server side service and BeeGFS client side service, modify the quota configuration of the BeeGFS server side, and then restart the BeeGFS server side service and the BeeGFS client side service;

According to some embodiments of the apparatus for configuring a BeeGFS quota for a cluster of the present invention, the modified quota configuring module is further configured to:

In view of the above object, another aspect of the embodiments of the present invention further provides a computer device, including: at least one processor; and a memory storing a computer program operable on the processor, the processor executing the program to perform the aforementioned method for configuring the BeeGFS quota by the cluster.

In another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, where a computer program is stored, and is characterized in that when being executed by a processor, the computer program performs the foregoing method for configuring a BeeGFS quota by a cluster.

Likewise, it will be understood by those skilled in the art that all of the embodiments, features and advantages set forth above with respect to the method for configuring a BeeGFS quota for a cluster according to the present invention equally apply to the apparatus, the computer device and the medium according to the present invention. For the sake of brevity of this disclosure, no further discussion is repeated herein.

It should be particularly noted that, steps in the embodiments of the method, apparatus, device and medium for configuring a BeeGFS quota by a cluster described above may be mutually intersected, replaced, added and deleted, and therefore, the method, apparatus, device and medium for configuring a BeeGFS quota by a cluster, which are transformed by reasonable permutation and combination, shall also belong to the protection scope of the present invention, and shall not limit the protection scope of the present invention to the embodiments.

Finally, it should be noted that, as those skilled in the art can understand, all or part of the processes in the method according to the foregoing embodiments may be implemented by instructing relevant hardware by a computer program, and the program of the method for configuring a BeeGFS quota in a cluster may be stored in a computer-readable storage medium, and when executed, the program may include the processes in the embodiments according to the foregoing methods. The storage medium of the program may be a magnetic disk, an optical disk, a read-only memory (ROM), or a Random Access Memory (RAM). The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments corresponding thereto.

Furthermore, the methods disclosed according to embodiments of the invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions as defined in the method disclosed by an embodiment of the invention.

Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.

Further, it should be understood that the computer-readable storage medium herein (e.g., memory) can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in a variety of forms such as synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.

The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The foregoing are exemplary embodiments of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The numbers of the embodiments disclosed in the above embodiments of the present invention are merely for description, and do not represent the advantages or disadvantages of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant only to be exemplary, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit or scope of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims

1. A method for configuring BeeGFS quota for cluster, the method comprising:

configuring nodes deploying BeeGFS service ends in a cluster as mutual secrecy, and configuring node environments deploying the nodes deploying the BeeGFS service ends;

configuring NIS service for the node deploying the BeeGFS service end, and testing whether the NIS service is configured successfully;

in response to the success of the NIS service configuration, closing BeeGFS service end services and BeeGFS client service, modifying quota configuration of the BeeGFS service end, and then restarting the BeeGFS service end services and the BeeGFS client service;

Creating a user in a file system of the BeeGFS server;

restarting management service in the BeeGFS server side service, and checking whether error log information appears in a log;

and responding to the log without the error report log information, and setting a quota for the user at the BeeGFS client.

2. The method for configuring the BeeGFS quota for the cluster of claim 1, wherein the configuring the node environment of the node deploying the BeeGFS service further comprises:

and closing and forbidding the firewall, the network manager and the selinux service of the nodes in the cluster.

3. The method for cluster provisioning of beegfr quotas according to claim 1, wherein said closing begfs server services and begfs client services and modifying the quota provisioning of the begfs server and subsequently restarting the begfs server services and the begfs client services in response to the NIS service provisioning success further comprises:

and in response to successful configuration, closing the BeeGFS client service, then closing the BeeGFS server service, then modifying quota configuration of the BeeGFS server, and then opening the BeeGFS server service, and then opening the BeeGFS client service.

4. The method for cluster configuration of the beegfr quota according to claim 1, wherein the setting a quota for the user at the begfs client in response to the failure of the error log information in the log further comprises:

and detecting whether the quota is effective or not through the management service every preset time.

5. The method of cluster configuration BeeGFS quota of claim 1, further comprising:

in response to the NIS service configuration being unsuccessful, checking whether the nodes deploying the BeeGFS service side in the cluster are mutually secret-free.

6. The method for cluster configuration of the beegfr quota according to claim 1, wherein the setting a quota for the user at the begfs client in response to the failure of the error log information in the log further comprises:

detecting whether the quota is in effect;

in response to the quota not being validated, checking whether the node environment of the node deploying the BeeGFS server is correct.

7. An apparatus for configuring a BeeGFS quota for a cluster, the apparatus comprising:

the environment configuration module is configured to configure nodes deploying BeeGFS service ends in the cluster as mutual secrecy-free and configure node environments deploying the nodes deploying the BeeGFS service ends;

An NIS service configuration module, configured to configure an NIS service for the node deploying the BeeGFS service end, and test whether the NIS service is configured successfully;

a quota modification configuration module configured to, in response to successful configuration of the NIS service, close a beegfr server service and a begfs client service and modify a quota configuration of the begfs server, and then restart the begfs server service and the begfs client service;

a creating user module configured to create a user in a file system of the BeeGFS server;

the log checking module is configured to restart management services in the BeeGFS server side services and check whether error log information appears in a log;

a quota setting module configured to set a quota for the user at the BeeGFS client in response to the error-reported log information not appearing in the log.

8. The apparatus for cluster configuration of BeeGFS quotas of claim 7, wherein the modify quota configuration module is further configured to:

9. A computer device, comprising:

at least one processor; and

memory storing a computer program operable on the processor, wherein the processor, when executing the program, performs the method of any of claims 1-6.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 6.