CN116719820A - Database operation and maintenance control method based on general model - Google Patents

Abstract

The application discloses a database operation and maintenance control method based on a general model, which relates to the technical field of computers and comprises the following steps: abstracting all databases into a combination of clusters and components, wherein each cluster is composed of a plurality of components, and extracting common characteristics of all the components; constructing a general model based on the common characteristics, and storing the common characteristic values of all databases and the corresponding operation and maintenance actions thereof in the general model; and responding to the common characteristic value of the database components customized by the database manufacturer, and selecting a corresponding operation and maintenance result from operation and maintenance actions stored by the general model to generate a complete operation and maintenance task stream. The application builds a general model of the database component, and can uniformly and dynamically generate a set of complete operation and maintenance task flow for any database by using the general model, thereby not only improving the code multiplexing degree and the user experience degree, but also reducing the access cost of database manufacturers.

Description

Database operation and maintenance control method based on general model

Technical Field

The application relates to the technical field of computers, in particular to a database operation and maintenance management and control method based on a general model.

Background

The existing database management and control system has the following general problems: one set of codes of the database has low multiplexing degree and high access cost of database manufacturers; the method is accessed in a command type task flow mode, and has no view angle of database research and development, a certain management and control research and development experience is needed, and the learning cost is high; the development and debugging cost is high, and the complete life cycle of the database product cannot be verified in the local environment; the user-oriented interfaces are not uniform, and each database has inconsistent user experience, such as different CR of each database in kubedb, and different interfaces mean corresponding to different controllers.

Disclosure of Invention

The application provides a database operation and maintenance control method based on a general model, which aims to solve the problems of low multiplexing degree of multiple database operation and maintenance control codes and low user experience in the prior art.

In order to achieve the above purpose, the present application adopts the following technical scheme:

the application discloses a database operation and maintenance management and control method based on a general model, which comprises the following steps:

abstracting all databases into a combination of clusters and components, wherein each cluster is composed of a plurality of components, and extracting common characteristics of all the components;

constructing a general model based on the common characteristics, and storing the common characteristic values of all databases and the corresponding operation and maintenance actions thereof in the general model;

and responding to the common characteristic value of the database components customized by the database manufacturer, and selecting a corresponding operation and maintenance result from the operation and maintenance actions stored by the general model to generate a complete operation and maintenance task stream.

Preferably, the common feature components belong to different dimensions including censoring of lifecycles, replication relationship types, availability, configuration, scheduling, networking, and monitoring.

Preferably, the adding, deleting and modifying of the life cycle specifically comprises a configuration container strategy, a stopping strategy, a horizontal capacity expansion strategy and a maximum unavailable number strategy.

Preferably, the operation and maintenance action corresponding to the container policy is to create a database cluster, which specifically includes a configuration execution command, an environment variable, a resource and a volume mount.

Preferably, the operation and maintenance action corresponding to the stopping strategy is to delete the database cluster according to the stopping strategy and selectively delete the data related to the database cluster.

Preferably, the operation and maintenance action corresponding to the horizontal capacity expansion strategy is to determine the front-end work of database capacity expansion according to the horizontal capacity expansion strategy and sequentially complete the front-end work and capacity expansion node work.

Preferably, the operation and maintenance action corresponding to the maximum unavailable number strategy is to control the updating concurrency mode according to the maximum unavailable number strategy and the copy relationship type.

Preferably, the replication relationship type includes a paxos type, a master-slave type, a replication relationship-free stateful type and a replication relationship-free stateless type.

Preferably, the method further comprises providing a declarative API for database vendors to enter custom database component common characteristic values.

Preferably, the responding to the common characteristic value of the database components customized by the database manufacturer selects a corresponding operation and maintenance result from the operation and maintenance actions stored by the universal model to generate a complete operation and maintenance task stream, which comprises the following steps:

and receiving the common characteristic value of the database vendor custom database components provided by the declarative API, and selecting all operation and maintenance actions corresponding to the common characteristic value of the custom database components from the operation and maintenance actions stored by the general model.

The application has the following beneficial effects:

the application provides the declarative API capable of describing the developed database characteristics for the database operator, and the general model can dynamically generate a set of complete operation and maintenance task flow for the database by the information provided by the API, thereby not only improving the code multiplexing degree, but also reducing the access cost of the database manufacturer; and meanwhile, a unified operation and maintenance API is provided for the user, so that the operation and maintenance experience of any database is kept consistent, and the user experience is improved.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the application, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.

Fig. 1 is a flowchart of a database operation and maintenance management and control method based on a general model.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terms "first," "second," and the like in the claims and the description of the application, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order, and it is to be understood that the terms so used may be interchanged, if appropriate, merely to describe the manner in which objects of the same nature are distinguished in the embodiments of the application by the description, and furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

The conventional database operation and maintenance management and control system is to configure a set of control logic for each type of database separately, so that the user experience is poor, and in practice, the databases of different types generally have certain similarity, unified features can be abstracted, a unified abstract model of a multi-database engine is constructed, and the unified management and control of the operation and maintenance of the multi-database engine is realized based on the model, based on the unified abstract model, as shown in fig. 1, the embodiment provides a database operation and maintenance management and control method based on a general model, which specifically comprises the following steps:

s110, abstracting all databases into a combination of clusters and components, wherein each cluster is composed of a plurality of components, and extracting common characteristics of all the components;

s120, constructing a general model based on the common characteristics, and storing the common characteristic values of all databases and corresponding operation and maintenance actions thereof in the general model;

and S130, responding to the common characteristic value of the database components customized by the database manufacturer, and selecting a corresponding operation and maintenance result from the operation and maintenance actions stored by the general model to generate a complete operation and maintenance task stream.

In this embodiment, all databases are first divided into two layers, the first layer is a cluster, which represents a complete database system, the cluster is composed of one or more components, each component can be used as an independent system to complete relatively independent work, the collaboration among multiple components forms a complete cluster work, one component usually contains one or more database nodes, which use the same binary program, and the stored data are copies of each other, and the main database, such as mysql, postgreql, redis, mongodb, etcd, clickhouse, kafka, can be abstracted into the two layers.

Next, in order to manage the lifecycle and the operation and maintenance of components, common features of all components need to be extracted, specifically including the following dimensions:

1. life cycle

Namely, adding, deleting and modifying the whole component, and particularly performing operation and maintenance actions on the database cluster, wherein the operation and maintenance actions generally comprise: creation, deletion, expansion, rolling updates, etc., to distinguish the lifecycle cases, this dimension abstracts several attributes:

(1) Container policy Containers, one or more Containers necessary for the database to run, which in particular also include a mirrored Image, typically expressed in docker, for controlling the version of the database executable; the script executed by the program in running can customize the executing Command of the starting behavior for each database; environmental variables Env needed in the container; resource Resources describing how many Resources (CPU, memory) are needed at minimum and how many Resources are available at maximum; and volume mount describing externally mounted persistent storage features.

Taking k8s as an example, the set contents related information includes Image, command, env, resources and VolumeMount, which respectively correspond to related fields in k8s pod, a state statefulset of k8s can be sequentially created, where the state includes a topology state and a storage state, and thus a database cluster is created.

(2) Stop policy termination policy: and describing which resources are cleaned when deleting, if so, whether the persistent storage needs to be deleted or whether the backup needs to be deleted.

Based on the information provided by the set termination policy field, the extent to which resources are deleted, i.e. the resources mentioned in the stop policy that should be cleaned, can be controlled when the cluster is deleted.

(3) Horizontal expansion strategy horizontality scaleplate policy: describing the strategy when expanding the node, if yes, whether the backup is needed to restore the data, and what mode is used for backup.

Based on k8s statefulset, a simple node adding and deleting function can be realized, but for the specific application of the database, the capacity expansion is often not only to add and delete nodes, but also some extra data synchronization work, the front-end work required when the current database is expanded can be known according to the strategy configured by the arranged HorizontalScalePolicy field, and if backup synchronization data is required for configuration, the backup recovery and capacity expansion nodes are completed in sequence when the capacity expansion is realized.

(4) The maximum unavailable number policy maxunnavailable is what is the maximum unavailable in the scroll update and is used for controlling the concurrency of the scroll update.

2. Copy relationship type

Communication exists among a plurality of nodes in a component, and the nodes generally have a certain replication relationship, and according to research on a main stream database, the replication relationship of the database can be divided into the following types:

(1) Paxos-like: often also referred to as three nodes, at least three nodes are required which can autonomously select one master after a replication relationship has been established and can select a new master after a master failure by means of an internal election algorithm.

(2) Master and slave types: at least two nodes are needed, the backup copy master data is used as real-time backup, the backup can be upgraded to the master executive master function when the master fails, and the availability is improved.

(3) Copy-free relational stateful types: copy relationships and role differentiation are not shown between nodes, but require persistent storage.

(4) Copy-free relationship stateless type: the copy relationship and the role distinction which are not shown among the nodes do not need to be stored in a lasting mode.

The main stream database is not limited to one of the four databases, and when describing a database component, only the copy relationship type of the database component is provided, so that the corresponding operation and maintenance operation can be performed according to the type of the database component.

According to the set maxunalfail, a PodDisruptionBudget in the availability is configured to control a basic updating concurrency strategy, but the database application always has a certain requirement on the updating sequence when updating based on the availability requirement, which depends on the feature of the dimension of a replication relation type, in the type with the replication relation, the updating strategy updateStrategy of statefulset is set as OnDeletete, and a master node in a master-level type or a master-primary node in a master-standby type in the paxos type is set as the last updating, some non-master nodes are updated preferentially, and meanwhile, the priority of a follower node in the paxos type is higher than that of a learner node, so that the whole updating sequence in the paxos type is learner- > leader, and the updating sequence in the master-standby type is secondery- > primary.

3. Availability of

Describing how to detect the availability of a database node and determine how to handle it when it fails, includes the following specific attributes:

(1) In operation detector RunningProbe: the node may be considered to be operating properly under what conditions are configured.

(2) Role switch detector RoleChangedProbe: how to probe the node roles is configured, which is typically required in paxos and active-standby types.

(3) Status detector status probe: how to probe the state of a node.

4. Configuration of

The database engine typically allows a user to adapt the behavior of the database process through the form of configuration to adapt to different database usage scenarios, thus describing how a database needs to be updated with its configuration, which dimension can abstract the following features:

(1) And (3) configuring a template: what the format of the configuration is, what the initial parameters are, is described.

(2) Parameter constraint: the values of the parameters may be filled in what types, and in what ranges.

(3) Validation policy: describing which parameters must be restarted to be effective after modification and which can be dynamically effective.

Based on the above configuration, the refreshing can be performed, all databases have a configuration file for user modification, so that the databases are better adapted to the service scene of the user, the configuration update depends on the implementation of the database kernel, some configuration updates need to restart the databases, some configuration can be dynamically updated, the refreshing behavior needs to be controlled according to the configuration characteristics in the process of realizing the refreshing, such as checking whether the validity of the parameters meets the description of the parameter constraint, controlling whether the node restarting is needed according to the effective strategy, and the like.

5. Scheduling

The distribution of the nodes is described, if it is desired that the nodes are distributed as much as possible (reducing latency) or if it is desired that the nodes are as decentralized as possible (increasing availability).

6. Network (service)

Whether the current component allows exposing services, which types of services are allowed to be exposed, is described.

7. Monitoring

How the monitoring data gets exposed, including on what path, through what port is exposed, is described.

And constructing a general model of the component according to the extracted common characteristics, wherein the general model also stores the common characteristic values of all databases and the mapping relation of the operation and maintenance actions corresponding to the common characteristic values.

In addition, in this embodiment, a declarative API is further provided, where a database vendor may declare a common feature value of its custom database component in the API, and then the API feeds back the common feature value of the custom database component to a generic model, where the generic model finds all operation and maintenance actions corresponding to the common feature value of the custom database component from its stored mapping relationship, and further outputs a complete operation and maintenance task flow of the database.

The method provided by the embodiment builds a general model of the database component, provides a declarative API capable of describing the developed database characteristics for a database manufacturer, and can dynamically generate a set of complete operation and maintenance task flow for the database by the information provided by the API, so that the code multiplexing degree is improved, and the access cost of the database manufacturer is reduced; and meanwhile, a unified operation and maintenance API is provided for the user, so that the operation and maintenance experience of any database is kept consistent, and the user experience is improved.

Another embodiment of the present application also provides an electronic device, including a memory and a processor, where the memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement a method for controlling a database operation and maintenance based on a general model as described above.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the electronic device described above may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

Another embodiment of the present application also provides a computer readable storage medium storing a computer program, where the computer program makes a computer execute a database operation and maintenance management method based on a general model as described above.

By way of example, a computer program may be divided into one or more modules/units stored in a memory and executed by a processor and the I/O interface transmission of data accomplished by an input interface and an output interface to accomplish the present application, and one or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions for describing the execution of the computer program in a computer device.

The computer device may be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device may include, but is not limited to, a memory, a processor, and it will be appreciated by those skilled in the art that the present embodiments are merely examples of computer devices and are not limiting of computer devices, may include more or fewer components, or may combine certain components, or different components, e.g., a computer device may also include an input, a network access device, a bus, etc.

The processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory may also be an external storage device of the computer device, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the computer device, and further, the memory may also include an internal storage unit of the computer device and an external storage device, and the memory may also be used to store a computer program and other programs and data required by the computer device, and the memory may also be used to temporarily store the program code in an output device, where the aforementioned storage medium includes a U-disk, a removable hard disk, a read-only memory ROM, a random access memory RAM, a disk or an optical disk and other various Media that can store program codes.

The foregoing is merely illustrative of specific embodiments of the present application, and the scope of the present application is not limited thereto, but any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. The database operation and maintenance control method based on the general model is characterized by comprising the following steps of:

abstracting all databases into a combination of clusters and components, wherein each cluster is composed of a plurality of components, and extracting common characteristics of all the components;

constructing a general model based on the common characteristics, and storing the common characteristic values of all databases and the corresponding operation and maintenance actions thereof in the general model;

and responding to the common characteristic value of the database components customized by the database manufacturer, and selecting a corresponding operation and maintenance result from the operation and maintenance actions stored by the general model to generate a complete operation and maintenance task stream.

2. The method of claim 1, wherein the common features belong to different dimensions including censoring, replication relationship types, availability, configuration, scheduling, networking and monitoring of life cycles.

3. The method for managing and controlling database operations and maintenance based on a general model according to claim 2, wherein the adding, deleting and modifying of the life cycle specifically includes a configuration container policy, a stop policy, a horizontal capacity expansion policy and a maximum unavailable number policy.

4. A method for managing and controlling database operations and dimensions based on a generic model according to claim 3, wherein the operation and dimension action corresponding to the container policy is to create a database cluster, which specifically includes configuration execution command, environment variable, resource and volume mount.

5. A method for controlling database operations and maintenance based on a generic model according to claim 3, wherein the operation and maintenance action corresponding to the stopping policy is to delete a database cluster according to the stopping policy and selectively delete data related to the database cluster.

6. The method for managing and controlling database operation and maintenance based on a universal model according to claim 3, wherein the operation and maintenance action corresponding to the horizontal capacity expansion strategy is to determine the front-end work of database capacity expansion according to the horizontal capacity expansion strategy and sequentially complete the front-end work and capacity expansion node work.

7. A method for controlling database operations and maintenance based on a common model according to claim 3, wherein the operation and maintenance action corresponding to the maximum unavailable number policy is to control the update concurrency mode according to the maximum unavailable number policy and the replication relationship type.

8. The method for managing and controlling database operations and maintenance based on a general model according to claim 2, wherein the replication relationship types include paxos type, master-slave type, replication relationship-free stateful type and replication relationship-free stateless type.

9. The method of claim 1, further comprising providing a declarative API for database vendors to enter custom database component common feature values.

10. The method for controlling operation and maintenance of a database based on a universal model according to claim 9, wherein the responding to the common characteristic value of the database components customized by a database manufacturer, selecting the corresponding operation and maintenance result from the operation and maintenance actions stored by the universal model to generate a complete operation and maintenance task stream, comprises:

and receiving the common characteristic value of the database vendor custom database components provided by the declarative API, and selecting all operation and maintenance actions corresponding to the common characteristic value of the custom database components from the operation and maintenance actions stored by the general model.