CN111158743B - Big data operation and maintenance management platform - Google Patents
Big data operation and maintenance management platform Download PDFInfo
- Publication number
- CN111158743B CN111158743B CN201911386466.9A CN201911386466A CN111158743B CN 111158743 B CN111158743 B CN 111158743B CN 201911386466 A CN201911386466 A CN 201911386466A CN 111158743 B CN111158743 B CN 111158743B
- Authority
- CN
- China
- Prior art keywords
- maintenance
- big data
- script
- component
- python
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
Abstract
The application discloses big data operation and maintenance management platform, including control node and a plurality of child node, every child node all has the python operation and maintenance storehouse of pre-construction and is responsible for the operation and maintenance operation of one or more big data assembly. The control node can provide an operation mode of the big data assembly cluster to the outside and send an operation instruction to each node; each child node calls a python operation and maintenance library to execute corresponding operation and maintenance operations on the target big data component according to the big data component operation and maintenance operation instruction sent by the control node; the python operation and maintenance library comprises a plurality of groups of operation and maintenance operation scripts and is used for providing a uniform application programming interface for each operation and maintenance operation, and each operation and maintenance operation script is packaged by using a pyhton language. The method and the device can shield the difference of specific operation and maintenance operation modes of different operation systems, simplify the writing of operation and maintenance operation scripts, and are beneficial to reducing the complexity of the system, the difficulty of development and the complexity of maintenance work.
Description
Technical Field
The application relates to the technical field of big data, in particular to a big data operation and maintenance management platform.
Background
With the rapid development of big data and cloud computing, a big data platform is widely applied to various industries as a big data analysis tool, and management, operation and maintenance of big data are an inevitable link.
Related art typically performs automatic installation deployment of big data components based on shell scripts or bat scripts. However, the support and behavior of the automatic installation and deployment mode based on the shell script or the bat script under different platforms are different, and the syntax and compatibility problems of different shells need to be considered when the script is written, so that the method is very complicated and is very easy to generate errors. In order to realize big data management, a management platform often needs to be developed by using multiple programming languages, so that the development and the cooperation among teams are not facilitated, and the complexity of a system and the difficulty of development are high.
Disclosure of Invention
The application provides a big data operation and maintenance management platform, which shields the difference of specific operation and maintenance operation modes of different operation systems, simplifies the writing of operation and maintenance operation scripts, and is beneficial to reducing the complexity of the system, the difficulty of development and the complexity of maintenance work.
In order to solve the above technical problems, embodiments of the present invention provide the following technical solutions:
the embodiment of the invention provides a big data operation and maintenance management platform, which comprises:
the method comprises a control node and a plurality of child nodes, wherein each child node is provided with a pre-constructed python operation and maintenance library;
the control node is used for providing an operation mode of the big data assembly cluster for an external system and sending an operation instruction to each node;
each child node calls the python operation and maintenance library to execute corresponding operation and maintenance operation on a target big data component according to the big data component operation and maintenance operation instruction sent by the control node;
the python operation and maintenance library comprises a plurality of sets of operation and maintenance operation scripts and is used for providing a uniform application programming interface for each operation and maintenance operation, and each operation and maintenance operation script is packaged by using a pyhton language.
Optionally, the control node further includes a component configuration plug-in;
the component configuration plug-in is used for updating the component configuration file of the first big data component based on the change information after receiving the change information of the first big data component of the first child node, and sending the updated component configuration file to the first child node;
the component configuration file comprises a component name, a component version, a component description, a component service, and a corresponding relation between each operation and maintenance operation of the component service and the operation and maintenance operation script.
Optionally, the control node includes a cluster state monitoring module, and further includes an alarm prompting module connected to the cluster state monitoring module;
the cluster state monitoring module is used for collecting and summarizing resource consumption information of each sub-node according to a preset frequency; the resource consumption information is heartbeat data fed back by the current child node and cluster state information of the big data assembly cluster to which the child node belongs;
and the alarm prompting module is used for carrying out alarm prompting when the resource consumption of the current child node is detected to exceed a preset energy consumption threshold or the running state of the current child node is abnormal.
Optionally, the operation mode may be any one or any combination of the following:
presentation layer state transition application programming interfaces, web pages, and command lines.
Optionally, the system version of the big data operation and maintenance management platform is compiled by a python programming language, the control node includes an upgrade module, and the upgrade module is configured to perform system version upgrade according to an update file list carried in an upgrade instruction; the updated file list comprises newly added files in the system version after the upgrade, replacement files modified compared with the system version before the upgrade and deleted files reduced compared with the system version before the upgrade;
the upgrading module is used for adding the newly added file on the basis of the system version before upgrading, deleting the deleted file from the system version before upgrading and replacing the corresponding file in the system version before upgrading by using the replacing file so as to finish upgrading the system version.
Optionally, the control node includes an upgrade module;
the updating module executes system version updating by using a patch pack mode of a git distributed version control system, the patch pack of the git distributed version control system is used for automatically generating an updating file list according to a system version before updating and a system version after updating, and the updating file list comprises the record information of each file which is compared with the system version before updating and the system version after updating to generate adding, deleting and changing operations.
Optionally, the python operation and maintenance library further includes a plurality of sub operation and maintenance libraries, and each sub operation and maintenance library corresponds to one type of big data component; each sub operation and maintenance library comprises a plurality of operation and maintenance script groups for realizing the same operation and maintenance operation of the corresponding big data assembly in different deployment environments, and each operation and maintenance script group has a unique label.
Optionally, the tag is activated by system architecture information of the corresponding deployment environment.
Optionally, each operation and maintenance operation script in the python operation and maintenance library further supports function call; correspondingly, each child node further comprises a script calling module, the script calling module is used for responding to a script calling instruction of the big data component of the cluster to which the child node belongs, and the script calling instruction is an instruction for calling a corresponding operation and maintenance operation script from the python operation and maintenance library according to each step sequence of target operation and maintenance operation by using a script calling function which is defined in advance when the corresponding big data component is integrated.
Optionally, the background development languages of the control node and each child node, the operation and maintenance operation language of the big data component, and the programming language of each operation and maintenance script in the python operation and maintenance library are all the same and are python languages.
The technical scheme provided by the application has the advantages that based on a system architecture of a Leader/Follower, each child node is responsible for one or more big data assemblies, a python operation and maintenance library is arranged on the child node, the control node provides an operation mode of the big data assemblies and issues an operation instruction, the child node executes corresponding operation and maintenance operations by using the python operation and maintenance library after receiving the operation instruction, the python operation and maintenance library uniformly uses python packaged operation and maintenance scripts for installation and deployment, differences of an operation system and different shell scripts are shielded, only one script is used for writing the same installation step under different operation systems, and writing and maintenance work of the installation scripts is greatly simplified; the python operation and maintenance library uses a uniform interface for each common system operation, so that a user does not need to consider too much system or platform differentiation parts when compiling the component operation and maintenance script and consider too much compatibility of the script, the compiling and maintenance work of the operation and maintenance operation script is further simplified, and the complexity of the system and the difficulty of development are reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the related art, the drawings required to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic diagram of a system architecture of a big data operation and maintenance management platform according to an embodiment of the present invention;
fig. 2 is a schematic system architecture diagram of another big data operation and maintenance management platform according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the python operation and maintenance library according to an embodiment of the present invention;
fig. 4 is a structural diagram of a specific implementation manner of a control node or a child node according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may include other steps or elements not expressly listed.
Having described the technical solutions of the embodiments of the present invention, various non-limiting embodiments of the present application are described in detail below.
Referring to fig. 1, fig. 1 is a schematic diagram of a system architecture of a big data operation and maintenance management platform provided in an embodiment of the present invention, where the embodiment of the present invention includes the following:
the system architecture of the big data operation and maintenance management platform may adopt a Leader/follow mode, that is, the big data operation and maintenance management platform may include a control node 1 and a plurality of child nodes, each of which may be denoted as 21, 22, … …, 2 n. The control node 1 is responsible for operating the whole cluster, can provide an operation mode of the big data assembly cluster for an external system, sends an operation instruction to each node, and collects feedback data information of each child node. The control node 1 may provide an operation clustering manner such as a presentation State Transfer Application Programming Interface (Rest API), a web page, and a command line to the outside, as shown in fig. 2. The control node 1 provides a reset API through the API server to realize operations such as operating a cluster big data assembly, modifying configuration and the like, and a user can directly use the interaction mode of the reset API and can also use a traditional command line or a web page mode to operate a big data cluster.
Wherein each child node interfaces with one or more big data components and each has a pre-built python operation and maintenance library. Each child node is responsible for performing operation and maintenance operations on one or more big data assemblies of the cluster to which the child node belongs, and feeding back state information of the cluster to the main control node 1. Specifically, each child node can call a python operation and maintenance library to execute corresponding operation and maintenance operations on the target big data component according to the big data component operation and maintenance operation instruction sent by the control node 1.
In the application, the python operation and maintenance library comprises a plurality of operation and maintenance operation scripts and is used for providing a uniform application programming interface for each operation and maintenance operation, each operation and maintenance operation script is packaged by using a pyhton language, and the operation and maintenance operation scripts such as a decompression script, a copy file script, a file regular replacement script, an installation software package script, a configuration software source script and the like. The python operation and maintenance library uses a uniform interface for each common system operation, and the bottom layer can provide different implementation details according to different operating systems and even platforms with different architectures, as shown in fig. 3. The unified API is provided for all operations, the difference of specific implementation steps of different operating systems is shielded, the consistency of program writing methods of the same operation under different systems is ensured, a user does not need to consider too much system or platform differentiation parts when writing the component operation and maintenance script, the compatibility of the script does not need to be considered too much, and the development process is simplified.
In the technical scheme provided by the embodiment of the invention, based on a system architecture of a Leader/Follower, each child node is responsible for one or more big data assemblies, a python operation and maintenance library is arranged on the child node, the control node provides an operation mode of the big data assemblies and issues an operation instruction, the child node executes corresponding operation and maintenance operations by using the python operation and maintenance library after receiving the operation instruction, the python operation and maintenance library uniformly uses python-packaged operation and maintenance scripts for installation and deployment, the difference of an operation system and different shell scripts is shielded, and only one script is written for the same installation step under different operation systems, so that the writing and maintenance work of the installation scripts is greatly simplified; the python operation and maintenance library uses a uniform interface for each common system operation, so that a user does not need to consider too much system or platform differentiation parts when compiling the component operation and maintenance script and consider too much compatibility of the script, the compiling and maintenance work of the operation and maintenance operation script is further simplified, and the complexity of the system and the difficulty of development are reduced.
It can be understood that multiple programming languages appear on the same management platform in the related art, which results in higher complexity of the whole system and complex maintenance operation, and is not beneficial to rapid development of projects. Based on this, the background development language of the control node 1 and each child node, the operation and maintenance operation language of the big data component, and the programming language of each operation and maintenance script in the python operation and maintenance library can all be the same, and can all be python language. The python language is used as a background development language of the big data management platform, and the operation and maintenance operation language related to the big data component also uses python. The operation and maintenance script is uniformly written by using the python language, so that the script writing work can be further simplified, the programming languages of the management system and the operation and maintenance part are unified, and the complexity and the development difficulty of the system can be greatly reduced. In addition, the method has the advantages that the python language is an interpretive language and compiling is not needed, the upgrading process of the big data management platform can be simplified, two upgrading modes of the big data management platform are provided, and the method can comprise the following steps:
in an embodiment, the control node 1 may include an upgrade module, where the upgrade module is configured to perform system version upgrade according to an update file list carried in an upgrade instruction; the updated file list includes newly added files in the system version after the upgrade, replaced files modified compared with the system version before the upgrade, and deleted files reduced compared with the system version before the upgrade. The upgrading module is used for adding a newly added file on the basis of the system version before upgrading, deleting the deleted file from the system version before upgrading and replacing the corresponding file in the system version before upgrading by using the replacing file so as to finish upgrading the system version. That is, the method supports a simple incremental upgrading mode, and only the files designed and modified in the project need to be directly replaced by the new version files when the upgrading operation is executed, for example, the modified file list can be listed first, and the new version files are used for replacement; meanwhile, files which do not exist in the new version are sorted out and deleted during upgrading, the whole management system does not need to be uninstalled and reinstalled, and upgrading of the system version is greatly simplified.
As another optional implementation manner, the upgrade module of the control node 1 may further perform system version upgrade in a patch package manner of the git distributed version control system, where the patch package of the git distributed version control system may be used to automatically generate an update file list according to the system version before upgrade and the system version after upgrade, where the update file list includes record information of operations of adding, deleting, and changing each file compared with the system version before upgrade and the system version after upgrade. In the method for upgrading by using the patch package of git, the adding and deleting conditions of each file in the big data management platform are listed in detail in the patch package of git. For the modified file, the add and delete records in the file are saved in detail, with the precision of the lines. Compared with the traditional project needing compiling, the method has the advantages that the upgrading operation only needs to use the git application patch, the new edition is released, the complicated processes of recompiling, packing, unloading, cleaning, installing and the like can be avoided, and the upgrading process is simplified.
The related art has poor support for heterogeneous platforms, such as x86, ppc64le, arm64 and the like, by means of automatic installation and deployment of big data components based on shell scripts or bat scripts, and different installation configuration scripts are not designed for multiple sets of heterogeneous platforms. In order to adapt to the heterogeneous platform, a management system corresponding to the target platform needs to be released, and development and version maintenance operations are very complicated. On the basis of the related technology, in order to realize big data management, a management platform often needs to be developed by using multiple programming languages, so that the development and the cooperation among teams are not facilitated, and the experience is poor. In view of this, the present application also extends the python operation and maintenance library in the above embodiment.
Although the python operation and maintenance library is provided for different operating systems or platforms with different architectures, deployment of some components requires different platforms to use different installation packages, or different installation scripts and the like. In other words, the operation and maintenance steps of the same component in different systems or heterogeneous platforms may be different. In order to solve the problem, the python operation and maintenance library of the present application further includes multiple sets of operation and maintenance script sets, and different sets of operation and maintenance script sets may be distinguished by using tags. Specifically, the python operation and maintenance library may include a plurality of child operation and maintenance libraries, each child operation and maintenance library corresponding to one type of big data component; each sub operation and maintenance library comprises a plurality of operation and maintenance script groups for realizing the same operation and maintenance operation of the corresponding big data assembly in different deployment environments, and each operation and maintenance script group has a unique label. For example, the big data component A follows step A in sequence in an x86 deployment environment1、A2、A3After the installation operation is executed, the steps A are sequentially carried out in the ppc64le deployment environment2、A1、A3When the installation operation is finished, only the step A needs to be executed in the arm64 deployment environment2And A3The installation operation can be performed. Based on the operation and maintenance method, a sub operation and maintenance library can be set for the A big data assembly, the operation and maintenance library comprises three operation and maintenance operation script groups, the label of each operation and maintenance operation script group can be the name or the number of the corresponding deployment environment, and the first operation and maintenance operation script group is formed by executing the A big data assembly1、A2、A3Is composed of scripts of operation and maintenance operations and is according to A1、A2、A3Executing the installation sequence; the second operation and maintenance operation script set is composed of an execution A1、A2、A3Is composed of scripts of operation and maintenance operations and is based onAccording to A2、A1、A3Executing the installation sequence; the third operation and maintenance operation script group is composed of execution A2、A3Is composed of scripts of operation and maintenance operations and is according to A2、A3The installation sequence is performed. Meanwhile, the corresponding tag activation function can be configured according to the system architecture, namely, the tag can be activated through the system architecture information of the corresponding deployment environment.
As can be seen from the above, the embodiment of the present invention provides the functions of the operation and maintenance script set, and can specify different heterogeneous platforms or call different sets of operation and maintenance scripts during operation under special conditions, so as to solve the problem of difference in operation and maintenance steps of different heterogeneous platforms, and implement the function of supporting multiple systems and heterogeneous platforms by using different operation and maintenance script sets.
In another application scenario, if the operating steps of the large data component deployed under different systems are not very different, for example, only a few lines of code, the use of different operation and maintenance script sets can be very cumbersome and use a large amount of repeated code. Based on this, each operation and maintenance operation script in the python operation and maintenance library of the application also supports function call; correspondingly, each child node further comprises a script calling module, the script calling module is used for responding to a script calling instruction of the big data assembly of the cluster to which the child node belongs, and the script calling instruction is an instruction for calling a corresponding operation and maintenance operation script from a python operation and maintenance library according to each step sequence of the target operation and maintenance operation by using a script calling function which is defined in advance when the corresponding big data assembly is integrated. That is to say, the method and the device support the function of defining the custom cross-platform operation and maintenance script function, and the function can be defined according to the actual application requirement when the big data assembly is integrated. The difference between the custom function and the interface provided in the python operation and maintenance library is the scope difference. The custom function is only used as a supplement to the python operation and maintenance library, the scope of action is a single component, and the scope of action of the python operation and maintenance library interface is the management system global.
In the embodiment of the invention, besides the operation and maintenance requirements of a special platform or system are met by utilizing the functions of the operation and maintenance script group, a user can also call the self-defined script when integrating the big data assembly, so that the operation and maintenance operation is more convenient for the user to use, and the operation and maintenance operation is simplified.
It can be understood that, for a big data component deployed on a child node, addition or deletion may be performed due to an actual situation, in order to implement flexible increase and decrease of the big data component, the application separates a managed big data component from a code logic of a big data management platform itself, and an operation and maintenance operation of the big data component may use a configured manner, which may specifically include the following:
in this embodiment of the present invention, the first child node is any child node under the control node 1, and the first big data component is a component that is changed in any cluster to which the first child node belongs, and for convenience of description, the first child node and the first big data component are named. The control node 1 may also comprise a component configuration plug-in; the component configuration plug-in is used for updating the component configuration file of the first big data component based on the change information after receiving the change information of the first big data component of the first child node, and sending the updated component configuration file to the first child node. The component configuration file comprises component names, component versions, component descriptions, component services and the corresponding relation between each operation and maintenance operation of the component services and the operation and maintenance operation script.
In the embodiment of the present invention, the control node 1 may preset a component configuration file template, then generate a component configuration file for all big data components in the cluster based on the component configuration file template, and manage the big data components by using the component configuration file. The component configuration file template comprises a component name, a component version, a component description, a component service, a corresponding relation between each operation and maintenance operation of the component service and an operation and maintenance operation script, the component service can support a plurality of operations, the operation and maintenance operations supported by the component services are different, for example, the component service 1 can support installation operation, starting operation and stopping operation, and the component service 2 can support decompression operation, encryption operation and the like.
Therefore, the embodiment of the invention can realize flexible increase and decrease of the big data assembly only by modifying the assembly configuration file.
It can be understood that each child node is responsible for maintaining and controlling connectivity of the node 1, and sends heartbeat data, monitoring data, task execution conditions of each child node and state information of the cluster to which the child node belongs to the control node 1 in a timed or real-time manner, and the control node 1 can collect the data information sent by each child node in a real-time or timed manner, so that the change of the working state of the child node can be perceived as soon as possible. Correspondingly, the control node 1 may include a cluster state monitoring module and an alarm prompting module connected with the cluster state monitoring module; the cluster state monitoring module is used for collecting and summarizing resource consumption information of each sub-node according to a preset frequency; the resource consumption information is heartbeat data fed back by the current child node and cluster state information of the big data assembly cluster to which the child node belongs; and the alarm prompting module is used for carrying out alarm prompting when the resource consumption of the current child node is detected to exceed a preset energy consumption threshold or the running state of the current child node is abnormal. The preset energy consumption threshold value may be determined according to an actual application scenario, which is not limited in this application.
Finally, from a hardware perspective, the control node 1 and each child node of the present application may include therein the memory 40 and the processor 41 shown in fig. 4. The memory 40 is used for storing a computer program; and the processor 41 is configured to implement, when executing the computer program, the steps of implementing the corresponding big data operation and maintenance management method in the big data operation and maintenance management platform according to any one of the above embodiments.
Processor 41 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so forth. The processor 21 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 41 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 41 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, processor 41 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
Memory 40 may include one or more computer-readable storage media, which may be non-transitory. Memory 40 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 40 is at least used for storing the following computer program 401, wherein after being loaded and executed by the processor 41, the computer program can implement the relevant steps of the big data operation and maintenance management method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 40 may also include an operating system 402, data 403, and the like, and the storage manner may be a transient storage or a permanent storage. Operating system 402 may include, among other things, Windows, Unix, Linux, and the like. Data 403 may include, but is not limited to, data corresponding to test results, and the like.
In some embodiments, the control node 1 or each sub-node may further include a display 42, an input/output interface 43, a communication interface 44, a power supply 45, and a communication bus 46. Those skilled in the art will appreciate that the configuration shown in FIG. 4 does not constitute a limitation of a big data operation and maintenance management device, and may include more or fewer components than those shown, such as sensors 47.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The above provides a detailed description of a big data operation and maintenance management platform provided by the present application. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present application.
Claims (8)
1. A big data operation and maintenance management platform is characterized by comprising a control node and a plurality of child nodes, wherein each child node is provided with a pre-constructed python operation and maintenance library;
the control node is used for providing an operation mode of the big data assembly cluster for an external system and sending an operation instruction to each node;
each child node calls the python operation and maintenance library to execute corresponding operation and maintenance operation on a target big data component according to the big data component operation and maintenance operation instruction sent by the control node;
the python operation and maintenance library comprises a plurality of groups of operation and maintenance operation scripts and is used for providing a uniform application programming interface for each operation and maintenance operation, and each operation and maintenance operation script is packaged by using a pyhton language;
the python operation and maintenance library also comprises a plurality of sub operation and maintenance libraries, and each sub operation and maintenance library corresponds to one type of big data assembly; each sub operation and maintenance library comprises a plurality of operation and maintenance script groups for realizing the same operation and maintenance operation of the corresponding big data assembly in different deployment environments, and each operation and maintenance script group has a unique label; the tags are activated by system architecture information of the respective deployment environment.
2. The big data operation and maintenance management platform according to claim 1, wherein the control node further comprises a component configuration plug-in;
the component configuration plug-in is used for updating the component configuration file of the first big data component based on the change information after receiving the change information of the first big data component of the first child node, and sending the updated component configuration file to the first child node;
the component configuration file comprises a component name, a component version, a component description, a component service, and a corresponding relation between each operation and maintenance operation of the component service and the operation and maintenance operation script.
3. The big data operation and maintenance management platform according to claim 2, wherein the control node comprises a cluster state monitoring module and an alarm prompting module connected with the cluster state monitoring module;
the cluster state monitoring module is used for collecting and summarizing resource consumption information of each sub-node according to a preset frequency; the resource consumption information is heartbeat data fed back by the current child node and cluster state information of the big data assembly cluster to which the child node belongs;
and the alarm prompting module is used for carrying out alarm prompting when the resource consumption of the current child node is detected to exceed a preset energy consumption threshold or the running state of the current child node is abnormal.
4. The big data operation and maintenance management platform according to claim 3, wherein the operation mode can be any one or any combination of the following:
presentation layer state transition application programming interfaces, web pages, and command lines.
5. The big data operation and maintenance management platform according to claim 1, wherein a system version of the big data operation and maintenance management platform is compiled using a python programming language, and the control node comprises an upgrade module, and the upgrade module is configured to perform system version upgrade according to an update file list carried in an upgrade instruction; the updated file list comprises newly added files in the system version after the upgrade, replacement files modified compared with the system version before the upgrade and deleted files reduced compared with the system version before the upgrade;
the upgrading module is used for adding the newly added file on the basis of the system version before upgrading, deleting the deleted file from the system version before upgrading and replacing the corresponding file in the system version before upgrading by using the replacing file so as to finish upgrading the system version.
6. The big data operation and maintenance management platform according to claim 1, wherein the control node comprises an upgrade module;
the updating module executes system version updating by using a patch pack mode of a git distributed version control system, the patch pack of the git distributed version control system is used for automatically generating an updating file list according to a system version before updating and a system version after updating, and the updating file list comprises the record information of each file which is compared with the system version before updating and the system version after updating to generate adding, deleting and changing operations.
7. The big data operation and maintenance management platform according to any one of claims 1 to 6, wherein each operation and maintenance operation script in the python operation and maintenance library further supports function call; correspondingly, each child node further comprises a script calling module, the script calling module is used for responding to a script calling instruction of the big data component of the cluster to which the child node belongs, and the script calling instruction is an instruction for calling a corresponding operation and maintenance operation script from the python operation and maintenance library according to each step sequence of target operation and maintenance operation by using a script calling function which is defined in advance when the corresponding big data component is integrated.
8. The big data operation and maintenance management platform according to claim 7, wherein the background development language of the control node and each child node, the operation and maintenance operation language of the big data component, and the programming language of each operation and maintenance script in the python operation and maintenance library are all the same and are python languages.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911386466.9A CN111158743B (en) | 2019-12-29 | 2019-12-29 | Big data operation and maintenance management platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911386466.9A CN111158743B (en) | 2019-12-29 | 2019-12-29 | Big data operation and maintenance management platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111158743A CN111158743A (en) | 2020-05-15 |
| CN111158743B true CN111158743B (en) | 2022-04-22 |
Family
ID=70558896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911386466.9A Active CN111158743B (en) | 2019-12-29 | 2019-12-29 | Big data operation and maintenance management platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111158743B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269714B (en) * | 2020-10-23 | 2023-02-28 | 山东云海国创云计算装备产业创新中心有限公司 | Server BMC centralized management system, method, device and medium |
| CN113434158B (en) * | 2021-07-08 | 2023-12-15 | 恒安嘉新（北京）科技股份公司 | Custom management method, device, equipment and medium for big data component |
| CN116056120A (en) * | 2021-10-28 | 2023-05-02 | 华为技术有限公司 | Operation and maintenance operation method, system and network equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103888287A (en) * | 2013-12-18 | 2014-06-25 | 北京首都国际机场股份有限公司 | Information system integrated operation and maintenance monitoring service early warning platform and realization method thereof |
| CN104022903A (en) * | 2014-06-13 | 2014-09-03 | 中国民航信息网络股份有限公司 | One-stop automatic operation and maintaining system |
| CN106055486A (en) * | 2016-08-19 | 2016-10-26 | 浪潮(北京)电子信息产业有限公司 | Automatic operation maintenance method and platform of distributed file system |
| CN110187896A (en) * | 2019-05-28 | 2019-08-30 | 北京海量数据技术股份有限公司 | A kind of automation operating system dispositions method |
| CN110276594A (en) * | 2019-06-21 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of banking operational system and method based on Ansible |
| CN110543328A (en) * | 2019-07-26 | 2019-12-06 | 苏州浪潮智能科技有限公司 | Cross-platform component management method, system, terminal and storage medium based on Ambari |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8438418B2 (en) * | 2010-06-30 | 2013-05-07 | Oracle International Corporation | Simplifying automated software maintenance of data centers |
-
2019
- 2019-12-29 CN CN201911386466.9A patent/CN111158743B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103888287A (en) * | 2013-12-18 | 2014-06-25 | 北京首都国际机场股份有限公司 | Information system integrated operation and maintenance monitoring service early warning platform and realization method thereof |
| CN104022903A (en) * | 2014-06-13 | 2014-09-03 | 中国民航信息网络股份有限公司 | One-stop automatic operation and maintaining system |
| CN106055486A (en) * | 2016-08-19 | 2016-10-26 | 浪潮(北京)电子信息产业有限公司 | Automatic operation maintenance method and platform of distributed file system |
| CN110187896A (en) * | 2019-05-28 | 2019-08-30 | 北京海量数据技术股份有限公司 | A kind of automation operating system dispositions method |
| CN110276594A (en) * | 2019-06-21 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of banking operational system and method based on Ansible |
| CN110543328A (en) * | 2019-07-26 | 2019-12-06 | 苏州浪潮智能科技有限公司 | Cross-platform component management method, system, terminal and storage medium based on Ambari |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111158743A (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111158743B (en) | Big data operation and maintenance management platform | |
| US10740114B2 (en) | Component invoking method and apparatus, and component data processing method and apparatus | |
| WO2021088909A1 (en) | Method and system for assisting operator development | |
| CN105022630B (en) | A kind of assembly management system and assembly management method | |
| US20140196022A1 (en) | Cloud Based Application Packaging | |
| CN111897570B (en) | Multi-dependency item file extraction method and device based on Maven plug-in | |
| CN104317642A (en) | Method and device for configuring software in cloud calculation environment | |
| CN110275722A (en) | Method, apparatus, equipment and storage medium for upgrade application | |
| CN103778178A (en) | Method and system for reconfiguring snapshot of virtual machine (VM) | |
| CN103608773A (en) | Deployment system for multi-node applications | |
| CN102156643A (en) | Software integration method and system thereof | |
| CN110673853A (en) | Compiling method, device and system | |
| CN114077423A (en) | Airport APP development container architecture based on mobile cross-platform | |
| US9542173B2 (en) | Dependency handling for software extensions | |
| WO2021037050A1 (en) | Code change method and device | |
| CN111752581A (en) | Distributed system upgrading method and device and computer system | |
| WO2024077885A1 (en) | Management method, apparatus and device for container cluster, and non-volatile readable storage medium | |
| CN113805882A (en) | Method and device for developing application program, electronic equipment and storage medium | |
| CN104699503A (en) | Method and device for replacing function execution logic in Android system | |
| CN114064083A (en) | Method for deploying cloud native application through self-defined template in configuration center and application | |
| CN107844300A (en) | Script processing method and system | |
| CN113760462A (en) | Method and device for constructing verification environment of dispatching automation system | |
| CN113656001A (en) | Platform component development method and device, computer equipment and storage medium | |
| CN111435312A (en) | Application program management method and device and electronic equipment | |
| CN110806891A (en) | Method and device for generating software version of embedded equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |