CN111026723B - Big data cluster management control method and device, computer equipment and storage medium - Google Patents
Big data cluster management control method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111026723B CN111026723B CN201911150335.0A CN201911150335A CN111026723B CN 111026723 B CN111026723 B CN 111026723B CN 201911150335 A CN201911150335 A CN 201911150335A CN 111026723 B CN111026723 B CN 111026723B
- Authority
- CN
- China
- Prior art keywords
- big data
- target site
- data cluster
- file
- management control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/161—Computing infrastructure, e.g. computer clusters, blade chassis or hardware partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the application discloses a big data cluster management control method, a big data cluster management control device, computer equipment and a storage medium, which comprise the following steps: acquiring environment information of each target site in a distributed server cluster; configuring a configuration file of each target site according to the environment information of each target site; and correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster image file. According to the embodiment of the application, the environment information of each target site in the distributed server cluster is obtained, the configuration file of each target site is configured according to the environment information of each target site, then the big data cluster system is installed on each target site correspondingly according to each configuration file and the preset big data cluster image file, and the installation and parameter configuration of the big data cluster systems are carried out in batches, so that the installation speed of the big data clusters can be effectively increased, and the operation and maintenance efficiency of the big data clusters can be improved.
Description
Technical Field
The embodiment of the application relates to the technical field of big data platforms, in particular to a big data cluster management control method, a big data cluster management control device, computer equipment and a storage medium.
Background
Entering the advanced information development stage, the role and position of data processing are higher and higher, and many companies are building large data platforms for creating, managing and monitoring large data platform system clusters.
However, the big data cluster generally needs multiple servers, so that it takes much time to build the big data cluster to perform repetitive work, and for the big data components in the big data cluster, parameter setting may need to be performed on multiple nodes, for example, time zones, languages and root user passwords need to be set, operation and maintenance personnel need to spend a great deal of time to participate in the installation process, and the installation speed and operation and maintenance efficiency of the big data cluster are reduced.
Disclosure of Invention
The embodiment of the application provides a big data cluster management control method, a big data cluster management control device, computer equipment and a storage medium, wherein the big data cluster management control method, the big data cluster management control device, the computer equipment and the storage medium can improve the installation efficiency of a big data cluster.
In order to solve the technical problems, the embodiment of the application adopts the following technical scheme: the big data cluster management control method comprises the following steps:
acquiring environment information of each target site in a distributed server cluster;
configuring a configuration file of each target site according to the environment information of each target site;
and correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster image file.
Optionally, before the step of obtaining the environmental information of each target site in the distributed server cluster, the method further includes the following steps:
obtaining kernel configuration parameters of a local system;
and setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
Optionally, the step of obtaining the kernel configuration parameters of the local system includes the following steps:
acquiring original software package management information of the local system;
and acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
Optionally, the step of configuring the configuration file of each target site according to the environment information of each target site includes the following steps:
acquiring each file sharing directory comprising file sharing terminal information of a corresponding target site, wherein any one of the file sharing terminal information comprises address information of a configuration file of the corresponding target site;
and acquiring configuration files of the corresponding target sites according to the address information, and configuring the configuration files according to the environment information of the corresponding target sites.
Optionally, before the step of obtaining the file sharing directory including the file sharing terminal information of each target site, the method further includes the following steps:
installing and starting a preset file sharing service on each target site;
and respectively creating file sharing catalogues comprising file sharing terminal information of the corresponding target sites according to the file sharing services of the corresponding target sites.
Optionally, the step of performing big data cluster system management control on each target site according to each configuration file and a preset big data cluster mirror file correspondingly includes the following steps:
acquiring local area network address information of each target site;
establishing a corresponding relation between the big data cluster image file and the corresponding target site according to the local area network address information of each target site;
and sending the big data cluster image files to corresponding target sites according to the corresponding relation, so that each target site installs the big data cluster image files to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration files.
In order to solve the above technical problem, an embodiment of the present application further provides a big data cluster management control device, including:
the first acquisition module is used for acquiring the environment information of each target site in the distributed server cluster;
the first processing module is used for configuring the configuration file of each target site according to the environment information of each target site;
and the first execution module is used for correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster image file.
Optionally, the method further comprises:
the second acquisition module is used for acquiring kernel configuration parameters of the local system;
and the second execution module is used for generating the big data cluster image file according to the basic parameters of the kernel configuration parameter setting preset original image.
Optionally, the method further comprises:
the first acquisition sub-module is used for acquiring the original software package management information of the local system;
and the first execution sub-module is used for acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
Optionally, the method further comprises:
a second obtaining sub-module, configured to obtain each file sharing directory including file sharing terminal information of a corresponding target site, where any one of the file sharing terminal information includes address information of a configuration file of the corresponding target site;
and the second execution sub-module is used for acquiring the configuration file of the corresponding target site according to the address information and configuring the configuration file according to the environment information of the corresponding target site.
Optionally, the method further comprises:
the first processing sub-module is used for installing and starting a preset file sharing service on each target site;
and the third execution sub-module is used for respectively creating file sharing catalogues comprising file sharing terminal information of the corresponding target sites according to the file sharing services corresponding to the target sites.
Optionally, the method further comprises:
a third obtaining sub-module, configured to obtain local area network address information of each target station;
the second processing sub-module is used for establishing the corresponding relation between the big data cluster image file and the corresponding target site according to the local area network address information of each target site;
and the third execution sub-module is used for sending the big data cluster image files to the corresponding target sites according to the corresponding relation, so that each target site installs the big data cluster image files to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration files.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, including a memory and a processor, where the memory stores computer readable instructions, and when the computer readable instructions are executed by the processor, the processor is caused to execute the steps of the big data cluster management control method.
To solve the above technical problem, an embodiment of the present application further provides a storage medium storing computer readable instructions, where the computer readable instructions when executed by one or more processors cause the one or more processors to execute the steps of the big data cluster management control method.
The embodiment of the application has the beneficial effects that: the method comprises the steps of obtaining environment information of each target site in a distributed server cluster, configuring configuration files of each target site according to the environment information of each target site, and correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster image file, and carrying out installation and parameter configuration of the big data cluster system in batches, so that the installation speed of the big data cluster can be effectively increased, and the operation and maintenance efficiency of the big data cluster can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a basic flow diagram of a big data cluster management control method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of a custom big data cluster image file according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating a method for obtaining kernel configuration parameters of a local system according to an embodiment of the present application;
FIG. 4 is a flowchart illustrating a configuration file configuration according to an embodiment of the present application;
FIG. 5 is a basic flow diagram of creating a file sharing directory according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a basic flow for installing a big data cluster system according to an embodiment of the present application;
fig. 7 is a schematic diagram of a basic structure of a big data cluster management control device according to an embodiment of the present application;
FIG. 8 is a basic block diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present application with reference to the accompanying drawings.
In some of the flows described in the specification and claims of the present application and in the foregoing figures, a plurality of operations occurring in a particular order are included, but it should be understood that the operations may be performed out of order or performed in parallel, with the order of operations such as 101, 102, etc., being merely used to distinguish between the various operations, the order of the operations themselves not representing any order of execution. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and are not limited to the "first" and the "second" being different types.
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to fall within the scope of the application.
Example 1
Referring to fig. 1 specifically, fig. 1 is a basic flow chart of a big data cluster management control method according to the present embodiment.
As shown in fig. 1, a big data cluster management control method includes the following steps:
s1100, acquiring environment information of each target site in a distributed server cluster;
the distributed server cluster is to split a set of systems into a plurality of subsystems and then deploy the subsystems on different servers, wherein a target site is a server, the environment information of the target site characterizes the running environment of the server, in the implementation, an Apache Ambari cluster is built on a CentOS system for example, centOS (Community Enterprise Operating System) is one of Linux release boards, which is compiled from Red Hat Enterprise Linux source codes released according to the open source code specification; apache Ambari is a Web-based tool supporting the creation, management and monitoring of Apache Hadoop clusters, ambari supports most Hadoop components including HDFS, mapReduce, hive, pig, hbase, zookeeper, sqoop and Hcataog, and the like, and in addition, ambari supports computing frameworks such as Spark, storm, and the like, and resource scheduling platforms YARN. The Ambari cluster comprises a plurality of servers, a corresponding big data cluster mirror image is installed on each server, corresponding configuration files are required to be set according to the environment of each server, the environment information of each target site can be uniformly managed through a file sharing method, each target site is provided with a shared file, the shared file comprises the environment information of the shared file, and the system can acquire the environment information of each target site by acquiring the shared file of each target site.
S1200, configuring configuration files of each target site according to the environment information of each target site;
the operating environments of different target sites are different, so that the configuration files of each target site can be correspondingly set according to the environment information of each target site, and in implementation, the environment information of the target site includes but is not limited to: the time zone, language, user password secret key-free ssh access, database and the like, the configuration file carries the installation configuration parameters of the corresponding target site, and the system can modify the installation configuration parameters in the corresponding configuration file through the environment information of the target site. In one embodiment, the system may also generate a configuration file for each target site based on the environmental information for each target site.
S1300, carrying out big data cluster system management control on each target site correspondingly according to each configuration file and a preset big data cluster image file.
The system manages and controls installation and parameter setting of a big data cluster system in a target site according to a corresponding configuration file and a preset big data cluster image file of the target site, when the system is implemented, the system is used for setting up Ambari clusters in a centOS operating system for example, environment information of each target site is obtained, for example, the system comprises a target site 1 and a target site 2, wherein a server mapped by the target site 1 is in an east-eighth time zone, a server mapped by the target site 2 is in an east-fifth time zone, corresponding time zones are required to be set when the big data cluster system service is installed on the target site 1 and the target site 2, the system correspondingly sets up the configuration file of each target site according to the environment information of each target site, the configuration file comprises but is not limited to time zones, languages, root user passwords, keyless ssh access, database services, network connection and other running environment information, then the big data cluster system is installed in the target site through the big data cluster image file, parameters of the big data cluster system are set up according to the corresponding configuration file, for example, the big data cluster system is set up in the server mapped by the target site 1 and the target site 2, the big data cluster system is not required to be set up in the east-eighth time zone, the system is set up according to the corresponding data cluster system, and the user passwords need to the data cluster system is not set up in the east-batch time zone, and the user time zone is set up in the large-free time zone, and the system is required to be accessed by the user, and the system is set up in a large-batch time zone and the system is required to be set up in a large-free to be set up time required to be accessed by a large-required to be accessed by a user.
According to the embodiment of the application, the environment information of each target site in the distributed server cluster is obtained, the configuration file of each target site is configured according to the environment information of each target site, and then the large data cluster system management control is correspondingly carried out on each target site according to each configuration file and the preset large data cluster image file, so that the installation and parameter configuration of the large data cluster system are carried out in batches, the installation speed of the large data cluster can be effectively increased, and the operation and maintenance efficiency of the large data cluster is improved.
In an alternative embodiment, referring to fig. 2, fig. 2 is a basic flowchart of a custom large data cluster image file in an embodiment of the present application.
As shown in fig. 2, before step S1100, the following steps are further included:
s1010, acquiring kernel configuration parameters of a local system;
the local system is an operating system of a server needing to install a big data cluster service, an Ambari cluster is built on the centOS system, the local system is the centOS system, the kernel configuration parameters are installation configuration parameters in the local system, the kernel configuration parameters include but are not limited to a system time zone, a language, a root user password, an ssh key and an authentication key, and when the local system is implemented, the target site is installed with a sharing server, the sharing server can share a sharing file comprising the kernel configuration parameters of the target site, and the system can further acquire the kernel configuration parameters of the local system by acquiring the sharing file.
S1020, setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
The system generates a system image file according to the acquired basic parameters of the kernel configuration parameters setting original images, wherein the original images are Ambari rpm (Red-Hat Package Manager, software package manager) files. When the method is implemented, the generation of the custom big data cluster image file needs to firstly determine the versions of a local system and the big data cluster system, taking the establishment of an Ambari cluster on a centOS system as an example, determining the versions of the centOS and the Ambari, acquiring an Ambari rpm file and a centOS original image file, and decompressing the centOS original image file to a temporary path; all rpm packets required in the CentOS system in the subsequent step are obtained from the network yum source by means of a simulated installation method and added to the temporary path, wherein the yum source is a software collective site; generating a yum source in the form of a local file using the rpm package described above; a kckstart file is written, which is a configuration file defining the Linux installation process, and centros provides a graphical kckstart configuration tool in which necessary system settings are made, including but not limited to: closing selinux, closing a firewall, setting a system time zone and a language, setting a root user password, an ssh key and an authentication key, setting kernel related parameters, generating a repo file for a yum source generated in the steps, adding the repo file into a system repo folder, and installing a necessary rpm package; adding the kckstart file into a temporary path; modifying an isolinux.cfg file in the temporary path, designating that the mirror image installation process needs to use the kckstart file in the above steps, and after the ISOLINUX is started, reading an isolinux.cfg file of the cd optical disc root directory by default, wherein the isolinux.cfg is similar to a menu.lst of grub, a designated kernel mirror image position and the like; calling a mkisofs command, wherein the mkisofs command can make the appointed directory and file into an image file in an ISO 9660 format, and make the content in the temporary path into a custom large data cluster image file; for a plurality of target sites, the steps can be repeated, and personalized kckstart files are manufactured, so that big data cluster image files corresponding to the target sites are manufactured.
In an alternative embodiment, referring to fig. 3, fig. 3 is a schematic flow chart illustrating a basic process of obtaining kernel configuration parameters of a local system according to an embodiment of the present application.
As shown in fig. 3, step S1010 includes the following steps:
s1001, acquiring original software package management information of the local system;
the original software package management information corresponds to an installation software package of the local system, and in implementation, the original software package management information can be obtained through a software package manager in the local system, and in one embodiment, the original software package management information carries address information of the installation software package of the local system.
S1002, acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
The system can acquire an installation software package of the local system according to the original software package management information, and when in implementation, the system can acquire the installation software package through the address information of the installation software package, wherein the installation software package comprises kernel configuration parameters of the local system, and the local system can keep the running environment parameters of the installation system, which is a server, into the installation software package when in installation.
In an alternative embodiment, referring to fig. 4, fig. 4 is a schematic flow diagram illustrating a configuration file according to an embodiment of the present application.
As shown in fig. 4, step S1200 includes the following steps:
s1210, acquiring each file sharing directory comprising file sharing terminal information of a corresponding target site, wherein any one of the file sharing terminal information comprises address information of a configuration file of the corresponding target site;
the server corresponding to the target site is provided with a file sharing server, the file sharing server creates a file sharing user, sets a password and sets a file sharing directory, the file sharing directory carries file sharing terminal information comprising configuration files of the corresponding target site, the file sharing terminal information comprises address information of the configuration files of the corresponding target site, when the file sharing server is implemented, the server corresponding to each target site stores the address information of the configuration files into the file sharing terminal information, the file sharing server creates the file sharing directory comprising the file sharing terminal information, and the system can acquire the address information of the configuration files of each target site through the file sharing directory of the corresponding target site.
S1220, acquiring configuration files of the corresponding target sites according to the address information, and configuring the configuration files according to the environment information of the corresponding target sites.
The system acquires corresponding configuration files according to the address information corresponding to each target site, and then sets the configuration files according to the environment information of the corresponding target site. In an alternative embodiment, referring to FIG. 5, a basic flow diagram of creating a file sharing directory in accordance with one embodiment of the present application is shown in FIG. 5.
As shown in fig. 5, step S1210 includes the following steps:
s1201, installing and starting a preset file sharing service on each target site;
the file sharing service is a preset service for sharing files on the servers in the system, and when the file sharing service is implemented, the preset file sharing service can be installed on the server corresponding to each target site.
S1202, respectively creating file sharing directories comprising file sharing terminal information of the corresponding target sites according to the file sharing services corresponding to the target sites.
The server of each target site starts the file sharing service to create a file sharing directory including file sharing terminal information of each target site. In one embodiment, configuration management of a large data cluster system may be accomplished through a file sharing service that includes: installing and starting a file sharing server on an Ambari server (target site), wherein the file sharing server provides a file sharing service; creating a file sharing user on an Ambari server, setting a password and setting a file sharing directory; all servers install file sharing clients; all servers mount the shared catalogue on the Ambari server by using the user and the password created in the steps, add the mounted catalogue into a startup item, and ensure that the mounting of the shared catalogue occurs before service startup; when a new service is added, the default parameters of the service are read from the service metadata configuration file at the Ambari server, the parameters are written into a database, and the parameters are written into the service configuration file corresponding to the file sharing directory. All servers link corresponding files or folders in the mounted shared directory to service configuration files or service configuration folders in a soft link mode, and then start the service; after detecting that a user modifies a parameter of a certain service, the Ambari server stores the modified parameter in a database, writes the parameter into a service configuration file corresponding to a file sharing directory, and restarts the service by sending a service restarting command to the agent so as to enable the latest parameter to be effective.
In an alternative embodiment, referring to fig. 6, fig. 6 is a schematic diagram illustrating a basic flow of installing a big data cluster system according to an embodiment of the present application.
As shown in fig. 6, step S1300 includes the following steps:
s1310, obtaining local area network address information of each target site;
the lan address information refers to MAC (Media Access Control Address) address of the destination station, and the MAC address is an address for identifying the location of the network device. In implementation, the method can be realized by the function of NetApi32.DLL built in Windows 9x/NT/Win2000, firstly, the number of network cards and the internal number of each network card are obtained by sending NCBENUM command, and then the NCBASIT command is sent to each network card label to obtain the MAC address of each network card.
S1320, establishing a corresponding relation between the big data cluster mirror image file and the corresponding target site according to local area network address information of each target site;
the corresponding big data cluster mirror image is installed in the target site, and when the method is implemented, the corresponding relation between each target site and the corresponding big data cluster mirror image file and the configuration file of the target site can be determined firstly by sending the big data cluster mirror image file to the corresponding target site Server, for example, the corresponding big data cluster mirror image file is determined according to the role of the Server to be installed, namely the role of the Server of the target site, the role of the Server (Server Roles) is used for describing the basic function of one Server, the MAC address of the role of the Server to be installed is obtained, and the corresponding relation between the MAC address and the big data cluster mirror image file of the corresponding target site is established, so that the function of establishing the corresponding relation between the big data cluster mirror image file and the corresponding target site is realized.
S1330, sending the big data cluster image file to the corresponding target sites according to the corresponding relation, so that each target site installs the big data cluster image file to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration file.
After the corresponding relation between each target site and the big data cluster image file and the configuration file of the target site is determined, the system sends the big data cluster image file to the server of the corresponding target site according to the corresponding relation, so that each target site installs the corresponding big data cluster system to generate the big data cluster system, and then the parameters of the corresponding big data cluster system are set according to the configuration file corresponding to the target site, and when the system is implemented, the system installation of the big data clusters can be realized by utilizing PXE batch, including that the big data cluster image files which are customized by utilizing the plurality of the user-defined files are placed on a BOOT server; according to the role of the server to be installed, the server of the target site, namely the installation server, determines a custom big data cluster mirror image file corresponding to the installation server; acquiring a corresponding relation between a role of a server to be installed and an MAC address; establishing a corresponding relation between the MAC address and a custom big data cluster mirror image file; placing the corresponding relation between the MAC address and the customized big data cluster image file on a BOOT server; when the server is installed, a corresponding customized big data cluster mirror image file is obtained from the BOOT server according to the MAC address; and the server uses the customized big data cluster image file and the personalized kckstart file in the image to install the Ambari related software. When the method is implemented, a plurality of custom big data cluster images are generated according to the needs, then PXE is utilized for quick batch installation, during batch installation, a server selects corresponding big data cluster images according to the MAC addresses of the big data cluster images to install and set, and when Ambari is utilized for unified management of big data components, the parameters of the big data components are managed in a unified way through a file sharing method, manual parameter setting is not needed for a plurality of nodes at the same time, and the installation speed of the big data clusters can be effectively accelerated.
In order to solve the technical problems, the embodiment of the application also provides a big data cluster management control device.
Referring to fig. 7 specifically, fig. 7 is a schematic diagram of a basic structure of a big data cluster management control device according to the present embodiment.
As shown in fig. 7, a big data cluster management control device includes: the system comprises a first acquisition module 2100, a first processing module 2200 and a first execution module 2300, wherein the first acquisition module 2100 is configured to acquire environmental information of each target site in a distributed server cluster; the first processing module 2200 is configured to configure a configuration file of each target site according to the environmental information of each target site; the first execution module 2300 is configured to correspondingly install the big data cluster system on each target site according to each configuration file and a preset big data cluster image file.
According to the embodiment, the environment information of each target site in the distributed server cluster is obtained, the configuration file of each target site is configured according to the environment information of each target site, and then the large data cluster system management control is carried out on each target site according to each configuration file and the preset large data cluster image file correspondingly, so that the large data cluster system is installed and the parameters are configured in batches, the installation speed of the large data clusters can be effectively increased, and the operation and maintenance efficiency of the large data clusters is improved.
In some embodiments, the big data cluster management control device further comprises: the system comprises a first acquisition module and a first execution module, wherein the first acquisition module is used for acquiring the kernel configuration parameters of a local system; the second execution module is used for generating the big data cluster image file according to the basic parameters of the kernel configuration parameter setting preset original image.
In some embodiments, the big data cluster management control device further comprises: the system comprises a first acquisition sub-module and a first execution sub-module, wherein the first acquisition sub-module is used for acquiring original software package management information of the local system; the first execution submodule is used for acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
In some embodiments, the big data cluster management control device further comprises: the second acquisition sub-module is used for acquiring a file sharing directory comprising file sharing terminal information of each target site, wherein the file sharing terminal information comprises address information of the configuration file; the second execution submodule is used for acquiring the configuration file according to the address information and configuring the configuration file according to the environment information of the corresponding target site.
In some embodiments, the big data cluster management control device further comprises: the first processing sub-module is used for installing and starting a preset file sharing service on each target site; and the third execution submodule is used for creating the file sharing directory comprising the file sharing terminal information according to the file sharing service.
In some embodiments, the big data cluster management control device further comprises: the second processing sub-module is used for determining the corresponding relation between each target site and the big data cluster image file and the configuration file; and the third execution submodule is used for sending the big data cluster image file to a corresponding target site for installation according to the corresponding relation, and setting parameters of the big data cluster system according to the corresponding configuration file.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
In order to solve the technical problems, the embodiment of the application also provides computer equipment. Referring specifically to fig. 8, fig. 8 is a basic structural block diagram of a computer device according to the present embodiment.
As shown in fig. 8, the internal structure of the computer device is schematically shown. As shown in fig. 8, the computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The nonvolatile storage medium of the computer device stores an operating system, a database and a computer readable instruction, the database can store a control information sequence, and the computer readable instruction can enable the processor to realize a big data cluster management control method when being executed by the processor. The processor of the computer device is used to provide computing and control capabilities, supporting the operation of the entire computer device. The memory of the computer device may have stored therein computer readable instructions that, when executed by the processor, cause the processor to perform a big data cluster management control method. The network interface of the computer device is for communicating with a terminal connection. It will be appreciated by persons skilled in the art that the structures shown in the drawings are block diagrams of only some of the structures associated with the inventive arrangements and are not limiting of the computer device to which the inventive arrangements may be implemented, and that a particular computer device may include more or less elements than those shown, or may be combined with some elements or have a different arrangement of elements.
The processor in this embodiment is configured to execute the first acquisition module 2100, the first processing module 2200, and the first execution module 2300 in fig. 7, and the memory stores program codes and various types of data required for executing the above modules. The network interface is used for data transmission between the user terminal or the server. The memory in this embodiment stores program codes and data required for executing all the sub-modules in the large data cluster management control device, and the server can call the program codes and data of the server to execute the functions of all the sub-modules.
The computer obtains the environment information of each target site in the distributed server cluster, configures the configuration file of each target site according to the environment information of each target site, and correspondingly installs the big data cluster system for each target site according to each configuration file and the preset big data cluster image file, and installs the big data cluster systems and configures parameters in batches, so that the installation speed of the big data clusters can be effectively increased, and the operation and maintenance efficiency of the big data clusters can be improved.
The present application also provides a storage medium storing computer readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the big data cluster management control method of any of the embodiments described above.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by way of a computer program stored in a computer-readable storage medium, which when executed, may comprise the steps of embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a random access Memory (Random Access Memory, RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The foregoing is only a partial embodiment of the present application, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present application, and such modifications and adaptations are intended to be comprehended within the scope of the present application.
Claims (8)
1. The big data cluster management control method is characterized by comprising the following steps:
acquiring environment information of each target site in a distributed server cluster;
acquiring each file sharing directory comprising file sharing terminal information of a corresponding target site, wherein any one of the file sharing terminal information comprises address information of a configuration file of the corresponding target site; acquiring configuration files of corresponding target sites according to the address information, and configuring the configuration files of each target site according to the environment information of each target site to obtain configuration files modified by each target site;
acquiring local area network address information of each target site; establishing a corresponding relation between the big data cluster image file and the corresponding target site according to the local area network address information of each target site; and sending the big data cluster image files to corresponding target sites according to the corresponding relation, and correspondingly carrying out big data cluster system management control on each target site according to the configuration files modified by each target site and the preset big data cluster image files, wherein the big data cluster system management control on each target site comprises the steps of installing the big data cluster image files on each target site to generate a big data cluster system, and setting parameters of the big data cluster system according to the modified configuration files.
2. The big data cluster management control method according to claim 1, wherein before the step of obtaining the environmental information of each target site in the distributed server cluster, the method further comprises the steps of:
obtaining kernel configuration parameters of a local system;
and setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
3. The big data cluster management control method according to claim 2, wherein the step of acquiring the kernel configuration parameters of the local system includes the steps of:
acquiring original software package management information of the local system;
and acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
4. The big data cluster management control method according to claim 1, wherein the step of acquiring the file sharing directory including the file sharing terminal information of each target site further comprises the steps of:
installing and starting a preset file sharing service on each target site;
and respectively creating file sharing catalogues comprising file sharing terminal information of the corresponding target sites according to the file sharing services of the corresponding target sites.
5. A big data cluster management control apparatus, comprising:
the first acquisition module is used for acquiring the environment information of each target site in the distributed server cluster;
the first processing module is used for acquiring each file sharing directory comprising file sharing terminal information of a corresponding target site, wherein any one of the file sharing terminal information comprises address information of a configuration file of the corresponding target site; acquiring configuration files of corresponding target sites according to the address information, and configuring the configuration files of each target site according to the environment information of each target site to obtain configuration files modified by each target site;
the first execution module is used for acquiring local area network address information of each target site; establishing a corresponding relation between the big data cluster image file and the corresponding target site according to the local area network address information of each target site; and sending the big data cluster image files to corresponding target sites according to the corresponding relation, and correspondingly carrying out big data cluster system management control on each target site according to the configuration files modified by each target site and the preset big data cluster image files, wherein the big data cluster system management control on each target site comprises the steps of installing the big data cluster image files on each target site to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration files.
6. The big data cluster management control device according to claim 5, further comprising:
the second acquisition module is used for acquiring kernel configuration parameters of the local system;
and the second execution module is used for generating the big data cluster image file according to the basic parameters of the kernel configuration parameter setting preset original image.
7. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the big data cluster management control method of any of claims 1 to 4.
8. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the big data cluster management control method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911150335.0A CN111026723B (en) | 2019-11-21 | 2019-11-21 | Big data cluster management control method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911150335.0A CN111026723B (en) | 2019-11-21 | 2019-11-21 | Big data cluster management control method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111026723A CN111026723A (en) | 2020-04-17 |
| CN111026723B true CN111026723B (en) | 2023-08-11 |
Family
ID=70206335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911150335.0A Active CN111026723B (en) | 2019-11-21 | 2019-11-21 | Big data cluster management control method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026723B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112702203A (en) * | 2020-12-22 | 2021-04-23 | 上海智迩智能科技有限公司 | Nginx cluster white screen configuration management method and system |
| CN113031964B (en) * | 2021-03-25 | 2023-12-26 | 恒安嘉新（北京）科技股份公司 | Big data application management method, device, equipment and storage medium |
| CN113726546B (en) * | 2021-07-12 | 2023-11-21 | 锐捷网络股份有限公司 | Configuration method, device, system, computing equipment and storage medium |
| CN114022331A (en) * | 2021-10-15 | 2022-02-08 | 金茂数字科技有限公司 | Wisdom thing allies oneself with data platform |
| CN115827392B (en) * | 2023-02-09 | 2023-11-21 | 中国证券登记结算有限责任公司 | Monitoring method, device and system of distributed system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7496783B1 (en) * | 2006-02-09 | 2009-02-24 | Symantec Operating Corporation | Merging cluster nodes during a restore |
| CN104468199A (en) * | 2014-11-23 | 2015-03-25 | 国云科技股份有限公司 | Frame and running method suitable for deployment and management of Linux distributed system |
| WO2016112825A1 (en) * | 2015-01-13 | 2016-07-21 | 华为技术有限公司 | Remote control method, terminal device, management server, and remote control system |
| WO2016127756A1 (en) * | 2015-02-15 | 2016-08-18 | 北京京东尚科信息技术有限公司 | Flexible deployment method for cluster and management system |
| CN106790483A (en) * | 2016-12-13 | 2017-05-31 | 武汉邮电科学研究院 | Hadoop group systems and fast construction method based on container technique |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7788335B2 (en) * | 2001-01-11 | 2010-08-31 | F5 Networks, Inc. | Aggregated opportunistic lock and aggregated implicit lock management for locking aggregated files in a switched file system |
| US8788768B2 (en) * | 2010-09-29 | 2014-07-22 | International Business Machines Corporation | Maintaining mirror and storage system copies of volumes at multiple remote sites |
-
2019
- 2019-11-21 CN CN201911150335.0A patent/CN111026723B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7496783B1 (en) * | 2006-02-09 | 2009-02-24 | Symantec Operating Corporation | Merging cluster nodes during a restore |
| CN104468199A (en) * | 2014-11-23 | 2015-03-25 | 国云科技股份有限公司 | Frame and running method suitable for deployment and management of Linux distributed system |
| WO2016112825A1 (en) * | 2015-01-13 | 2016-07-21 | 华为技术有限公司 | Remote control method, terminal device, management server, and remote control system |
| WO2016127756A1 (en) * | 2015-02-15 | 2016-08-18 | 北京京东尚科信息技术有限公司 | Flexible deployment method for cluster and management system |
| CN106790483A (en) * | 2016-12-13 | 2017-05-31 | 武汉邮电科学研究院 | Hadoop group systems and fast construction method based on container technique |
Non-Patent Citations (1)
| Title |
|---|
| Design of the Server Cluster for the Scalable Networked Virtual Environment;Jiung-yao Huang 等;《semantic scholar》;第1-7页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111026723A (en) | 2020-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111026723B (en) | Big data cluster management control method and device, computer equipment and storage medium | |
| US11301287B2 (en) | Pattern-based orchestration of cloud provisioning tasks at runtime | |
| CN110622129B (en) | Method, system, and portal for accelerating aspects of data analysis application development and deployment using software containers | |
| CN109274722B (en) | Data sharing method and device and electronic equipment | |
| US9483281B2 (en) | Methods, systems, and computer readable mediums for updating components in a converged infrastructure system | |
| CN111782232A (en) | Cluster deployment method and device, terminal equipment and storage medium | |
| CN109687987A (en) | A kind of cloud platform dispositions method, device, electronic equipment and readable storage medium storing program for executing | |
| US20150358392A1 (en) | Method and system of virtual desktop infrastructure deployment studio | |
| US20150095473A1 (en) | Automatic configuration of applications based on host metadata using application-specific templates | |
| US11144292B2 (en) | Packaging support system and packaging support method | |
| CN111580926A (en) | Model publishing method, model deploying method, model publishing device, model deploying device, model publishing equipment and storage medium | |
| CN113626286A (en) | Multi-cluster instance processing method and device, electronic equipment and storage medium | |
| CN111414391A (en) | Method and system for accessing multiple data sources | |
| CN112346818A (en) | Container application deployment method and device, electronic equipment and storage medium | |
| CN113419813B (en) | Method and device for deploying bare engine management service based on container platform | |
| CN117112122A (en) | Cluster deployment method and device | |
| CN114489954A (en) | Tenant creation method based on virtualization platform, tenant access method and equipment | |
| CN116208498A (en) | Method, device, equipment and medium for node differential configuration of OpenStack cloud platform | |
| CN110162312A (en) | A kind of BeeGFS configuration method and device based on IML | |
| CN115525396A (en) | Application management method and device based on cloud protogenesis | |
| US11425203B2 (en) | Commissioning a virtualized network function | |
| CN114564211A (en) | Cluster deployment method, cluster deployment device, equipment and medium | |
| CN110971642B (en) | Data processing method and device for cloud computing platform | |
| Mardan et al. | Getting Node. js Apps Production Ready | |
| Telang | Containerizing Microservices Using Kubernetes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |