CN111026723A - Big data cluster management control method and device, computer equipment and storage medium - Google Patents
Big data cluster management control method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111026723A CN111026723A CN201911150335.0A CN201911150335A CN111026723A CN 111026723 A CN111026723 A CN 111026723A CN 201911150335 A CN201911150335 A CN 201911150335A CN 111026723 A CN111026723 A CN 111026723A
- Authority
- CN
- China
- Prior art keywords
- target site
- big data
- data cluster
- file
- management control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/161—Computing infrastructure, e.g. computer clusters, blade chassis or hardware partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Stored Programmes (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a big data cluster management control method, a device, computer equipment and a storage medium, which comprises the following steps: acquiring environment information of each target site in a distributed server cluster; configuring a configuration file of each target site correspondingly according to the environmental information of each target site; and correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster mirror image file. According to the embodiment of the invention, the installation speed of the big data cluster can be effectively accelerated and the operation and maintenance efficiency of the big data cluster can be improved by acquiring the environmental information of each target site in the distributed server cluster, configuring the configuration file of each target site according to the environmental information of each target site, correspondingly installing the big data cluster system on each target site according to each configuration file and the preset big data cluster mirror image file, and carrying out the installation and parameter configuration of the big data cluster system in batches.
Description
Technical Field
The embodiment of the invention relates to the technical field of big data platforms, in particular to a big data cluster management control method and device, computer equipment and a storage medium.
Background
In the stage of high development of information, the role and the position of data processing are higher and higher, and many companies construct their own big data platforms for creating, managing and monitoring big data platform system clusters.
However, a large data cluster generally needs a plurality of servers, so that it takes a lot of time to perform repetitive work to build the large data cluster, and for a large data component in the large data cluster, parameter setting may need to be performed on a plurality of nodes, for example, a time zone, a language, or even a root user password needs to be set, and an operation and maintenance worker needs to spend a lot of time to participate in an installation process, thereby reducing the installation speed and the operation and maintenance efficiency of the large data cluster.
Disclosure of Invention
The embodiment of the invention provides a large data cluster management control method and device, computer equipment and a storage medium, which can improve the installation efficiency of a large data cluster.
In order to solve the above technical problem, the embodiment of the present invention adopts a technical solution that: the big data cluster management control method comprises the following steps:
acquiring environment information of each target site in a distributed server cluster;
configuring a configuration file of each target site according to the environment information of each target site correspondingly;
and correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster mirror image file.
Optionally, before the step of obtaining the environment information of each target site in the distributed server cluster, the method further includes the following steps:
acquiring kernel configuration parameters of a local system;
and setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
Optionally, the step of obtaining the kernel configuration parameter of the local system includes the following steps:
acquiring original software package management information of the local system;
and acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
Optionally, the step of configuring the configuration file of each target site according to the environment information of each target site includes the following steps:
acquiring each file sharing directory including file sharing terminal information of a corresponding target site, wherein any file sharing terminal information includes address information of a configuration file of the corresponding target site;
and acquiring a configuration file of the corresponding target site according to the address information, and configuring the configuration file according to the environment information of the corresponding target site.
Optionally, before the step of obtaining the file sharing directory including the file sharing terminal information of each target site, the method further includes the following steps:
installing and starting a preset file sharing service on each target site;
and respectively creating a file sharing directory including the file sharing terminal information of the corresponding target site according to the file sharing service corresponding to each target site.
Optionally, the step of correspondingly performing big data cluster system management control on each target site according to each configuration file and a preset big data cluster image file includes the following steps:
acquiring local area network address information of each target station;
establishing a corresponding relation between the big data cluster mirror image file and the corresponding target site according to the local area network address information of each target site;
and sending the big data cluster mirror image file to the corresponding target sites according to the corresponding relation so that each target site installs the big data cluster mirror image file to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration file.
In order to solve the above technical problem, an embodiment of the present invention further provides a big data cluster management control apparatus, including:
the first acquisition module is used for acquiring the environmental information of each target site in the distributed server cluster;
the first processing module is used for configuring a configuration file of each target site according to the environment information of each target site correspondingly;
and the first execution module is used for correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster mirror image file.
Optionally, the method further comprises:
the second acquisition module is used for acquiring the kernel configuration parameters of the local system;
and the second execution module is used for setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
Optionally, the method further comprises:
the first acquisition submodule is used for acquiring the original software package management information of the local system;
and the first execution submodule is used for acquiring an installation software package corresponding to the local system according to the management information of the original software package, wherein the installation software package comprises the kernel configuration parameters.
Optionally, the method further comprises:
the second obtaining submodule is used for obtaining each file sharing directory comprising file sharing terminal information of a corresponding target site, wherein any file sharing terminal information comprises address information of a configuration file of the corresponding target site;
and the second execution submodule is used for acquiring the configuration file of the corresponding target site according to the address information and configuring the configuration file according to the environment information of the corresponding target site.
Optionally, the method further comprises:
the first processing submodule is used for installing and starting a preset file sharing service on each target site;
and the third execution submodule is used for respectively creating a file sharing directory comprising the file sharing terminal information of the corresponding target site according to the file sharing service corresponding to each target site.
Optionally, the method further comprises:
the third acquisition submodule is used for acquiring the local area network address information of each target site;
the second processing submodule is used for establishing a corresponding relation between the big data cluster mirror image file and the corresponding target site according to the local area network address information of each target site;
and the third execution submodule is used for sending the big data cluster mirror image file to the corresponding target sites according to the corresponding relation so as to enable each target site to install the big data cluster mirror image file to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration file.
In order to solve the above technical problem, an embodiment of the present invention further provides a computer device, including a memory and a processor, where the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to execute the steps of the above big data cluster management control method.
To solve the above technical problem, an embodiment of the present invention further provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to execute the steps of the above large data cluster management control method.
The embodiment of the invention has the beneficial effects that: the method comprises the steps of acquiring the environmental information of each target site in the distributed server cluster, configuring the configuration file of each target site according to the environmental information of each target site, correspondingly managing and controlling the big data cluster system of each target site according to each configuration file and the preset big data cluster mirror image file, and carrying out installation and parameter configuration on the big data cluster system in batches, so that the installation speed of the big data cluster can be effectively accelerated, and the operation and maintenance efficiency of the big data cluster is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a basic flow chart of a big data cluster management control method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of customizing a big data cluster mirror image file according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of acquiring a kernel configuration parameter of a local system according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a process of setting a configuration file according to an embodiment of the present invention;
FIG. 5 is a basic flowchart illustrating the creation of a file sharing directory according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a basic flow for installing a big data cluster system according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a basic structure of a big data cluster management control apparatus according to an embodiment of the present invention;
FIG. 8 is a block diagram of the basic structure of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.
In some of the flows described in the present specification and claims and in the above figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, with the order of the operations being indicated as 101, 102, etc. merely to distinguish between the various operations, and the order of the operations by themselves does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
Specifically, referring to fig. 1, fig. 1 is a basic flowchart of the big data cluster management control method according to the embodiment.
As shown in fig. 1, a big data cluster management control method includes the following steps:
s1100, acquiring environment information of each target site in the distributed server cluster;
the distributed server cluster is that a set of system is divided into a plurality of subsystems and then deployed on different servers, wherein a target site refers to a server, environment information of the target site represents an operating environment of the server, and in implementation, an Apache Ambari cluster is built on a CentOS system, wherein a CentOS (Community Enterprise operating System) is one of Linux distribution versions and is compiled by source codes released by Red Hat Enterprise Linux according to open source code regulations; apache Ambari is a Web-based tool and supports the creation, management and monitoring of Apache Hadoop clusters, Ambari supports most Hadoop components including HDFS, MapReduce, Hive, Pig, Hbase, Zookeeper, Sqoop, Hcatalog and the like, and in addition, Ambari also supports computing frameworks such as Spark, Storm and the like and a resource scheduling platform YARN. The Ambari cluster comprises a plurality of servers, a corresponding big data cluster mirror image is installed on each server, a corresponding configuration file needs to be set according to the environment of each server, the environment information of each target site can be uniformly managed through a file sharing method, each target site is provided with a shared file, the shared file comprises the environment information of each target site, and the system can acquire the environment information of each target site by acquiring the shared file of each target site.
S1200, configuring a configuration file of each target site correspondingly according to the environment information of each target site;
the operating environments of different target sites are different, so configuration files of each target site can be correspondingly set according to the environment information of each target site, and in implementation, the environment information of the target site includes but is not limited to: the time zone, the language, the user password are accessed without a secret key ssh, a database and the like, the configuration file carries the installation configuration parameters of the corresponding target site, and the system can modify the installation configuration parameters in the corresponding configuration file through the environment information of the target site. In one embodiment, the system may also generate a configuration file for each target site based on the environmental information for each target site.
S1300, correspondingly managing and controlling the big data cluster system of each target site according to each configuration file and a preset big data cluster mirror image file.
The system manages and controls installation and parameter setting of a big data cluster system in a target site according to a configuration file corresponding to the target site and a preset big data cluster mirror image file aiming at each target site, and in implementation, taking building an Ambari cluster in a CentOS operating system as an example, the system acquires environment information of each target site, such as target site 1 and target site 2, wherein a server mapped by the target site 1 is in an eight-east time zone, a server mapped by the target site 2 is in a five-east time zone, corresponding time zones need to be set when big data cluster system services are installed on the target site 1 and the target site 2, the system correspondingly sets a configuration file of each target site according to the environment information of each target site, and the configuration file includes, but is not limited to, time zone, language, root user password, no-key ssh access, database service, Network connection and other operation environment information are obtained, then a big data cluster system is installed in a target site through a big data cluster mirror image file, parameters of the big data cluster system are set according to corresponding configuration files, for example, the big data cluster system is installed on servers of the target site 1 and a target site 2, but the big data cluster system in the target site 1 is correspondingly configured to be an east eight time zone, the big data cluster system in the target site 2 is correspondingly configured to be an east five time zone, and of course, language, root user password, no-key ssh access setting, a database and the like are set according to information carried in the configuration files, so that the big data cluster system is set up in batches, and operation and maintenance personnel do not need to spend time to participate in the installation process to set the parameters.
The embodiment of the invention can effectively accelerate the installation speed of the big data cluster and improve the operation and maintenance efficiency of the big data cluster by acquiring the environmental information of each target site in the distributed server cluster, configuring the configuration file of each target site according to the environmental information of each target site, correspondingly managing and controlling the big data cluster system of each target site according to each configuration file and the preset big data cluster mirror image file, and carrying out the installation and parameter configuration of the big data cluster system in batches.
In an alternative embodiment, please refer to fig. 2, fig. 2 is a basic flowchart of customizing a big data cluster image file according to an embodiment of the present invention.
As shown in fig. 2, before step S1100, the following steps are further included:
s1010, obtaining kernel configuration parameters of a local system;
the local system is an operating system of a server needing to install big data cluster service, for example, an Ambari cluster is built on a CentOS system, the local system is the CentOS system, the kernel configuration parameters are installation configuration parameters in the local system, the kernel configuration parameters include but are not limited to a system time zone, a language, a root user password, a ssh key and an authentication key, when the implementation is carried out, the target sites are all provided with shared servers, the shared servers can share a shared file including the kernel configuration parameters of the target sites, and the system can further obtain the kernel configuration parameters of the local system by obtaining the shared file.
S1020, setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
The system sets basic parameters of an original image according to the acquired kernel configuration parameters to generate a system image file, wherein the original image is an Ambari rpm (Red-Hat Package Manager) file. In implementation, the generation of a custom big data cluster image file requires determining the versions of a local system and a big data cluster system, taking the construction of an Ambari cluster on a CentOS system as an example, determining the versions of the CentOS and the Ambari, acquiring an Ambari rpm file and a CentOS original image file, and decompressing the CentOS original image file to a temporary path; all rpm packets required in the CentOS system in the subsequent steps are obtained from a network yum source through a simulation installation method, and are added into a temporary path, wherein the yum source is a software assembly place; generating yum source in local file form by using the rpm package; writing a kickstart file, which is a configuration file defining the Linux installation process, and centros provides a graphical kickstart configuration tool, in which necessary system settings are made, including but not limited to: closing selinux, closing a firewall, setting a system time zone and language, setting a root user password, a ssh key and an authentication key, setting related parameters of a kernel, generating a repo file for an yum source generated in the steps, adding the repo file into a system repo folder, and installing a necessary rpm package; adding the kickstart file to a temporary path; modifying the isolinux.cfg file in the temporary path, specifying that the image installation process needs to use the kickstart file in the above step, and after starting the ISOLINUX, reading an isolinux.cfg file of the cd optical disk root directory by default, wherein the isolinux.cfg is similar to menu.lst of grub, and specifying the kernel image position and the like; calling an mkisofs command, wherein the mkisofs command can make a specified directory and a specified file into a mapping file in an ISO 9660 format, and make the content in a temporary path into a self-defined big data cluster mirror image file; for a plurality of target sites, the steps can be repeated to produce an individualized kickstart file, and further produce a big data cluster mirror image file corresponding to each target site.
In an alternative embodiment, please refer to fig. 3, where fig. 3 is a schematic diagram illustrating a basic flow for acquiring kernel configuration parameters of a local system according to an embodiment of the present invention.
As shown in fig. 3, step S1010 includes the following steps:
s1001, acquiring original software package management information of the local system;
the original software package management information corresponds to an installation software package of the local system, and in implementation, the original software package management information may be obtained through a software package manager in the local system.
S1002, acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
The system can obtain the installation software package of the local system according to the management information of the original software package, and when the system is implemented, the system can obtain the installation software package through the address information of the installation software package, wherein the installation software package comprises the kernel configuration parameters of the local system, and the local system can keep the operation environment parameters of the installation system, which is a server, in the installation software package.
In an alternative embodiment, please refer to fig. 4, where fig. 4 is a basic flowchart illustrating setting a configuration file according to an embodiment of the present invention.
As shown in fig. 4, step S1200 includes the following steps:
s1210, obtaining each file sharing directory including file sharing terminal information of a corresponding target site, wherein any file sharing terminal information includes address information of a configuration file of the corresponding target site;
the method comprises the steps that a server corresponding to a target site is provided with a file sharing server, the file sharing server creates a file sharing user and sets a password, and sets a file sharing directory, the file sharing directory carries file sharing terminal information comprising the corresponding target site, the file sharing terminal information comprises address information of a configuration file of the corresponding target site, when the method is implemented, the server corresponding to each target site saves the address information of the configuration file into the file sharing terminal information, the file sharing server creates the file sharing directory comprising the file sharing terminal information, and a system can obtain the address information of the configuration file of each target site through the file sharing directory of the corresponding target site.
S1220, obtaining a configuration file of the corresponding target site according to the address information, and configuring the configuration file according to the environment information of the corresponding target site.
The system acquires corresponding configuration files according to the address information corresponding to each target site, and then sets the configuration files according to the environment information of the corresponding target sites. In an alternative embodiment, please refer to fig. 5, which is a schematic diagram illustrating a basic flow of creating a file sharing directory according to an embodiment of the present invention in fig. 5.
As shown in fig. 5, step S1210 includes the following steps:
s1201, installing and starting a preset file sharing service on each target site;
the file sharing service is a service preset in the system and used for sharing files on the servers, and during implementation, the preset file sharing service can be installed on the server corresponding to each target site.
S1202, respectively creating a file sharing directory including file sharing terminal information of the corresponding target sites according to the file sharing service corresponding to each target site.
The server of each target site starts the file sharing service to create a file sharing directory including file sharing terminal information of each target site. In one embodiment, configuration management of a big data cluster system may be accomplished through a file sharing service that includes: installing and starting a file sharing server on an Ambari server (target site), wherein the file sharing server provides a file sharing service; establishing a file sharing user on the Ambari server, setting a password, and setting a file sharing directory; all servers install file sharing clients; all servers use the user and the password created in the step to mount the shared directory on the Ambari server, and the mounted directory is added into the starting item, so that the mounting of the shared directory is ensured to occur before the service is started; when a new service is added, reading the default parameters of the service from the service metadata configuration file at the Ambari server, writing the parameters into the database, and writing the parameters into the service configuration file corresponding to the file sharing directory. All servers link corresponding files or folders in the mounted shared directory to service configuration files or service configuration folders in a soft link mode, and then start the service; after detecting that a user modifies the parameters of a certain service, the Ambari server stores the modified parameters into a database, writes the parameters into a service configuration file corresponding to the file sharing directory, and enables the service to be restarted so as to enable the latest parameters to take effect by sending a service restarting command to the agent.
In an alternative embodiment, please refer to fig. 6, where fig. 6 is a basic flowchart illustrating the installation of a big data cluster system according to an embodiment of the present invention.
As shown in fig. 6, step S1300 includes the following steps:
s1310, obtaining local area network address information of each target site;
the lan address information refers to a MAC (media Access Control address) address of the target station, and the MAC address is an address for confirming the location of the network device. In implementation, the method can be realized through the function of NetApi32.DLL built in Windows 9x/NT/Win2000, firstly, the number of the network cards and the internal number of each network card are obtained by sending NCBENUM commands, and then, the MAC address of each network card is obtained by sending NCBASTAT commands to the labels of the network cards.
S1320, establishing a corresponding relation between the big data cluster mirror image file and the corresponding target site according to the local area network address information of each target site;
the corresponding big data cluster mirror image is installed in the target site, the big data cluster mirror image file can be sent to the corresponding target site Server, when in implementation, the corresponding relation between each target site and the big data cluster mirror image file and the configuration file corresponding to the target site can be determined firstly, for example, the big data cluster mirror image file and the Server role (Server rolls) corresponding to the target site can be determined according to the role of the Server to be installed, namely the Server role of the target site, the Server role (Server rolls) is used for describing the basic function of one Server, the MAC address of the role of the Server to be installed is obtained, the corresponding relation between the MAC address and the big data cluster mirror image file of the corresponding target site is established, and therefore the function of establishing the corresponding relation between the big data cluster mirror image file and the corresponding target site is achieved.
And S1330, sending the big data cluster image file to corresponding target sites according to the corresponding relation so that each target site installs the big data cluster image file to generate a big data cluster system, and setting parameters of the big data cluster system according to corresponding configuration files.
After the corresponding relation between each target site and the big data cluster mirror image file and the configuration file of each target site is determined, the system sends the big data cluster mirror image file to the server of the corresponding target site according to the corresponding relation aiming at each target site, so that each target site installs the corresponding big data cluster system to generate the big data cluster system, and then sets the parameter of the corresponding big data cluster system according to the configuration file corresponding to the target site; according to the role of a server to be installed, a server of a target site, namely the installation server, determines a self-defined big data cluster mirror image file corresponding to the server; acquiring the corresponding relation between the role of the server to be installed and the MAC address; establishing a corresponding relation between the MAC address and the self-defined big data cluster mirror image file; placing the corresponding relation between the MAC address and the self-defined big data cluster mirror image file on a BOOT server; when the server is installed, acquiring a corresponding self-defined big data cluster mirror image file from a BOOT server according to the MAC address; and the server uses the self-defined big data cluster image file and the personalized kickstart file in the image to install Ambari related software. When the method is implemented, firstly, a plurality of self-defined big data cluster mirror images are generated according to needs, then PXE is used for quickly installing in batches, when the big data cluster mirror images are installed and set in batches, the server selects corresponding big data cluster mirror images according to MAC addresses of the server, when Ambari is used for uniformly managing big data assemblies, parameters of the big data assemblies are uniformly managed by a file sharing method, manual parameter setting of a plurality of nodes is not needed, and the installation speed of the big data clusters can be effectively accelerated.
In order to solve the above technical problem, an embodiment of the present invention further provides a big data cluster management control device.
Referring to fig. 7, fig. 7 is a schematic diagram of a basic structure of a big data cluster management control device according to the present embodiment.
As shown in fig. 7, a large data cluster management control apparatus includes: the system comprises a first obtaining module 2100, a first processing module 2200 and a first executing module 2300, wherein the first obtaining module 2100 is configured to obtain environment information of each target site in a distributed server cluster; the first processing module 2200 is configured to configure a configuration file of each target site according to the environment information of each target site; the first executing module 2300 is configured to perform big data cluster system installation on each target site correspondingly according to each configuration file and a preset big data cluster image file.
In this embodiment, the installation speed of the big data cluster can be effectively increased and the operation and maintenance efficiency of the big data cluster can be improved by acquiring the environmental information of each target site in the distributed server cluster, configuring the configuration file of each target site according to the environmental information of each target site, and correspondingly performing the big data cluster system management control on each target site according to each configuration file and the preset big data cluster image file.
In some embodiments, the big data cluster management control apparatus further includes: the second acquisition module is used for acquiring the kernel configuration parameters of the local system; and the second execution module is used for setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
In some embodiments, the big data cluster management control apparatus further includes: the system comprises a first acquisition submodule and a first execution submodule, wherein the first acquisition submodule is used for acquiring the management information of an original software package of the local system; the first execution submodule is used for acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
In some embodiments, the big data cluster management control apparatus further includes: the second obtaining submodule is used for obtaining a file sharing directory comprising file sharing terminal information of each target site, wherein the file sharing terminal information comprises address information of the configuration file; and the second execution submodule is used for acquiring the configuration file according to the address information and configuring the configuration file according to the environment information of the corresponding target site.
In some embodiments, the big data cluster management control apparatus further includes: the system comprises a first processing submodule and a third execution submodule, wherein the first processing submodule is used for installing and starting a preset file sharing service on each target site; and the third execution sub-module is used for creating the file sharing directory comprising the file sharing terminal information according to the file sharing service.
In some embodiments, the big data cluster management control apparatus further includes: the second processing submodule is used for determining the corresponding relation between each target site and the image file and the configuration file of the big data cluster; and the third execution submodule is used for sending the big data cluster image file to a corresponding target site for installation according to the corresponding relation and setting parameters of the big data cluster system according to a corresponding configuration file.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
In order to solve the above technical problem, an embodiment of the present invention further provides a computer device. Referring to fig. 8, fig. 8 is a block diagram of a basic structure of a computer device according to the present embodiment.
As shown in fig. 8, the internal structure of the computer device is schematically illustrated. As shown in fig. 8, the computer apparatus includes a processor, a nonvolatile storage medium, a memory, and a network interface connected through a system bus. The non-volatile storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store control information sequences, and the computer readable instructions can enable the processor to realize a big data cluster management control method when being executed by the processor. The processor of the computer device is used for providing calculation and control capability and supporting the operation of the whole computer device. The memory of the computer device may have stored therein computer readable instructions that, when executed by the processor, cause the processor to perform a big data cluster management control method. The network interface of the computer device is used for connecting and communicating with the terminal. It will be appreciated by those skilled in the art that the configurations shown in the figures are block diagrams of only some of the configurations relevant to the present application, and do not constitute a limitation on the computing devices to which the present application may be applied, and that a particular computing device may include more or less components than those shown in the figures, or may combine certain components, or have a different arrangement of components.
In this embodiment, the processor is configured to execute the first obtaining module 2100, the first processing module 2200, and the first executing module 2300 in fig. 7, and the memory stores program codes and various data required for executing the modules. The network interface is used for data transmission to and from a user terminal or a server. The memory in this embodiment stores program codes and data necessary for executing all the submodules in the large data cluster management and control device, and the server can call the program codes and data of the server to execute the functions of all the submodules.
The computer acquires the environmental information of each target site in the distributed server cluster, configures the configuration file of each target site according to the environmental information of each target site, and correspondingly installs the big data cluster system on each target site according to each configuration file and the preset big data cluster mirror image file, so that the installation speed and the parameter configuration of the big data cluster system can be effectively accelerated, and the operation and maintenance efficiency of the big data cluster is improved.
The present invention also provides a storage medium storing computer-readable instructions, which when executed by one or more processors, cause the one or more processors to perform the steps of the big data cluster management control method according to any of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A big data cluster management control method is characterized by comprising the following steps:
acquiring environment information of each target site in a distributed server cluster;
configuring a configuration file of each target site according to the environment information of each target site correspondingly;
and correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster mirror image file.
2. The big data cluster management control method according to claim 1, wherein the step of obtaining the environment information of each target site in the distributed server cluster further comprises the following steps:
acquiring kernel configuration parameters of a local system;
and setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
3. The big data cluster management control method according to claim 2, wherein the step of obtaining the kernel configuration parameters of the local system comprises the steps of:
acquiring original software package management information of the local system;
and acquiring an installation software package corresponding to the local system according to the original software package management information, wherein the installation software package comprises the kernel configuration parameters.
4. The big data cluster management control method according to claim 1, wherein the step of configuring the configuration file of each target site according to the environment information of each target site comprises the steps of:
acquiring each file sharing directory including file sharing terminal information of a corresponding target site, wherein any file sharing terminal information includes address information of a configuration file of the corresponding target site;
and acquiring a configuration file of the corresponding target site according to the address information, and configuring the configuration file according to the environment information of the corresponding target site.
5. The big data cluster management control method according to claim 4, wherein before the step of obtaining the file sharing directory including the file sharing terminal information of each target site, the method further comprises the steps of:
installing and starting a preset file sharing service on each target site;
and respectively creating a file sharing directory including the file sharing terminal information of the corresponding target site according to the file sharing service corresponding to each target site.
6. The big data cluster management control method according to claim 1, wherein the step of correspondingly performing big data cluster system management control on each target site according to each configuration file and a preset big data cluster image file comprises the following steps:
acquiring local area network address information of each target station;
establishing a corresponding relation between the big data cluster mirror image file and the corresponding target site according to the local area network address information of each target site;
and sending the big data cluster mirror image file to the corresponding target sites according to the corresponding relation so that each target site installs the big data cluster mirror image file to generate a big data cluster system, and setting parameters of the big data cluster system according to the corresponding configuration file.
7. A big data cluster management control device, comprising:
the first acquisition module is used for acquiring the environmental information of each target site in the distributed server cluster;
the first processing module is used for configuring a configuration file of each target site according to the environment information of each target site correspondingly;
and the first execution module is used for correspondingly carrying out big data cluster system management control on each target site according to each configuration file and a preset big data cluster mirror image file.
8. The big data cluster management control device according to claim 7, further comprising:
the second acquisition module is used for acquiring the kernel configuration parameters of the local system;
and the second execution module is used for setting basic parameters of a preset original image according to the kernel configuration parameters to generate the big data cluster image file.
9. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the big data cluster management control method according to any of claims 1 to 6.
10. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the big data cluster management control method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911150335.0A CN111026723B (en) | 2019-11-21 | 2019-11-21 | Big data cluster management control method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911150335.0A CN111026723B (en) | 2019-11-21 | 2019-11-21 | Big data cluster management control method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111026723A true CN111026723A (en) | 2020-04-17 |
| CN111026723B CN111026723B (en) | 2023-08-11 |
Family
ID=70206335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911150335.0A Active CN111026723B (en) | 2019-11-21 | 2019-11-21 | Big data cluster management control method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026723B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112702203A (en) * | 2020-12-22 | 2021-04-23 | 上海智迩智能科技有限公司 | Nginx cluster white screen configuration management method and system |
| CN113031964A (en) * | 2021-03-25 | 2021-06-25 | 恒安嘉新（北京）科技股份公司 | Management method, device, equipment and storage medium for big data application |
| CN113726546A (en) * | 2021-07-12 | 2021-11-30 | 锐捷网络股份有限公司 | Configuration method, device, system, computing equipment and storage medium |
| CN114022331A (en) * | 2021-10-15 | 2022-02-08 | 金茂数字科技有限公司 | Wisdom thing allies oneself with data platform |
| CN115827392A (en) * | 2023-02-09 | 2023-03-21 | 中国证券登记结算有限责任公司 | Monitoring method, device and system for distributed system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040133652A1 (en) * | 2001-01-11 | 2004-07-08 | Z-Force Communications, Inc. | Aggregated opportunistic lock and aggregated implicit lock management for locking aggregated files in a switched file system |
| US7496783B1 (en) * | 2006-02-09 | 2009-02-24 | Symantec Operating Corporation | Merging cluster nodes during a restore |
| US20120079224A1 (en) * | 2010-09-29 | 2012-03-29 | International Business Machines Corporation | Maintaining mirror and storage system copies of volumes at multiple remote sites |
| CN104468199A (en) * | 2014-11-23 | 2015-03-25 | 国云科技股份有限公司 | Frame and running method suitable for deployment and management of Linux distributed system |
| WO2016112825A1 (en) * | 2015-01-13 | 2016-07-21 | 华为技术有限公司 | Remote control method, terminal device, management server, and remote control system |
| WO2016127756A1 (en) * | 2015-02-15 | 2016-08-18 | 北京京东尚科信息技术有限公司 | Flexible deployment method for cluster and management system |
| CN106790483A (en) * | 2016-12-13 | 2017-05-31 | 武汉邮电科学研究院 | Hadoop group systems and fast construction method based on container technique |
-
2019
- 2019-11-21 CN CN201911150335.0A patent/CN111026723B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040133652A1 (en) * | 2001-01-11 | 2004-07-08 | Z-Force Communications, Inc. | Aggregated opportunistic lock and aggregated implicit lock management for locking aggregated files in a switched file system |
| US7496783B1 (en) * | 2006-02-09 | 2009-02-24 | Symantec Operating Corporation | Merging cluster nodes during a restore |
| US20120079224A1 (en) * | 2010-09-29 | 2012-03-29 | International Business Machines Corporation | Maintaining mirror and storage system copies of volumes at multiple remote sites |
| CN104468199A (en) * | 2014-11-23 | 2015-03-25 | 国云科技股份有限公司 | Frame and running method suitable for deployment and management of Linux distributed system |
| WO2016112825A1 (en) * | 2015-01-13 | 2016-07-21 | 华为技术有限公司 | Remote control method, terminal device, management server, and remote control system |
| WO2016127756A1 (en) * | 2015-02-15 | 2016-08-18 | 北京京东尚科信息技术有限公司 | Flexible deployment method for cluster and management system |
| CN106790483A (en) * | 2016-12-13 | 2017-05-31 | 武汉邮电科学研究院 | Hadoop group systems and fast construction method based on container technique |
Non-Patent Citations (3)
| Title |
|---|
| JIUNG-YAO HUANG 等: "Design of the Server Cluster for the Scalable Networked Virtual Environment", 《SEMANTIC SCHOLAR》, pages 1 - 7 * |
| 李永昌: "面向服务器整合的高效物理机到虚拟机转换系统", 《中国优秀硕士学位论文全文数据库》, no. 6, pages 2 - 3 * |
| 王洪发: "一种云环境下的镜像文件存储系统", 《电信科学》, pages 82 - 87 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112702203A (en) * | 2020-12-22 | 2021-04-23 | 上海智迩智能科技有限公司 | Nginx cluster white screen configuration management method and system |
| CN113031964A (en) * | 2021-03-25 | 2021-06-25 | 恒安嘉新（北京）科技股份公司 | Management method, device, equipment and storage medium for big data application |
| CN113031964B (en) * | 2021-03-25 | 2023-12-26 | 恒安嘉新（北京）科技股份公司 | Big data application management method, device, equipment and storage medium |
| CN113726546A (en) * | 2021-07-12 | 2021-11-30 | 锐捷网络股份有限公司 | Configuration method, device, system, computing equipment and storage medium |
| CN113726546B (en) * | 2021-07-12 | 2023-11-21 | 锐捷网络股份有限公司 | Configuration method, device, system, computing equipment and storage medium |
| CN114022331A (en) * | 2021-10-15 | 2022-02-08 | 金茂数字科技有限公司 | Wisdom thing allies oneself with data platform |
| CN115827392A (en) * | 2023-02-09 | 2023-03-21 | 中国证券登记结算有限责任公司 | Monitoring method, device and system for distributed system |
| CN115827392B (en) * | 2023-02-09 | 2023-11-21 | 中国证券登记结算有限责任公司 | Monitoring method, device and system of distributed system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111026723B (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111026723B (en) | Big data cluster management control method and device, computer equipment and storage medium | |
| US11301287B2 (en) | Pattern-based orchestration of cloud provisioning tasks at runtime | |
| CN108809722B (en) | Method, device and storage medium for deploying Kubernetes cluster | |
| CN108536519B (en) | Method for automatically building Kubernetes main node and terminal equipment | |
| CN109274722B (en) | Data sharing method and device and electronic equipment | |
| US6324690B1 (en) | Installation of application software through a network from a source computer system on to a target computer system | |
| CN111782232A (en) | Cluster deployment method and device, terminal equipment and storage medium | |
| US9473356B2 (en) | Automatic configuration of applications based on host metadata using application-specific templates | |
| CN109889377B (en) | Method and device for deploying VNF in NFV system based on Openstack | |
| CN111414391A (en) | Method and system for accessing multiple data sources | |
| CN113419813B (en) | Method and device for deploying bare engine management service based on container platform | |
| CN116208498A (en) | Method, device, equipment and medium for node differential configuration of OpenStack cloud platform | |
| CN110162312A (en) | A kind of BeeGFS configuration method and device based on IML | |
| CN115865718A (en) | Service arrangement control method and device based on Yaml template | |
| CN113254158B (en) | Deployment method and device of deep learning system | |
| US11757976B2 (en) | Unified application management for heterogeneous application delivery | |
| CN110971642B (en) | Data processing method and device for cloud computing platform | |
| CN110489134B (en) | PXE server and high-performance cluster environment deployment method and system | |
| CN112564979A (en) | Execution method and device for construction task, computer equipment and storage medium | |
| CN113268252A (en) | Mirror image manufacturing method and system based on virtual chemical industry control equipment and storage medium | |
| CN110716764A (en) | Sharing method and device of public configuration, computer equipment and storage medium | |
| GB2583904A (en) | Commissioning a virtualised network function | |
| CN116614323B (en) | Cloud storage enterprise network management method and system based on Rclone | |
| CN113824790B (en) | Cloud native PaaS management platform supporting enterprise-level application | |
| WO2023193609A1 (en) | Selective privileged container augmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |