CN116009866A - Web system and method for realizing dynamic optimization of Hadoop platform parameters - Google Patents
Web system and method for realizing dynamic optimization of Hadoop platform parameters Download PDFInfo
- Publication number
- CN116009866A CN116009866A CN202310125789.2A CN202310125789A CN116009866A CN 116009866 A CN116009866 A CN 116009866A CN 202310125789 A CN202310125789 A CN 202310125789A CN 116009866 A CN116009866 A CN 116009866A
- Authority
- CN
- China
- Prior art keywords
- button
- configuration
- configuration file
- service
- hadoop platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000005457 optimization Methods 0.000 title claims description 17
- 230000001960 triggered effect Effects 0.000 claims abstract description 12
- 238000012423 maintenance Methods 0.000 claims description 74
- 238000012986 modification Methods 0.000 claims description 28
- 230000004048 modification Effects 0.000 claims description 28
- 238000013515 script Methods 0.000 claims description 16
- 238000004891 communication Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 description 38
- 230000000007 visual effect Effects 0.000 description 20
- 230000006870 function Effects 0.000 description 9
- 230000000694 effects Effects 0.000 description 6
- 238000012360 testing method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000008571 general function Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Landscapes
- Stored Programmes (AREA)
Abstract
The invention belongs to the technical field of dynamic parameter tuning of an open source Hadoop platform, and particularly provides a web system and a method for realizing dynamic parameter tuning of the Hadoop platform, wherein the system comprises a web interface, is used for maintaining configuration files, service components and server information for deploying the Hadoop platform through a list, and provides an adding button, a modifying button and a deleting button which are communicated with a management background and are used for editing the configuration files, the service components and the server information respectively; after receiving trigger information of the editing buttons, the management background automatically reads XML configuration files according to names and catalog information corresponding to the editing buttons triggered in the list, analyzes the XML configuration files, displays configuration parameters on a WEB interface in a form, and receives editing of the configuration files by a user on the WEB interface. The time of the tuning process is saved, and the efficiency is improved.
Description
Technical Field
The invention relates to the technical field of dynamic parameter tuning of an open-source Hadoop platform, and particularly provides a web system and a web method for realizing dynamic parameter tuning of the Hadoop platform.
Background
Hadoop is a distributed system infrastructure developed by the Apache foundation. The user may develop the distributed program without knowing the details of the distributed underlying layer. And the power of the clusters is fully utilized to perform high-speed operation and storage. Hadoop implements a distributed file system (Distributed File System) in which one component is HDFS (Hadoop Distributed File System). HDFS is characterized by high fault tolerance and is designed to be deployed on inexpensive (low-cost) hardware; and it provides high throughput (high throughput) to access data of applications, suitable for those applications with very large data sets (large data sets). HDFS relaxes the requirements of (relay) POSIX and can access (streaming access) data in the file system in the form of a stream. The most core design of the Hadoop framework is as follows: HDFS and MapReduce. HDFS provides storage for massive amounts of data, while MapReduce provides computation for massive amounts of data.
Hadoop is a reliable, extensible and distributed open source framework for computing and development, and is mainly applied to mass data storage and mass data analysis and computation; the Hadoop TestDFSIO read-write test and the TeraSort ordering test are used as common reference tests to reflect the file processing and analysis computing capacity of a big data platform, and further reflect the hardware performance of a server, and parameter tuning is needed when the performance part is involved.
The existing mature large data platform software such as CDH\HDP and the like has the advantages that the configuration of optimal parameters is realized by dynamically adjusting Hadoop parameters based on a WEB platform so as to realize the optimal performance configuration of Hadoop, the Hadoop software like an open source can not realize the dynamic adjustment of parameters based on a WEB interface at present, configuration files can only be manually modified, then services are restarted, and each time the parameters are adjusted, the servers for deploying the Hadoop platform are connected to carry out file modification and parameter adjustment, so that the visual operation can not be realized, and the operation is difficult for non-professional personnel and the operation steps are complex.
Disclosure of Invention
The Hadoop software of the open source can not realize the dynamic adjustment of the parameters based on the WEB interface at present, only can manually modify the configuration file, then restart the service, and each time the parameters are adjusted, the server for deploying the Hadoop platform is connected for file modification and parameter adjustment, the visual operation can not be realized in the mode, and the operation is difficult and the operation steps are complicated for non-professional personnel.
In a first aspect, the present invention provides a web system for implementing dynamic parameter tuning of a Hadoop platform, including a web interface and a management background;
the web interface is used for maintaining the configuration files, the service components and the server information for deploying the Hadoop platform through the list, and providing an adding button, a modifying button and a deleting button which are communicated with the management background and used for editing the configuration files, the service components and the server information respectively;
and the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
The method comprises the steps of establishing a WEB system-based dynamic optimization of open-source Hadoop parameters, displaying a commonly used configuration file of the open-source Hadoop platform through a visual interface, adjusting the displayed parameter value, automatically updating the adjusted parameter into a target configuration file, and therefore realizing visual editing operation of the configuration parameter, integrating a Hadoop platform service starting and stopping function by a WEB platform, and conveniently adjusting the parameter value without connecting to a server for file modification during Hadoop optimization.
As a further limitation of the technical scheme of the invention, the web interface is provided with a server information maintenance module, a configuration file maintenance module and a service component maintenance module;
the server information maintenance module is used for maintaining server information for deploying the Hadoop platform and providing a server information adding button, a server information deleting button and a server information modifying button which are communicated with the management background;
the configuration file maintenance module is used for maintaining the Hadoop platform configuration file to display the configuration in a form and provide a configuration adding button, a configuration modifying button and a configuration deleting button which are communicated with the management background;
the service component maintenance module is used for maintaining a service component of the Hadoop platform and providing a service adding button, a service modifying button, a service deleting button, a service starting button and a service stopping button which are communicated with the management background. The method solves the complex process of modifying the parameters of the open source Hadoop platform software, realizes the visual maintenance of parameter tuning based on the WEB platform, can simultaneously manage a plurality of Hadoop platform configuration parameters through the set of flow methods, finds the optimal configuration of Hadoop performance through the dynamic adjustment of the parameters, greatly reduces the time for manually modifying files and the complicated configuration process through the interfacial operation of the WEB platform, saves the time of the tuning process, realizes the effect of once and improves the efficiency.
As a further limitation of the technical scheme of the present invention, the server information maintenance module is specifically configured to maintain IP address, user name, and password information, and log in or transmit a file to a designated server through the information using SSH and SCP instructions.
As a further limitation of the technical solution of the present invention, the configuration file maintenance module is specifically configured to maintain a configuration file that is commonly used for tuning and optimizing in a configuration file list, and receive modification to the configuration file by triggering a configuration file modification button.
As a further limitation of the technical solution of the present invention, the profile maintenance module is further provided with a profile replication button that communicates with the management background, and copies the profile to another server by triggering the profile replication button.
As a further limitation of the technical scheme of the invention, the service component maintenance module is used for maintaining a common service component or a service component related to a configuration file list, and the start and stop control of the service component is realized by executing a script under the directory where the service is located.
As a further limitation of the technical scheme of the present invention, the configuration file maintenance module includes a configuration file list, where the configuration file list includes a configuration file name, a directory where the file is located, and a server node where the file is located.
As a further limitation of the technical solution of the present invention, the service component maintenance module includes a service list, where the service list includes a service name and a script directory.
The method solves the complex process of modifying the parameters of the open source Hadoop platform software, realizes the visual maintenance of parameter tuning based on the WEB platform, can simultaneously manage a plurality of Hadoop platform configuration parameters through the set of flow methods, finds the optimal configuration of Hadoop performance through the dynamic adjustment of the parameters, greatly reduces the time for manually modifying files and the complicated configuration process through the interfacial operation of the WEB platform, saves the time of the tuning process, realizes the effect of once and improves the efficiency.
In a second aspect, the present invention further provides a method for implementing dynamic tuning of parameters of a Hadoop platform based on a web system, where the web system includes the web system according to the first aspect, and the method includes the following steps:
maintaining a configuration file which is commonly used for tuning by the Hadoop platform into a configuration file list, and triggering a configuration modification button in a configuration file maintenance module;
and the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
As a further limitation of the technical solution of the present invention, the method further comprises:
after the configuration file is edited, judging whether restarting service is needed;
if yes, the start and stop control of the service component is realized by executing the script under the directory where the service is located;
if not, ending.
From the above technical scheme, the invention has the following advantages: the method solves the complex process of modifying the software parameters of the open source Hadoop platform, realizes the visual maintenance of parameter tuning based on the WEB platform, can simultaneously manage a plurality of Hadoop platform configuration parameters through the set of flow methods, finds the optimal configuration of Hadoop performance through the dynamic adjustment of the parameters, greatly reduces the time for manually modifying files and the complicated configuration process through the interfacial operation of the WEB platform, saves the time of the tuning process and improves the efficiency.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
It can be seen that the present invention has outstanding substantial features and significant advances over the prior art, as well as its practical advantages.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic block diagram of a system of one embodiment of the present invention.
Fig. 2 is a schematic flow chart of a method of one embodiment of the invention.
Detailed Description
The existing mature large data platform software such as CDH\HDP and the like has the advantages that the configuration of optimal parameters is realized by dynamically adjusting Hadoop parameters based on a WEB platform so as to realize the optimal performance configuration of Hadoop, the Hadoop software like an open source can not realize the dynamic adjustment of parameters based on a WEB interface at present, configuration files can only be manually modified, then services are restarted, and each time the parameters are adjusted, the servers for deploying the Hadoop platform are connected to carry out file modification and parameter adjustment, so that the visual operation can not be realized, and the operation is difficult for non-professional personnel and the operation steps are complex. The method comprises the steps of establishing a WEB system-based dynamic optimization of open-source Hadoop parameters, displaying a commonly used configuration file of the open-source Hadoop platform through a visual interface, adjusting the displayed parameter value, automatically updating the adjusted parameter into a target configuration file, and therefore realizing visual editing operation of the configuration parameter, integrating a Hadoop platform service starting and stopping function by a WEB platform, and conveniently adjusting the parameter value without connecting to a server for file modification during Hadoop optimization. In order to make the technical solution of the present invention better understood by those skilled in the art, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
As shown in FIG. 1, the embodiment of the invention provides a web system for realizing the dynamic tuning of parameters of a Hadoop platform, which comprises a web interface and a management background;
the web interface is used for maintaining the configuration files, the service components and the server information for deploying the Hadoop platform through the list, and providing an adding button, a modifying button and a deleting button which are communicated with the management background and used for editing the configuration files, the service components and the server information respectively;
when the web interface realizes maintenance of configuration files and service components of the Hadoop platform and server information for deploying the Hadoop platform, the server information for deploying the Hadoop platform is maintained through a server list, node names, maintenance IP addresses, user names and password information in the list are maintained, and corresponding operation buttons are correspondingly arranged, so that the maintained node names, maintenance IP addresses, user names and password information can be modified or deleted when the corresponding buttons are triggered, and corresponding node information can be added through adding the buttons; the files are logged in or transferred to the designated server using SSH, SCP instructions through the information.
Maintaining the configuration files for deploying the Hadoop platform through a configuration file list, wherein the configuration file list comprises file names, file catalogues, node names and operation buttons; when the configuration file modification button is triggered, the XML configuration file is automatically read according to the file name, the file directory and the node name corresponding to the modification button in the configuration file list, the XML configuration file is analyzed, the configuration parameters are displayed on the WEB interface in the form of the form, and the modification of the configuration file on the WEB interface by a user is received. The configuration file is copied to another server by triggering the configuration file copy button.
The service components for deploying the Hadoop platform are maintained through a service list, wherein the service list comprises service names, script catalogues and corresponding operation buttons, and the service components which are commonly used or are related to the configuration file list are maintained.
And the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
Here, the name refers to the name of the server where the configuration file is located, and the directory information refers to the directory where the configuration file is located.
The method comprises the steps of establishing a WEB system-based dynamic optimization of open-source Hadoop parameters, displaying a commonly used configuration file of the open-source Hadoop platform through a visual interface, adjusting the displayed parameter value, automatically updating the adjusted parameter into a target configuration file, and therefore realizing visual editing operation of the configuration parameter, integrating a Hadoop platform service starting and stopping function by a WEB platform, and conveniently adjusting the parameter value without connecting to a server for file modification during Hadoop optimization. The method solves the complex process of modifying the parameters of the open source Hadoop platform software, realizes the visual maintenance of parameter tuning based on the WEB platform, can simultaneously manage a plurality of Hadoop platform configuration parameters through the set of flow methods, finds the optimal configuration of Hadoop performance through the dynamic adjustment of the parameters, greatly reduces the time for manually modifying files and the complicated configuration process through the interfacial operation of the WEB platform, saves the time of the tuning process, realizes the effect of once and improves the efficiency.
The embodiment of the invention provides a web system for realizing dynamic optimization of Hadoop platform parameters, which comprises a web interface and a management background;
the web interface is used for maintaining the configuration files, the service components and the server information for deploying the Hadoop platform through the list, and providing an adding button, a modifying button and a deleting button which are communicated with the management background and used for editing the configuration files, the service components and the server information respectively;
and the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
The web interface is provided with a server information maintenance module, a configuration file maintenance module and a service component maintenance module;
the server information maintenance module is used for maintaining server information for deploying the Hadoop platform and providing a server information adding button, a server information deleting button and a server information modifying button which are communicated with the management background;
the configuration file maintenance module is used for maintaining the Hadoop platform configuration file to display the configuration in a form and provide a configuration adding button, a configuration modifying button and a configuration deleting button which are communicated with the management background;
the service component maintenance module is used for maintaining a service component of the Hadoop platform and providing a service adding button, a service modifying button, a service deleting button, a service starting button and a service stopping button which are communicated with the management background. The method comprises the steps of establishing a WEB system-based dynamic optimization of open-source Hadoop parameters, displaying a commonly used configuration file of the open-source Hadoop platform through a visual interface, adjusting the displayed parameter value, automatically updating the adjusted parameter into a target configuration file, and therefore realizing visual editing operation of the configuration parameter, integrating a Hadoop platform service starting and stopping function by a WEB platform, and conveniently adjusting the parameter value without connecting to a server for file modification during Hadoop optimization.
The embodiment of the invention provides a web system for realizing dynamic optimization of Hadoop platform parameters, which comprises a web interface and a management background;
the web interface is used for maintaining the configuration file, the service component and the server information for deploying the Hadoop platform of the Hadoop platform, and providing an adding button, a modifying button and a deleting button which are communicated with the management background and used for editing the configuration file, the service component and the server information respectively;
and the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
The web interface is provided with a server information maintenance module, a configuration file maintenance module and a service component maintenance module;
the server information maintenance module is used for maintaining server information for deploying the Hadoop platform and providing a server information adding button, a server information deleting button and a server information modifying button which are communicated with the management background;
the configuration file maintenance module is used for maintaining the Hadoop platform configuration file to display the configuration in a form and provide a configuration adding button, a configuration modifying button and a configuration deleting button which are communicated with the management background;
the service component maintenance module is used for maintaining a service component of the Hadoop platform and providing a service adding button, a service modifying button, a service deleting button, a service starting button and a service stopping button which are communicated with the management background.
The server information maintenance module is specifically used for maintaining IP address, user name and password information, and using SSH and SCP instructions to log in or transmit files to the appointed server through the information. The configuration file maintenance module is specifically configured to maintain a configuration file which is commonly used for tuning and optimizing by Hadoop into a configuration file list, and receive modification to the configuration file by triggering a configuration file modification button. The configuration file maintenance module is also provided with a configuration file copying button communicated with the management background, and the configuration file is copied to another server by triggering the configuration file copying button. The service component maintenance module is used for maintaining a common service component or a service component related to the configuration file list, and starting and stopping control of the service component is realized by executing a script under a directory where the service is located. The configuration file maintenance module comprises a configuration file list, wherein the configuration file list comprises a configuration file name, a directory where the file is located and a server node where the file is located. The service component maintenance module includes a service list including service names and script directories.
The method solves the complex process of modifying the parameters of the open source Hadoop platform software, realizes the visual maintenance of parameter tuning based on the WEB platform, can simultaneously manage a plurality of Hadoop platform configuration parameters through the set of flow methods, finds the optimal configuration of Hadoop performance through the dynamic adjustment of the parameters, greatly reduces the time for manually modifying files and the complicated configuration process through the interfacial operation of the WEB platform, saves the time of the tuning process, realizes the effect of once and improves the efficiency.
The established WEB interface has the general functions as above and mainly relates to three functional modules, namely server information maintenance, configuration file maintenance and service component maintenance;
the server configuration information mainly needs to maintain information such as an IP address, a user name, a password and the like, and can log in or transmit files to a designated server by using commands such as an SSH, an SCP and the like through the information, such as operations of reading configuration file contents, updating configuration files, copying configuration files and the like.
The server list is used for maintaining server information of Hadoop software, wherein the IP address, the user name of an operating system and the password lamp of the operating system are mainly used for modifying, distributing, starting scripts and other operations of configuration files.
The configuration file list is firstly required to maintain the configuration files (such as yam-site.xml, hdfs-site.xml and other XML files, the names of the configuration files, the catalogue of the files, the server node of the files and the like) which are commonly used for tuning and optimizing into the configuration file list, and when the configuration files are clicked and edited, the background can automatically read the XML configuration files according to the information of the server and the catalogue of the files and analyze the read XML configuration files, wherein the main configuration parameters in the XML files are in the following formats:
<property>
<name>dfs.namenode.name.dir</name>
<value>/data1/hadoop/data/dfs/nn</value>
</property>
by writing functional codes, the configuration parameters are displayed on a WEB interface in a form, and a user can modify the configuration files on the WEB interface to realize visual modification of the parameters; if deployed in a cluster, the configuration file may exist on multiple servers, and the configuration file may be copied to another server through a copy button of the web interface.
The service list is mainly used for maintaining commonly used service components or service components related to the configuration file list, and the start and stop control of the service components is realized through the information of a server where the service components are located, a directory where the script is located and the like.
The function of the service component maintenance module is mainly used for maintaining the service components of the Hadoop platform, after the configuration file is modified, some service components can be validated after restarting, the function can realize the service starting of a single component, the starting of all the service components can also be realized, and the starting and stopping control of the service is realized by executing a starting or stopping script under a catalog. A single start-up refers to a certain service under a certain server being started up at a time, and starting all components refers to starting all services under all servers.
Because the deployment methods of the common version of the open source Hadoop platform are basically consistent, and the configuration files and the service components are also approximately the same, a plurality of Hadoop platform configuration parameters can be managed simultaneously through the set of flow methods, the optimal configuration of the Hadoop performance is found through dynamic adjustment of the parameters, the time for manually modifying the files and the complicated configuration process are greatly reduced through the interfacial operation of the WEB platform, the time of the tuning process is saved, the effect of once and forever is realized, and the efficiency is improved.
The technical scheme of the invention also provides a method for realizing the dynamic optimization of the Hadoop platform parameters based on a web system, wherein the web system comprises a web interface and a management background; the web interface is used for maintaining configuration files, service components and server information for deploying the Hadoop platform of the Hadoop platform and providing an adding button, a modifying button and a deleting button for communication with the management background; and the management background is used for automatically reading the XML configuration file according to the server and directory information of the configuration file after receiving the trigger information of the modification button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the modification of the configuration file on the WEB interface by a user.
The web interface is provided with a server information maintenance module, a configuration file maintenance module and a service component maintenance module; the server information maintenance module is used for maintaining server information for deploying the Hadoop platform and providing a server information adding button, a server information deleting button and a server information modifying button which are communicated with the management background; the configuration file maintenance module is used for maintaining the Hadoop platform configuration file to display the configuration in a form and provide a configuration adding button, a configuration modifying button and a configuration deleting button which are communicated with the management background; the service component maintenance module is used for maintaining a service component of the Hadoop platform and providing a service adding button, a service modifying button, a service deleting button, a service starting button and a service stopping button which are communicated with the management background; as shown in fig. 2, the method comprises the steps of:
step 1: maintaining a configuration file which is commonly used for tuning by the Hadoop platform into a configuration file list, and triggering a configuration modification button in a configuration file maintenance module;
step 2: and the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
The method comprises the steps of establishing a WEB system-based dynamic optimization of open-source Hadoop parameters, displaying a commonly used configuration file of the open-source Hadoop platform through a visual interface, adjusting the displayed parameter value, automatically updating the adjusted parameter into a target configuration file, and therefore realizing visual editing operation of the configuration parameter, integrating a Hadoop platform service starting and stopping function by a WEB platform, and conveniently adjusting the parameter value without connecting to a server for file modification during Hadoop optimization.
The technical scheme of the invention also provides a method for realizing the dynamic optimization of the Hadoop platform parameters based on a web system, wherein the web system comprises a web interface and a management background; the web interface is used for maintaining configuration files, service components and server information for deploying the Hadoop platform of the Hadoop platform and providing an adding button, a modifying button and a deleting button for communication with the management background; and the management background is used for automatically reading the XML configuration file according to the server and directory information of the configuration file after receiving the trigger information of the modification button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the modification of the configuration file on the WEB interface by a user.
The web interface is provided with a server information maintenance module, a configuration file maintenance module and a service component maintenance module; the server information maintenance module is used for maintaining server information for deploying the Hadoop platform and providing a server information adding button, a server information deleting button and a server information modifying button which are communicated with the management background; the configuration file maintenance module is used for maintaining the Hadoop platform configuration file to display the configuration in a form and provide a configuration adding button, a configuration modifying button and a configuration deleting button which are communicated with the management background; the service component maintenance module is used for maintaining a service component of the Hadoop platform and providing a service adding button, a service modifying button, a service deleting button, a service starting button and a service stopping button which are communicated with the management background; the method comprises the following steps:
step 1: maintaining a configuration file which is commonly used for tuning by the Hadoop platform into a configuration file list, and triggering a configuration modification button in a configuration file maintenance module;
step 2: the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the editing button triggered in the list after receiving the triggering information of the editing button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the editing of the configuration file by a user on the WEB interface;
step 3: after the configuration file is edited, judging whether restarting service is needed;
if yes, the start and stop control of the service component is realized by executing the script under the directory where the service is located;
if not, ending.
It should be noted here that, if the cluster is deployed, the configuration file is copied to another server through the copy button of the web interface.
The established WEB interface has the general functions as above and mainly relates to three functional modules, namely server information maintenance, configuration file maintenance and service component maintenance;
the server configuration information mainly needs to maintain information such as an IP address, a user name, a password and the like, and can log in or transmit files to a designated server by using commands such as an SSH, an SCP and the like through the information, such as operations of reading configuration file contents, updating configuration files, copying configuration files and the like.
The server list is used for maintaining server information of Hadoop software, wherein the IP address, the user name of an operating system and the password lamp of the operating system are mainly used for modifying, distributing, starting scripts and other operations of configuration files.
The configuration file list is firstly required to maintain the configuration files (such as yam-site.xml, hdfs-site.xml and other XML files, the names of the configuration files, the catalogue of the files, the server node of the files and the like) which are commonly used for tuning and optimizing into the configuration file list, and when the configuration files are clicked and edited, the background can automatically read the XML configuration files according to the information of the server and the catalogue of the files and analyze the read XML configuration files, wherein the main configuration parameters in the XML files are in the following formats:
<property>
<name>dfs.namenode.name.dir</name>
<value>/data1/hadoop/data/dfs/nn</value>
</property>
by writing functional codes, the configuration parameters are displayed on a WEB interface in a form, and a user can modify the configuration files on the WEB interface to realize visual modification of the parameters; if deployed in a cluster, the configuration file may exist on multiple servers, and the configuration file may be copied to another server through a copy button of the web interface.
The service list is mainly used for maintaining commonly used service components or service components related to the configuration file list, and the start and stop control of the service components is realized through the information of a server where the service components are located, a directory where the script is located and the like.
The function of the service component maintenance module is mainly used for maintaining the service components of the Hadoop platform, after the configuration file is modified, some service components can be validated after restarting, the function can realize the service starting of a single component, the starting of all the service components can also be realized, and the starting and stopping control of the service is realized by executing a starting or stopping script under a catalog. A single start-up refers to a certain service under a certain server being started up at a time, and starting all components refers to starting all services under all servers.
Because the deployment methods of the common version of the open source Hadoop platform are basically consistent, and the configuration files and the service components are also approximately the same, a plurality of Hadoop platform configuration parameters can be managed simultaneously through the set of flow methods, the optimal configuration of the Hadoop performance is found through dynamic adjustment of the parameters, the time for manually modifying the files and the complicated configuration process are greatly reduced through the interfacial operation of the WEB platform, the time of the tuning process is saved, the effect of once and forever is realized, and the efficiency is improved.
Although the present invention has been described in detail by way of preferred embodiments with reference to the accompanying drawings, the present invention is not limited thereto. Various equivalent modifications and substitutions may be made in the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and it is intended that all such modifications and substitutions be within the scope of the present invention/be within the scope of the present invention as defined by the appended claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. The web system for realizing the dynamic optimization of the Hadoop platform parameters is characterized by comprising a web interface and a management background;
the web interface is used for maintaining the configuration files, the service components and the server information for deploying the Hadoop platform through the list, and providing an adding button, a modifying button and a deleting button which are communicated with the management background and used for editing the configuration files, the service components and the server information respectively;
and the management background is used for automatically reading the XML configuration file according to the name and directory information corresponding to the edit button triggered in the list after receiving the trigger information of the edit button, analyzing the XML configuration file, displaying the configuration parameters on the WEB interface in a form, and receiving the edit of the configuration file by the user on the WEB interface.
2. The web system for realizing the dynamic tuning of the Hadoop platform parameters according to claim 1, wherein the web interface is provided with a server information maintenance module, a configuration file maintenance module and a service component maintenance module;
the server information maintenance module is used for maintaining server information for deploying the Hadoop platform and providing a server information adding button, a server information deleting button and a server information modifying button which are communicated with the management background;
the configuration file maintenance module is used for maintaining the Hadoop platform configuration file to display the configuration in a form and provide a configuration adding button, a configuration modifying button and a configuration deleting button which are communicated with the management background;
the service component maintenance module is used for maintaining a service component of the Hadoop platform and providing a service adding button, a service modifying button, a service deleting button, a service starting button and a service stopping button which are communicated with the management background.
3. The web system for implementing the dynamic tuning of Hadoop platform parameters according to claim 2, wherein the server information maintenance module is specifically configured to maintain IP address, user name, and password information, and log in or transmit a file to a designated server through the information using SSH and SCP instructions.
4. The web system for implementing dynamic tuning of Hadoop platform parameters according to claim 3, wherein the configuration file maintenance module is specifically configured to maintain a configuration file of Hadoop commonly used for tuning into a configuration file list, and receive a modification to the configuration file by triggering a configuration file modification button.
5. The web system for implementing dynamic tuning of Hadoop platform parameters according to claim 4, wherein the profile maintenance module is further provided with a profile copy button in communication with the management background, and the profile is copied to another server by triggering the profile copy button.
6. The web system for realizing dynamic tuning of Hadoop platform parameters according to claim 5, wherein the service component maintenance module is configured to maintain a commonly used service component or a service component related to a configuration file list, and realize start-stop control of the service component by executing a script under a directory where the service is located.
7. The web system for implementing dynamic tuning of Hadoop platform parameters according to claim 6, wherein the profile maintenance module comprises a profile list comprising a profile name, a directory in which the files reside, and a server node in which the files reside.
8. The web system for implementing Hadoop platform parameter dynamic tuning of claim 7, wherein the service component maintenance module comprises a service list comprising service names and script directories.
9. A method for realizing dynamic tuning of Hadoop platform parameters based on a web system, wherein the web system comprises the web system according to any one of claims 1-8, the method comprising the steps of:
maintaining a configuration file which is commonly used for tuning by the Hadoop platform into a configuration file list, and triggering an editing button in a configuration file maintenance module;
after receiving trigger information of the editing buttons, the management background automatically reads XML configuration files according to names and catalog information corresponding to the editing buttons triggered in the list, analyzes the XML configuration files, displays configuration parameters on a WEB interface in a form, and receives editing of the configuration files by a user on the WEB interface.
10. The method for realizing dynamic tuning of parameters of a Hadoop platform based on a web system according to claim 9, wherein the method further comprises:
after the configuration file is edited, judging whether restarting service is needed;
if yes, the start and stop control of the service component is realized by executing the script under the directory where the service is located; if not, ending.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310125789.2A CN116009866A (en) | 2023-02-16 | 2023-02-16 | Web system and method for realizing dynamic optimization of Hadoop platform parameters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310125789.2A CN116009866A (en) | 2023-02-16 | 2023-02-16 | Web system and method for realizing dynamic optimization of Hadoop platform parameters |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116009866A true CN116009866A (en) | 2023-04-25 |
Family
ID=86021253
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310125789.2A Pending CN116009866A (en) | 2023-02-16 | 2023-02-16 | Web system and method for realizing dynamic optimization of Hadoop platform parameters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116009866A (en) |
-
2023
- 2023-02-16 CN CN202310125789.2A patent/CN116009866A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107797767B (en) | One kind is based on container technique deployment distributed memory system and its storage method | |
| US8370828B2 (en) | Techniques to perform gradual upgrades | |
| US20150237113A1 (en) | Method and system for file transmission | |
| CN111614733B (en) | Deployment method, device and storage medium of distributed multi-fragmentation cluster | |
| JP2010518490A (en) | Synchronization framework for irregularly connected applications | |
| CN103297275A (en) | Dynamic discovery and loading system and method for Web network management client module | |
| WO2018204865A1 (en) | Data integration for distributed and massively parallel processing environments | |
| CN112711397A (en) | Method and system for establishing system based on micro-service and top-level design | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| CN116009866A (en) | Web system and method for realizing dynamic optimization of Hadoop platform parameters | |
| CN114443294B (en) | Big data service component deployment method, system, terminal and storage medium | |
| CN114500289B (en) | Control plane recovery method, device, control node and storage medium | |
| CN116028463A (en) | Method for constructing large data platform with separated storage and calculation | |
| CN115994124A (en) | File scanning method, device, electronic equipment and computer readable storage medium | |
| CN112685486B (en) | Data management method and device for database cluster, electronic equipment and storage medium | |
| Cisco | Cisco Info Server | |
| CN111026413B (en) | Requirement-oriented complex system 'pull' mode deployment system and method thereof | |
| CN113438095B (en) | Method, device and equipment for managing configuration data and storage medium | |
| CN113760836B (en) | Wide table calculation method and device | |
| CN117170692A (en) | Database deployment method, device, equipment and storage medium | |
| Peterson | Modernization of Center-to-Center Data Communication Standards: Sample Implementation Administration and User Guide | |
| CN114816667A (en) | Mirror image file processing method and device, storage medium and electronic equipment | |
| CN114064658A (en) | Method and device for updating dimension table in cluster | |
| CN118092982A (en) | Multi-cluster operation and maintenance method, equipment and medium for cloud native application | |
| Du et al. | Application Research on Cloud Platform Forensics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |