CN112398669A - Hadoop deployment method and device - Google Patents
Hadoop deployment method and device Download PDFInfo
- Publication number
- CN112398669A CN112398669A CN201910753380.9A CN201910753380A CN112398669A CN 112398669 A CN112398669 A CN 112398669A CN 201910753380 A CN201910753380 A CN 201910753380A CN 112398669 A CN112398669 A CN 112398669A
- Authority
- CN
- China
- Prior art keywords
- deployment
- hadoop
- commands
- maintenance platform
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0266—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using meta-data, objects or commands for formatting management information, e.g. using eXtensible markup language [XML]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0253—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using browsers or web-pages for accessing management information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a Hadoop deployment method and device, and relates to the technical field of computers. One embodiment of the method comprises: receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, Hadoop clusters for executing the deployment commands or one or more Hadoop nodes; merging or splitting the one or more deployment commands; and sending the merged or split one or more deployment commands to a server of the automatic operation and maintenance platform, so that the Hadoop node managed by the server of the automatic operation and maintenance platform executes the deployment commands. According to the implementation mode, the memory consumed by the automatic operation and maintenance platform when the deployment command is processed is reduced, the throughput of the automatic operation and maintenance platform is improved, and the Hadoop deployment efficiency is further improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a Hadoop deployment method and device.
Background
Hadoop is a distributed system infrastructure developed by the Apache Foundation, and has two core designs: HDFS and MapReduce; the hdfs (Hadoop Distributed File system), namely, a Distributed File system, provides storage for mass data, and the MapReduce provides possibility for calculation of mass data, so that the Hadoop not only can provide access to application data with high throughput, but also is applicable to application programs with large data sets. Therefore, it is often necessary to deploy Hadoop clusters in bulk or to execute commands in bulk. At present, a commonly used Hadoop deployment method is to implement Hadoop deployment by means of an automated operation and maintenance tool or an automated operation and maintenance platform.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: with the great increase of the number of deployed Hadoop nodes, especially when tens of thousands of Hadoop nodes need to be deployed at the same time, the processing of a large number of deployment commands and Hadoop node execution results causes great pressure on a processor, the throughput of a system is limited, and the Hadoop deployment efficiency is reduced.
Disclosure of Invention
In view of this, the invention provides a Hadoop deployment method and device, which can realize distribution control of deployment commands by splitting or combining the deployment commands, greatly reduce the great pressure on a processor caused by the deployment commands and the execution results of Hadoop nodes, and improve the Hadoop deployment efficiency.
To achieve the above object, according to a first aspect of the present invention, there is provided a Hadoop deployment method, including: receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, Hadoop clusters for executing the deployment commands or one or more Hadoop nodes; merging or splitting the one or more deployment commands; and sending the merged or split one or more deployment commands to a server of the automatic operation and maintenance platform, so that the Hadoop node managed by the server of the automatic operation and maintenance platform executes the deployment commands.
Optionally, the one or more deployment commands are merged or split according to one or more of: the number of deployment commands, the type of Hadoop node executing the deployment commands, and the number of Hadoop nodes executing the deployment commands.
Optionally, the server side of the automation operation and maintenance platform sends the merged or split one or more deployment commands to the Hadoop node through a Redis queue.
Optionally, the method further comprises: receiving one or more execution results returned by a result returner of the automated operation and maintenance platform; and combining the one or more execution results, and returning the combined execution results to the deployment foreground.
Optionally, the automated operation and maintenance platform is Saltstack or Ansible.
To achieve the above object, according to a second aspect of the present invention, there is provided a Hadoop deployment device comprising: the system comprises a deployment request receiving module, a command management module and a command sending module; the deployment request receiving module is configured to receive a deployment request sent by a deployment foreground, where the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster executing the deployment commands, or one or more Hadoop nodes; the command management module is used for merging or splitting the one or more deployment commands; the command sending module is used for sending the combined or split one or more deployment commands to a server of the automated operation and maintenance platform, so that the Hadoop node managed by the server of the automated operation and maintenance platform executes the deployment commands.
Optionally, the command management module is configured to merge or split the one or more deployment commands according to one or more of the following: the number of deployment commands, the type of Hadoop node executing the deployment commands, and the number of Hadoop nodes executing the deployment commands.
Optionally, the server side of the automation operation and maintenance platform sends the merged or split one or more deployment commands to the Hadoop node through a Redis queue.
Optionally, the method further comprises: an execution result processing module; the execution result processing module is used for receiving one or more execution results returned by a result returner of the automatic operation and maintenance platform; and combining the one or more execution results, and returning the combined execution results to the deployment foreground.
Optionally, the automated operation and maintenance platform is Saltstack or Ansible.
To achieve the above object, according to a third aspect of the present invention, there is provided a server for Hadoop deployment, comprising: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method as any one of the Hadoop deployment methods described above.
To achieve the above object, according to a fourth aspect of the present invention, a computer readable medium is stored with a computer program, characterized in that the program, when executed by a processor, implements any of the Hadoop deployment methods described above.
Due to the adoption of the technical scheme, the invention has the following advantages or beneficial effects: by merging or splitting the deployment command in the deployment request, the deployment command is controlled to be distributed, the pressure of the excessive deployment command on the automatic operation and maintenance platform is avoided, and the efficiency of the automatic operation and maintenance platform in deploying the command to the Hadoop node is improved; meanwhile, the pressure of the automatic operation and maintenance platform server side in sending commands to the managed Hadoop nodes is further relieved through the Redis queue; in addition, the method of directly returning the execution result through the result returning device of the automatic operation and maintenance platform greatly reduces the pressure of the query or the acquisition of the execution result on the server where the automatic operation and maintenance platform is located, and simultaneously improves the efficiency of acquiring the execution result, so that the deployment foreground can monitor the deployment progress or the progress of the Hadoop node executing the deployment command in real time.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a Hadoop deployment method according to an embodiment of the invention;
FIG. 2 is a schematic diagram of the major modules of a Hadoop deployment device, according to an embodiment of the present invention;
FIG. 3 is a schematic main flow diagram of a Hadoop deployment apparatus application method according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As shown in fig. 1, an embodiment of the present invention provides a Hadoop deployment method, which may specifically include the following steps:
step S101, receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, Hadoop clusters executing the deployment commands, or one or more Hadoop nodes.
The deployment foreground is a WEB module of the whole deployment system and provides all visual operations including task progress control, execution result query display, software management and the like. The deployment foreground can store the execution result into the local database after receiving the execution result returned according to the deployment command, and further can query the execution result in real time or according to actual requirements and visually display the execution result. The deploy command may include any command that is executed on the Hadoop node, such as deleting a directory, creating a directory, starting a Hadoop service, etc.
Furthermore, in order to realize asynchronous processing of sending a deployment request and receiving an execution result by a deployment foreground and increase the throughput of the system during Hadoop deployment, the sending of the deployment request and the receiving of the execution result can be performed through message queues such as ActiveMQ, RabbitMQ, Kafka and the like, so that the efficiency of sending a deployment command and receiving the execution result by the deployment foreground is improved.
Step S102, merging or splitting the one or more deployment commands.
With the increase of the large data volume, the number of the corresponding Hadoop nodes which need to be deployed simultaneously or in batches is also increased sharply, and based on the increase, the deployment command is considered to be split or merged, so that the shunting control of the deployment command is realized. For example, when the same software package is deployed to ten thousand Hadoop nodes or the update of the same software package is executed, ten thousand deployment commands need to be simultaneously sent to the managed Hadoop nodes through the automated operation and maintenance platform, so that the number of processes which need to be simultaneously run by the automated operation and maintenance platform is greatly increased, the processor pressure is too high, and the efficiency of Hadoop deployment is limited to a great extent.
It can be understood that, in the actual execution process, the splitting of the deployment command can be realized according to the actual requirements, such as the number of the deployment commands, the type of the Hadoop nodes executing the deployment command, and the number of the Hadoop nodes executing the deployment command. Specifically, when the number of the deployment commands is large, the deployment commands are processed in batches; when the number of the deployment commands is small, the deployment commands can be combined into a new deployment command, so that the number of Hadoop nodes executing the deployment commands is reduced, and load balancing is realized. Meanwhile, when the deployment commands are merged or split, different deployment commands can be allocated to different types of Hadoop nodes according to different types of deployed Hadoop nodes or different services, such as DataNodes and NameNodes.
Step S103, sending the merged or split one or more deployment commands to a server of the automated operation and maintenance platform, so that the Hadoop node managed by the server of the automated operation and maintenance platform executes the deployment commands.
In the actual execution process, a server where each Hadoop node is located or a client of the automated operation and maintenance platform is arranged at the position of the server, so that the server of the automated operation and maintenance platform can send a deployment command to the client of the automated operation and maintenance platform where the Hadoop node is located; and after receiving the deployment command, the client of the automatic operation and maintenance platform calls the managed Hadoop node to execute the deployment command.
In an alternative embodiment, the automated operation and maintenance platform is a Saltstack or an Anchor. The SaltStack is a centralized management platform of a server infrastructure, has functions of configuration management, remote execution, monitoring and the like, is realized based on Python language, and is constructed by combining a lightweight message queue (zeroMQ) and Python third-party modules (Pyzmq, PyCrypto, Pyjinjia2, Python-msgpack, PyYAML and the like); through the SaltStack, batch execution commands can be performed on tens of millions of servers, and centralized management of configuration, file distribution, server data collection, operating system foundation and software package management can be performed according to different services. The infrastructure is a Python-based development automation operation and maintenance platform, and realizes functions of batch system configuration, batch program deployment, batch operation commands and the like.
In an optional implementation manner, the server of the automation operation and maintenance platform sends the merged or split one or more deployment commands to the Hadoop node through a Redis queue. Therefore, the pressure of the server side of the automatic operation and maintenance platform when sending the deployment command to the client side of the automatic operation and maintenance platform is further reduced through the cache of the Redis queue, the efficiency of the automatic operation and maintenance platform for processing the deployment command is improved, and the efficiency of Hadoop deployment is further improved.
In an optional implementation manner, one or more execution results returned by a result returner of the automatic operation and maintenance platform are received; and combining the one or more execution results, and returning the combined execution results to the deployment foreground. Specifically, after the Hadoop node executes the deployment command, an execution result is directly returned to the Hadoop deployment device by triggering a result Returner (such as a Salt Return of Saltstack) of the automated operation and maintenance platform; after receiving the execution result returned by the Salt Return, the Hadoop deployment device locally stores the execution result, merges the execution result, and sends the merged execution result to the deployment foreground through the message queue between the Hadoop deployment device and the deployment foreground. That is to say, the execution result is prevented from being returned to the automated operation and maintenance platform server through the automated operation and maintenance platform client, or the automated operation and maintenance platform server actively inquires the automated operation and maintenance platform client about a large amount of memory consumed by the execution result, so that the efficiency of the deployment command through the automated operation and maintenance platform server is improved, meanwhile, the efficiency of the deployment foreground for receiving the execution result is improved through combining the execution result, and the possibility of the deployment foreground for monitoring the deployment progress in real time is provided.
Based on the embodiment, the deployment command is merged or split in the deployment request, so that the deployment command is controlled to be distributed, the pressure of the excessive deployment command on the automatic operation and maintenance platform is avoided, and the efficiency of the automatic operation and maintenance platform in deploying the command to the Hadoop node is improved; meanwhile, the pressure of the automatic operation and maintenance platform server side in sending commands to the managed Hadoop nodes is further relieved through the Redis queue; in addition, the method of directly returning the execution result through the result returning device of the automatic operation and maintenance platform greatly reduces the pressure of the query or the acquisition of the execution result on the server where the automatic operation and maintenance platform is located, and simultaneously improves the efficiency of acquiring the execution result, so that the deployment foreground can monitor the deployment progress or the progress of the Hadoop node executing the deployment command in real time.
Referring to fig. 2, an embodiment of the present invention provides a Hadoop deployment device 200, including: a deployment request receiving module 201, a command management module 202 and a command sending module 203; wherein the content of the first and second substances,
the deployment request receiving module 201 is configured to receive a deployment request sent by a deployment foreground, where the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster executing the deployment commands, or one or more Hadoop nodes;
the command management module 202 is configured to merge or split the one or more deployment commands;
the command sending module 203 is configured to send the merged or split one or more deployment commands to a server of the automation operation and maintenance platform, so that the Hadoop node managed by the server of the automation operation and maintenance platform executes the deployment commands.
In an alternative embodiment, the command management module 202 is configured to merge or split the one or more deployment commands according to one or more of the following: the number of deployment commands, the type of Hadoop node executing the deployment commands, and the number of Hadoop nodes executing the deployment commands.
In an optional implementation manner, the server of the automation operation and maintenance platform sends the merged or split one or more deployment commands to the Hadoop node through a Redis queue.
In an optional embodiment, the method further comprises: an execution result processing module 204; the execution result processing module 204 is configured to receive one or more execution results returned by a result returner of the automation operation and maintenance platform; and combining the one or more execution results, and returning the combined execution results to the deployment foreground.
In an alternative embodiment, the automated operation and maintenance platform is a Saltstack or an Anchor.
Referring to fig. 3, on the basis of the foregoing embodiment, an embodiment of the present invention provides an application method of a Hadoop deployment device, which is specifically as follows:
the Hadoop deployment device 200 receives, through a message queue between the deployment foreground and the deployment foreground, a deployment request sent by the deployment foreground, where the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster executing the deployment commands, or one or more Hadoop nodes.
The Hadoop deployment device 200 merges or splits one or more deployment commands.
The Hadoop deployment device 200 sends the merged or split deployment command or commands to the server of the automation operation and maintenance platform, so that the Hadoop node managed by the server of the automation operation and maintenance platform executes the deployment command.
Specifically, the Hadoop deployment device 200 sends the merged or split deployment command to a service end (Salt Master) of the automated operation and maintenance platform; the Salt Master sends a deployment command to a client (Salt Minion) of an automatic operation and maintenance platform deployed on a Hadoop node executing the deployment command through a message queue; calling the managed Hadoop node by the Salt Minion to enable the Hadoop node to execute a deployment command; after the Hadoop node executes the deployment command, triggering a result Returner (Salt Returner) of the automated operation and maintenance platform, so that the Salt Returner directly returns the execution result to the Hadoop deployment device 200; after receiving the execution result returned by the Salt return, the Hadoop deployment device 200 locally stores the execution result, merges the execution result, and sends the merged execution result to the deployment foreground through the message queue between the Hadoop deployment device and the deployment foreground.
FIG. 4 illustrates an exemplary system architecture 400 to which the Hadoop deployment method or Hadoop deployment apparatus of embodiments of the invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have various communication client applications installed thereon, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 401, 402, and 403. The background management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (e.g., a combined execution result) to the terminal device.
It should be noted that the Hadoop deployment method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the Hadoop deployment apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: the names of these modules do not in some cases constitute a limitation on the module itself, for example, the command management module may also be described as a "module that merges or splits the one or more deployment commands".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, Hadoop clusters for executing the deployment commands or one or more Hadoop nodes; merging or splitting the one or more deployment commands; and sending the merged or split one or more deployment commands to a server of the automatic operation and maintenance platform, so that the Hadoop node managed by the server of the automatic operation and maintenance platform executes the deployment commands.
According to the technical scheme of the embodiment of the invention, the control on the distribution of the deployment command is realized by merging or splitting the deployment command in the deployment request, the pressure of the excessive deployment command on the automatic operation and maintenance platform is avoided, and the efficiency of the automatic operation and maintenance platform in deploying the command to the Hadoop node is improved; meanwhile, the pressure of the automatic operation and maintenance platform server side in sending commands to the managed Hadoop nodes is further relieved through the Redis queue; in addition, the method of directly returning the execution result through the result returning device of the automatic operation and maintenance platform greatly reduces the pressure of the query or the acquisition of the execution result on the server where the automatic operation and maintenance platform is located, and simultaneously improves the efficiency of acquiring the execution result, so that the deployment foreground can monitor the deployment progress or the progress of the Hadoop node executing the deployment command in real time.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (12)
1. A Hadoop deployment method is characterized by comprising the following steps:
receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, Hadoop clusters for executing the deployment commands or one or more Hadoop nodes;
merging or splitting the one or more deployment commands;
and sending the merged or split one or more deployment commands to a server of the automatic operation and maintenance platform, so that the Hadoop node managed by the server of the automatic operation and maintenance platform executes the deployment commands.
2. The Hadoop deployment method of claim 1,
merging or splitting the one or more deployment commands according to one or more of: the number of deployment commands, the type of Hadoop node executing the deployment commands, and the number of Hadoop nodes executing the deployment commands.
3. The Hadoop deployment method according to claim 1, wherein the server of the automation operation and maintenance platform sends the merged or split one or more deployment commands to the Hadoop node through a Redis queue.
4. The Hadoop deployment method of claim 1, further comprising:
receiving one or more execution results returned by a result returner of the automated operation and maintenance platform;
and combining the one or more execution results, and returning the combined execution results to the deployment foreground.
5. The Hadoop deployment method of claim 1 wherein the automated operation and maintenance platform is Saltstack or Ansible.
6. A Hadoop deployment device, comprising: the system comprises a deployment request receiving module, a command management module and a command sending module; wherein the content of the first and second substances,
the deployment request receiving module is used for receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, Hadoop clusters for executing the deployment commands or one or more Hadoop nodes;
the command management module is used for merging or splitting the one or more deployment commands;
the command sending module is used for sending the combined or split one or more deployment commands to a server of the automated operation and maintenance platform, so that the Hadoop node managed by the server of the automated operation and maintenance platform executes the deployment commands.
7. The Hadoop deployment device of claim 6, wherein the command management module is configured to merge or split the one or more deployment commands according to one or more of: the number of deployment commands, the type of Hadoop node executing the deployment commands, and the number of Hadoop nodes executing the deployment commands.
8. The Hadoop deployment device according to claim 6, wherein the server of the automated operation and maintenance platform sends the merged or split one or more deployment commands to the Hadoop node through a Redis queue.
9. The Hadoop deployment device of claim 6 further comprising: an execution result processing module; the execution result processing module is used for
Receiving one or more execution results returned by a result returner of the automated operation and maintenance platform;
and combining the one or more execution results, and returning the combined execution results to the deployment foreground.
10. The Hadoop deployment device of claim 6 wherein the automated operation and maintenance platform is Saltstack or Ansible.
11. A server for Hadoop deployment, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.
12. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910753380.9A CN112398669B (en) | 2019-08-15 | 2019-08-15 | Hadoop deployment method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910753380.9A CN112398669B (en) | 2019-08-15 | 2019-08-15 | Hadoop deployment method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112398669A true CN112398669A (en) | 2021-02-23 |
| CN112398669B CN112398669B (en) | 2023-09-26 |
Family
ID=74601565
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910753380.9A Active CN112398669B (en) | 2019-08-15 | 2019-08-15 | Hadoop deployment method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112398669B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127150A (en) * | 2021-03-18 | 2021-07-16 | 同盾控股有限公司 | Rapid deployment method and device of cloud native system, electronic equipment and storage medium |
| CN113377385A (en) * | 2021-06-07 | 2021-09-10 | 中国工商银行股份有限公司 | Client automatic deployment method and device |
| CN113656147A (en) * | 2021-08-20 | 2021-11-16 | 北京百度网讯科技有限公司 | Cluster deployment method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064742A (en) * | 2012-12-25 | 2013-04-24 | 中国科学院深圳先进技术研究院 | Automatic deployment system and method of hadoop cluster |
| CN105893545A (en) * | 2016-04-01 | 2016-08-24 | 浪潮电子信息产业股份有限公司 | Efficient Hadoop cluster deployment method |
| CN106445611A (en) * | 2016-09-30 | 2017-02-22 | 广州特道信息科技有限公司 | Big data node system and automatic deploying method |
| US20180239646A1 (en) * | 2014-12-12 | 2018-08-23 | Nec Corporation | Information processing device, information processing system, task processing method, and storage medium for storing program |
| CN109284272A (en) * | 2018-09-07 | 2019-01-29 | 郑州云海信息技术有限公司 | A kind of dispositions method of distributed file system, device and equipment |
-
2019
- 2019-08-15 CN CN201910753380.9A patent/CN112398669B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064742A (en) * | 2012-12-25 | 2013-04-24 | 中国科学院深圳先进技术研究院 | Automatic deployment system and method of hadoop cluster |
| US20180239646A1 (en) * | 2014-12-12 | 2018-08-23 | Nec Corporation | Information processing device, information processing system, task processing method, and storage medium for storing program |
| CN105893545A (en) * | 2016-04-01 | 2016-08-24 | 浪潮电子信息产业股份有限公司 | Efficient Hadoop cluster deployment method |
| CN106445611A (en) * | 2016-09-30 | 2017-02-22 | 广州特道信息科技有限公司 | Big data node system and automatic deploying method |
| CN109284272A (en) * | 2018-09-07 | 2019-01-29 | 郑州云海信息技术有限公司 | A kind of dispositions method of distributed file system, device and equipment |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127150A (en) * | 2021-03-18 | 2021-07-16 | 同盾控股有限公司 | Rapid deployment method and device of cloud native system, electronic equipment and storage medium |
| CN113127150B (en) * | 2021-03-18 | 2023-10-17 | 同盾控股有限公司 | Rapid deployment method and device of cloud primary system, electronic equipment and storage medium |
| CN113377385A (en) * | 2021-06-07 | 2021-09-10 | 中国工商银行股份有限公司 | Client automatic deployment method and device |
| CN113656147A (en) * | 2021-08-20 | 2021-11-16 | 北京百度网讯科技有限公司 | Cluster deployment method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112398669B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108629029B (en) | Data processing method and device applied to data warehouse | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN112398669B (en) | Hadoop deployment method and device | |
| CN111478781B (en) | Message broadcasting method and device | |
| CN113672357A (en) | Task scheduling method, device and system | |
| CN112000734A (en) | Big data processing method and device | |
| CN115794262A (en) | Task processing method, device, equipment, storage medium and program product | |
| CN111858040A (en) | Resource scheduling method and device | |
| CN112084042A (en) | Message processing method and device | |
| CN110795328A (en) | Interface testing method and device | |
| CN113742389A (en) | Service processing method and device | |
| CN113760638A (en) | Log service method and device based on kubernets cluster | |
| CN111767126A (en) | System and method for distributed batch processing | |
| CN113672671A (en) | Method and device for realizing data processing | |
| CN111831503A (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN110764769A (en) | Method and device for processing user request | |
| CN112685481A (en) | Data processing method and device | |
| CN112667368A (en) | Task data processing method and device | |
| CN113779122B (en) | Method and device for exporting data | |
| CN112860447B (en) | Interaction method and system between different applications | |
| CN112688982B (en) | User request processing method and device | |
| CN114070889A (en) | Configuration method, traffic forwarding method, device, storage medium, and program product | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN112559001A (en) | Method and device for updating application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |