CN112398669B - Hadoop deployment method and device - Google Patents
Hadoop deployment method and device Download PDFInfo
- Publication number
- CN112398669B CN112398669B CN201910753380.9A CN201910753380A CN112398669B CN 112398669 B CN112398669 B CN 112398669B CN 201910753380 A CN201910753380 A CN 201910753380A CN 112398669 B CN112398669 B CN 112398669B
- Authority
- CN
- China
- Prior art keywords
- deployment
- hadoop
- commands
- maintenance platform
- automatic operation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012423 maintenance Methods 0.000 claims abstract description 75
- 238000012545 processing Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 10
- 150000003839 salts Chemical class 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 108010015780 Viral Core Proteins Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0266—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using meta-data, objects or commands for formatting management information, e.g. using eXtensible markup language [XML]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0253—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using browsers or web-pages for accessing management information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a Hadoop deployment method and device, and relates to the technical field of computers. One embodiment of the method comprises the following steps: receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more pieces of deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster for executing the deployment commands or one or more Hadoop nodes; merging or splitting the one or more deployment commands; and sending the one or more combined or split deployment commands to a server side of the automatic operation and maintenance platform, so that Hadoop nodes managed by the server side of the automatic operation and maintenance platform execute the deployment commands. According to the implementation mode, the memory required to be consumed when the automatic operation and maintenance platform processes the deployment command is reduced, the throughput of the automatic operation and maintenance platform is improved, and further the Hadoop deployment efficiency is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a Hadoop deployment method and device.
Background
Hadoop is a distributed system infrastructure developed by the Apache foundation, with two major core designs: HDFS and MapReduce; the HDFS (Hadoop Distributed File System), i.e. the distributed file system, provides storage for massive data, while the MapReduce provides possibility for computing the massive data, so that the Hadoop not only can provide high-throughput application data access, but also is suitable for application programs with large data sets. Therefore, there is often a need to deploy Hadoop clusters in batches or execute commands in batches. Currently, a commonly used Hadoop deployment method is to implement Hadoop deployment by means of an automated operation and maintenance tool or an automated operation and maintenance platform.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art: with the great increase of the number of deployed Hadoop nodes, particularly when tens of thousands of Hadoop nodes are required to be deployed at the same time, great pressure is caused to a processor by processing a large number of deployment commands and execution results of the Hadoop nodes, the throughput of a system is limited, and the efficiency of Hadoop deployment is reduced.
Disclosure of Invention
In view of the above, the invention provides a Hadoop deployment method and device, which can realize the distribution control of deployment commands by splitting or merging the deployment commands, greatly reduce the pressure on a processor caused by the deployment commands and the execution results of Hadoop nodes, and improve the Hadoop deployment efficiency.
To achieve the above object, according to a first aspect of the present invention, there is provided a Hadoop deployment method, including: receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more pieces of deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster for executing the deployment commands or one or more Hadoop nodes; merging or splitting the one or more deployment commands; and sending the one or more combined or split deployment commands to a server side of the automatic operation and maintenance platform, so that Hadoop nodes managed by the server side of the automatic operation and maintenance platform execute the deployment commands.
Optionally, the one or more deployment commands are merged or split according to one or more of: the method comprises the steps of deploying the number of commands, the type of Hadoop nodes executing the deploying commands and the number of Hadoop nodes executing the deploying commands.
Optionally, the server side of the automated operation and maintenance platform sends the one or more deployment commands after being combined or split to the Hadoop node through a Redis queue.
Optionally, the method further comprises: receiving one or more execution results returned by a result reflector of the automatic operation and maintenance platform; and merging the one or more execution results, and returning the merged execution results to the deployment foreground.
Optionally, the automated operation and maintenance platform is Saltstack or an Anstable.
To achieve the above object, according to a second aspect of the present invention, there is provided a Hadoop deployment device, including: a deployment request receiving module, a command management module and a command sending module; the deployment request receiving module is used for receiving a deployment request sent by a deployment foreground, the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, hadoop clusters for executing the deployment commands or one or more Hadoop nodes; the command management module is used for merging or splitting the one or more deployment commands; the command sending module is used for sending the one or more combined or split deployment commands to the service end of the automatic operation and maintenance platform, so that Hadoop nodes managed by the service end of the automatic operation and maintenance platform execute the deployment commands.
Optionally, the command management module is configured to merge or split the one or more deployment commands according to one or more of: the method comprises the steps of deploying the number of commands, the type of Hadoop nodes executing the deploying commands and the number of Hadoop nodes executing the deploying commands.
Optionally, the server side of the automated operation and maintenance platform sends the one or more deployment commands after merging or splitting to the Hadoop node through a Redis queue.
Optionally, the method further comprises: an execution result processing module; the execution result processing module is used for receiving one or more execution results returned by a result reflector of the automatic operation and maintenance platform; and merging the one or more execution results, and returning the merged execution results to the deployment foreground.
Optionally, the automated operation and maintenance platform is Saltstack or an Anstable.
To achieve the above object, according to a third aspect of the present invention, there is provided a server for Hadoop deployment, comprising: one or more processors; and a storage device for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the Hadoop deployment methods described above.
To achieve the above object, according to a fourth aspect of the present invention, a computer readable medium has stored thereon a computer program, characterized in that the program, when executed by a processor, implements a method according to any one of the Hadoop deployment methods described above.
Due to the adoption of the technical scheme, the invention has the following advantages or beneficial effects: by merging or splitting the deployment commands in the deployment request, the control of the deployment command distribution is realized, the pressure of excessive deployment commands on the automatic operation and maintenance platform is avoided, and the efficiency of the automatic operation and maintenance platform for deploying commands to Hadoop nodes is improved; meanwhile, the pressure of the automatic operation and maintenance platform server side when sending commands to the managed Hadoop nodes is further relieved through the Redis queue; in addition, by means of the mode that the result returning device of the automatic operation and maintenance platform directly returns the execution result, the pressure of inquiring or obtaining the execution result on the server where the automatic operation and maintenance platform is located is greatly reduced, meanwhile, the efficiency of obtaining the execution result is improved, and the deployment foreground can monitor the deployment progress or the Hadoop node execution deployment command progress in real time.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a Hadoop deployment method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the major modules of a Hadoop deployment apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic flow diagram of a Hadoop deployment device application method according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 5 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As shown in fig. 1, the embodiment of the invention provides a Hadoop deployment method, which specifically includes the following steps:
step S101, receiving a deployment request sent by a deployment foreground, where the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster for executing the deployment commands, or one or more Hadoop nodes.
The deployment foreground is a WEB module of the whole deployment system and provides all visual operations including task progress control, execution result query display, software management and the like. The deployment foreground can store the execution result into the local database after receiving the execution result returned according to the deployment command, so that the execution result can be queried in real time or according to actual requirements, and the execution result is visually displayed. The deployment command may then include any command that executes on the Hadoop node, such as deleting a directory, creating a directory, launching a Hadoop service, etc.
Furthermore, in order to realize asynchronous processing of sending the deployment request and receiving the execution result by the deployment foreground, the throughput of the system in Hadoop deployment is increased, and the sending of the deployment request and the receiving of the execution result can be performed through a ActiveMQ, rabbitMQ, kafka message queue, so that the efficiency of sending the deployment command and receiving the execution result by the deployment foreground is improved.
Step S102, merging or splitting the one or more deployment commands.
Along with the increase of large data volume, the number of corresponding Hadoop nodes needing to be deployed simultaneously or in batches is also rapidly increased, and based on the number, splitting or merging of deployment commands is considered, so that the shunting control of the deployment commands is realized. For example, when deploying the same software package on ten thousand Hadoop nodes or executing the update of the same software package, the ten thousand deployment commands need to be sent to the managed Hadoop nodes through the automatic operation and maintenance platform at the same time, so that the number of processes needing to be operated simultaneously by the automatic operation and maintenance platform is increased suddenly, the pressure of a processor is overlarge, the efficiency of Hadoop deployment is limited to a great extent, and therefore, the ten thousand deployment commands are considered to be split into a plurality of batches for execution, such as 5000 execution at a time, and the like, thereby reducing the number of deployment commands which need to be processed at one time by the automatic operation and maintenance platform and relieving the pressure of a system when the deployment commands are processed.
It can be appreciated that in the actual execution process, the splitting of the deployment command can be implemented according to the actual requirement, such as the number of deployment commands, the type of Hadoop nodes executing the deployment command, and the number of Hadoop nodes executing the deployment command. Specifically, when the number of the deployment commands is large, the deployment commands are processed in batches; when the number of the deployment commands is small, the deployment commands can be combined into new deployment commands, so that the number of Hadoop nodes for executing the deployment commands is reduced, and load balancing is realized. Meanwhile, when the deployment commands are merged or split, different deployment commands can be distributed to different types of Hadoop nodes according to different types of deployed Hadoop nodes or different provided services, such as DataNode, nameNode.
Step S103, the one or more combined or split deployment commands are sent to a server side of the automatic operation and maintenance platform, so that Hadoop nodes managed by the server side of the automatic operation and maintenance platform execute the deployment commands.
In the actual execution process, a server or a client of an automatic operation and maintenance platform is arranged at the position of each Hadoop node, so that the server of the automatic operation and maintenance platform can send a deployment command to the client of the automatic operation and maintenance platform where the Hadoop node is located; and after receiving the deployment command, the client of the automatic operation and maintenance platform calls the managed Hadoop node to execute the deployment command.
In an alternative embodiment, the automated operation and maintenance platform is Saltstack or Ancable. The SaltStack is a centralized management platform of a server infrastructure, has functions of configuration management, remote execution, monitoring and the like, is realized based on Python language, and is constructed by combining a lightweight message queue (zeroMQ) with a Python third party module (Pyzmq, pyCrypto, pyjinjia, python-msgpack, pyYAML and the like); the SaltStack can realize batch execution of commands on thousands of servers, and perform configuration centralized management, file distribution, server data acquisition, operating system foundation and software package management according to different services. Anstable is an automatic operation and maintenance platform based on Python development, and functions such as batch system configuration, batch program deployment, batch operation commands and the like are realized.
In an optional implementation manner, the server side of the automated operation and maintenance platform sends the one or more deployment commands after being combined or split to the Hadoop node through a Redis queue. Therefore, the pressure of the server side of the automatic operation and maintenance platform when sending the deployment command to the client side of the automatic operation and maintenance platform is further reduced through the cache of the Redis queue, the efficiency of the automatic operation and maintenance platform in processing the deployment command is improved, and further the efficiency of Hadoop deployment is improved.
In an alternative embodiment, one or more execution results returned by a result reflector of the automatic operation and maintenance platform are received; and merging the one or more execution results, and returning the merged execution results to the deployment foreground. Specifically, after the Hadoop node executes the deployment command, the execution result is directly returned to the Hadoop deployment device by triggering a result reflector (such as a Salt return of a Saltstack) of the automatic operation and maintenance platform; after receiving the execution result returned by the Salt return, the Hadoop deployment device locally stores the execution result, merges the execution result, and sends the merged execution result to the deployment foreground through a message queue between the Hadoop deployment device and the deployment foreground. That is, the method avoids returning the execution result to the automatic operation and maintenance platform server through the automatic operation and maintenance platform client or actively inquiring a large amount of memory consumed by the execution result to the automatic operation and maintenance platform client by the automatic operation and maintenance platform server, improves the efficiency of the deployment command through the automatic operation and maintenance platform server, improves the efficiency of the deployment foreground in receiving the execution result through combining the execution result, and provides possibility for the real-time monitoring of the deployment progress of the deployment foreground.
Based on the embodiment, the deployment commands in the deployment request are combined or split, so that the control of the deployment command distribution is realized, the pressure of excessive deployment commands on the automatic operation and maintenance platform is avoided, and the efficiency of the automatic operation and maintenance platform for deploying the commands to Hadoop nodes is improved; meanwhile, the pressure of the automatic operation and maintenance platform server side when sending commands to the managed Hadoop nodes is further relieved through the Redis queue; in addition, by means of the mode that the result returning device of the automatic operation and maintenance platform directly returns the execution result, the pressure of inquiring or obtaining the execution result on the server where the automatic operation and maintenance platform is located is greatly reduced, meanwhile, the efficiency of obtaining the execution result is improved, and the deployment foreground can monitor the deployment progress or the Hadoop node execution deployment command progress in real time.
Referring to fig. 2, an embodiment of the present invention provides a Hadoop deployment apparatus 200, including: a deployment request receiving module 201, a command management module 202, and a command transmitting module 203; wherein,,
the deployment request receiving module 201 is configured to receive a deployment request sent by a deployment foreground, where the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, hadoop clusters for executing the deployment commands, or one or more Hadoop nodes;
the command management module 202 is configured to merge or split the one or more deployment commands;
the command sending module 203 is configured to send the one or more merged or split deployment commands to a server of the automated operation and maintenance platform, so that a Hadoop node managed by the server of the automated operation and maintenance platform executes the deployment commands.
In an alternative embodiment, the command management module 202 is configured to merge or split the one or more deployment commands according to one or more of: the method comprises the steps of deploying the number of commands, the type of Hadoop nodes executing the deploying commands and the number of Hadoop nodes executing the deploying commands.
In an optional implementation manner, the server side of the automated operation and maintenance platform sends the one or more deployment commands after being combined or split to the Hadoop node through a Redis queue.
In an alternative embodiment, the method further comprises: an execution result processing module 204; the execution result processing module 204 is configured to receive one or more execution results returned by a result reflector of the automated operation and maintenance platform; and merging the one or more execution results, and returning the merged execution results to the deployment foreground.
In an alternative embodiment, the automated operation and maintenance platform is Saltstack or Ancable.
Referring to fig. 3, on the basis of the foregoing embodiment, the embodiment of the present invention provides an application method of a Hadoop deployment device, which specifically includes:
the Hadoop deployment device 200 receives a deployment request sent by a deployment foreground through a message queue between the Hadoop deployment device and the deployment foreground, wherein the deployment request indicates one or more deployment data, and the deployment data indicates one or more deployment commands, hadoop clusters for executing the deployment commands, or one or more Hadoop nodes.
Hadoop deployment device 200 merges or splits one or more deployment commands.
The Hadoop deployment device 200 sends the one or more deployment commands after merging or splitting to the server side of the automated operation and maintenance platform, so that the Hadoop node managed by the server side of the automated operation and maintenance platform executes the deployment command.
Specifically, the Hadoop deployment device 200 sends the deployment command after being combined or split to a service end (Salt Master) of the automated operation and maintenance platform; the Salt Master sends the deployment command to a client (Salt Minion) of an automated operation and maintenance platform deployed on a Hadoop node executing the deployment command through a message queue; the Hadoop node managed by the Salt Minion is called, so that the Hadoop node executes a deployment command; after the Hadoop node executes the deployment command, triggering a result return device (Salt return) of the automatic operation and maintenance platform, so that the Salt return directly returns the execution result to the Hadoop deployment device 200; after receiving the execution result returned by the Salt return, the Hadoop deployment device 200 locally stores the execution result, merges the execution result, and sends the merged execution result to the deployment foreground through a message queue between the merged execution result and the deployment foreground.
FIG. 4 illustrates an exemplary system architecture 400 in which a Hadoop deployment method or device of embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 is used as a medium to provide communication links between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 405 via the network 404 using the terminal devices 401, 402, 403 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc., may be installed on the terminal devices 401, 402, 403.
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server providing support for shopping-type websites browsed by the user using the terminal devices 401, 402, 403. The background management server may analyze and process the received data such as the product information query request, and feed back the processing result (for example, the combined execution result) to the terminal device.
It should be noted that, in the embodiment of the present invention, the Hadoop deployment method is generally executed by the server 405, and accordingly, the Hadoop deployment device is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, there is illustrated a schematic diagram of a computer system 500 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output portion 507 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as needed so that a computer program read therefrom is mounted into the storage section 508 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509, and/or installed from the removable media 511. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 501.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a deployment request receiving module, a command management module, and a command sending module, where the names of the modules do not in some cases constitute a limitation on the module itself, e.g., the command management module may also be described as a "module that merges or splits the one or more deployment commands".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates one or more pieces of deployment data, and the deployment data indicates one or more deployment commands, a Hadoop cluster for executing the deployment commands or one or more Hadoop nodes; merging or splitting the one or more deployment commands; and sending the one or more combined or split deployment commands to a server side of the automatic operation and maintenance platform, so that Hadoop nodes managed by the server side of the automatic operation and maintenance platform execute the deployment commands.
According to the technical scheme of the embodiment of the invention, the deployment command distribution control is realized by merging or splitting the deployment command in the deployment request, so that the pressure of excessive deployment commands on the automatic operation and maintenance platform is avoided, and the efficiency of the automatic operation and maintenance platform for deploying commands to Hadoop nodes is improved; meanwhile, the pressure of the automatic operation and maintenance platform server side when sending commands to the managed Hadoop nodes is further relieved through the Redis queue; in addition, by means of the mode that the result returning device of the automatic operation and maintenance platform directly returns the execution result, the pressure of inquiring or obtaining the execution result on the server where the automatic operation and maintenance platform is located is greatly reduced, meanwhile, the efficiency of obtaining the execution result is improved, and the deployment foreground can monitor the deployment progress or the Hadoop node execution deployment command progress in real time.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (12)
1. A Hadoop deployment method, comprising:
receiving a deployment request sent by a deployment foreground, wherein the deployment request indicates a plurality of deployment data, and the deployment data indicates a plurality of deployment commands and Hadoop clusters or one or more Hadoop nodes for executing the deployment commands;
merging or splitting the plurality of deployment commands;
and sending the one or more combined or split deployment commands to a server side of the automatic operation and maintenance platform, so that Hadoop nodes managed by the server side of the automatic operation and maintenance platform execute the deployment commands.
2. The Hadoop deployment method according to claim 1, wherein,
merging or splitting the plurality of deployment commands according to one or more of: the method comprises the steps of deploying the number of commands, the type of Hadoop nodes executing the deploying commands and the number of Hadoop nodes executing the deploying commands.
3. The Hadoop deployment method of claim 1, wherein the server of the automated operation and maintenance platform sends the one or more deployment commands after merging or splitting to the Hadoop node through a Redis queue.
4. The Hadoop deployment method of claim 1, further comprising:
receiving one or more execution results returned by a result reflector of the automatic operation and maintenance platform;
and merging the one or more execution results, and returning the merged execution results to the deployment foreground.
5. The Hadoop deployment method of claim 1, wherein the automated operation and maintenance platform is a Saltstack or an stable.
6. A Hadoop deployment device, comprising: a deployment request receiving module, a command management module and a command sending module; wherein,,
the deployment request receiving module is used for receiving a deployment request sent by a deployment foreground, the deployment request indicates a plurality of deployment data, and the deployment data indicates a plurality of deployment commands and Hadoop clusters or one or more Hadoop nodes for executing the deployment commands;
the command management module is used for merging or splitting the plurality of deployment commands;
the command sending module is used for sending the one or more combined or split deployment commands to the service end of the automatic operation and maintenance platform, so that Hadoop nodes managed by the service end of the automatic operation and maintenance platform execute the deployment commands.
7. The Hadoop deployment device of claim 6, wherein the command management module is configured to merge or split the plurality of deployment commands according to one or more of: the method comprises the steps of deploying the number of commands, the type of Hadoop nodes executing the deploying commands and the number of Hadoop nodes executing the deploying commands.
8. The Hadoop deployment device of claim 6, wherein the server of the automated operation and maintenance platform sends the one or more deployment commands after merging or splitting to the Hadoop node through a Redis queue.
9. The Hadoop deployment device of claim 6, further comprising: an execution result processing module; the execution result processing module is used for
Receiving one or more execution results returned by a result reflector of the automatic operation and maintenance platform;
and merging the one or more execution results, and returning the merged execution results to the deployment foreground.
10. The Hadoop deployment device of claim 6, wherein the automated operation and maintenance platform is a Saltstack or an onsable.
11. A server for Hadoop deployment, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-5.
12. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910753380.9A CN112398669B (en) | 2019-08-15 | 2019-08-15 | Hadoop deployment method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910753380.9A CN112398669B (en) | 2019-08-15 | 2019-08-15 | Hadoop deployment method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112398669A CN112398669A (en) | 2021-02-23 |
| CN112398669B true CN112398669B (en) | 2023-09-26 |
Family
ID=74601565
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910753380.9A Active CN112398669B (en) | 2019-08-15 | 2019-08-15 | Hadoop deployment method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112398669B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127150B (en) * | 2021-03-18 | 2023-10-17 | 同盾控股有限公司 | Rapid deployment method and device of cloud primary system, electronic equipment and storage medium |
| CN113377385A (en) * | 2021-06-07 | 2021-09-10 | 中国工商银行股份有限公司 | Client automatic deployment method and device |
| CN113656147B (en) * | 2021-08-20 | 2023-03-31 | 北京百度网讯科技有限公司 | Cluster deployment method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064742A (en) * | 2012-12-25 | 2013-04-24 | 中国科学院深圳先进技术研究院 | Automatic deployment system and method of hadoop cluster |
| CN105893545A (en) * | 2016-04-01 | 2016-08-24 | 浪潮电子信息产业股份有限公司 | Efficient Hadoop cluster deployment method |
| CN106445611A (en) * | 2016-09-30 | 2017-02-22 | 广州特道信息科技有限公司 | Big data node system and automatic deploying method |
| CN109284272A (en) * | 2018-09-07 | 2019-01-29 | 郑州云海信息技术有限公司 | A kind of dispositions method of distributed file system, device and equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6107801B2 (en) * | 2014-12-12 | 2017-04-05 | 日本電気株式会社 | Information processing apparatus, information processing system, task processing method, and program |
-
2019
- 2019-08-15 CN CN201910753380.9A patent/CN112398669B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064742A (en) * | 2012-12-25 | 2013-04-24 | 中国科学院深圳先进技术研究院 | Automatic deployment system and method of hadoop cluster |
| CN105893545A (en) * | 2016-04-01 | 2016-08-24 | 浪潮电子信息产业股份有限公司 | Efficient Hadoop cluster deployment method |
| CN106445611A (en) * | 2016-09-30 | 2017-02-22 | 广州特道信息科技有限公司 | Big data node system and automatic deploying method |
| CN109284272A (en) * | 2018-09-07 | 2019-01-29 | 郑州云海信息技术有限公司 | A kind of dispositions method of distributed file system, device and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112398669A (en) | 2021-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108629029B (en) | Data processing method and device applied to data warehouse | |
| CN112398669B (en) | Hadoop deployment method and device | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN112860451A (en) | Multi-tenant data processing method and device based on SaaS | |
| CN111767157B (en) | Request processing method and device based on service grid | |
| CN110321252B (en) | Skill service resource scheduling method and device | |
| CN111478781B (en) | Message broadcasting method and device | |
| CN113050940A (en) | Method for previewing small program, related device and computer program product | |
| CN112000734A (en) | Big data processing method and device | |
| CN112084042B (en) | Message processing method and device | |
| CN113760488A (en) | Method, device, equipment and computer readable medium for scheduling task | |
| CN113672357A (en) | Task scheduling method, device and system | |
| CN112947919A (en) | Method and device for constructing service model and processing service request | |
| CN113282589A (en) | Data acquisition method and device | |
| CN110795328A (en) | Interface testing method and device | |
| CN111382953A (en) | Dynamic process generation method and device | |
| CN110764769A (en) | Method and device for processing user request | |
| CN111831503A (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113779122B (en) | Method and device for exporting data | |
| CN112860447B (en) | Interaction method and system between different applications | |
| CN114896244A (en) | Method, device and equipment for configuring database table and computer readable medium | |
| CN112688982B (en) | User request processing method and device | |
| CN112559001B (en) | Method and device for updating application | |
| CN110909269B (en) | Log reporting method and device | |
| CN110019445A (en) | Method of data synchronization and device calculate equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |