KR101352699B1 - Method and Apparatus for Provisioning Parallel Processing-Based Virtual Cluster Computing System - Google Patents

Abstract

A method and apparatus for provisioning a parallel processing based virtual cluster computing system is disclosed in accordance with one embodiment of the present invention. In one embodiment, a method of provisioning a virtual cluster system includes: executing work node virtual machines and a master virtual machine according to user-specific system configuration information of the virtual cluster computing system; Installing, by a main distributed configuration manager, a sub distributed configuration manager on the master virtual machine; And installing, by the sub-distributed configuration manager, a work node agent on the work node virtual machines, respectively.

Description

Method and Apparatus for Provisioning Parallel Cluster Based Computing System {Method and Apparatus for Provisioning Parallel Processing-Based Virtual Cluster Computing System}

The present invention relates to a method and apparatus for provisioning a virtual cluster computing system based on parallel processing, and to prevent overload of the distributed configuration manager in a scalable manner to meet the needs of users even when a plurality of users request creation of a virtual cluster. Provisioning method, configuration automation, and service method that can provide virtual cluster computing system system for parallel processing.

Recently, the amount of data to be processed is increasing due to the increase in the use of mobile devices and the rapid increase in the size of media data, and the possibility of increasing the added value of the service through the analysis of the large amount of data collected during the automated data collection process through the service is various. It is developing. Analyzing large amounts of data typically requires purchasing system resources and configuring and operating complex distributed computing systems using distributed computing software.

Distributed Configuration Manager is required to manage distributed computing systems. However, even with distributed configuration manager, it is very difficult for general users to build distributed computing system for processing large data, tune performance and apply it to actual work. For this reason, the importance of virtual cluster technology has recently increased, by using cloud computing technology and virtualization technology to automate the installation and configuration of resources and distributed or parallel processing software that users need, so that users can perform only necessary data analysis. It became feasible.

When multiple users simultaneously request configuration of distributed computing systems on physical resources, the load on the distributed configuration manager increases, thereby reducing the tasks and responsiveness requested by each user, and successfully creating a virtual machine resource that satisfies the user's request. There is still a risk of not being able to run it. . There is a need for distributed computing technology that reliably scales and scales data center resources to meet growing user requirements and scales to meet growing user requirements and resource requests.

According to an embodiment of the present invention, if a plurality of users simultaneously request the configuration of the virtual cluster system, the problem may be caused by overloading the distributed configuration manager and the virtual infrastructure manager or delayed access to the configuration management data in the configuration management repository. It is possible to provide a provisioning method and apparatus for a virtual cluster computing system that can be solved.

According to an embodiment of the present invention, a virtual cluster computing system may be used to increase the reliability and response success rate of user operations by using asynchronous communication between a virtual machine of a virtual cluster system through a message queue, a distributed coordinator, and a distributed configuration manager. Provision can be made and a virtual cluster device using a virtual machine can be provided.

In addition, according to an embodiment of the present invention, it is possible to provide a recording medium on which a program capable of performing the provisioning method of the virtual cluster computing system is recorded.

According to an embodiment of the present invention, there is provided a method for provisioning a parallel cluster-based virtual cluster computing system, comprising: executing work node virtual machines and a master virtual machine according to user-specific system configuration information of the virtual cluster computing system. ; Installing, by a main distributed configuration manager, a sub distributed configuration manager on the master virtual machine; And installing, by the sub distributed configuration manager, a work node agent on the work node virtual machines, respectively.

According to an embodiment of the present invention, in a method for provisioning a parallel cluster-based virtual cluster computing system, the first work node virtual machines and the first master virtual machine may be configured according to the system configuration information of the virtual cluster of the first user. Executing; Installing, by a main distributed configuration manager, a first sub distributed configuration manager on the first master virtual machine; Executing the second working node virtual machines and the second master virtual machine according to the system configuration information of the virtual cluster of the second user; Installing, by the main distributed configuration manager, a second sub distributed configuration manager on the second master virtual machine, wherein the first and second sub distributed configuration managers each of the first working node virtual machines. And installing a work node agent in each of the second work node virtual machines, and the work node agent installs software for parallel processing in each work node virtual machine.

According to an embodiment of the present invention, an apparatus for provisioning a virtual cluster based on parallel processing, the apparatus comprising: a virtual infrastructure manager capable of executing a master virtual machine and one or more work node virtual machines according to configuration information of the virtual cluster; And a first distributed configuration manager capable of creating the virtual cluster, wherein the first distributed configuration manager can install a second distributed configuration manager in the master virtual machine according to a request of the master virtual machine. Can be provided.

According to an embodiment of the present invention, in an apparatus for provisioning a virtual cluster based on parallel processing, the first master virtual machine and the at least one first working node virtual machine may be executed according to configuration information of the virtual cluster of the first user. A virtual infrastructure manager capable of executing a second master virtual machine and one or more second work node virtual machines according to configuration information of the virtual cluster of the second user; And a main distributed configuration manager capable of creating the first and second virtual clusters, respectively, wherein the main distributed configuration manager connects to the first master virtual machine at the request of the first master virtual machine. A provisioning apparatus may be provided in which a first sub distributed configuration manager may be installed and a second sub distributed configuration manager may be installed in the second master virtual machine in response to a request of the second master virtual machine.

According to an embodiment of the present invention, a recording medium having a computer readable program recorded thereon capable of performing the method can be provided.

According to an embodiment of the present invention, if a plurality of users simultaneously request the configuration of the virtual cluster system, the problem may be caused by overloading the distributed configuration manager and the virtual infrastructure manager or delayed access to the configuration management data in the configuration management repository. I can solve it.

According to an embodiment of the present invention, by using asynchronous communication between the virtual machine of the virtual cluster system through the message queue, the distributed coordinator and the distributed configuration manager, it is possible to increase the stability of the user task performance and the success rate of response.

1 is a block diagram provided to explain a method and apparatus for provisioning a parallel processing based distributed computing system according to an embodiment of the present invention;
2 is a flowchart illustrating a method of provisioning a parallel computing based distributed computing system and processing data at the request of a user according to an embodiment of the present invention;
3 is a flowchart illustrating a method for a user connecting to a parallel processing based distributed computing system via a virtual private network according to an embodiment of the present invention;
4 is a block diagram schematically illustrating a distributed configuration manager 40 provisioning a parallel processing based distributed computing system for each user according to an embodiment of the present invention;
5A and 5B are flow charts illustrating a method for provisioning a parallel processing based distributed computing system in accordance with one embodiment of the present invention;
6 to 9 are block diagrams illustrating a method for provisioning a parallel computing based distributed computing system according to an embodiment of the present invention;
10 is a flowchart illustrating a method of processing user data in a parallel computing based distributed computing system according to an embodiment of the present invention;
11 is a block diagram illustrating a method of processing data in a parallel computing based distributed computing system according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features, and advantages of the present invention will become more readily apparent from the following description of preferred embodiments with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. Rather, the embodiments disclosed herein are provided so that the disclosure can be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

In the present specification, that the first component is operated (or executed) on the second component means that the first component is operated (or executed) in an environment in which the second component is operated (or executed) or Or means in which the first component is operated (or executed) through interaction with the second component directly or indirectly.

When a component, device, or system is referred to as including a component made up of a program or software, even if not expressly stated, the component, device, or system may be a piece of hardware necessary for the program or software to execute or operate. For example, it may be understood to include a memory, a CPU, etc.) or other programs or software (eg, an operating system or a driver required to drive hardware).

It is also to be understood that unless otherwise stated in the implementation of any component, the component may be implemented in software, hardware, or any form of software and hardware.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used in the specification, 'comprises' and / or 'comprising' does not exclude the presence or addition of one or more other components in addition to the components mentioned.

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the following specific embodiments, various specific details are set forth in order to explain and understand the invention in more detail. However, those skilled in the art can understand that the present invention can be used without these various specific details. In some instances, it should be noted that portions of the invention that are well known in the description of the invention and are not significantly related to the invention do not describe confusion in describing the invention.

1 is a block diagram provided to explain a method and apparatus for provisioning a parallel processing based distributed computing system according to an embodiment of the present invention.

According to one preferred embodiment of the present invention, the cloud computing system 30 may be an apparatus for provisioning a parallel computing based distributed computing system. The cloud computing system 30 communicates with the service server 20 via a wired or wireless network.

The service server 20 receives a request for processing a specific service from the user terminal 10 and provides the cloud computing system 30 with information necessary for provisioning and service processing of the distributed computing system for processing the service, and the result of the service processing. Received from the cloud computing system 30 may transmit the result to the user terminal 10.

In the illustrated embodiment, the service server 20 includes a message processing interface 201, a user input interpreter 202, a rights manager 203, a data store 204, a virtual private network manager 205, a wait queue ( 206, virtual infrastructure request client 207, and management module 208.

The message processing interface 201 is an interface that receives input information such as resource scale, configuration requirements, and job execution information of the distributed computing system from the user terminal 10. In one embodiment, the input information from the user terminal 10 may be provided in the form of a message. The user input interpreter 202 interprets the information provided from the user terminal 10 into a data structure that can be processed in the service server 20 by interpreting the message containing the requirements for the distributed computing system, On the contrary, the service server 20 converts the information provided to the user into a form that the user can understand and use.

The authority manager 203 is a module for authenticating whether a user who has requested a service through the user terminal 10 is a legitimate user and confirming service use rights and restrictions allowed by the user. The data store 204 is a data storage device in a distributed file system. The data store 204 is a device for storing and viewing data and files necessary for executing a user's job, and result data and outputs generated after the job is executed.

The virtual private network manager 205 configures a virtual private network channel that a user can access to the virtual network of the virtual infrastructure manager, maps and manages the user's personal authentication and access information, and authenticates and requests each time a user requests. The authorization manager can provide a channel for accessing a virtual private network through single sign-on.

The work waiting queue 206 queues and manages work requests until the cloud computing resources in which the user's distributed computing system provisioning requests and job execution requests are available within the cloud computing system 30 are available. Service module. The virtual infrastructure request client 207 is a module for requesting a virtual resource from the cloud computing system 30 according to a user's request.

The management module 208 is a module for monitoring and managing provisioning status of a distributed computing system, execution status of user tasks in a distributed computing system, and the like.

With respect to the above-described components of the service server 20, some functions may be added or limited according to the embodiment of the invention. In addition, according to an embodiment of the present invention, new components may be added and some of the aforementioned components may be omitted. For example, if the system does not provide a private network, the virtual private network manager 205 may be omitted. In addition, according to an embodiment of the present invention, the two or more components described above may operate as one integrated module, for example, a message interface 201 communicating with the user terminal 10 and a user input interpreter 202 for interpreting user input. ) May be implemented as one integrated module.

In one embodiment of the invention, the cloud computing system 30 is an apparatus for provisioning a distributed computing system 50 based on parallel processing. The cloud computing system 30 may also provision a parallel cluster based virtual cluster for each user. Here, "provisioning" refers to disposing and providing specific requested computer resources, for example, identifying resources not being used in the cloud computing system 30 and allocating resources according to user requests to provide services.

In addition, the "parallel processing based distributed computing system" 50 referred to herein is referred to simply as a "distributed computing system" throughout the specification, and is an embodiment for provisioning a parallel processing based distributed computing system. A method of provisioning a "virtual MapReduce cluster system" (hereinafter referred to simply as "map reduce system", "virtual cluster system", or "virtual cluster") for each user will be described. However, the present invention is not limited thereto and may include a method and apparatus for provisioning a distributed computing system based on any type of parallel processing.

In one embodiment, cloud computing system 30 includes hardware 301, hypervisor 302. The hardware 301 may include, for example, one or more servers, storage devices, and network devices for performing cloud computing.

Hypervisor 302 is also referred to as a Virtual Machine Monitor (VMM) as a virtual platform for concurrently running multiple operating systems on hardware 301 of cloud computing system 30.

The virtual infrastructure manager 303 is a system for creating, operating, and managing virtual resources such as virtual machines, virtual storage devices, and virtual networks. The virtual infrastructure manager API 305 is a set of protocols and command interfaces that allow commands to operate the virtual infrastructure manager 303 to be executed locally or remotely. In the illustrated embodiment, the virtual infrastructure manager 303 may communicate with the virtual infrastructure request client 207 and / or the management module 208 of the service server 20 via the virtual infrastructure manager API 305.

The virtual private network manager 304 is a module for configuring, operating, and managing the virtual private network, and in one illustrated embodiment, communicates with the virtual private network manager 205 of the service server 20.

The main distributed configuration manager 40 can dynamically manipulate and change the configuration and operation of the system and the services in the system according to the user's request in order to create one MapReduce cluster system for each user in the distributed computing system 50. Module.

The above-described components of the cloud computing system 30 are according to an embodiment of the present invention, and some functions may be added or limited to each component according to the embodiment of the present invention. In addition, according to an embodiment of the present invention, new components may be added to the cloud computing system 30, and some of the above components may be omitted.

2 is a flowchart illustrating a method of provisioning a parallel processing based distributed computing system and processing data at the request of a user according to an embodiment of the present invention.

First, in step S201, the user executes data and tasks to be processed in the distributed computing system 50 to be virtually generated and provided to the requirements specification describing the size, specifications and requirements of the distributed computing system required by the user. User input information such as files is transmitted to the service server 20 through the user terminal 10.

The message processing interface 201 of the service server 20 delivers the user input information to the user input interpreter 202, which interprets the user's computing resources and system configuration requirements and job descriptions. Thus, system configuration information for configuring the distributed computing system 50 required by the user, and job execution information required for executing the job in the distributed computing system are generated.

Thereafter, in step S203, the user authentication and authority confirmation step by the authority manager 203 is performed. That is, it verifies whether the user is a legitimate user to use the service and checks the user's resource use level and authority by checking the service use authority. When authentication and authorization check are made in the authority manager 203, control is returned to the user input interpreter 202 again.

Next, in step S205, information for configuration and job processing of the distributed computing system, that is, system configuration information and job execution information is stored in the data store 204, and in step S207, the job request is waited on. Deliver to queue 206 to wait. In one embodiment, the wait queue 206, through communication between the virtual infrastructure request client 207 and the virtual infrastructure manager 301, may perform a distributed computing system request of a user waiting in the wait queue 206. It periodically checks if there are available resources in the cloud computing system 30 and waits for the work request in the work waiting queue 206 until the user can run the requested distributed computing system 50. Thereafter, when the virtual infrastructure manager 301 reserves resources for executing the requested distributed computing system, the virtual infrastructure manager 301 forwards the work request queued in the work waiting queue 206 to the virtual infrastructure request client 207.

In step S209, the virtual infrastructure request client 207 transmits the received user's work request to the virtual infrastructure manager 301 of the cloud computing system 30. In this case, system configuration information for configuring a distributed computing system and job execution information required for job execution may be included in the job request or transmitted to the virtual infrastructure manager 301 separately from the job request.

Thereafter, in step S211, the virtual infrastructure manager 301 uses the received system configuration information to store the virtual machine, virtual storage device, and virtual server server, storage resource, and network resource in the cloud computing system 30. Generate and schedule network resources to generate distributed computing system 50. The generated distributed computing system 50 runs on the physical resources through the hypervisor 302.

Next, in step S213, the main distributed configuration manager 304 configures the distributed computing system 50 to meet the requirements of the user according to the system configuration information. For example, the virtual map reduce cluster system may be configured by installing map reduce software in the distributed computing system 50 according to a user's requirement.

In step S213, the distributed computing system 50 is equipped with a system for executing a user request job, and then, in step S215, the user's request job is processed.

In one embodiment of the present invention, the distributed computing system 50 transmits the generated result data and files to the data store 204 of the service server 20 after performing a task requested by the user. The resulting data and files received in the data store 204 are converted or generated in response messages in the user input interpreter 202 and transmitted to the user terminal 10. In addition, the virtual machines in the distributed computing system 50 configured according to the user's request may be suspended and destroyed, or maintained until the user's next job execution request.

Meanwhile, according to an embodiment of the present invention, the user may monitor and manage the execution status of the distributed computing system 50 and the tasks executed in the distributed computing system 50 through the management module 208. In one specific embodiment, the user requests information to monitor the operation of the distributed computing system 50 or the execution of his or her task or manage the operation of the distributed computing system or his task through the management module 208. A predetermined administrative task can be requested, and this information request or administrative task request is transmitted to the virtual infrastructure manager 303. The result of the information request or management task request is then returned to the management module 208 for viewing by the user.

3 is a flowchart illustrating a method for a user connecting to a parallel processing based distributed computing system through a virtual private network according to an embodiment of the present invention.

First, in step S301, an access request from a user is sent to the virtual private network manager 205. That is, when a user requests direct access to the distributed computing system 50, the access requirement of the user may be virtualized by the service server 20 to open a virtual private network (VPN) for the direct access of the user. Request to the private network manager 205.

Then, in step S303, the single private sign-on (SSO) required by the virtual private network manager 205 to open the virtual private network, and the authorization check operation on whether the user can use the virtual private network are authorized. It is performed at manager 203.

If the authorization manager 203 is a legitimate user as a result of authentication and authorization check, in step S305, the virtual private network manager 205 configures the virtual private network available to the user in the cloud computing system 30. The virtual private network manager 304.

Then, in step S307, the virtual private network manager 304 requests the virtual infrastructure manager 301 to receive the virtual network information configured for the execution of the distributed computing system 50 of the current user, and configures the virtual private network. Virtual private network information is communicated to the virtual private network manager 205 of the service server 20. Then, in step S309, the virtual private network manager 205 transmits the virtual private network information and the access information for the user connection to the user terminal 10 through the message processing interface 201, whereby the user Through the distributed computing system 50 to be accessed.

4 is a block diagram schematically illustrating a distributed configuration manager 40 provisioning a parallel processing based distributed computing system for each user according to an embodiment of the present invention.

4 is a simplified diagram of the cloud computing system 30 shown in FIG. 1. For convenience of description, the virtual infrastructure manager 303, the virtual private network manager 304, and the virtual infrastructure manager API 305 are illustrated. Omitted.

Referring to FIG. 4, when provisioning the distributed computing system 50, a virtual map reduce cluster system may be generated for each user in the distributed computing system 50 using the main distributed configuration manager 40. That is, when a user receives a configuration and work request of a virtual cluster, the main distributed configuration manager 40 installs another distributed configuration manager (hereinafter referred to as "sub-distributed configuration manager") on one virtual machine, This sub-distributed configuration manager can configure and install MapReduce software on all virtual machines in the virtual cluster to form a virtual cluster at the request of the user.

Accordingly, as shown in FIG. 4, for example, when three users request work at the same time, the main distributed configuration manager 40 may configure each of the first to third map reduce clusters 50a, 50b, and 50c configured for each user. Only the sub-distributed configuration managers need to be created in the server, and the sub-distributed configuration managers in each cluster then perform the remaining provisioning procedures for forming their clusters ((50a, 50b, 50c), and thus the main distributed configuration manager (40). Therefore, if many users simultaneously request configuration of the distributed computing system 50, the problem may be caused by overloading the main distributed configuration manager 40 or delayed access to configuration management data in the configuration management repository. And further expand the resources of the cloud computing system 30 in accordance with additional system expansion requirements. Suga is easily scalable.

5A and 5B are flowcharts illustrating a method of provisioning a parallel processing based distributed computing system according to an embodiment of the present invention, and FIGS. 6 to 9 are parallel processing based distributed according to an embodiment of the present invention. A block diagram illustrating a method of provisioning a computing system.

FIG. 6 is a schematic diagram of the cloud computing system 30 shown in FIG. 1, and for convenience of description, the hardware 301, the hypervisor 302, the virtual private network manager 304, and the virtual infrastructure manager API. (305) is omitted. In addition, the message communication processor 60 and the distributed coordinator 70 are further illustrated for explanation in accordance with an embodiment of the present invention.

In one embodiment, the message communication processor 60 includes a message queue 601. The message queue 601 may be a storage device operating in a first-in-first-out (FIFO) manner, and may receive a predetermined request from the virtual machine 51 and output a response accordingly.

The distributed coordinator 70 may include information or variables (e.g., information necessary for configuring each virtual cluster of the distributed computing system, master virtual machines and operations within each virtual cluster) that are shared and used in the overall system resources related to the distributed computing system 50. It stores and manages information necessary for the configuration of node virtual machine. When a plurality of virtual machines are installed in the distributed computing system 50, configuration information or IP information of each virtual machine is shared by a plurality of modules or servers such as a main distributed configuration manager 40 and a virtual infrastructure manager 303. Information to use. The distributed coordinator 70 may limit simultaneous access to this information to ensure data consistency. That is, when executing a predetermined command is executed through the message queue 601 and the distributed coordinator 70, so that it is managed in one distributed coordinator 70 without having to distribute the information to obtain data consistency and synchronization for variable values Can be secured.

In addition, the distributed coordinator 70 may further include a role of storing and changing state information of the distributed computing system 50 through the message queue 601, and thus, may be predetermined from the distributed computing system 50. Each time a request is received, the function of changing the state information of the distributed computing system 50 is performed.

Meanwhile, according to the embodiment, the main distributed configuration manager 40, the distributed computing system 50, the message communication processor 60, and the distributed coordinator 70 may be installed in separate servers to communicate with each other through a network. . In addition, each of the message communication processor 60 and the distributed coordinator 70 may be implemented as one or more servers.

Referring now to FIG. 5A, an embodiment of provisioning a distributed computing system 50, first, in step S501, a virtual infrastructure manager 303 that has received a user's resource requirements from a service server 20 receives a user's request. Schedule and run one or more virtual machines with specifications (see arrows in FIG. 6). In the illustrated embodiment, the virtual infrastructure manager 303 may execute one master virtual machine 51 and one or more work node virtual machines 52a, 52b, 52c. At this time, only the operating system (OS) is installed in the master virtual machine 51 and the work node virtual machines 52a, 52b, and 52c, except that the master virtual machine 51 further includes a message transceiver 513. Are the same as the working node virtual machines 52a, 52b, and 52c.

Thereafter, the master virtual machine 51 should send a request to the main distributed configuration manager 40 to install the master agent. For this purpose, in step S503, the message transceiver 513 of the master virtual machine 51 The installation request of the master agent is transmitted to the message queue 601 of the message communication processor 60 (see FIG. 7).

Then, in step S505, the message communication processor 60 communicates with the distributed coordinator 70 to obtain a master agent installation command and deliver it to the main distributed configuration manager 40.

According to one specific embodiment of this process, first, the message queue 601 forwards the master agent installation request to the distributed coordinator 70, the distributed coordinator 70 based on the installation request, the user's work in the distributed coordinator The state of the virtual map reduce system corresponding to the ID is changed from the "virtual machine_start" state to the "configuration" state, for example. Accordingly, an event handler executed upon switching of state variable values transmits a master agent installation command for automatic configuration to the message queue 601, and the message queue 601 sends the master agent installation command to the main distributed configuration manager 40. The message is transmitted to the transceiver 312.

On the other hand, in an alternative embodiment of the present invention, the message communication processor 60 may not exist. In this case, as an alternative embodiment to the steps S503 and S505, communication between the master virtual machine 51, the distributed coordinator 70, and the main distributed configuration manager 40 may be performed without passing through the message communication processor 60. Can be. That is, in step S503, the master virtual machine 51 sends the master agent installation request directly to the distributed coordinator 70, and the main distributed configuration manager 40 sends the master agent installation command from the distributed coordinator 70. Can be received directly.

Again in the embodiment of Figure 5A, in step S507, the main distributed configuration manager 40 obtains configuration information for installation of the master agent. In the illustrated embodiment, the message transceiver 312 forwards the master agent installation command to the distributed configuration controller 401, which distributes configuration information corresponding to the requested installation command to the configuration information store 402. ) To obtain configuration information for installation of the master agent by receiving from the repository 402.

Next, in step S509, the main distributed configuration manager 40 installs the master agent 511 in the master virtual machine 51. In the illustrated embodiment, the distributed configuration controller 401 of the main distributed configuration manager 40 installs and configures the master agent 511 on the master virtual machine 51. Here, the master agent 511 may be implemented as a module that configures the virtual machine according to a purpose under the control of the main distributed configuration manager 40 while operating in the form of a daemon service in the virtual machine.

In step S511, the master virtual machine 51 transmits the installation request of the sub distributed configuration manager to the main distributed configuration manager 40 (see FIG. 8). In the illustrated embodiment, the master agent 511 of the master virtual machine 51 may deliver the installation request to the distributed configuration controller 401 of the main distributed configuration manager 40.

Thereafter, in step S513 of FIG. 5B, the main distributed configuration manager 40 obtains installation information of the sub distributed configuration manager. For example, the distributed configuration controller 311 of the main distributed configuration manager 40 may obtain configuration information by requesting the configuration information store 402 for configuration information for installation of the sub distributed configuration manager. In one embodiment, in this step S513, the distributed configuration controller 401 may also obtain mapreduce software installation and configuration information from the configuration information store 402 for formation of the map reduce cluster system.

Next, based on the configuration information obtained in step S513, the main distributed configuration manager 40 installs the sub distributed configuration manager 512 in the master virtual machine 51 in step S515. The sub-distributed configuration manager 512 is a module capable of autonomously configuring the virtual map reduce system, and is preferably installed in the master virtual machine 51 to install and configure the map reduce cluster system.

The sub distributed configuration manager 512 is basically the same as the main distributed configuration manager 40. That is, like the main distributed configuration manager 40, the distributed configuration controller 401 and the configuration information store 402 may include a distributed configuration controller (not shown) and a configuration information store (not shown), respectively. However, in the case of the configuration information store, the configuration information store 402 of the main distributed configuration manager 40 stores all the configuration information in the system, whereas the configuration information store of the sub-distributed configuration manager 512 is the master virtual server. Only configuration information necessary to form the virtual cluster to which the machine 51 belongs may be stored. Since the master virtual machine 51 includes the message transceiver 513, the sub distributed configuration manager 512 does not need to have a separate message transceiver.

When the sub distributed configuration manager 512 is installed, the sub distributed configuration manager 512 installs the work node agent 521 in each work node virtual machine 52a, 52b, 52c in step S517. For example, the sub-distributed configuration manager 512 of the master virtual machine 51 uses the information of the work node virtual machines 52a, 52b, and 52c received from the virtual infrastructure manager 303 to each work node virtual machine. The work node agent 521 may be installed.

Here, the work node agent 521 has a configuration similar to that of the master agent 511, and each node virtual machine 52a, 52b, 52c of the job under the control of the main distributed configuration manager 40 or the sub distributed configuration manager 512. It can be implemented as a module to configure the according to the purpose.

The operation node agent 521 is installed in the work node virtual machines 52a, 52b, and 52c by the execution up to step S517, and all the virtual machines 51, 52a, 52b, and 52c are still fully mapped. It is not integrated into the Deuce cluster system. Therefore, in step S519, the master virtual machine 51 transmits the installation request of the MapReduce cluster system to the message communication processor 60 (see Fig. 9).

Then, in step S521, the message communication processor 60 obtains an installation command of the MapReduce cluster system and transfers the command to the master virtual machine 51. In a specific embodiment, when the message queue 601 of the message communication processor 60 receives the installation request of the MapReduce cluster system from the master virtual machine 51 and delivers it to the distributed coordinator 70, the distributed coordinator 70 Based on this installation request, the state variable value of the MapReduce cluster system in the distributed coordinator is changed from, for example, the "configuration" state to the "map reduce_configuration" state. Accordingly, the event handler executed when the state variable value is switched is executed, and the installation command of the map reduce system is transmitted to the message queue 601. The message queue 601 forwards the installation command to the message transceiver 513 of the master virtual machine 51, and the message transceiver 513 forwards this installation command to the sub distributed configuration manager 512.

On the other hand, even in the above steps S519 and S521, the message communication processor 60 may not exist according to an alternative embodiment of the present invention. In this case, in step S519, the master virtual machine 51 may directly transmit the installation request of the map reduce cluster to the distributed coordinator 70. In response thereto, the distributed coordinator 70 may perform the map reduce cluster in step S521. Can be directly sent to the master virtual machine (51).

Again in the embodiment of FIG. 5B, in step S523, the work node agent 521 in each work node virtual machine, in accordance with the instruction of the sub-distributed configuration manager 512, is configured for each work node virtual machine 52a, 52b, 52c. By installing MapReduce software, the MapReduce virtual cluster system is configured according to the request of the user. Accordingly, after the MapReduce software is installed, the state of the MapReduce cluster system of the distributed coordinator 70 is changed from, for example, the "MapReduce_Configuration" state to the "MapReduce_Start" state and waits for the user to execute a job. You can switch.

10 is a flowchart illustrating a method of processing user data in a parallel processing based distributed computing system according to an embodiment of the present invention, and FIG. 11 is a parallel processing based distributed computing system according to an embodiment of the present invention. A block diagram for explaining a method of processing data.

When the MapReduce virtual cluster system is provisioned through the steps described with reference to FIGS. 5A and 5B, the MapReduce system is in a state of waiting for a user to execute a job. Thereafter, in step S1001 of FIG. 10, the user job execution request may be registered in the message queue 601 of the message communication processor 60 to execute the user's map reduce job.

Next, in step S1003, the message communication processor 60 obtains a user work file request command and passes it to the data store 204. In a specific embodiment, if a user job execution request is passed from the message queue 601 to the distributed coordinator 70, the distributed coordinator 70 may display the state of the map reduce system, for example, in the "map reduce_start" state. A command is registered in the message queue 314 requesting to put user work files into the data store 204 by the event handler executed with the MapReduce system state change. The request command registered in the message queue 314 is then transferred to the data store 204, and the user work file is transmitted to the master virtual machine 51 in step S1005. In an embodiment, step S1005 may be performed by the master virtual machine 51 pulling a user work file from the data store 204.

Thereafter, in step S1007, the user work file is transferred to each work node virtual machine 52a, 52b, 52c in the map reduce cluster, and the map reduce job is executed in each work node virtual machine. And in one embodiment, when the user task is executed, the map reduce system state of the distributed coordinator 70 may be changed from the "task_submit" state to the "task_start" state, for example.

As described above, according to an embodiment of the present invention, the MapReduce virtual cluster using the main distributed configuration manager 40 and the sub distributed configuration manager 512 executed in the master virtual machine 51 operated in the cloud computing system. Automated MapReduce software configuration and installation. With this configuration, even if a plurality of users simultaneously request the installation and configuration of the MapReduce system, the service can be easily extended according to the expansion of system resources, and the load of the main distributed configuration manager 40 is increased by the sub distributed configuration manager (on the virtual machine). 512).

According to another embodiment of the present invention, the stability of user task performance by using asynchronous communication between the virtual machine of the MapReduce system and the distributed coordinator 70 and the main distributed configuration manager 40 through the message queue 601 And increased responsiveness.

In addition, according to another embodiment of the present invention, the configuration and execution of the MapReduce system in an event-driven manner is automated by using an event handler that is performed at the state transition and state change of the MapReduce system using the distributed coordinator 70 Therefore, unnecessary delay time and waste of resources can be minimized.

The technology related to the provisioning apparatus and method of the parallel processing-based virtual cluster computing system of the present invention described above may be implemented in hardware or software, or a suitable combination of the two. Thus, for example, program code (ie, instructions) embodied in a tangible medium such as a floppy diskette, CD-ROM, hard drive, or any other computer readable storage medium, all or part of a method of provisioning a virtual cluster computing system. ), And when such program code is loaded on or executed by a machine such as a computer, the machine may be part of the provisioning device of a distributed computing system.

While the present invention has been described with reference to the particular embodiments and drawings, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. This is possible.

10: user terminal 20: service server
30: cloud computing system 302: hypervisor
303: Virtual Infrastructure Manager 40: Main Distributed Configuration Manager
50: distributed computing system 51: master virtual machine
512: Sub-distributed configuration manager 52a, 52b, 52c: work node virtual machine
60: Message Communication Handler 70: Distributed Coordinator

Claims (22)

A method of provisioning a parallel processing based virtual cluster computing system by a cloud computing system,
Executing, by the cloud computing system, working node virtual machines and a master virtual machine according to user-specific system configuration information of the virtual cluster computing system;
Installing, by a main distributed configuration manager, a sub distributed configuration manager on the master virtual machine; And
And installing, by the sub distributed configuration manager, a work node agent on the work node virtual machines, respectively. The method of claim 1,
Provisioning method further comprising the step of installing the master agent on the master virtual machine, the main distributed configuration manager. The method of claim 2, wherein the step of installing the master agent,
Sending, by the master virtual machine, a request for installation of a master agent to the main distributed configuration manager;
Acquiring, by the main distributed configuration manager, configuration information for installation of a master agent;
And the main distributed configuration manager installs the master agent based on the obtained configuration information. The method of claim 3, wherein the step of sending the installation request of the master agent to the main distributed configuration manager,
Sending, by the master virtual machine, a request for installation of the master agent to a distributed coordinator; And
The distributed coordinator sending a master agent installation command to the main distributed configuration manager. The method of claim 2, wherein installing the sub distributed configuration manager comprises:
The master agent requesting the main distributed configuration manager to install the sub distributed configuration manager; And
Acquiring, by the main distributed configuration manager, configuration information for installation of a sub distributed configuration manager;
And the main distributed configuration manager installs the sub distributed configuration manager based on the obtained configuration information. The method of claim 1,
And the work node agents installing software for parallel processing on each work node virtual machine. 7. The method of claim 6, wherein installing the software comprises:
Sending, by the master virtual machine, a virtual cluster system installation request to a distributed coordinator;
The distributed coordinator sending a virtual cluster system installation command to the master virtual machine; And
Based on the installation instructions sent to the master virtual machine, the work node agents installing the software on each work node virtual machine. A method of provisioning a parallel processing based virtual cluster computing system by a cloud computing system,
Executing, by the cloud computing system, first work node virtual machines and first master virtual machine according to system configuration information of a virtual cluster of a first user;
Installing, by a main distributed configuration manager, a first sub distributed configuration manager on the first master virtual machine;
Executing, by the cloud computing system, second work node virtual machines and a second master virtual machine according to system configuration information of a virtual cluster of a second user;
Installing, by the main distributed configuration manager, a second sub distributed configuration manager on the second master virtual machine;
The first and second sub-distributed configuration managers install a work node agent on each of the first work node virtual machines and on each of the second work node virtual machines, and the work node agent performs parallel processing on each work node virtual machine. Provisioning method, characterized in that to install the software for. The method of claim 8,
Installing the first sub-distributed configuration manager comprises installing a first master agent on the first master virtual machine,
And installing the second sub distributed configuration manager comprises installing a second master agent on the second master virtual machine. The method of claim 9, wherein the step of installing the first master agent,
Sending, by the first master virtual machine, a first master agent installation request to a distributed coordinator;
The distributed coordinator sending a first master agent installation command to a main distributed configuration manager; And
Acquiring, by the main distributed configuration manager, configuration information for installing a first master agent;
And the main distributed configuration manager installs a first master agent on the first master virtual machine based on the obtained configuration information. 11. The method of claim 10, wherein installing the first sub distributed configuration manager comprises:
Sending, by the first master virtual machine, the installation request of the first sub distributed configuration manager to the main distributed configuration manager; And
Acquiring, by the main distributed configuration manager, configuration information for installation of a first sub distributed configuration manager;
And the main distributed configuration manager installs a first sub distributed configuration manager on the first master virtual machine based on the obtained configuration information. The method of claim 9, wherein the step of installing the second master agent,
Sending, by the second master virtual machine, a second master agent installation request to a distributed coordinator;
Obtaining, by the distributed coordinator, a second master agent installation command, and transmitting it to a main distributed configuration manager; And
Acquiring, by the main distributed configuration manager, configuration information for installing a second master agent;
And the main distributed configuration manager installs a second master agent on the second master virtual machine based on the obtained configuration information. 13. The method of claim 12, wherein installing the second sub distributed configuration manager comprises:
Sending, by the second master virtual machine, the installation request of the second sub distributed configuration manager to the main distributed configuration manager; And
Acquiring, by the main distributed configuration manager, configuration information for installation of a second sub distributed configuration manager;
And the main distributed configuration manager installs a second sub distributed configuration manager on the second master virtual machine based on the obtained configuration information. 13. The method of claim 10 or 12, wherein the distributed coordinator changes the state of a user-specific virtual cluster in the distributed coordinator when the distributed coordinator receives the first or second master agent installation request. An apparatus for provisioning a virtual cluster based on parallelism,
A virtual infrastructure manager capable of executing a master virtual machine and one or more work node virtual machines according to configuration information of the virtual cluster; And
A first distributed configuration manager capable of creating the virtual cluster;
And the first distributed configuration manager can install a second distributed configuration manager in the master virtual machine according to a request of the master virtual machine. 16. The system of claim 15, wherein the second distributed configuration manager may install a work node agent on each of the one or more work node virtual machines, wherein each work node agent is configured for parallel processing on each work node virtual machine. A device for provisioning a virtual cluster, wherein software can be installed. The apparatus of claim 16, wherein the apparatus for provisioning the virtual cluster comprises:
And a distributed coordinator for storing and managing information necessary for configuring the cluster, the master virtual machine, and the working node virtual machine. 18. The system of claim 17, further comprising the distributed coordinator storing and changing state information of the cluster.
And whenever the distributed coordinator receives a request of the master virtual machine, the distributed coordinator may change state information of the cluster. 18. The apparatus of claim 17, wherein the first distributed configuration manager and the distributed coordinator are implemented on physically different servers. An apparatus for provisioning a virtual cluster based on parallelism,
Run the first master virtual machine and the one or more first work node virtual machines according to the configuration information of the first user's virtual cluster, and according to the configuration information of the second user's virtual cluster, the second master virtual machine and the one or more second A virtual infrastructure manager capable of executing a work node virtual machine; And
A main distributed configuration manager capable of creating the first and second virtual clusters, respectively;
The main distributed configuration manager installs a first sub distributed configuration manager on the first master virtual machine according to the request of the first master virtual machine, and requests the second master virtual machine according to the request of the second master virtual machine. And provisioning a second sub-distributed configuration manager in the virtual cluster. 21. The method of claim 20,
The first sub-distributed configuration manager may install a first work node agent in each of the one or more first work node virtual machines, and the first work node agent is parallel according to configuration information of the virtual cluster of the first user. Install software for processing,
The second sub-distributed configuration manager may install a second work node agent on each of the one or more second work node virtual machines, and the second work node agent may be parallel according to configuration information of the virtual cluster of the second user. A device for provisioning a virtual cluster, characterized in that it can install software for processing. A computer-readable recording medium having recorded thereon a program for causing a computer to execute at least one step of the method for provisioning a parallel processing based virtual cluster computing system according to any one of claims 1 to 7.

Images