CN107797874A

CN107797874A - A kind of resource management-control method based on embedded jetty and spark on yarn frameworks

Info

Publication number: CN107797874A
Application number: CN201710947052.3A
Authority: CN
Inventors: 邹涛; 滕飞; 刘冉
Original assignee: Nanjing Sinovatio Technology LLC
Current assignee: Nanjing Sinovatio Technology LLC
Priority date: 2017-10-12
Filing date: 2017-10-12
Publication date: 2018-03-13
Anticipated expiration: 2037-10-12
Also published as: CN107797874B

Abstract

The invention discloses a kind of resource management-control method based on embedded jetty and spark on yarn frameworks, comprise the following steps：(1) hadoop cluster and spark clusters are built, disposes front and back tomcat servers；(2) user initiates spark application application resource requests to system；(3) backstage tomcat receives request, starts Jetty services by submit methods, loads servlet, generate Resident Process；Initiation message queue, initialization spark context generation spark applications, and most application is submitted in yarn clusters at last；(4) backstage tomcat will start returns to foreground tomcat using result；(5) foreground tomcat sends the Jetty Resident Process that follow-up spark tasks are submitted to background application；(6) background task Resident Process receives and performs task, returns to user's final result.Beneficial effects of the present invention are：User can apply for multiple spark application resources from a yarn resource queue, ensure that maximum resource utilization；Follow-up work under multiple different applications can perform simultaneously.

Description

A kind of resource management-control method based on embedded jetty and spark on yarn frameworks

Technical field

It is especially a kind of to be based on embedded jetty and spark the present invention relates to spark application resource application control fields The resource management-control method of on yarn frameworks.

Background technology

With the development of big data technology, the value of data is highlighted increasingly, and increasing enterprise is absorbed in from magnanimity number Business opportunity is found according to central analysis and excavation, and spark excavates most popular framework as current data, by more and more enterprises Favor.Spark provides a kind of resource management and control based on yarn, i.e., spark tasks is submitted into yarn clusters, yarn frameworks The concept of the resource queue of offer realizes that yarn cluster resources are relatively isolated.

As complete data mining platform, the foreground of user's operation and the backstage of processing request will possess.And user Whole process is asked to pay close attention to following problem to backstage application spark application resources by operating foreground：First, backstage Need uniformly to receive and return the framework of foreground message；Secondly, while there are multiple application resource applications to ask, while perform multiple Spark tasks, and resource isolation is realized in each spark applications；Again, it must have from the background and resided corresponding to spark applications Process, ensure that task dynamic is submitted；Finally, multiple spark applications can be submitted in same yarn resource queues, realize team Row resource makes full use of.

The content of the invention

The technical problems to be solved by the invention are, there is provided one kind is based on embedded jetty and spark on yarn frames It the resource management-control method of frame, can realize that backstage performs the spark application resources application request initiated on foreground, and realize resource pipe Control.

In order to solve the above technical problems, the present invention provide it is a kind of based on embedded jetty and spark on yarn frameworks Resource management-control method, comprises the following steps：

(1) hadoop cluster and spark clusters are built, disposes front and back tomcat servers；

(2) user initiates spark application application resource requests to system；

(3) backstage tomcat receives request, starts Jetty services by submit methods, loads servlet, generation is normal In process；Initiation message queue, initialization spark context generation spark applications, and most application is submitted to yarn clusters at last In；

(4) backstage tomcat will start returns to foreground tomcat using result；

(5) foreground tomcat sends the Jetty Resident Process that follow-up spark tasks are submitted to background application；

(6) background task Resident Process receives and performs task, returns to user's final result.

Preferably, in step (3), backstage tomcat starts jetty services, ensures each jetty Resident Process corresponding one Individual spark applications, the task requests that spark applications are persistently sent for subsequent user are prepared.

Preferably, in step (3), in Jetty service activation process, comprising loading servlet, front and back interaction is started Message queue, spark contexts are initialized, to yarn cluster application resources, controlled, protected by the realization of spring case mechanisms The succession of card event and the uniformity of environmental variance.

Preferably, in step (3), Jetty services are started by submit patterns, ensure that multiple spark application can be with It is submitted in a yarn resource queue.

Preferably, in step (5), foreground tomcat is follow-up spark tasks by different to row name and task mark Knowledge is sent in different message queues, and the backstage permanent processes of different Jetty receive the message in message queue, and then perform Different spark tasks.

Beneficial effects of the present invention are：User can by multiple spark application resources from a yarn resource queue Shen Please, it ensure that maximum resource utilization；Follow-up work under multiple different applications can perform simultaneously.

Brief description of the drawings

Fig. 1 is the structural framing schematic diagram of the present invention.

Fig. 2 is the method flow schematic diagram of the present invention.

Embodiment

Fig. 1 is the knot for the resource management and control based on embedded jetty and spark on yarn frameworks implemented according to the present invention Structure schematic diagram.Wherein include backstage tomcat modules, jetty engine modules, message queue module etc..

Backstage tomcat modules are mainly responsible for the http request for the spark application resource applications that receiving front-end comes, and start Jetty is serviced, record Jetty attribute informations to mysql databases, mistake rollback and return request results.

Jetty engine modules are mainly responsible for starting jetty Resident Process, load servlet, front and back interaction message team Row, spark contexts are initialized, submit task to yarn resource queues, and wrong rollback mechanism.

Message queue module is mainly responsible for follow-up front and back message communicating.User has applied for that the spark of resource should to some With new task requests are submitted, corresponding jetty resident service processes, corresponding jetty are transmitted directly to by message queue Spark tasks corresponding to response and processing.

Fig. 2 is the stream for the resource management and control based on embedded Jetty and spark on yarn frameworks implemented according to the present invention Journey schematic diagram.Specific step is described below：

(1) user initiates application resource request to backstage tomcat；

(2) backstage tomcat receives request, starts jetty services by submit methods,

(3) jetty calls start methods, loads servlet, generates Resident Process；

(4) spring frameworks built in jetty drive case mechanism initiation message queue successively, initialize spark contexts Spark applications are generated, and most application is submitted in yarn clusters at last.

(5) backstage tomcat will start returns to foreground using result, and errors excepted rollback simultaneously returns to error result；

By above-mentioned basic procedure, user can by multiple spark application resources from a yarn resource queue Shen Please, maximum resource utilization is ensure that, and the follow-up work under multiple different applications can simultaneously be performed and is independent of each other.

The present invention is based on spark on yarn Frame Designs, is realized by adding embedded Jetty services by foreground The request of spark application resources is converted to the resource allocation of yarn frameworks identification.

By using such scheme, user can easily carry out spark application resources point according to different business complexity Match somebody with somebody, and do not interfere with each other more using the operation of multitask notice, suitable for actual production requirement.

Although the present invention is illustrated and described with regard to preferred embodiment, it is understood by those skilled in the art that Without departing from scope defined by the claims of the present invention, variations and modifications can be carried out to the present invention.

Claims

1. a kind of resource management-control method based on embedded jetty and spark on yarn frameworks, it is characterised in that including such as Lower step：

(2) user initiates spark application application resource requests to system；

(3) backstage tomcat receive request, by submit methods start Jetty service, load servlet, generation reside into Journey；Initiation message queue, initialization spark context generation spark applications, and most application is submitted in yarn clusters at last；

(4) backstage tomcat will start returns to foreground tomcat using result；

2. the resource management-control method as claimed in claim 1 based on embedded jetty and spark on yarn frameworks, it is special Sign is, in step (3), backstage tomcat starts jetty services, ensures that the corresponding spark of each jetty Resident Process should With the task requests that spark applications are persistently sent for subsequent user are prepared.

3. the resource management-control method as claimed in claim 1 based on embedded jetty and spark on yarn frameworks, it is special Sign is, in step (3), in Jetty service activation process, comprising loading servlet, starts the message team that front and back interacts Row, spark contexts are initialized, to yarn cluster application resources, controlled by the realization of spring case mechanisms, ensure event Succession and environmental variance uniformity.

4. the resource management-control method as claimed in claim 1 based on embedded jetty and spark on yarn frameworks, it is special Sign is, in step (3), starts Jetty services by submit patterns, ensure that multiple spark applications can be submitted to one In individual yarn resource queues.

5. the resource management-control method as claimed in claim 1 based on embedded jetty and spark on yarn frameworks, it is special Sign is, in step (5), foreground tomcat is sent to follow-up spark tasks to row name and task identification by different In different message queues, the backstage permanent processes of different Jetty receive the message in message queue, and then perform different Spark tasks.