CN110659131B - Task processing method, electronic device, computer equipment and storage medium - Google Patents
Task processing method, electronic device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110659131B CN110659131B CN201910754313.9A CN201910754313A CN110659131B CN 110659131 B CN110659131 B CN 110659131B CN 201910754313 A CN201910754313 A CN 201910754313A CN 110659131 B CN110659131 B CN 110659131B
- Authority
- CN
- China
- Prior art keywords
- task
- application program
- processed
- target application
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 claims abstract description 140
- 238000012545 processing Methods 0.000 claims abstract description 38
- 238000004590 computer program Methods 0.000 claims description 12
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a task processing method, an electronic device, computer equipment and a storage medium, wherein parameter information of a resource package to be called is set in an application program; determining a target application program corresponding to a task to be processed according to a request message carrying the task to be processed, which is sent by client equipment to server equipment, and starting a driving process for driving the target application program in the server equipment; and the driving process determines a resource package required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device. According to the method and the device for processing the task, the driving process is started in the server side equipment, and the server side equipment calls the corresponding resource package from the resource server through the calling interface to process the task to be processed, so that the problem of local resource occupation during processing of the task to be processed can be solved, and the operation efficiency of task processing is effectively improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a task processing method, an electronic device, a computer device, and a storage medium.
Background
Spark is a distributed computing system based on memory, and the system is stable in operation, exquisite in design and concise in grammar, and is a big data processing system preferred by many companies.
Currently, a Spark-based task processing method is that a client device submits a task to a server device through a main node, the server device starts a working node to be responsible for resource management of a current node, the working node informs the main node of current CPU, memory and other information at regular time, and is responsible for creating an executor process, and the client device processes the task through applying for the executor process. The client device starts a driving process, the driving process is responsible for specific running and tracking of tasks, and the server device returns running results to the client device. Because the main node and the working node are in different devices, and the driving process needs to communicate with the executor process, the task processing has the problems of low operation efficiency, large occupied local resources and the like.
Disclosure of Invention
In view of this, the present invention provides a task processing method, an electronic device, a computer device, and a storage medium, in which a server device processes a task to be processed, so that the problem of local resource occupation can be solved, and the operation efficiency of task processing can be effectively improved.
First, in order to achieve the above object, the present invention provides a task processing method, which includes the steps of:
setting parameter information of a resource package to be called in an application program, wherein the application program is used for processing a task to be processed; and
Determining a target application program corresponding to a task to be processed according to a request message carrying the task to be processed, which is sent by client equipment to server equipment, and starting a driving process for driving the target application program in the server equipment;
and the driving process determines a resource package required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device.
Further, the task to be processed is a Spark task, and the step of sending, according to the client device, the request message carrying the task to be processed to the server device further includes:
generating a parameter setting process for processing Spark tasks in an application program of the client device; and
And sending a request message carrying the Spark task to be processed to the server device through the parameter setting process.
Further, before the step of determining the resource package required to be invoked by the target application program through the parameter information of the target application program, the driving process further comprises the following steps of
Determining the positions of a resource manager and a driving process required to be used by the target application program through a parameter setting process, and applying resources to a server device to start an executor process;
and when the success of the application of the executor process is monitored, ending the parameter setting process.
Further, the driving process is further used for creating a Spark environment, and the Spark environment is responsible for communicating with a cluster manager of the server device and performing resource application, task allocation and monitoring.
Further, before the step of determining the target application program corresponding to the task to be processed, the method further includes:
and modifying a corresponding configuration file in an application program of the task to be processed so as to send a memory resource request instruction to increase the memory resource space allocated by the server-side equipment for processing the task to be processed.
Further, after the step of determining the target application program corresponding to the task to be processed, the method further includes:
starting a main application program of a target application program in a container distributed for the target application program by a resource manager of a server side device;
registering with a resource manager of the server device through the host application to view the running state of the application at the client device through the resource manager.
Further, the method further comprises:
sending request information for requesting to process the execution result of the task to be processed to the server device;
and receiving an execution result returned by the server-side equipment.
In addition, to achieve the above object, the present invention also provides an electronic device, including:
the setting module is suitable for setting parameter information of a resource package to be called in an application program, and the application program is used for processing a task to be processed; and
The sending module is suitable for determining a target application program corresponding to the task to be processed according to a request message carrying the task to be processed, which is sent by the client device to the server device, and starting a driving process for driving the target application program in the server device;
and the driving process determines a resource package required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device.
Further, the task to be processed is a Spark task, and the sending module is further adapted to generate a parameter setting process for processing the Spark task in an application program of the client device; and
And sending a request message carrying the Spark task to be processed to the server device through the parameter setting process.
Further, the sending module is further adapted to determine, through a parameter setting process, a location of a resource manager and a driving process that are required to be used by the target application program, and apply resources to a server device to start an executor process;
and when the success of the application of the executor process is monitored, ending the parameter setting process.
Further, the driving process is further used for creating a Spark environment, and the Spark environment is responsible for communicating with a cluster manager of the server device and performing resource application, task allocation and monitoring.
Further, the setting module is further adapted to modify the memory resource request instruction in the application program of the task to be processed, so as to increase the memory resource space allocated by the server device for processing the task to be processed.
Further, the sending module is further adapted to start a main application of the target application in a container allocated by a resource manager of the server device for the target application;
registering with a resource manager of the server device through the host application to view the running state of the application at the client device through the resource manager.
Further, the sending module is further adapted to send request information for requesting to process the execution result of the task to be processed to the server device; and
And receiving an execution result returned by the server-side equipment.
To achieve the above object, the present invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.
Compared with the prior art, the task processing method, the electronic device, the computer equipment and the storage medium provided by the invention have the advantages that the resource package and the calling interface are arranged in the resource server, the driving process is started in the server equipment, and the server equipment calls the corresponding resource package through the calling interface to process the task to be processed, so that the problem of local resource occupation during processing the task to be processed can be solved, and the operation efficiency of task processing is effectively improved.
Drawings
FIG. 1 is a flow chart of a task processing method according to an exemplary embodiment of the invention;
FIG. 2 is a flow chart of a task processing method according to an exemplary embodiment of the invention;
FIG. 3 is a flow chart of a task processing method according to an exemplary embodiment of the present invention;
FIG. 4 is a flow chart of a task processing method according to an exemplary embodiment of the invention;
FIG. 5 is a flow chart of a task processing method according to an exemplary embodiment of the invention;
FIG. 6 is a flow chart of a task processing method according to an exemplary embodiment of the invention;
FIG. 7 is a program module schematic diagram of an electronic device according to an exemplary embodiment of the present invention;
fig. 8 is a schematic diagram of a hardware architecture of an electronic device according to an exemplary embodiment of the present invention.
Reference numerals:
| electronic device | 20 |
| Memory device | 21 |
| Processor and method for controlling the same | 22 |
| Network interface | 23 |
| Memory | 24 |
| Setting module | 201 |
| Transmitting module | 202 |
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the description of "first", "second", etc. in this disclosure is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
In the prior art, when a client device submits a task to be processed to a server device, a request message carrying the task to be processed is submitted to a resource management system, and when the server device processes the task to be processed, the server device needs to communicate with the client device to obtain a resource package and the like required by processing the task to be processed, so that when the task to be processed is processed, a corresponding process in the client device must exist all the time until the task to be processed is finished in operation, and thus the problems of low operation efficiency, large local resource occupation and the like in task processing are caused. In view of the above, the present invention provides a task processing method, an electronic device, a computer apparatus, and a storage medium.
Referring to fig. 1, a flow chart of a task processing method according to an exemplary embodiment of the present invention is shown, and the method includes the following steps:
step S110, setting parameter information of a resource package to be called in an application program, wherein the application program is used for processing a task to be processed; and
Step S120, determining a target application program corresponding to a task to be processed according to a request message carrying the task to be processed, which is sent by a client device to a server device, and starting a driving process for driving the target application program in the server device;
and step S130, the driving process determines a resource package required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device.
Setting a resource package and a calling interface in a resource server, namely storing the resource package needed by the server equipment when processing the task to be processed in the resource server, for example, setting the resource package and an HTTP (Hyper Text Transfer Protocol ) interface capable of calling the resource package in the resource server such as a hundred-degree network disk. The resource package and the calling interface set in the resource server can be free, so that the service end equipment and the like can call the resource package at will. Setting parameter information of a resource package to be called in an application program of a task to be processed, so that when the server side equipment processes the task to be processed and needs to call the resource package, a path for acquiring the resource package can be obtained through analyzing the parameter information, and the resource package is called from a resource server through the path to process the task to be processed.
In one embodiment, the task to be processed is a Spark task, which is a memory-based distributed computing system, and for ease of understanding, the basic concept in Spark will be briefly described below.
(1) Application (Application): the application program written by the user and submitted to the server device for processing Spark tasks comprises a Driver code and an Executor code.
(2) Drive (Driver): a main () function is represented for creating and closing a Spark environment (Spark Context). The Spark environment is responsible for communication with a Cluster Manager (Cluster Manager) of the server device, and performs resource application, task allocation, monitoring and the like. And closing the Spark environment process after the application program of the Spark task is executed.
(3) Actuator (Executor): an Application program (Application) runs on a process on a work node (Worker node), and an Executor process (Executor process) is responsible for running certain work units (tasks) and for storing data in a memory or a disk. In the resource management system mode (Spark on Yarn mode), the actuator is named coarse-grained actuator backend (Coarse Grained Executor Backend), a coarse-grained actuator backend process has and has only one actuator process object, which holds a thread pool, the actuator process is responsible for wrapping the work units into Task management tools (Task Runner), and extracting an idle thread running work unit from the thread pool, so that the number of parallel running work units per coarse-grained actuator backend depends on the number of CPUs assigned to it.
(4) Cluster Manager (Cluster Manager): refers to an external service that obtains resources on a cluster.
(5) Working node (Worker): any node in the cluster that can run application code.
(6) Work unit (Task): the work units sent to a certain executor are basic units for running an application program, a plurality of work units form a work single station (Stage), and the scheduling and management of the work units are responsible for a Task debugging program (Task Schedule).
And sending a request message carrying the task to be processed to the server-side equipment through the client-side equipment, wherein the content in the request message carrying the task to be processed comprises a main application program (Application Master) of the task to be processed, a command for starting the main application program and the like.
As shown in fig. 2, in an embodiment of the present invention, the step of determining the target application program corresponding to the task to be processed may include the following steps:
step S201, a main application program of a target application program is started in a container allocated to the target application program by a resource manager of a server side device;
in step S202, the host application registers with the resource manager of the server device, so as to view the running state of the application at the client device through the resource manager.
After receiving the request, the Resource Manager (Resource Manager) of the server device selects a Node Manager (Node Manager) in the cluster, allocates a Container (Container) for the application program, and requests the Container to start a main application program of the application program in the Resource, wherein the main application program performs initialization of Spark environment and the like.
The host application registers with the resource manager so that a user can view the running state of the application directly through the resource manager, and the resource manager can apply for resources for various tasks in a polling manner through the RPC (Remote Procedure Call ) protocol and monitor their running state until the end of the run.
After the main application program applies for the resource, the main application program communicates with a corresponding Node Manager (Node Manager) and requires the Node Manager to start the coarse-grained executor back end in the obtained resource, and after the coarse-grained executor back end is started, the main application program registers with the Spark environment in the main application program and applies for a working unit. When the Spark environment is initialized in the Spark application program, the rear end of the coarse-grained scheduler is matched with a cluster scheduler to schedule tasks, wherein the cluster scheduler is a simple package for a task scheduler (Task Scheduler Impl), and waiting logic and the like for an executor are added.
After receiving a request message carrying a task to be processed sent by a client device, the server device starts a driving process according to a driver in a received main application program, analyzes parameter information in a target application program corresponding to the task to be processed, determines a resource package to be called according to the parameter information of the target application program and the parameter information of the resource package, and calls the corresponding resource package through a calling interface, so that the task to be processed can be processed in the server device.
As shown in fig. 3, in an embodiment of the present invention, the task to be processed is a Spark task, and the step of sending, according to a request message carrying the task to be processed to a server device by a client device may include the following steps:
step S301, generating a parameter setting process for processing Spark task in an application program of a client device; and
Step S302, a request message carrying a Spark task to be processed is sent to a server device through the parameter setting process.
When an application program of a task to be processed is submitted to a server device, a Spark environment object is initialized, the Spark environment is responsible for coordinating the running of the application program on a cluster, and a procedure demonstration (Demo) submitted by the program is packaged. And generating a parameter setting (Spark-sub) process in the client device, wherein the parameter setting process is used for submitting the task to the Spark cluster, and sending a request message for processing the task to be processed to the server device by loading the environment variable to Submit an application program of the task to be processed.
The specific process of the parameter setting process is as follows:
1. determining an entry for an application (e.g., org. Apache. Spark. Samples. Spark pi);
2. determining the master node address, which is the URL address of the master in the cluster (e.g., spark://192.168.1.1: 1234);
3. determining a deployment mode;
4. determining a spark configuration form, a key-value pair form;
5. determining an application path;
6. parameters required by the user application are determined.
As shown in fig. 4, in an embodiment of the present invention, before the step of determining, by the driver process according to the parameter information of the target application, a resource package to be invoked by the target application, the method further includes:
step S401, determining the positions of a resource manager and a driving process required to be used by the target application program through a parameter setting process, and applying resources to a server device to start an executor process;
step S402, ending the parameter setting process when the success of the application of the executor process is monitored.
The client device submits the application program of the task to be processed through the parameter setting process, the parameter setting process determines which resource manager to use according to the user code and the configuration, and the driver is started at a proper position. The parameter setting process is used to set how many executor processes to perform the task to be processed in total. When the driver applies for resources to the cluster manager, the cluster manager sets each working node of the cluster according to the setting of the application program of the task to be processed, and starts the corresponding number of executor processes. The application program for submitting the task to be processed comprises an executor code, and the executor process is started by applying resources to the server equipment.
Generating a parameter setting process at the client device, sending a request message carrying a task to be processed to the server device through the parameter setting process, applying for an executor process, and starting a master node and a working node in the nodes of the cluster by the server device, wherein the master node informs the working node to start a driving process. After the server side equipment receives the request message, the parameter setting process is ended, so that the resources of the client side equipment are saved.
As shown in fig. 5, in an embodiment of the present invention, before the step of determining the target application program corresponding to the task to be processed, the method may further include the following steps:
in step S501, the corresponding configuration file is modified in the application program of the task to be processed, so as to send a memory resource request instruction to increase the memory resource space allocated by the server device for processing the task to be processed.
The cluster manager sets each working node of the cluster as far as possible according to the setting of the application program of the task to be processed, and starts the corresponding number of executor processes. The memory resource parameter is used for setting the memory of each executor process.
The memory resource parameter is used for setting the number of CPU cores of each executor process. This parameter determines the ability of each executor process to execute a unit of work (Task) thread in parallel. Because each CPU core can only execute one work unit thread at the same time, the more CPU cores of each executor process, the more quickly all work unit threads allocated to itself can be executed. The memory resource request instruction can be increased by a plurality of times by modifying the configuration file in the application program of the task to be processed so as to increase the memory resource space allocated for processing the task to be processed, thereby increasing the speed for processing the task to be processed.
As shown in fig. 6, in an embodiment of the present invention, the task processing method may further include the following steps:
step S601, request information for requesting to process the execution result of a task to be processed is sent to a server device;
and
Step S602, receiving an execution result returned by the server device.
The result of the application program of the task to be processed is output in Stdout (Standard output) of the node running the driving process, and since the driving process in the present invention runs in the server device, the result of the application program cannot be directly displayed on the screen of the client device. In order to facilitate the user to check the result of the application program, in one embodiment of the present invention, by sending request information for requesting to process the execution result of the task to be processed to the server device, the server device returns the execution result to the client device sending the request information after receiving the request information, so that the user can check the result of the application program in the client device conveniently.
In an embodiment of the present invention, a Spark environment in a main application program allocates a working unit to a coarse-granularity executor back-end for execution, and the coarse-granularity executor back-end runs the working unit and reports the running state and progress to the main application program, so that the main application program can master the running state of each task at any time, and can restart the task when the task fails.
The directed acyclic graph Scheduler (Directed Acyclic Graph Scheduler, DAG Scheduler) divides tasks to be processed: dividing a task to be processed into a work single-station task, dividing the work single-station task into Pipeline (Pipeline) tasks, packaging all operations in the Pipeline tasks into work unit tasks, and sending the work unit tasks to a work unit thread in an executor process for execution. And when the executor process monitors that the driving process is finished before all the working unit threads are executed, namely the task to be processed is not processed, restarting the driving process, and processing the task to be processed again.
After the application program of the task to be processed is operated, the main application program applies for logging off and closing the main application program to the resource manager so as to reduce the occupation of resources in the client device and the server device.
According to the method and the device for processing the task, the resource package and the calling interface are arranged in the resource server, the driving process is started in the server device, and the server device calls the corresponding resource package through the calling interface to process the task to be processed, so that the problem of occupation of local resources when the task to be processed is processed can be solved, and the operation efficiency of task processing is effectively improved.
The invention further provides an electronic device. Referring to fig. 7, a schematic diagram of a program module of the electronic device 20 according to an exemplary embodiment of the invention is shown.
The electronic device 20 includes:
the setting module 201 is adapted to set parameter information of a resource package to be called in an application program, wherein the application program is used for processing a task to be processed; and
The sending module 202 is adapted to determine a target application program corresponding to a task to be processed according to a request message carrying the task to be processed sent by the client device to the server device, and start a driving process for driving the target application program in the server device;
and the driving process determines a resource package required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device.
Further, the task to be processed is a Spark task, and the sending module 202 is further adapted to generate a parameter setting process for processing the Spark task in an application program of the client device; and
And sending a request message carrying the Spark task to be processed to the server device through the parameter setting process.
Further, the sending module 202 is further adapted to determine, through a parameter setting process, a location of a resource manager and a driving process that need to be used by the target application program, and apply resources to a server device to start an executor process;
and when the success of the application of the executor process is monitored, ending the parameter setting process.
Further, the driving process is further used for creating a Spark environment, and the Spark environment is responsible for communicating with a cluster manager of the server device and performing resource application, task allocation and monitoring.
Further, the setting module 201 is further adapted to modify the memory resource request instruction in the application program of the task to be processed, so as to increase the memory resource space allocated by the server device for processing the task to be processed.
Further, the sending module 202 is further adapted to start a main application of the target application in a container allocated by a resource manager of the server device for the target application;
registering with a resource manager of the server device through the host application to view the running state of the application at the client device through the resource manager.
Further, the sending module 202 is further adapted to send request information for requesting to process the execution result of the task to be processed to the server device; and
And receiving an execution result returned by the server-side equipment.
The method and the device start the driving process in the server equipment, and the server equipment calls the corresponding resource package through the calling interface to process the task to be processed, so that the problem of local resource occupation during processing the task to be processed can be solved, and the operation efficiency of task processing is effectively improved.
To achieve the above object, as shown in fig. 8, the present invention further provides a computer device 20, including a memory 21, a processor 22, and a computer program stored on the memory 21 and executable on the processor 22, wherein the processor 22 implements the steps of the above method when executing the computer program. The computer program may be stored in the memory 24.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.
The invention also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server or a cabinet server (comprising independent servers or a server cluster formed by a plurality of servers) and the like which can execute programs. The computer device of the present embodiment includes at least, but is not limited to: memory, processors, etc. that may be communicatively coupled to each other via a system bus.
The present embodiment also provides a computer-readable storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor, performs the corresponding functions. The computer readable storage medium of the present embodiment is used to store the electronic device 20, and when executed by the processor 22, implements the task processing method of the present invention.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (9)
1. A method of task processing, the method comprising the steps of:
setting parameter information of a resource package to be called in an application program, wherein the application program is used for processing a task to be processed; and
Determining a target application program corresponding to a task to be processed according to a request message carrying the task to be processed, which is sent by client equipment to server equipment, and starting a driving process for driving the target application program in the server equipment;
determining a resource package required to be called by the target application program through the parameter information of the target application program, and calling a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device;
the task to be processed is a Spark task, and the step of sending a request message carrying the task to be processed to the server device according to the client device further comprises:
generating a parameter setting process for processing Spark tasks in an application program of the client device; and
And sending a request message carrying the Spark task to be processed to the server device through the parameter setting process.
2. The task processing method according to claim 1, wherein before the step of determining a resource package required to be invoked by the target application by the driving process through the target application parameter information, further comprising
Determining the positions of a resource manager and a driving process required to be used by the target application program through a parameter setting process, and applying resources to a server device to start an executor process;
and when the success of the application of the executor process is monitored, ending the parameter setting process.
3. The method of claim 1, wherein the driving process is further configured to create a Spark environment, and the Spark environment is responsible for communicating with a cluster manager of the server device, and performing resource application, task allocation and monitoring.
4. The task processing method according to claim 1, further comprising, before the step of determining a target application corresponding to the task to be processed:
and modifying a corresponding configuration file in an application program of the task to be processed so as to send a memory resource request instruction to increase the memory resource space allocated by the server-side equipment for processing the task to be processed.
5. The task processing method according to claim 1, further comprising, after the step of determining the target application corresponding to the task to be processed:
starting a main application program of a target application program in a container distributed for the target application program by a resource manager of a server side device;
registering with a resource manager of the server device through the host application to view the running state of the application at the client device through the resource manager.
6. The task processing method according to claim 2, characterized in that the method further comprises:
sending request information for requesting to process the execution result of the task to be processed to the server device; and
And receiving an execution result returned by the server-side equipment.
7. An electronic device, comprising:
the setting module is suitable for setting parameter information of a resource package to be called in an application program, and the application program is used for processing a task to be processed; and
The sending module is suitable for determining a target application program corresponding to the task to be processed according to a request message carrying the task to be processed, which is sent by the client device to the server device, and starting a driving process for driving the target application program in the server device;
the driving process determines a resource package required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource package through a calling interface so that the target application program processes the task to be processed in the server device;
the task to be processed is a Spark task, and the sending module is further adapted to generate a parameter setting process for processing the Spark task in an application program of the client device; and sending a request message carrying the Spark task to be processed to the server device through the parameter setting process.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the task processing method according to any of claims 1 to 6 when the computer program is executed by the processor.
9. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program, when executed by a processor, implements the steps of the task processing method as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910754313.9A CN110659131B (en) | 2019-08-15 | 2019-08-15 | Task processing method, electronic device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910754313.9A CN110659131B (en) | 2019-08-15 | 2019-08-15 | Task processing method, electronic device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110659131A CN110659131A (en) | 2020-01-07 |
| CN110659131B true CN110659131B (en) | 2024-04-02 |
Family
ID=69037586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910754313.9A Active CN110659131B (en) | 2019-08-15 | 2019-08-15 | Task processing method, electronic device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110659131B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000949B (en) * | 2020-08-26 | 2023-06-16 | 中国联合网络通信集团有限公司 | Program package calling method, system, terminal device and computer readable storage medium |
| CN113778658A (en) * | 2020-09-29 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Task allocation method and device, electronic equipment and storage medium |
| CN113760262A (en) * | 2021-01-21 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Task processing method, device, computer system and computer readable storage medium |
| CN112783978A (en) * | 2021-01-28 | 2021-05-11 | 北京东方优播网络科技有限公司 | Data transmission method, device, computer system and storage medium |
| CN114237752B (en) * | 2021-12-20 | 2023-09-26 | 百度在线网络技术（北京）有限公司 | Push page display method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5301326A (en) * | 1991-09-24 | 1994-04-05 | Microsoft Corporation | Method and system for controlling the execution of an application program |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN108710529A (en) * | 2018-04-28 | 2018-10-26 | 四川斐讯信息技术有限公司 | A kind of remote task processing method, system and wireless router |
| CN109936593A (en) * | 2017-12-15 | 2019-06-25 | 网宿科技股份有限公司 | A kind of method and system of message distribution |
| CN110069335A (en) * | 2019-05-07 | 2019-07-30 | 江苏满运软件科技有限公司 | Task processing system, method, computer equipment and storage medium |
-
2019
- 2019-08-15 CN CN201910754313.9A patent/CN110659131B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5301326A (en) * | 1991-09-24 | 1994-04-05 | Microsoft Corporation | Method and system for controlling the execution of an application program |
| CN109936593A (en) * | 2017-12-15 | 2019-06-25 | 网宿科技股份有限公司 | A kind of method and system of message distribution |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN108710529A (en) * | 2018-04-28 | 2018-10-26 | 四川斐讯信息技术有限公司 | A kind of remote task processing method, system and wireless router |
| CN110069335A (en) * | 2019-05-07 | 2019-07-30 | 江苏满运软件科技有限公司 | Task processing system, method, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110659131A (en) | 2020-01-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110659131B (en) | Task processing method, electronic device, computer equipment and storage medium | |
| CN107766101B (en) | Method, device and equipment for processing App starting event | |
| CA3000422C (en) | Workflow service using state transfer | |
| CN109451051B (en) | Service request processing method and device, electronic equipment and storage medium | |
| US20190155660A1 (en) | Scheduling system for computational work on heterogeneous hardware | |
| CN108874549B (en) | Resource multiplexing method, device, terminal and computer readable storage medium | |
| CN106897299B (en) | Database access method and device | |
| CN110162397B (en) | Resource allocation method, device and system | |
| CN111079048B (en) | Page loading method and device | |
| WO2019228346A1 (en) | Method and apparatus for executing task by scheduling device | |
| CN113849312A (en) | Data processing task allocation method and device, electronic equipment and storage medium | |
| CN108900627B (en) | Network request method, terminal device and storage medium | |
| CN110659104B (en) | Service monitoring method and related equipment | |
| CN112491940B (en) | Request forwarding method and device of proxy server, storage medium and electronic equipment | |
| CN110381143B (en) | Job submission execution method, device, equipment and computer storage medium | |
| US10073689B2 (en) | Managing application lifecycles within a federation of distributed software applications | |
| CN114610446B (en) | Method, device and system for automatically injecting probe | |
| CN108062224B (en) | Data reading and writing method and device based on file handle and computing equipment | |
| CN111190725B (en) | Task processing method, device, storage medium and server | |
| CN109634721B (en) | Method and related device for starting communication between virtual machine and host | |
| CN114385351A (en) | Cloud management platform load balancing performance optimization method, device, equipment and medium | |
| CN114301970A (en) | Service calling method and device, electronic equipment and storage medium | |
| CN113656161A (en) | Method, device and system for scheduling pressure generation service in performance test | |
| CN111352710A (en) | Process management method and device, computing equipment and storage medium | |
| US10303531B2 (en) | Console application through web service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |