CN110659131A - Task processing method, electronic device, computer device, and storage medium - Google Patents
Task processing method, electronic device, computer device, and storage medium Download PDFInfo
- Publication number
- CN110659131A CN110659131A CN201910754313.9A CN201910754313A CN110659131A CN 110659131 A CN110659131 A CN 110659131A CN 201910754313 A CN201910754313 A CN 201910754313A CN 110659131 A CN110659131 A CN 110659131A
- Authority
- CN
- China
- Prior art keywords
- task
- application program
- processed
- target application
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 141
- 238000012545 processing Methods 0.000 claims abstract description 49
- 238000004590 computer program Methods 0.000 claims description 12
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Abstract
The invention discloses a task processing method, an electronic device, computer equipment and a storage medium, wherein parameter information of a resource package to be called is set in an application program; determining a target application program corresponding to the task to be processed according to a request message carrying the task to be processed and sent to the server device by the client device, and starting a driving process for driving the target application program in the server device; and the driving process determines a resource packet required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource packet through a calling interface so that the target application program processes the task to be processed in the server-side equipment. According to the invention, the driving process is started in the server-side equipment, and the server-side equipment calls the corresponding resource packet from the resource server through the calling interface to process the task to be processed, so that the problem of local resource occupation during processing the task to be processed can be solved, and the running efficiency of task processing is effectively improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a task processing method, an electronic device, a computer device, and a storage medium.
Background
Spark is a distributed computing system based on a memory, the system has stable operation, delicate design and simple grammar, and is a preferred big data processing system of many companies.
At present, a task processing method based on Spark is that a client device submits a task to a server device through a master node, the server device starts a work node to take charge of resource management of a current node, the work node regularly informs the current information such as a CPU, a memory and the like to the master node and takes charge of creating an actuator process, and the client device processes the task by applying for the actuator process. The client device starts a driving process, the driving process is responsible for specific running and tracking of tasks, and the server device returns a running result to the client device. The main node and the working node are in different devices, and the driving process needs to communicate with the executor process, so that the problems of low running efficiency, large occupied local resources and the like in task processing are caused.
Disclosure of Invention
In view of this, the present invention provides a task processing method, an electronic apparatus, a computer device, and a storage medium, where a server device processes a task to be processed, so that the problem of local resource occupation can be solved, and the operation efficiency of task processing can be effectively improved.
First, in order to achieve the above object, the present invention provides a task processing method, including:
setting parameter information of a resource package to be called in an application program, wherein the application program is used for processing a task to be processed; and
determining a target application program corresponding to a task to be processed according to a request message carrying the task to be processed and sent to a server device by a client device, and starting a driving process for driving the target application program in the server device;
and the driving process determines a resource packet required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource packet through a calling interface so that the target application program processes the task to be processed in the server-side equipment.
Further, the task to be processed is a spare task, and the step of sending a request message carrying the task to be processed to the server device according to the client device further includes:
generating a parameter setting process for processing a Spark task in an application program of the client device; and
and sending a request message carrying the spare task to be processed to the server equipment through the parameter setting process.
Further, before the step of determining, by the driver process, the resource package to be called by the target application according to the parameter information of the target application, the method further includes
Determining the positions of a resource manager and a driving process which are required to be used by the target application program through a parameter setting process, and applying for resources to server-side equipment to start an actuator process;
and when the actuator process application is successfully monitored, ending the parameter setting process.
Further, the driver process is further configured to create a Spark environment, where the Spark environment is responsible for communicating with the cluster manager of the server device, and applying for resources, and allocating and monitoring tasks.
Further, before the step of determining the target application program corresponding to the task to be processed, the method further includes:
and modifying the corresponding configuration file in the application program of the task to be processed so as to send a memory resource request instruction to increase the memory resource space allocated by the server equipment for processing the task to be processed.
Further, after the step of determining the target application program corresponding to the task to be processed, the method further includes:
starting a main application program of a target application program in a container allocated to the target application program by a resource manager of server equipment;
the method includes registering with a resource manager of the server device through the host application to view an operating state of the application through the resource manager at the client device.
Further, the method further comprises:
sending request information for requesting to process the execution result of the task to be processed to the server-side equipment;
and receiving an execution result returned by the server-side equipment.
In addition, to achieve the above object, the present invention also provides an electronic device, including:
the system comprises a setting module, a processing module and a processing module, wherein the setting module is suitable for setting parameter information of a resource package to be called in an application program, and the application program is used for processing a task to be processed; and
the system comprises a sending module, a processing module and a processing module, wherein the sending module is suitable for determining a target application program corresponding to a task to be processed according to a request message which is sent to a server device by a client device and carries the task to be processed, and starting a driving process for driving the target application program in the server device;
and the driving process determines a resource packet required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource packet through a calling interface so that the target application program processes the task to be processed in the server-side equipment.
Further, the task to be processed is a spare task, and the sending module is further adapted to generate a parameter setting process for processing the spare task in an application program of the client device; and
and sending a request message carrying the spare task to be processed to the server equipment through the parameter setting process.
Furthermore, the sending module is also suitable for determining the positions of a resource manager and a driving process which are required to be used by the target application program through a parameter setting process, and applying for resources to the server-side equipment to start the actuator process;
and when the actuator process application is successfully monitored, ending the parameter setting process.
Further, the driver process is further configured to create a Spark environment, where the Spark environment is responsible for communicating with the cluster manager of the server device, and applying for resources, and allocating and monitoring tasks.
Furthermore, the setting module is further adapted to modify the memory resource request instruction in the application program of the task to be processed, so as to increase the memory resource space allocated by the server device for processing the task to be processed.
Further, the sending module is further adapted to start a main application program of the target application program in a container allocated to the target application program by a resource manager of the server device;
the method includes registering with a resource manager of the server device through the host application to view an operating state of the application through the resource manager at the client device.
Further, the sending module is also adapted to send request information requesting for processing the execution result of the task to be processed to the server device; and
and receiving an execution result returned by the server-side equipment.
To achieve the above object, the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above method.
Compared with the prior art, the task processing method, the electronic device, the computer equipment and the storage medium provided by the invention have the advantages that the resource packet and the calling interface are arranged in the resource server, the driving process is started in the server equipment, and the server equipment calls the corresponding resource packet through the calling interface to process the task to be processed, so that the problem of occupation of local resources during processing of the task to be processed can be solved, and the running efficiency of task processing is effectively improved.
Drawings
FIG. 1 is a flowchart illustrating a method of task processing according to an exemplary embodiment of the invention;
FIG. 2 is a flowchart illustrating a method of task processing according to an exemplary embodiment of the invention;
FIG. 3 is a flowchart illustrating a method of task processing according to an exemplary embodiment of the invention;
FIG. 4 is a flowchart illustrating a method of task processing according to an exemplary embodiment of the invention;
FIG. 5 is a flowchart illustrating a method of task processing according to an exemplary embodiment of the invention;
FIG. 6 is a flowchart illustrating a method of task processing according to an exemplary embodiment of the invention;
FIG. 7 is a schematic diagram illustrating program modules of an electronic device in accordance with an exemplary embodiment of the present invention;
fig. 8 is a schematic diagram of a hardware architecture of an electronic device according to an exemplary embodiment of the present invention.
Reference numerals:
| |
20 |
| Memory device | 21 |
| Processor with a memory having a plurality of memory cells | 22 |
| Network interface | 23 |
| Memory device | 24 |
| Setting module | 201 |
| Transmission module | 202 |
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
In the prior art, generally, when a client device submits a task to be processed to a server device, a request message carrying the task to be processed is submitted to a resource management system, and when the server device processes the task to be processed, the server device needs to communicate with the client device to obtain a resource packet and the like required by processing the task to be processed. In view of the above, the present invention provides a task processing method, an electronic apparatus, a computer device, and a storage medium.
Referring to fig. 1, a flowchart of a task processing method according to an exemplary embodiment of the present invention is shown, where the method includes the following steps:
step S110, setting parameter information of a resource packet to be called in an application program, wherein the application program is used for processing a task to be processed; and
step S120, according to a request message carrying a task to be processed and sent from a client device to a server device, determining a target application program corresponding to the task to be processed, and starting a driving process for driving the target application program in the server device;
step S130, the driver determines, through the parameter information of the target application program, a resource packet that needs to be called by the target application program, and calls, through a call interface, a corresponding resource packet so that the target application program processes the to-be-processed task in the server device.
The resource server is provided with a resource package and a call interface, that is, the resource package required by the server device to process the task to be processed is stored in the resource server, for example, the resource package and an HTTP (hypertext Transfer Protocol) interface capable of calling the resource package are provided in the resource servers such as a hundred-degree network disk. The resource packet and the calling interface set in the resource server may be free, so that the service end device and the like may call the resource packet arbitrarily. The method comprises the steps of setting parameter information of a resource packet to be called in an application program of a task to be processed, so that when the service end equipment processes the task to be processed and needs to call the resource packet, obtaining a path of the resource packet by analyzing the parameter information, and calling the resource packet from a resource server through the path to process the task to be processed.
In an embodiment, the task to be processed is a Spark task, which is a distributed computing system based on a memory, and for ease of understanding, the basic concept in Spark is briefly introduced next.
(1) Application program (Application): the application program written by the user and submitted to the server device to process the Spark task comprises Driver (Driver) code and Executor (Executor) code.
(2) Drive (Driver): represents a main function (main () function) for creating and closing a Spark environment (Spark Context). The Spark environment is responsible for communicating with a Cluster Manager (Cluster Manager) of the server device, and applying for resources, distributing and monitoring tasks and the like are performed. And closing the Spark environment process after the application program of the Spark task is executed.
(3) Actuator (actuator): an Application program (Application) runs on a process on a Worker node (Worker node), and an Executor process (Executor process) is responsible for running some work units (tasks) and storing data in a memory or a disk. In the resource management system mode (Spark on Yarn mode), the Executor is named as Coarse-Grained Executor Backend (Coarse Grained Executor Backend) and one Coarse-Grained Executor Backend process has only one Executor process object and holds one thread pool, and the Executor process is responsible for packaging work units into Task management tools (Task Runner) and extracting one idle thread from the thread pool to run the work units, so that the number of the work units which can be run in parallel by each Coarse-Grained Executor Backend depends on the number of the CPUs allocated to the Coarse-Grained Executor Backend.
(4) Cluster Manager (Cluster Manager): refers to an external service that acquires resources on the cluster.
(5) Working node (Worker): any node in the cluster that can run application code.
(6) Working unit (Task): the work units sent to a certain executor are basic units for running the application program, a plurality of work units form a work single station (Stage), and the scheduling and management of the work units are responsible for a Task debugging program (Task Schedule).
Sending a request message carrying a task to be processed to a server device through a client device, wherein the content in the request message carrying the task to be processed comprises a main Application program (Application Master) of the task to be processed, a command for starting the main Application program and the like.
As shown in fig. 2, in an embodiment of the present invention, the step of determining the target application corresponding to the task to be processed may include the following steps:
step S201, starting a main application program of a target application program in a container allocated to the target application program by a resource manager of server equipment;
step S202, registering with a resource manager of the server device through the main application program, so as to check the running state of the application program through the resource manager at the client device.
After receiving the request, the Resource Manager (Resource Manager) of the server device selects a Node Manager (Node Manager) in the cluster, allocates a Container (Container) to the application program, and requests it to start the main application program of the application program in the Resource, wherein the main application program initializes the Spark environment and the like.
The main application program registers with the resource manager so that a user can directly check the running state of the application program through the resource manager, and the resource manager can apply for resources for each task through an RPC (Remote Procedure Call) protocol in a polling mode and monitor the running state of the tasks until the running is finished.
After the main application program applies for the resources, the main application program communicates with a corresponding Node Manager (Node Manager) to request the Node Manager to start the back end of the coarse-grained executor in the obtained resources, and the back end of the coarse-grained executor registers in the Spark environment in the main application program and applies for a work unit after being started. When the Spark environment is initialized in the Spark application program, the rear end of the coarse-granularity scheduler is matched with a cluster scheduler to schedule tasks, wherein the cluster scheduler is a simple package of a task scheduler (task scheduler Impl), and waiting logic of an actuator is increased.
After receiving a request message carrying a task to be processed and sent by client equipment, the server equipment starts a driving process according to a received driver in a main application program, analyzes parameter information in a target application program corresponding to the task to be processed, determines a resource packet to be called according to the parameter information of the target application program and the parameter information of the resource packet, and calls the corresponding resource packet through a calling interface, so that the task to be processed can be processed in the server equipment.
As shown in fig. 3, in an embodiment of the present invention, the to-be-processed task is a spare task, and the step of sending, to the server device, a request message carrying the to-be-processed task by the client device may include the following steps:
step S301, generating a parameter setting process for processing a Spark task in an application program of the client device; and
step S302, sending a request message carrying the spare task to be processed to the server device through the parameter setting process.
When an application program of a task to be processed is submitted to a server device, a Spark environment object is initialized firstly, the Spark environment is responsible for coordinating the application program to run on a cluster, and a flow demonstration (Demo) submitted by the program is packaged. And generating a parameter setting (Spark-Submit) process in the client equipment, wherein the parameter setting process is used for submitting the task to a Spark cluster, submitting the application program of the task to be processed by loading the environment variable, and sending a request message for processing the task to be processed to the server equipment.
The specific process of the parameter setting process is as follows:
1. determine an entry for the application (e.g., org. app. spark. instances. sparkpi);
2. determining the host node address, which is the URL address of the master in the cluster (e.g., spark://192.168.1.1: 1234);
3. determining a deployment mode;
4. determining a spark configuration form and a key-value pair form;
5. determining an application program path;
6. parameters required by the user application are determined.
As shown in fig. 4, in an embodiment of the present invention, before the step of determining, by the driver process, the resource package that needs to be called by the target application according to the parameter information of the target application, the method further includes:
step S401, determining the positions of a resource manager and a driving process which are needed to be used by the target application program through a parameter setting process, and applying for resources to server-side equipment to start an actuator process;
and step S402, when the actuator process application is monitored to be successful, ending the parameter setting process.
The client device submits the application program of the task to be processed through the parameter setting process, the parameter setting process determines which resource manager is used according to the user code and the configuration, and the driver is started at a proper position. The parameter setting process is used to set how many executor processes are to be used in total for the task to be processed. When the driver applies for resources from the cluster manager, the cluster manager sets each working node of the cluster according to the setting of the application program of the task to be processed, and starts corresponding number of executor processes. The application program submitting the task to be processed comprises an executor code, and the executor code is started by applying for resources from the server-side equipment.
The method comprises the steps that a parameter setting process is generated on client equipment, a request message carrying a task to be processed is sent to server equipment through the parameter setting process, an actuator process is applied, the server equipment starts a main node and a working node in a cluster node, and the main node informs the working node to start a driving process. After the server device receives the request message, the parameter setting process is ended to save resources of the client device.
As shown in fig. 5, in an embodiment of the present invention, before the step of determining the target application program corresponding to the task to be processed, the following steps may be further included:
step S501, modify a corresponding configuration file in an application program of a to-be-processed task, so as to send a memory resource request instruction to increase a memory resource space allocated by a server device for processing the to-be-processed task.
The cluster manager is arranged at each working node of the cluster as far as possible according to the setting of the application program of the task to be processed, and starts corresponding number of executor processes. The memory resource parameter is used for setting the memory of each actuator process.
The memory resource parameter is used for setting the number of CPU cores of each actuator process. This parameter determines the ability of each executor process to execute a unit of work (Task) thread in parallel. Because each CPU core can only execute one unit of work thread at a time, the greater the number of CPU cores per actuator process, the faster it can be to execute all the unit of work threads assigned to it. The memory resource request instruction can be increased by multiple times by modifying the configuration file in the application program of the task to be processed, so that the memory resource space allocated for processing the task to be processed is increased, and the speed of processing the task to be processed is increased.
As shown in fig. 6, in an embodiment of the present invention, the task processing method may further include the following steps:
step S601, sending request information for requesting to process the execution result of the task to be processed to the server device;
and
step S602, receiving an execution result returned by the server device.
The result of the application program of the task to be processed is output in a Standard output (Stdout) of the node running the driver process. In order to facilitate the user to view the result of the application program, in an embodiment of the present invention, request information requesting for processing the execution result of the task to be processed is sent to the server device, and the server device returns the execution result to the client device that sent the request information after receiving the request information, so that the user can conveniently view the result of the application program in the client device.
In an embodiment of the present invention, the Spark environment in the main application program allocates the working unit to the rear end of the coarse-grained actuator for execution, and the rear end of the coarse-grained actuator runs the working unit and reports the running state and progress to the main application program, so that the main application program can master the running state of each task at any time, and thus the task can be restarted when the task fails.
A Directed Acyclic Graph Scheduler (direct Acyclic Graph Scheduler, DAG Scheduler for short) divides tasks to be processed: dividing a task to be processed into a work single station task, dividing the work single station task into Pipeline (Pipeline) tasks, packaging all operations in the Pipeline tasks into work unit tasks, and sending the work unit tasks to work unit threads in an actuator process for execution. And when the executor process monitors that the driving process is finished before all the threads of the working units are executed, namely the task to be processed is not processed, restarting the driving process and processing the task to be processed again.
After the application program of the task to be processed is operated, the main application program applies for logout to the resource manager and closes the main application program, so that the occupation of resources in the client equipment and the server equipment is reduced.
According to the invention, the resource packet and the calling interface are arranged in the resource server, the driving process is started in the server side equipment, and the server side equipment calls the corresponding resource packet through the calling interface to process the task to be processed, so that the problem of local resource occupation during processing the task to be processed can be solved, and the operation efficiency of task processing is effectively improved.
The invention further provides an electronic device. Fig. 7 is a schematic diagram illustrating program modules of the electronic device 20 according to an exemplary embodiment of the invention.
The electronic device 20 includes:
the setting module 201 is adapted to set parameter information of a resource package to be called in an application program, where the application program is used to process a task to be processed; and
the sending module 202 is adapted to determine a target application program corresponding to a task to be processed according to a request message carrying the task to be processed sent from the client device to the server device, and start a driving process for driving the target application program in the server device;
and the driving process determines a resource packet required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource packet through a calling interface so that the target application program processes the task to be processed in the server-side equipment.
Further, the task to be processed is a spare task, and the sending module 202 is further adapted to generate a parameter setting process for processing the spare task in an application program of the client device; and
and sending a request message carrying the spare task to be processed to the server equipment through the parameter setting process.
Further, the sending module 202 is further adapted to determine, through a parameter setting process, positions of a resource manager and a driver process that are required to be used by the target application program, and apply for resources from a server device to start an actuator process;
and when the actuator process application is successfully monitored, ending the parameter setting process.
Further, the driver process is further configured to create a Spark environment, where the Spark environment is responsible for communicating with the cluster manager of the server device, and applying for resources, and allocating and monitoring tasks.
Further, the setting module 201 is further adapted to modify the memory resource request instruction in the application program of the to-be-processed task, so as to increase the memory resource space allocated by the server device for processing the to-be-processed task.
Further, the sending module 202 is further adapted to start a main application of the target application in a container allocated to the target application by a resource manager of the server device;
the method includes registering with a resource manager of the server device through the host application to view an operating state of the application through the resource manager at the client device.
Further, the sending module 202 is further adapted to send request information requesting for processing the execution result of the task to be processed to the server device; and
and receiving an execution result returned by the server-side equipment.
According to the invention, the driving process is started in the server-side equipment, and the server-side equipment calls the corresponding resource packet through the calling interface to process the task to be processed, so that the problem of local resource occupation during processing the task to be processed can be solved, and the running efficiency of task processing is effectively improved.
To achieve the above object, as shown in fig. 8, the present invention further provides a computer device 20, which includes a memory 21, a processor 22, and a computer program stored on the memory 21 and executable on the processor 22, wherein the processor 22 implements the steps of the above method when executing the computer program. The computer program may be stored in the memory 24.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above method.
The present invention also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers) capable of executing programs, and the like. The computer device of the embodiment at least includes but is not limited to: memory, processor, etc. communicatively coupled to each other via a system bus.
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer readable storage medium of the embodiment is used for storing the electronic device 20, and when being executed by the processor 22, the computer readable storage medium implements the task processing method of the invention.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A method for processing a task, the method comprising the steps of:
setting parameter information of a resource package to be called in an application program, wherein the application program is used for processing a task to be processed; and
determining a target application program corresponding to a task to be processed according to a request message carrying the task to be processed and sent to a server device by a client device, and starting a driving process for driving the target application program in the server device;
and determining a resource packet required to be called by the target application program according to the parameter information of the target application program, and calling the corresponding resource packet through a calling interface so that the target application program processes the task to be processed in the server-side equipment.
2. The task processing method according to claim 1, wherein the task to be processed is a Spark task, and the step of sending the request message carrying the task to be processed to the server device according to the client device further comprises:
generating a parameter setting process for processing a Spark task in an application program of the client device; and
and sending a request message carrying the spare task to be processed to the server equipment through the parameter setting process.
3. A task processing method according to claim 2, wherein the step of determining the resource package to be called by the target application by the driver process through the parameter information of the target application is preceded by the step of determining the resource package to be called by the target application, further comprising
Determining the positions of a resource manager and a driving process which are required to be used by the target application program through a parameter setting process, and applying for resources to server-side equipment to start an actuator process;
and when the actuator process application is successfully monitored, ending the parameter setting process.
4. The task processing method of claim 2, wherein the driver process is further configured to create a Spark environment, and the Spark environment is responsible for communicating with the cluster manager of the server device, and applying for resources, allocating and monitoring tasks.
5. The task processing method according to claim 1, wherein before the step of determining the target application program corresponding to the task to be processed, the method further comprises:
and modifying the corresponding configuration file in the application program of the task to be processed so as to send a memory resource request instruction to increase the memory resource space allocated by the server equipment for processing the task to be processed.
6. The task processing method according to claim 1, wherein after the step of determining the target application program corresponding to the task to be processed, the task processing method further comprises:
starting a main application program of a target application program in a container allocated to the target application program by a resource manager of server equipment;
the method includes registering with a resource manager of the server device through the host application to view an operating state of the application through the resource manager at the client device.
7. A task processing method according to claim 3, characterized in that the method further comprises:
sending request information for requesting to process the execution result of the task to be processed to the server-side equipment; and
and receiving an execution result returned by the server-side equipment.
8. An electronic device, comprising:
the system comprises a setting module, a processing module and a processing module, wherein the setting module is suitable for setting parameter information of a resource package to be called in an application program, and the application program is used for processing a task to be processed; and
the system comprises a sending module, a processing module and a processing module, wherein the sending module is suitable for determining a target application program corresponding to a task to be processed according to a request message which is sent to a server device by a client device and carries the task to be processed, and starting a driving process for driving the target application program in the server device;
and the driving process determines a resource packet required to be called by the target application program through the parameter information of the target application program, and calls a corresponding resource packet through a calling interface so that the target application program processes the task to be processed in the server-side equipment.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the task processing method according to any of claims 1 to 7 are implemented by the processor when executing the computer program.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when being executed by a processor, realizes the steps of the task processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910754313.9A CN110659131B (en) | 2019-08-15 | 2019-08-15 | Task processing method, electronic device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910754313.9A CN110659131B (en) | 2019-08-15 | 2019-08-15 | Task processing method, electronic device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110659131A true CN110659131A (en) | 2020-01-07 |
| CN110659131B CN110659131B (en) | 2024-04-02 |
Family
ID=69037586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910754313.9A Active CN110659131B (en) | 2019-08-15 | 2019-08-15 | Task processing method, electronic device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110659131B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000949A (en) * | 2020-08-26 | 2020-11-27 | 中国联合网络通信集团有限公司 | Program package calling method, system, terminal device and computer readable storage medium |
| CN112783978A (en) * | 2021-01-28 | 2021-05-11 | 北京东方优播网络科技有限公司 | Data transmission method, device, computer system and storage medium |
| CN114237752A (en) * | 2021-12-20 | 2022-03-25 | 百度在线网络技术（北京）有限公司 | Display method and device of pushed page, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5301326A (en) * | 1991-09-24 | 1994-04-05 | Microsoft Corporation | Method and system for controlling the execution of an application program |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN108710529A (en) * | 2018-04-28 | 2018-10-26 | 四川斐讯信息技术有限公司 | A kind of remote task processing method, system and wireless router |
| CN109936593A (en) * | 2017-12-15 | 2019-06-25 | 网宿科技股份有限公司 | A kind of method and system of message distribution |
| CN110069335A (en) * | 2019-05-07 | 2019-07-30 | 江苏满运软件科技有限公司 | Task processing system, method, computer equipment and storage medium |
-
2019
- 2019-08-15 CN CN201910754313.9A patent/CN110659131B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5301326A (en) * | 1991-09-24 | 1994-04-05 | Microsoft Corporation | Method and system for controlling the execution of an application program |
| CN109936593A (en) * | 2017-12-15 | 2019-06-25 | 网宿科技股份有限公司 | A kind of method and system of message distribution |
| CN108255619A (en) * | 2017-12-28 | 2018-07-06 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN108710529A (en) * | 2018-04-28 | 2018-10-26 | 四川斐讯信息技术有限公司 | A kind of remote task processing method, system and wireless router |
| CN110069335A (en) * | 2019-05-07 | 2019-07-30 | 江苏满运软件科技有限公司 | Task processing system, method, computer equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000949A (en) * | 2020-08-26 | 2020-11-27 | 中国联合网络通信集团有限公司 | Program package calling method, system, terminal device and computer readable storage medium |
| CN112000949B (en) * | 2020-08-26 | 2023-06-16 | 中国联合网络通信集团有限公司 | Program package calling method, system, terminal device and computer readable storage medium |
| CN112783978A (en) * | 2021-01-28 | 2021-05-11 | 北京东方优播网络科技有限公司 | Data transmission method, device, computer system and storage medium |
| CN114237752A (en) * | 2021-12-20 | 2022-03-25 | 百度在线网络技术（北京）有限公司 | Display method and device of pushed page, electronic equipment and storage medium |
| CN114237752B (en) * | 2021-12-20 | 2023-09-26 | 百度在线网络技术（北京）有限公司 | Push page display method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110659131B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10033816B2 (en) | Workflow service using state transfer | |
| CN110659131B (en) | Task processing method, electronic device, computer equipment and storage medium | |
| US20190158626A1 (en) | Method, apparatus and computer readable storage medium for processing service | |
| CN108874549B (en) | Resource multiplexing method, device, terminal and computer readable storage medium | |
| CN106897299B (en) | Database access method and device | |
| CN112256417B (en) | Data request processing method and device and computer readable storage medium | |
| WO2019228346A1 (en) | Method and apparatus for executing task by scheduling device | |
| CN108900627B (en) | Network request method, terminal device and storage medium | |
| CN110162397B (en) | Resource allocation method, device and system | |
| CN110659104B (en) | Service monitoring method and related equipment | |
| US20200310828A1 (en) | Method, function manager and arrangement for handling function calls | |
| CN114816709A (en) | Task scheduling method, device, server and readable storage medium | |
| CN115617497A (en) | Thread processing method, scheduling component, monitoring component, server and storage medium | |
| CN111586140A (en) | Data interaction method and server | |
| CN113157411A (en) | Reliable configurable task system and device based on Celery | |
| CN112685132A (en) | Koji task execution method, device, equipment and readable storage medium | |
| CN110888726A (en) | Multitask concurrent processing method and system | |
| CN114385227A (en) | Service processing method, device, equipment and storage medium | |
| CN115114022A (en) | Method, system, device and medium for using GPU resources | |
| CN114385351A (en) | Cloud management platform load balancing performance optimization method, device, equipment and medium | |
| CN109634721B (en) | Method and related device for starting communication between virtual machine and host | |
| CN110109747B (en) | Apache Spark-based data exchange method, system and server | |
| CN109062702B (en) | Computing resource allocation method, related device and readable storage medium | |
| CN112486502A (en) | Distributed task deployment method and device, computer equipment and storage medium | |
| US20140343930A1 (en) | Systems and Methods for Voice Data Processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |