WO2024130565A1 - 任务处理方法、装置及电子设备 - Google Patents

任务处理方法、装置及电子设备 Download PDF

Info

Publication number
WO2024130565A1
WO2024130565A1 PCT/CN2022/140507 CN2022140507W WO2024130565A1 WO 2024130565 A1 WO2024130565 A1 WO 2024130565A1 CN 2022140507 W CN2022140507 W CN 2022140507W WO 2024130565 A1 WO2024130565 A1 WO 2024130565A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
executed
computer device
computing node
parameters
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/140507
Other languages
English (en)
French (fr)
Inventor
杨恢亮
颜旭
黎宇翔
成祥源
曾涛
曹耀方
王志扬
董宇亮
章文蔚
徐讯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BGI Shenzhen Co Ltd
Original Assignee
BGI Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BGI Shenzhen Co Ltd filed Critical BGI Shenzhen Co Ltd
Priority to PCT/CN2022/140507 priority Critical patent/WO2024130565A1/zh
Priority to CN202280101155.7A priority patent/CN120077365A/zh
Publication of WO2024130565A1 publication Critical patent/WO2024130565A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements

Definitions

  • the present application relates to the field of data processing technology, and in particular to a task processing method, device and electronic equipment.
  • various asynchronous task processing frameworks can only meet the asynchronous task processing requirements on a single machine, such as an asynchronous task processing system that runs multiple computing nodes on a single machine by calling multiple threads.
  • the present application provides a task processing method, device and electronic device, the main purpose of which is to improve the technical problem in the current prior art that a large number of asynchronous tasks are uniformly stacked on a single machine to run, which not only makes computing resources tight, but also affects the efficiency of asynchronous task processing.
  • the present application provides a task processing method, comprising:
  • a task execution instruction is sent to the target computing node, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • the present application provides a task processing method, applied to a first computer device, the method comprising:
  • the present application provides a task processing method, applied to a second computer device, the method comprising:
  • the task processing state is an idle state, obtaining the task parameters of the current task to be executed;
  • a task execution instruction is sent to the first computer device, where the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.
  • the present application provides a task processing device, comprising:
  • a monitoring module is configured to monitor the task processing status of each computing node in the distributed system, wherein the computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location;
  • a determination module is configured to determine a target computing node whose task processing state is an idle state
  • An acquisition module is configured to acquire task parameters of a current task to be executed
  • the sending module is configured to send a task execution instruction to the target computing node according to the task parameters, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • the present application provides a task processing device, applied to a first computer device, the device comprising:
  • a sending module configured to send the current task processing status of the first computer device to the second computer device
  • a receiving module configured to receive a task execution instruction sent by the second computer device, wherein the task execution instruction is sent by the second computer device when determining that the task processing state is an idle state, and the task execution instruction carries a task parameter of a current task to be executed;
  • the execution module is configured to execute the current task to be executed according to the task parameters.
  • the present application provides a task processing device, applied to a second computer device, the device comprising:
  • a receiving module is configured to request the first computer device to obtain a current task processing status of the first computer device
  • an acquisition module configured to acquire task parameters of the current task to be executed if the task processing state is an idle state
  • the sending module is configured to send a task execution instruction to the first computer device according to the task parameters, wherein the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.
  • the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the task processing method described in the first aspect, the second aspect, or the third aspect is implemented.
  • the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the task processing method described in the first aspect, the second aspect, or the third aspect when executing the computer program.
  • the present application provides a task processing method, device and electronic device. Compared with the current prior art, the present application can effectively improve the degree of freedom of asynchronous task scheduling, and can mobilize computing nodes deployed on multiple machines to perform distributed processing of asynchronous tasks. Specifically, the task processing status of each computing node in the distributed system is first monitored.
  • the computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location; then determine the target computing node whose task processing status is idle; then send a task execution instruction to the target computing node based on the task parameters of the current task to be executed, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • FIG1 is a schematic diagram showing a flow chart of a task processing method provided in an embodiment of the present application.
  • FIG2 is a schematic diagram showing a flow chart of another task processing method provided in an embodiment of the present application.
  • FIG3 shows a schematic diagram of the structure of a distributed system architecture provided in an embodiment of the present application
  • FIG4 is a schematic diagram showing an example of a front-end display effect provided by an embodiment of the present application.
  • FIG5 is a schematic diagram showing an example architecture provided in an embodiment of the present application.
  • FIG6 is a schematic diagram showing a flow chart of another task processing method provided in an embodiment of the present application.
  • FIG. 7 is a schematic diagram showing a flow chart of another task processing method provided in an embodiment of the present application.
  • FIG8 shows a schematic diagram of the structure of a task processing device provided in an embodiment of the present application.
  • FIG9 shows a schematic diagram of the structure of another task processing device provided in an embodiment of the present application.
  • FIG. 10 shows a schematic diagram of the structure of another task processing device provided in an embodiment of the present application.
  • This embodiment provides a task processing method, as shown in Figure 1, the method includes:
  • Step 101 Monitor the task processing status of each computing node in the distributed system.
  • the computing nodes in the distributed system can be different computer devices (such as network computers, servers and other devices), and different computing nodes can read the business data in the preset storage location.
  • This embodiment adopts the decentralized concept of distributed systems, distributing all computing nodes to various computer devices to facilitate development and maintenance.
  • a unified scheduling platform (Monitor) as a manager to distribute asynchronous tasks.
  • the scheduling platform can be a program with a front-end and a back-end, and it is unique in the distributed system. It can be deployed on any computer device in the same network segment. It is responsible for the deployment of tasks. Users submit tasks in the foreground, and the background records the task information submitted by users into the database to undertake monitoring and distribution of these tasks.
  • the preset storage location can be a database, folder or data table, etc., which can be used to store business-related data, such as all sequencing data.
  • the sequencing data generated by the sequencing instrument will also be transferred to the preset storage location for storage.
  • the path of the preset storage location will be mounted on each computer device, so that different computing nodes can read the business data (data required to perform tasks) in the preset storage location.
  • Step 102 Determine a target computing node whose task processing state is an idle state.
  • computing node A there are five computing nodes in the distributed system, namely computing node A, computing node B, computing node C, computing node D, and computing node E.
  • computing node A only the task processing status of computing node A is idle, while the other four computing nodes are in the task processing status, then computing node A can be determined as the target computing node.
  • Step 103 Obtain task parameters of the current task to be executed.
  • the task parameters may be the parameters referenced by the task execution (such as the task execution subject, the task execution action, the task execution purpose, etc.).
  • the target computing node When the subsequent task is executed, the target computing node will execute according to the task parameters.
  • Step 104 Send a task execution instruction to the target computing node according to the task parameters of the current task to be executed.
  • the target computing node is enabled to execute the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • computing node A whose task processing status is idle, parses the received task execution instruction to obtain the task information (including task parameters) that needs to be executed, and then reads the business data corresponding to the task parameters in the preset storage location according to the path of the preset storage location, and then executes the current task to be executed assigned to the computing node A.
  • this embodiment can effectively improve the degree of freedom of asynchronous task scheduling, and can mobilize computing nodes deployed on multiple machines to perform distributed processing of asynchronous tasks. Specifically, first monitor the task processing status of each computing node in the distributed system.
  • the computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location; then determine the target computing node whose task processing status is idle; then send a task execution instruction to the target computing node based on the task parameters of the current task to be executed, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • a distributed system for realizing free scheduling of asynchronous tasks is provided, and a large number of asynchronous tasks can be distributedly processed by computing nodes deployed on multiple machines, which can effectively improve the problem of tight computing resources of a single machine and improve the efficiency of asynchronous task processing.
  • this embodiment provides a specific method as shown in FIG. 2, which includes:
  • Step 201 monitor the task processing status of each computing node in the distributed system at a preset time interval.
  • the computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location.
  • this embodiment can use the redis database (only for illustrative purposes, other persistent storage databases can also be used) and a timed polling script to implement the role of the message middleware in the asynchronous framework, so that the scheduling of asynchronous tasks is freed from the celery framework, so that developers can process the redis database and modify the script to achieve the purpose of adapting to various data analysis scenarios.
  • this embodiment may provide a distributed system for realizing free scheduling of asynchronous tasks, wherein the distributed system is a system in which hardware or software systems are distributed on different computer devices and can communicate and coordinate with each other through message passing.
  • the distributed system a group of independent computer devices presents a unified whole to the user.
  • the system has a variety of general physical and logical resources, which can dynamically allocate tasks, and the dispersed physical and logical resources realize information exchange through a computer network.
  • the distributed system may include: a data center (preset storage location), a task message middleware, a scheduling platform (Monitor), computing nodes, a scheduling layer front-end page, a scheduling layer backend, etc.
  • the data center (preset storage location, such as NFS server) will store all business data (such as sequencing data.
  • the sequencing data generated by the sequencing instrument will also be transferred to the data center for storage).
  • the path for storing business data will be mounted to each computing node, so that even if the service is deployed on other computer devices, the business data stored in the data center can be read.
  • the database can be a redis database. It should be noted that other persistent storage databases can also be selected according to actual needs, and this embodiment does not limit it.
  • each task information created will be stored in the redis database.
  • the redis database will also store the parameter configuration of each asynchronous task. Specifically, it may include the following:
  • ConfigDB Redis database, which stores the parameter configuration of each task, including parameter options, parameter default values, the maximum number of workers for each task, etc.
  • Pending queue database (first preset database, PendingQueueDB): redis database. Any created task information is first stored in this pending queue database waiting for processing.
  • Running status database (second preset database, RunningBufferDB): Redis database, records the running tasks. If a task enters the running status database, it means that it is already in the running state.
  • Scheduling platform (scheduler, Monitor): a role that manages the task queue. It can be implemented as a scheduled script. Its main function is to push the tasks to be executed in the queue in the first preset database to the idle computing nodes for execution.
  • the real executor of the asynchronous task can be a package deployed on a computer device (such as a server), which can complete the task logic through the remote Secure Shell Protocol (SSH) command line call.
  • the worker is equivalent to a software virtual device, that is, a package of a data analysis function deployed on a single computer device (a single computing node). It can be considered as a background program that responds to these ssh command lines and finally actually executes the asynchronous task.
  • Scheduling layer front page For example, users can select sequencing data and corresponding parameters on the page and then click Submit Task, and the front desk will send the parameters to the back desk.
  • the front desk will also display all currently queued/running task information, and provide relevant interfaces for operators to add, delete, modify, and query the task queue.
  • Scheduling layer backend The backend for scheduling tasks can use Python's Django backend framework, which provides many interfaces to the front-end page to complete the modification of task information and the replacement of task queues. Specifically, it can perform various operations such as reading, writing, and modifying the redis database.
  • the task processing method provided in this embodiment is specifically executed, wherein the executor of the method of this embodiment can be the scheduling platform, which can monitor the task processing status of each computing node in the distributed system at a preset time interval (the timing is to monitor once every 1 second), and then timely discover the target computing nodes whose task processing status is idle, and assign them the tasks to be executed queued in the first preset database.
  • the executor of the method of this embodiment can be the scheduling platform, which can monitor the task processing status of each computing node in the distributed system at a preset time interval (the timing is to monitor once every 1 second), and then timely discover the target computing nodes whose task processing status is idle, and assign them the tasks to be executed queued in the first preset database.
  • Step 202 Determine a target computing node whose task processing state is an idle state.
  • Step 203 extracting the task parameters of the tasks to be executed that are ranked high from the first preset database according to the arrangement order of the tasks to be executed.
  • the arrangement order of the tasks to be executed in the first preset database may be determined according to the creation time of the tasks to be executed and/or the corresponding set priorities. After extracting the task parameters of the tasks to be executed from the first preset database, the tasks to be executed and the task parameters of the tasks to be executed will be deleted from the first preset database to save storage space of the first preset database.
  • the earlier the creation time of the task to be executed the higher its arrangement order in the first preset database (the higher its ranking), and the later the creation time of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking).
  • the user can set the priority of the task to be executed according to actual needs. The higher the priority of the task to be executed, the higher its arrangement order in the first preset database (the higher its ranking), and the lower the priority of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking).
  • Step 204 The extracted task parameters are combined to obtain command line information corresponding to the task to be executed.
  • Step 205 Send command line information to the target computing node through the secure shell protocol.
  • a software tool is installed on the target computing node, and the target computing node executes the current task to be executed in response to the command line information through the software tool.
  • the target computing node responds to the ssh command line through a software tool and executes an asynchronous task in a background program, wherein the software tool may be a software virtual device with a data analysis function, namely, a worker.
  • the task parameters of the top-ranked tasks to be executed in PendingQueueDB are extracted, the command line to be executed corresponding to the task is spliced out, and then sent to computing node A via SSH, so that computing node A executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • step 205 may specifically include: obtaining a key according to the current time information and the identification of the local computer device (such as the MAC address, IP address, motherboard ID number, etc. of the terminal) (the current time information and the identification of the local computer device may be combined to calculate a hash value or an MD5 value, etc. to obtain the key); after encrypting the command line information with the key, sending it to the target computing node through the secure shell protocol, so that the target computing node can infer the key according to the time information of the received command line information and the identification of each computer device that has established trust with the target computing node, and use the key to decrypt the command line information.
  • the target computing node after encrypting the command line information with the key, sending it to the target computing node through the secure shell protocol, so that the target computing node can infer the key according to the time information of the received command line information and the identification of each computer device that has established trust with the target computing node, and use the key to decrypt the command line information.
  • the target computing node obtains the time within 3 seconds (such as 9:59:59, 9:59:58, 9:59:57) based on the time information of receiving the command line information (such as 10:00:00), and then combines the time information within 3 seconds with the identification of each computer device that has established trust with the target computing node to calculate a hash value or MD5 value, and then uses the hash value or MD5 value one by one to try to decrypt the encrypted command line information until the decryption is successful, thereby obtaining the decrypted command line information.
  • 3 seconds such as 9:59:59, 9:59:58, 9:59:57
  • the target computing node uses the hash value or MD5 value one by one to try to decrypt the encrypted command line information until the decryption is successful, thereby obtaining the decrypted command line information.
  • the method of this embodiment may also include: creating a running record of the current task to be executed in a second preset database (RunningBufferDB), wherein the running records of each task created in the second preset database can be provided to the client for display, so that the user can view the progress of task processing in real time, thereby improving the user's experience.
  • RuleBufferDB a second preset database
  • the method of this embodiment may also include: responding to an update setting instruction of the distributed system, obtaining the number of computing nodes to be updated and set; and then determining the computing nodes to be started or shut down according to the number of computing nodes based on the computing performance information and real-time task processing status of each computing node in the distributed system.
  • users can set the number of computing nodes in a distributed system, which can be set before or during the operation of the distributed system.
  • computing nodes with better computing performance can be started first; and during the operation of the distributed system, based on the number of computing nodes set by the user, appropriate computing nodes can be started or shut down according to computing performance information and real-time task processing status, so as to achieve the optimal overall computing performance of the distributed system.
  • the basecall task specifically converts the identified read result (reads) signal into a site (base). As shown in Figure 4, select the file and task parameters to be analyzed on the front-end page, and click the Add button (ADD) to create a task. At this time, the task record will appear in PendingQueueDB (the first preset database). Due to the limitation of the number of computing nodes, many submitted tasks will be queued according to the order of submission. It can be observed that the analysis task just submitted is at the 169th position in the queue. Then in the redis database of the stack structure, the position is the first.
  • PendingQueueDB the first preset database
  • the backend interface can be called through the front-end page to process the task queue, thereby improving the priority position of certain tasks. If the task is swapped from the end of the queue to the first place, when the task is in the queue, the task position and task parameters can still be changed by changing the redis database.
  • the computing node is idle, the task in the stack structure will be extracted, and the parameters will be spliced into the corresponding SSH remote command line, which will be sent to the idle computing node for execution.
  • a new task record will be created in RunningBuffer (the second preset database) to track progress.
  • RunningBuffer the second preset database
  • This embodiment transforms the single-machine task scheduling mode into a distributed system scheduling mode, which can fully utilize the computing nodes of multiple computing devices and effectively solve problems that are difficult to solve in a single-machine performance bottleneck scenario.
  • a method as shown in FIG. 6 is provided, which can be applied to a first computer device (a single computing node), and the method includes:
  • Step 301 A first computer device sends a current task processing status of the first computer device to a second computer device.
  • the first computer device may be any computing node in a distributed system
  • the second computer device is a computer device deployed with a scheduling platform.
  • Step 302 The first computer device receives a task execution instruction sent by the second computer device.
  • the task execution instruction is sent by the second computer device when it is determined that the task processing state is an idle state.
  • the task execution instruction carries the task parameters of the current task to be executed.
  • step 302 may specifically include: the first computer device receives command line information sent by the second computer device through a secure shell protocol, where the command line information is obtained by splicing according to task parameters;
  • a first computer device receives command line information sent by a second computer device via a secure shell protocol, which may specifically include: the first computer device receives encrypted command line information sent by the second computer device via a secure shell protocol; then, based on time information of receiving the command line information and identifications of each computer device that has established trust with the first computer device, determines a key required to decrypt the command line information; and then uses the key to decrypt the command line information.
  • step 205 The specific decryption process can be found in the corresponding description in step 205, which will not be repeated here.
  • Step 303 The first computer device executes the current task to be executed according to the task parameters.
  • step 303 may specifically include: executing the current task to be executed by executing command line information.
  • step 303 may specifically include: executing the current task to be executed based on the business data corresponding to the task parameters in a preset storage location, wherein the preset storage location stores business data corresponding to different task parameters.
  • This embodiment transforms the single-machine task scheduling mode into a distributed system scheduling mode, which can fully utilize the computing nodes of multiple computing devices and effectively solve problems that are difficult to solve in a single-machine performance bottleneck scenario.
  • a method as shown in FIG. 7 is provided, which can be applied to a second computer device (which can be a computer device on which the scheduling platform is deployed), and the method includes:
  • Step 401 The second computer device requests the first computer device to obtain the current task processing status of the first computer device.
  • the first computer device may be any computing node in a distributed system
  • the second computer device is a computer device deployed with a scheduling platform.
  • Step 402 If the current task processing state of the first computer device is an idle state, the second computer device obtains the task parameters of the current task to be executed.
  • obtaining the task parameters of the current task to be executed in step 402 may specifically include: extracting the task parameters of the top-ranked tasks to be executed from the first preset database according to the arrangement order of the tasks to be executed, wherein the arrangement order of the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and/or the corresponding set priority.
  • the earlier the creation time of the task to be executed the higher its arrangement order in the first preset database (the higher its ranking), and the later the creation time of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking).
  • the user can set the priority of the task to be executed according to actual needs. The higher the priority of the task to be executed, the higher its arrangement order in the first preset database (the higher its ranking), and the lower the priority of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking).
  • Step 403 The second computer device sends a task execution instruction to the first computer device according to the task parameters.
  • the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.
  • step 403 may specifically include: splicing the task parameters to obtain command line information to be executed corresponding to the task; and sending the command line information to the first computer device via a secure shell protocol.
  • the sending of command line information to the first computer device through the secure shell protocol may specifically include: obtaining a key according to the current time information and the identification of the local computer device; after encrypting the command line information using the key, sending it to the first computer device through the secure shell protocol.
  • the specific decryption process can be found in the corresponding descriptions in steps 205 and 302, which will not be repeated here.
  • the method of this embodiment may also include: creating a running record of the current task to be executed in the second preset database; receiving a viewing instruction for the task running record; and determining a display result corresponding to the viewing instruction according to the running records of each task created in the second preset database, so that the user can view the progress of the task processing in real time, thereby improving the user's experience.
  • This embodiment transforms the single-machine task scheduling mode into a distributed system scheduling mode, which can fully utilize the computing nodes of multiple computing devices and effectively solve problems that are difficult to solve in a single-machine performance bottleneck scenario.
  • this embodiment provides a task processing device, as shown in FIG. 8 , which includes: a monitoring module 51 , a determining module 52 , an acquiring module 53 , and a sending module 54 .
  • a monitoring module 51 is configured to monitor the task processing status of each computing node in the distributed system, where the computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location;
  • a determination module 52 is configured to determine a target computing node whose task processing state is an idle state
  • the acquisition module 53 is configured to acquire the task parameters of the current task to be executed
  • the sending module 54 is configured to send a task execution instruction to the target computing node according to the task parameters, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.
  • the acquisition module 53 is specifically configured to extract the task parameters of the top-ranked tasks to be executed from the first preset database according to the arrangement order of the tasks to be executed, wherein the arrangement order between the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and/or the corresponding set priority.
  • the determination module 52 is further configured to create a running record of the current task to be executed in a second preset database, wherein the running record of each task created in the second preset database can be provided to the client for display.
  • the sending module 54 is specifically configured to splice the task parameters into command line information to be executed corresponding to the task; and send the command line information to the target computing node through the secure shell protocol.
  • the sending module 54 is further configured to obtain a key based on the current time information and the identification of the local computer device; after encrypting the command line information using the key, it is sent to the target computing node through the secure shell protocol, so that the target computing node can infer the key based on the time information of receiving the command line information and the identification of each computer device that has established trust with the target computing node, and use the key to decrypt the command line information.
  • the acquisition module 53 is further configured to obtain the number of computing nodes to be updated in response to an update setting instruction of the distributed system
  • the determination module 52 is further configured to determine the computing nodes to be started or shut down according to the number of computing nodes based on the computing performance information and the real-time task processing status of each computing node in the distributed system.
  • the monitoring module 51 is specifically configured to monitor the task processing status of each computing node in the distributed system at a preset time interval.
  • this embodiment provides a task processing device, which is applied to a first computer device.
  • the device includes: a sending module 61 , a receiving module 62 , and an executing module 63 .
  • a sending module 61 configured to send the current task processing status of the first computer device to the second computer device;
  • the receiving module 62 is configured to receive a task execution instruction sent by the second computer device, wherein the task execution instruction is sent by the second computer device when the task processing state is determined to be an idle state, and the task execution instruction carries a task parameter of the current task to be executed;
  • the execution module 63 is configured to execute the current task to be executed according to the task parameters.
  • the execution module 63 is specifically configured to execute the current task to be executed based on the business data corresponding to the task parameters in a preset storage location, and the preset storage location stores business data corresponding to different task parameters.
  • the receiving module 62 is specifically configured to receive command line information sent by the second computer device through a secure shell protocol, wherein the command line information is obtained by splicing according to the task parameters;
  • the execution module 63 is specifically configured to execute the current task to be executed by executing the command line information.
  • the receiving module 62 is further configured to receive the encrypted command line information sent by the second computer device through the secure shell protocol; determine the key required to decrypt the command line information based on the time information of receiving the command line information and the identification of each computer device that has established trust with the first computer device; and use the key to decrypt and obtain the command line information.
  • this embodiment provides a task processing device, which is applied to a second computer device.
  • the device includes: a receiving module 71 , an acquiring module 72 , and a sending module 73 .
  • the receiving module 71 is configured to request the first computer device to obtain the current task processing status of the first computer device;
  • the acquisition module 72 is configured to acquire the task parameters of the current task to be executed if the task processing state is an idle state;
  • the sending module 73 is configured to send a task execution instruction to the first computer device according to the task parameters, wherein the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.
  • the acquisition module 72 is specifically configured to extract the task parameters of the top-ranked tasks to be executed from the first preset database according to the arrangement order of the tasks to be executed, wherein the arrangement order between the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and/or the corresponding set priority.
  • the acquisition module 72 is also configured to create an operation record of the current task to be executed in a second preset database; receive a viewing instruction for the task operation record; and determine a display result corresponding to the viewing instruction based on the operation records of each task created in the second preset database.
  • the sending module 73 is specifically configured to splice the task parameters into command line information to be executed corresponding to the task; and send the command line information to the first computer device through the secure shell protocol.
  • the sending module 73 is further configured to obtain a key according to current time information and an identification of a local computer device; after encrypting the command line information using the key, the key is sent to the first computer device via a secure shell protocol.
  • this embodiment further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method shown in FIG. 1 and FIG. 2 is implemented.
  • this embodiment further provides another computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method shown in FIG. 6 is implemented.
  • this embodiment further provides another computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method shown in FIG. 7 is implemented.
  • the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.
  • a non-volatile storage medium which can be a CD-ROM, USB flash drive, mobile hard disk, etc.
  • a computer device which can be a personal computer, server, or network device, etc.
  • the embodiment of the present application also provides an electronic device, which may specifically include a personal computer, a server, a network computer and other devices, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above-mentioned method as shown in Figures 1 and 2.
  • the embodiment of the present application also provides another electronic device, which may specifically include a personal computer, a server, a network computer and other devices, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above method as shown in Figure 6.
  • the embodiment of the present application also provides another electronic device, which may specifically include a personal computer, a server, a network computer and other devices, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above method as shown in Figure 7.
  • the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc.
  • the user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc.
  • the network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
  • the storage medium may also include an operating system and a network communication module.
  • the operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and/or programs.
  • the network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
  • the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware.
  • the degree of freedom of asynchronous task scheduling can be effectively improved, and computing nodes deployed on multiple machines can be mobilized to perform distributed processing of asynchronous tasks.
  • a distributed system for realizing free scheduling of asynchronous tasks is provided, and a large number of asynchronous tasks can be distributedly processed by computing nodes deployed on multiple machines, which can effectively improve the problem of tight computing resources on a single machine and improve the efficiency of asynchronous task processing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Multi Processors (AREA)

Abstract

本申请涉及一种任务处理方法、装置及电子设备，涉及数据处理技术领域，其中方法包括：首先监测分布式系统中的每个计算节点的任务处理状态，分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；再确定任务处理状态为空闲状态的目标计算节点；然后根据当前待执行任务的任务参数，向目标计算节点发送任务执行指令，以使得目标计算节点基于预设存储位置中与任务参数相应的业务数据，执行当前待执行任务。通过应用本申请的技术方案，提供了一种实现异步任务自由调度的分布式系统，可有效改善单机计算资源紧张的问题，提高了异步任务处理的效率。

Description

任务处理方法、装置及电子设备 技术领域
本申请涉及数据处理技术领域，具体涉及一种任务处理方法、装置及电子设备。
背景技术
随着移动互联网、云计算、大数据时代的到来，需要处理的信息任务量越来越多，其中，异步任务处理一直是开发者关注的问题。
目前，各种异步任务处理的框架仅能满足在单机上的异步任务处理，如在单台机器上通过调用多个线程来运行多个计算节点的异步任务处理系统。
然而，统一将大量的异步任务都堆叠在单机上运行，会使计算资源紧张。并且受到单机性能的限制，导致处理任务的速度会被限制，进而会影响异步任务处理的效率。
发明内容
有鉴于此，本申请提供了一种任务处理方法、装置及电子设备，主要目的在于改善目前现有技术中由于统一将大量的异步任务都堆叠在单机上运行，不仅会使计算资源紧张，并且还会影响异步任务处理的效率的技术问题。
第一方面，本申请提供了一种任务处理方法，包括：
监测分布式系统中的每个计算节点的任务处理状态，所述分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；
确定所述任务处理状态为空闲状态的目标计算节点；
获取当前待执行任务的任务参数；
根据所述任务参数，向所述目标计算节点发送任务执行指令，以使得所述目标计算节点基于所述预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务。
第二方面，本申请提供了一种任务处理方法，应用于第一计算机设备，所述方法包括：
向第二计算机设备发送所述第一计算机设备当前的任务处理状态；
接收所述第二计算机设备发送的任务执行指令，所述任务执行指令是所述第二计算机设备在确定所述任务处理状态为空闲状态时发送的，所述任务执行指令中携带有当前待执行任务的任务参数；
根据所述任务参数，执行所述当前待执行任务。
第三方面，本申请提供了一种任务处理方法，应用于第二计算机设备，所述方法包括：
向第一计算机设备请求获取所述第一计算机设备当前的任务处理状态；
若所述任务处理状态为空闲状态，则获取当前待执行任务的任务参数；
根据所述任务参数，向所述第一计算机设备发送任务执行指令，所述任务执行指令用于指示所述第一计算机设备根据所述任务参数执行所述当前待执行任务。
第四方面，本申请提供了一种任务处理装置，包括：
监测模块，被配置为监测分布式系统中的每个计算节点的任务处理状态，所述分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；
确定模块，被配置为确定所述任务处理状态为空闲状态的目标计算节点；
获取模块，被配置为获取当前待执行任务的任务参数；
发送模块，被配置为根据所述任务参数，向所述目标计算节点发送任务执行指令，以使得所述目标计算节点基于所述预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务。
第五方面，本申请提供了一种任务处理装置，应用于第一计算机设备，所述装置包括：
发送模块，被配置为向第二计算机设备发送所述第一计算机设备当前的任务处理状态；
接收模块，被配置为接收所述第二计算机设备发送的任务执行指令，所述任务执行指令是所述第二计算机设备在确定所述任务处理状态为空闲状态时发送的，所述任务执行指令中携带有当前待执行任务的任务参数；
执行模块，被配置为根据所述任务参数，执行所述当前待执行任务。
第六方面，本申请提供了一种任务处理装置，应用于第二计算机设备，所述装置包括：
接收模块，被配置为向第一计算机设备请求获取所述第一计算机设备当前的任务处理状态；
获取模块，被配置为若所述任务处理状态为空闲状态，则获取当前待执行任务的任务参数；
发送模块，被配置为根据所述任务参数，向所述第一计算机设备发送任务执行指令，所述任务执行指令用于指示所述第一计算机设备根据所述任务参数执行所述当前待执行任务。
第七方面，本申请提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现第一方面或第二方面或第三方面所述的任务处理方法。
第八方面，本申请提供了一种电子设备，包括存储介质、处理器及存储在存储介质上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现第一方面或第二方面或第三方面所述的任务处理方法。
借由上述技术方案，本申请提供的一种任务处理方法、装置及电子设备，与目前现有技术相比，本申请可以有效提高异步任务调度的自由度，可以调动部署于多台机器的计算节点进行异步任务的分布式处理。具体首先监测分布式系统中的每个计算节点的任务处理状态，分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；再确定任务处理状态为空闲状态的目标计算节点；然后根据当前待执行任务的任务参数，向目标计算节点发送任务执行指令，以使得目标计算节点基于预设存储位置中与该任务参数相应的业务数据，执行当前待执行任务。通过应用本申请的技术 方案，提供了一种实现异步任务自由调度的分布式系统，可将大量的异步任务通过部署于多台机器的计算节点进行分布式处理，可有效改善单机计算资源紧张的问题，提高了异步任务处理的效率。
上述说明仅是本申请技术方案的概述，为了能够更清楚了解本申请的技术手段，而可依照说明书的内容予以实施，并且为了让本申请的上述和其它目的、特征和优点能够更明显易懂，以下特举本申请的具体实施方式。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1示出了本申请实施例提供的一种任务处理方法的流程示意图；
图2示出了本申请实施例提供的另一种任务处理方法的流程示意图；
图3示出了本申请实施例提供的一种分布式系统架构的结构示意图；
图4示出了本申请实施例提供的一种前端显示效果示例的示意图；
图5示出了本申请实施例提供的一种示例架构的示意图；
图6示出了本申请实施例提供的又一种任务处理方法的流程示意图；
图7示出了本申请实施例提供的再一种任务处理方法的流程示意图；
图8示出了本申请实施例提供的一种任务处理装置的结构示意图；
图9示出了本申请实施例提供的另一种任务处理装置的结构示意图；
图10示出了本申请实施例提供的又一种任务处理装置的结构示意图。
具体实施方式
为了能够更清楚地理解本申请的上述目的、特征和优点，下面将对本申请的方案进行进一步描述。需要说明的是，在不冲突的情况下，本申请的实施例及实施例中的特征可以相互组合。
为了能高效调度运行于不同服务器的异步任务来解决单机的性能瓶颈,并使异步任务流程透明可控。本实施例提供了一种任务处理方法，如图1所示，该方法包括：
步骤101、监测分布式系统中的每个计算节点的任务处理状态。
分布式系统中的计算节点可为不同的计算机设备(如网络计算机、服务器等设备)，且不同的计算节点均能读取预设存储位置中的业务数据。
本实施例采用分布式系统的去中心化思想，将所有计算节点分散到各个计算机设备中,方便开发维护。同时，针对于本分布式系统的使用，有一个统一的调度平台(Monitor)作为管理者角色去对异步任务进行分发。调度平台可以是一个具有前后台的程序，且在分布 式系统中它是唯一的，可以部署于同网段的任意一台计算机设备上，由它负责任务的调配，用户在前台提交任务，后台将用户提交的任务信息记录到数据库中承担监控，分发这些任务的功能等。
预设存储位置具体可为数据库、文件夹或数据表等，可用于存放业务相关的数据，如可存放所有的测序数据,测序仪器产生的测序数据也会转移到该预设存储位置中进行存放。预设存储位置的路径会挂载到各个计算机设备上，使得不同的计算节点均能读取预设存储位置中的业务数据(执行任务所需的数据)。
步骤102、确定任务处理状态为空闲状态的目标计算节点。
例如，分布式系统中存在5个计算节点，分别为计算节点A、计算节点B、计算节点C、计算节点D、计算节点E。其中，只有计算节点A的任务处理状态为空闲状态，而其余4个计算节点为任务处理中的状态，则可将计算节点A确定为该目标计算节点。
步骤103、获取当前待执行任务的任务参数。
任务参数可为任务执行所参照的参数(如任务被执行主体、任务执行动作、任务执行目的等)，在后续任务执行时，目标计算节点会根据该任务参数来执行。
步骤104、根据当前待执行任务的任务参数，向目标计算节点发送任务执行指令。
进一步的，以使得目标计算节点基于预设存储位置中与该任务参数相应的业务数据，执行当前待执行任务。
例如，任务处理状态为空闲状态的计算节点A，根据接收到的任务执行指令，解析得到需要执行的任务信息(包括任务参数)，然后依据预设存储位置的路径，读取预设存储位置中与该任务参数相应的业务数据，进而执行分配给该计算节点A的当前待执行任务。
与目前现有技术相比，本实施例可以有效提高异步任务调度的自由度，可以调动部署于多台机器的计算节点进行异步任务的分布式处理。具体首先监测分布式系统中的每个计算节点的任务处理状态，分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；再确定任务处理状态为空闲状态的目标计算节点；然后根据当前待执行任务的任务参数，向目标计算节点发送任务执行指令，以使得目标计算节点基于预设存储位置中与该任务参数相应的业务数据，执行当前待执行任务。通过应用本实施例的技术方案，提供了一种实现异步任务自由调度的分布式系统，可将大量的异步任务通过部署于多台机器的计算节点进行分布式处理，可有效改善单机计算资源紧张的问题，提高了异步任务处理的效率。
进一步的，作为上述实施例的细化和扩展，为了完整说明本实施例方法的具体实现过程，本实施例提供了如图2所示的具体方法，该方法包括：
步骤201、按照预设时间间隔，监测分布式系统中的每个计算节点的任务处理状态。
分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据。
为了提高流畅性，可将异步任务放在后台运行。在python语言中比较常见且被广泛使 用的就是celery框架，结合第三方消息服务(Broker)来实现任务调度，但是当需要深入业务的时候，就会发现该celery框架的局限性，不支持python的原生进程池，即使使用兼容的billboard进程池，也仍然会有内存泄漏的问题，而且任务的代码维护会牵一发而动全身，维护起来也很困难，对任务也很难做到灵活的增删改查，最主要的一点是无法对分布在多台计算机设备的任务节点进行调度。因此如何使用统一调度平台来调度分布在多台计算机设备的任务节点，完全自由的管理任务是一个值得思考和解决的技术问题。
为了解决上述技术问题，本实施例可使用redis数据库(仅示例性说明，还可使用其他持久化存储的数据库)和定时轮询脚本实现了异步框架中消息中间件的作用，使异步任务的调度从celery框架中解放出来，从而使得开发人员可以通过对redis数据库进行处理以及修改脚本，达到适配各种数据分析场景的目的。
具体的，本实施例可提出一种实现异步任务自由调度的分布式系统，该分布式系统是一个硬件或软件系统分布在不同的计算机设备上，彼此之间可通过消息传递进行通信和协调的系统。在本分布式系统中，一组独立的计算机设备展现给用户的是一个统一的整体。系统拥有多种通用的物理和逻辑资源，可以动态的分配任务，分散的物理和逻辑资源通过计算机网络实现信息交换。
如图3所示，本分布式系统可包括：数据中心(预设存储位置)、任务消息中间件、调度平台(Monitor)、计算节点、调度层前端页面、调度层后台等。
其中，数据中心(预设存储位置，如NFS server)会存放所有的业务数据(如测序数据，测序仪器产生的测序数据也会转移到数据中心存放)，同时，如图3所示，业务数据存放的路径会挂载到各个计算节点，这样即使服务部署在其它计算机设备，也能读取数据中心存放的业务数据。
任务消息中间件：数据库选型可为redis数据库，需要说明的是，根据实际需求也可以选用其他持久化存储的数据库，本实施例不做限定。用户在前端页面选择相应参数和数据名称之后创建的每条任务信息都会存入到redis数据库中，同时redis数据库中也会存放每种异步任务的参数配置。具体可包括以下几种：
配置信息数据库(ConfigDB)：redis数据库，存放各个任务的参数配置，具体可包含参数选项、参数默认值、每个任务的最大worker数目等等。
待处理队列数据库(第一预设数据库，PendingQueueDB)：redis数据库，任何创建的任务信息首先存放到这个待处理队列数据库中等待处理。
运行状态数据库(第二预设数据库，RunningBufferDB)：redis数据库，记录正在运行的任务，如果任务进入到了该运行状态数据库，说明已经处于运行中状态。
调度平台(调度者，Monitor)：管理任务队列的角色，实现形式可以为定时脚本，主要功能是将第一预设数据库中排队的待执行任务推送给空闲的计算节点来执行。
计算节点对应的工作者(worker)：实际上异步任务的真正执行者，具体可以是部署在某个计算机设备(如服务器)上的一个package，可通过远程安全外壳协议(Secure Shell  Protocol，SSH)命令行调用的形式完成任务逻辑。worker相当于是软件虚拟设备，也就是部署在单个计算机设备(单个计算节点)上的某一种数据分析功能的package。可认为是响应这些ssh命令行，最后真正去执行异步任务的一个后台程序。
调度层前台页面：例如，用户可在页面上选择测序数据以及相应参数之后点击提交任务，前台将参数发送给后台。前台还会展示目前所有排队/运行的任务信息，也提供相关界面供操作者对任务队列做增删改查等。
调度层后台：调度任务的后台可采用的是python的Django后端框架，提供诸多接口给到前端页面，完成对任务信息的修改，以及任务队列的调换，具体可以是对redis数据库进行读、写、改等多种操作。
基于上述分布式系统，具体执行本实施例提供的任务处理方法，其中本实施例方法的执行主体可为该调度平台，可按照预设时间间隔(定时为每隔1s进行一次监测)，监测分布式系统中的每个计算节点的任务处理状态，进而可及时发现那些任务处理状态为空闲状态的目标计算节点，并向其分配第一预设数据库中排队的待执行任务。
步骤202、确定任务处理状态为空闲状态的目标计算节点。
步骤203、从第一预设数据库中，按照待执行任务之间的排列顺序，提取排名靠前的待执行任务的任务参数。
其中，第一预设数据库中的各个待执行任务之间的排列顺序可以是按照待执行任务的创建时间和/或对应设置的优先级确定的。在从第一预设数据库中提取待执行任务的任务参数之后，该待执行任务和该待执行任务的任务参数会在第一预设数据库中被删除，以节省第一预设数据库的存储空间。
例如，待执行任务的创建时间越早，其在第一预设数据库中的排列顺序越靠前(排名靠前)，待执行任务的创建时间越晚，其在第一预设数据库中的排列顺序越靠后(排名靠后)。再例如，用户可根据实际需求设置待执行任务的优先级，待执行任务的优先级越高，其在第一预设数据库中的排列顺序越靠前(排名靠前)，待执行任务的优先级越低，其在第一预设数据库中的排列顺序越靠后(排名靠后)。
步骤204、将提取的任务参数拼接出任务对应所要执行的命令行信息。
步骤205、通过安全外壳协议向目标计算节点发送命令行信息。
目标计算节点上安装有软件工具，目标计算节点通过软件工具响应于命令行信息执行当前待执行任务。
具体的，目标计算节点通过软件工具响应ssh命令行，在后台程序中执行异步任务，其中，软件工具可为具有数据分析功能的软件虚拟设备，软件工具即worker。
例如，在确定计算节点A空闲的前提下，将PendingQueueDB(第一预设数据库)中排名靠前的待执行任务的任务参数提取出来，拼接出任务对应所要执行的命令行，然后通过SSH的形式发送到计算节点A，以使得计算节点A基于预设存储位置中与该任务参数相应的业务数据，执行当前待执行任务。
为了保证数据传输的安全性，可选的，步骤205具体可包括：根据当前时间信息和本地计算机设备的标识(如终端的MAC地址、IP地址、主板ID号等)，获取密钥(可将当前时间信息和本地计算机设备的标识组合后，计算哈希值或者MD5值等，得到密钥)；在利用密钥对命令行信息进行加密后，通过安全外壳协议发送给目标计算节点，以使得目标计算节点根据接收到命令行信息的时间信息和与目标计算节点建立信任的各个计算机设备的标识推算出该密钥，并利用该密钥解密得到命令行信息。
例如，目标计算节点根据接收到命令行信息的时间信息(如10:00:00)，向前获取3秒内的时间(如9:59:59，9:59:58，9:59:57)，然后将这3秒内的时间信息和与目标计算节点建立信任的各个计算机设备的标识分别进行组合，计算出哈希值或者MD5值，然后逐个使用哈希值或者MD5值去尝试解密该加密的命令行信息，直到解密成功，得到解密后的命令行信息。
进一步可选的，本实施例方法还可包括：在第二预设数据库(RunningBufferDB)中创建当前待执行任务的运行记录，其中，第二预设数据库中创建的各个任务的运行记录能够提供给客户端进行显示，以便用户可以实时查看到任务处理的进度，提升用户的使用体验。
为了满足更多的需求，进一步可选的，本实施例方法还可包括：响应于分布式系统的更新设置指令，获取更新设置的计算节点个数；然后根据分布式系统中各个计算节点的计算性能信息和实时的任务处理状态，按照该计算节点个数确定启动或关闭的计算节点。
例如，用户可以设置分布式系统的计算节点个数，具体可以在分布式系统的运行前或者是运行中进行设置。在分布式系统的运行前，根据用户设置的计算节点个数，可以优先启动那些计算性能较好的计算节点；而在分布式系统的运行中，根据用户设置的计算节点个数，可以按照计算性能信息和实时的任务处理状态，启动或关闭合适的计算节点，以达到分布式系统整体计算性能最优。
为了说明上述实施例的具体实现过程，给出如下应用场景，但不限于此：
以basecall任务为例，该basecall任务具体将识别出来的读取结果(reads)信号转变为位点(base)。如图4所示，在前端页面选择需要做分析的文件和任务参数，点击添加按键(ADD)创建任务。此时任务记录会出现在PendingQueueDB(第一预设数据库)，因为计算节点数目的限制，众多提交的任务会根据提交的先后次序进行排队处理，可以观察到刚刚提交的分析任务处在队列第169的位置。那么在堆栈结构的redis数据库中，所处的位置即为第一个。如果任务优先级较高，可以通过前端页面调用后台接口，对任务队列进行处理，进而提升某些任务的优先级位置。如任务从排队的末尾被调换到了第一位，在任务处于排队的情况下，仍然可以通过对redis数据库的改动，改变任务位置和任务参数。当计算节点存在空闲时，会将堆栈结构中的任务提取出来，将其中参数拼接成相应的SSH远程命令行，发送到该空闲的计算节点执行，同时也会在RunningBuffer(第二预设数据库)新建该条任务记录，用于跟踪进度。其中整体的示例图，可如图5所示。
本实施例将单机任务调度模式转变成了分布式系统的调度模式，可以充分利用到多台 计算设备的计算节点，能有效解决在单机性能瓶颈场景下难以解决的问题。
进一步的，以单个计算节点的处理过程为例，提供了如图6所示的方法，可应用于第一计算机设备(单个计算节点)，该方法包括：
步骤301、第一计算机设备向第二计算机设备发送第一计算机设备当前的任务处理状态。
第一计算机设备可为分布式系统中的任一个计算节点，而第二计算机设备为部署有调度平台的计算机设备。
步骤302、第一计算机设备接收第二计算机设备发送的任务执行指令。
该任务执行指令是第二计算机设备在确定该任务处理状态为空闲状态时发送的，该任务执行指令中携带有当前待执行任务的任务参数。
为了提高数据传输安全性，可选的，步骤302具体可包括：第一计算机设备接收第二计算机设备通过安全外壳协议发送的命令行信息，命令行信息是根据任务参数拼接得到的；
为了进一步保证数据传输的安全性，在一些示例中，第一计算机设备接收第二计算机设备通过安全外壳协议发送的命令行信息，具体可包括：第一计算机设备接收第二计算机设备通过安全外壳协议发送的加密后的命令行信息；然后根据接收到命令行信息的时间信息和与第一计算机设备建立信任的各个计算机设备的标识，确定解密该命令行信息所需的密钥；进而可利用该密钥解密得到命令行信息。
具体的解密过程可参见步骤205中的对应描述，在此不再赘述。
步骤303、第一计算机设备根据任务参数，执行当前待执行任务。
基于步骤302的可选方式，相应的，步骤303具体可包括：通过执行命令行信息实现执行当前待执行任务。
示例性的，步骤303具体可包括：基于预设存储位置中与该任务参数相应的业务数据，执行当前待执行任务，其中，该预设存储位置中保存有不同的任务参数分别对应的业务数据。
本实施例将单机任务调度模式转变成了分布式系统的调度模式，可以充分利用到多台计算设备的计算节点，能有效解决在单机性能瓶颈场景下难以解决的问题。
进一步的，以调度平台的处理过程为例，提供了如图7所示的方法，可应用于第二计算机设备(可以是部署了调度平台的计算机设备)，该方法包括：
步骤401、第二计算机设备向第一计算机设备请求获取第一计算机设备当前的任务处理状态。
第一计算机设备可为分布式系统中的任一个计算节点，而第二计算机设备为部署有调度平台的计算机设备。
步骤402、若第一计算机设备当前的任务处理状态为空闲状态，则第二计算机设备获取当前待执行任务的任务参数。
可选的，步骤402中获取当前待执行任务的任务参数，具体可包括：从第一预设数据 库中，按照待执行任务之间的排列顺序，提取排名靠前的待执行任务的任务参数，其中，第一预设数据库中的各个待执行任务之间的排列顺序是按照待执行任务的创建时间和/或对应设置的优先级确定的。
例如，待执行任务的创建时间越早，其在第一预设数据库中的排列顺序越靠前(排名靠前)，待执行任务的创建时间越晚，其在第一预设数据库中的排列顺序越靠后(排名靠后)。再例如，用户可根据实际需求设置待执行任务的优先级，待执行任务的优先级越高，其在第一预设数据库中的排列顺序越靠前(排名靠前)，待执行任务的优先级越低，其在第一预设数据库中的排列顺序越靠后(排名靠后)。
步骤403、第二计算机设备根据任务参数，向第一计算机设备发送任务执行指令。
任务执行指令用于指示第一计算机设备根据任务参数执行当前待执行任务。
为了提高数据传输安全性，可选的，步骤403具体可包括：将任务参数拼接出任务对应所要执行的命令行信息；通过安全外壳协议向第一计算机设备发送命令行信息。
为了进一步保证数据传输的安全性，在一些示例中，上述通过安全外壳协议向第一计算机设备发送命令行信息，具体可包括：根据当前时间信息和本地计算机设备的标识，获取密钥；在利用该密钥对命令行信息进行加密后，通过安全外壳协议发送给第一计算机设备。具体的解密过程可参见步骤205和302中的对应描述，在此不再赘述。
进一步可选的，本实施例方法还可包括：在第二预设数据库中创建当前待执行任务的运行记录；接收任务运行记录的查看指令；根据第二预设数据库中创建的各个任务的运行记录，确定与该查看指令对应的显示结果。以便用户可以实时查看到任务处理的进度，提升用户的使用体验。
本实施例将单机任务调度模式转变成了分布式系统的调度模式，可以充分利用到多台计算设备的计算节点，能有效解决在单机性能瓶颈场景下难以解决的问题。
进一步的，作为图1和图2所示方法的具体实现，本实施例提供了一种任务处理装置，如图8所示，该装置包括：监测模块51、确定模块52、获取模块53、发送模块54。
监测模块51，被配置为监测分布式系统中的每个计算节点的任务处理状态，所述分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；
确定模块52，被配置为确定所述任务处理状态为空闲状态的目标计算节点；
获取模块53，被配置为获取当前待执行任务的任务参数；
发送模块54，被配置为根据所述任务参数，向所述目标计算节点发送任务执行指令，以使得所述目标计算节点基于所述预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务。
在具体的应用场景中，获取模块53，具体被配置为从第一预设数据库中，按照待执行任务之间的排列顺序，提取排名靠前的待执行任务的任务参数，其中，所述第一预设数据库中的各个待执行任务之间的排列顺序是按照待执行任务的创建时间和/或对应设置的优 先级确定的。
在具体的应用场景中，确定模块52，还被配置为在第二预设数据库中创建所述当前待执行任务的运行记录，其中，所述第二预设数据库中创建的各个任务的运行记录能够提供给客户端进行显示。
在具体的应用场景中，发送模块54，具体被配置为将所述任务参数拼接出任务对应所要执行的命令行信息；通过安全外壳协议向所述目标计算节点发送所述命令行信息。
在具体的应用场景中，发送模块54，具体还被配置为根据当前时间信息和本地计算机设备的标识，获取密钥；在利用所述密钥对所述命令行信息进行加密后，通过安全外壳协议发送给所述目标计算节点，以使得所述目标计算节点根据接收到所述命令行信息的时间信息和与所述目标计算节点建立信任的各个计算机设备的标识推算出所述密钥，并利用所述密钥解密得到所述命令行信息。
在具体的应用场景中，获取模块53，还被配置为响应于所述分布式系统的更新设置指令，获取更新设置的计算节点个数；
确定模块52，还被配置为根据所述分布式系统中各个计算节点的计算性能信息和实时的任务处理状态，按照所述计算节点个数确定启动或关闭的计算节点。
在具体的应用场景中，监测模块51，具体被配置为按照预设时间间隔，监测所述分布式系统中的每个计算节点的任务处理状态。
需要说明的是，本实施例提供的一种任务处理装置所涉及各功能单元的其它相应描述，可以参考图1和图2中的对应描述，在此不再赘述。
进一步的，作为图6所示方法的具体实现，本实施例提供了一种任务处理装置，应用于第一计算机设备，如图9所示，该装置包括：发送模块61、接收模块62、执行模块63。
发送模块61，被配置为向第二计算机设备发送所述第一计算机设备当前的任务处理状态；
接收模块62，被配置为接收所述第二计算机设备发送的任务执行指令，所述任务执行指令是所述第二计算机设备在确定所述任务处理状态为空闲状态时发送的，所述任务执行指令中携带有当前待执行任务的任务参数；
执行模块63，被配置为根据所述任务参数，执行所述当前待执行任务。
在具体的应用场景中，执行模块63，具体被配置为基于预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务，所述预设存储位置中保存有不同的任务参数分别对应的业务数据。
在具体的应用场景中，接收模块62，具体被配置为接收所述第二计算机设备通过安全外壳协议发送的命令行信息，所述命令行信息是根据所述任务参数拼接得到的；
执行模块63，具体被配置为通过执行所述命令行信息实现执行所述当前待执行任务。
在具体的应用场景中，接收模块62，具体还被配置为接收所述第二计算机设备通过安全外壳协议发送的加密后的所述命令行信息；根据接收到所述命令行信息的时间信息和与 所述第一计算机设备建立信任的各个计算机设备的标识，确定解密所述命令行信息所需的密钥；利用所述密钥解密得到所述命令行信息。
需要说明的是，本实施例提供的一种任务处理装置所涉及各功能单元的其它相应描述，可以参考图6中的对应描述，在此不再赘述。
进一步的，作为图7所示方法的具体实现，本实施例提供了一种任务处理装置，应用于第二计算机设备，如图10所示，该装置包括：接收模块71、获取模块72、发送模块73。
接收模块71，被配置为向第一计算机设备请求获取所述第一计算机设备当前的任务处理状态；
获取模块72，被配置为若所述任务处理状态为空闲状态，则获取当前待执行任务的任务参数；
发送模块73，被配置为根据所述任务参数，向所述第一计算机设备发送任务执行指令，所述任务执行指令用于指示所述第一计算机设备根据所述任务参数执行所述当前待执行任务。
在具体的应用场景中，获取模块72，具体被配置为从第一预设数据库中，按照待执行任务之间的排列顺序，提取排名靠前的待执行任务的任务参数，其中，所述第一预设数据库中的各个待执行任务之间的排列顺序是按照待执行任务的创建时间和/或对应设置的优先级确定的。
在具体的应用场景中，获取模块72，还被配置为在第二预设数据库中创建所述当前待执行任务的运行记录；接收任务运行记录的查看指令；根据所述第二预设数据库中创建的各个任务的运行记录，确定与所述查看指令对应的显示结果。
在具体的应用场景中，发送模块73，具体被配置为将所述任务参数拼接出任务对应所要执行的命令行信息；通过安全外壳协议向所述第一计算机设备发送所述命令行信息。
在具体的应用场景中，发送模块73，具体还被配置为根据当前时间信息和本地计算机设备的标识，获取密钥；在利用所述密钥对所述命令行信息进行加密后，通过安全外壳协议发送给所述第一计算机设备。
需要说明的是，本实施例提供的一种任务处理装置所涉及各功能单元的其它相应描述，可以参考图7中的对应描述，在此不再赘述。
基于上述如图1和图2所示方法，相应的，本实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述如图1和图2所示的方法。基于上述如图6所示方法，相应的，本实施例还提供了另一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述如图6所示的方法。基于上述如图7所示方法，相应的，本实施例还提供了又一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述如图7所示的方法。
基于这样的理解，本申请的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中，包括若干指 令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施场景的方法。
基于上述如图1和图2所示的方法，以及图8所示的虚拟装置实施例，为了实现上述目的，本申请实施例还提供了一种电子设备，具体可包括个人计算机、服务器、网络计算机等设备，该设备包括存储介质和处理器；存储介质，用于存储计算机程序；处理器，用于执行计算机程序以实现上述如图1和图2所示的方法。
基于上述如图6所示的方法，以及图9所示的虚拟装置实施例，为了实现上述目的，本申请实施例还提供了另一种电子设备，具体可包括个人计算机、服务器、网络计算机等设备，该设备包括存储介质和处理器；存储介质，用于存储计算机程序；处理器，用于执行计算机程序以实现上述如图6所示的方法。
基于上述如图7所示的方法，以及图10所示的虚拟装置实施例，为了实现上述目的，本申请实施例还提供了又一种电子设备，具体可包括个人计算机、服务器、网络计算机等设备，该设备包括存储介质和处理器；存储介质，用于存储计算机程序；处理器，用于执行计算机程序以实现上述如图7所示的方法。
可选的，上述实体设备还可以包括用户接口、网络接口、摄像头、射频(Radio Frequency，RF)电路，传感器、音频电路、WI-FI模块等等。用户接口可以包括显示屏(Display)、输入单元比如键盘(Keyboard)等，可选用户接口还可以包括USB接口、读卡器接口等。网络接口可选的可以包括标准的有线接口、无线接口(如WI-FI接口)等。
本领域技术人员可以理解，本实施例提供的上述实体设备结构并不构成对该实体设备的限定，可以包括更多或更少的部件，或者组合某些部件，或者不同的部件布置。
存储介质中还可以包括操作系统、网络通信模块。操作系统是管理上述实体设备硬件和软件资源的程序，支持信息处理程序以及其它软件和/或程序的运行。网络通信模块用于实现存储介质内部各组件之间的通信，以及与信息处理实体设备中其它硬件和软件之间通信。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到本申请可以借助软件加必要的通用硬件平台的方式来实现，也可以通过硬件实现。通过应用本实施例的方案，可以有效提高异步任务调度的自由度，可以调动部署于多台机器的计算节点进行异步任务的分布式处理。提供了一种实现异步任务自由调度的分布式系统，可将大量的异步任务通过部署于多台机器的计算节点进行分布式处理，可有效改善单机计算资源紧张的问题，提高了异步任务处理的效率。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固 有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本申请的具体实施方式，使本领域技术人员能够理解或实现本申请。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本申请的精神或范围的情况下，在其它实施例中实现。因此，本申请将不会被限制于本文所述的这些实施例，而是要符合与本文所申请的原理和新颖特点相一致的最宽的范围。

Claims (19)

  1. 一种任务处理方法，其特征在于，包括：
    监测分布式系统中的每个计算节点的任务处理状态，所述分布式系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；
    确定所述任务处理状态为空闲状态的目标计算节点；
    获取当前待执行任务的任务参数；
    根据所述任务参数，向所述目标计算节点发送任务执行指令，以使得所述目标计算节点基于所述预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务。
  2. 根据权利要求1所述的方法，其特征在于，所述获取当前待执行任务的任务参数，包括：
    从第一预设数据库中，按照待执行任务之间的排列顺序，提取排名靠前的待执行任务的任务参数，其中，所述第一预设数据库中的各个待执行任务之间的排列顺序是按照待执行任务的创建时间和/或对应设置的优先级确定的。
  3. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    在第二预设数据库中创建所述当前待执行任务的运行记录，其中，所述第二预设数据库中创建的各个任务的运行记录能够提供给客户端进行显示。
  4. 根据权利要求1所述的方法，其特征在于，所述根据所述任务参数，向所述目标计算节点发送任务执行指令，包括：
    将所述任务参数拼接出任务对应所要执行的命令行信息；
    通过安全外壳协议向所述目标计算节点发送所述命令行信息。
  5. 根据权利要求4所述的方法，其特征在于，所述通过安全外壳协议向所述目标计算节点发送所述命令行信息，包括：
    根据当前时间信息和本地计算机设备的标识，获取密钥；
    在利用所述密钥对所述命令行信息进行加密后，通过安全外壳协议发送给所述目标计算节点，以使得所述目标计算节点根据接收到所述命令行信息的时间信息和与所述目标计算节点建立信任的各个计算机设备的标识推算出所述密钥，并利用所述密钥解密得到所述命令行信息。
  6. 根据权利要求4所述的方法，其特征在于，所述目标计算节点上安装有软件工具，所述目标计算节点通过所述软件工具响应于所述命令行信息执行所述当前待执行任务。
  7. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    响应于所述分布式系统的更新设置指令，获取更新设置的计算节点个数；
    根据所述分布式系统中各个计算节点的计算性能信息和实时的任务处理状态，按照所述计算节点个数确定启动或关闭的计算节点。
  8. 根据权利要求1所述的方法，其特征在于，所述监测分布式系统中的每个计算节点 的任务处理状态，包括：
    按照预设时间间隔，监测所述分布式系统中的每个计算节点的任务处理状态。
  9. 一种任务处理方法，其特征在于，应用于第一计算机设备，所述方法包括：
    向第二计算机设备发送所述第一计算机设备当前的任务处理状态；
    接收所述第二计算机设备发送的任务执行指令，所述任务执行指令中携带有当前待执行任务的任务参数；
    基于预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务，所述预设存储位置中保存有不同的任务参数分别对应的业务数据。
  10. 根据权利要求9所述的方法，其特征在于，所述接收所述第二计算机设备发送的任务执行指令，包括：
    接收所述第二计算机设备通过安全外壳协议发送的命令行信息，所述命令行信息是根据所述任务参数拼接得到的；
    所述根据所述任务参数，执行所述当前待执行任务，包括：
    通过执行所述命令行信息实现执行所述当前待执行任务。
  11. 一种任务处理方法，其特征在于，应用于第二计算机设备，所述方法包括：
    向第一计算机设备请求获取所述第一计算机设备当前的任务处理状态；
    若所述任务处理状态为空闲状态，则获取当前待执行任务的任务参数；
    根据所述任务参数，向所述第一计算机设备发送任务执行指令，所述任务执行指令用于指示所述第一计算机设备根据所述任务参数执行所述当前待执行任务。
  12. 根据权利要求11所述的方法，其特征在于，所述获取当前待执行任务的任务参数，包括：
    从第一预设数据库中，按照待执行任务之间的排列顺序，提取排名靠前的待执行任务的任务参数，其中，所述第一预设数据库中的各个待执行任务之间的排列顺序是按照待执行任务的创建时间和/或对应设置的优先级确定的。
  13. 根据权利要求11所述的方法，其特征在于，所述方法还包括：
    在第二预设数据库中创建所述当前待执行任务的运行记录；
    接收任务运行记录的查看指令；
    根据所述第二预设数据库中创建的各个任务的运行记录，确定与所述查看指令对应的显示结果。
  14. 根据权利要求11所述的方法，其特征在于，所述根据所述任务参数，向所述第一计算机设备发送任务执行指令，包括：
    将所述任务参数拼接出任务对应所要执行的命令行信息；
    通过安全外壳协议向所述第一计算机设备发送所述命令行信息。
  15. 一种任务处理装置，其特征在于，包括：
    监测模块，被配置为监测分布式系统中的每个计算节点的任务处理状态，所述分布式 系统中的计算节点为不同的计算机设备，且不同的计算节点均能读取预设存储位置中的业务数据；
    确定模块，被配置为确定所述任务处理状态为空闲状态的目标计算节点；
    获取模块，被配置为获取当前待执行任务的任务参数；
    发送模块，被配置为根据所述任务参数，向所述目标计算节点发送任务执行指令，以使得所述目标计算节点基于所述预设存储位置中与所述任务参数相应的业务数据，执行所述当前待执行任务。
  16. 一种任务处理装置，其特征在于，应用于第一计算机设备，所述装置包括：
    发送模块，被配置为向第二计算机设备发送所述第一计算机设备当前的任务处理状态；
    接收模块，被配置为接收所述第二计算机设备发送的任务执行指令，所述任务执行指令是所述第二计算机设备在确定所述任务处理状态为空闲状态时发送的，所述任务执行指令中携带有当前待执行任务的任务参数；
    执行模块，被配置为根据所述任务参数，执行所述当前待执行任务。
  17. 一种任务处理装置，其特征在于，应用于第二计算机设备，所述装置包括：
    接收模块，被配置为向第一计算机设备请求获取所述第一计算机设备当前的任务处理状态；
    获取模块，被配置为若所述任务处理状态为空闲状态，则获取当前待执行任务的任务参数；
    发送模块，被配置为根据所述任务参数，向所述第一计算机设备发送任务执行指令，所述任务执行指令用于指示所述第一计算机设备根据所述任务参数执行所述当前待执行任务。
  18. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至14中任一项所述的方法。
  19. 一种电子设备，包括存储介质、处理器及存储在存储介质上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至14中任一项所述的方法。
PCT/CN2022/140507 2022-12-20 2022-12-20 任务处理方法、装置及电子设备 Ceased WO2024130565A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2022/140507 WO2024130565A1 (zh) 2022-12-20 2022-12-20 任务处理方法、装置及电子设备
CN202280101155.7A CN120077365A (zh) 2022-12-20 2022-12-20 任务处理方法、装置及电子设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/140507 WO2024130565A1 (zh) 2022-12-20 2022-12-20 任务处理方法、装置及电子设备

Publications (1)

Publication Number Publication Date
WO2024130565A1 true WO2024130565A1 (zh) 2024-06-27

Family

ID=91587261

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/140507 Ceased WO2024130565A1 (zh) 2022-12-20 2022-12-20 任务处理方法、装置及电子设备

Country Status (2)

Country Link
CN (1) CN120077365A (zh)
WO (1) WO2024130565A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776604A (zh) * 2018-05-23 2018-11-09 网易（杭州）网络有限公司 目标任务的执行方法及系统
CN109800070A (zh) * 2019-01-02 2019-05-24 深圳壹账通智能科技有限公司 任务处理方法和系统
US20200073710A1 (en) * 2018-08-30 2020-03-05 Baidu Online Network Technology (Beijing) Co., Ltd. Task scheduling method, apparatus, electronic device and storage medium
CN112596885A (zh) * 2020-12-25 2021-04-02 网易（杭州）网络有限公司 任务调度方法、装置、设备及存储介质
CN113553178A (zh) * 2021-07-15 2021-10-26 北京字跳网络技术有限公司 任务处理方法、装置和电子设备
CN113986534A (zh) * 2021-10-15 2022-01-28 腾讯科技（深圳）有限公司 任务调度方法、装置、计算机设备和计算机可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776604A (zh) * 2018-05-23 2018-11-09 网易（杭州）网络有限公司 目标任务的执行方法及系统
US20200073710A1 (en) * 2018-08-30 2020-03-05 Baidu Online Network Technology (Beijing) Co., Ltd. Task scheduling method, apparatus, electronic device and storage medium
CN109800070A (zh) * 2019-01-02 2019-05-24 深圳壹账通智能科技有限公司 任务处理方法和系统
CN112596885A (zh) * 2020-12-25 2021-04-02 网易（杭州）网络有限公司 任务调度方法、装置、设备及存储介质
CN113553178A (zh) * 2021-07-15 2021-10-26 北京字跳网络技术有限公司 任务处理方法、装置和电子设备
CN113986534A (zh) * 2021-10-15 2022-01-28 腾讯科技（深圳）有限公司 任务调度方法、装置、计算机设备和计算机可读存储介质

Also Published As

Publication number Publication date
CN120077365A (zh) 2025-05-30

Similar Documents

Publication Publication Date Title
US12314752B2 (en) Execution of auxiliary functions in an on-demand network code execution system
JP7197612B2 (ja) オンデマンドネットワークコード実行システム上での補助機能の実行
US11243953B2 (en) Mapreduce implementation in an on-demand network code execution system and stream data processing system
US11119813B1 (en) Mapreduce implementation using an on-demand network code execution system
CN103092698B (zh) 云计算应用自动部署系统及方法
CN103297395B (zh) 一种互联网业务的实现方法、系统以及装置
CN106533713B (zh) 一种应用部署方法及设备
KR20210019533A (ko) 온-디맨드 네트워크 코드 실행 시스템에서의 운영 체제 커스터마이제이션
US9104488B2 (en) Support server for redirecting task results to a wake-up server
CN108681777A (zh) 一种基于分布式系统的机器学习程序运行的方法和装置
CN109298937A (zh) 文件解析方法及网络设备
KR20110083084A (ko) 가상화를 이용한 서버 운영 장치 및 방법
CN112104679B (zh) 处理超文本传输协议请求的方法、装置、设备和介质
CN115268949A (zh) 一种镜像预热方法、装置、设备及存储介质
CN114296953A (zh) 一种多云异构系统及任务处理方法
WO2019237587A1 (zh) 脚本的执行方法、服务器管理系统及存储介质
CN116263768B (zh) 基于云平台的信息推送方法及装置
CN118331686B (zh) 基于多种虚拟化技术的云服务器密码机系统及实现方法
CN116112497B (zh) 一种云主机集群的节点调度方法、装置、设备及介质
WO2018188607A1 (zh) 流处理方法及装置
WO2024130565A1 (zh) 任务处理方法、装置及电子设备
CN109783201A (zh) 数据交换方法及其系统
WO2024251107A1 (zh) 容器编排与数据访问方法、电子设备及存储介质
CN118550651A (zh) 云应用启动方法
CN103761128A (zh) 一种指令执行方法、装置及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22968867

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280101155.7

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 202280101155.7

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22968867

Country of ref document: EP

Kind code of ref document: A1