CN113010286A - Parallel task scheduling method and device, computer equipment and storage medium - Google Patents
Parallel task scheduling method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113010286A CN113010286A CN202110271991.7A CN202110271991A CN113010286A CN 113010286 A CN113010286 A CN 113010286A CN 202110271991 A CN202110271991 A CN 202110271991A CN 113010286 A CN113010286 A CN 113010286A
- Authority
- CN
- China
- Prior art keywords
- execution
- task
- processor
- subtask
- core
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5018—Thread allocation
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Devices For Executing Special Programs (AREA)
Abstract
The invention discloses a parallel task scheduling method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: determining at least one execution subtask according to the calculation dimension information of the parallel task; establishing a core execution thread and a task distribution queue corresponding to the core execution thread for each processor based on the current working state of each processor; distributing each execution subtask to each task distribution queue according to a preset rule; and controlling each core execution thread to process the execution subtasks in the corresponding task distribution queue. According to the embodiment of the invention, the task is divided into a plurality of execution subtasks according to different calculation dimension information, and the plurality of execution subtasks are processed in parallel by using the multidimensional core execution thread, so that the execution efficiency of the task is improved, the compiling difficulty of the task is reduced, and the controllability in the task synchronous execution process is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a parallel task scheduling method, a parallel task scheduling device, computer equipment and a storage medium.
Background
With the development of computer technology, the processor of modern computers gradually becomes a multi-core processor, the multi-core processor can perform parallel processing on tasks, and the processing performance of the computers is greatly improved. However, the amount of information processed in the internet industry is large at present, and the information is often required to be stored in different databases and different data tables respectively. However, the processing of the above situation data in the prior art has the following technical problems:
1) the computer adopts the full-scale table to carry out sequential retrieval, which causes the multi-core characteristic waste of the computer, and the multi-core processor is not in a working state at the same time, thereby reducing the processing efficiency of data; 2) the computer adopts the multithreading technology to realize parallel query, however, because the current general thread scheduling rule is one-dimensional linear execution, the compiling of the multithreading task is complex for the multidimensional data storage condition that data is stored in different data tables of different databases, and the task synchronous execution process is difficult to control.
Disclosure of Invention
The invention provides a parallel task scheduling method, a parallel task scheduling device, computer equipment and a storage medium, which are used for simultaneously executing tasks by using a plurality of threads, improving the execution efficiency of the tasks, reducing the task compiling difficulty under the condition of multiple data sources and improving the controllability of the task synchronous execution process.
In a first aspect, an embodiment of the present invention provides a parallel task scheduling method, where the method includes:
determining at least one execution subtask according to the calculation dimension information of the parallel task;
establishing a core execution thread and a task distribution queue corresponding to the core execution thread for each processor based on the current working state of each processor;
distributing each execution subtask to each task distribution queue according to a preset rule;
and controlling each core execution thread to process the execution subtasks in the corresponding task distribution queue.
In a second aspect, an embodiment of the present invention further provides a parallel task scheduling apparatus, where the apparatus includes:
the subtask module is used for determining at least one execution subtask according to the calculation dimension information of the parallel task;
the execution thread module is used for establishing a core execution thread and a task distribution queue corresponding to the core execution thread for each processor based on the current working state of each processor;
the task allocation module is used for allocating each execution subtask to each task allocation queue according to a preset rule;
and the task execution module is used for controlling each core execution thread to process the execution subtasks in the corresponding task distribution queue.
In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
a memory for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the parallel task scheduling method according to any of the embodiments of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the parallel task scheduling method according to any one of the embodiments of the present invention.
In the embodiment of the invention, the parallel task is divided into at least one execution subtask according to the preset calculation dimension information, creating a core execution thread and a task allocation queue of the core execution thread for the processors according to the current working state of each processor, allocating each execution subtask to each task allocation queue, controlling each core execution thread to process the execution subtask in each task allocation queue, realizing multidimensional core execution threads, fully utilizing the characteristics of the multi-core processor, leading each processor to process and schedule tasks based on one core execution thread respectively, can improve the execution efficiency of scheduling tasks, adopts multidimensional core execution threads to process the scheduling of multiple data sources, the core execution threads are attached to the scenes of the multiple data sources, the tasks can be divided according to the data sources, the compiling difficulty of the tasks is reduced, and the controllability in the task execution process can be improved.
Drawings
Fig. 1 is a flowchart of a parallel task scheduling method according to an embodiment of the present invention;
fig. 2 is a flowchart of another parallel task scheduling method according to a second embodiment of the present invention;
fig. 3 is an exemplary diagram of a parallel task scheduling method according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of a parallel task scheduling apparatus according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only a part of the structures related to the present invention, not all of the structures, are shown in the drawings, and furthermore, embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
Example one
Fig. 1 is a flowchart of a parallel task scheduling method provided in an embodiment of the present invention, where this embodiment is applicable to a situation of parallel task scheduling in a scenario with multiple data sources, and the method may be executed by a parallel task scheduling device, and the device may be implemented in a hardware and/or software manner, referring to fig. 1, the method provided in the embodiment of the present invention specifically includes the following steps:
and step 110, determining at least one execution subtask according to the calculation dimension information of the parallel task.
The parallel tasks may be scheduling tasks to be executed, and the parallel tasks may specifically be a set composed of one or more atomic tasks, for example, the modification task may be a set of a search atomic task and an update atomic task, and the parallel tasks may be executed in parallel by one or more processors. The calculation information dimension may be an attribute dimension used in a processing process of the parallel task, and the calculation information dimension may include the number of atomic tasks that can be split by the parallel task, the number of data sources of the parallel task, and the like. The execution subtasks may be tasks executed by the processor, and the execution subtasks may be obtained by parallel task decomposition.
In the embodiment of the invention, the calculation dimension information of the parallel task can be acquired, and the calculation dimension information can be preset by a user or can be automatically analyzed and generated according to the attribute information of the parallel task. The parallel task may be divided into a plurality of execution subtasks according to the obtained computation dimension information, an atomic task included in the execution subtasks may correspond to one dimension of the original parallel task, for example, a data source of one parallel task is a and B, the atomic task included in the parallel task is query and update, the parallel task is split according to the data source of the parallel task and the atomic task, and the execution subtasks may be four execution subtasks, such as query in a, update in a, query in B, and update in B.
And step 120, creating a core execution thread and a task allocation queue of the corresponding core execution thread for each processor based on the current working state of each processor.
The processors may be arithmetic and control cores in the computer device, may be final execution units for information processing and program running, and the number of the processors may be at least two. The current operating state may be a state in which the processor performs data processing, and may include an operating state and an idle state, and the current operating state may be determined by a current load condition of the processor. The kernel execution thread may be a thread for processing execution subtasks, different processing may exist in the respective kernel execution thread, and resources of the kernel execution threads in different processors may be independent of each other. The task allocation queues may correspond to the core execution threads one to one, and the tasks in the task allocation queues may be sequentially executed by the corresponding core execution threads according to the storage time.
Specifically, the working state of each processor may be monitored, when each execution subtask needs to be executed, the working state of each processor at the current time may be acquired as the current working state, the current working state may be compared with a threshold requirement, a core execution thread is created in the processor that meets the threshold requirement, and a task allocation queue corresponding to each core execution thread is created, where the threshold requirement may be a requirement for determining that the processor can process the execution subtask, and may include the number of threads of the processor and loads such as the remaining processing performance of the processor.
And step 130, distributing each execution subtask to each task distribution queue according to a preset rule.
The preset rule may be a rule for allocating execution of the subtasks, and may be preset. The preset rules may include quantity-balanced distribution of the execution subtasks, quantity-unbalanced distribution of the execution subtasks, execution of the subtasks according to current load distribution of the processor, execution of the subtasks according to processing performance distribution of the processor, and the like.
In the embodiment of the present invention, after creating the task allocation queue for the core execution thread of the processor, each execution subtask may be allocated to each task allocation queue, and the preset rule is not limited herein, for example, the execution task queue may be allocated to the task allocation queue of the corresponding core execution thread according to the load condition of the processor, the less execution subtasks are allocated to the corresponding task allocation queue when the processor load is larger, or each execution subtask may be allocated to the task allocation queue corresponding to the core execution thread of each processor in a balanced amount, the same number of execution subtasks are allocated to the task allocation queue of the core execution thread of each processor, and each execution subtask may be allocated to the task allocation queue of the core execution thread of each processor according to the processing performance of the processor, and the processing performance of the processor is higher, the greater the number of executing subtasks allocated to the corresponding task allocation queue.
And step 140, controlling each core execution thread to process the execution subtasks in the corresponding task allocation queue.
Specifically, a core execution thread of the processor may obtain one execution subtask from a corresponding task allocation queue each time, and the core execution thread may calculate the execution subtask to obtain a scheduling result. It can be understood that the core execution threads of each processor are in a parallel execution state, and the processing of each execution subtask is not affected.
According to the embodiment of the invention, the parallel task is divided into at least one execution subtask through calculating the dimension information, the core execution thread and the task distribution queue are established for the processors according to the current working state of each processor, wherein the task distribution queue corresponds to the core execution thread one to one, each execution subtask is distributed to each task distribution queue, the core execution thread is controlled to process the execution subtask in each task distribution queue, the multidimensional core execution thread is realized, the characteristics of the multi-core processor are fully utilized, the execution efficiency of the scheduling task is improved, the scheduling of a multi-data source is processed by adopting the multidimensional core execution thread, the compiling difficulty of the task is reduced, and the controllability in the task execution process can be improved.
Example two
Fig. 2 is a flowchart of another parallel task scheduling method provided in the second embodiment of the present invention, and the second embodiment of the present invention is embodied on the basis of the foregoing second embodiment of the present invention, and referring to fig. 2, the method provided in the second embodiment of the present invention specifically includes the following steps:
The data source information may be information of a data source required for executing the parallel task, and may include database information, data table information, and the like, and the data source information of the parallel task may be one or more, for example, the execution of the parallel task needs to depend on information stored in different databases in multiple clusters. The split task information may be information of an atomic task included in the parallel task, for example, the parallel task may include a query atomic task and an update atomic task, and the split task information may be related information of the query atomic task and the update atomic task.
Specifically, before the parallel task is processed, the calculation dimension information for defining the parallel task may be input through an operation command or through a script file, and the defined calculation dimension information may include data source information and split task information.
And step 220, splitting the parallel task into at least one execution subtask according to the data source information and the split task information.
In the embodiment of the invention, the parallel task can be split according to the predefined data source information and split task information, and each split execution subtask can only correspond to one data source information and one checking task information.
And step 230, acquiring the running load of each processor as the current working state.
The operation load may be information indicating a current state of the processor, and the operation load may include a Central Processing Unit (CPU) utilization rate, a connection thread number, and the like.
In the embodiment of the invention, the performance of each processor for processing the parallel tasks can be monitored, and the information such as the CPU utilization rate, the number of connecting threads and the like of each processor at the current moment can be used as the current working state.
And step 240, recording the processors with the running loads less than or equal to the load threshold as execution processors.
The load threshold may be a critical load for determining that the processor can perform parallel task processing, and when the operating load of the processor is less than or equal to the load threshold, the processor may be considered to be available for performing parallel task processing. The execution processor may be a processor selected from all processors to process parallel tasks, and the execution processor may be marked in a physical or logical sense, for example, by identifying an identification number of the processor to implement the physical marking as the execution processor, or by regarding a processor having an operating load less than a load threshold as the execution processor to implement the logical marking as the execution processor.
Specifically, the operation load of each processor may be compared with a load threshold, and if the operation load of the processor is less than or equal to the load threshold, the processor is regarded as an execution processor. It is understood that the value of the load threshold may be preset or dynamically set according to actual conditions, for example, the value of the load threshold may be set to a larger value at night to execute the parallel task using a processor that is idle, or the value of the load threshold may be set to a smaller value at a higher temperature to execute the parallel task using a processor that is computationally more powerful.
In the embodiment of the present invention, a corresponding core execution thread may be created for each execution processor, the core execution threads do not affect each other, and a task allocation queue may be created for each core execution thread, and tasks in the task allocation queue are processed only by the corresponding core execution thread.
The unique identification information may be information identifying the execution subtask queue, such as a UID or a task name, and the unique identification information of different execution subtasks may be different.
Specifically, a unique identification information may be set for each execution subtask, the unique identification information may be randomly allocated to each task allocation queue in a disorder manner, and the number of the unique identification information in each task allocation queue may be the same or different, for example, different numbers of unique identification information may be allocated according to the processing capability of the core execution thread corresponding to the task allocation queue.
Further, on the basis of the above embodiment of the present invention, the unique identification information is represented in a form of coordinates, where elements in the coordinates represent a data source and a split task corresponding to the execution of the subtask, respectively.
In the embodiment of the present invention, the unique identification information of the executed subtask may be coordinate information, each coordinate represents a data source to which the executed subtask belongs or a split task, for example, the data source in the parallel task is A, B, C three databases, each database includes four data tables q, w, e, and r, the parallel task includes three split tasks x, y, and z, the unique identification information may be three-dimensional coordinates, the first dimension uses 0, 1, and 2 to identify the database A, B, C, the second dimension uses 0, 1,2, and 3 to identify the data tables q, w, e, and r, the third dimension uses 0, 1, and 2 to identify the split tasks x, y, and z, one data source for executing the subtask is a w data table in the a database, and the split task for executing the subtask is y, the unique identification information for executing the subtask may be (0, 1,1).
And step 270, controlling each core execution thread to acquire unique identification information from the corresponding task allocation queue.
Specifically, the core execution threads in each execution processor may obtain unique identification information from the corresponding task allocation queue, so that the core execution threads may perform calculation according to the unique identification information. It can be understood that the kernel execution threads in different execution processors are in a parallel running state, and the kernel execution threads do not influence each other.
And step 280, setting a memory isolation barrier corresponding to the execution subtask in each core execution thread according to the unique identification information.
The memory isolation barrier may be a technology for setting independent resources of each core execution Thread, and may be specifically implemented by a Thread-local technology.
In the embodiment of the present invention, after the core execution Thread acquires the unique identification information in the corresponding task allocation queue, a Thread-local technology may be used to set a memory isolation barrier according to the unique identification information, so that the execution subtask corresponding to the unique identification information is only executed by the local core execution Thread.
And step 290, controlling each core execution thread to process and execute the subtasks.
Specifically, each core execution thread may obtain a corresponding execution subtask according to the obtained unique identification information, and each core execution thread processes its execution subtask in parallel, thereby implementing multidimensional calculation of threads.
The error queue may be a queue for storing execution subtasks that have failed to execute, may be created for each processor, or may be created for all processors.
Specifically, when the core execution thread fails to process the execution subtask, the unique identification information of the execution subtask may be stored in the error queue, which facilitates subsequent error processing, for example, the core execution thread in the processor may re-process the execution subtask that has failed to execute in the error queue.
The embodiment of the invention divides the parallel task into a plurality of execution subtasks based on data source information and splitting task information in the calculation dimension information by acquiring the predefined parallel task and the calculation dimension information, detects the operation load of each processor, marks the processor with the operation load less than or equal to a load threshold as the execution processor, creates a core execution thread and a task distribution queue aiming at each execution processor, generates unique identification information of each execution subtask, randomly distributes each unique identification information to each task distribution queue, controls each core execution thread to acquire the unique identification information from the corresponding task distribution queue, creates a memory isolation barrier for executing the subtasks according to the unique identification information, controls each core execution thread to process each execution subtask, and adds the execution subtask with execution failure to an error queue, the multi-core thread parallel processing of the parallel tasks is realized, the characteristics of the multi-core processor are matched, the processing efficiency of the parallel tasks is improved, the data competition conflict in the task execution process is reduced by randomly distributing and executing the subtasks, and the processing performance of the processor can be further enhanced.
Fig. 3 is an exemplary diagram of a parallel task scheduling method according to a second embodiment of the present invention, and referring to fig. 3, a retrieval task scheduling may specifically include the following steps: 1. before the retrieval task is started, the dimension (1 dimension, 2 dimensions or 3 dimensions) of the task participating in the calculation and the execution process of each subtask are defined, and then the defined data is submitted to a scheduler, such as a submit (dim (1,2,3), a sub-task, wherein the sub-task is marked by the sub-task. 2. The scheduler determines the number of subtasks according to the defined dimensions, which may be represented by the formula sum x y z (x >0 and y >0 and z >0), where x, y and z represent different dimensions of the tasks, respectively. 3. And the scheduler creates core execution threads with the same quantity according to the quantity of the current available CPUs, allocates a task queue for each core execution thread, creates coordinates of each subtask according to the subtask quantity determined in the step 2, and equally allocates the coordinates to the task queue of each CPU, wherein the coordinates are inserted into different queues in a disorder mode in the allocation process. 4. After the core execution Thread extracts the coordinates in the task queue of the core execution Thread each time, setting the coordinate values to a memory isolation barrier through a Thread-Local technology, ensuring that the core execution Thread only processes the subtasks corresponding to the coordinates in the task queue, and then executing the sub-task submitted in the step 1 according to the coordinates, wherein the current coordinates can be obtained through the Thread-Local technology in the execution process of the sub-task to perform multidimensional calculation. 5. And when a sub-task fails to execute, inserting the failed coordinate into an error queue. Before each subtask is executed, the corresponding coordinate can be written into the thread local of the core execution thread, so that the coordinate value stored by the current thread can be acquired when the thread executes the subtask, and the mutual influence among different threads is prevented. The coordinates of each subtask are inserted into each task queue in a disorder mode, so that accurate data conflict in the task execution process can be reduced, and the task scheduling efficiency can be further improved.
EXAMPLE III
Fig. 4 is a schematic structural diagram of a parallel task scheduling apparatus provided in a third embodiment of the present invention, and the apparatus shown in fig. 4 can execute the parallel task scheduling method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software and/or hardware, and specifically comprises: a subtask module 301, a thread of execution module 302, a task assignment module 303, and a task execution module 304.
And the subtask module 301 is configured to determine at least one execution subtask according to the calculation dimension information of the parallel task.
The thread-executing module 302 is configured to create a core thread and a task allocation queue corresponding to the core thread for each processor based on the current working state of each processor.
And the task allocation module 303 is configured to allocate each execution subtask to each task allocation queue according to a preset rule.
And the task execution module 304 is configured to control each core execution thread to process the execution subtasks in the corresponding task allocation queue.
According to the embodiment of the invention, the parallel task is divided into at least one execution subtask through the calculation dimension information of the subtask module, the execution thread module creates a core execution thread and a task distribution queue for the processors according to the current working state of each processor, wherein the task distribution queue is in one-to-one correspondence with the core execution thread, the task distribution module distributes each execution subtask to each task distribution queue, and the task execution module controls the core execution thread to process the execution subtasks in each task distribution queue, so that the multidimensional core execution thread is realized, the characteristics of a multi-core processor are fully utilized, the execution efficiency of a scheduling task is improved, the scheduling of multiple data sources is processed by adopting the multidimensional core execution thread, the compiling difficulty of the task is reduced, and the controllability in the task execution process can be improved.
Further, on the basis of the above embodiment of the present invention, the subtask module 301 includes:
the information determining unit is used for acquiring a predefined parallel task and calculation dimension information of the parallel task, wherein the calculation dimension information at least comprises data source information and splitting task information.
And the task splitting unit is used for splitting the parallel task into at least one execution subtask according to the data source information and the split task information.
Further, on the basis of the above embodiment of the present invention, the thread executing module 302 includes:
and the state detection unit is used for acquiring the running load of each processor as the current working state.
And the processor determining unit is used for recording the processor with the running load less than or equal to the load threshold as an execution processor.
And the thread establishing unit is used for establishing a core execution thread for the execution processor and establishing a task distribution queue corresponding to the core execution thread.
Further, on the basis of the above embodiment of the present invention, the task allocation module 303 includes:
and the identification generating unit is used for creating unique identification information for each execution subtask.
And the information distribution unit is used for randomly distributing each piece of unique identification information to each task distribution queue.
Further, on the basis of the above embodiment of the present invention, the task execution module 304 includes:
and the linear acquisition unit is used for controlling each core execution thread to acquire the unique identification information from the corresponding task allocation queue.
And the barrier setting unit is used for setting the memory isolation barrier corresponding to the execution subtask in each core execution thread according to the unique identification information.
And the task execution unit is used for controlling each core execution thread to process the execution subtasks.
Further, on the basis of the above embodiment of the invention, the apparatus further includes: and the fault processing module is used for adding the execution subtasks to the error queue when the execution of the execution subtasks fails.
Further, on the basis of the above embodiment of the present invention, the unique identification information in the task allocation module 303 is represented in a form of coordinates, where elements in the coordinates respectively represent a data source corresponding to the execution of the subtask and the split task.
Example four
Fig. 5 is a schematic structural diagram of a computer apparatus according to a fourth embodiment of the present invention, as shown in fig. 5, the computer apparatus includes a processor 40, a memory 41, an input device 42, and an output device 43; the number of processors 40 in the computer device may be one or more, and one processor 40 is taken as an example in fig. 5; the processor 40, the memory 41, the input device 42 and the output device 43 in the computer apparatus may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 5.
The memory 41, which is a computer-readable storage medium, may be used to store software programs, computer-executable programs, and modules, such as program modules corresponding to the parallel task scheduling method in the embodiment of the present invention (e.g., the subtask module 301, the thread of execution module 302, the task assignment module 303, and the task execution module 304 in the parallel task scheduling apparatus). The processor 40 executes various functional applications of the computer device and data processing by running software programs, instructions and modules stored in the memory 41, that is, implements the parallel task scheduling method described above.
The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 41 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 41 may further include memory located remotely from processor 40, which may be connected to a computer device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 42 is operable to receive input numeric or character information and to generate key signal inputs relating to user settings and function controls of the computer apparatus. The output device 73 may include a display device such as a display screen.
EXAMPLE five
An embodiment of the present invention further provides a computer-readable storage medium containing a computer program, which when executed by a computer processor is configured to perform a method for parallel task scheduling, the method including:
determining at least one execution subtask according to the calculation dimension information of the parallel task;
establishing a core execution thread and a task distribution queue corresponding to the core execution thread for each processor based on the current working state of each processor;
distributing each execution subtask to each task distribution queue according to a preset rule;
and controlling each core execution thread to process the execution subtasks in the corresponding task distribution queue.
Of course, the storage medium containing the computer-executable instructions provided by the embodiments of the present invention is not limited to the method operations described above, and may also perform related operations in the parallel task scheduling method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the parallel task scheduling device, each included unit and module are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for scheduling parallel tasks, the method comprising:
determining at least one execution subtask according to the calculation dimension information of the parallel task;
establishing a core execution thread and a task distribution queue corresponding to the core execution thread for each processor based on the current working state of each processor;
distributing each execution subtask to each task distribution queue according to a preset rule;
and controlling each core execution thread to process the execution subtasks in the corresponding task distribution queue.
2. The method of claim 1, wherein determining at least one execution subtask according to the computation dimension information of the parallel task comprises:
the method comprises the steps of obtaining a predefined parallel task and calculation dimension information of the parallel task, wherein the calculation dimension information at least comprises data source information and split task information;
and splitting the parallel task into at least one execution subtask according to the data source information and the split task information.
3. The method of claim 1, wherein creating a kernel thread and a task allocation queue corresponding to the kernel thread for each processor based on the current working state of each processor comprises:
acquiring the running load of each processor as the current working state;
recording the processor with the running load less than or equal to a load threshold value as an execution processor;
and creating a core execution thread for the execution processor and creating a task distribution queue corresponding to the core execution thread.
4. The method according to claim 1, wherein said allocating each of said execution subtasks to each of said task allocation queues according to a predetermined rule comprises:
creating unique identification information for each execution subtask;
and randomly distributing each piece of unique identification information to each task distribution queue.
5. The method according to claim 4, wherein said controlling each of the core execution threads to process the execution subtasks in the corresponding task allocation queue comprises:
controlling each core execution thread to acquire the unique identification information from the corresponding task allocation queue;
setting a memory isolation barrier corresponding to the execution subtask in each core execution thread according to the unique identification information;
and controlling each core execution thread to process the execution subtask.
6. The method of claim 1, further comprising:
and when the execution of the execution subtask is determined to fail, adding the execution subtask to an error queue.
7. The method according to claim 4 or 5, wherein the unique identification information is represented in a form of coordinates, wherein elements in the coordinates represent a data source and a split task corresponding to the execution of the subtask, respectively.
8. An apparatus for parallel task scheduling, the apparatus comprising:
the subtask module is used for determining at least one execution subtask according to the calculation dimension information of the parallel task;
the execution thread module is used for establishing a core execution thread and a task distribution queue corresponding to the core execution thread for each processor based on the current working state of each processor;
the task allocation module is used for allocating each execution subtask to each task allocation queue according to a preset rule;
and the task execution module is used for controlling each core execution thread to process the execution subtasks in the corresponding task distribution queue.
9. A computer device, characterized in that the computer device comprises:
one or more processors;
a memory for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a method of parallel task scheduling according to any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out a method for parallel task scheduling according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110271991.7A CN113010286A (en) | 2021-03-12 | 2021-03-12 | Parallel task scheduling method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110271991.7A CN113010286A (en) | 2021-03-12 | 2021-03-12 | Parallel task scheduling method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113010286A true CN113010286A (en) | 2021-06-22 |
Family
ID=76406472
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110271991.7A Pending CN113010286A (en) | 2021-03-12 | 2021-03-12 | Parallel task scheduling method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113010286A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113535361A (en) * | 2021-07-23 | 2021-10-22 | 百果园技术(新加坡)有限公司 | Task scheduling method, device, equipment and storage medium |
| CN114116068A (en) * | 2021-12-02 | 2022-03-01 | 重庆紫光华山智安科技有限公司 | Service starting optimization method and device, electronic equipment and readable storage medium |
| CN117076092A (en) * | 2023-10-13 | 2023-11-17 | 成都登临科技有限公司 | Multi-dimensional data task processing method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5940612A (en) * | 1995-09-27 | 1999-08-17 | International Business Machines Corporation | System and method for queuing of tasks in a multiprocessing system |
| US20150135183A1 (en) * | 2013-11-12 | 2015-05-14 | Oxide Interactive, LLC | Method and system of a hierarchical task scheduler for a multi-thread system |
| JP2016126426A (en) * | 2014-12-26 | 2016-07-11 | 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． | Multicore system, multicore processor, parallel processing method, and parallel processing control program |
| CN110795222A (en) * | 2019-10-25 | 2020-02-14 | 北京浪潮数据技术有限公司 | Multithreading task scheduling method, device, equipment and readable medium |
| CN111176811A (en) * | 2019-12-27 | 2020-05-19 | 国家计算机网络与信息安全管理中心 | Device and method for realizing rapid aging of large-batch table entries based on multi-core processor |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
| CN112000455A (en) * | 2020-09-10 | 2020-11-27 | 华云数据控股集团有限公司 | Multithreading task processing method and device and electronic equipment |
-
2021
- 2021-03-12 CN CN202110271991.7A patent/CN113010286A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5940612A (en) * | 1995-09-27 | 1999-08-17 | International Business Machines Corporation | System and method for queuing of tasks in a multiprocessing system |
| US20150135183A1 (en) * | 2013-11-12 | 2015-05-14 | Oxide Interactive, LLC | Method and system of a hierarchical task scheduler for a multi-thread system |
| JP2016126426A (en) * | 2014-12-26 | 2016-07-11 | 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． | Multicore system, multicore processor, parallel processing method, and parallel processing control program |
| CN110795222A (en) * | 2019-10-25 | 2020-02-14 | 北京浪潮数据技术有限公司 | Multithreading task scheduling method, device, equipment and readable medium |
| CN111176811A (en) * | 2019-12-27 | 2020-05-19 | 国家计算机网络与信息安全管理中心 | Device and method for realizing rapid aging of large-batch table entries based on multi-core processor |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
| CN112000455A (en) * | 2020-09-10 | 2020-11-27 | 华云数据控股集团有限公司 | Multithreading task processing method and device and electronic equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113535361A (en) * | 2021-07-23 | 2021-10-22 | 百果园技术(新加坡)有限公司 | Task scheduling method, device, equipment and storage medium |
| CN114116068A (en) * | 2021-12-02 | 2022-03-01 | 重庆紫光华山智安科技有限公司 | Service starting optimization method and device, electronic equipment and readable storage medium |
| CN114116068B (en) * | 2021-12-02 | 2023-06-02 | 重庆紫光华山智安科技有限公司 | Service start optimization method and device, electronic equipment and readable storage medium |
| CN117076092A (en) * | 2023-10-13 | 2023-11-17 | 成都登临科技有限公司 | Multi-dimensional data task processing method and device, electronic equipment and storage medium |
| CN117076092B (en) * | 2023-10-13 | 2024-01-19 | 成都登临科技有限公司 | Multi-dimensional data task processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9959337B2 (en) | Independent data processing environments within a big data cluster system | |
| WO2020211579A1 (en) | Processing method, device and system for distributed bulk processing system | |
| CN113010286A (en) | Parallel task scheduling method and device, computer equipment and storage medium | |
| CN107239335B (en) | Job scheduling system and method for distributed system | |
| CN102792271B (en) | Cross over many-core systems DYNAMIC DISTRIBUTION multidimensional working set | |
| WO2016078008A1 (en) | Method and apparatus for scheduling data flow task | |
| CN111427681A (en) | Real-time task matching scheduling system and method based on resource monitoring in edge computing | |
| US10942824B2 (en) | Programming model and framework for providing resilient parallel tasks | |
| JP2012511204A (en) | How to reorganize tasks to optimize resources | |
| JP5730386B2 (en) | Computer system and parallel distributed processing method | |
| US20130219395A1 (en) | Batch scheduler management of tasks | |
| CN111708639A (en) | Task scheduling system and method, storage medium and electronic device | |
| US20210374319A1 (en) | Dynamic allocation of computing resources for electronic design automation operations | |
| CN112699040A (en) | Pressure testing method, device, equipment and computer readable storage medium | |
| US10360079B2 (en) | Architecture and services supporting reconfigurable synchronization in a multiprocessing system | |
| US20040093477A1 (en) | Scalable parallel processing on shared memory computers | |
| CN111143063B (en) | Task resource reservation method and device | |
| Nasr et al. | Task scheduling algorithm for high performance heterogeneous distributed computing systems | |
| CN111506400A (en) | Computing resource allocation system, method, device and computer equipment | |
| CN116881003A (en) | Resource allocation method, device, service equipment and storage medium | |
| Yang et al. | Study on static task scheduling based on heterogeneous multi-core processor | |
| CN110297693B (en) | Distributed software task allocation method and system | |
| CN111290868B (en) | Task processing method, device and system and flow engine | |
| CN110415162B (en) | Adaptive graph partitioning method facing heterogeneous fusion processor in big data | |
| GB2504812A (en) | Load balancing in a SAP (RTM) system for processors allocated to data intervals based on system load |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: Jingdong Digital Technology Holding Co., Ltd |