CN113687951A - Task processing method and device, electronic equipment and computer readable storage medium - Google Patents
Task processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113687951A CN113687951A CN202111020326.7A CN202111020326A CN113687951A CN 113687951 A CN113687951 A CN 113687951A CN 202111020326 A CN202111020326 A CN 202111020326A CN 113687951 A CN113687951 A CN 113687951A
- Authority
- CN
- China
- Prior art keywords
- task
- processed
- slave
- computing system
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000000034 method Methods 0.000 claims abstract description 53
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000004891 communication Methods 0.000 claims abstract description 19
- 230000006855 networking Effects 0.000 claims description 45
- 230000008569 process Effects 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 9
- 238000011176 pooling Methods 0.000 claims description 7
- 238000012549 training Methods 0.000 claims description 5
- 230000000977 initiatory effect Effects 0.000 claims description 4
- 239000012634 fragment Substances 0.000 description 16
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17306—Intercommunication techniques
- G06F15/17331—Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Abstract
The invention relates to a task processing method, a task processing device, electronic equipment and a computer-readable storage medium, wherein the method is applied to main equipment in a computing system, the main equipment is in communication connection with other slave equipment included in the computing system, and the method comprises the following steps: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching corresponding slave equipment for each subtask, and sending the subtask to the corresponding slave equipment for processing. By the method, the utilization rate of resources can be improved.
Description
Technical Field
The application belongs to the field of data processing, and particularly relates to a task processing method and device, an electronic device and a computer-readable storage medium.
Background
In recent years, with the development of science and technology, artificial intelligence technology has come into the sight of people. Most artificial intelligence products are realized based on a deep learning algorithm, and most processing objects of the deep learning algorithm are data with high dimensionality such as pictures and videos, so that the normal operation of the deep learning algorithm has high requirements on computing resources of electronic equipment.
Based on the above premise, in the prior art, for an electronic device a receiving a task a, if the computing resources required by the task a exceed the computing resources left by the electronic device a at present, or the computing resources required by the task a exceed the total computing resources of the electronic device a, the electronic device a cannot successfully process the task a, and only the task a can be handed over to another electronic device B with more computing resources for processing. After the operation, the computing resources of the electronic device a are not fully utilized, and the utilization rate of the resources is reduced.
Disclosure of Invention
In view of the above, an object of the present application is to provide a task processing method, a task processing apparatus, an electronic device, and a computer-readable storage medium, which can improve resource utilization.
The embodiment of the application is realized as follows:
in a first aspect, an embodiment of the present application provides a task processing method, which is applied to a master device in a computing system, where the master device is in communication connection with other slave devices included in the computing system, and the method includes: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching corresponding slave equipment for each subtask, and sending the subtask to the corresponding slave equipment for processing. In the above process, the master device may split the task to be processed, and allocate the split sub-tasks to different slave devices for processing, so as to successfully process the task to be processed requiring a large amount of computation. In addition, the resource utilization rate of each slave device can be improved by slicing the computing resources of the slave devices.
With reference to the embodiment of the first aspect, in a possible implementation manner, each of the to-be-processed tasks stores a corresponding designated slave device; before the splitting of the task to be processed, the method comprises the following steps: and determining that the current computing resources of the specified slave equipment corresponding to the task to be processed are not enough to process the task to be processed.
With reference to the embodiment of the first aspect, in a possible implementation manner, before the receiving a task to be processed, the method further includes: sending a computing resource pooling instruction to the slave device to enable the slave device to slice the computing resources included in the slave device according to a preset size; the current computing resource is a slice in an idle state.
With reference to the embodiment of the first aspect, in a possible implementation manner, before the receiving a task to be processed, the method further includes: when detecting that newly accessed electronic equipment meets networking conditions, initiating a networking request and first networking configuration information which is stored in advance to the newly accessed electronic equipment; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment into a current computing system of the equipment according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device is a new slave device. Under the embodiment, the capacity of the computing system can be rapidly expanded, so that the total computing resources of the computing system are increased, and the computing system can process more tasks.
With reference to the embodiment of the first aspect, in a possible implementation manner, the networking condition is: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic device is located and the local area network where the device is located can be in network communication.
With reference to the first aspect, in one possible implementation manner, the RDMA-based communication may be performed between the devices included in the computing system. The RDMA technology is used for communication, so that the communication between the electronic devices can be rapidly carried out on the premise that a CPU does not participate, and the communication loss is reduced.
With reference to the embodiment of the first aspect, in a possible implementation manner, the task to be processed is any one of a model training task, a model prediction task, and a model online service.
In a second aspect, an embodiment of the present application provides a task processing apparatus, which is applied to a master device in a computing system, where the master device is communicatively connected to other slave devices included in the computing system, and the apparatus includes: the device comprises a receiving module, a splitting module and a sending module.
The receiving module is used for receiving the tasks to be processed;
the splitting module is used for splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks;
and the sending module is used for matching the corresponding slave equipment for each subtask and sending the subtask to the corresponding slave equipment for processing.
With reference to the embodiment of the second aspect, in a possible implementation manner, each of the to-be-processed tasks stores a corresponding designated slave device; the apparatus further comprises a determination module to: and determining that the current computing resources of the specified slave equipment corresponding to the task to be processed are not enough to process the task to be processed.
With reference to the second aspect, in a possible implementation manner, the apparatus further includes a pooling module configured to: sending a computing resource pooling instruction to the slave device to enable the slave device to slice the computing resources included in the slave device according to a preset size; the current computing resource is a slice in an idle state.
With reference to the embodiment of the second aspect, in a possible implementation manner, the apparatus further includes a networking module, configured to: when detecting that newly accessed electronic equipment meets networking conditions, initiating a networking request and first networking configuration information which is stored in advance to the newly accessed electronic equipment; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment into a current computing system of the equipment according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device is a new slave device.
With reference to the embodiment of the second aspect, in a possible implementation manner, the networking condition is: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic device is located and the local area network where the device is located can be in network communication.
With reference to the second aspect, in one possible implementation manner, the RDMA-based communication between the devices included in the computing system may be performed.
With reference to the second aspect, in a possible implementation manner, the task to be processed is any one of a model training task, a model prediction task, and a model online service.
In a third aspect, an embodiment of the present application further provides an electronic device, including: a memory and a processor, the memory and the processor connected; the memory is used for storing programs; the processor calls a program stored in the memory to perform the method of the first aspect embodiment and/or any possible implementation manner of the first aspect embodiment.
In a fourth aspect, the present application further provides a non-transitory computer-readable storage medium (hereinafter, referred to as a computer-readable storage medium), on which a computer program is stored, where the computer program is executed by a computer to perform the method in the foregoing first aspect and/or any possible implementation manner of the first aspect.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts. The foregoing and other objects, features and advantages of the application will be apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the drawings. The drawings are not intended to be to scale as practical, emphasis instead being placed upon illustrating the subject matter of the present application.
Fig. 1 shows a flowchart of a task processing method provided in an embodiment of the present application.
Fig. 2 is a block diagram illustrating a structure of a task processing device according to an embodiment of the present application.
Fig. 3 shows a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Icon: 100-an electronic device; 110-a processor; 120-a memory; 400-a task processing device; 410-a receiving module; 420-splitting module; 430-sending module.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, relational terms such as "first," "second," and the like may be used solely in the description herein to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Further, the term "and/or" in the present application is only one kind of association relationship describing the associated object, and means that three kinds of relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone.
In addition, the defects (reduced resource utilization rate of the device) existing in the task processing scheme in the prior art are the result of the practical and careful study of the applicant, and therefore, the discovery process of the above defects and the solution proposed by the embodiment of the present application to the above defects in the following text should be considered as the contribution of the applicant to the present application.
In order to solve the above problem, embodiments of the present application provide a task processing method and apparatus, an electronic device, and a computer-readable storage medium, which can improve the utilization rate of resources of the electronic device.
The technology can be realized by adopting corresponding software, hardware and a combination of software and hardware. The following describes embodiments of the present application in detail.
The following description will be directed to a task processing method provided in the present application.
The embodiment of the application provides a task processing method, which is applied to a master device in a computing system, wherein the master device is in communication connection with other slave devices included in the computing system.
It is noted that a computing system, when formed, requires that a host device be elected through an election system. After the identity of the master device is determined, the identities of the remaining devices are slave devices.
The election system is a common prior art, and is not described herein again.
The steps involved will be described below with reference to fig. 1.
Step S110: and receiving a task to be processed.
In the embodiment of the application, the master device receives the to-be-processed task in preference to other slave devices.
Of course, in some embodiments, other slave devices may acquire the to-be-processed task before the master device. At this time, the slave device needs to synchronize the to-be-processed task to the master device, so that the master device determines a policy for processing the to-be-processed task according to the current computing resource after receiving the to-be-processed task.
It should be noted that the to-be-processed task referred to in the embodiments of the present application may be a task for training a neural network model, a task for operating the neural network model, or any one of a model prediction task and a model online service.
Step S120: and splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks.
Step S130: and matching corresponding slave equipment for each subtask, and sending the subtask to the corresponding slave equipment for processing.
In an optional implementation manner, after the to-be-processed task is obtained, the main device may directly split the to-be-processed task, so as to obtain a plurality of sub-tasks.
In another optional implementation manner, after the main device acquires the task to be processed, task information corresponding to the task to be processed may be acquired synchronously. The task information comprises the appointed slave device corresponding to the task to be processed, and the appointed slave device is used for representing that the task to be processed needs to be processed by the corresponding appointed slave device preferentially.
In this embodiment, before the master device splits the task to be processed, it further needs to first acquire the current computing resource of the designated slave device corresponding to the task to be processed, and determine whether the current computing resource of the corresponding designated slave device is greater than the computing resource required by the running of the task to be processed, that is, determine whether the current computing resource of the corresponding designated slave device is sufficient for processing the task to be processed.
When the master device determines that the current computing resources of the corresponding designated slave device are not enough to process the task to be processed, the task to be processed is split; otherwise, the master device may directly allocate the task to be processed to the corresponding designated slave device for processing.
As for the process of splitting the to-be-processed task by the master device, the following manner may be adopted.
Before the master device receives the task to be processed, the master device may send a computing resource pooling instruction to the slave devices, so that each slave device slices the computing resources included in itself according to a preset size. Wherein, the computing resource included in one segment is a unit resource.
In addition, in the task information of each task to be processed, the total computing resources required by the task to be processed during running can be further included.
After acquiring the total computing resource required by the to-be-processed task, the master device may divide the total computing resource by the unit resource (i.e., the above-mentioned fragment), thereby determining that the to-be-processed task requires several fragments. In addition, the master device obtains the current computing resource of each slave device, and the current computing resource is the fragment in the idle state.
After determining the number of fragments required by the task to be processed and the number of fragments in an idle state currently possessed by each slave device, if the master device detects that the task to be processed does not have a corresponding designated slave device, M (M is an integer greater than 1) target slave devices can be randomly screened from each slave device, so as to ensure that the number of fragments in an idle state included by the M target slave devices is greater than the number of fragments required by the task to be processed. Subsequently, the master device splits the task to be processed into M subtasks, and matches the M subtasks with the M target slave devices one by one.
And the number of the fragments in the idle state included by the target slave device is greater than or equal to the number of the fragments required by the subtasks matched with the target slave device.
Of course, it should be noted that there is no dependency on the number of pieces required by each of the split sub-tasks.
Optionally, when the target slave device is screened, the master device may preferentially determine, as the target slave device, the slave device with the smaller number of fragments in the idle state, so as to fully utilize fragmented fragment resources.
In another optional implementation manner, if the master device detects that the to-be-processed task has the corresponding designated slave device, the master device first determines whether the number of fragments in the idle state currently included by the corresponding designated slave device is greater than the number of fragments required by the to-be-processed task, if so, the master device does not split the to-be-processed task, and if not, the master device splits the to-be-processed task according to the splitting process.
After the tasks to be processed are split, the master device sends each subtask to the slave device matched with the master device for processing.
The following will exemplify the above process:
assume that there are 10 slaves and that there are 10 slices per slave. Assuming that 12 pieces of fragments are needed for the currently acquired task to be processed, according to the prior art, none of the 10 slave devices can process the task to be processed. In this embodiment of the present application, the task to be processed may be split into a plurality of sub-tasks, for example, split into 2 sub-tasks, where 4 pieces of fragments are required for sub-task 1, and 6 pieces of fragments are required for sub-task 2, and then the master device may successfully process the task to be processed by allocating sub-task 1 and sub-task 2 to different slave devices for processing.
In the above process, the master device may split the task to be processed, and allocate the split sub-tasks to different slave devices for processing, so as to successfully process the task to be processed requiring a large amount of computation. In addition, the resource utilization rate of each slave device can be improved by slicing the computing resources of the slave devices.
Moreover, it is worth pointing out that, in some embodiments, the capacity of the computing system including the master device and the slave device may be expanded, so as to increase the total computing resources of the computing system, so that the computing system may process more tasks.
Specifically, when detecting that the newly accessed electronic device satisfies the networking condition, the master device may initiate a networking request and pre-stored first networking configuration information to the newly accessed electronic device.
Wherein the first networking configuration information is pre-stored in the master device by a background worker.
After receiving the networking request, the newly accessed electronic device may feed back second networking configuration information stored by itself to the master device based on the networking request.
Therefore, both the master device and the newly accessed electronic device acquire networking configuration information of each other.
Subsequently, the main device adds the newly accessed electronic device to the current computing system of the device according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device becomes a new slave device.
In some optional embodiments, the networking condition may be: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic equipment is located and the local area network where the electronic equipment is located can be in network communication.
In addition, in some embodiments, the computing system may include devices (including a master device and a slave device) that communicate with each other based on RDMA (Remote Direct Memory Access). Under the embodiment, the electronic equipment can rapidly carry out the communication between the electronic equipment on the premise that the CPU does not participate, and the communication loss is reduced.
As shown in fig. 2, an embodiment of the present application further provides a task processing apparatus 400, which is applied to a master device in a computing system, where the master device is communicatively connected to other slave devices included in the computing system, and the task processing apparatus 400 may include: a receiving module 410, a splitting module 420, and a sending module 430.
A receiving module 410, configured to receive a task to be processed;
a splitting module 420, configured to split the to-be-processed task according to current computing resources of each slave device included in the computing system, so as to obtain multiple sub-tasks;
a sending module 430, configured to match a corresponding slave device for each sub-task, and send the sub-task to the corresponding slave device for processing.
In one possible implementation, each of the tasks to be processed stores a corresponding designated slave device; the device further comprises a determining module, configured to determine that a current computing resource of a specified slave device corresponding to the to-be-processed task is insufficient to process the to-be-processed task.
In a possible embodiment, the apparatus further comprises a pooling module for: sending a computing resource pooling instruction to the slave device to enable the slave device to slice the computing resources included in the slave device according to a preset size; the current computing resource is a slice in an idle state.
In a possible implementation, the apparatus further includes a networking module configured to: when detecting that newly accessed electronic equipment meets networking conditions, initiating a networking request and first networking configuration information which is stored in advance to the newly accessed electronic equipment; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment into a current computing system of the equipment according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device is a new slave device.
In one possible embodiment, the networking condition is: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic device is located and the local area network where the device is located can be in network communication.
In one possible implementation, the computing system includes devices that can communicate over RDMA.
In a possible implementation manner, the task to be processed is any one of a model training task, a model prediction task and a model online service.
The task processing device 400 provided in the embodiment of the present application has the same implementation principle and the same technical effect as those of the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments for the parts of the embodiment that are not mentioned in the description of the present application.
In addition, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a computer, the computer program performs the steps included in the task processing method.
In addition, please refer to fig. 3 to describe an electronic device 100 for implementing a task processing method and apparatus according to an embodiment of the present application.
Optionally, the electronic Device 100 may be, but is not limited to, a hardware Device such as a Personal Computer (PC), a smart phone, a tablet computer, a Mobile Internet Device (MID), a Personal digital assistant, and a server. The server may be, but is not limited to, a web server, a database server, a cloud server, and the like.
Among them, the electronic device 100 may include: a processor 110, a memory 120.
It should be noted that the components and structure of electronic device 100 shown in FIG. 3 are exemplary only, and not limiting, and electronic device 100 may have other components and structures as desired.
The processor 110, memory 120, and other components that may be present in the electronic device 100 are electrically connected to each other, directly or indirectly, to enable the transfer or interaction of data. For example, the processor 110, the memory 120, and other components that may be present may be electrically coupled to each other via one or more communication buses or signal lines.
The memory 120 is used for storing programs, such as programs corresponding to the task processing methods described above or the task processing devices described above. Optionally, when the task processing device is stored in the memory 120, the task processing device includes at least one software functional module that can be stored in the memory 120 in the form of software or firmware (firmware).
Alternatively, the software function module included in the task processing device may also be solidified in an Operating System (OS) of the electronic device 100.
The processor 110 is adapted to execute executable modules stored in the memory 120, such as software functional modules or computer programs comprised by the task processing device. When the processor 110 receives the execution instruction, it may execute the computer program, for example, to perform: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching corresponding slave equipment for each subtask, and sending the subtask to the corresponding slave equipment for processing.
Of course, the method disclosed in any of the embodiments of the present application can be applied to the processor 110, or implemented by the processor 110.
In summary, the task processing method, the task processing apparatus, the electronic device, and the computer-readable storage medium according to the embodiments of the present invention are applied to a master device in a computing system, where the master device is in communication connection with other slave devices included in the computing system, and the method includes: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching corresponding slave equipment for each subtask, and sending the subtask to the corresponding slave equipment for processing. In the above process, the master device may split the task to be processed, and allocate the split sub-tasks to different slave devices for processing, so as to successfully process the task to be processed requiring a large amount of computation. In addition, the resource utilization rate of each slave device can be improved by slicing the computing resources of the slave devices.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a notebook computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application.
Claims (10)
1. A task processing method is applied to a master device in a computing system, wherein the master device is in communication connection with other slave devices included in the computing system, and the method comprises the following steps:
receiving a task to be processed;
splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks;
and matching corresponding slave equipment for each subtask, and sending the subtask to the corresponding slave equipment for processing.
2. The method according to claim 1, wherein each of the pending tasks stores a corresponding designated slave device; before the splitting of the task to be processed, the method comprises the following steps:
and determining that the current computing resources of the specified slave equipment corresponding to the task to be processed are not enough to process the task to be processed.
3. The method of claim 1, wherein prior to said receiving a pending task, the method further comprises:
sending a computing resource pooling instruction to the slave device to enable the slave device to slice the computing resources included in the slave device according to a preset size;
the current computing resource is a slice in an idle state.
4. The method of claim 1, further comprising:
when detecting that newly accessed electronic equipment meets networking conditions, initiating a networking request and first networking configuration information which is stored in advance to the newly accessed electronic equipment;
receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request;
adding the newly accessed electronic equipment into a current computing system of the equipment according to the first networking configuration information and the second networking configuration information;
wherein the newly accessed device is a new slave device.
5. The method of claim 4, wherein the networking condition is: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic device is located and the local area network where the device is located can be in network communication.
6. The method of any of claims 1-5, wherein the computing system includes RDMA-based communications between the devices.
7. The method according to any one of claims 1-5, wherein the task to be processed is any one of a model training task, a model prediction task and an online service of a model.
8. A task processing apparatus applied to a master device in a computing system, the master device being communicatively connected to other slave devices included in the computing system, the apparatus comprising:
the receiving module is used for receiving the tasks to be processed;
the splitting module is used for splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks;
and the sending module is used for matching the corresponding slave equipment for each subtask and sending the subtask to the corresponding slave equipment for processing.
9. An electronic device, comprising: a memory and a processor, the memory and the processor connected;
the memory is used for storing programs;
the processor calls a program stored in the memory to perform the method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when executed by a computer, performs the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111020326.7A CN113687951A (en) | 2021-09-01 | 2021-09-01 | Task processing method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111020326.7A CN113687951A (en) | 2021-09-01 | 2021-09-01 | Task processing method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113687951A true CN113687951A (en) | 2021-11-23 |
Family
ID=78584695
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111020326.7A Pending CN113687951A (en) | 2021-09-01 | 2021-09-01 | Task processing method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113687951A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170083381A1 (en) * | 2015-09-21 | 2017-03-23 | Alibaba Group Holding Limited | System and method for processing task resources |
| CN112363821A (en) * | 2021-01-12 | 2021-02-12 | 湖南大学 | Computing resource scheduling method and device and computer equipment |
| CN112540841A (en) * | 2020-12-28 | 2021-03-23 | 智慧神州(北京)科技有限公司 | Task scheduling method and device, processor and electronic equipment |
| CN112990727A (en) * | 2021-03-26 | 2021-06-18 | 中国人民财产保险股份有限公司深圳市分公司 | Robot task execution control method, device, system and medium |
-
2021
- 2021-09-01 CN CN202111020326.7A patent/CN113687951A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170083381A1 (en) * | 2015-09-21 | 2017-03-23 | Alibaba Group Holding Limited | System and method for processing task resources |
| CN112540841A (en) * | 2020-12-28 | 2021-03-23 | 智慧神州(北京)科技有限公司 | Task scheduling method and device, processor and electronic equipment |
| CN112363821A (en) * | 2021-01-12 | 2021-02-12 | 湖南大学 | Computing resource scheduling method and device and computer equipment |
| CN112990727A (en) * | 2021-03-26 | 2021-06-18 | 中国人民财产保险股份有限公司深圳市分公司 | Robot task execution control method, device, system and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220179682A1 (en) | Task processing method, apparatus, and system based on distributed system | |
| US11087234B2 (en) | Method and system for distributed deep machine learning | |
| US8806488B2 (en) | System for managing personalization information of virtual machine based on cloud computing and method thereof | |
| CN113656179B (en) | Scheduling method and device of cloud computing resources, electronic equipment and storage medium | |
| CN113411404A (en) | File downloading method, device, server and storage medium | |
| CN112311820A (en) | Edge device scheduling method, connection method, device and edge device | |
| CN113553178A (en) | Task processing method and device and electronic equipment | |
| CN109951551B (en) | Container mirror image management system and method | |
| CN108112268B (en) | Managing load balancers associated with auto-extension groups | |
| CN111309447A (en) | Container scheduling method and device | |
| CN109033184B (en) | Data processing method and device | |
| CN112822044B (en) | Distributed cluster deployment method and device, electronic equipment and readable storage medium | |
| US20240143436A1 (en) | Techniques to provide self-healing data pipelines in a cloud computing environment | |
| CN116069493A (en) | Data processing method, device, equipment and readable storage medium | |
| CN112631756A (en) | Distributed regulation and control method and device applied to space flight measurement and control software | |
| CN114327846A (en) | Cluster capacity expansion method and device, electronic equipment and computer readable storage medium | |
| CN111930542A (en) | Data acquisition method and device, electronic equipment and computer readable storage medium | |
| CN112527901A (en) | Data storage system, method, computing device and computer storage medium | |
| CN113687951A (en) | Task processing method and device, electronic equipment and computer readable storage medium | |
| CN108833532B (en) | Service processing method, device and system based on Internet of things | |
| WO2023020432A1 (en) | Computing power request method, computing power allocation method, computing power execution method, and terminal and network-side device | |
| CN107465725B (en) | Heterogeneous long transaction processing system and method based on client information control system | |
| US11797353B2 (en) | Method and system for performing workloads in a data cluster | |
| CN106293929B (en) | Data processing method and first electronic device | |
| CN110874256B (en) | Computing cluster updating system, method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |