CN113687951B - Task processing method, device, electronic equipment and computer readable storage medium - Google Patents
Task processing method, device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113687951B CN113687951B CN202111020326.7A CN202111020326A CN113687951B CN 113687951 B CN113687951 B CN 113687951B CN 202111020326 A CN202111020326 A CN 202111020326A CN 113687951 B CN113687951 B CN 113687951B
- Authority
- CN
- China
- Prior art keywords
- task
- processed
- slave
- equipment
- fragments
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17306—Intercommunication techniques
- G06F15/17331—Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Mathematical Physics (AREA)
- Stored Programmes (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention relates to a task processing method, a device, an electronic device and a computer readable storage medium, wherein the method is applied to a master device in a computing system, the master device is in communication connection with other slave devices included in the computing system, and the method comprises the following steps: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching the corresponding slave device for each subtask, and sending the subtasks to the corresponding slave device for processing. By the method, the utilization rate of resources can be improved.
Description
Technical Field
The application belongs to the field of data processing, and particularly relates to a task processing method, a task processing device, electronic equipment and a computer readable storage medium.
Background
In recent years, with the development of scientific technology, artificial intelligence technology has come into the line of sight of people. Most of artificial intelligence products are realized based on a deep learning algorithm, and most of processing objects of the deep learning algorithm are data with higher dimensionality such as pictures, videos and the like, so that the normal operation of the deep learning algorithm has higher requirements on computing resources of electronic equipment.
Based on the above premise, in the prior art, for an electronic device a that receives a task a, if the computing resources required by the task a exceed the computing resources currently left by the electronic device a, or the computing resources required by the task a exceed the total computing resources of the electronic device a, the electronic device a cannot successfully process the task a, and can only transfer the task a to another electronic device B with more computing resources for processing. After the operation, the computing resources of the electronic equipment A are not fully utilized, and the utilization rate of the resources is reduced.
Disclosure of Invention
In view of the above, an object of the present application is to provide a task processing method, apparatus, electronic device, and computer readable storage medium, which can improve the utilization rate of resources.
Embodiments of the present application are implemented as follows:
In a first aspect, an embodiment of the present application provides a task processing method, applied to a master device in a computing system, where the master device is communicatively connected to other slave devices included in the computing system, and the method includes: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching the corresponding slave device for each subtask, and sending the subtasks to the corresponding slave device for processing. In the process, the main equipment can split the task to be processed and distribute the sub-tasks obtained by splitting to different auxiliary equipment for processing, so that the task to be processed with larger calculation force can be successfully processed. In addition, by slicing the computing resources of the slave devices, the resource utilization of the respective slave devices can also be improved.
With reference to the first aspect embodiment, in a possible implementation manner, each task to be processed stores a corresponding designated slave device; before the splitting of the task to be processed, the method comprises the following steps: and determining that the appointed slave equipment corresponding to the task to be processed is insufficient to process the task to be processed in the current computing resource.
With reference to the first aspect embodiment, in a possible implementation manner, before the receiving a task to be processed, the method further includes: sending a computing resource pooling instruction to the slave equipment so that the slave equipment fragments the computing resources included by the slave equipment according to a preset size; the current computing resource is a slice in an idle state.
With reference to the first aspect embodiment, in a possible implementation manner, before the receiving a task to be processed, the method further includes: when the newly accessed electronic equipment is detected to meet networking conditions, a networking request and pre-stored first networking configuration information are initiated to the newly accessed electronic equipment; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment to a computing system where the equipment is currently located according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device is a new slave device. In such an embodiment, the computing system may be quickly expanded, thereby increasing the overall computing resources of the computing system so that the computing system may handle more tasks.
With reference to the first aspect embodiment, in a possible implementation manner, the networking conditions are: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic equipment is located and the local area network where the newly accessed electronic equipment is located can be in network communication.
In combination with the first aspect embodiment, in a possible implementation manner, communication between devices included in the computing system may be performed based on RDMA. The RDMA technology is used for communication, so that the electronic devices can quickly communicate with each other on the premise that a CPU is not involved, and communication loss is reduced.
With reference to the embodiment of the first aspect, in a possible implementation manner, the task to be processed is any one task of a model training task, a model predicting task and a model online service.
In a second aspect, an embodiment of the present application provides a task processing device, applied to a master device in a computing system, where the master device is communicatively connected to other slave devices included in the computing system, and the device includes: the device comprises a receiving module, a splitting module and a sending module.
The receiving module is used for receiving the task to be processed;
The splitting module is used for splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks;
and the sending module is used for matching the corresponding slave equipment for each subtask and sending the subtasks to the corresponding slave equipment for processing.
With reference to the second aspect embodiment, in a possible implementation manner, each task to be processed stores a corresponding designated slave device; the apparatus further comprises a determination module for: and determining that the appointed slave equipment corresponding to the task to be processed is insufficient to process the task to be processed in the current computing resource.
With reference to the second aspect embodiment, in a possible implementation manner, the apparatus further includes a pooling module, configured to: sending a computing resource pooling instruction to the slave equipment so that the slave equipment fragments the computing resources included by the slave equipment according to a preset size; the current computing resource is a slice in an idle state.
With reference to the second aspect of the embodiment, in a possible implementation manner, the apparatus further includes a networking module, configured to: when the newly accessed electronic equipment is detected to meet networking conditions, a networking request and pre-stored first networking configuration information are initiated to the newly accessed electronic equipment; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment to a computing system where the equipment is currently located according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device is a new slave device.
With reference to the second aspect of the embodiment, in a possible implementation manner, the networking conditions are: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic equipment is located and the local area network where the newly accessed electronic equipment is located can be in network communication.
In combination with the second aspect embodiment, in one possible implementation manner, communication between devices included in the computing system may be performed based on RDMA.
With reference to the second aspect of the embodiment, in a possible implementation manner, the task to be processed is any one task of a model training task, a model predicting task and a model online service.
In a third aspect, an embodiment of the present application further provides an electronic device, including: the device comprises a memory and a processor, wherein the memory is connected with the processor; the memory is used for storing programs; the processor invokes a program stored in the memory to perform the above-described first aspect embodiment and/or the method provided in connection with any one of the possible implementations of the first aspect embodiment.
In a fourth aspect, embodiments of the present application further provide a non-volatile computer readable storage medium (hereinafter referred to as computer readable storage medium), on which a computer program is stored, which when executed by a computer performs the above-described embodiments of the first aspect and/or the method provided in connection with any one of the possible implementations of the embodiments of the first aspect.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. The above and other objects, features and advantages of the present application will become more apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the several views of the drawings. The drawings are not intended to be drawn to scale, with emphasis instead being placed upon illustrating the principles of the application.
Fig. 1 shows a flowchart of a task processing method provided by an embodiment of the present application.
Fig. 2 shows a block diagram of a task processing device according to an embodiment of the present application.
Fig. 3 shows a schematic structural diagram of an electronic device according to an embodiment of the present application.
Icon: 100-an electronic device; 110-a processor; 120-memory; 400-task processing device; 410 a receiving module; 420-splitting module; 430-a transmitting module.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, relational terms such as "first," "second," and the like may be used solely to distinguish one entity or action from another entity or action in the description of the application without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Furthermore, the term "and/or" in the present application is merely an association relationship describing the association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone.
Moreover, the drawbacks of the task processing solutions of the prior art (reducing the resource utilization of the device) are the results of the applicant after practice and careful study, and therefore the discovery process of the above-mentioned drawbacks and the solutions proposed by the embodiments of the present application hereinafter for the above-mentioned drawbacks should be considered as contributions of the applicant to the present application.
In order to solve the above problems, embodiments of the present application provide a task processing method, apparatus, electronic device, and computer readable storage medium, which can improve the utilization rate of resources of the electronic device.
The technology can be realized by adopting corresponding software, hardware and a combination of the software and the hardware. Embodiments of the present application are described in detail below.
The task processing method provided by the application will be described below.
The embodiment of the application provides a task processing method which is applied to a master device in a computing system, wherein the master device is in communication connection with other slave devices included in the computing system.
It is noted that the computing system, when formed, requires the election of a master device via an election regimen. After the identity of the master device is determined, the identities of the remaining devices are slaves.
The election system is a common prior art, and will not be described herein.
The steps involved will be described below in connection with fig. 1.
Step S110: and receiving a task to be processed.
In the embodiment of the application, the master device receives the task to be processed in preference to other slave devices.
Of course, in some embodiments, other slave devices may acquire the task to be processed prior to the master device. At this time, the slave device is required to synchronize the task to be processed to the master device, so that after the master device receives the task to be processed, the policy for processing the task to be processed is determined according to the current computing resource.
It is worth pointing out that the task to be processed in the embodiment of the application can be a task for training the neural network model, a task for running the neural network model, or any one of a model prediction task and a model online service.
Step S120: and splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks.
Step S130: and matching the corresponding slave device for each subtask, and sending the subtasks to the corresponding slave device for processing.
In an alternative embodiment, after the main device acquires the task to be processed, the main device may directly split the task to obtain a plurality of subtasks.
In another optional implementation manner, after the master device acquires the task to be processed, the master device may synchronously acquire task information corresponding to the task to be processed. The task information comprises a designated slave device corresponding to the task to be processed, and the designated slave device is used for representing that the task to be processed needs to be processed by the corresponding designated slave device preferentially.
In this embodiment, before splitting the task to be processed, the master device needs to acquire the current computing resources of the designated slave device corresponding to the task to be processed, and determine whether the current computing resources of the corresponding designated slave device are greater than the computing resources required by the task to be processed when the task to be processed is running, that is, determine whether the current computing resources of the corresponding designated slave device are sufficient to process the task to be processed.
When the master device determines that the corresponding appointed slave device is insufficient to process the task to be processed in the current computing resource, the task to be processed is split; otherwise, the master device can directly distribute the task to be processed to the corresponding appointed slave device for processing.
As for the process in which the master device splits for the task to be processed, the following may be adopted.
Before the master device receives the task to be processed, the master device may send a computing resource pooling instruction to the slave devices, so that each slave device segments the computing resources included in the slave device according to a preset size. Wherein, the computing resource included in one slice is unit resource.
In addition, the task information of each task to be processed can also include total computing resources required by the task to be processed in running.
After obtaining the total computation resources required for the task to be processed, the master device may divide the total computation resources by the unit resources (i.e., the tiles presented above) to determine that the task to be processed requires several tiles. In addition, the master device obtains the current computing resource of each slave device, and the current computing resource is the slice in the idle state.
After determining the number of fragments required by the task to be processed and the number of fragments in an idle state of each slave device, if the master device detects that the task to be processed does not have a corresponding designated slave device, the master device can randomly screen M (M is an integer greater than 1) target slave devices from each slave device so as to ensure that the number of fragments in the idle state included by the M target slave devices is greater than the number of fragments required by the task to be processed. Subsequently, the master device splits the task to be processed into M subtasks and matches the M subtasks with the M target slave devices one by one.
The number of the fragments in the idle state, which are included by the target slave device, is greater than or equal to the number of fragments required by the subtasks matched with the target slave device.
Of course, it should be noted that there is no correlation in the number of slices required for each sub-task that is split.
Optionally, when screening the target slave device, the master device may preferentially determine, as the target slave device, the slave device that includes fewer fragments in the idle state, so as to fully utilize the fragmented fragment resources.
In another optional implementation manner, if the master device detects that the task to be processed has the corresponding designated slave device, the master device first determines whether the number of fragments in an idle state currently included by the corresponding designated slave device is greater than the number of fragments required by the task to be processed, if so, the master device does not split the task to be processed, and if not, the master device splits the task to be processed according to the splitting process.
After splitting the task to be processed, the master device sends each sub-task to the slave device matched with the sub-task to process.
The above procedure will be exemplified as follows:
Assume that there are 10 slaves and there are 10 slices at each slave. Assuming that the currently acquired task to be processed requires 12 slices, according to the prior art, none of the 10 slaves can process the task to be processed. In the embodiment of the present application, the task to be processed may be split into a plurality of subtasks, for example, split into 2 subtasks, where the subtask 1 needs 4 slices and the subtask 2 needs 6 slices, and then the master device may successfully process the task to be processed by distributing the subtask 1 and the subtask 2 to different slave devices.
In the process, the main equipment can split the task to be processed and distribute the sub-tasks obtained by splitting to different auxiliary equipment for processing, so that the task to be processed with larger calculation force can be successfully processed. In addition, by slicing the computing resources of the slave devices, the resource utilization of the respective slave devices can also be improved.
Furthermore, it should be noted that in some embodiments, the foregoing computing system including the master device and the slave device may be further expanded, thereby increasing the total computing resources of the computing system, so that the computing system may process more tasks.
Specifically, when the master device detects that the newly accessed electronic device meets the networking condition, the master device may initiate a networking request and pre-stored first networking configuration information to the newly accessed electronic device.
The first networking configuration information is pre-stored in the main equipment by a background staff.
After receiving the networking request, the newly accessed electronic device can feed back the second networking configuration information stored by itself to the main device based on the networking request.
So far, the master device and the newly accessed electronic device all acquire networking configuration information of each other.
Subsequently, the main equipment adds the newly accessed electronic equipment to a current computing system of the equipment according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device becomes the new slave device.
In some alternative embodiments, the networking conditions may be: the newly accessed electronic equipment and the equipment are positioned in the same local area network; or the local area network where the newly accessed electronic equipment is located and the local area network where the newly accessed electronic equipment is located can be in network communication.
Further, in some implementations, communication between various devices included in the computing system, including the master device and the slave device, may be based on RDMA (Remote Direct Memory Access ). Under the implementation mode, the electronic equipment can quickly communicate with each other on the premise that the CPU is not involved, and the communication loss is reduced.
As shown in fig. 2, an embodiment of the present application further provides a task processing device 400, which is applied to a master device in a computing system, where the master device is communicatively connected to other slave devices included in the computing system, and the task processing device 400 may include: a receiving module 410, a splitting module 420, and a transmitting module 430.
A receiving module 410, configured to receive a task to be processed;
The splitting module 420 is configured to split the task to be processed according to current computing resources of each slave device included in the computing system, so as to obtain multiple subtasks;
and the sending module 430 is configured to match each subtask with a corresponding slave device, and send the subtask to the corresponding slave device for processing.
In one possible implementation manner, each task to be processed stores a corresponding designated slave device; the device also comprises a determining module, which is used for determining that the appointed slave equipment corresponding to the task to be processed is insufficient to process the task to be processed in the current computing resource.
In one possible embodiment, the apparatus further comprises a pooling module for: sending a computing resource pooling instruction to the slave equipment so that the slave equipment fragments the computing resources included by the slave equipment according to a preset size; the current computing resource is a slice in an idle state.
In a possible implementation manner, the device further comprises a networking module, configured to: when the newly accessed electronic equipment is detected to meet networking conditions, a networking request and pre-stored first networking configuration information are initiated to the newly accessed electronic equipment; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment to a computing system where the equipment is currently located according to the first networking configuration information and the second networking configuration information; wherein the newly accessed device is a new slave device.
In one possible embodiment, the networking conditions are: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic equipment is located and the local area network where the newly accessed electronic equipment is located can be in network communication.
In one possible implementation, RDMA-based communication may be performed between devices included in the computing system.
In one possible implementation manner, the task to be processed is any one task of a model training task, a model predicting task and a model online service.
The task processing device 400 according to the embodiment of the present application has the same implementation principle and technical effects as those of the foregoing method embodiment, and for brevity, reference may be made to the corresponding content in the foregoing method embodiment where the device embodiment is not mentioned.
In addition, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program which executes the steps included in the task processing method when being executed by a computer.
In addition, referring to fig. 3, an electronic device 100 for implementing the task processing method and apparatus according to the embodiments of the present application will be described.
Alternatively, the electronic device 100 may be, but is not limited to, a personal computer (Personal computer, PC), a smart phone, a tablet computer, a Mobile internet device (Mobile INTERNET DEVICE, MID), a personal digital assistant, a server, and other hardware devices. The server may be, but is not limited to, a web server, a database server, a cloud server, etc.
Wherein the electronic device 100 may include: a processor 110, a memory 120.
It should be noted that the components and structures of the electronic device 100 shown in fig. 3 are exemplary only and not limiting, as the electronic device 100 may have other components and structures as desired.
The processor 110, the memory 120, and other components that may be present in the electronic device 100 are electrically connected to each other, either directly or indirectly, to enable transmission or interaction of data. For example, the processor 110, the memory 120, and possibly other components may be electrically connected to each other by one or more communication buses or signal lines.
The memory 120 is used for storing programs, for example, programs corresponding to the task processing methods appearing in the foregoing or task processing devices appearing in the foregoing. Alternatively, when the task processing device is stored in the memory 120, the task processing device includes at least one software functional module that can be stored in the memory 120 in the form of software or firmware (firmware).
Alternatively, the software functional modules included in the task processing device may be incorporated into an Operating System (OS) of the electronic apparatus 100.
The processor 110 is configured to execute executable modules stored in the memory 120, such as software functional modules or computer programs included in the task processing device. When the processor 110 receives the execution instructions, it may execute a computer program, for example, to perform: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching the corresponding slave device for each subtask, and sending the subtasks to the corresponding slave device for processing.
Of course, the methods disclosed in any of the embodiments of the present application may be applied to the processor 110 or implemented by the processor 110.
In summary, the task processing method, apparatus, electronic device and computer readable storage medium according to the embodiments of the present invention are applied to a master device in a computing system, where the master device is communicatively connected to other slave devices included in the computing system, and the method includes: receiving a task to be processed; splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks; and matching the corresponding slave device for each subtask, and sending the subtasks to the corresponding slave device for processing. In the process, the main equipment can split the task to be processed and distribute the sub-tasks obtained by splitting to different auxiliary equipment for processing, so that the task to be processed with larger calculation force can be successfully processed. In addition, by slicing the computing resources of the slave devices, the resource utilization of the respective slave devices can also be improved.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described as different from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a notebook computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application.
Claims (7)
1. A method of task processing for a master device in a computing system, the master device communicatively coupled to other slave devices included in the computing system, the method comprising:
Sending a computing resource pooling instruction to the slave equipment so that the slave equipment fragments the computing resources included by the slave equipment according to a preset size; the method comprises the steps that a master device obtains current computing resources of each slave device, wherein the current computing resources are fragments in an idle state; the computing resource included in one slice is a unit resource;
receiving a task to be processed; the task information of the task to be processed comprises total computing resources required by the task to be processed in running;
Determining that the appointed slave equipment corresponding to the task to be processed is insufficient to process the task to be processed in the current computing resource; each task to be processed has corresponding appointed slave equipment;
Splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks;
Matching corresponding slave devices for each subtask, and sending the subtasks to the corresponding slave devices for processing;
the method comprises the steps of splitting the task to be processed according to the current computing resources of each slave device included in the computing system, and obtaining a plurality of subtasks comprises:
dividing the total computing resource by the unit resource to determine the number of fragments required by the task to be processed;
If the task to be processed does not have the corresponding designated slave device, randomly screening M target slave devices from the slave devices to ensure that the number of fragments in an idle state, which are included in the M target slave devices, is larger than the number of fragments required by the task to be processed; m is an integer greater than 1;
splitting a task to be processed into M subtasks;
correspondingly, matching the corresponding slave device for each subtask includes:
Matching the M subtasks with the M target slave devices one by one; the method comprises the steps that the number of fragments in an idle state, which are included by target slave equipment, is larger than or equal to the number of fragments required by a subtask matched with the target slave equipment; under the condition of screening the target slave device, preferentially determining the slave device with less fragments in an idle state as the target slave device;
the method further comprises the steps of:
When the newly accessed electronic equipment is detected to meet networking conditions, a networking request and pre-stored first networking configuration information are initiated to the newly accessed electronic equipment; the first networking configuration information is pre-stored in the main equipment by a background staff;
receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request;
Adding the newly accessed electronic equipment to a computing system where the equipment is currently located according to the first networking configuration information and the second networking configuration information;
wherein the newly accessed electronic device is a new slave device.
2. The method according to claim 1, wherein the networking conditions are: the newly accessed electronic equipment and the equipment are in the same local area network; or the local area network where the newly accessed electronic equipment is located and the local area network where the newly accessed electronic equipment is located can be in network communication.
3. The method of any of claims 1-2, wherein RDMA-based communication is performed between devices included in the computing system.
4. The method according to any of claims 1-2, wherein the task to be processed is any one of a model training task, a model predicting task, a model online service.
5. A task processing device for use with a master device in a computing system, the master device being communicatively coupled to other slave devices included in the computing system, the device comprising:
The pooling module is used for sending a computing resource pooling instruction to the slave equipment so that the slave equipment can segment the computing resources included by the slave equipment according to a preset size; the method comprises the steps that a master device obtains current computing resources of each slave device, wherein the current computing resources are fragments in an idle state; the computing resource included in one slice is a unit resource;
the receiving module is used for receiving the task to be processed; the task information of the task to be processed comprises total computing resources required by the task to be processed in running;
the determining module is used for determining that the appointed slave equipment corresponding to the task to be processed is insufficient in current computing resources to process the task to be processed; each task to be processed has corresponding appointed slave equipment;
The splitting module is used for splitting the task to be processed according to the current computing resources of each slave device included in the computing system to obtain a plurality of subtasks;
the sending module is used for matching corresponding slave equipment for each subtask and sending the subtasks to the corresponding slave equipment for processing;
The splitting module is specifically configured to divide the total computing resource by a unit resource, and determine the number of fragments required by the task to be processed; if the task to be processed does not have the corresponding designated slave device, randomly screening M target slave devices from the slave devices to ensure that the number of fragments in an idle state, which are included in the M target slave devices, is larger than the number of fragments required by the task to be processed; m is an integer greater than 1; splitting a task to be processed into M subtasks;
The sending module is specifically configured to match the M subtasks with the M target slave devices one by one; the method comprises the steps that the number of fragments in an idle state, which are included by target slave equipment, is larger than or equal to the number of fragments required by a subtask matched with the target slave equipment; under the condition of screening the target slave device, preferentially determining the slave device with less fragments in an idle state as the target slave device;
The networking module is used for initiating a networking request and pre-stored first networking configuration information to the newly accessed electronic equipment when the newly accessed electronic equipment is detected to meet the networking condition; the first networking configuration information is pre-stored in the main equipment by a background staff; receiving second networking configuration information fed back by the newly accessed electronic equipment according to the networking request; adding the newly accessed electronic equipment to a computing system where the equipment is currently located according to the first networking configuration information and the second networking configuration information; wherein the newly accessed electronic device is a new slave device.
6. An electronic device, comprising: the device comprises a memory and a processor, wherein the memory is connected with the processor;
the memory is used for storing programs;
the processor invokes a program stored in the memory to perform the method of any one of claims 1-4.
7. A computer-readable storage medium, on which a computer program is stored, which computer program, when run by a computer, performs the method according to any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111020326.7A CN113687951B (en) | 2021-09-01 | 2021-09-01 | Task processing method, device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111020326.7A CN113687951B (en) | 2021-09-01 | 2021-09-01 | Task processing method, device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113687951A CN113687951A (en) | 2021-11-23 |
| CN113687951B true CN113687951B (en) | 2024-09-20 |
Family
ID=78584695
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111020326.7A Active CN113687951B (en) | 2021-09-01 | 2021-09-01 | Task processing method, device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113687951B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116886677A (en) * | 2023-07-13 | 2023-10-13 | 西安广和通无线软件有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN118673891B (en) * | 2024-06-11 | 2025-02-25 | 深圳海规网络科技有限公司 | Document generation method, device and electronic equipment |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112363821A (en) * | 2021-01-12 | 2021-02-12 | 湖南大学 | Computing resource scheduling method and device and computer equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970520B (en) * | 2013-01-31 | 2017-06-16 | 国际商业机器公司 | Method for managing resource, device and architecture system in MapReduce frameworks |
| CN106548262B (en) * | 2015-09-21 | 2020-11-06 | 阿里巴巴集团控股有限公司 | Scheduling method, device and system for resources for processing tasks |
| CN110609749B (en) * | 2019-09-06 | 2023-07-14 | 创新先进技术有限公司 | A distributed task operation method, system and device |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
| CN112540841B (en) * | 2020-12-28 | 2021-11-12 | 智慧神州(北京)科技有限公司 | Task scheduling method and device, processor and electronic equipment |
| CN112990727A (en) * | 2021-03-26 | 2021-06-18 | 中国人民财产保险股份有限公司深圳市分公司 | Robot task execution control method, device, system and medium |
-
2021
- 2021-09-01 CN CN202111020326.7A patent/CN113687951B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112363821A (en) * | 2021-01-12 | 2021-02-12 | 湖南大学 | Computing resource scheduling method and device and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113687951A (en) | 2021-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12346723B2 (en) | Task processing method, apparatus, and system based on distributed system | |
| US8806488B2 (en) | System for managing personalization information of virtual machine based on cloud computing and method thereof | |
| US11815994B2 (en) | Techniques to provide self-healing data pipelines in a cloud computing environment | |
| CN112035238B (en) | Task scheduling processing method and device, cluster system and readable storage medium | |
| KR20120018178A (en) | Swarm-based synchronization of networked object stores | |
| US9807152B2 (en) | Distributed processing device and distributed processing system as well as distributed processing method | |
| JP2022017588A (en) | Training method of deep-running framework, device, and storage medium | |
| CN113687951B (en) | Task processing method, device, electronic equipment and computer readable storage medium | |
| CN114124968B (en) | Load balancing method, device, equipment and medium based on market data | |
| Li et al. | Optimization of Face Recognition System Based on Azure IoT Edge. | |
| CN116756282A (en) | Task processing methods, servers, language prediction systems and human-computer interaction systems | |
| CN113900598A (en) | Block chain based data storage method, device, equipment and storage medium | |
| CN117608823A (en) | Resource management method, device, equipment and storage medium | |
| CN112269628A (en) | Resource scheduling system and method | |
| CN117519996B (en) | Data processing method, device, equipment and storage medium | |
| CN114327846B (en) | Cluster expansion method, device, electronic device and computer-readable storage medium | |
| CN109033184B (en) | Data processing method and device | |
| CN112286662B (en) | Task pushing method, device and equipment based on shared resources and storage medium | |
| CN113407331B (en) | A task processing method, device and storage medium | |
| CN112114968A (en) | Recommendation method and device, electronic equipment and storage medium | |
| CN111930542A (en) | Data acquisition method and device, electronic equipment and computer readable storage medium | |
| CN107465725B (en) | Heterogeneous long transaction processing system and method based on client information control system | |
| CN112631756A (en) | Distributed regulation and control method and device applied to space flight measurement and control software | |
| CN116032932B (en) | Cluster management method, system, equipment and medium for edge server | |
| US11394780B2 (en) | System and method for facilitating deduplication of operations to be performed |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |