WO2022057940A1 - 一种计算节点资源信息更新方法、节点和存储介质 - Google Patents

一种计算节点资源信息更新方法、节点和存储介质 Download PDF

Info

Publication number
WO2022057940A1
WO2022057940A1 PCT/CN2021/119600 CN2021119600W WO2022057940A1 WO 2022057940 A1 WO2022057940 A1 WO 2022057940A1 CN 2021119600 W CN2021119600 W CN 2021119600W WO 2022057940 A1 WO2022057940 A1 WO 2022057940A1
Authority
WO
WIPO (PCT)
Prior art keywords
computing
task
sample data
node
historical
Prior art date
Application number
PCT/CN2021/119600
Other languages
English (en)
French (fr)
Inventor
王凯
王晴
范晓辉
Original Assignee
中国移动通信有限公司研究院
中国移动通信集团有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国移动通信有限公司研究院, 中国移动通信集团有限公司 filed Critical 中国移动通信有限公司研究院
Publication of WO2022057940A1 publication Critical patent/WO2022057940A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs

Definitions

  • the present disclosure relates to the field of edge computing, in particular to a computing node resource information update method, node and storage medium.
  • the embodiments of the present disclosure provide a computing node resource information update method, node and storage medium.
  • Embodiments of the present disclosure provide a method for updating computing node resource information, the method comprising:
  • the computing resource information update interval is determined based on the processing duration of the multiple subtasks; the computing resource information update interval is consistent with the processing duration of the divided subtasks, or the computing resource information update interval is the same as the divided subtasks.
  • the minimum processing time in the processing time is the same.
  • the dividing the multiple computing tasks to obtain subtasks corresponding to each computing task includes:
  • the computing tasks are divided based on the task amount of each computing task and the state of each computing node to obtain subtasks.
  • the computing tasks are divided based on the task amount of each computing task and the state of each computing node to obtain subtasks, including:
  • the size of the subtask is obtained based on the task amount of the computing task, the state of each computing node and the pre-obtained computing model, the computing task is divided based on the size of the subtask, and multiple subtasks corresponding to the computing task are obtained .
  • the method further includes: obtaining a historical sample data set; the historical sample data set includes multiple groups of historical sample data; each group of historical sample data includes the history of the computing node at each historical moment Status and task volume of historical computing tasks;
  • the computing model is trained based on the historical sample data in the historical sample data set.
  • the training of the computing model based on the historical sample data includes:
  • n is a positive integer
  • N is a positive integer
  • Modeling is performed based on the target variable set and the N sample subsets to obtain the computational model.
  • An embodiment of the present disclosure further provides a scheduling node, where the scheduling node includes: an obtaining unit, a dividing unit, and a determining unit; wherein,
  • the obtaining unit is configured to obtain multiple computing tasks to be processed by the computing node
  • the dividing unit is configured to divide the plurality of computing tasks to obtain subtasks corresponding to each computing task
  • the determining unit is configured to determine the computing resource information update interval based on the processing duration of the multiple subtasks; the computing resource information update interval is consistent with the processing duration of the divided subtasks, or the computing resource information update interval It is consistent with the minimum processing duration among the processing durations of the divided subtasks.
  • the computing resource information update interval is the processing duration of the multiple subtasks
  • the computing resource information update interval is the minimum processing duration among the processing durations of the multiple subtasks.
  • the dividing unit is configured to divide the computing task based on the task amount of each computing task and the state of each computing node to obtain subtasks.
  • the dividing unit is configured to obtain the size of the subtask based on the task amount of the computing task, the state of each computing node, and a pre-obtained computing model, and based on the size of the subtask
  • the computing task is divided to obtain a plurality of subtasks corresponding to the computing task.
  • the scheduling node further includes a modeling unit configured to obtain a historical sample data set; the historical sample data set includes multiple groups of historical sample data; each group of historical sample data includes a computing node The historical state at each historical moment and the task amount of the historical computing task; the computing model is trained based on the historical sample data in the historical sample data set.
  • a modeling unit configured to obtain a historical sample data set; the historical sample data set includes multiple groups of historical sample data; each group of historical sample data includes a computing node The historical state at each historical moment and the task amount of the historical computing task; the computing model is trained based on the historical sample data in the historical sample data set.
  • the modeling unit is configured to use a feature processing algorithm for each group of historical sample data in the historical sample data set to obtain n features, and select the n features according to preset rules Some of the features in the features are used as the target variable set; n is a positive integer; the historical sample data in the historical sample data set is randomly sampled to form N sample subsets; N is a positive integer; The N sample subsets are modeled to obtain the computational model.
  • the embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, implements the steps of the methods described in the embodiments of the present disclosure.
  • An embodiment of the present disclosure also provides a scheduling node, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the method described in the embodiment of the present disclosure when the processor executes the program A step of.
  • the computing node resource information update method, node, and storage medium provided by the embodiments of the present disclosure include: obtaining multiple computing tasks to be processed by the computing node; dividing the multiple computing tasks, and obtaining corresponding computing tasks subtask; determine the computing resource information update interval based on the processing duration of the multiple subtasks; the computing resource information update interval is consistent with the processing duration of the divided subtasks, or, the computing resource information update interval is the same as the The minimum processing duration among the processing durations of multiple subtasks is the same.
  • the node resource information update interval can be dynamically adjusted based on the processing time of the subtask. On the one hand, it avoids the waste of resources caused by updating the computing node resource information at a fixed small time interval, and also avoids updating the computing node resource information at a fixed large time interval. The resulting task accumulation or even overflow, resulting in data loss, tasks that cannot be completed on time, etc.; on the other hand, if the resource situation of the computing node changes, the scheduling node can also update the computing resource information within the period of the update interval, that is, the sub-task.
  • the resource change of the computing node is obtained, so that the scheduling node can schedule tasks according to the current actual resource situation of the computing node, thereby reducing the computing resource information of the scheduling node due to the resource change of the computing node.
  • FIG. 1 is a schematic diagram of a system architecture to which a computing node resource information update method according to an embodiment of the present disclosure is applied;
  • FIG. 2 is a schematic flowchart of a method for updating computing node resource information according to an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of the establishment of a computing model in a method for updating computing node resource information according to an embodiment of the present disclosure
  • FIG. 4 is a schematic diagram 1 of the composition structure of a scheduling node according to an embodiment of the present disclosure
  • FIG. 5 is a second schematic diagram of the composition and structure of a scheduling node according to an embodiment of the present disclosure
  • FIG. 6 is another schematic diagram of a system architecture to which the method for updating computing node resource information according to an embodiment of the present disclosure is applied;
  • FIG. 7 is a schematic diagram of a hardware structure of a scheduling node according to an embodiment of the present disclosure.
  • FIG. 1 is a schematic diagram of a system architecture of an application of a computing node resource information update method according to an embodiment of the disclosure; as shown in FIG. 1 , the system may include: a computing module, a scheduling and resource allocation module, and a consumer module. in,
  • the consumer module can generate task data, accept the calculation results corresponding to the task data, and carry out more extended applications based on the calculation results.
  • the calculation results of speech analysis and recognition can be consumed by the voice box for further communication input, and can also be provided to other intelligent devices such as air conditioners and lights for smart home control operations.
  • the consumer module can be a webcam, and the video data collected by the webcam can be used as task data.
  • the consumer module may be a microphone, and the audio data collected by the microphone may be used as task data.
  • the computing module provides the general computing capability as a module to the scheduling and resource allocation module, and can calculate the task data.
  • Exemplary computing capabilities may include: image analysis and recognition capabilities, speech analysis and recognition capabilities, and the like.
  • Scheduling and resource allocation module manages idle computing resources according to the request of the consumer module, optimizes the matching between demand and supply, and realizes the completion of the task.
  • computing node in the following embodiments of the present disclosure is equivalent to the computing module shown in FIG. 1 ;
  • scheduling node is equivalent to the scheduling and resource allocation module shown in FIG. 1 ;
  • consumer node is equivalent to The consumer module shown in Figure 1.
  • FIG. 2 is a schematic flowchart of a method for updating computing node resource information according to an embodiment of the disclosure; as shown in FIG. 2 , the method includes:
  • Step 101 Obtain multiple computing tasks to be processed by the computing node
  • Step 102 Divide the multiple computing tasks to obtain subtasks corresponding to each computing task
  • Step 103 Determine the computing resource information update interval based on the processing duration of multiple subtasks; the computing resource information update interval is consistent with the processing duration of the divided subtasks, or, the computing resource information update interval is the same as the divided multiple subtasks.
  • the minimum processing time in the processing time of the task is the same.
  • the scheduling node obtains the computing task from the consumer node.
  • the computing task may be video data collected by the webcam.
  • the scheduling node obtains the task data of the computing task from the consumer node; on the other hand, the scheduling node obtains the task processing request from the The obtained task data is divided to obtain multiple subtasks.
  • the task processing request may include instruction information representing the task; the task may be, for example, identifying a target person in an image, identifying the number of people in an image, or identifying a target person in audio data, and so on.
  • each task and the corresponding identifier may be preconfigured in the scheduling node; the indication identifier carried in the task processing request is identified, and the corresponding target task may be determined according to the indication identifier.
  • the scheduling node divides each computing task, and divides each computing task into multiple subtasks.
  • the processing durations of the subtasks corresponding to each computing task are the same or in a proportional relationship.
  • the processing durations of multiple subtasks corresponding to one computing task are the same.
  • the dividing the plurality of computing tasks to obtain subtasks corresponding to each computing task includes: performing a calculation based on the task amount of each computing task and the state of each computing node The task is divided into subtasks.
  • the scheduling node may obtain the status of each computing node according to a preset rule.
  • the state of the computing node may include: information about various types of resources, information about memory, idle computing power, information about used computing resources, and the like.
  • the memory-related information may include memory usage.
  • the types of resources may include: Central Processing Unit (CPU, Central Processing Unit), Graphics Processing Unit (GPU, Graphics Processing Unit), Neural Network Processing Unit (NPU, Neural Network Processing Unit), etc.; information about each type of resource Can include CPU usage, GPU usage, NPU usage, etc.
  • the state of the computing node may further include: the number of existing processes, the temperature of the node device, the task type, and the like.
  • the scheduling node may obtain the status of each computing node according to a preset period; wherein, the preset period may be set according to actual requirements, which is not limited in this embodiment.
  • the scheduling node may divide each computing task based on the state of each computing node and the task amount of each computing task, so that the divided subtasks corresponding to each computing node can process the corresponding computing node for a long time. the same or an equal relationship.
  • the computing resource information update interval is the processing duration of the multiple subtasks; if the processing durations of the multiple subtasks are in an equal ratio relationship , the computing resource information update interval is the minimum processing duration among the processing durations of the multiple subtasks.
  • the update interval of the computing resource information is the above t_3.
  • the computing resource information update interval is the processing durations of the subtasks corresponding to the three computing tasks.
  • the scheduling node divides the computing task according to the current running state of the computing node, the processing time of the computing task, and the waiting time of the previous task to obtain subtasks.
  • the scheduling node divides subtasks, it is necessary to consider the amount of tasks allocated according to the current running state of the computing node.
  • the computing capacity of the node is full, the subtasks need to be divided according to the task processing time and the waiting time of the previous task.
  • dividing the computing task based on the task amount of the computing task and the state of each computing node to obtain multiple subtasks includes: based on the task amount of the computing task , the state of each computing node and the pre-obtained computing model to obtain the size of each subtask, and divide the computing task based on the size of the subtask to obtain a plurality of subtasks.
  • the method further includes: obtaining a historical sample data set; the historical sample data set includes multiple groups of historical sample data; each group of historical sample data includes the history of the computing node at each historical moment state and the task amount of the historical computing task; the computing model is trained based on the historical sample data in the historical sample data set.
  • the scheduling node may establish a calculation model in advance, and determine the task amount of the subtasks of each calculation task based on the calculation model.
  • a historical sample data set is obtained, the historical sample data set includes multiple groups of historical sample data, and each group of historical sample data may include the historical state X of the computing nodes at N historical moments and the task size Y of the historical computing task, The computing model is trained based on the historical state X and the task size Y of the historical computing task.
  • the historical state X may include at least one of the following: CPU usage, GPU usage, NPU usage, memory usage, the number of existing processes, the temperature of the node device, the task type, and so on.
  • the training of the computing model based on the historical sample data includes: using a feature processing algorithm for each group of historical sample data in the historical sample data set to obtain n features, according to a preset The rule selects part of the features in the n features as the target variable set; n is a positive integer; the historical sample data in the historical sample data set is randomly sampled to form N sample subsets; N is a positive integer; The target variable set and the N sample subsets are modeled to obtain the computational model.
  • the feature processing algorithm includes, but is not limited to, principal component analysis method, variance selection method, correlation coefficient method, and the like.
  • a Principal Component Analysis (PCA, Principal Component Analysis) method is used for each group of historical sample data to obtain n features, and some of the n features are selected according to preset rules Features ⁇ VS 1 , VS 2 , VS 3 . value.
  • each feature in the target variable set is a weighted combination of some features in the historical sample data.
  • modeling is performed based on the target variable set and the N sample subsets to obtain the computational model.
  • the modeling based on the target variable set and the N sample subsets to obtain the calculation model includes: modeling each sample subset based on the target variable set.
  • a linear regression algorithm can be used for modeling to obtain a model corresponding to each sample subset, and based on the model corresponding to each sample subset, a prediction result corresponding to the task volume Y of the historical computing tasks in each sample subset can be obtained.
  • the node resource information update interval can be dynamically adjusted based on the processing time of the subtask. On the one hand, it avoids the waste of resources caused by updating the computing node resource information at a fixed small time interval, and also avoids updating the computing node resource information at a fixed large time interval.
  • the scheduling node can also update the computing resource information within this period of the computing resource information.
  • the resource change of the computing node is obtained, so that the scheduling node can schedule tasks according to the current actual resource situation of the computing node, thereby reducing the computing resource information of the scheduling node due to the resource change of the computing node. Task delays caused by untimely updates, while reducing computing resources consumed by task coordination.
  • the system connects multiple edge devices to form a row of video processing channels.
  • Multiple video processing channels enable parallel processing of real-time video.
  • the real-time video is divided into multiple video task packages by slicing, which are respectively assigned to each video processing channel for processing.
  • the processed video data are fused together to form a real-time video processing result stream.
  • the consumer module (such as a camera) provides real-time video data and requests video processing capabilities from the scheduling node.
  • the scheduling node requests the video processing task data to be sent to the corresponding computing node on the one hand.
  • the real-time video data is cut and distributed to the corresponding computing nodes, and the corresponding processing tasks are performed by the computing nodes.
  • the scheduling node also continuously updates the resource status of each computing node, and adjusts task scheduling when necessary.
  • FIG. 4 is a schematic diagram of the composition structure of a scheduling node according to an embodiment of the present disclosure; as shown in FIG. 4 , the scheduling node includes: an obtaining unit 21, a dividing unit 22, and a determining unit 23; wherein,
  • the obtaining unit 21 is configured to obtain multiple computing tasks to be processed by the computing node;
  • the dividing unit 22 is configured to divide the plurality of computing tasks to obtain subtasks corresponding to each computing task;
  • the determining unit 23 is configured to determine the computing resource information update interval based on the processing durations of multiple subtasks; the computing resource information updating interval is consistent with the processing duration of the divided subtasks, or the computing resource information updating interval is the same as the processing duration of the divided subtasks.
  • the minimum processing duration among the processing durations of the divided subtasks is the same.
  • the dividing unit 22 is configured to divide the computing tasks based on the task amount of each computing task and the state of each computing node to obtain subtasks.
  • the dividing unit divides subtasks, it is necessary to consider the amount of tasks allocated according to the current running state of the computing node.
  • the computing capacity of the node is full, the subtasks need to be divided according to the task processing time and the waiting time of the previous task.
  • the dividing unit 22 is configured to obtain the size of each subtask based on the task amount of the computing task, the state of each computing node, and a pre-obtained computing model, and based on the size of the subtask The computing task is divided to obtain a plurality of subtasks corresponding to the computing task.
  • the scheduling node further includes a modeling unit 24 configured to obtain a historical sample data set; the historical sample data set includes multiple groups of historical sample data; each The group of historical sample data includes the historical state of the computing node at each historical moment and the task amount of the historical computing task; the computing model is trained based on the historical sample data in the historical sample data set.
  • the modeling unit 24 is configured to use a feature processing algorithm for each group of historical sample data in the historical sample data set to obtain n features, and select the n features according to preset rules Some of the features are used as the target variable set; n is a positive integer; the historical sample data in the historical sample data set is randomly sampled to form N sample subsets; N is a positive integer; based on the target variable set and The N sample subsets are modeled to obtain the computational model.
  • the feature processing algorithm includes, but is not limited to, principal component analysis method, variance selection method, correlation coefficient method, and the like.
  • the dividing unit 22, the determining unit 23, and the modeling unit 24 in the scheduling node can all be composed of a central processing unit (CPU, Central Processing Unit), a digital signal in the scheduling unit in practical applications processor (DSP, Digital Signal Processor), Microcontroller Unit (MCU, Microcontroller Unit) or Programmable Gate Array (FPGA, Field-Programmable Gate Array); the acquisition unit 21 in the scheduling node can be used in practical applications. It is realized by communication modules (including basic communication suites, operating systems, communication modules, standardized interfaces and protocols, etc.) and transceiver antennas.
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • MCU Microcontroller Unit
  • FPGA Field-Programmable Gate Array
  • the scheduling node provided by the above embodiment updates the computing node resource information
  • only the division of the above program modules is used as an example for illustration.
  • the above processing can be allocated to different program modules as required.
  • To complete that is, to divide the internal structure of the scheduling node into different program modules to complete all or part of the above-described processing.
  • the scheduling node and computing node resource information updating method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.
  • a scheduling node may include: an information collection submodule, a resource allocation submodule, a task analysis submodule, and a task segmentation sub-module;
  • the information collection sub-module is used to organize computing resource information, establish resource queues, record resource types (such as GPU/CPU/NPU), memory size, idle computing power, existing computing modules and other information; and regularly update resource information, So that the resource allocation sub-module can perform scheduling optimization in real time.
  • resource types such as GPU/CPU/NPU
  • memory size such as GPU/CPU/NPU
  • idle computing power such as GPU/CPU/NPU
  • existing computing modules and other information such as GPU/CPU/NPU
  • resource allocation sub-module can perform scheduling optimization in real time.
  • the task analysis sub-module is used to collect consumer demand, establish a job queue, record the priority of each job, job submission time, and completion time of required calculation.
  • the resource allocation sub-module is used for allocating corresponding resource nodes for each task according to the task requirements and computing resources, and also for the resource nodes to be issued from the computing module provider module request module.
  • the task segmentation sub-module is used to determine how to divide the computing task into sub-tasks according to the task requirements and computing resource allocation results, considering the different computing power of different devices, so that the processing time of the sub-tasks of each device is the same; According to the actual requirements, it is determined that different computing tasks are divided into sub-tasks, so that the sub-tasks of different tasks have the same processing time or a proportional relationship.
  • FIG. 7 is a schematic diagram of the hardware structure of a scheduling node according to an embodiment of the disclosure; as shown in FIG. 7 , the scheduling node includes a memory 32, a processor 31, and a computer program stored in the memory 32 and running on the processor 31, When the processor 31 executes the program, the steps of the computing node resource information update method described in the embodiment of the present disclosure are implemented.
  • the scheduling node also includes one or more network interfaces 33 .
  • the various components in the scheduling unit are coupled together by a bus system 34 .
  • the bus system 34 is used to implement the connection communication between these components.
  • the bus system 34 also includes a power bus, a control bus and a status signal bus.
  • the various buses are labeled as bus system 34 in FIG. 7 .
  • the memory 32 may be either volatile memory or non-volatile memory, and may include both volatile and non-volatile memory.
  • the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-only memory) Only Memory), Electrically Erasable Programmable Read-Only Memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), Magnetic Random Access Memory (FRAM, ferromagnetic random access memory), Flash Memory (Flash Memory), Magnetic Surface Memory , CD-ROM, or CD-ROM (Compact Disc Read-Only Memory); magnetic surface memory can be disk memory or tape memory.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • SSRAM Synchronous Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • SDRAM Synchronous Dynamic Random Access Memory
  • DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • ESDRAM Enhanced Type Synchronous Dynamic Random Access Memory
  • SLDRAM Synchronous Link Dynamic Random Access Memory
  • DRRAM Direct Rambus Random Access Memory
  • the memory 32 described in the embodiments of the present disclosure is intended to include, but not be limited to, these and any other suitable types of memory.
  • the methods disclosed in the above embodiments of the present disclosure may be applied to the processor 31 or implemented by the processor 31 .
  • the processor 31 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by a hardware integrated logic circuit in the processor 31 or an instruction in the form of software.
  • the above-mentioned processor 31 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.
  • the processor 31 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present disclosure.
  • a general purpose processor may be a microprocessor or any conventional processor or the like.
  • the steps of the methods disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software module may be located in a storage medium, and the storage medium is located in the memory 32, and the processor 31 reads the information in the memory 32 and completes the steps of the foregoing method in combination with its hardware.
  • the scheduling node may be implemented by one or more Application Specific Integrated Circuit (ASIC, Application Specific Integrated Circuit), DSP, Programmable Logic Device (PLD, Programmable Logic Device), Complex Programmable Logic Device (CPLD, Complex Programmable Logic Device), FPGA, general-purpose processor, controller, MCU, Microprocessor (Microprocessor), or other electronic components implemented for performing the aforementioned method.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Signal Processing Unit
  • PLD Programmable Logic Device
  • CPLD Complex Programmable Logic Device
  • FPGA general-purpose processor
  • controller MCU
  • Microprocessor Microprocessor
  • an embodiment of the present disclosure further provides a computer-readable storage medium, such as a memory 32 including a computer program, and the computer program can be executed by the processor 31 of the scheduling node to complete the steps of the foregoing method.
  • the computer-readable storage medium can be memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one or any combination of the above memories.
  • the computer-readable storage medium provided by the embodiment of the present disclosure stores a computer program thereon, and when the program is executed by the processor, implements the steps of the method for updating resource information of a computing node described in the embodiment of the present disclosure.
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented.
  • the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.
  • the unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present disclosure may be all integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration
  • the unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.
  • the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute It includes the steps of the above method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk and other media that can store program codes.
  • the above-mentioned integrated units of the present disclosure are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
  • the technical solutions of the embodiments of the present disclosure essentially or the parts that make contributions to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present disclosure.
  • the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk and other mediums that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种计算节点资源信息更新方法、节点和存储介质，涉及边缘计算领域。所述方法包括：获得计算节点待处理的多个计算任务（101）；对所述多个计算任务进行分割，获得对应于各计算任务的子任务（102）；基于多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致（103）。

Description

一种计算节点资源信息更新方法、节点和存储介质
相关申请的交叉引用
本公开基于申请号为202010997282.2、申请日为2020年09月21日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本公开。
技术领域
本公开涉及边缘计算领域，具体涉及一种计算节点资源信息更新方法、节点和存储介质。
背景技术
目前，并未有计算节点资源信息更新策略。若是以固定的小时间间隔对计算节点资源信息进行更新，则持续高频率的更新对调度并没有帮助，反而会造成资源的浪费。若是以固定的大时间间隔对计算节点资源信息进行更新，过于低频的更新可能会造成某些任务无法完成原定目标。例如，如果某个设备发生闲置计算资源的大量减少，在系统进行资源状态更新前，调度节点会继续根据原有状态分发新任务到该节点，而这些新任务可能就会堆积甚至溢出，导致数据丢失、任务无法按时完成等问题。
发明内容
为解决现有存在的技术问题，本公开实施例提供一种计算节点资源信息更新方法、节点和存储介质。
为达到上述目的，本公开实施例的技术方案是这样实现的：
本公开实施例提供了一种计算节点资源信息更新方法，所述方法包括：
获得计算节点待处理的多个计算任务；
对所述多个计算任务进行分割，获得对应于各计算任务的子任务；
基于所述多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。
在本公开的一些可选实施例中，所述对所述多个计算任务进行分割，获得对应于各计算任务的子任务，包括：
基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务。
在本公开的一些可选实施例中，所述基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务，包括：
基于计算任务的任务量、各计算节点的状态以及预先获得的计算模型获得子任务的大小，基于所述子任务的大小对所述计算任务进行分割，获得对应于所述计算任务的多个子任务。
在本公开的一些可选实施例中，所述方法还包括：获得历史样本数据集；所述历史样本数据集包括多组历史样本数据；各组历史样本数据包括计算节点在各个历史时刻的历史状态以及历史计算任务的任务量；
基于所述历史样本数据集中的历史样本数据训练计算模型。
在本公开的一些可选实施例中，所述基于所述历史样本数据训练计算模型，包括：
对所述历史样本数据集中的各组历史样本数据采用特征处理算法得到n个特征，按照预设规则选取所述n个特征中的部分特征作为目标变量集；n为正整数；
对所述历史样本数据集中的历史样本数据进行随机采样，形成N个样 本子集；N为正整数；
基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
本公开实施例还提供了一种调度节点，所述调度节点包括：获取单元、分割单元和确定单元；其中，
所述获取单元，配置为获得计算节点待处理的多个计算任务；
所述分割单元，配置为对所述多个计算任务进行分割，获得对应于各计算任务的子任务；
所述确定单元，配置为基于所述多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。
在本公开的一些可选实施例中，若所述多个子任务的处理时长相同，则所述计算资源信息更新间隔为所述多个子任务的处理时长；
若所述多个子任务的处理时长呈等比关系，则所述计算资源信息更新间隔为所述多个子任务的处理时长中的最小处理时长。
在本公开的一些可选实施例中，所述分割单元，配置为基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务。
在本公开的一些可选实施例中，所述分割单元，配置为基于计算任务的任务量、各计算节点的状态以及预先获得的计算模型获得子任务的大小，基于所述子任务的大小对所述计算任务进行分割，获得对应于所述计算任务的多个子任务。
在本公开的一些可选实施例中，所述调度节点还包括建模单元，配置为获得历史样本数据集；所述历史样本数据集包括多组历史样本数据；各组历史样本数据包括计算节点在各个历史时刻的历史状态以及历史计算任 务的任务量；基于所述历史样本数据集中的历史样本数据训练计算模型。
在本公开的一些可选实施例中，所述建模单元，配置为对所述历史样本数据集中的各组历史样本数据采用特征处理算法得到n个特征，按照预设规则选取所述n个特征中的部分特征作为目标变量集；n为正整数；对所述历史样本数据集中的历史样本数据进行随机采样，形成N个样本子集；N为正整数；基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本公开实施例所述方法的步骤。
本公开实施例还提供了一种调度节点，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现本公开实施例所述方法的步骤。
本公开实施例提供的计算节点资源信息更新方法、节点和存储介质，所述方法包括：获得计算节点待处理的多个计算任务；对所述多个计算任务进行分割，获得对应于各计算任务的子任务；基于所述多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。采用本公开实施例的技术方案，通过将各计算任务分割为子任务，并将计算资源信息更新间隔确定为各子任务的处理时长或者各子任务的处理时长中的最小处理时长，实现了计算节点资源信息更新间隔可基于子任务的处理时长而动态调整，一方面，避免以固定的小时间间隔更新计算节点资源信息带来的资源浪费，也避免以固定的大时间间隔更新计算节点资源信息导致的任务堆积甚至溢出，导致数据丢失、任务无法按时完成等问题；另一方面，若计算节点的资源情况发生变化，调度节点也能够在计算资源信息更新间隔这个周期内， 也即子任务的处理完成之前或者子任务处理完成时，获得计算节点的资源变化情况，以便于调度节点能够根据计算节点的当前实际资源情况进行任务调度，从而降低因为计算节点的资源变化而调度节点的计算资源信息更新不及时导致的任务延迟，同时减少任务协同所消耗的计算资源。
附图说明
图1为本公开实施例的计算节点资源信息更新方法应用的系统架构的一种示意图；
图2为本公开实施例的计算节点资源信息更新方法的流程示意图；
图3为本公开实施例的计算节点资源信息更新方法中的计算模型建立的示意图；
图4为本公开实施例的调度节点的组成结构示意图一；
图5为本公开实施例的调度节点的组成结构示意图二；
图6为本公开实施例的计算节点资源信息更新方法应用的系统架构的另一种示意图；
图7为本公开实施例的调度节点的硬件结构示意图。
具体实施方式
下面结合附图及具体实施例对本公开作进一步详细的说明。
图1为本公开实施例的计算节点资源信息更新方法应用的系统架构示意图；如图1所示，系统可包括：计算模块、调度与资源分配模块和消费者模块。其中，
消费者模块，作为计算资源的消费者，能够产生任务数据，并接受任务数据对应的计算结果，基于计算结果开展更多的扩展应用。例如，语音分析与识别的计算结果可以被语音盒子消费，用于进一步交流的输入，也可以提供给空调、灯等其他智能化设备，进行智能家居控制操作。在一种 示例中，消费者模块可以是网络摄像头，则网络摄像头采集到的视频数据可作为任务数据。在另一种示例中，消费者模块可以是麦克风，则麦克风采集到的音频数据可作为任务数据。
计算模块，将通用的计算能力作为模块提供给调度与资源分配模块，能够对任务数据进行计算。示例性的计算能力可包括：图像分析与识别能力、语音分析与识别能力等等。
调度与资源分配模块，该模块根据消费者模块的请求，管理闲置的计算资源，将需求与供给进行优化匹配，实现任务的完成。
基于图1所示的系统架构，提出本公开以下各实施例。
需要说明的是，本公开以下各实施例中的计算节点，相当于图1中所示的计算模块；调度节点，相当于图1中所示的调度与资源分配模块；消费者节点，相当于图1中所示的消费者模块。
本公开实施例提供了一种计算节点资源信息更新方法，应用于调度节点中。图2为本公开实施例的计算节点资源信息更新方法的流程示意图；如图2所示，所述方法包括：
步骤101：获得计算节点待处理的多个计算任务；
步骤102：对所述多个计算任务进行分割，获得对应于各计算任务的子任务；
步骤103：基于多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。
本实施例中，调度节点从消费者节点中获得计算任务。示例性的，若消费者节点为网络摄像头，则所述计算任务可以是网络摄像头采集的视频数据。
在一些可选实施例中，一方面，调度节点从消费者节点中获得计算任务的任务数据；另一方面，调度节点从消费者节点中获得任务处理请求，则调度节点基于该任务处理请求对获得的任务数据进行分割，得到多个子任务。可选地，所述任务处理请求中可包括表示任务的指示信息；该任务例如可以是识别图像中的目标人物、识别图像中的人物数量、或者识别音频数据中的目标人物等等。示例性的，调度节点中可预先配置各任务以及对应的标识；识别任务处理请求中携带的指示标识，根据该指示标识可确定对应的目标任务。
本实施例中，调度节点对各个计算任务进行分割，将每个计算任务分割为多个子任务。其中，对应于各计算任务的子任务的处理时长相同或呈等比关系。示例性的，对应于一个计算任务的多个子任务的处理时长相同。
在本公开的一些可选实施例中，所述对所述多个计算任务进行分割，获得对应于各计算任务的子任务，包括：基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务。
本实施例中，调度节点可按照预设规则获得各计算节点的状态。其中，计算节点的状态可包括：各类型资源的相关信息、内存的相关信息、闲置算力、已使用的计算资源的相关信息等等。其中，内存的相关信息可包括内存使用率。资源的类型可包括：中央处理器(CPU，Central Processing Unit)、图形处理器(GPU，Graphics Processing Unit)、神经网络处理器(NPU，Neural Network Processing Unit)等等；则各类型资源的相关信息可包括CPU使用率、GPU使用率、NPU使用率等等。可选地，计算节点的状态还可包括：已有进程数、节点设备的温度、任务类型等等。
示例性的，调度节点可按照预设周期获得各计算节点的状态；其中，所述预设周期可按照实际需求设置，本实施例中对此不做限定。
本实施例中，调度节点可基于各计算节点的状态结合各计算任务的任 务量大小对各计算任务进行分割，从而使得分割后的对应于各计算节点的子任务在对应的计算节点的处理时长相同或呈等比关系。
例如，计算任务1分割后，其子任务的处理时长为t_1；计算任务2由于其流程需要减少数据读取次数，其子任务设置略大，处理时长为t_2＝2×t_1；计算任务3由于其处理时长要求尽可能短，所以将其子任务设置略小，处理时长为t_3＝t_1/2；所以三个计算任务的子任务的处理时长为等比关系，即t_1：t_2：t_3＝2：4：1。
在一些可选实施例中，若所述多个子任务的处理时长相同，则所述计算资源信息更新间隔为所述多个子任务的处理时长；若所述多个子任务的处理时长呈等比关系，则所述计算资源信息更新间隔为所述多个子任务的处理时长中的最小处理时长。
一个示例中，以上述示例为例，则所述计算资源信息更新间隔为上述t_3。
另一个示例中，若三个计算任务对应的子任务的处理时长均相同，则所述计算资源信息更新间隔三个计算任务对应的子任务的处理时长。
在一些可选实施例中，调度节点根据计算节点当前运行状态、计算任务的处理时长以及上一任务的等待时长对计算任务进行分割，获得子任务。
示例性的，调度节点分割子任务时，需考根据计算节点当前运行状态考虑分配任务量，当节点计算能力被占满时，需根据任务处理时长与上一任务等待时长分割子任务。
在本公开的一些可选实施例中，所述基于所述计算任务的任务量以及各计算节点的状态对所述计算任务进行分割，获得多个子任务，包括：基于所述计算任务的任务量、各计算节点的状态以及预先获得的计算模型获得个子任务的大小，基于所述子任务的大小对所述计算任务进行分割，获得多个子任务。
在本公开的一些可选实施例中，所述方法还包括：获得历史样本数据集；所述历史样本数据集包括多组历史样本数据；各组历史样本数据包括计算节点在各个历史时刻的历史状态以及历史计算任务的任务量；基于所述历史样本数据集中的历史样本数据训练计算模型。
本实施例中，调度节点可预先建立计算模型，基于计算模型确定各计算任务的子任务的任务量大小。
示例性的，获得历史样本数据集，历史样本数据集中包括多组历史样本数据，每组历史样本数据中可包括N个历史时刻的计算节点的历史状态X以及历史计算任务的任务量大小Y，基于历史状态X和历史计算任务的任务量大小Y训练计算模型。其中，历史状态X可包括以下至少之一：CPU使用率、GPU使用率、NPU使用率、内存使用率、已有进程数、节点设备的温度、任务类型等等。
在本公开的一些可选实施例中，所述基于所述历史样本数据训练计算模型，包括：对所述历史样本数据集中的各组历史样本数据采用特征处理算法得到n个特征，按照预设规则选取所述n个特征中的部分特征作为目标变量集；n为正整数；对所述历史样本数据集中的历史样本数据进行随机采样，形成N个样本子集；N为正整数；基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
示例性的，所述特征处理算法包括但不限于主成分分析方法、方差选择法、相关系数法等。
本实施例中，具体可参照图3所示，首先对各组历史样本数据采用主成分分析(PCA，Principal Component Analysis)方法得到n个特征，按照预设规则选取所述n个特征中的部分特征{VS 1，VS 2，VS 3…VS m}作为目标变量集；其中，所述预设规则包括：根据不同特征在整体中的贡献，按照特定比例进行保留，以及保留所有大于1的特征值。其中，目标变量集中 的各特征是历史样本数据中的部分特征的加权组合。
其次，对历史样本数据集中的历史样本数据进行随机采样，得到N个样本子集。
最后，基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
示例性的，所述基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型，包括：基于目标变量集，对各个样本子集进行建模。例如可采用线性回归算法进行建模，得到各个样本子集对应的模型，并基于各样本子集对应的模型得到各个样本子集中的历史计算任务的任务量大小Y对应的预测结果，该预测结果为任务量大小Y对应的预测的子任务大小y i；采用Ensemble集成算法、Bagging、boosting或随机森林的方式将多个样本子集对应的模型进行融合得到本实施例中的计算模型。
采用本公开实施例的技术方案，通过将各计算任务分割为子任务，并将计算资源信息更新间隔确定为各子任务的处理时长或者各子任务的处理时长中的最小处理时长，实现了计算节点资源信息更新间隔可基于子任务的处理时长而动态调整，一方面，避免以固定的小时间间隔更新计算节点资源信息带来的资源浪费，也避免以固定的大时间间隔更新计算节点资源信息导致的任务堆积甚至溢出，导致数据丢失、任务无法按时完成等问题；另一方面，若计算节点的资源情况发生变化，调度节点也能够在计算资源信息更新间隔这个周期内，也即子任务的处理完成之前或者子任务处理完成时，获得计算节点的资源变化情况，以便于调度节点能够根据计算节点的当前实际资源情况进行任务调度，从而降低因为计算节点的资源变化而调度节点的计算资源信息更新不及时导致的任务延迟，同时减少任务协同所消耗的计算资源。
示例性的，本实施例的系统可应用于以下场景：
以视频任务为例，系统把多个边缘设备连接形成行的视频处理通道。多个视频处理通道可实现对实时视频的并行处理。实时视频通过切片，分为多个视频任务包，分别分配到各个视频处理通道进行处理。处理完的视频数据融合到一起形成实时的视频处理结果流。
在视频任务的场景中，消费者模块(如摄像头)提供实时视频数据，向调度节点请求视频处理能力，调度节点根据各计算节点的状态，一方面请求视频处理任务数据下发到相应的计算节点，另一方面将实时视频数据进行切割，分配给相应的计算节点，由计算节点执行相应的处理任务。调度节点还会持续更新各计算节点的资源状况，当必要时进行任务调度的调整。
基于前述方法实施例，本公开实施例还提供了一种调度节点。图4为本公开实施例的调度节点的组成结构示意图；如图4所示，所述调度节点包括：获取单元21、分割单元22和确定单元23；其中，
所述获取单元21，配置为获得计算节点待处理的多个计算任务；
所述分割单元22，配置为对所述多个计算任务进行分割，获得对应于各计算任务的子任务；
所述确定单元23，配置为基于多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。
在本公开的一些可选实施例中，所述分割单元22，配置为基于各个计算任务的任务量以及各计算节点的状态对所述计算任务进行分割，获得子任务。
示例性的，分割单元分割子任务时，需考根据计算节点当前运行状态考虑分配任务量，当节点计算能力被占满时，需根据任务处理时长与上一 任务等待时长分割子任务。
在本公开的一些可选实施例中，所述分割单元22，配置为基于计算任务的任务量、各计算节点的状态以及预先获得的计算模型获得个子任务的大小，基于所述子任务的大小对所述计算任务进行分割，获得对应于所述计算任务的多个子任务。
在本公开的一些可选实施例中，如图5所示，所述调度节点还包括建模单元24，配置为获得历史样本数据集；所述历史样本数据集包括多组历史样本数据；各组历史样本数据包括计算节点在各个历史时刻的历史状态以及历史计算任务的任务量；基于所述历史样本数据集中的历史样本数据训练计算模型。
在本公开的一些可选实施例中，所述建模单元24，配置为对所述历史样本数据集中的各组历史样本数据采用特征处理算法得到n个特征，按照预设规则选取所述n个特征中的部分特征作为目标变量集；n为正整数；对所述历史样本数据集中的历史样本数据进行随机采样，形成N个样本子集；N为正整数；基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
示例性的，所述特征处理算法包括但不限于主成分分析方法、方差选择法、相关系数法等。
本公开实施例中，所述调度节点中的分割单元22、确定单元23和建模单元24，在实际应用中均可由所述调度单元中的中央处理器(CPU，Central Processing Unit)、数字信号处理器(DSP，Digital Signal Processor)、微控制单元(MCU，Microcontroller Unit)或可编程门阵列(FPGA，Field－Programmable Gate Array)实现；所述调度节点中的获取单元21，在实际应用中可通过通信模组(包含：基础通信套件、操作系统、通信模块、标准化接口和协议等)及收发天线实现。
需要说明的是：上述实施例提供的调度节点在进行计算节点资源信息更新时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将调度节点的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的调度节点与计算节点资源信息更新方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
图6为本公开实施例的计算节点资源信息更新方法应用的系统架构的另一种示意图；如图6所示，调度节点可包括：信息收集子模块、资源分配子模块、任务分析子模块和任务分割子模块；其中，
信息收集子模块，用于整理计算资源信息，建立资源队列，记录资源的类型(例如GPU/CPU/NPU)、内存大小、闲置算力、已有计算模块等信息；并定期进行资源信息更新，以便资源分配子模块即时进行调度优化。
任务分析子模块，用于收集消费者需求，建立作业队列，记录每个作业的优先级、作业提交时间、要求计算完毕时间。
资源分配子模块，用于根据任务需求与计算资源，为每个任务分配相应的资源节点，也为资源节点从计算模组提供者模块请求模块下发。
任务分割子模块，用于根据任务需求与计算资源分配结果，考虑不同设备的算力不同，确定计算任务如何进行分割为子任务，使得每个设备的子任务处理时间相同；考虑不同计算任务的实际需求，确定不同计算任务分为子任务，使得不同任务的子任务处理时间相同或者为等比关系。
可以理解，上述信息收集子模块、资源分配子模块、任务分析子模块和任务分割子模块相当于图4中所示的获取单元、分割单元和确定单元，仅是各处理单元的功能划分方式的不同。
本公开实施例还提供了一种调度节点。图7为本公开实施例的调度节点的硬件结构示意图；如图7所示，所述调度节点包括存储器32、处理器 31及存储在存储器32上并可在处理器31上运行的计算机程序，所述处理器31执行所述程序时实现本公开实施例所述的计算节点资源信息更新方法的步骤。
可选地，调度节点还包括一个或多个网络接口33。调度单元中的各个组件通过总线系统34耦合在一起。可理解，总线系统34用于实现这些组件之间的连接通信。总线系统34除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图7中将各种总线都标为总线系统34。
可以理解，存储器32可以是易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(ROM，Read Only Memory)、可编程只读存储器(PROM，Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM，ferromagnetic random access memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM，Compact Disc Read-Only Memory)；磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM，Random Access Memory)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM，Static Random Access Memory)、同步静态随机存取存储器(SSRAM，Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM，Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM，Synchronous Dynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM，Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态 随机存取存储器(ESDRAM，Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM，SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM，Direct Rambus Random Access Memory)。本公开实施例描述的存储器32旨在包括但不限于这些和任意其它适合类型的存储器。
上述本公开实施例揭示的方法可以应用于处理器31中，或者由处理器31实现。处理器31可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器31中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器31可以是通用处理器、DSP，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。处理器31可以实现或者执行本公开实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本公开实施例所公开的方法的步骤，可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中，该存储介质位于存储器32，处理器31读取存储器32中的信息，结合其硬件完成前述方法的步骤。
在示例性实施例中，调度节点可以被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，Programmable Logic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable Logic Device)、FPGA、通用处理器、控制器、MCU、微处理器(Microprocessor)、或其他电子元件实现，用于执行前述方法。
在示例性实施例中，本公开实施例还提供了一种计算机可读存储介质，例如包括计算机程序的存储器32，上述计算机程序可由调度节点的处理器31执行，以完成前述方法所述步骤。计算机可读存储介质可以是FRAM、ROM、PROM、EPROM、EEPROM、Flash Memory、磁表面存储器、光盘、 或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备。
本公开实施例提供的计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本公开实施例所述的计算节点资源信息更新方法的步骤。
本公开所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得到新的方法实施例。
本公开所提供的几个产品实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的产品实施例。
本公开所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
在本公开所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元，即可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。
另外，在本公开各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可 以采用硬件加软件功能单元的形式实现。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
或者，本公开上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。

Claims (12)

  1. 一种计算节点资源信息更新方法，所述方法包括：
    获得计算节点待处理的多个计算任务；
    对所述多个计算任务进行分割，获得对应于各计算任务的子任务；
    基于多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。
  2. 根据权利要求1所述的方法，其中，所述对所述多个计算任务进行分割，获得对应于各计算任务的子任务，包括：
    基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务。
  3. 根据权利要求2所述的方法，其中，所述基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务，包括：
    基于计算任务的任务量、各计算节点的状态以及预先获得的计算模型获得子任务的大小，基于所述子任务的大小对所述计算任务进行分割，获得对应于所述计算任务的多个子任务。
  4. 根据权利要求3所述的方法，其中，所述方法还包括：
    获得历史样本数据集；所述历史样本数据集包括多组历史样本数据；各组历史样本数据包括计算节点在各个历史时刻的历史状态以及历史计算任务的任务量；
    基于所述历史样本数据集中的历史样本数据训练计算模型。
  5. 根据权利要求4所述的方法，其中，所述基于所述历史样本数据训练计算模型，包括：
    对所述历史样本数据集中的各组历史样本数据采用特征处理算法得到n个特征，按照预设规则选取所述n个特征中的部分特征作为目标变量集； n为正整数；
    对所述历史样本数据集中的历史样本数据进行随机采样，形成N个样本子集；N为正整数；
    基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
  6. 一种调度节点，所述调度节点包括：获取单元、分割单元和确定单元；其中，
    所述获取单元，配置为获得计算节点待处理的多个计算任务；
    所述分割单元，配置为对所述多个计算任务进行分割，获得对应于各计算任务的子任务；
    所述确定单元，配置为基于多个子任务的处理时长确定计算资源信息更新间隔；所述计算资源信息更新间隔与分割后的子任务的处理时长一致，或者，所述计算资源信息更新间隔与分割后的多个子任务的处理时长中的最小处理时长一致。
  7. 根据权利要求6所述的调度节点，其中，所述分割单元，配置为基于各个计算任务的任务量以及各计算节点的状态对计算任务进行分割，获得子任务。
  8. 根据权利要求7所述的调度节点，其中，所述分割单元，配置为基于计算任务的任务量、各计算节点的状态以及预先获得的计算模型获得子任务的大小，基于所述子任务的大小对所述计算任务进行分割，获得对应于所述计算任务的多个子任务。
  9. 根据权利要求8所述的调度节点，其中，所述调度节点还包括建模单元，配置为获得历史样本数据集；所述历史样本数据集包括多组历史样本数据；各组历史样本数据包括计算节点在各个历史时刻的历史状态以及历史计算任务的任务量；基于所述历史样本数据集中的历史样本数据训练 计算模型。
  10. 根据权利要求9所述的调度节点，其中，所述建模单元，配置为对所述历史样本数据集中的各组历史样本数据采用特征处理算法得到n个特征，按照预设规则选取所述n个特征中的部分特征作为目标变量集；n为正整数；对所述历史样本数据集中的历史样本数据进行随机采样，形成N个样本子集；N为正整数；基于所述目标变量集和所述N个样本子集进行建模，得到所述计算模型。
  11. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现权利要求1至5任一项所述方法的步骤。
  12. 一种调度节点，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1至5任一项所述方法的步骤。
PCT/CN2021/119600 2020-09-21 2021-09-22 一种计算节点资源信息更新方法、节点和存储介质 WO2022057940A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010997282.2A CN114253695A (zh) 2020-09-21 2020-09-21 一种计算节点资源信息更新方法、节点和存储介质
CN202010997282.2 2020-09-21

Publications (1)

Publication Number Publication Date
WO2022057940A1 true WO2022057940A1 (zh) 2022-03-24

Family

ID=80778244

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/119600 WO2022057940A1 (zh) 2020-09-21 2021-09-22 一种计算节点资源信息更新方法、节点和存储介质

Country Status (2)

Country Link
CN (1) CN114253695A (zh)
WO (1) WO2022057940A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114697209A (zh) * 2022-03-30 2022-07-01 广州穗华能源科技有限公司 一种云边协同计算资源配置方法及配置系统
CN114978908A (zh) * 2022-05-11 2022-08-30 量子科技长三角产业创新中心 算力网络节点评价及操作方法和装置
CN115426363A (zh) * 2022-08-29 2022-12-02 广东鑫光智能系统有限公司 智能板材加工工厂的数据采集方法及终端
CN117278100A (zh) * 2023-09-21 2023-12-22 之江实验室 一种基于天基信息系统的业务执行方法、装置及存储介质
WO2024082692A1 (zh) * 2022-10-21 2024-04-25 华为技术有限公司 执行任务的方法和异构服务器

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8990383B1 (en) * 2011-11-23 2015-03-24 Amazon Technologies, Inc. Elastic service level agreement for metrics accuracy computation
CN109298940A (zh) * 2018-09-28 2019-02-01 考拉征信服务有限公司 计算任务分配方法、装置、电子设备及计算机存储介质
CN109815011A (zh) * 2018-12-29 2019-05-28 东软集团股份有限公司 一种数据处理的方法和装置
CN111338791A (zh) * 2020-02-12 2020-06-26 平安科技（深圳）有限公司 集群队列资源的调度方法、装置、设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8990383B1 (en) * 2011-11-23 2015-03-24 Amazon Technologies, Inc. Elastic service level agreement for metrics accuracy computation
CN109298940A (zh) * 2018-09-28 2019-02-01 考拉征信服务有限公司 计算任务分配方法、装置、电子设备及计算机存储介质
CN109815011A (zh) * 2018-12-29 2019-05-28 东软集团股份有限公司 一种数据处理的方法和装置
CN111338791A (zh) * 2020-02-12 2020-06-26 平安科技（深圳）有限公司 集群队列资源的调度方法、装置、设备及存储介质

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114697209A (zh) * 2022-03-30 2022-07-01 广州穗华能源科技有限公司 一种云边协同计算资源配置方法及配置系统
CN114697209B (zh) * 2022-03-30 2023-12-22 广州穗华能源科技有限公司 一种云边协同计算资源配置方法及配置系统
CN114978908A (zh) * 2022-05-11 2022-08-30 量子科技长三角产业创新中心 算力网络节点评价及操作方法和装置
CN114978908B (zh) * 2022-05-11 2023-09-26 量子科技长三角产业创新中心 算力网络节点评价及操作方法和装置
CN115426363A (zh) * 2022-08-29 2022-12-02 广东鑫光智能系统有限公司 智能板材加工工厂的数据采集方法及终端
WO2024082692A1 (zh) * 2022-10-21 2024-04-25 华为技术有限公司 执行任务的方法和异构服务器
CN117278100A (zh) * 2023-09-21 2023-12-22 之江实验室 一种基于天基信息系统的业务执行方法、装置及存储介质
CN117278100B (zh) * 2023-09-21 2024-04-26 之江实验室 一种基于天基信息系统的业务执行方法、装置及存储介质

Also Published As

Publication number Publication date
CN114253695A (zh) 2022-03-29

Similar Documents

Publication Publication Date Title
WO2022057940A1 (zh) 一种计算节点资源信息更新方法、节点和存储介质
WO2021139438A1 (zh) 大数据资源处理方法、装置、终端及存储介质
CN115422284B (zh) 存储设备、分布式存储系统以及数据处理方法
US8756209B2 (en) Computing resource allocation based on query response analysis in a networked computing environment
JP2015133112A (ja) ジョブスケジューリング方法、データアナライザ、データ解析装置、コンピュータシステム及びコンピュータ可読媒体
WO2022171066A1 (zh) 基于物联网设备的任务分配方法、网络训练方法及装置
CN112148468A (zh) 一种资源调度方法、装置、电子设备及存储介质
WO2021022660A1 (zh) 一种特征预测方法、系统及引擎
CN111124644B (zh) 任务调度资源的确定方法、装置及系统
CN115794341A (zh) 基于人工智能的任务调度方法、装置、设备及存储介质
CN115586961A (zh) 一种ai平台计算资源任务调度方法、装置及介质
CN113568747A (zh) 基于任务分类与时序预测的云机器人资源调度方法及系统
CN112182111A (zh) 基于区块链的分布式系统分层处理方法和电子设备
Sandholm et al. QoS-based pricing and scheduling of batch jobs in openstack clouds
CN116896591A (zh) 网络数据分析模型调度方法、装置和计算机设备
CN113742059B (zh) 任务分配方法、装置、计算机设备和存储介质
CN115826729A (zh) 硬盘电源管理方法、装置、电子设备、介质及产品
CN115658287A (zh) 一种用于调度运行单元的方法、设备、介质及程序产品
CN114764372A (zh) 数据处理方法、装置、电子设备和存储介质
CN114296965A (zh) 特征检索方法、装置、电子设备及计算机存储介质
JP6753521B2 (ja) 計算資源管理装置、計算資源管理方法、及びプログラム
US11354592B2 (en) Intelligent computation acceleration transform utility
CN112953993A (zh) 资源调度方法、设备、网络系统及存储介质
Volpe et al. A Deep Reinforcement Learning Approach for Competitive Task Assignment in Enterprise Blockchain
WO2023246709A1 (zh) 数据处理方法、装置、设备和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21868762

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM XXXX DATED 03/07/2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21868762

Country of ref document: EP

Kind code of ref document: A1