CN116521335A - Distributed task scheduling method and system for inclined image model production - Google Patents

Abstract

The invention relates to resource allocation scheduling, in particular to a distributed task scheduling method and system for oblique image model production, wherein the method comprises the following steps: the real-time hardware score of each computing node is collected and updated regularly; dividing the task to be processed into task packets; and aiming at different task types, setting limiting indexes, selecting corresponding calculation nodes with highest scores according to real-time indexes, respectively carrying out feature matching calculation, block adjustment calculation and fusion adjustment calculation on the task package to derive xml files and bin files corresponding to the task package, and selecting the calculation node with highest comprehensive scores of the current CPU to carry out modeling calculation. Therefore, by collecting hardware parameters and monitoring real-time indexes, available resources are accurately scored on each computing node, and accordingly task packages are distributed to the computing node with the highest current adaptation degree for computing, node performance is fully exerted, and computing efficiency is improved. And the dispatching node can be accessed by multiple platforms and is suitable for various system environments.

Description

Distributed task scheduling method and system for oblique image model production

technical field

The invention relates to the technical field of resource allocation and scheduling, in particular to a distributed task scheduling method and system for oblique image model production.

Background technique

Oblique photography refers to synchronously collecting images from one vertical angle of view and four oblique angles of view, and then obtains a high-precision and high-resolution real-world 3D model through 3D reconstruction. Both have substantial advantages. In recent years, drones equipped with oblique photography equipment have been applied on a large scale, which has greatly reduced the hardware threshold for oblique photography, and thus collected and produced a huge amount of original image data. On the other hand, the 3D reconstruction work has great requirements on the computing power of the host computer. Users may have the ability to collect original image data, but lack the computing power to efficiently perform model production calculations on the original image data. Therefore, at present, task subcontracting is mostly carried out for computing tasks, and distributed computing is realized through distributed computing frameworks such as Hadoop and Storm to avoid the problem of insufficient local computing power.

However, the above conventional distributed computing frameworks are not friendly enough for the Windows environment, and the computing resources required for different computing tasks are quite different. For example, high GPU performance is required for feature extraction, and high GPU performance is required for feature matching. High CPU multi-thread computing capability, in addition to CPU multi-thread computing capability, also requires high memory capacity when performing 3D modeling, while the existing conventional distributed computing framework lacks a resource allocation mechanism, and task division is not reasonable enough , and after the task is divided into multiple task operators, the task operator cannot be allocated to the computing node with the highest computing efficiency, the performance of each computing node cannot be fully utilized, and the computing efficiency is not good.

Contents of the invention

The purpose of the present invention is to solve the deficiencies of the prior art. The present invention provides a distributed task scheduling method and system for oblique image model production. By collecting hardware parameters and monitoring real-time indicators, the available resources of each computing node can be accurately scored. Based on this, the task package is distributed to the computing node with the highest degree of fitness for calculation, so as to give full play to the performance of the node and improve the computing efficiency. And the scheduling node can be accessed by multiple platforms, adapting to various system environments.

The first aspect of the present invention discloses a distributed task scheduling method for oblique image model production, which is characterized in that it includes:

Regularly collect and update the real-time hardware score of each computing node;

The tasks to be processed are divided into several task packages;

Select the computing node with the highest current GPU score to perform feature extraction calculations for each task package;

Select the computing node with the highest current CPU multi-core score, and perform feature matching calculation on the task package with successful feature extraction and calculation;

Select the computing node with the highest comprehensive score of the current CPU, and perform block adjustment calculation for the task package whose feature matching calculation is successful;

Select the computing node with the highest comprehensive score of the current CPU, and perform fusion adjustment calculation on the task package whose block adjustment calculation is successful;

Export the corresponding xml file and bin file for the task package successfully calculated by fusion adjustment;

Select the computing node with the highest comprehensive score of the current CPU, and perform modeling calculation on the xml file and the bin file.

Preferably, based on the scoring items and corresponding device parameters of the hardware devices included in each computing node, evaluate and obtain the benchmark hardware score of each computing node;

The real-time hardware score is obtained by weighting and adjusting the benchmark hardware score based on the real-time occupancy rate of the hardware devices included in each computing node;

Wherein, the hardware device includes at least CPU, memory and GPU;

Moreover, the scoring items for CPU include at least the number of CPU cores and CPU turbo frequency, the scoring items for memory include memory space, and the scoring items for GPU include at least the number of GPU stream processors and GPU memory space.

Preferably, a benchmark value is set for each scoring item, and a benchmark score corresponding to the benchmark value is set;

Among them, the benchmark value of the number of CPU cores is 4, and the benchmark score of CPU cores is 50;

The CPU turbo frequency benchmark value is 3GHz, and the CPU turbo frequency benchmark score is 50;

The memory space benchmark value is 16GB, and the memory benchmark score is 50;

The benchmark value for the number of GPU stream processors is 1280, and the benchmark score for GPU stream processors is 50;

The GPU memory space benchmark value is 4GB, and the GPU memory benchmark score is 50.

Preferably, based on the benchmark value and the benchmark score, evaluate the CPU score corresponding to the CPU performance of each computing node, the memory score corresponding to the memory performance, the GPU score corresponding to the GPU performance, and the corresponding CPU and memory comprehensive performance Comprehensive score of CPU and memory;

Wherein, the CPU score includes at least a CPU single-core score, a CPU multi-core score, and a CPU comprehensive score.

Preferably, CPU single-core score=current CPU turbo value/CPU turbo benchmark value*CPU core benchmark score;

CPU multi-core score = (current CPU turbo frequency value/CPU turbo frequency benchmark value)*(current CPU core number/CPU core number benchmark value)*CPU core benchmark score;

CPU comprehensive score = CPU single-core score * 0.99 + CPU multi-core score * 0.01;

Memory score = (current memory space value/memory space benchmark value)*memory benchmark score;

GPU score = (current GPU stream processor number/GPU stream processor number benchmark value)*GPU video memory benchmark score;

Comprehensive score of CPU and memory = CPU multi-core score + memory score.

Preferably, several limit indicators are set to calibrate the equipment performance requirements corresponding to different tasks;

Set the first limit index as CPU usage < 90%, and remaining GPU memory space > 1GB;

Set the second limit index as CPU usage < 90%, and remaining memory space > 1GB;

Set the third limit index as CPU usage < 90%, and remaining memory space > 4GB;

Set the fourth limit index as CPU usage < 90%, and remaining memory space > (task package contains the number of movies * 0.0015) GB;

Set the fifth limit index as CPU usage < 90%, and remaining memory space > 12GB.

Preferably, the task package that fails in feature extraction, feature matching, block adjustment, fusion adjustment or modeling calculation is marked as a redistribution task, and the number of calculation failures is updated;

Among them, the calculation failure refers to an error in the calculation process or a calculation result verification that does not conform to the rules;

Re-select the current optimal computing node for redistribution tasks with less than 3 failure times, and perform feature extraction, feature matching, block adjustment, fusion adjustment or modeling calculation.

Preferably, the redistribution task whose number of failures reaches 3 is marked as a failed task, the dispatch and calculation process of the failed task is terminated, and the calculation failure information is returned

A second aspect of the present invention discloses a system, characterized in that the system includes:

Data nodes, scheduling nodes, and computing nodes;

Each data node is connected to at least one scheduling node data;

Each computing node is connected to at least one scheduling node data;

The data node is used to preprocess the task to be processed into several task packages;

The dispatching node performs access control through the visualization terminal to collect hardware parameters and update real-time indicators of the computing nodes connected to it.

Preferably, the scheduling node is also used to select the optimal calculation for feature extraction, feature matching, block adjustment, fusion adjustment or modeling process of each task package according to the hardware parameters and real-time indicators of each computing node. Nodes perform task subcontracting.

It can be seen that by collecting hardware parameters and monitoring real-time indicators, the available resources of each computing node are accurately scored, and the task package is distributed to the computing node with the highest current fitness for calculation, so as to fully utilize the node performance and improve computing efficiency. And the scheduling node can be accessed by multiple platforms, adapting to various system environments.

Description of drawings

Fig. 1 is a schematic flow chart of a distributed task scheduling method for oblique image model production of the present invention;

Fig. 2 is a schematic structural diagram of a distributed task scheduling system for oblique image model production of the present invention;

Fig. 3 is a schematic diagram of the operation flow of a distributed task scheduling system for oblique image model production according to the present invention.

Detailed ways

In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples of implementation and accompanying drawings. The present invention can be implemented in the following ways:

Embodiment one

Please refer to Figure 1, a distributed task scheduling method for oblique image model production, which may include the following steps:

101. Periodically collect and update the real-time hardware score of each computing node.

In this embodiment, based on the scoring items of the hardware devices included in each computing node and the corresponding device parameters, the benchmark hardware score of each computing node is obtained through evaluation;

The real-time hardware score is obtained by weighting and adjusting the benchmark hardware score based on the real-time occupancy rate of the hardware devices included in each computing node;

Among them, hardware devices include at least CPU, memory and GPU;

Moreover, the scoring items for CPU include at least the number of CPU cores and CPU turbo frequency, the scoring items for memory include memory space, and the scoring items for GPU include at least the number of GPU stream processors and GPU memory space.

As an optional implementation, computing nodes can be used as workstations for professional computing services, or as personal computers that have opened cloud computing services, and even if they are both workstations or personal computers, there are differences between their configurations. A huge difference. Different types of computing tasks have specific hardware requirements, so the online computing nodes can be scored and sorted in real time according to the benchmark hardware scores, so that tasks can be preferentially dispatched to high-scoring computing nodes for processing to obtain the highest processing efficiency.

In this embodiment, considering that the computing nodes may be processing other tasks at the same time, if tasks are dispatched based on their benchmark hardware scores, only their idle computing resources can be used for computing processing after dispatching, that is, the actual computing capabilities of the computing nodes and their benchmark The hardware scores are inconsistent, and their actual computing power fluctuates normally. Therefore, real-time hardware scores are introduced here to evaluate the actual idle computing power based on the current actual occupancy of each hardware device in the computing node.

As an optional implementation, the real-time occupancy rate of computing nodes can be directly multiplied by the benchmark hardware score of computing nodes to calculate the actual idle computing power of each computing node, and then calculate the Sorting is carried out, and task dispatching is carried out accordingly, so as to avoid dispatching processing tasks to computing nodes with insufficient redundancy, which will cause blockage of the task queue.

As another optional implementation, the idle computing power of the hardware device and its actual processing capacity are not an accurate linear relationship. When the processing efficiency is substantially reduced, although it has a 10% idle rate, due to the limitations of equipment heat dissipation and data channel and other factors, the calculation power of 10% of the equipment parameters cannot be fully invoked. Therefore, multiple stages can be set according to the operating characteristics of the hardware equipment. For example, in the case of idle rate ≥ 50%, the real-time hardware score is directly multiplied by the idle rate by the benchmark hardware score; in the case of 50% > idle rate ≥ 10%, the real-time hardware score is multiplied by the idle rate The benchmark hardware score is multiplied by a weighting factor of 0.9; when the idle rate is less than 10%, the real-time hardware score is multiplied by the idle rate by the benchmark hardware score and then multiplied by a weighting factor of 0.6; in order to more accurately measure each calculation The actual computing power of the nodes ensures that there is no blockage in the task processing process, and the task packages are allocated more reasonably.

In this embodiment, a benchmark value is set for each scoring item, and a benchmark score corresponding to the benchmark value is set;

Among them, the benchmark value of the number of CPU cores is 4, and the benchmark score of CPU cores is 50;

The CPU turbo frequency benchmark value is 3GHz, and the CPU turbo frequency benchmark score is 50;

The memory space benchmark value is 16GB, and the memory benchmark score is 50;

The benchmark value for the number of GPU stream processors is 1280, and the benchmark score for GPU stream processors is 50;

The GPU memory space benchmark value is 4GB, and the GPU memory benchmark score is 50.

Here, by setting various benchmark parameters and benchmark scores, it is possible to efficiently and accurately achieve standardized scoring for hardware devices of different computing nodes, so that the computing performance of computing nodes can be clearly seen, and it is convenient to accurately allocate task packages.

In this embodiment, based on the benchmark value and the benchmark score, evaluate the CPU score corresponding to the CPU performance of each computing node, the memory score corresponding to the memory performance, the GPU score corresponding to the GPU performance, and the corresponding CPU and memory comprehensive performance Comprehensive score of CPU and memory;

Among them, the CPU score includes at least a CPU single-core score, a CPU multi-core score, and a CPU comprehensive score.

Here, according to the common hardware performance requirements of image processing, the above data-based scores are generated, so that only by scoring, the computing nodes can be ranked according to different performance requirements, without involving too many complex algorithms, reducing operation and computing load to improve subcontracting efficiency.

As an optional implementation, CPU single-core score=current CPU turbo value/CPU turbo benchmark value*CPU core benchmark score;

CPU multi-core score = (current CPU turbo frequency value/CPU turbo frequency benchmark value)*(current CPU core number/CPU core number benchmark value)*CPU core benchmark score;

CPU comprehensive score = CPU single-core score * 0.99 + CPU multi-core score * 0.01;

Memory score = (current memory space value/memory space benchmark value)*memory benchmark score;

GPU score = (current GPU stream processor number/GPU stream processor number benchmark value)*GPU video memory benchmark score;

Comprehensive score of CPU and memory = CPU multi-core score + memory score.

Specifically, the above scoring mechanism is constructed based on oblique image model processing. If there are different computing services, the hardware scoring items, scoring weights, and scoring formulas of the above scoring mechanism should be modified and adjusted according to the actual needs of computing services to adapt to different computing services. business needs.

In this embodiment, several limit indicators are set to calibrate the equipment performance requirements corresponding to different tasks;

Set the first limit index as CPU usage < 90%, and remaining GPU memory space > 1GB;

Set the second limit index as CPU usage < 90%, and remaining memory space > 1GB;

Set the third limit index as CPU usage < 90%, and remaining memory space > 4GB;

Set the fourth limit index as CPU usage < 90%, and remaining memory space > (task package contains the number of movies * 0.0015) GB;

Set the fifth limit index as CPU usage < 90%, and remaining memory space > 12GB.

Among them, the first restriction index is used for feature extraction task, the second restriction index is used for feature matching task, the third restriction index is used for block adjustment task, the fourth restriction index is used for fusion network leveling task, and the fifth restriction index is used for for modeling tasks.

Specifically, the hardware requirements and constraints of common image processing tasks are as follows:

Therefore, the above constraints can be used as pre-screening conditions to further prevent characteristic modeling tasks from being assigned to computing nodes that are not suitable for the current processing, effectively avoiding task congestion.

Here, assuming that the current task package needs to perform feature extraction processing, according to the above table, it can be seen that it pays attention to GPU performance, and requires the computing node to have more than 1GB of GPU memory space, and the CPU usage of the computing node should be less than 90%, so it can be According to the restrictive conditions, the computing nodes whose hardware scores meet the standard but are currently occupied and unable to efficiently perform other computing tasks are screened out;

Furthermore, after filtering out high-occupancy computing nodes based on the above constraints, a computing node queue X [A(60), B(55), C(54), ..., N(41 )], and the queue X is a descending queue, that is, the feature extraction task is assigned to the computing node A with the highest current GPU score for processing.

It is understandable that, in addition to the scheduling node actively querying the running status of the computing node, after the task subcontracting is completed, the real-time occupancy rate and real-time scoring of computing node A will be actively uploaded to the scheduling node for feedback and update to improve the accuracy of real-time scoring Responsibility, avoiding the allocation of additional tasks to the current computing node, causing computing congestion.

102. Divide the task to be processed into several task packages.

In this embodiment, in order to give full play to the computing power of the distributed computing nodes, the task to be processed is divided into multiple task packages first, so as to ensure that each task package can be efficiently calculated in a short period of time, so that the large amount of data A single task can be quickly completed under the division of labor of many computing nodes.

As an optional implementation, for tasks to be processed with task categories marked, each task package is gradually sent to the optimal computing node corresponding to the task category after block processing, or each task package is evenly distributed to Calculations can be efficiently completed on multiple computing nodes at the forefront of the scoring queue.

103. Set the first limit index, and select the computing node with the highest current GPU score accordingly, and perform feature extraction calculation for each task package.

104. Set the second limit index, and select the computing node with the highest current CPU multi-core score accordingly, and perform feature matching calculation on the task package whose feature extraction calculation is successful.

105. Set the third limit index, and select the computing node with the highest current CPU comprehensive score based on this, and perform block adjustment calculation for the task package whose feature matching calculation is successful.

106. Set the fourth limit index, and select the computing node with the highest current CPU comprehensive score based on this, and perform fusion adjustment calculation on the task package whose block adjustment calculation is successful.

107. Export the corresponding xml file and bin file for the task package successfully calculated by fusion adjustment.

108. Set the fifth limit index, and select the computing node with the highest comprehensive score of the current CPU according to this, and perform modeling calculation on the xml file and the bin file.

In this embodiment, the task package that fails in feature extraction, feature matching, block adjustment, fusion adjustment or modeling calculation is marked as a redistribution task, and the number of calculation failures is updated;

Among them, the calculation failure refers to an error in the calculation process or a calculation result verification that does not conform to the rules;

Re-select the current optimal computing node for redistribution tasks with less than 3 failure times, and perform feature extraction, feature matching, block adjustment, fusion adjustment or modeling calculation.

Here, considering that there is a time difference in the update of the real-time indicators of the computing nodes, data errors may occur during the task package splitting process, calculation errors may occur during the task calculation process, hardware failures in the computing nodes and other factors, the calculation of the task package may be wrong, or the calculation is successful , but the verification of the calculation results does not conform to the rules. At this time, the task package needs to be redistributed and recalculated to ensure that the output data is accurate and valid.

As an optional implementation, the redistribution task whose failure count reaches 3 is marked as a failed task, the dispatch and calculation process of the failed task is terminated, and the calculation failure information is returned.

Here, the failed tasks that repeatedly fail to calculate are discarded, and the failure information is returned for analysis by the task uploader or operation and maintenance personnel to find out the cause of the failure.

To sum up, by collecting hardware parameters and monitoring real-time indicators, the available resources of each computing node are accurately scored, and the task package is distributed to the current computing node with the highest degree of fitness for calculation, so as to give full play to the node performance and improve computing efficiency. And the scheduling node can be accessed by multiple platforms, adapting to various system environments.

Embodiment two

Please refer to Figure 2, a distributed task scheduling system for oblique image model production, including:

Data nodes, scheduling nodes, and computing nodes;

Each data node is connected to at least one scheduling node data;

Each computing node is connected to at least one scheduling node data;

The data node is used to preprocess the task to be processed into several task packages;

The scheduling node performs access control through the visual terminal to collect hardware parameters and update real-time indicators of the computing nodes connected to it.

Among them, the scheduling node is also used to select the optimal computing node for the feature extraction, feature matching, block adjustment, fusion adjustment or modeling process of each task package according to the hardware parameters and real-time indicators of each computing node, and execute Task subcontracting.

Here, the visualization terminal can be a terminal device with data transmission function and control operation function such as a personal computer, a smart phone, and a photography terminal, so that the photography terminal can upload the image data it shoots in real time, and the user can also use it on a personal terminal such as a personal computer. Log in to the scheduling node, where you upload the required calculation tasks to the scheduling node.

It can be seen that this process can be conveniently realized through the data bus or TCP/IP protocol, does not involve the construction and interaction of the platform architecture, and is widely compatible with various software and hardware platforms and operating systems.

Claims (10)

1. A distributed task scheduling method for oblique image model production, characterized in that, comprising: Regularly collect and update the real-time hardware score of each computing node; The tasks to be processed are divided into several task packages; Select the computing node with the highest current GPU score to perform feature extraction calculations for each task package; Select the computing node with the highest current CPU multi-core score, and perform feature matching calculation on the task package with successful feature extraction and calculation; Select the computing node with the highest comprehensive score of the current CPU, and perform block adjustment calculation for the task package whose feature matching calculation is successful; Select the computing node with the highest comprehensive score of the current CPU, and perform fusion adjustment calculation on the task package whose block adjustment calculation is successful; Export the corresponding xml file and bin file for the task package successfully calculated by fusion adjustment; Select the computing node with the highest comprehensive score of the current CPU, and perform modeling calculation on the xml file and the bin file. 2. The distributed task scheduling method of oblique image model production according to claim 1, characterized in that, comprising: Based on the scoring items of the hardware devices included in each computing node and the corresponding device parameters, evaluate and obtain the benchmark hardware score of each computing node; The real-time hardware score is obtained by weighting and adjusting the benchmark hardware score based on the real-time occupancy rate of the hardware devices included in each computing node; Wherein, the hardware device includes at least CPU, memory and GPU; Moreover, the scoring items for CPU include at least the number of CPU cores and CPU turbo frequency, the scoring items for memory include memory space, and the scoring items for GPU include at least the number of GPU stream processors and GPU memory space. 3. The distributed task scheduling method of oblique image model production according to claim 2, is characterized in that, comprises: setting a benchmark value for each scoring item, and setting a benchmark score corresponding to the benchmark value; Among them, the benchmark value of the number of CPU cores is 4, and the benchmark score of CPU cores is 50; The CPU turbo frequency benchmark value is 3GHz, and the CPU turbo frequency benchmark score is 50; The memory space benchmark value is 16GB, and the memory benchmark score is 50; The benchmark value for the number of GPU stream processors is 1280, and the benchmark score for GPU stream processors is 50; The GPU memory space benchmark value is 4GB, and the GPU memory benchmark score is 50. 4. The distributed task scheduling method of oblique image model production according to claim 3, is characterized in that, comprises: Based on the benchmark value and benchmark score, evaluate the CPU score corresponding to the CPU performance of each computing node, the memory score corresponding to the memory performance, the GPU score corresponding to the GPU performance, and the CPU and memory corresponding to the comprehensive performance of the CPU and memory Overall rating; Wherein, the CPU score includes at least a CPU single-core score, a CPU multi-core score, and a CPU comprehensive score. 5. The distributed task scheduling method of oblique image model production according to claim 4, is characterized in that, comprises: CPU single-core score = current CPU turbo frequency value / CPU turbo frequency benchmark value * CPU core benchmark score; CPU multi-core score = (current CPU turbo frequency value/CPU turbo frequency benchmark value)*(current CPU core number/CPU core number benchmark value)*CPU core benchmark score; CPU comprehensive score = CPU single-core score * 0.99 + CPU multi-core score * 0.01; Memory score = (current memory space value/memory space benchmark value)*memory benchmark score; GPU score = (current GPU stream processor number/GPU stream processor number benchmark value)*GPU video memory benchmark score; Comprehensive score of CPU and memory = CPU multi-core score + memory score. 6. The distributed task scheduling method of oblique image model production according to claim 5, characterized in that, comprising: Set several limit indicators to calibrate the equipment performance requirements corresponding to different tasks; Set the first limit index as CPU usage < 90%, and remaining GPU memory space > 1GB; Set the second limit index as CPU usage < 90%, and remaining memory space > 1GB; Set the third limit index as CPU usage < 90%, and remaining memory space > 4GB; Set the fourth limit index as CPU usage < 90%, and remaining memory space > (task package contains the number of movies * 0.0015) GB; Set the fifth limit index as CPU usage < 90%, and remaining memory space > 12GB. 7. The distributed task scheduling method of oblique image model production according to claim 1, characterized in that, comprising: For feature extraction, feature matching, block adjustment, fusion adjustment or modeling calculation failures, the task package is marked as a redistribution task, and the number of calculation failures is updated; Among them, the calculation failure refers to an error in the calculation process or a calculation result verification that does not conform to the rules; Re-select the current optimal computing node for redistribution tasks with less than 3 failure times, and perform feature extraction, feature matching, block adjustment, fusion adjustment or modeling calculation. 8. The distributed task scheduling method of oblique image model production according to claim 7, characterized in that, comprising: Mark the redistribution task whose failure times reach 3 as a failed task, terminate the dispatch and calculation process of the failed task, and return the calculation failure information. 9. A system employing a distributed task scheduling method produced by any oblique image model of claims 1 to 8, characterized in that the system comprises: Data nodes, scheduling nodes and computing nodes; Each data node is connected to at least one scheduling node data; Each computing node is connected to at least one scheduling node data; The data node is used to preprocess the task to be processed into several task packages; The dispatching node performs access control through the visualization terminal to collect hardware parameters and update real-time indicators of the computing nodes connected to it. 10. The system of claim 9, comprising: The scheduling node is also used to select the optimal computing node for the feature extraction, feature matching, block adjustment, fusion adjustment or modeling process of each task package according to the hardware parameters and real-time indicators of each computing node, and execute Task subcontracting.