CN114445688B - A distributed multi-camera spherical unmanned system target detection method - Google Patents

Abstract

The present invention discloses a distributed multi-camera spherical unmanned system target detection method, wherein the unmanned system is composed of a plurality of quad-rotor drones, and the plurality of quad-rotor drones are assembled into a spherical unmanned system; a monocular camera is installed on the drone, and when assembled into a spherical state, the unmanned system is a rigid body carrying a plurality of cameras; the method comprises the following steps: designing a distributed camera network topology structure for the multi-camera unmanned system; performing image fusion processing at the feature level using a multi-camera data fusion algorithm; establishing a target detection algorithm based on deep learning, compressing a neural network model, and performing target detection processing on the fused image using the compressed neural network model to complete the target detection task. The method improves the target detection speed and the accuracy of target detection under conditions such as occlusion, and ensures that the spherical unmanned system successfully completes the task.

Description

A distributed multi-camera spherical unmanned system target detection method

Technical Field

The present invention relates to the technical field of multi-camera target detection, and in particular to a moving target detection method of a distributed multi-camera network structure, and more specifically to a target detection method of a spherical multi-camera unmanned system in a rolling mode.

Background technique

In recent years, with the development of science and technology, mobile robots have gradually entered modern social life and started to play an increasingly important role in industrial production and other fields. Due to the rapid expansion of the robot market and the development of artificial intelligence, various target detection, recognition or tracking technologies based on mobile robots have also attracted more and more attention from researchers. Compared with wheeled robots and legged robots, spherical robots are quite different in mechanical structure and motion characteristics, and there are fewer restrictions on related research. Therefore, the research on the motion target detection method of spherical unmanned systems has certain scientific innovation and practical significance.

At present, the detection, recognition or tracking technology for various targets (vehicles, pedestrians, faces, gestures, etc.) in images or videos has become the mainstream direction of computer vision intelligence. Video image analysis and processing technology has practical application value and broad development prospects, including but not limited to smart cities, social security management, home security systems, live broadcast and analysis of sports events, medical monitoring, etc. The system using a single camera is subject to its own hardware constraints (video resolution, transmission problems, etc.), especially the limitation of its field of view, resulting in the inability of traditional video image acquisition technology to meet the demand for data quantity and quality; in contrast, the collaborative work of multiple cameras can make up for the limitations of a single camera at work, and has become one of the current research hotspots.

Summary of the invention

The present invention studies a moving target detection method for a three-dimensional spherical modular self-assembling unmanned system. The unmanned system can switch between three states: flight mode, rolling mode, and navigation mode. It is composed of multiple four-rotor drones, and each drone can independently or in combination perform corresponding tasks such as search, exploration, and communication. Each individual sub-module drone of the spherical unmanned system is equipped with a monocular camera. When assembled into a spherical rolling state, the six sub-module drones can be regarded as a unified rigid body carrying six cameras. Through collaborative formation, real-time communication, and distributed control among the module units, the three-dimensional spherical modular self-assembling unmanned system can be realized to roll in all directions on the ground, meeting the task requirements of the spherical unmanned system such as front-line reconnaissance, target detection and tracking. In order to solve the problem that when the spherical multi-camera unmanned system performs moving target detection, the single camera's viewing angle is limited and cannot provide accurate recognition of the target, the present invention proposes a target detection method for a distributed spherical multi-camera unmanned system.

The technical solution adopted by the present invention is to design a distributed camera network topology structure for a multi-camera system, as well as a data fusion algorithm between multiple cameras, and use a compressed neural network model for target detection. The technical solution adopted is as follows:

A distributed multi-camera spherical unmanned system target detection method, wherein the unmanned system is composed of a plurality of quad-rotor drones, and the plurality of quad-rotor drones are assembled into a spherical unmanned system; the drones are equipped with monocular cameras, and when assembled into a spherical state, the unmanned system is a rigid body carrying a plurality of cameras; the method comprises the following steps:

The first step is to design a distributed camera network topology for the multi-camera unmanned system, so that each camera can perform independent feature extraction processing on the captured scene images;

The second step is to use the multi-camera data fusion algorithm to perform feature-level image fusion processing on the image data provided by multiple cameras so that they can be fused into a complete image.

The third step is to establish a target detection algorithm based on deep learning, compress the neural network model, and use the compressed neural network model to perform target detection on the fused image obtained in the second step to complete the target detection task.

Furthermore, in the first step, the distributed camera network topology structure is specifically as follows: a processor is configured for each camera node as a processing unit of the camera node of the spherical unmanned system, and the topic-subscription mechanism in ROS is used to realize data communication of any number of camera nodes. Each camera node runs independently and performs independent feature extraction processing on the captured scene image.

Furthermore, in the second step, the multi-camera data fusion algorithm is specifically as follows:

(1) Obtaining target image feature points from multi-view images provided by multiple cameras;

(2) The feature point descriptors are transmitted to the central processing unit to perform feature matching on the input multi-view images so that the matched multi-view images are fused into a complete image.

Furthermore, an improved weighted smoothing algorithm is introduced when fusion of multi-view images, namely:

Let f(x,y) represent the image after overlapping area fusion, which is obtained by weighted average of two images to be fused, _fL and _fR , that is:

f(x,y)＝α× _fL (x,y)+(1-α) _fR (x,y)

Among them, α is an adjustable factor, 0＜α＜1, that is, in the image intersection area, along the direction from the view 1 image to the view 2 image, α gradually changes from 1 to 0, achieving smooth fusion of the intersection area; in order to establish a greater correlation between the two images, the following formula is used for fusion processing:

make Then/> Wherein d ₁ and d ₂ represent the distances from a point in the intersection region to the left boundary and the right boundary of the intersection region of two images of different viewing angles, respectively.

Furthermore, the third step of compressing the neural network model is specifically as follows:

(1) Create a target dataset for training and use it for basic training;

(2) Setting the learning rate to perform sparse training on the network will cause many scaling factors in the network to approach zero;

(3) Perform network pruning, sort the scaling factors, and prune the channels corresponding to the scaling factors with smaller values;

(4) Perform knowledge distillation and use the teacher network to guide the training of the pruned student network to obtain a more compact neural network model after compression.

Furthermore, after knowledge distillation, the sparsification training, network pruning, and knowledge distillation steps are repeated to compress the model multiple times.

Furthermore, the network pruning is specifically as follows:

(1) Introduce a scaling factor, multiply it by the output of the channel, and calculate the newly introduced gradient terms associated with all filters;

(2) Jointly train the network weights and scaling factors, and perform sparse regularization on the scaling factors;

(3) Fine-tune the channel with a small factor so that all weights at the input or output of the same channel tend to zero at the same time during training, so as to cut off all input and output connections related to the channel and achieve channel refinement;

(4) Prune the channel with a scaling factor approximately equal to 0 and remove all its connections and corresponding weights to obtain the pruned network model.

Furthermore, the knowledge distillation is specifically as follows:

(1) Training large models;

(2) The training target is updated from the traditional true value label to the soft target, and the training knowledge of the large model is transferred to the small model to obtain a more compact neural network model after compression.

The beneficial effects of this application compared to the prior art are as follows:

(1) Compared with a single camera with limited viewing angle, installing multiple cameras at appropriate locations on a spherical unmanned system can obtain information from different viewing angles of the same target, which can make up for the positive information lost when the relative angle of a single camera changes, and also provide an effective solution to the occlusion problem.

(2) Considering that the computing power of the onboard processor of the UAV is limited and the use of multiple cameras introduces a huge amount of data, compressing the neural network model can improve the real-time performance of target detection to enable the unmanned system to successfully complete its mission.

(3) Designing a distributed camera network topology for multiple cameras in a spherical unmanned system can give the system distributed computing and communication characteristics, as well as high mobility and stability. Processor resources are configured at each node to ensure local computing power, so that even if the function of a node is unbalanced, the multi-camera system can continue to maintain its working state.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is an amphibious spherical modular self-assembling unmanned system;

Figure 2 is a schematic diagram of the decomposition of the modules of the amphibious spherical modular self-assembling unmanned system;

Figure 3 is a schematic diagram of a terrestrial spherical modular self-assembling unmanned system camera;

FIG4 is a technical roadmap of a distributed multi-camera spherical unmanned system target detection method;

FIG5 is a diagram of a distributed camera network topology of a multi-camera system;

Figure 6 shows the neural network model compression process.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

The target detection method of the distributed multi-camera spherical unmanned system proposed in the present invention is shown in Figures 1-3. The unmanned system is composed of multiple quad-rotor drones, and the multiple quad-rotor drones are assembled into a spherical unmanned system; a monocular camera is installed on the drone, and when assembled into a spherical state, the unmanned system is a rigid body carrying multiple cameras; as shown in Figure 4, the entire process of the distributed spherical multi-camera unmanned system to achieve target detection mainly includes: the design of the distributed camera network topology structure of the spherical unmanned multi-camera system and the image feature extraction and processing, the design of the multi-camera data fusion algorithm, the establishment of a target detection algorithm based on deep learning and the compression processing of the neural network model. The specific technical scheme is as follows:

Step 1: Design of distributed camera network topology for spherical unmanned multi-camera system. As shown in Figure 5, the use of a distributed camera network topology can make the spherical multi-camera unmanned system have distributed computing and communication characteristics as well as high mobility and stability. In terms of system structure, in order to avoid the hidden danger that the entire system will crash once an error occurs in the main server in the centralized structure, a distributed structure is adopted. Each node is equipped with processor resources to ensure local computing power. When the function of any node is unbalanced, the array system can continue to maintain its working state. In order to meet the computing performance of the distributed system, considering the difficulty of configuration and use and evaluating the cost and size range of the processing unit, each camera node is configured with a Raspberry Pi 4b as the processing unit of the camera node of the spherical unmanned system to complete the preparation work before multi-camera data fusion, such as image information acquisition and target image feature extraction.

And referring to the multi-machine communication scheme suitable for distributed structure, the topic-subscription mechanism in ROS is used to realize data communication of any number of camera nodes. Each camera node runs independently and processes the captured scene images for moving target detection. The advantage of the distributed camera network topology is that each camera node is equipped with certain computing resources to ensure local processing capabilities, there is no dependency between the operation of each camera node, and each node can exchange data with any other node.

Among them, the ORB feature point extraction algorithm is used to extract the feature points of the target image, and its steps are as follows: first, the first step is to perform rough extraction, select a point P from the image, draw a circle with a radius of 3 pixels with P as the center, and if there are n consecutive pixels on the circumference whose grayscale values are greater or smaller than the grayscale value of point P, then P is considered to be a feature point; the second step is to train a decision tree through machine learning methods, input 16 pixels on the circumference of the feature point into the decision tree, and screen out the optimal FAST feature point at one time; the third step is to remove locally dense feature points through the non-maximum suppression algorithm; the fourth step is to establish a pyramid to achieve multi-scale invariance of feature points; the fifth step is to use the moment method to determine the direction of the FAST feature point to achieve rotation invariance of the feature point.

Step 2: Design a multi-camera data fusion algorithm. When multiple cameras of a spherical multi-camera unmanned system work together to detect a target, they can obtain information from different perspectives of the same target. In order to better complete the target detection task, the image features of the same target or scene collected by multiple cameras are processed by a specific feature matching algorithm (such as a brute force matching algorithm) for image fusion, which can make up for the positive information of the target lost when the relative angle of a single camera changes. For example, although multispectral images are rich in spectral information, they have low resolution, while panchromatic images have high resolution but low color recognition. Image fusion can make full use of the complementary information of the two. The fused image can better provide scene information, which is convenient for human eye observation and further machine processing. Therefore, designing a multi-camera data fusion algorithm for a spherical unmanned multi-camera system can improve the accuracy and credibility of target detection and increase the fault tolerance of the system.

First, the target image feature points of the multi-view images provided by multiple cameras are obtained; then the feature point descriptors are transmitted to the central processing unit, and feature matching is performed on the input multi-view images, so that the matched multiple images are fused into a complete image. Considering that in practical problems, images collected by cameras at different angles will produce color aberration after fusion, the present invention introduces an improved weighted smoothing algorithm when fusing multi-view images to solve the problem of color aberration when fusing images at different angles.

The weighted smoothing algorithm is as follows: f(x, y) is used to represent the image after the overlapping area is fused, which is obtained by weighted averaging the two images to be fused, _fL and _fR , that is:

f(x,y)＝α× _fL (x,y)+(1-α) _fR (x,y)

Among them, α is an adjustable factor. Generally, 0＜α＜1, that is, in the image intersection area, along the direction from view 1 image to view 2 image, α gradually changes from 1 to 0, thereby achieving smooth fusion of the intersection area. In order to establish a greater correlation between the two images, the following formula is used for fusion processing:

make Then/> Wherein d ₁ and d ₂ represent the distances from a point in the intersection region to the left boundary and the right boundary of the intersection region of two images of different viewing angles, respectively.

In particular, the basis of multi-image fusion is pairwise fusion. In actual scenes, it is not necessary to use six images to detect targets. First, the probability of each camera detecting the target is ranked; then, the camera with the highest detection probability is taken as the center (it is most likely to be facing the target), and 3-4 images above, below, left and right are fused for detection to increase the credibility and accuracy of target detection.

Step 3: Establish a target detection algorithm based on deep learning and compress the neural network model. Due to the capacity limitations of the spherical unmanned multi-camera system platform, the computing power of its onboard processor is limited, so it is necessary to select a suitable target detection algorithm to meet the requirements of real-time detection and target detection accuracy. Compared with traditional sliding window-based target detection, the deep learning-based target detection algorithm YOLO v3 adopts a more direct approach. It does not select candidate areas, but directly predicts the location and category of the target, which greatly improves the target detection accuracy and rate.

The YOLO model grids the input image by covering it with an S×S grid map. If the center of an object falls within a grid unit, then this grid unit is responsible for detecting the object and predicting the category information and confidence of the bounding box. By setting the confidence to filter the bounding box, the bounding boxes with low confidence will be discarded, and the bounding boxes with high confidence that are retained will be subjected to non-maximum suppression to delete highly redundant bounding boxes. The improved YOLO v3 algorithm draws on the idea of skip connections in the residual network and adopts a new network structure Darknet-53. It uses three feature maps of different scales for multi-scale targets for target detection, and uses a logistic regression classifier instead of a normalized exponential function classifier to predict categories. It can predict multiple categories at the same time and detect multi-label target objects. While maintaining the speed advantage, the YOLO v3 algorithm improves the prediction accuracy, especially the recognition ability of small objects.

However, when deep learning algorithms are transplanted on airborne platforms with limited computing resources and complex conditions, the issue of computing resource usage must be considered. Therefore, it is necessary to optimize the target detection algorithm, lightweight the neural network model, compress the model to reduce the number of parameters, and increase the detection speed without much decrease in accuracy, so as to better adapt to the special needs of the airborne platform hardware and complete the target detection task.

The neural network model compression process used in the present invention is shown in Figure 6. There are a large number of redundant parameters in the deep learning network model. Most of the activation values of neurons in the convolutional layer or the fully connected layer tend to 0, and these neurons can still show the same model expression ability after being removed. This is the so-called over-parameterization. Therefore, in the process of using data sets to train neural networks, the present invention uses network pruning and knowledge distillation methods to remove network parameters, remove redundant network nodes and weight connections, and even remove redundant convolution kernels, so as to make the network structure more streamlined.

Network pruning reduces the number of parameters by removing redundant and unimportant connections. Network pruning methods mainly include two categories: sparse constraints during training and pruning after training. Sparse constraints refer to adding sparse constraints to the optimization function to make the network structure sparse without the need for pre-training models. The sparse constraints of the network loss function are mainly achieved by introducing L1 regularization and L2 regularization constraints; post-training pruning is to remove relatively unimportant parts of the network to make the network sparse and streamlined, which is currently the simplest and most effective method. Post-training network pruning starts from the existing training model and gradually eliminates redundant information in the network to avoid losses caused by network retraining.

According to the different pruning scales, there are mainly intra-core pruning, channel pruning, inter-layer pruning, and k×k core pruning. The present invention uses a channel pruning method to prune the network. In order to achieve channel refinement, all input and output connections related to the channel must be cut off, which makes the method of directly pruning the weights on the pre-trained model invalid, because pruning needs to prune the weights that tend to zero, but it is impossible for all weights at the input or output end of the channel to tend to zero. Therefore, the present invention solves this problem by implementing sparse regularization in the objective function of training. Specifically, the grouped Lasso method is used to make the same channel of all filters tend to 0 at the same time during training. When calculating the newly introduced gradient terms related to all filters, the present invention introduces a scaling factor for each channel and multiplies it by the output of the channel; then the network weights and these scaling factors are jointly trained to sparsely regularize the latter; finally, a small factor is used to fine-tune these channels. The above objective function is defined as follows:

Among them, L is the objective function, f indicates that there is a mapping relationship between x and W, l indicates that there is a mapping relationship between f(x, W) and y, which has no actual meaning, Γ indicates the coefficient set of the feature, (x, y) represents the training input and target, W represents the trainable weight, the first summation term corresponds to the loss of normal training of the convolutional neural network, g(γ) is the sparsity penalty on the scaling factor, and λ is the balance factor of the two terms. Here, g(γ) = |γ| is selected, that is, L1 regularization, and sub-gradient descent is used as the optimization method for the non-smooth L1 penalty term.

After sparse normalization is performed on each channel, many scaling factors in the model become 0, and then the channels with scaling factors close to 0 can be pruned, and all their connections and corresponding weights can be removed. For all layers, the present invention sets a global threshold to prune channels according to the values of all scaling factors at a certain ratio, so that the network parameters and calculation operations will be much less, and the memory usage at runtime will be saved.

Knowledge distillation refers to the process of training a small model, using the knowledge that the large model has learned as a guide, so that the small model has a similar detection effect to the large model with a relatively low number of parameters. There are two ideas for knowledge distillation. One is to train the large model and the small model at the same time; the other is to train the large model first, and then refine the small model on its basis. In fact, a more universal complex model can be trained. Through the knowledge transfer of the complex model, the training cost of small-scale special tasks can be reduced on a large scale, and a high-performance model can be obtained. Therefore, the method of knowledge distillation adopted by the present invention is to transfer the training knowledge of the large model to a special small model. When training the small model, the training target is updated from the traditional true value label to the so-called soft target. The soft target can provide greater information entropy during the training process and better transfer the trained model knowledge to the new model. And because the gradient variance between different training samples is small under this goal, the training data required for small model learning will be greatly reduced, a higher learning rate can be used, and the iteration speed of the model will be significantly accelerated.

The soft target mentioned above is actually the output probability of the normalized exponential function layer of the trained complex model, and the "distillation" method introduces a "temperature" parameter T in the normalized exponential function layer:

Among them, z _i is the probability of each classification, q _i is the output of the normalized exponential function given by the student model (small model), and the temperature parameter T is usually set to 1. The simplest form of distillation is to use the soft target obtained by the complex model as the target when training the small model, use the higher T in the complex model, and change T to 1 after training.

The entire neural network model compression process is: first, create the target data set for training, which is the data set of the target to be detected, that is, the set of labeled pictures, and use the data set for basic training; then set the learning rate, perform sparse training on the network, and many scaling factors in the network will approach zero; then perform network pruning, sort the scaling factors, and cut off the channels corresponding to the scaling factors with smaller values; finally, perform knowledge distillation, and use the teacher network to guide the pruned student network for training, so that a more compact neural network model can be obtained after compression. It should be noted that after knowledge distillation, sparse training, network pruning, knowledge distillation and other steps can be repeated to compress the model multiple times.

The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims (3)

1. A distributed multi-camera spherical unmanned system target detection method, characterized in that the unmanned system is composed of a plurality of quad-rotor drones, and the plurality of quad-rotor drones are assembled into a spherical unmanned system; a monocular camera is installed on the drones, and when assembled into a spherical state, the unmanned system is a rigid body carrying a plurality of cameras; comprising the following steps: The first step is to design a distributed camera network topology for the multi-camera unmanned system, so that each camera can perform independent feature extraction processing on the captured scene images; The second step is to use the multi-camera data fusion algorithm to perform feature-level image fusion processing on the image data provided by multiple cameras so that they can be fused into a complete image. The third step is to establish a target detection algorithm based on deep learning, compress the neural network model, and use the compressed neural network model to perform target detection on the fused image obtained in the second step to complete the target detection task; In the first step, the distributed camera network topology structure is specifically as follows: a processor is configured for each camera node as a processing unit of the camera node of the spherical unmanned system, and a topic-subscription mechanism in ROS is used to realize data communication of any number of camera nodes, and each camera node runs independently to perform independent feature extraction processing on the captured scene image; In the second step, the multi-camera data fusion algorithm is specifically as follows: (1) Obtaining target image feature points from multi-view images provided by multiple cameras; (2) The feature point descriptors are transmitted to the central processing unit, and feature matching is performed on the input multi-view images so that the matched multi-view images are fused into a complete image; An improved weighted smoothing algorithm is introduced in multi-view image fusion, namely: Let f(x,y) represent the image after overlapping area fusion, which is obtained by weighted average of two images to be fused, _fL and _fR , that is: f(x,y)＝α× _fL (x,y)+(1-α) _fR (x,y) Among them, α is an adjustable factor, 0<α<1, that is, in the image intersection area, along the direction from the view 1 image to the view 2 image, α gradually changes from 1 to 0, achieving smooth fusion of the intersection area; in order to establish a greater correlation between the two images, the following formula is used for fusion processing: make Then/> Where d ₁ and d ₂ represent the distances from a point in the intersection region to the left and right boundaries of the intersection region of two images of different view angles, respectively; The third step of compressing the neural network model is specifically as follows: (1) Create a target dataset for training and use it for basic training; (2) Setting the learning rate to perform sparse training on the network will cause many scaling factors in the network to approach zero; (3) Perform network pruning, sort the scaling factors, and prune the channels corresponding to the scaling factors with smaller values; (4) Perform knowledge distillation and use the teacher network to guide the pruned student network for training to obtain a more compact neural network model after compression; The knowledge distillation is specifically as follows: (1) Training large models; (2) The training target is updated from the traditional true value label to the soft target, and the training knowledge of the large model is transferred to the small model to obtain a more compact neural network model after compression. 2. The distributed multi-camera spherical unmanned system target detection method according to claim 1 is characterized in that after knowledge distillation, sparse training, network pruning, and knowledge distillation steps are re-performed to compress the model multiple times. 3. The distributed multi-camera spherical unmanned system target detection method according to claim 1, characterized in that the network pruning is specifically: (1) Introduce a scaling factor, multiply it by the output of the channel, and calculate the newly introduced gradient terms associated with all filters; (2) Jointly train the network weights and scaling factors, and perform sparse regularization on the scaling factors; (3) Fine-tune the channel with a small factor so that all weights at the input or output of the same channel tend to zero at the same time during training, so as to cut off all input and output connections related to the channel and achieve channel refinement; (4) Prune the channel with a scaling factor approximately equal to 0 and remove all its connections and corresponding weights to obtain the pruned network model.