CN110719444A - Multi-sensor fusion omnidirectional monitoring and intelligent camera method and system - Google Patents

Abstract

The invention relates to a multi-sensor fusion omnidirectional monitoring and intelligent camera method and system. The system includes a 360° fixed camera module, a 360° scanning camera module, an intelligent processing and control module and a comprehensive information display terminal. It integrates the perception information of four fixed-focus cameras, a zoom camera and a radar sensor to realize the all-round monitoring of "seeing the whole", "seeing clearly" and "understanding" and the monitoring target of intelligent camera. It can be widely used in important places such as prisons, gun depots, oil depots, etc., to achieve all-round monitoring and intelligent camera in sensitive areas, while supporting the capture and early warning of objects of interest, and improving the intelligence level of security monitoring.

Description

Multi-sensor fusion omnidirectional monitoring and intelligent camera method and system

technical field

The invention relates to a multi-sensor fusion omnidirectional monitoring and intelligent camera method and system.

Background technique

With the advancement of science and technology, video surveillance has been widely used in various fields, such as bank safety supervision, traffic violation records, the sky eye system for public security investigation and investigation, monitoring in the examination room, and power system detection. It can be said that video surveillance is integrated into all aspects of our daily life, thus contributing to the huge market space currently owned by video surveillance.

The monitoring equipment currently on the market can be roughly divided into two categories. A type of monitoring equipment is a monitoring unit composed of fixed-focus cameras. This type of equipment has a simple structure and low cost, but is only suitable for monitoring a small area, and because the focal length is fixed, the screen cannot be zoomed in and out. Therefore, this type of monitoring equipment Detailed information cannot be obtained. For example, the device can monitor that someone enters the monitoring area, but it cannot achieve a clear zoom-in and zoom-in process to obtain detailed information of the target. The second type of monitoring equipment is a monitoring unit composed of a variable-focus camera. This type of equipment has a high cost and a wide range of variable focal lengths. The market application can reach a maximum of 50 times optical zoom, but the zoom of this type of monitoring equipment requires manual operation and control. For example, in the above-mentioned situation, it is necessary to manually adjust the focal length to lock the target and zoom in, which means that intelligent automatic monitoring cannot be realized, which consumes human resources.

In the field of security monitoring, traditional fixed-focus cameras have problems such as being unable to obtain detailed information of distant targets and limited monitoring range. Variable-focus cameras need to manually control the orientation and focal length, or automatically cruise according to certain rules, and cannot actively record the target. , and can not achieve comprehensive monitoring.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention proposes a multi-sensor fusion omnidirectional monitoring and intelligent camera method and system.

The multi-sensor fusion omnidirectional monitoring and intelligent camera system proposed by the present invention includes a 360° fixed camera module, a 360° scanning camera module, an intelligent processing and control module, and a comprehensive information display terminal;

The 360° fixed camera module includes 4 fixed-focus cameras to realize all-round monitoring and display, so that the monitoring scene can be seen more comprehensively;

The 360° scanning camera module includes a zoom camera, a radar, and a pan/tilt. The pan/tilt realizes 360° scanning. The radar senses the target distance, automatically controls the zoom camera to adjust the focal length, and realizes optical amplification of targets at different distances, so that the monitoring target can see the target. more clearly;

The intelligent processing and control module deploys a target detection and identification algorithm to intelligently identify the target of interest in the monitoring scene, so that the monitoring scene is easier to understand;

The intelligent processing and control module realizes the gimbal control through a rectangular wave generator with a PWM pulse frequency of 50Hz and an adjustable duty cycle. Different duty cycles correspond to the zero-degree rotation of the gimbal and a fixed angle. The duty cycle of the rectangular wave is 5%- 95%, the corresponding gimbal rotation angle is 0°-360°, which controls the gimbal rotation;

The intelligent processing and control module connects the laser ranging signal through the USB to TTL interface, and converts it into the distance information of the target; the zoom camera uses the preset position to adjust the focus, and there are 6 preset positions corresponding to 6 distance segments respectively. The distance information is preset, and then the zoom camera is controlled to adjust the focus through the USB serial port to obtain a clear image;

After the intelligent processing and control module receives the laser signal and controls the zoom camera to focus, it intelligently recognizes the images captured by the zoom camera. Specifically, a multi-model fusion target detection side algorithm is deployed on the ARM, and the detection result of the algorithm is pedestrians. to capture the image;

The comprehensive information display terminal displays all-round monitoring information and performs intelligent photography, so as to meet the video monitoring requirements of high security levels.

The invention integrates the perception information of four fixed-focus cameras, one variable-focus camera and one radar sensor, and realizes the all-round monitoring of "seeing the whole", "seeing clearly" and "understanding" and the monitoring target of intelligent camera.

Description of drawings

Figure 1 Schematic diagram of the 360° fixed camera module (top view),

Figure 2 Schematic diagram of 360° scanning camera module,

Figure 3 VGG network structure,

Figure 4 ResNet50 network structure,

Figure 5 Multi-model fusion architecture diagram,

Figure 6 Comprehensive information display terminal system interface.

Detailed ways

The invention provides a multi-sensor fusion omnidirectional monitoring and intelligent camera method and system. It can be widely used in important places such as prisons, gun depots, oil depots and other important places to realize all-round monitoring and intelligent camera in sensitive areas. The target is captured and early-warned to improve the intelligence level of security monitoring.

(1) 360° fixed camera module

The 360° fixed camera module is responsible for shooting 360° panorama images in the horizontal direction. It is assembled by 4 fixed-focus cameras. The angle between the adjacent 4 cameras is 90°, the focal length of the camera lens is 3.6mm, and the monitoring range is 85°～105°, the monitoring distance is 0～10m. The pitch angles of each camera are the same, and they are all inclined downward by 30° in the horizontal direction. In this way, the horizontal monitoring angle of the four cameras is greater than 360°, which can realize 360° all-round monitoring in the horizontal direction, and achieve the monitoring goal of "seeing the whole". The overlapping angle of the two adjacent cameras in the horizontal direction is greater than 10°, which facilitates accurate stitching of images captured by adjacent cameras. The schematic diagram of 360° fixed camera module assembly is shown in Figure 1.

(2) 360° scanning camera module

The 360° scanning camera module is responsible for capturing high-definition images of the target of interest, and consists of a PTZ, a laser ranging sensor and a zoom camera, as shown in Figure 2. Among them, the gimbal can be rotated 360°, the laser ranging sensor and the zoom camera are fixed on the same support column, the horizontal orientation is the same, and the vertical tilt angle is the same, both of which are inclined downward by 30°. The central server obtains the distance information measured by the laser ranging sensor through the serial port, analyzes it, calculates the focal length of the zoom camera, and sends the control command to the control module to adjust the focal length of the zoom camera, obtain clear images, and achieve "See clearly" monitoring purpose.

(3) Intelligent processing and control module

The processor of the intelligent processing and control module is an ARM board, which mainly realizes three functions:

(3.1) PTZ control

The intelligent processing and control module realizes the control of the gimbal motor through a rectangular wave generator with a PWM pulse frequency of 50Hz and an adjustable duty cycle. Different duty cycles correspond to a fixed angle of zero-degree rotation of the motor, and the duty cycle of the rectangular wave is 5%. -95%, the corresponding gimbal motor rotation angle is 0°-360°, which controls the gimbal rotation.

(3.2) Laser ranging signal processing and zoom camera control

The intelligent processing and control module connects the laser ranging signal through the USB to TTL interface, and converts it into the distance information of the target; the zoom camera uses the preset position to adjust the focus, and there are 6 preset positions corresponding to 6 distance segments respectively. The distance information is preset, and then the zoom camera is controlled to adjust the focus through the USB serial port to obtain a clear image.

(3.3) Intelligent recognition and snapshot

After the intelligent processing and control module receives the laser signal and controls the zoom camera to focus, it intelligently recognizes the images captured by the zoom camera. Specifically, a multi-model fusion target detection side algorithm is deployed on the ARM, and the detection result of the algorithm is pedestrians. image is captured. The target detection side algorithm of multi-model fusion uses the transfer learning method to transfer the model pre-trained for target detection on the official website to the specific target recognition, freezing the network in the fully connected layer of the deep convolutional neural network, The parameters in these frozen network layers are not updated by gradient during the training process of the model, and the parameters that can be optimized are only all the parameters of the feature extractor that is not frozen, which we call the feature extractor to obtain the feature map. Then, the feature maps are connected, and the fully connected layer responsible for the output classification of the entire model is reconstructed. Three fully connected layers are used, and the optimization method of random deactivation is used to optimize the parameters and improve the robustness of the model. Specifically, the present invention integrates the VGG16 network and the ResNet50 network. The VGG16 feature extractor refers to the convolution block structure of the VGG16 network, and the ResNet50 feature extractor refers to the residual block structure of ResNet50;

The VGG16 network consists of 13 convolutional layers and 3 fully connected layers. The network structure is shown in Figure 3. The biggest feature is the combination and stacking of 3×3 filters to extract image features. For a specific target, this method extracts rich detailed features and enhances the feature's ability to distinguish between regions of interest and regions of non-interest. The VGG16 network feature extractor used in the present invention is the convolution block structure part of the dotted box in FIG. 3 .

The ResNet50 network contains 49 convolutional layers and 1 fully connected layer, and the network structure is shown in Figure 4. Since the identity mapping layer is added to the network, the shallow network and the deep network are directly connected. The function of the connection method in this way is that it does not degenerate with the increase of the network depth, and the convergence effect is good. Using this feature, the problem of missing and under-fitting VGG16 network features can be solved. The ResNet50 network feature extractor used in the present invention is the residual block structure part of the dashed box in FIG. 4 .

The model diagram after fusion of VGG16 and ResNet50 feature extractor is shown in Fig. 5.

Traditional machine learning frameworks require a large amount of calibration training data, which will consume a lot of manpower and material resources. Without a large amount of labeled data, many studies and applications related to learning cannot be carried out. Traditional machine learning assumes that the training data and test data follow the same data distribution, and in many cases, this same distribution assumption is not satisfied. On the other hand, if we have a large amount of training data under different distributions, it is also very wasteful to discard this data completely. How to use these data reasonably is the main problem solved by transfer learning. Transfer learning can transfer knowledge from existing data to aid future learning. The goal of transfer learning is to use the knowledge learned from one environment to help the learning task in the new environment. Therefore, transfer learning does not make the same distribution assumption as traditional machine learning. The current work on transfer learning can be divided into the following three parts: instance-based transfer learning in homogeneous space, feature-based transfer learning in homogeneous space and transfer learning in heterogeneous space. The invention adopts feature-based transfer learning in isomorphic space, finds common feature representation in the feature space of source domain and target domain, narrows the difference between the two domains, and improves the detection performance of the model in the target domain. Since the specific targets selected by the present invention are people and vehicles, compared with the relatively complete data sets in the target detection field, the high-level network feature layer has universality in terms of shape, texture, etc., using a large amount of labeled data contained in the target detection field. Performing transfer learning can greatly reduce the time to train the model and achieve good results. The spatial distribution of the features extracted by the feature extraction layers of different models is different. The transfer learning method can transfer the feature extraction capabilities of different models and build a new fully connected layer. Retrain to better fit different models. The present invention adopts the VGG16 and ResNet50 models provided by the Keras library, and uses the pre-training weights trained from the ImageNet data set for migration learning.

Based on the transfer learning method, the VGG16 and ResNet50 feature extractors are transferred from the target detection field to the pedestrian detection field, and the two models are fused. Compared with a single model, the multi-model fusion network structure extracts a total of 2560 images to be detected7 *7 feature maps, of which the VGG16 feature extractor extracted 512 feature maps, and the ResNet50 feature extractor extracted 2048. Since the ResNet50 network model adopts the skip connection mechanism, the image information of the specific target is passed to the deeper layers of the network, such as The information such as texture and edge makes the extracted pedestrian image information richer. The invention fuses the feature extractors in the migrated model, re-builds the fully connected layer, freezes the feature extractors of the two models, and adopts the method of randomly initializing the fully connected layer parameters to train the model after fusion.

The target detection side algorithm of multi-model fusion uses the transfer learning method to speed up the training speed of the fusion model, and adopts the multi-model fusion strategy to improve the reliability of pedestrian detection.

(4) Integrated information display terminal

The comprehensive information display terminal displays the monitoring data of the 360° fixed camera module, the 360° scanning camera module, and the intelligent analysis and snapshot results of the intelligent processing and control module. As shown in Figure 6, the system interface is divided into three areas: "see all", "see clearly" and "understand", as shown in the figure. The "see full" area displays the all-round monitoring information obtained by the 4-way fixed-focus cameras; the "clear" area displays the clear images obtained by the zoom camera, which can automatically adjust the focal length; the "understand" area displays interesting Smart capture images of pedestrian targets.

Claims (4)

1. The omni-directional monitoring and intelligent camera system of multi-sensor fusion, including 360° fixed camera module, 360° scanning camera module, intelligent processing and control module and comprehensive information display terminal, is characterized in that, The 360° fixed camera module includes 4 fixed-focus cameras to realize all-round monitoring and display, so that the monitoring scene can be seen comprehensively; The included angle between the four adjacent cameras is 90°, the focal length of the camera lens is 3.6mm, the monitoring range is 85°～105°, and the monitoring distance is 0～10m; The 360° scanning camera module includes a zoom camera, a radar, and a pan/tilt. The pan/tilt realizes 360° scanning. The radar senses the target distance, automatically controls the zoom camera to adjust the focal length, and realizes optical amplification of targets at different distances, so that the monitoring target can see the target. clear; The pan/tilt rotates 360°, the laser ranging sensor and the zoom camera are fixed on the same support column, the horizontal orientation is the same, and the vertical tilt angle is the same, and both are inclined downward by 30°; The central server obtains the distance information measured by the laser ranging sensor through the serial port, analyzes it, calculates the focal length of the zoom camera, and sends the control command to the control module to adjust the focal length of the zoom camera, obtain clear images, and achieve "Clear" monitoring purposes; The intelligent processing and control module deploys a target detection and identification algorithm to intelligently identify the target of interest in the monitoring scene; The intelligent processing and control module realizes the gimbal control through a rectangular wave generator with a PWM pulse frequency of 50Hz and an adjustable duty cycle. Different duty cycles correspond to the zero-degree rotation of the gimbal and a fixed angle. The duty cycle of the rectangular wave is 5%- 95%, the corresponding gimbal rotation angle is 0°-360°, which controls the gimbal rotation; The intelligent processing and control module connects the laser ranging signal through the USB to TTL interface, and converts it into the distance information of the target; the zoom camera uses the preset position to adjust the focus, and there are 6 preset positions corresponding to 6 distance segments respectively. The distance information is preset, and then the zoom camera is controlled to adjust the focus through the USB serial port to obtain a clear image; After the intelligent processing and control module receives the laser signal and controls the zoom camera to focus, it intelligently recognizes the images captured by the zoom camera. Specifically, a multi-model fusion target detection side algorithm is deployed on the ARM, and the detection result of the algorithm is pedestrians. to capture the image; The comprehensive information display terminal displays all-round monitoring information and performs intelligent photography, so as to meet the video monitoring requirements of high security levels. 2. The omnidirectional monitoring and intelligent camera system of multi-sensor fusion according to claim 1, is characterized in that, the pitch angles of described 4 cameras are the same, and are all horizontally inclined downward by 30°, and 4 cameras are horizontally oriented. The monitoring angle is greater than 360°, enabling 360° all-round monitoring in the horizontal direction, and the overlapping angle of two adjacent cameras in the horizontal direction is greater than 10°, enabling accurate stitching of images captured by adjacent cameras. 3. The omni-directional monitoring and intelligent camera system of multi-sensor fusion according to claim 1, wherein the target detection side algorithm migrates the pre-trained model for target detection to the specific target recognition , freeze the network in the fully connected layer of the deep convolutional neural network, so that the parameters in these frozen network layers do not undergo gradient updates during the training process of the model, and the parameters that can be optimized are only the feature extractors that are not frozen All the parameters of , we call the feature extractor, and get the feature map; then, connect the feature maps to reconstruct the fully connected layer that undertakes the classification work of the entire model output, using 3 layers of fully connected layers, using the optimization method of random deactivation , perform parameter optimization, specifically, fuse the VGG16 network with the ResNet50 network, the VGG16 feature extractor refers to the convolution block structure of the VGG16 network, and the ResNet50 feature extractor refers to the ResNet50 residual block structure; The VGG16 network consists of 13 convolutional layers and 3 fully connected layers. Image features are extracted by combining and stacking 3×3 filters. The ResNet50 network contains 49 convolutional layers and 1 fully connected layer. The identity mapping layer is added to the network to directly connect the shallow network and the deep network to solve the problems of feature loss and underfitting of the VGG16 network. 4. The omni-directional monitoring and intelligent camera system of multi-sensor fusion according to claim 3, wherein the transfer learning adopts feature-based transfer learning in a homogeneous space, and finds the feature space of the source domain and the target domain To reduce the differences between the two fields, the VGG16 and ResNet50 models that come with the Keras library are used, and the pre-trained weights trained from the ImageNet dataset are used for transfer learning.

Images