CN112163483A - Target quantity detection system - Google Patents
Target quantity detection system Download PDFInfo
- Publication number
- CN112163483A CN112163483A CN202010975300.7A CN202010975300A CN112163483A CN 112163483 A CN112163483 A CN 112163483A CN 202010975300 A CN202010975300 A CN 202010975300A CN 112163483 A CN112163483 A CN 112163483A
- Authority
- CN
- China
- Prior art keywords
- module
- target
- computing platform
- image
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Abstract
The invention discloses a target quantity detection system, and belongs to the technical field of target detection and machine vision. The target quantity detection system comprises a universal unmanned aerial vehicle platform, a multiband image acquisition module, an embedded computing platform, a ground end module and a wireless network module. The target quantity detection system of the invention has the following characteristics: (1) the all-weather detection capability and the living body number detection capability of the whole system can be realized through feature fusion by acquiring data through a visible light and infrared thermal imaging vision sensor; (2) the system is directly carried on a universal unmanned aerial vehicle platform, has the characteristics of high maneuverability and wide visual angle, and is suitable for counting requirements of moving targets in a large range; (3) the image with the fused features is transmitted to the mobile terminal App through wireless image transmission, so that more convenient image viewing and target tracking selection can be realized.
Description
Technical Field
The application belongs to the technical field of unmanned aerial vehicles, the image processing technology and the deep learning field, and particularly relates to a target quantity detection system.
Background
The target quantity detection is an important application of a target detection algorithm, and the counting of the target quantity in a given area can be applied to people flow and traffic flow statistics, so that the congestion condition of a certain area or road is evaluated, or the commercial operation condition of a specific place is evaluated; it can also be used for counting the number of specific populations, such as the number of specific animal populations in the natural reserve.
The current target quantity check out test set that commonly uses is mostly fixed visible light surveillance camera head, and its working process is: and transmitting the collected visible light video data to a local storage device or a server, running a target number detection program at a local PC end or a server end, and further summarizing to obtain a target counting statistical result in a given area in a certain time range.
In addition, the existing target quantity detection technology often has the following problems:
first, prior art is more based on fixed visual angle camera equipment, receives the mounted position and the angle of vision influence of fixed visual angle camera equipment, and the scope that current target quantity detection device can count is comparatively limited, hardly adapts to the count demand that removes the target in the great within range.
Secondly, in the visible light camera device used more in the prior art, the video image output by the device is greatly affected by external conditions such as resolution, ambient light change, and object appearance covering, and especially the detection accuracy of the number of objects in bad weather or at night is low, and the effect is not good.
Disclosure of Invention
In order to solve the defects of the prior art, the invention provides a target quantity detection system which is directly mounted on a universal unmanned aerial vehicle, can realize all-weather detection capability by utilizing multiband images through a feature fusion algorithm, has the characteristics of high maneuverability and wide visual angle, and is suitable for counting requirements of moving targets under various conditions and in a larger range.
The technical scheme adopted by the invention is as follows:
a target quantity detection system comprising the following modules:
the multiband image acquisition module is used for acquiring visible light image data and infrared image data;
the embedded computing platform is used for deploying a neural network model for target detection and counting, carrying out multi-band image data fusion on the image data acquired by the multi-band image acquisition module and carrying out target detection and counting by adopting the neural network model;
the wireless network module is used for establishing a wireless transmission channel between the nodes;
the universal unmanned aerial vehicle platform is used for carrying the multiband image acquisition module, the embedded computing platform and the wireless network module;
and the ground end module is used for controlling the universal unmanned aerial vehicle platform and displaying a target detection result.
Among the above-mentioned technical scheme, furtherly, general unmanned aerial vehicle mainly include hardware modules such as frame, power supply unit, flight control device, power device, navigation module and camera cloud platform, possess and stabilize gesture take off and land, hover and cruise function. The system provided by the invention is directly mounted on a universal unmanned aerial vehicle platform, so that the whole system has higher flexibility, wider visual angle, lower operation cost and relatively easy flight control design, and can adapt to the counting requirement of moving targets in a larger range (such as the counting of the number of animals with various behavior patterns in a natural conservation area).
Furthermore, the multiband image acquisition module is deployed on a universal unmanned aerial vehicle platform and comprises an infrared band image acquisition device and a visible light band image acquisition device, the module can transmit a multiband image sequence acquired by acquisition to the embedded computing platform for image fusion, image data acquired by fusion has visible details such as colors and textures and remarkable edge gradient characteristics and heat-sensitive characteristics of infrared bands, and the adaptability of the target quantity detection system to complex weather and light conditions and the detection capability of living targets can be effectively improved.
Further, the embedded computing platform is deployed on a general unmanned aerial vehicle platform, and specifically includes the following functions:
1) the method for carrying out feature fusion on the image data acquired by the multiband image acquisition module comprises the following steps: correcting image distortion according to factory data of the multiband image acquisition module and a mobile plane checkerboard calibration method; performing spatial registration operation by using a scale invariant feature transform matching algorithm (SIFT feature matching algorithm) disclosed in the field, thereby obtaining visible light and infrared image sequences with uniform size and matched features, and finally realizing fusion of multiband feature information by using an image fusion algorithm to obtain the visible light and infrared fusion image sequences;
2) deploying a neural network model for target detection and counting on the embedded computing platform, training the neural network model by adopting a visible light and infrared fusion image data set which is labeled in advance, and using the trained model for a newly obtained fusion image sequence; in addition, a boundary box of a target detection result can be drawn on the original fusion image sequence, and the targets of a specific category are counted to obtain a target number counting result and an image file drawn with the target boundary box, so that the functions of detecting and counting the multiband and multi-type targets are realized;
3) the embedded computing platform and the wireless network module can establish bidirectional data connection through a USB-A interface, transmit the target quantity counting result and the image file to the wireless network module to be sent outwards, and receive a control signal from the wireless network module.
Further, ground end device, this ground end device is including being responsible for unmanned aerial vehicle state monitoring and remote control's PC or remote controller to and be used for the removal end App that the testing result shows. The PC and the remote controller are connected with the unmanned aerial vehicle platform through a control link established by a wireless channel, and the PC monitors and controls the state parameters of the unmanned aerial vehicle platform by using software matched with the flight control of the general unmanned aerial vehicle platform. The mobile terminal App comprises a video picture module, a map module and a target detection module, and can allow an operator to select a specific target type to display, and the selected specific target ID is transmitted back to the embedded computing platform through the wireless network module. The target quantity statistical result and the image file generated by the embedded computing platform can be transmitted to the mobile terminal App through a UDP protocol; after the control personnel select a specific target at the mobile terminal App, the information of the corresponding target can be transmitted to the embedded computing platform through a TCP protocol, and the embedded computing platform performs detection and counting.
The invention has the beneficial effects that:
the target quantity detection system of the invention has the following characteristics:
(1) the all-weather detection capability and the living body number detection capability of the whole system can be realized through feature fusion by acquiring data through a visible light and infrared thermal imaging vision sensor;
(2) the system is directly carried on a universal unmanned aerial vehicle platform, has the characteristics of high maneuverability and wide visual angle, and is suitable for counting requirements of moving targets in a large range;
(3) the image with the fused features is transmitted to the mobile terminal App through wireless image transmission, so that more convenient image viewing and target tracking selection can be realized.
Drawings
FIG. 1 is a flow chart of an embodiment of the present invention;
FIG. 2 is a block diagram of the modules of the target quantity detection system according to an embodiment of the present invention;
fig. 3 is a block diagram of the mobile terminal App according to an embodiment of the present invention.
Detailed Description
The following describes embodiments of the present invention in further detail with reference to examples. The specific embodiments described herein are merely illustrative of the invention and do not delimit the invention.
The present embodiment provides a target number detection system based on a quad-rotor drone platform, as shown in fig. 2, the target number detection system includes the following five modules:
general unmanned aerial vehicle platform: adopt four rotor unmanned aerial vehicle platforms, mainly use the carbon fiber frame to build, model aeroplane and model ship battery power supply, main control unit, IMU and GPS select for use the integrated product of big jiangjiang flight control.
The multiband image acquisition module comprises: for carrying on the visible light camera and the infrared thermal imaging sensor of unmanned aerial vehicle platform, this imaging assembly passes through the triaxial and increases steady cloud platform and carry in unmanned aerial vehicle ventral front portion, realizes the multi-angle of multiband image acquisition module and shoots and the function of stably obtaining the video picture in the flight.
The data processing process of the multiband image acquisition module specifically comprises the following steps: firstly, distortion correction is carried out on imaging results of heterogeneous sensors in visible and infrared bands, and internal and external parameters and distortion coefficients of two types of imaging elements are respectively calculated mainly by a camera calibration method of moving a plane checkerboard; then, respectively matching the edges and the corners of the images with different wave bands by using a scale invariant feature transform matching algorithm (SIFT algorithm), solving the influence of factors such as different imaging element perspectives, illumination conditions and noise on the images, and obtaining a multi-band image sequence with consistent scene and uniform size; finally, an image fusion algorithm is applied to realize fusion of multiband characteristic information, and the embodiment provides a self-coding neural network method to realize multiband characteristic information fusion so as to retain details of a fusion image sequence to the greatest extent, which is specifically as follows: the self-coding neural network method is an infrared and visible light image fusion self-coder network architecture constructed based on a convolutional neural network, comprises three parts of a coder, a fusion layer and a decoder, and is used for respectively extracting all significant features of a visible light image and an infrared image and fusing the significant features to generate a multiband feature fusion graph; the encoder performs convolution operation on an input visible light image and an input infrared image sequence to realize data dimension reduction and obtain a low-dimensional feature map containing multiband significant features; then, carrying out weighted fusion on the low-dimensional visible light and infrared characteristic images through a fusion layer; and finally, the low-dimensional fusion feature map is subjected to up-sampling through a decoder network to be restored to a high-dimensional feature fusion image with the original image size.
An embedded computing platform: a high-performance embedded platform supporting neural network operation acceleration is selected, and the high-performance embedded platform comprises an Invitta Jetson TX2 or an Xavier platform. The high-performance embedded platform is provided with a neural network model and can be used for target detection and counting functions. The neural network model described in this embodiment may be any neural network model for target Detection and counting, and in this example, a customized MultiBand image target Detection network model (MultiBand-NET) based on a YOLOv3 structure is provided, which is a feature extraction backbone network composed of a DBL convolution network unit and a resialblock residual network unit and a multiscale Detection head network composed of an upscale upsampling module, a Concat splicing fusion layer, and a Detection module, where a convolution layer, a pooling layer, a loss function, and an activation function module in deep learning frames tens and Keras are called in the embedded computing platform, and a MultiBand constructed target Detection network model MultiBand-NET of a MultiBand image includes a DBL convolution network unit and a resialblock residual network unit; and after the construction is finished, iteratively training the hyper-parameters of the network model according to the pre-labeled visible light and infrared band fusion image data set. And applying the trained detection model Multiband-NET to the fusion image sequence to perform an actual frame-by-frame target quantity detection task. The detection model outputs the boundary frame coordinates, the target detection result ID and the confidence probability value information of the preset targets in the picture, records the number of the targets in the same category in each frame of picture, draws the detection result to the original image to form a video file, and outputs the video file to the ground end for display.
A wireless network module: the high-performance embedded platform performs integrated management, and a wireless transmission link between the mobile terminal device and the high-performance embedded computing platform can be constructed.
A ground end device: including PC, the remote controller that is responsible for unmanned aerial vehicle state monitoring and remote control to and be used for the removal end App that the testing result shows. The PC and the remote controller are connected with the unmanned aerial vehicle platform through a control link established by a wireless channel, and the PC monitors and controls the state parameters of the unmanned aerial vehicle platform by using software matched with the flight control of the universal unmanned aerial vehicle platform. The mobile terminal APP comprises a video picture module, a map module and an object detection module, as shown in FIG. 3. The video picture module establishes a UDP (user datagram protocol) protocol with a high-performance embedded platform deployed on the unmanned aerial vehicle, receives fusion image data processed by the high-performance embedded platform, and immediately displays the processed video picture on a mobile terminal window. The map module establishes a UDP protocol with a high-performance embedded platform deployed on the unmanned aerial vehicle, receives GPS signal data, and immediately displays the position of the unmanned aerial vehicle on a mobile terminal window. The target detection module and a high-performance embedded platform deployed on the unmanned aerial vehicle establish a TCP (transmission control protocol) protocol and support control personnel to select a specific target. After the control personnel select the specific target, the target detection module transmits the specific target ID to the high-performance embedded platform.
Claims (7)
1. A target quantity detection system, comprising the following modules:
the multiband image acquisition module is used for acquiring visible light image data and infrared image data;
the embedded computing platform is used for deploying a neural network model for target detection and counting, carrying out multi-band image data fusion on the image data acquired by the multi-band image acquisition module and carrying out target detection and counting by adopting the neural network model;
the wireless network module is used for establishing a wireless transmission channel between the nodes;
the universal unmanned aerial vehicle platform is used for carrying the multiband image acquisition module, the embedded computing platform and the wireless network module;
and the ground end module is used for controlling the universal unmanned aerial vehicle platform and displaying a target detection result.
2. The system of claim 1, wherein the universal drone platform comprises a frame, a power supply device, a flight control device, a power device and a camera pan-tilt, and has functions of stable attitude take-off and landing, hovering and cruising.
3. The system of claim 1, wherein the multi-band image capture module comprises an infrared band image capture device and a visible band image capture device.
4. A target quantity detection system as claimed in claim 1, wherein: the embedded computing platform sequentially performs distortion correction, spatial registration and feature fusion processing on image data acquired by the multiband image acquisition module, and specifically comprises the following steps: firstly, distortion correction is carried out on imaging results of visible light and infrared bands, and internal and external parameters and distortion coefficients of two types of imaging elements are respectively calculated by a camera calibration method of moving a plane checkerboard; matching the edges and the angular points of the images with different wave bands respectively by using an SIFT algorithm to obtain a multi-band image sequence with consistent scene and uniform size; and finally, realizing the fusion of multiband characteristic information by applying an image fusion algorithm.
5. The system of claim 1, wherein the wireless network module is configured to establish a wireless transmission channel between the embedded computing platform and the ground mobile terminal, and the wireless transmission channel is used for data transmission and image transmission.
6. The system for detecting the number of targets in the aircraft according to claim 1, wherein the ground segment module comprises a PC or a remote controller responsible for unmanned aerial vehicle state monitoring and remote control, and a mobile terminal App for detecting result display.
7. The system for detecting the number of targets according to claim 6, wherein the mobile terminal App comprises a video picture module, a map module and a target detection module; the video picture module and the embedded platform establish a UDP protocol, receive the fused image data processed by the embedded computing platform and display the fused image data on an App window of the mobile terminal in real time; the map module and the embedded computing platform establish a UDP protocol, receive GPS signal data and instantly display the position of the unmanned aerial vehicle at a mobile terminal App window, the target detection module and the embedded computing platform establish a TCP protocol and support a controller to select a specific target, after the specific target is selected, the target detection module transmits the information of the specific target to the embedded computing platform, and the embedded computing platform detects and counts the specific target.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010975300.7A CN112163483A (en) | 2020-09-16 | 2020-09-16 | Target quantity detection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010975300.7A CN112163483A (en) | 2020-09-16 | 2020-09-16 | Target quantity detection system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112163483A true CN112163483A (en) | 2021-01-01 |
Family
ID=73858124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010975300.7A Pending CN112163483A (en) | 2020-09-16 | 2020-09-16 | Target quantity detection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112163483A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112949579A (en) * | 2021-03-30 | 2021-06-11 | 上海交通大学 | Target fusion detection system and method based on dense convolution block neural network |
| CN113280706A (en) * | 2021-03-22 | 2021-08-20 | 中国十七冶集团有限公司 | Foldable measuring scale for square number of ground cover seedlings and using method |
| CN113724250A (en) * | 2021-09-26 | 2021-11-30 | 新希望六和股份有限公司 | Animal target counting method based on double-optical camera |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107911429A (en) * | 2017-11-04 | 2018-04-13 | 南京奇蛙智能科技有限公司 | A kind of online traffic flow monitoring method in unmanned plane high in the clouds based on video |
| CN109685078A (en) * | 2018-12-17 | 2019-04-26 | 浙江大学 | Infrared image recognition based on automatic marking |
| CN109934148A (en) * | 2019-03-06 | 2019-06-25 | 华瑞新智科技（北京）有限公司 | A kind of real-time people counting method, device and unmanned plane based on unmanned plane |
| CN111247558A (en) * | 2018-12-04 | 2020-06-05 | 深圳市大疆创新科技有限公司 | Image processing method, device, unmanned aerial vehicle, system and storage medium |
-
2020
- 2020-09-16 CN CN202010975300.7A patent/CN112163483A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107911429A (en) * | 2017-11-04 | 2018-04-13 | 南京奇蛙智能科技有限公司 | A kind of online traffic flow monitoring method in unmanned plane high in the clouds based on video |
| CN111247558A (en) * | 2018-12-04 | 2020-06-05 | 深圳市大疆创新科技有限公司 | Image processing method, device, unmanned aerial vehicle, system and storage medium |
| CN109685078A (en) * | 2018-12-17 | 2019-04-26 | 浙江大学 | Infrared image recognition based on automatic marking |
| CN109934148A (en) * | 2019-03-06 | 2019-06-25 | 华瑞新智科技（北京）有限公司 | A kind of real-time people counting method, device and unmanned plane based on unmanned plane |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113280706A (en) * | 2021-03-22 | 2021-08-20 | 中国十七冶集团有限公司 | Foldable measuring scale for square number of ground cover seedlings and using method |
| CN112949579A (en) * | 2021-03-30 | 2021-06-11 | 上海交通大学 | Target fusion detection system and method based on dense convolution block neural network |
| CN113724250A (en) * | 2021-09-26 | 2021-11-30 | 新希望六和股份有限公司 | Animal target counting method based on double-optical camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10645300B2 (en) | Methods and apparatus for image processing | |
| US11748898B2 (en) | Methods and system for infrared tracking | |
| US10650235B2 (en) | Systems and methods for detecting and tracking movable objects | |
| CN112163483A (en) | Target quantity detection system | |
| US10802509B2 (en) | Selective processing of sensor data | |
| US10599149B2 (en) | Salient feature based vehicle positioning | |
| CN108351649B (en) | Method and apparatus for controlling a movable object | |
| CN104504748B (en) | A kind of infrared 3-D imaging system of unmanned plane oblique photograph and modeling method | |
| US11644839B2 (en) | Systems and methods for generating a real-time map using a movable object | |
| CN112462802A (en) | System and method for providing autonomous photography and photography | |
| AU2015305406A1 (en) | Methods for agronomic and agricultural monitoring using unmanned aerial systems | |
| US10891483B2 (en) | Texture classification of digital images in aerial inspection | |
| WO2019100219A1 (en) | Output image generation method, device and unmanned aerial vehicle | |
| US10602064B2 (en) | Photographing method and photographing device of unmanned aerial vehicle, unmanned aerial vehicle, and ground control device | |
| CN115240089A (en) | Vehicle detection method of aerial remote sensing image | |
| CN107985577A (en) | A kind of safe unmanned plane | |
| WO2018211396A1 (en) | Detection of powerlines in aerial images | |
| CN114727028B (en) | Image exposure method and device and unmanned aerial vehicle | |
| CN113063401A (en) | Unmanned aerial vehicle aerial survey system | |
| CN111273134A (en) | Ultraviolet and short wave infrared band fused corona monitoring device | |
| CN115115957A (en) | Airplane identification system, airplane identification method and airplane identification equipment based on image analysis | |
| CN117110160A (en) | Night PM2.5 monitoring method and system based on multi-data fusion | |
| CN117789063A (en) | Unmanned aerial vehicle-mounted AI image processing equipment and method | |
| CN117635889A (en) | Real-time rendering method, system and device for laser point cloud data | |
| JP2022112431A (en) | Electronic apparatus, method for controlling electronic apparatus, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210101 |