CN115035470A - Low, small and slow target identification and positioning method and system based on mixed vision - Google Patents

Low, small and slow target identification and positioning method and system based on mixed vision Download PDF

Info

Publication number
CN115035470A
CN115035470A CN202210644599.7A CN202210644599A CN115035470A CN 115035470 A CN115035470 A CN 115035470A CN 202210644599 A CN202210644599 A CN 202210644599A CN 115035470 A CN115035470 A CN 115035470A
Authority
CN
China
Prior art keywords
target
low
small
camera
slow
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210644599.7A
Other languages
Chinese (zh)
Inventor
何鎏
孔德磊
徐庶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanhu Research Institute Of Electronic Technology Of China
Original Assignee
Nanhu Research Institute Of Electronic Technology Of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanhu Research Institute Of Electronic Technology Of China filed Critical Nanhu Research Institute Of Electronic Technology Of China
Priority to CN202210644599.7A priority Critical patent/CN115035470A/en
Publication of CN115035470A publication Critical patent/CN115035470A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/417—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section involving the use of neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Remote Sensing (AREA)
  • Multimedia (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Medical Informatics (AREA)
  • Electromagnetism (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

The invention discloses a low-small slow target identification and positioning method and system based on mixed vision, wherein a target in a visual field range is detected through an event camera and tracked to obtain a target coordinate position, after the target coordinate position information is obtained, a holder visible light camera is triggered to carry out zoom stretching, a stretched target image is collected, the stretched target image is subjected to category detection to obtain category information corresponding to the target, and if the target is an interested low-small slow target, a laser radar is triggered to carry out ranging on the target to obtain distance information of the low-small slow target; and the main control computer calculates the running track of the low, small and slow target and carries out human-computer interaction with an operator through a human-computer interaction interface. The invention improves the accuracy of low, small and slow target identification and positioning and reduces the false alarm rate.

Description

Low, small and slow target identification and positioning method and system based on mixed vision
Technical Field
The application belongs to the technical field of target identification and positioning, and particularly relates to a hybrid vision-based low-small slow target identification and positioning method and system.
Background
The civil unmanned aerial vehicle technology is mature day by day, and accidents caused by the civil unmanned aerial vehicle and related safety problems are increasingly prominent while the application is wider. The unmanned aerial vehicle can easily carry out activities such as photographing and shooting by virtue of the flight flexibility and a complete shooting technology. In some sensitive areas, such as civil aviation airports, important material warehouses, meeting sites and the like, chaos and even great loss can be caused due to the intrusion of the unmanned aerial vehicle. Some equipment unmanned aerial vehicle are because the quality is too poor and control the reason and probably lead to taking place accidents such as air collision, falling, catching fire, explosion, constitute very big dangerous hidden danger.
Therefore, detecting such "low, small and slow" targets of unmanned aerial vehicles is an important topic for researchers in this field. At present, target detection and interception and striking equipment for a low-speed and small-speed unmanned aerial vehicle is developed by multiple units or mechanisms at home and abroad, and various low-speed and small-speed target detection means such as radar detection, optical detection, acoustic detection, radio frequency spectrum monitoring and the like appear.
For example, according to the FDA-MIMO radar-based low-small slow target detection method, electromagnetic wave signals are transmitted and received by constructing a multi-path multi-transmission and multi-reception frequency diversity array, discrete sampling is carried out on the adjusted received signals, an echo data array is constructed and expressed in a vector form, then a target fluctuation model is introduced, and an FDA-MIMO radar low-small slow target is deduced by adopting a generalized likelihood ratio detection method. Under a certain false alarm rate probability, the probability of target detection can be effectively improved. For another example, the target of "low, small and slow" is detected by using an image detection method, a target training sequence is formed by collecting a target image in advance and stacking a plurality of frames of the target image, the target image training sequence is labeled to form a target training sample, a target image detection model is trained based on the target training sample set, and the target image to be detected is detected to detect whether the target to be detected is "low, small and slow".
However, radar detection has a detection blind area, cannot detect low altitude, has a small radar scattering surface, is difficult to find a small unmanned aerial vehicle, and has the defects of high manufacturing cost, large size, difficult deployment and the like. The optical detection is greatly influenced by light and environmental factors, and target texture details cannot be obtained in a large visual field range, so that the detection fails; the detection distance of the acoustic detection is short, so that the noise of the unmanned aerial vehicle is easy to be integrated with the background noise, and the anti-interference capability is poor; radio frequency spectrum monitoring only supports the detection of a few types of unmanned aerial vehicle brands such as Xinjiang, and has more false alarms. At present, common visible light cameras and infrared cameras are mainly used for the short-distance detection method. But there is the contradiction problem between surveying the large scale and obtaining the target detail, because unmanned aerial vehicle volume is too little, it is difficult to detect unmanned aerial vehicle flight in the air through the form of visible light in a long distance.
Disclosure of Invention
The application aims to provide a method and a system for identifying and positioning a low-small slow target based on mixed vision, so as to overcome the technical defects and improve the accuracy of identifying the low-small slow target.
In order to achieve the purpose, the technical scheme of the application is as follows:
a mixed vision-based low-small slow target recognition and positioning method is applied to a low-small slow target recognition and positioning system, the low-small slow target recognition and positioning system comprises an event camera, a holder visible light camera, a laser radar and a main control computer, and the mixed vision-based low-small slow target recognition and positioning method comprises the following steps:
acquiring an event acquired by an event camera, detecting and tracking a target in a visual field range, and acquiring a coordinate position of the target;
after the target coordinate position information is obtained, triggering a holder visible light camera to perform zoom stretching, and collecting a stretched target image;
performing category detection on the stretched target image to obtain category information corresponding to the target, if the target is an interested low-small slow target, performing the next operation, and if not, not paying attention to the target;
triggering a laser radar to measure the distance of the low and small slow targets of interest to obtain the distance information of the low and small slow targets;
and sending target coordinate position information obtained by detecting the event camera, category information detected by the holder visible light camera and distance information obtained by detecting the laser radar to a main control computer, calculating the running track of the low, small and slow targets by the main control computer, and performing man-machine interaction with an operator through a man-machine interaction interface.
Further, after the target coordinate position information is obtained, triggering the holder visible light camera to perform variable-magnification stretching, wherein the variable-magnification stretching multiple corresponding to the variable-magnification stretching satisfies the following relational expression:
scale＝100/max(boundbox(w,h))
wherein scale represents the magnification-varying stretching ratio, boundbox (w, h) represents the detection frame, and w and h represent the width and height of the detection frame.
Furthermore, the event camera, the holder visible light camera and the laser radar are installed on the same horizontal line.
Further, the acquiring an event collected by the event camera, detecting a target in a visual field and tracking the target includes:
and training the impulse neural network model, and inputting an event sequence acquired by the event camera into the trained impulse neural network model to detect and track the target.
Further, the performing category detection on the stretched target image to obtain category information corresponding to the target includes:
and performing target detection on the stretched target image by using a convolutional neural network to obtain the class information corresponding to the target.
The application also provides a low-small slow target recognition and positioning system based on mixed vision, which comprises an event camera, a holder visible light camera, a laser radar and a main control computer, wherein:
the event camera is used for collecting events, detecting and tracking a target in a visual field range and acquiring a coordinate position of the target;
the holder visible light camera is used for performing zooming stretching after obtaining the coordinate position information of the target, acquiring a stretched target image, and performing category detection on the stretched target image to obtain category information corresponding to the target;
the laser radar is used for ranging the target when the target is detected to be the low-small slow target of interest to obtain the distance information of the low-small slow target;
and the main control computer is used for calculating the running track of the low, small and slow target according to the target coordinate position information obtained by the detection of the sister receiving event camera, the category information detected by the holder visible light camera and the distance information obtained by the detection of the laser radar, and performing man-machine interaction with an operator through a man-machine interaction interface.
Further, the holder visible light camera performs variable-magnification stretching, wherein a variable-magnification stretching multiple corresponding to the variable-magnification stretching satisfies the following relational expression:
scale＝100/max(boundbox(w,h))
wherein scale represents the zoom stretch ratio, boundbox (w, h) represents the event camera detection box, and w and h represent the width and height of the detection box.
Furthermore, the event camera, the holder visible light camera and the laser radar are installed on the same horizontal line.
Further, the event camera detects and tracks a target in a visual field, including:
and training the impulse neural network model, and inputting an event sequence acquired by the event camera into the trained impulse neural network model to detect and track the target.
Further, the cloud deck visible light camera performs category detection on the stretched target image to obtain category information corresponding to the target, and the method includes:
and performing target detection on the stretched target image by using a convolutional neural network to obtain the class information corresponding to the target.
According to the method and the system for recognizing and positioning the low, small and slow targets based on the hybrid vision, the low, small and slow targets are detected and positioned through a multi-sensor fusion means, the dynamic event camera DVS with low power consumption is adopted for monitoring the low altitude in a long-term flight, the extremely low resource consumption is achieved, the method and the system can be used as a supplement of a radar scheme and a vision scheme, targets which cannot be detected by the radar and the vision scheme can be achieved with higher precision, and the method and the system are advanced. In addition, after DVS triggers, the laser range finder and the RGB camera are started to be linked, and the cooperation, resource scheduling and energy conservation are realized. By researching the fusion sensing technology of the DVS, the infrared thermal imaging RGB camera and the laser range finder, an integrated detection device is constructed, the rapid detection and secondary identification verification of the aerial low-low slow moving target are carried out, and the distance confirmation is carried out by the laser range finder matched with a high-precision two-degree-of-freedom servo system, so that the target with higher accuracy and lower false alarm rate is realized compared with the current detection technology based on common vision.
Drawings
FIG. 1 is a schematic diagram of a low, small and slow target identification and positioning system according to an embodiment of the present application;
FIG. 2 is a flowchart of a hybrid vision-based low-small slow target identification and positioning method according to the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
The application provides a low-small slow target recognition and positioning method based on mixed vision, which is applied to a low-small slow target recognition and positioning system, wherein the low-small slow target recognition and positioning system is shown in figure 1 and comprises a DVS camera, a holder visible light camera, a laser radar, a main control computer and a display. The DVS camera is used for detecting and tracking a target, the holder visible light camera is used for carrying out secondary identification verification on the target, and the laser radar is used for ranging and positioning the target. The DVS camera, the holder visible light camera and the laser radar are installed in the same horizontal line, the coordinate point of the DVS camera is (x1, y), the holder visible light camera consists of an infrared camera and a visible light camera, the coordinate of the center point of the holder visible light camera is (x2, y), the relative position (x2+ delta x, y) of the camera can be calculated according to the axis of the camera, and the coordinate of the center of the laser radar is (x3, y), so that the world coordinate relationship among the three can be easily obtained.
In one embodiment, as shown in fig. 2, there is provided a hybrid vision-based low-small slow target recognition and positioning method, including:
and step S1, acquiring the event collected by the event camera, detecting and tracking the target in the visual field range, and acquiring the coordinate position of the target.
The event camera is also called a dynamic vision sensor, and is a novel sensor which senses the change of light intensity and outputs a discrete pulse signal to represent the change of pixel brightness. Different from the traditional camera for shooting a complete image taking a frame as a unit, the event camera outputs a pulse event, and has the advantages of high dynamic range, quick response under low light conditions and the like (the characteristics of insensitivity to a static target and quick response to a moving target). An event has three elements: timestamp, pixel coordinates, and polarity. An event expresses "at what time, which pixel point, an increase or decrease in brightness has occurred".
The method adopts a pulse neural network model to detect the moving target, the pulse neural network is also called as a third-generation artificial neural network, the method is different from a convolution neural network to process real values, neurons are activated to participate in operation in each iteration, the pulse neural network processes a pulse sequence, the neuron model is closer to the working mode of a human brain, whether the neurons need to be activated or not is determined by whether the change of membrane potential reaches a set threshold value, and therefore the pulse neural network has an event triggering characteristic and lower computing resource consumption.
Aiming at the problems that in the existing visual scheme, in the process of detecting low, medium and small slow targets, motion blurred targets are not clear, particularly the targets are not visible when light is poor, and an artificial intelligence algorithm for processing visual input is high in calculation power and high in power consumption requirement, the method uses a pulse neural network to process event streams output by an event camera (DVS camera). Firstly, training a pulse neural network model by adopting a training sample, then detecting a moving target of an event sequence output by an event camera by adopting the trained pulse neural network model, and outputting target coordinate position information and corresponding detection frame information.
In this embodiment, training the impulse neural network model, inputting an event sequence acquired by the event camera into the trained impulse neural network model for target detection and tracking, and detecting and tracking a target based on the event camera to obtain continuous position coordinate information of the target is a well-established technology in the art, and is not described herein again.
And step S2, after the target coordinate position information is obtained, triggering the holder visible light camera to perform zoom stretching, and collecting the stretched target image.
In the previous step, the target is found based on the event camera detection, and the coordinate position information of the target is continuously output. Therefore, the position information of the target relative to the event camera can be obtained through conversion according to the world coordinates of the event camera. Meanwhile, the position information triggers the pan-tilt visible light camera to perform variable-magnification stretching on the range of the position (the stretching magnification is determined by the event camera detection frame), and the formula is as follows:
scale＝100/max(boundbox(w,h))
wherein scale represents the variable-magnification stretching ratio, and boundbox (w, h) represents the event camera detection box.
For example, the length and width of the event camera detection target are (10,8), and the zoom stretch factor 10 is obtained through conversion. If the length and width of the event camera detection target are (20,20), the zoom stretching ratio is converted to be 5.
And after the lifting multiple is obtained, adjusting the lifting multiple of the holder visible light camera. The pan-tilt-zoom visible light camera supports 30 times of variable-magnification stretching. And acquiring a target image by the variable-magnification holder visible light camera to obtain the stretched target image.
And step S3, performing category detection on the stretched target image to obtain category information corresponding to the target, if the target is an interested low-small slow target, performing the next operation, and if not, paying no attention to the target.
In the embodiment, a convolutional neural network is used for carrying out secondary verification detection on a stretched target image, the characteristic that the stretched characteristic is rich compared with the characteristic before stretching is utilized, the yoloV5 is adopted for detecting the target, and whether the target belongs to the type of the unmanned aerial vehicle in the sample library is judged.
It should be noted that, the target detection using the convolutional neural network is a mature technology in the technical field, and is not described herein again.
However, since the visible light camera is directly adopted to detect the low and low speed, the range of the picture is large, the details of the target cannot be seen clearly due to the small target, and the visible light camera and the background are easily mixed into a whole, so that the detection cannot be carried out. If people want to see details clearly, the picture can be stretched, and a small range is seen, so that the defects of monitoring a large scene and detecting an object exist. The method and the device have the advantages that the advantages of the DVS event camera are utilized, the small targets in a large range can be detected, after the suspicious targets are detected, the target types cannot be confirmed, the registered cradle head visible light camera is triggered to conduct secondary stretching, verification is identified again, the real target types are confirmed, namely the targets adopting the cradle head camera are the real types of the confirmed targets and are not real low and small slow targets, and therefore false detection is reduced.
And step S4, triggering the laser radar to measure the distance of the low and small slow targets to obtain the distance information of the low and small slow targets.
Because both the event camera and the holder visible light camera do not have the distance measurement capability and can only acquire the coordinate position information and the category information of the target, the embodiment adopts the laser radar to detect the point cloud of the low-small slow target, obtains the distance information of the point cloud, calculates the average distance of the point cloud in the registration area, and acquires the real distance of the low-small slow target.
And step S5, sending the target coordinate position information obtained by the event camera detection, the category information detected by the holder visible light camera and the distance information obtained by the laser radar detection to a main control computer, calculating the running track of the low, small and slow target by the main control computer, and performing man-machine interaction with an operator through a man-machine interaction interface.
The method and the device have the advantages that the targets are detected and found (coordinate position information is obtained through detection of the event camera), the targets are accurately identified (category information is obtained through detection of the holder visible light camera), and the target distance is accurately positioned (laser radar detection distance information) through cooperative processing of the event camera, the holder visible light camera and the laser radar.
And finally, sending the fused information to a main control computer, calculating the running track of the low, small and slow target by a program run by the main control computer, and performing human-computer interaction with an operator through a human-computer interaction interface.
In a specific embodiment, the target coordinate position information obtained by the event camera detection, the category information obtained by the holder visible light camera detection and the distance information obtained by the laser radar detection are fused. That is, the fusion is performed by using a multi-sensor fusion technology, which includes:
1. the 'low-small-slow' targets detected by different sensor data are converted into a world coordinate system for representation.
The main work includes the coordinate conversion of the target position of low, small and slow speed and the conversion of the attitude coordinate, namely, the target position and attitude information of low, small and slow speed in the sensor coordinate system are converted into the world coordinate system through the position coordinate conversion matrix and the attitude conversion matrix.
2. The position coordinate conversion relation can be derived by using the installation position of the sensor and the position of the center of the world coordinate system.
The expression of the sensor mounting position in the world coordinate system is as follows:
P BtoA ＝[p x p y p z ] T
in the formula, p x ，p y And p z Is the mounting position of the sensor in the world coordinate system.
3. The attitude information transformation matrix is a 3x3 matrix. The sensor coordinate system is defined as B, the world coordinate system is defined as A, and the posture of the coordinate system B relative to the coordinate system A is
Figure BDA0003683582410000071
：
Figure BDA0003683582410000081
In the formula, X A ，Y A ，Z A The unit vector of the principal axis direction of the world coordinate system we choose; x B ， Y B ，Z B Unit vector of the principal axis direction of the sensor coordinate system.
4. Converting the low, small and slow target detected by the sensor into position and attitude information in a world coordinate system, and expressing the position and attitude information as follows:
Figure BDA0003683582410000082
the DVS obtains coordinate position information of a target through multi-sensor data fusion, the visible light camera is triggered through converting coordinates to stretch and then detect images in the area, type information of the detected target is judged, and after the type of the target is confirmed, a registered laser radar is triggered to carry out ranging on the low, small and slow target.
The technical scheme is applied to boundary protection of cities, sentries and sensitive courtyards, and is used for monitoring low, small and slow objects of invasion all weather, analyzing in real time and giving an alarm. Due to the integration of the DVS dynamic vision sensor, the traditional RGB camera and the laser radar, the intrusion target can be autonomously positioned and identified under the condition of GPS limitation. Compared with the traditional radar, the DVS camera has small volume and low price, and can avoid the defects of high radar manufacturing cost, large volume, difficult deployment and the like in a limited way. The visible light camera is used for checking the target, and the convolutional neural network has the characteristic of high detection accuracy on the target in the image, so that the defect that the noise of the unmanned aerial vehicle is easily integrated with the background noise and the anti-interference capability is poor due to the short detection distance in the acoustic detection is effectively overcome. Use laser radar to carry out accurate location to the target, possess to measure the rate of accuracy height, measure precision height and so on a bit, effectively solved radio frequency spectrum monitoring and only supported the detection of a small number of types of unmanned aerial vehicle brands such as big jiang, and the more shortcoming of false-alarm.
The DVS camera acquires event flow information, processes the event flow information through the pulse neural network, accurately detects moving objects, and tracks the 'slow and slow' target on a time sequence by using an extended Kalman filtering and Hungary algorithm. And performing secondary verification on the image output by the visible light camera by using a convolutional neural network, and tracking the target in the image sequence by using a kernel correlation filtering algorithm. And detecting and tracking an object in a 3D point cloud picture scanned by the laser radar by using a 3D convolutional neural network, and matching the object detected and identified in the visible light image with the result detected in the laser radar point cloud in terms of time and space so as to realize the identification, tracking and positioning of the low, small and slow target.
In one specific example, a DVS camera (a specific propheiesee camera, 100 ten thousand pixels) is used, with detection of a small target using a spiking neural network, to obtain coordinate position information of the target. The pan-tilt visible light camera adopts 400 ten thousand pixels and 2K resolution, the focal length f is 10-800mm, 80 times of optical zoom and 16 times of digital zoom, the detected target is at the distance of 3000 meters at the maximum, the pixel value of a four-rotor unmanned aerial vehicle with the length of 30 centimeters and the width of 30 centimeters in an image is 30-20, and the accuracy detection rate of a convolutional neural network is 98%. After a suspicious target is detected by the DVS, the holder visible light camera is triggered to stretch the specified position, and then the detection experiment work of the low, small and slow target is carried out at the same time to confirm the category. And finally triggering the laser radar again to measure the distance of the target, and finishing the whole process.
In another embodiment, a hybrid vision based low-small slow target recognition and positioning system comprises an event camera, a pan-tilt visible light camera, a laser radar and a master control computer, wherein:
the event camera is used for collecting events, detecting and tracking a target in a visual field range and acquiring a coordinate position of the target;
the holder visible light camera is used for performing zooming stretching after obtaining the coordinate position information of the target, acquiring a stretched target image, and performing category detection on the stretched target image to obtain category information corresponding to the target;
the laser radar is used for ranging the target when the target is detected to be the interested low-small slow target, so as to obtain the distance information of the low-small slow target;
and the main control computer is used for calculating the running track of the low, small and slow target according to the target coordinate position information obtained by the detection of the sister receiving event camera, the category information detected by the holder visible light camera and the distance information obtained by the detection of the laser radar, and performing man-machine interaction with an operator through a man-machine interaction interface.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A mixed vision-based low-small slow target recognition and positioning method is applied to a low-small slow target recognition and positioning system, and is characterized in that the low-small slow target recognition and positioning system comprises an event camera, a holder visible light camera, a laser radar and a main control computer, and the mixed vision-based low-small slow target recognition and positioning method comprises the following steps:
acquiring an event acquired by an event camera, detecting and tracking a target in a visual field range, and acquiring a coordinate position of the target;
after the target coordinate position information is obtained, triggering a holder visible light camera to perform zoom stretching, and collecting a stretched target image;
performing category detection on the stretched target image to obtain category information corresponding to the target, if the target is an interested low-small slow target, performing the next operation, and if not, not paying attention to the target;
triggering a laser radar to measure the distance of the low and small slow targets of interest to obtain the distance information of the low and small slow targets;
and sending target coordinate position information obtained by detecting the event camera, category information detected by the holder visible light camera and distance information obtained by detecting the laser radar to a main control computer, calculating the running track of the low, small and slow targets by the main control computer, and performing man-machine interaction with an operator through a man-machine interaction interface.
2. The hybrid vision-based low-small-slow target recognition and positioning method according to claim 1, wherein after the target coordinate position information is obtained, a holder visible light camera is triggered to perform variable-magnification stretching, wherein a variable-magnification stretching multiple corresponding to the variable-magnification stretching satisfies the following relation:
scale＝100/max(boundbox(w,h))
wherein scale represents the variable-magnification stretching ratio, boundbox (w, h) represents the detection frame, and w and h represent the width and height of the detection frame.
3. The hybrid vision-based low-small slow target recognition and positioning method according to claim 1, wherein the event camera, the pan-tilt visible light camera and the lidar are installed on the same horizontal line.
4. The hybrid vision-based low-small slow target recognition and positioning method according to claim 1, wherein the acquiring events collected by an event camera, detecting targets in a visual field and tracking the targets comprises:
and training a pulse neural network model, and inputting an event sequence acquired by an event camera into the trained pulse neural network model to detect and track the target.
5. The hybrid vision-based low-small slow target recognition and positioning method according to claim 1, wherein the performing of the class detection on the stretched target image to obtain the class information corresponding to the target comprises:
and performing target detection on the stretched target image by using a convolutional neural network to obtain the class information corresponding to the target.
6. The low-small slow target recognition and positioning system based on the hybrid vision is characterized by comprising an event camera, a holder visible light camera, a laser radar and a main control computer, wherein:
the event camera is used for collecting events, detecting and tracking targets in a visual field range and acquiring the coordinate positions of the targets;
the holder visible light camera is used for performing zooming stretching after obtaining the coordinate position information of the target, acquiring a stretched target image, and performing category detection on the stretched target image to obtain category information corresponding to the target;
the laser radar is used for ranging the target when the target is detected to be the low-small slow target of interest to obtain the distance information of the low-small slow target;
and the main control computer is used for calculating the running track of the low, small and slow target according to the target coordinate position information obtained by the detection of the sister receiving event camera, the category information detected by the holder visible light camera and the distance information obtained by the detection of the laser radar, and performing man-machine interaction with an operator through a man-machine interaction interface.
7. The hybrid vision based low-small-slow target recognition and positioning system according to claim 6, wherein the pan-tilt visible light camera performs variable-magnification stretching, and a variable-magnification stretching ratio corresponding to the variable-magnification stretching satisfies the following relation:
scale＝100/max(boundbox(w,h))
wherein scale represents the variable-magnification stretching ratio, boundbox (w, h) represents the event camera detection frame, and w and h represent the width and height of the detection frame.
8. The hybrid vision based low-small slow target recognition and positioning system according to claim 6, wherein the event camera, the pan-tilt-visible camera and the lidar are installed on the same horizontal line.
9. The hybrid vision based low-small slow target recognition and positioning system according to claim 6, wherein the event camera detects and tracks targets in a field of view, comprising:
and training the impulse neural network model, and inputting an event sequence acquired by the event camera into the trained impulse neural network model to detect and track the target.
10. The hybrid vision-based low-small slow target recognition and positioning system according to claim 6, wherein the holder visible camera performs class detection on the stretched target image to obtain class information corresponding to a target, and the class information comprises:
and performing target detection on the stretched target image by using a convolutional neural network to obtain class information corresponding to the target.
CN202210644599.7A 2022-06-08 2022-06-08 Low, small and slow target identification and positioning method and system based on mixed vision Pending CN115035470A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210644599.7A CN115035470A (en) 2022-06-08 2022-06-08 Low, small and slow target identification and positioning method and system based on mixed vision

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210644599.7A CN115035470A (en) 2022-06-08 2022-06-08 Low, small and slow target identification and positioning method and system based on mixed vision

Publications (1)

Publication Number Publication Date
CN115035470A true CN115035470A (en) 2022-09-09

Family

ID=83123703

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210644599.7A Pending CN115035470A (en) 2022-06-08 2022-06-08 Low, small and slow target identification and positioning method and system based on mixed vision

Country Status (1)

Country Link
CN (1) CN115035470A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116958142A (en) * 2023-09-20 2023-10-27 安徽大学 Target detection and tracking method based on compound eye event imaging and high-speed turntable

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116958142A (en) * 2023-09-20 2023-10-27 安徽大学 Target detection and tracking method based on compound eye event imaging and high-speed turntable
CN116958142B (en) * 2023-09-20 2023-12-15 安徽大学 Target detection and tracking method based on compound eye event imaging and high-speed turntable

Similar Documents

Publication Publication Date Title
KR102129893B1 (en) Ship tracking method and system based on deep learning network and average movement
CN111932588B (en) Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning
CN109686109B (en) Parking lot safety monitoring management system and method based on artificial intelligence
CN109872483B (en) Intrusion alert photoelectric monitoring system and method
Rivera et al. Post-disaster rescue facility: Human detection and geolocation using aerial drones
CN106335646A (en) Interference-type anti-UAV (Unmanned Aerial Vehicle) system
CN109255286A (en) A kind of quick detection recognition method of unmanned plane optics based on YOLO deep learning network frame
CN112068111A (en) Unmanned aerial vehicle target detection method based on multi-sensor information fusion
CN114419825B (en) High-speed rail perimeter intrusion monitoring device and method based on millimeter wave radar and camera
CN105759834A (en) System and method of actively capturing low altitude small unmanned aerial vehicle
CN104378582A (en) Intelligent video analysis system and method based on PTZ video camera cruising
CN110081982B (en) Unmanned aerial vehicle target positioning method based on double-spectrum photoelectric search
CN106485245A (en) A kind of round-the-clock object real-time tracking method based on visible ray and infrared image
CN105306892B (en) A kind of generation of ship video of chain of evidence form and display methods
CN110619276B (en) Anomaly and violence detection system and method based on unmanned aerial vehicle mobile monitoring
CN103455822A (en) Inspection robot system in complex industrial environment and multi-face dynamic tracking method
CN111968046A (en) Radar photoelectric sensor target association fusion method based on topological structure
CN111830470A (en) Combined calibration method and device, and target object detection method, system and device
CN115035470A (en) Low, small and slow target identification and positioning method and system based on mixed vision
CN111466892A (en) Remote temperature measurement system and method based on thermal imaging and face recognition
CN114200442A (en) Road target detection and correlation method based on millimeter wave radar and vision
CN107045805B (en) Method and system for monitoring small aircraft and airborne objects
KR20150031530A (en) Method and apparatus for video surveillance by using surveillance apparatus of unmanned aerial vehicle
Choubisa et al. An optical-camera complement to a PIR sensor array for intrusion detection and classfication in an outdoor environment
CN115979250B (en) Positioning method based on UWB module, semantic map and visual information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination