CN114638898A - Small-sized flight target detection method and device - Google Patents
Small-sized flight target detection method and device Download PDFInfo
- Publication number
- CN114638898A CN114638898A CN202210559631.1A CN202210559631A CN114638898A CN 114638898 A CN114638898 A CN 114638898A CN 202210559631 A CN202210559631 A CN 202210559631A CN 114638898 A CN114638898 A CN 114638898A
- Authority
- CN
- China
- Prior art keywords
- image
- target
- binocular camera
- coordinate system
- plane coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The application relates to the field of target detection and positioning, and discloses a small flying target detection method and a device, wherein the method comprises the following steps: acquiring a first image and a second image containing a small flying target by using a binocular camera; inputting the first image into the pruned yolov5s network for target detection by using a target detection module of the general processor to obtain a first image plane coordinate of the target; based on the polar line constraint principle of a binocular camera, utilizing a preprocessing module of a general processor to obtain a cutting range of a second image, cutting the second image according to the cutting range, and then utilizing a target detection module to obtain a second image plane coordinate of a target; and calculating the spatial position of the target by using a target positioning module of the general processor according to the coordinates of the two image planes and the internal and external parameters of the binocular camera. Therefore, the processing data of the second image can be reduced, so that the memory occupation is reduced, the detection efficiency and the detection precision are improved, and the method is suitable for a general processor with limited computing resources.
Description
Technical Field
The invention relates to the field of target detection and positioning, in particular to a small flying target detection method and a small flying target detection device.
Background
Small flying targets are typically referred to as "low-to-small-slow" flying targets, which are flying targets that have slow flying speeds, small sizes, low flying heights, and are not easily detected by military and civilian radars. In view of the safety consideration of low-altitude airspace, high-time-efficiency and high-accuracy detection and positioning of small flying targets are needed.
Due to the high efficiency of the autonomous learning features of the convolutional neural network, the convolutional neural network becomes the mainstream research direction in the field of small target detection. The convolutional neural network optimizes a network model mainly by deepening network hierarchy so as to improve detection accuracy. However, as the network hierarchy deepens, the hardware requirements required by the training model are simultaneously increased, and the intensive computation required by the deep convolutional neural network makes it difficult to realize real-time detection and positioning in a general-purpose processor with limited resources (such as raspberry pi).
In order to solve the problems, the existing technical scheme mainly compresses the deep convolutional neural network, so that the memory occupation and the calculation power of the deep neural network model meet the requirement of low configuration, but the cost is that the detection accuracy is greatly reduced. When the detected target is a small flying target, the scene is often a large-resolution image to detect a small target, the small target is directly input into a detection network for detection, if simple down-sampling is carried out, data information is easily lost when the down-sampling multiple is too large, and the network is difficult to learn the characteristic information of the target; when the down-sampling multiple is too small, the calculation resources required by a large amount of feature maps required by network forward propagation and stored in the memory cannot be guaranteed, and further normal network training cannot be continued.
Disclosure of Invention
In view of this, the present invention provides a method and an apparatus for detecting a small flying target, which can reduce memory usage and improve detection efficiency and detection accuracy. The specific scheme is as follows:
a small flying target detection method, comprising:
acquiring an image containing a small flying target by using a binocular camera; the images include a first image acquired with a first camera and a second image acquired with a second camera;
inputting the first image into the pruned yolov5s network for target detection by using a target detection module of a general processor, and acquiring a first image plane coordinate of a target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in the image column direction;
based on the polar line constraint principle of the binocular camera, obtaining a cutting range of the second image by utilizing a preprocessing module of the general processor according to the maximum value and the minimum value of the longitudinal axis, and cutting the second image according to the cutting range to obtain a cut second image;
inputting the second cut image to the pruned yolov5s network by using the object detection module to perform object detection, and acquiring a second image plane coordinate of the object;
and calculating the space position of the target by utilizing a target positioning module of the general processor according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera.
Preferably, in the method for detecting a small flying target provided in an embodiment of the present invention, after the binocular camera is used to acquire the image containing the small flying target, the method further includes:
establishing an image coordinate system;
establishing a focal plane coordinate system according to the image coordinate system, the resolution and the pixel size of the binocular camera;
establishing an ideal image space coordinate system according to the focal plane coordinate system; the origin of the ideal spatial coordinate system is located at the projection center,xof axes and said focal plane coordinate systemxThe axes are parallel to each other and the axis is parallel,yof axes and said focal plane coordinate systemyThe axes are parallel to each other and the axis is parallel,zthe axis is the camera principal axis direction.
Preferably, in the above small flying target detection method provided by the embodiment of the present invention, before the target detection module of the general-purpose processor inputs the first image to the pruned yolov5s network for target detection, the method further includes:
preprocessing the first image by utilizing a preprocessing module of the general processor in a time-space domain fusion mode;
before inputting the cropped second image to the pruned yolov5s network for object detection by using an object detection module of the general processor, the method further comprises the following steps:
and utilizing a preprocessing module of the general processor to carry out preprocessing of time-space domain fusion on the second cut image.
Preferably, in the method for detecting a small flying target according to an embodiment of the present invention, the preprocessing of performing time-space domain fusion on the first image includes:
performing optimization processing on the first image by adopting Gaussian filtering on a spatial domain to obtain a spatial domain result diagram;
performing clutter suppression processing on the first image by adopting a sequence frame difference method on a time domain to obtain a time domain result graph;
and fusing and outputting the spatial domain result graph and the time domain result graph to obtain the preprocessed first image.
Preferably, in the above method for detecting a small flying target provided by the embodiment of the present invention, before the obtaining an image containing the small flying target by using a binocular camera, the method further includes:
and calibrating the binocular camera to obtain the internal and external parameters of the binocular camera.
Preferably, in the above small flying target detection method provided in the embodiment of the present invention, calculating a spatial position of a target according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera includes:
obtaining the depth information of the target by the second image plane coordinate and the focal length parameter of the binocular camera;
and performing coordinate conversion on the first image plane coordinate of the target, and calculating the spatial position of the target under a binocular camera coordinate system by combining the obtained depth information.
Preferably, in the method for detecting a small flying target provided in an embodiment of the present invention, coordinate conversion is performed on the first image plane coordinate of the target, and the obtained depth information is combined to calculate a spatial position of the target in a binocular camera coordinate system, where the method includes:
calculating focal plane coordinates of the target in the focal plane coordinate system according to the first image plane coordinates of the target;
according to the calculated focal plane coordinates, calculating ideal image space coordinates of the target under the ideal image space coordinate system;
calculating a target view vector of the binocular camera according to the calculated ideal image space coordinate and the obtained depth information;
and obtaining the space position of the target under the coordinate system of the binocular camera according to the target view vector of the binocular camera.
The embodiment of the invention also provides a small flying target detection device, which comprises: a binocular camera and a general purpose processor; the general processor comprises a preprocessing module, a target detection module and a target positioning module; wherein the content of the first and second substances,
the binocular camera is used for acquiring an image containing a small flying target; the images include a first image acquired with a first camera and a second image acquired with a second camera;
the target detection module is used for inputting the first image into the pruned yolov5s network for target detection, and acquiring a first image plane coordinate of the target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in the image column direction;
the preprocessing module is used for obtaining a cutting range of the second image according to the maximum value and the minimum value of the longitudinal axis based on the polar line constraint principle of the binocular camera, and cutting the second image according to the cutting range to obtain a cut second image;
the target detection module is further configured to input the cut second image to the pruned yolov5s network for target detection, and obtain a second image plane coordinate of the target;
the target positioning module is used for calculating the space position of the target according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera.
Preferably, in the above small flying-object detecting device provided in the embodiment of the present invention, the general processor further includes a coordinate system establishing module;
the coordinate system establishing module is used for establishing an image coordinate system; establishing a focal plane coordinate system according to the image coordinate system, the resolution and the pixel size of the binocular camera; establishing an ideal image space coordinate system according to the focal plane coordinate system; said ideal spatial coordinate systemxShaft and theOf focal plane coordinate systemxThe axes are parallel to each other and the axis is parallel,yof axes and said focal plane coordinate systemyThe axes are parallel to each other and the axis is parallel,zthe axis is the main axis direction of the camera.
Preferably, in the above small flying-target detecting device provided in the embodiment of the present invention, the general processor further includes a parameter obtaining module;
the parameter acquisition module is used for calibrating the binocular camera and acquiring the internal and external parameters of the binocular camera.
According to the technical scheme, the small flying target detection method provided by the invention comprises the following steps: acquiring an image containing a small flying target by using a binocular camera; the images include a first image acquired with a first camera and a second image acquired with a second camera; inputting the first image into a pruned yolov5s network for target detection by using a target detection module of a general processor, and acquiring a first image plane coordinate of the target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in the image column direction; based on the polar line constraint principle of a binocular camera, obtaining a cutting range of a second image by utilizing a preprocessing module of a general processor according to the maximum value and the minimum value of a longitudinal axis, and cutting the second image according to the cutting range to obtain a cut second image; inputting the second cut image into the pruned yolov5s network by using an object detection module to perform object detection, and acquiring a second image plane coordinate of the object; and calculating the space position of the target by using a target positioning module of the general processor according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera.
According to the small flying target detection method provided by the invention, the first image and the second image of the small flying target are obtained by using the binocular camera, then the second image is cut according to the detection result of the first image based on the polar constraint principle of the binocular camera, the processing data of the second image is greatly reduced, so that the memory occupation is reduced, the detection efficiency is improved, the target detection is carried out by matching with the pruned yolov5s network, the method is suitable for a general processor with limited computing resources, and the detection of the small flying target can be ensured to have higher accuracy. In addition, the invention also provides a corresponding device for the small flying target detection method, so that the method has higher practicability and the device has corresponding advantages.
Drawings
In order to more clearly illustrate the embodiments of the present invention or technical solutions in related arts, the drawings used in the description of the embodiments or related arts will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a method for detecting a small flying target according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a coordinate system provided in an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a small flying-target detecting device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The invention provides a small flying target detection method, as shown in figure 1, comprising the following steps:
s101, acquiring an image containing a small flying target by using a binocular camera; the images include a first image acquired with a first camera and a second image acquired with a second camera;
in practical application, the binocular camera is formed by back-to-back combination of two fisheye cameras with the field angle exceeding 180 degrees, so that all the scenes in the azimuth angle of 360 degrees can be acquired by one-time shooting. The size of the pictures at the input end of the detection network can be reduced in the invention, for example, the size of the pictures obtained from the binocular camera can be 1280 × 720 × 2 (length × width).
S102, inputting the first image into a pruned yolov5S network for target detection by using a target detection module of a general processor, and acquiring a first image plane coordinate of the target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in an image column direction;
in practical application, the general processor can be a raspberry pie, namely, the binocular camera in the small flight target detection method is responsible for observing and imaging, and the raspberry pie can bear the functions of the processor.
It should be noted that the detection network used in the present invention is an improved yolov5s network, and the present invention prunes the model of the original yolov5s network, prunes the network structure and weight, thereby reducing the network model scale and speeding up the network processing. The specific operation can be that L1 norm regularization is used for constraining the Batch Normalization (BN layer) coefficient in the network, the BN layer coefficient is thinned, then the layer with small sparseness is cut, the process is iterated repeatedly, and finally the obtained network model with smaller volume and small loss of detection effect is obtained.
S103, based on the polar line constraint principle of the binocular camera, obtaining a cutting range of the second image by utilizing a preprocessing module of the general processor according to the maximum value and the minimum value of the longitudinal axis, and cutting the second image according to the cutting range to obtain a cut second image;
the epipolar constraint refers to the projective relation of each pixel in the two images, and is related to the internal parameters of the camera and the shooting positions of the two images; the camera internal reference and relative position can be obtained by calibration.
In the specific implementation, the epipolar constraint principle of a binocular camera is utilized according to the maximum value of a longitudinal axis
And the minimum value of the vertical axis
Obtaining a cropping range of the second image as
(ii) a Cutting the second image according to the cutting range to obtain a cut second image; cropping the second image to a size of
Wherein, in the step (A),
lateral resolution values for images taken with a binocular camera.
It should be noted that, the epipolar constraint between the left and right images of the binocular camera is used to obtain the range of the target in the other image, so that the redundant information in the input characteristic diagram of the detection model data is reduced while the precision loss is very small, the data volume of the network input end is greatly compressed, and the speed is effectively increased. Therefore, the requirements on hardware memory and computing power are reduced on the premise of ensuring the detection effect, and the deployment of a deep learning detection network on the microprocessor is realized.
S104, inputting the cut second image to a pruned yolov5S network for target detection by using a target detection module, and acquiring a second image plane coordinate of the target;
and S105, calculating the space position of the target by using a target positioning module of the general processor according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera.
In the method for detecting the small flying target provided by the embodiment of the invention, the first image and the second image of the small flying target are obtained by using the binocular camera, then the second image is cut according to the detection result of the first image based on the polar line constraint principle of the binocular camera, the processing data of the second image is greatly reduced, the memory occupation is reduced, the detection efficiency is improved, the target detection is carried out by matching with the pruned yolov5s network, the method is suitable for a general processor with limited computing resources, and the detection of the small flying target can be ensured to have higher accuracy.
Further, when embodied, provided in embodiments of the present inventionIn the above method for detecting a small flying target, in order to achieve the purpose of detecting and positioning the target, an accurate coordinate system needs to be established, and after the step S101 is executed and the image containing the small flying target is acquired by using the binocular camera, the method may further include: establishing an image coordinate system at the center position of an upper left corner pixel of the image; the image plane coordinates of the target are coordinates in an image coordinate system; establishing a focal plane coordinate system according to the image coordinate system, the resolution ratio of the binocular camera and the pixel size; establishing an ideal image space coordinate system according to the focal plane coordinate system; it is assumed that the origin of the spatial coordinate system is located at the projection center,xof axis and focal plane coordinate systemsxThe axes are parallel to each other and the axis is parallel,yof axis and focal plane coordinate systemsyThe axes are parallel to each other, and the axial direction is parallel to each other,zthe axis is the camera principal axis direction.
Specifically, as shown in FIG. 2, the image coordinate system
Origin pointO 1At the center of the top left pixel of the image, downwards along the image column directionjCoordinate axes, to the right in the image directioniAnd coordinate axes. Focal plane coordinate system
Realizing the conversion from pixel coordinates to physical coordinates in an image coordinate system, and obtaining the origin of a focal plane coordinate systemO 2Corresponding to the physical location (derived from a binocular camera resolution of 1280 × 720) at the pels of the image coordinate system (640, 360) in millimeters, the image coordinate system is transformed to the focal plane coordinate system as follows:
wherein the content of the first and second substances,dis the pixel size in millimeters. Ideal image space coordinate system
Origin of (2)
At the center of the projection. The principal point has coordinates in the ideal spatial coordinate system of
. The conversion formula from the focal plane coordinate system to the ideal image space coordinate system is as follows:
here, the ideal image space coordinate system is transferred to the binocular camera. The invention sets the coordinate system of the binocular camera as the world coordinate system, takes the origin of the space coordinate system of the left camera as the center of the lens of the left camera,Zthe axis is parallel to the main axis of the camera,Xthe axis points to the center of the right camera lens andZthe axis is vertical to the axis of the device,Ythe axis meets the right-hand criterion, pointing vertically to the ground. The right camera space coordinate system in the binocular camera coordinate system is defined according to the same rule, namely the origin of the right camera space coordinate system is the center of the right camera lens,Zthe axis is parallel to the optical axis of the camera,Xthe axis points to the center of the left camera lens andZthe axis is vertical to the axis of the device,Ythe axis meets the right-hand criterion, pointing vertically to the ground.
It should be noted that the target image plane coordinates of the present invention refer to coordinates in the image coordinate system, and the spatial coordinates refer to coordinates in the binocular camera coordinate system.
In practical application, the preprocessing module of the general processor can be used for performing rapid threshold processing on the first image and the second image to screen out a plurality of images which possibly contain a target area; then, the targets can be marked, namely, a boundary frame and category information of each target to be detected appearing in each image are marked; data enhancement operation is carried out on the existing images, so that the trained network has a better effect; unifying and normalizing the image size; optimizing the image based on the number of positive and negative samples, dividing to obtain a training image set and a testing image set, training the yolov5s network by using the training image set, testing by using the testing set, and debugging parameters according to the testing result to obtain an optimal model.
In addition, in a specific implementation, in the above small flying target detection method provided by the embodiment of the present invention, before performing step S102, inputting the first image to the pruned yolov5S network for target detection by using the target detection module of the general-purpose processor, the method may further include: and utilizing a preprocessing module of the general processor to perform preprocessing of time-space domain fusion on the first image.
Similarly, before the step S104 is executed to input the cropped second image to the pruned yolov5S network for object detection by using the object detection module of the general-purpose processor, the method may further include: and utilizing a preprocessing module of the general processor to perform preprocessing of time-space domain fusion on the second cut image.
Specifically, the preprocessing of performing time-space domain fusion on the first image may specifically include: performing optimization processing on the first image by adopting Gaussian filtering on a spatial domain to obtain a spatial domain result graph; performing clutter suppression processing on the first image by adopting a sequence frame difference method on a time domain to obtain a time domain result graph; and fusing and outputting the spatial domain result graph and the time domain result graph to obtain a preprocessed first image.
Similarly, the preprocessing of the time-space domain fusion is performed on the second image to be cut, and specifically may include: optimizing the second cut image by adopting Gaussian filtering in a spatial domain to obtain a spatial domain result graph; performing clutter suppression processing on the cut second image by adopting a sequence frame difference method on a time domain to obtain a time domain result graph; and fusing and outputting the spatial domain result graph and the time domain result graph to obtain a preprocessed cutting second image.
It should be noted that, the present invention realizes the large compression of the data amount of the network input end under the minimum precision loss, ensures the network feature extraction and feature fusion effect, reduces the requirements on the hardware memory and the computing power on the premise of ensuring the detection effect, realizes the deployment of the deep learning detection network on the microprocessor, and ensures the speed and the precision of the detection and the positioning. Therefore, the problems that the detection and the positioning of the image target based on the deep neural network are difficult to realize on the universal micro platform and the detection accuracy and the instantaneity of the image target detection method based on the lightweight network on the universal micro platform are low are overcome.
In specific implementation, in the above method for detecting a small flying target provided by the embodiment of the present invention, before the step S101 of acquiring an image containing the small flying target by using a binocular camera, the method may further include: and calibrating the binocular camera to obtain the internal and external parameters of the binocular camera. Specifically, the calibration process of the binocular camera may include: the method comprises the steps of firstly performing monocular calibration on a first camera and a second camera (namely a left camera and a right camera), acquiring an internal reference matrix and a distortion parameter vector, then calibrating a binocular camera, acquiring a center distance, and finally performing stereo correction on the binocular camera.
In specific implementation, in the method for detecting a small flying target provided in the embodiment of the present invention, the step S103 calculates a spatial position of the target according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera, and specifically may include: obtaining depth information of the target according to the first image plane coordinate and the second image plane coordinate of the target and the focal length parameter of the binocular camera; and carrying out coordinate conversion on the first image plane coordinate of the target, and calculating the space position of the target under a binocular camera coordinate system by combining the obtained depth information.
Specifically, the depth information of the target may be calculated using the following formula:
wherein the content of the first and second substances,
is a parameter of the focal length of the lens,
is the distance between the two cameras and is,
is the x-axis coordinate in the first planar coordinate,
is the x-axis coordinate in the second planar coordinate,
representing an absolute value operation.
Specifically, in the implementation, coordinate conversion is performed on a first image plane coordinate of the target, and the obtained depth information is combined to calculate a spatial position of the target in a binocular camera coordinate system, including: calculating focal plane coordinates of the target in a focal plane coordinate system according to the first image plane coordinates of the target; calculating ideal image space coordinates of the target under an ideal image space coordinate system according to the calculated focal plane coordinates; calculating a target view vector of the binocular camera according to the calculated space coordinates of the ideal image and the obtained depth information; and calculating the space position of the target under the coordinate system of the binocular camera according to the target view vector of the binocular camera.
The method for detecting the small flying target provided by the embodiment of the invention is exemplified by a specific example, and the specific steps are as follows:
firstly, calibrating by a binocular camera to obtain internal and external parameters including a focal length, a distance between the two cameras, distortion parameters and the like.
And step two, acquiring an image containing the small aircraft target by adopting a binocular camera, and transmitting the acquired image to a raspberry group for processing.
And thirdly, preprocessing one 1280 x 720 image (taking the first image as an example, the sequence of the two images can be exchanged) acquired by the binocular camera through an image preprocessing module of a raspberry group, namely preprocessing time-space domain fusion is performed on the first image, the detected image is optimized by adopting Gaussian filtering in a space domain and is combined with gradient characteristics, the purpose of improving the local contrast is achieved, clutter is inhibited by using a sequence frame difference method in a time domain, and the acquired space domain result image and the time domain result image are fused and output to obtain the preprocessed first image.
Step four, target detection through raspberry groupThe detection module inputs the preprocessed first image into a pruned yolov5s network for target detection, and obtains a first image plane coordinate of the target and a maximum value of a longitudinal axis of the target in the direction of an image column
And the minimum value of the vertical axis
。
Step five, obtaining the cutting range of the second image by utilizing a preprocessing module of the raspberry pi according to the maximum value of the longitudinal axis and the minimum value of the longitudinal axis, and only keeping the size of 1280 x
Size, target within this range. And cutting the second image according to the cutting range to obtain a cut second image. And then, preprocessing the second image to obtain a preprocessed second image.
And step six, inputting the preprocessed cut second image into the pruned yolov5s network by using a raspberry target detection module to perform target detection, and acquiring a second image plane coordinate of the target.
And seventhly, performing coordinate conversion by using a target positioning module of the raspberry pi according to the first image plane coordinate, the second image plane coordinate and the imaging parameter. Specifically, depth information of the target is obtained according to a first image plane coordinate and a second image plane coordinate of the target and a focal length parameter of a binocular camera; calculating focal plane coordinates according to the first image plane coordinates of the target; calculating ideal image space coordinates according to focal plane coordinates; calculating coordinates (namely target view vectors) of a binocular camera coordinate system according to the space coordinates of the ideal image and the obtained depth information; and obtaining the position of the calculated target under the binocular camera coordinate system according to the target view vectors of the left camera and the right camera. Thus, the automatic detection and positioning of the small flying target on the raspberry is completed.
Based on the same inventive concept, the embodiment of the invention also provides a small flying target detection device, and as the principle of solving the problems of the device is similar to that of the small flying target detection method, the implementation of the device can refer to the implementation of the small flying target detection method, and repeated parts are not described again.
In specific implementation, the small flying target detection device provided by the embodiment of the present invention, as shown in fig. 3, specifically includes: a binocular camera 11 and a general processor 12; the general purpose processor 12 includes an object detection module 121, a pre-processing module 122, and an object location module 123; wherein the content of the first and second substances,
a binocular camera 11 for acquiring an image containing a small flying target; the images include a first image acquired with a first camera and a second image acquired with a second camera;
the target detection module 121 is configured to perform target detection on the yolov5s network obtained by inputting the first image into pruning, and obtain a first image plane coordinate of the target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in the image column direction;
the preprocessing module 122 is configured to obtain a clipping range of the second image according to the maximum value and the minimum value of the longitudinal axis based on an epipolar constraint principle of the binocular camera, and clip the second image according to the clipping range to obtain a clipped second image;
the target detection module 121 is further configured to input the cut second image to the pruned yolov5s network for target detection, and obtain a second image plane coordinate of the target;
and the target positioning module 123 is configured to calculate a spatial position of the target according to the first image plane coordinate and the second image plane coordinate of the target, and the internal and external parameters of the binocular camera.
In the small flying target detection device provided by the embodiment of the invention, the images are acquired by using the binocular camera, the detection and the positioning of the target are carried out by combining the general processor with limited computing resources, the second image is cut according to the detection result of the first image based on the epipolar constraint principle of the binocular camera during the image processing, the processing data of the second image can be greatly reduced, the memory occupation is reduced, the detection efficiency is improved, the target detection is carried out by matching with the pruned yolov5s network, and the detection of the small flying target can be ensured to have higher accuracy.
In a specific implementation, in the above small flying-target detecting device provided in the embodiment of the present invention, the general processor 12 may further include a coordinate system establishing module;
the coordinate system establishing module is used for establishing an image coordinate system; establishing a focal plane coordinate system according to the image coordinate system, the resolution ratio of the binocular camera and the pixel size; establishing an ideal image space coordinate system according to the focal plane coordinate system; imagining a spatial coordinate systemxOf axes and focal plane coordinate systemsxThe axes are parallel to each other and the axis is parallel,yof axis and focal plane coordinate systemsyThe axes are parallel to each other, and the axial direction is parallel to each other,zthe axis is the main axis direction of the camera.
In specific implementation, in the small flying object detecting device provided in the embodiment of the present invention, the general processor 12 may further include a parameter obtaining module;
and the parameter acquisition module is used for calibrating the binocular camera and acquiring the internal and external parameters of the binocular camera.
For more specific working processes of the above modules, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not described herein again.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
To sum up, the method for detecting a small flying target provided by the embodiment of the invention comprises the following steps: acquiring an image containing a small flying target by using a binocular camera; the images include a first image acquired with a first camera and a second image acquired with a second camera; inputting the first image into a pruned yolov5s network for target detection by using a target detection module of a general processor to obtain a first image plane coordinate of a target; and a vertical axis maximum and a vertical axis minimum of the object in the image column direction; based on the polar line constraint principle of the binocular camera, obtaining a cutting range of the second image by utilizing a preprocessing module of the general processor according to the maximum value and the minimum value of the longitudinal axis, and cutting the second image according to the cutting range to obtain a cut second image; inputting the second cut image into the pruned yolov5s network by using an object detection module to perform object detection, and acquiring a second image plane coordinate of the object; and calculating the space position of the target by using a target positioning module of the general processor according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera. According to the small flying target detection method, the first image and the second image of the small flying target are obtained by the binocular camera, then the second image is cut according to the detection result of the first image based on the polar line constraint principle of the binocular camera, the processing data of the second image are greatly reduced, the memory occupation is reduced, the detection efficiency is improved, the yolov5s network after pruning is matched for target detection, the method is suitable for a general processor with limited computing resources, and the detection of the small flying target can be guaranteed to have higher accuracy. In addition, the invention also provides a corresponding device for the small flying target detection method, so that the method has higher practicability and the device has corresponding advantages.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The method and the device for detecting the small flying target provided by the invention are described in detail, a specific example is applied in the text to explain the principle and the implementation mode of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method for detecting a small flying target, comprising:
acquiring an image containing a small flying target by using a binocular camera; the images include a first image acquired with a first camera and a second image acquired with a second camera;
inputting the first image into a pruned yolov5s network for target detection by using a target detection module of a general processor, and acquiring a first image plane coordinate of a target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in an image column direction;
based on the polar line constraint principle of the binocular camera, obtaining a cutting range of the second image by utilizing a preprocessing module of the general processor according to the maximum value and the minimum value of the longitudinal axis, and cutting the second image according to the cutting range to obtain a cut second image;
inputting the second cut image to the pruned yolov5s network by using the target detection module for target detection to obtain a second image plane coordinate of the target;
and calculating the space position of the target by utilizing a target positioning module of the general processor according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera.
2. The small flying target detection method according to claim 1, further comprising, after acquiring the image containing the small flying target with a binocular camera:
establishing an image coordinate system;
establishing a focal plane coordinate system according to the image coordinate system, the resolution and the pixel size of the binocular camera;
establishing an ideal image space coordinate system according to the focal plane coordinate system; the origin of the ideal image space coordinate system is located at the projection center, the x axis is parallel to the x axis of the focal plane coordinate system, the y axis is parallel to the y axis of the focal plane coordinate system, and the z axis is the direction of the main axis of the camera.
3. The small flying object detecting method according to claim 2, wherein before inputting the first image to the pruned yolov5s network for object detection by using an object detecting module of a general-purpose processor, the method further comprises:
preprocessing the first image by utilizing a preprocessing module of the general processor in a time-space domain fusion mode;
before inputting the second cropped image to the cropped yolov5s network for object detection by using an object detection module of the general-purpose processor, the method further comprises the following steps:
and utilizing a preprocessing module of the general processor to carry out preprocessing of time-space domain fusion on the second cut image.
4. The small flying target detection method of claim 3, wherein the preprocessing of the time-space domain fusion of the first image comprises:
performing optimization processing on the first image by adopting Gaussian filtering on a spatial domain to obtain a spatial domain result graph;
performing clutter suppression processing on the first image by adopting a sequence frame difference method on a time domain to obtain a time domain result graph;
and fusing and outputting the spatial domain result graph and the time domain result graph to obtain the preprocessed first image.
5. The small flying target detecting method according to claim 4, wherein before the images containing the small flying target are acquired by the binocular camera, the method further comprises:
and calibrating the binocular camera to obtain the internal and external parameters of the binocular camera.
6. The small flying target detecting method according to claim 5, wherein calculating the spatial position of the target according to the first and second image plane coordinates of the target and the internal and external parameters of the binocular camera comprises:
obtaining depth information of the target according to the first image plane coordinate and the second image plane coordinate of the target and the focal length parameter of the binocular camera;
and performing coordinate conversion on the first image plane coordinate of the target, and calculating the spatial position of the target under a binocular camera coordinate system by combining the obtained depth information.
7. The small flying target detecting method according to claim 6, wherein the coordinate transformation of the first image plane coordinate of the target is performed, and the spatial position of the target under the binocular camera coordinate system is calculated by combining the obtained depth information, and the method comprises:
calculating focal plane coordinates of the target in the focal plane coordinate system according to the first image plane coordinates of the target;
according to the calculated focal plane coordinates, calculating ideal image space coordinates of the target in the ideal image space coordinate system;
calculating a target view vector of the binocular camera according to the calculated ideal image space coordinate and the obtained depth information;
and obtaining the space position of the target under the coordinate system of the binocular camera according to the target view vector of the binocular camera.
8. A small flying target detection device, comprising: a binocular camera and a general purpose processor; the general processor comprises a preprocessing module, a target detection module and a target positioning module; wherein the content of the first and second substances,
the binocular camera is used for acquiring an image containing a small flying target; the images include a first image acquired with a first camera and a second image acquired with a second camera;
the target detection module is used for inputting the first image into the pruned yolov5s network for target detection, and acquiring a first image plane coordinate of the target, and a longitudinal axis maximum value and a longitudinal axis minimum value of the target in the image column direction;
the preprocessing module is used for obtaining a cutting range of the second image according to the maximum value and the minimum value of the longitudinal axis based on the polar line constraint principle of the binocular camera, and cutting the second image according to the cutting range to obtain a cut second image;
the target detection module is further configured to input the cut second image to the pruned yolov5s network for target detection, and obtain a second image plane coordinate of the target;
the target positioning module is used for calculating the space position of the target according to the first image plane coordinate and the second image plane coordinate of the target and the internal and external parameters of the binocular camera.
9. The small flying object detecting device of claim 8, wherein the general purpose processor further comprises a coordinate system establishing module;
the coordinate system establishing module is used for establishing an image coordinate system; establishing a focal plane coordinate system according to the image coordinate system, the resolution and the pixel size of the binocular camera; establishing an ideal image space coordinate system according to the focal plane coordinate system; the x axis of the ideal image space coordinate system is parallel to the x axis of the focal plane coordinate system, the y axis is parallel to the y axis of the focal plane coordinate system, and the z axis is the direction of the main axis of the camera.
10. The small flying object detecting device according to claim 9, wherein the general purpose processor further comprises a parameter acquisition module;
the parameter acquisition module is used for calibrating the binocular camera and acquiring the internal and external parameters of the binocular camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210559631.1A CN114638898A (en) | 2022-05-23 | 2022-05-23 | Small-sized flight target detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210559631.1A CN114638898A (en) | 2022-05-23 | 2022-05-23 | Small-sized flight target detection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114638898A true CN114638898A (en) | 2022-06-17 |
Family
ID=81952977
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210559631.1A Pending CN114638898A (en) | 2022-05-23 | 2022-05-23 | Small-sized flight target detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114638898A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116883945A (en) * | 2023-07-21 | 2023-10-13 | 江苏省特种设备安全监督检验研究院 | Personnel identification positioning method integrating target edge detection and scale invariant feature transformation |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107506711A (en) * | 2017-08-15 | 2017-12-22 | 江苏科技大学 | Binocular vision obstacle detection system and method based on convolutional neural networks |
| CN107680139A (en) * | 2017-10-17 | 2018-02-09 | 中国人民解放军国防科技大学 | Universality calibration method of telecentric binocular stereo vision measurement system |
| CN111666792A (en) * | 2019-03-07 | 2020-09-15 | 阿里巴巴集团控股有限公司 | Image recognition method, image acquisition and recognition method and commodity recognition method |
| CN112529960A (en) * | 2020-12-17 | 2021-03-19 | 珠海格力智能装备有限公司 | Target object positioning method and device, processor and electronic device |
-
2022
- 2022-05-23 CN CN202210559631.1A patent/CN114638898A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107506711A (en) * | 2017-08-15 | 2017-12-22 | 江苏科技大学 | Binocular vision obstacle detection system and method based on convolutional neural networks |
| CN107680139A (en) * | 2017-10-17 | 2018-02-09 | 中国人民解放军国防科技大学 | Universality calibration method of telecentric binocular stereo vision measurement system |
| CN111666792A (en) * | 2019-03-07 | 2020-09-15 | 阿里巴巴集团控股有限公司 | Image recognition method, image acquisition and recognition method and commodity recognition method |
| CN112529960A (en) * | 2020-12-17 | 2021-03-19 | 珠海格力智能装备有限公司 | Target object positioning method and device, processor and electronic device |
Non-Patent Citations (2)
| Title |
|---|
| 明悦: "《多源视觉信息感知与识别》", 31 August 2020 * |
| 陈志军: "《智能网联环境下车辆运动行为理解方法》", 31 December 2020 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116883945A (en) * | 2023-07-21 | 2023-10-13 | 江苏省特种设备安全监督检验研究院 | Personnel identification positioning method integrating target edge detection and scale invariant feature transformation |
| CN116883945B (en) * | 2023-07-21 | 2024-02-06 | 江苏省特种设备安全监督检验研究院 | Personnel identification positioning method integrating target edge detection and scale invariant feature transformation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111563923B (en) | Method for obtaining dense depth map and related device | |
| CN109685842B (en) | Sparse depth densification method based on multi-scale network | |
| CN112444242B (en) | Pose optimization method and device | |
| CN106529495A (en) | Obstacle detection method of aircraft and device | |
| CN106529538A (en) | Method and device for positioning aircraft | |
| CN111862180B (en) | Camera set pose acquisition method and device, storage medium and electronic equipment | |
| CN111829532B (en) | Aircraft repositioning system and method | |
| CN110889873A (en) | Target positioning method and device, electronic equipment and storage medium | |
| CN108764080B (en) | Unmanned aerial vehicle visual obstacle avoidance method based on point cloud space binarization | |
| CN115410167A (en) | Target detection and semantic segmentation method, device, equipment and storage medium | |
| CN113361365A (en) | Positioning method and device, equipment and storage medium | |
| US8509522B2 (en) | Camera translation using rotation from device | |
| CN111383264B (en) | Positioning method, positioning device, terminal and computer storage medium | |
| CN114638898A (en) | Small-sized flight target detection method and device | |
| Sahin | Comparison and calibration of mobile phone fisheye lens and regular fisheye lens via equidistant model | |
| CN113808202A (en) | Multi-target detection and space positioning method and system thereof | |
| CN113706391B (en) | Real-time splicing method, system, equipment and storage medium for aerial images of unmanned aerial vehicle | |
| US20230053952A1 (en) | Method and apparatus for evaluating motion state of traffic tool, device, and medium | |
| CN113312435A (en) | High-precision map updating method and device | |
| CN113129422A (en) | Three-dimensional model construction method and device, storage medium and computer equipment | |
| CN113239936B (en) | Unmanned aerial vehicle visual navigation method based on deep learning and feature point extraction | |
| CN116246119A (en) | 3D target detection method, electronic device and storage medium | |
| CN114387532A (en) | Boundary identification method and device, terminal, electronic equipment and unmanned equipment | |
| CN113223163A (en) | Point cloud map construction method and device, equipment and storage medium | |
| CN113554754A (en) | Indoor positioning method based on computer vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220617 |