CN117115252A - Bionic ornithopter space pose estimation method based on vision - Google Patents
Bionic ornithopter space pose estimation method based on vision Download PDFInfo
- Publication number
- CN117115252A CN117115252A CN202310990249.0A CN202310990249A CN117115252A CN 117115252 A CN117115252 A CN 117115252A CN 202310990249 A CN202310990249 A CN 202310990249A CN 117115252 A CN117115252 A CN 117115252A
- Authority
- CN
- China
- Prior art keywords
- image
- bionic ornithopter
- ornithopter
- bionic
- wing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000011664 nicotinic acid Substances 0.000 title claims abstract description 122
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000001514 detection method Methods 0.000 claims abstract description 60
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 52
- 238000012549 training Methods 0.000 claims abstract description 34
- 238000013135 deep learning Methods 0.000 claims abstract description 28
- 238000012360 testing method Methods 0.000 claims abstract description 16
- 238000002372 labelling Methods 0.000 claims abstract description 11
- 238000004364 calculation method Methods 0.000 claims abstract description 8
- 239000011159 matrix material Substances 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 7
- 238000000605 extraction Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000000354 decomposition reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T90/00—Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation
Abstract
The invention discloses a bionic ornithopter space pose estimation method based on vision, which comprises the following steps: acquiring an image training data set; training the image training data set and the labeling information by using a deep learning detection algorithm; acquiring internal and external parameter information of an image acquisition device, and acquiring a flight image of the bionic ornithopter; according to a deep learning detection algorithm, testing the flight image of the bionic ornithopter to obtain a key point image coordinate system test result and an airfoil state of the bionic ornithopter; measuring three-dimensional coordinates of key points of a coordinate system of the bionic ornithopter in different wing-shaped states and wing-shaped states; and obtaining the spatial position and posture result of the bionic ornithopter through calculation. According to the method, the aerofoil state labeling definition is carried out according to different flight attitudes, the solution is carried out after the deep learning training detection, the spatial position and the attitude are obtained, and the problems that the space pose of the unmanned aerial vehicle is difficult to solve through an external image due to the complex structure and various aerofoil states of the unmanned aerial vehicle are solved.
Description
Technical Field
The invention relates to the technical field of unmanned aerial vehicle space pose estimation, in particular to a bionic ornithopter space pose estimation method based on vision.
Background
The detection, tracking and pose estimation based on vision are important means of the anti-unmanned aerial vehicle, a plurality of target images are obtained through monocular RGB sensors, key points on the target unmanned aerial vehicle are obtained through extraction of an image detection technology, and then the position and the pose of the target are resolved through the geometric relationship. However, in practical application, due to the fact that trees, telegraph poles, buildings and the like exist in a low-spatial area of a city, the background is complex, foreign matter interference is large, and the problems of detection omission, false detection, low pose estimation precision and the like often occur by adopting results obtained by a traditional image detection and pose estimation method.
The prior art discloses a monocular vision unmanned aerial vehicle pose estimation method, which uses a key point detection and positioning network to acquire 2D coordinates of a target, and calculates the pose of the target unmanned aerial vehicle according to 3D key points and internal parameters of an image acquisition device. The pose estimation method of the unmanned aerial vehicle has the defects that most of the targets are fixed-wing and rotor wing-like approximately rigid-body aircrafts, key points of the bionic unmanned aerial vehicle in the flying process are similar to bird wings, the unmanned aerial vehicle has the characteristic of deformability, the error of pose estimation can be greatly increased due to the difference caused by deformation, and the requirement on the pose estimation of the bionic ornithopter is difficult to meet.
Disclosure of Invention
The invention aims to solve the problems of the prior art, and provides a visual-based bionic ornithopter space pose estimation method, which can effectively solve the problems that the space position and pose information sources of the existing bionic ornithopter are concentrated in an inertial navigation sensor, and the space position and pose estimation is carried out on the existing bionic ornithopter by lacking external information, so that the accuracy and the comprehensiveness of monitoring the state of the bionic ornithopter are improved.
In order to achieve the above purpose of the present invention, the following technical scheme is adopted:
a bionic ornithopter space pose estimation method based on vision comprises the following steps:
labeling the obtained bionic ornithopter unmanned aerial vehicle image, wherein labeling contents comprise image coordinate key points and wing profile state information of the bionic ornithopter unmanned aerial vehicle, and obtaining a bionic ornithopter unmanned aerial vehicle image training dataset;
training the bionic ornithopter image training dataset and the labeling information by using a deep learning detection algorithm to obtain training weights, updating the deep learning detection algorithm, and obtaining a trained deep learning detection algorithm;
performing inside and outside parameter calibration on the image acquisition device to obtain inside and outside parameter information of the image acquisition device, and acquiring a flight image of the bionic ornithopter through the image acquisition device;
according to the trained deep learning detection algorithm, testing the flight image of the bionic ornithopter to obtain a key point image coordinate system test result and the wing shape state of the bionic ornithopter;
according to the wing section state classification, measuring three-dimensional coordinate information of key points of a body coordinate system of the bionic ornithopter in different wing section states, and recording the wing section states;
according to the internal and external parameter information of the image acquisition device, the testing result of the key point image coordinate system, the wing section state of the bionic ornithopter, and the three-dimensional coordinate information of key points of the body coordinate system of the bionic ornithopter under different wing section states; and (3) calculating by using a spatial position and posture estimation algorithm to obtain the spatial position and posture result of the bionic ornithopter.
Preferably, the image coordinate key points of the bionic ornithopter are respectively a nose of the bionic ornithopter, a wing tip of a left wing, a wing tip of a right wing, a wing tip of a left tail wing and a wing tip of a right tail wing;
the airfoil states are divided into three categories, respectively: a first airfoil state when the wing is above the fuselage level, a second airfoil state when the wing is below the fuselage level, and a third airfoil state when the wing is below the fuselage level.
Further, the bionic ornithopter image is marked with image coordinate key points and wing states, and the method specifically comprises the following steps:
firstly, calibrating the positions of the image coordinate key points by using a rectangular frame, and then calculating the center coordinates of the rectangular frame to obtain the two-dimensional image coordinates of the image coordinate key points; wherein the position of the rectangular box is represented using the upper left corner xy coordinates and the lower right corner xy coordinates;
the wing section states are marked according to the flight attitude of the bionic ornithopter in the bionic ornithopter image and are divided into three types;
the final calibration data of each image are 16 groups of data including 5 image coordinate key points, two-dimensional image coordinates corresponding to the 5 image coordinate key points and 1 airfoil state;
and after the calibration is completed, obtaining an image training data set of the bionic ornithopter.
Preferably, the deep learning detection algorithm adopts a yolov5 target detection algorithm, and two output channels are added on the basis of the yolov5 target detection algorithm to perform feature training on the key points of the image coordinates and the wing profile state.
Further, two additional branches are introduced into the network architecture of the deep learning-based one-stage yolov5 target detection algorithm;
one branch is used for processing an image coordinate key point detection task of the bionic ornithopter unmanned aerial vehicle and generating image coordinates of the image coordinate key point on the bionic ornithopter unmanned aerial vehicle;
the other branch is used for classifying the wing section states of the bionic ornithopter unmanned aerial vehicle, outputting the probability that the bionic ornithopter unmanned aerial vehicle is in different wing section states, and selecting the motion state with the highest probability as the wing section state of the bionic ornithopter unmanned aerial vehicle based on the output probability value.
Preferably, the image acquisition device is calibrated with internal and external parameters to obtain the internal and external parameter information of the image acquisition device, and the method specifically comprises the following steps: adopting a Zhang Zhengyou checkerboard calibration method to calibrate squares with the checkerboard size of L1 and black and white squares with the size of L2 to form alternately, taking the checkerboard as a calibration reference, and capturing images by changing the orientation of the checkerboard for a plurality of times so as to obtain rich coordinate information;
and then extracting angular points for calibration calculation, deleting images with average errors higher than a threshold value from the calibration result through an average error bar graph, and finally performing Export Camera Parameters calculation to obtain the internal and external parameter information of the image acquisition device.
And (3) resolving by using a spatial position and posture estimation algorithm to obtain a spatial position and posture result of the bionic ornithopter, specifically calculating three-dimensional coordinates of the image coordinate key points under a world coordinate system, and representing a posture cosine matrix and a position matrix of the position information of the bionic ornithopter through a perspective projection model Jie Suanchu.
Further, the three-dimensional coordinates of the image coordinate key points in the world coordinate system are calculated according to the following formula:
wherein P is the world three-dimensional coordinates of the image coordinate key points of the bionic ornithopter unmanned plane to be solved; p' is the corresponding image coordinate, K is the internal parameter matrix of the image acquisition device; alpha and beta are scaled focal lengths; s is a skew parameter; (u) 0 ,v 0 ) Is an optical center; w is the scale factor of the image point; r andis the external parameter information of the image acquisition device and represents the required 3D rotation and 3D flatteningAnd (5) moving.
A computer system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, said processor implementing the steps of the method as described above when said computer program is executed.
A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a method as described above.
The beneficial effects of the invention are as follows:
the invention provides a visual-based bionic ornithopter space pose estimation method, which does not need to construct a CAD model of a strange object, predefines the coordinate positions of key points of a bionic ornithopter wing-shaped state and a body system thereof, finds the key points on an image through a deep learning detection algorithm, and realizes the estimation of the space position pose of the bionic ornithopter through matching PnP of the key points of the image coordinates. According to the invention, the wing section state marking definition is carried out according to different flight attitudes of the bionic ornithopter, the space position and the attitude of the bionic ornithopter are obtained by solving after the deep learning training detection, and the problem that the space pose of the bionic ornithopter is difficult to solve through an external image due to the fact that the body structure of the bionic ornithopter is complex and the wing section state is various is solved. Therefore, the invention can adapt to more application scenes, complex machine types and better pose estimation performance.
Drawings
Fig. 1 is a flow chart of a bionic ornithopter space pose estimation method.
FIG. 2 is a schematic diagram of the establishment of a body coordinate system for the ornithopter.
FIG. 3 is another schematic view of the body coordinate system of the ornithopter.
Fig. 4 is a further schematic diagram of the establishment of a body coordinate system for the ornithopter.
Fig. 5 is a schematic diagram of selecting image coordinate keypoints.
FIG. 6 is a schematic illustration of a first airfoil condition.
FIG. 7 is another schematic illustration of a first airfoil state.
FIG. 8 is a schematic illustration of a second airfoil condition.
FIG. 9 is another schematic view of a second airfoil state.
Fig. 10 is a schematic view of a third airfoil configuration.
Fig. 11 is another schematic view of the third airfoil state.
Fig. 12 is a network model schematic of the object detection algorithm.
Fig. 13 is a schematic diagram of a checkerboard.
Fig. 14 is a schematic diagram of deleting an image in which the average error is higher than the threshold value.
Fig. 15 is a diagram of actual key point detection effects.
Detailed Description
Further advantages and effects of the present invention will become readily apparent to those skilled in the art from the disclosure herein, by referring to the accompanying drawings and the preferred embodiments. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be understood that the preferred embodiments are presented by way of illustration only and not by way of limitation.
It should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present invention by way of illustration, and only the components related to the present invention are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
Example 1
As shown in fig. 1, a method for estimating the space pose of a bionic ornithopter based on vision comprises the following steps:
s1: labeling the obtained bionic ornithopter unmanned aerial vehicle image, wherein labeling contents comprise image coordinate key points and wing profile state information of the bionic ornithopter unmanned aerial vehicle, and obtaining a bionic ornithopter unmanned aerial vehicle image training dataset;
s2: training the bionic ornithopter image training dataset and the labeling information by using a deep learning detection algorithm to obtain training weights, updating the deep learning detection algorithm, and obtaining a trained deep learning detection algorithm;
s3: performing inside and outside parameter calibration on the image acquisition device to obtain inside and outside parameter information of the image acquisition device, and acquiring a flight image of the bionic ornithopter through the image acquisition device;
s4: according to the trained deep learning detection algorithm, testing the flight image of the bionic ornithopter to obtain a key point image coordinate system test result and the wing shape state of the bionic ornithopter;
s5: according to the wing section state classification, measuring three-dimensional coordinate information of key points of a body coordinate system of the bionic ornithopter in different wing section states, and recording the wing section states;
s6: according to the internal and external parameter information of the image acquisition device, the testing result of the key point image coordinate system, the wing section state of the bionic ornithopter, and the three-dimensional coordinate information of key points of the body coordinate system of the bionic ornithopter under different wing section states, a spatial position and posture estimation algorithm is used for resolving, and the spatial position and posture result of the bionic ornithopter is obtained.
In a specific embodiment, a bionic ornithopter image is acquired, specifically as follows: and collecting and using various shooting devices (such as mobile phones, image collecting devices, ipad and other shooting devices with shooting functions) for data collection. After the bionic ornithopter unmanned aerial vehicle is controlled to take off, ground shooting personnel use shooting equipment to acquire data of the bionic ornithopter unmanned aerial vehicle, the shooting process keeps the bionic ornithopter unmanned aerial vehicle in the center of an image as much as possible, and keeps the stability of an image acquisition device, so that high-quality data can be acquired as much as possible. The data acquisition is divided into a plurality of sections of video shooting, the shooting time length of each section is 3-5 minutes, the video definition is 1080P, and the shot video is divided into image sets after shooting is completed.
And (3) establishing a body coordinate system of the ornithopter in a mode shown in fig. 2, 3 and 4, wherein the origin of the body coordinate system is selected on a fixed buckle on the back of the ornithopter.
The image coordinate key points of the bionic ornithopter are selected to be the nose of the bionic ornithopter, the wingtip of the left wing, the wingtip of the right wing, the wingtip of the left tail wing and the wingtip of the right tail wing respectively, as shown in fig. 5.
The wing shape states of the unmanned aerial vehicle are divided into three types according to the positions of the wings of the bionic ornithopter, namely, a first wing shape state (fig. 6 and 7) when the wings are positioned above the horizontal line of the airframe, a second wing shape state (fig. 8 and 9) when the wings are positioned below the horizontal line of the airframe, and a second wing shape state (fig. 10 and 11) when the wings are positioned below the horizontal line of the airframe.
In the three wing section states, the coordinates of the key points of the image coordinates in the ornithopter body coordinate system are as follows:
left_tail＝[-0.29 -0.1 -0.05]
right_tail＝[-0.29 0.1 -0.05]
left_wing_up＝[0 -0.18 0.17]
right_wing_up＝[0 0.18 0.17]
left_wing_mid＝[0 -0.26 0.17]
right_wing_mid＝[0 0.26 0.17]
left_wing_down＝[0 -0.18 -0.17]
right_wing_down＝[0 0.18 -0.17]
head＝[0.13 0 0]
wherein left_tail represents the coordinates of the key point 4 in the body coordinate system in fig. 5, right_tail represents the coordinates of the key point 5 in the body coordinate system in fig. 5, left_wing_up represents the coordinates of the key point 2 in the first airfoil state in the body coordinate system in fig. 5, right_wing_up represents the coordinates of the key point 3 in the first airfoil state in the body coordinate system in fig. 5, left_wing_mid represents the coordinates of the key point 2 in the second airfoil state in the body coordinate system in fig. 5, right_wing_mid represents the coordinates of the key point 3 in the second airfoil state in the body coordinate system in fig. 5, left_wing_down represents the coordinates of the key point 2 in the third airfoil state in the body coordinate system in fig. 5, and right_wing_down represents the coordinates of the key point 3 in the third airfoil state in the body coordinate system in fig. 5.
In a specific embodiment, the image coordinate key points and the wing shape states of the bionic ornithopter unmanned aerial vehicle image are marked, and the method specifically comprises the following steps: the data marking uses labelimg data calibration software, and the marking format is yolo.
Firstly, calibrating the positions of the image coordinate key points by using a rectangular frame, and then calculating the center coordinates of the rectangular frame to obtain the two-dimensional image coordinates of the image coordinate key points; wherein the position of the rectangular box is represented using the upper left corner xy coordinates and the lower right corner xy coordinates;
the wing section states are marked according to the flight attitude of the bionic ornithopter in the bionic ornithopter image and are divided into three types;
the final calibration data of each image are 16 groups of data including 5 image coordinate key points, two-dimensional image coordinates corresponding to the 5 image coordinate key points and 1 airfoil state;
and after the calibration is completed, obtaining an image training data set of the bionic ornithopter.
In a specific embodiment, the deep learning detection algorithm is modified and optimized as follows: the deep learning detection algorithm adopts a yolov5 target detection algorithm, and an output double channel is added on the basis of the yolov5 target detection algorithm to perform feature training on the key points of the image coordinates and the wing profile state.
Introducing two additional branches into a network architecture of a deep learning-based one-stage yolov5 target detection algorithm;
one branch is used for processing an image coordinate key point detection task of the bionic ornithopter unmanned aerial vehicle and generating image coordinates of the image coordinate key point on the bionic ornithopter unmanned aerial vehicle;
the other branch is used for classifying the wing section states of the bionic ornithopter unmanned aerial vehicle, outputting the probability that the bionic ornithopter unmanned aerial vehicle is in different wing section states, and selecting the motion state with the highest probability as the wing section state of the bionic ornithopter unmanned aerial vehicle based on the output probability value.
Through the improvements, the capability of a yolov5 target detection algorithm is enhanced, and more accurate and comprehensive information is provided for the key point detection and the wing section state estimation of the bionic ornithopter unmanned plane. The improved yolov5 network structure is shown in figure 12.
The training process of the deep learning detection algorithm is mainly divided into three parts, namely environment construction, model training and model detection. The environment construction needs to carry out corresponding deep learning algorithm environment configuration according to the model of the hardware equipment. And then under a configured environment, dividing the manufactured ornithopter detection data set into a training set and a testing set, training the deep learning detection algorithm under different parameters by using the training set, testing and verifying the trained deep learning detection algorithm by using the testing set after training, and selecting a model with highest performance precision as a detection model for final use.
The improved yolov5 target detection algorithm comprises a feature extraction network CSP-Darknet53, a multi-scale feature fusion network FPN and an improved prediction network Head. The feature extraction network comprises a downsampling Focus module, four basic convolution network modules Conv, four feature extraction modules C3 and a pooling module SPPF; the improved prediction network Head comprises a feature point category prediction network, a location prediction network and an airfoil state prediction network, wherein the feature point category prediction network and the location prediction network are represented in one output channel, and the airfoil state prediction network is represented in the other output channel by a PAN module.
The training set is input into an improved yolov5 target detection algorithm for training, the iteration number is set to be 120, the learning rate is set to be 1.2e-4, the optimizer uses Adam, the CSP-Darknet53 network part training attenuation rate is set to be 1e-4, the FPN and Head network part attenuation rate is set to be 1.2e-5, the training batch size is set to be 32, and the loss function consists of classification loss, positioning loss and confidence loss, wherein the classification loss comprises characteristic point type loss and wing type state type loss. And finishing training after the appointed iteration times are reached, and obtaining the bionic ornithopter detection network.
The detection process through the yolov5 target detection algorithm specifically comprises the following steps: each frame of image is scaled to 640x640x3 through up-down sampling, firstly, a Focus sampling module is used, then a convolution Conv module and a feature extraction module C3 are sequentially and alternately used for 4 times, and then a pooling module SPPF module is used for obtaining an image global feature map; inputting an image global feature map into a multi-scale feature fusion network FPN, outputting three-channel dimensional feature maps through downsampling, wherein the dimensions are respectively 80x80x (3 x 10), 40x40x (3 x 10), 20x20x (3 x 10), wherein 80, 40 and 20 are respectively feature map width and height, 3 represents that each network is preset with three anchor frames with different sizes, and 10 represents that each anchor frame needs to predict 5 feature point category information, target frame (x, y, w, h) position information and confidence coefficient by 10 values; the three-dimensional feature map is divided into two channels for processing, wherein one channel respectively and directly predicts the feature point types and the position information of the bionic ornithopter for the three-dimensional feature map, the other channel carries out up-sampling feature fusion on the three-dimensional feature map and sums the three-dimensional feature map on a third dimension to obtain an 80x80x3 feature map, and three airfoil-shaped state probabilities of the bionic ornithopter are predicted on the feature map.
In a specific embodiment, the image acquisition device is calibrated by internal and external parameters to obtain the internal and external parameter information of the image acquisition device, which is specifically as follows: a Zhang Zhengyou checkerboard calibration method is adopted to calibrate squares with the checkerboard size of L1=82.5 mm and black and white squares with the size of L2=10x7mm are alternately formed (fig. 13), the checkerboard is used as a calibration reference, the complex three-dimensional object is easier to process, and the image is captured by changing the orientation of the checkerboard for a plurality of times so as to obtain rich coordinate information;
and then extracting angular points for calibration calculation, deleting images (figure 14) with average errors higher than a threshold value from the calibration result through an average error bar graph, and finally performing Export Camera Parameters calculation to obtain the internal and external parameter information of the image acquisition device.
In this embodiment, the test image of the bionic ornithopter unmanned plane is obtained as follows: firstly, fixing the position of an image acquisition device, defining the pose of the image acquisition device under a world coordinate system, and then carrying out data acquisition again. And then, the collected data are subjected to the detection of the unmanned plane key points and the wing profile states of the bionic ornithopter by using a trained target detection algorithm. Fig. 15 is a diagram of the actual key point detection effect.
In a specific embodiment, the internal and external parameter information of the image acquisition device, the testing result of the key point image coordinate system, the wing type state of the bionic ornithopter unmanned aerial vehicle and the three-dimensional coordinate information of the key point of the body coordinate system of the bionic ornithopter unmanned aerial vehicle under different wing type states are input into a spatial position and posture estimation algorithm to be calculated, the spatial position and posture of the unmanned aerial vehicle relative to the image acquisition device are obtained, the calibrated external parameters of the image acquisition device are utilized to obtain the position and posture of the image acquisition device in the spatial coordinates, and the coordinate conversion is carried out, so that the position and posture of the unmanned aerial vehicle in the spatial coordinates are finally obtained.
And (3) performing calculation by using a spatial position and posture estimation algorithm (such as an EPnP pose estimation algorithm) to obtain spatial positions and posture results of the bionic ornithopter, specifically, calculating three-dimensional coordinates of key points of image coordinates under a world coordinate system, and representing a posture cosine matrix and a position matrix of pose information of the bionic ornithopter through a perspective projection model Jie Suanchu.
In this embodiment, the three-dimensional coordinates of the image coordinate key points in the world coordinate system are calculated according to the following formula:
wherein P is the world three-dimensional coordinates of the image coordinate key points of the bionic ornithopter unmanned plane to be solved; p' is the corresponding image coordinate, K is the internal parameter matrix of the image acquisition device; alpha and beta are scaled focal lengths; s is a skew parameter; (u) 0 ,v 0 ) Is an optical center; w is the scale factor of the image point; r andis the external parameter information of the image acquisition device, and represents the required 3D rotation and 3D translation; (u, v) is the coordinates of the image keypoints in the pixel coordinate system; r is (r) 11 ～r 33 Is a camera external parameter rotation matrix [ t ] x ,t y ,t z ]Is a camera external parameter translation matrix, [ x, y, z ]]Is the key point of the image is in the worldAnd three-dimensional coordinates under a boundary coordinate system.
And solving P, wherein the solving algorithm adopts an EPnP method, and the solving flow is as follows:
d1: four points not in the same plane are selected as control points c in the world coordinate system (represented by the table w above) w To uniquely represent the coordinates p of the already selected image coordinate key point w The method comprises the following steps:wherein: />
Wherein,representing 4 control points in world coordinate system, < >>Representing selected image keypoints, alpha ij Representing a weight matrix.
D2: under the coordinate system of the image acquisition device, the weight is unchanged, namely:
wherein,representing 4 control points in the camera coordinate system.
D3: taking the obtained key point image coordinates as the input of an EPnP algorithm, and regarding any one of the image coordinate key pointsConsider the transformation relationship of the image acquisition device system to the pixel coordinate system:
wherein,representing the z-axis direction coordinate of the key point in a camera coordinate system, f x ，f y Representing the normalized focal length of the camera,representing the coordinates of the control point in the camera coordinate system.
Unfolding it into:
wherein,representing the z-axis direction coordinate of the control point in the camera coordinate system, [ u ] i ,v i ]Representing the coordinates of the key points in the pixel coordinate system.
Selecting 5 image coordinate key points to obtain 10 equations, and expressing the 10 equations into a matrix form as follows:
M 2n×12 x 12×1 ＝0
wherein:is the coordinates of the control point in the coordinate system of the image acquisition device. M is M 2n×12 For the characteristic equation coefficient matrix, < >>And the coordinates under the image coordinate system of each control point to be solved.
D4: the solution to x can be converted into a zero eigenvalue vector problem of solution M, and can be converted into solution matrix M T And (3) a problem of the eigenvector corresponding to the M zero eigenvalue.
Wherein N represents the number of control points, beta i Is the inverse of M, v i Is a zero vector matrix.
D5: after coordinates of the image coordinate key points under the image acquisition device system are obtained, the optimal posture matrix and the position matrix are solved by Singular Value Decomposition (SVD).
And (3) making:H＝B T a, wherein->
Wherein,transposed matrix representing key points in world coordinate system, < ->Transposed matrix representing key points in camera coordinate system, A, B is decomposition matrix, +.>Represents the key point coordinates under the camera coordinate system, +.>Representing the coordinates of the key points in the world coordinate system.
D6: and decomposing the singular value of H to obtain:
H＝U∑V T
wherein U, V represents a decomposition matrix.
D7: obtaining an attitude cosine matrix:
R＝UV T
d8: further obtain displacement vector:
d9: solving Euler angles
The Euler angle representation method is used for representing the gesture of the bionic ornithopter by adopting (phi, theta, phi), wherein phi is a rolling angle, theta is a pitch angle, phi is a yaw angle, and the rotation sequence is phi, theta, phi.
The known conversion relation from euler angle to attitude cosine is:
the Euler angle can be obtained by the inverse solution of the conversion relation:
d10: then according to the pose of the image acquisition device fixed during image acquisition relative to the world coordinate system, carrying out coordinate transformation, and finally outputting pose information of the ornithopter relative to the world coordinate system
In this embodiment, the image capturing device includes, but is not limited to, a mobile phone, a camera, an ipad, and other photographing devices.
In this embodiment, the target detection algorithm may be yolov5 target detection algorithm, and may be replaced by other detection algorithms, including but not limited to Faster RCNN, DETR, etc.
In this embodiment, the spatial position and posture estimation algorithm includes, but is not limited to, EPnP method, DLT method, RPnP method, and method capable of optimizing EPnP result based on gaussian newton optimization method.
Example 2
A computer system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, said processor implementing the steps of the method according to embodiment 1 when said computer program is executed.
Example 3
A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method as described in embodiment 1.
It is to be understood that the above examples of the present invention are provided by way of illustration only and not by way of limitation of the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are desired to be protected by the following claims.
Claims (10)
1. A bionic ornithopter space pose estimation method based on vision is characterized by comprising the following steps: the method comprises the following steps:
labeling the obtained bionic ornithopter unmanned aerial vehicle image, wherein labeling contents comprise image coordinate key points and wing profile state information of the bionic ornithopter unmanned aerial vehicle, and obtaining a bionic ornithopter unmanned aerial vehicle image training dataset;
training the bionic ornithopter image training dataset and the labeling information by using a deep learning detection algorithm to obtain training weights, updating the deep learning detection algorithm, and obtaining a trained deep learning detection algorithm;
performing inside and outside parameter calibration on the image acquisition device to obtain inside and outside parameter information of the image acquisition device, and acquiring a flight image of the bionic ornithopter through the image acquisition device;
according to the trained deep learning detection algorithm, testing the flight image of the bionic ornithopter to obtain a key point image coordinate system test result and the wing shape state of the bionic ornithopter;
according to the wing section state classification, measuring three-dimensional coordinate information of key points of a body coordinate system of the bionic ornithopter in different wing section states, and recording the wing section states;
according to the internal and external parameter information of the image acquisition device, the testing result of the key point image coordinate system, the wing section state of the bionic ornithopter, and the three-dimensional coordinate information of key points of the body coordinate system of the bionic ornithopter under different wing section states, a spatial position and posture estimation algorithm is used for resolving, and the spatial position and posture result of the bionic ornithopter is obtained.
2. The visual-based bionic ornithopter space pose estimation method according to claim 1, wherein the method comprises the following steps of: the image coordinate key points of the bionic ornithopter are respectively a nose of the bionic ornithopter, a wingtip of a left wing, a wingtip of a right wing, a wingtip of a left tail wing and a wingtip of a right tail wing;
the airfoil states are divided into three categories, respectively: a first airfoil state when the wing is above the fuselage level, a second airfoil state when the wing is below the fuselage level, and a third airfoil state when the wing is below the fuselage level.
3. The visual-based bionic ornithopter space pose estimation method according to claim 2, wherein the method comprises the following steps: the bionic ornithopter unmanned aerial vehicle image is marked with image coordinate key points and wing profile states, and the method specifically comprises the following steps:
firstly, calibrating the positions of the image coordinate key points by using a rectangular frame, and then calculating the center coordinates of the rectangular frame to obtain the two-dimensional image coordinates of the image coordinate key points; wherein the position of the rectangular box is represented using the upper left corner xy coordinates and the lower right corner xy coordinates;
the wing section states are marked according to the flight attitude of the bionic ornithopter in the bionic ornithopter image and are divided into three types;
the final calibration data of each image are 16 groups of data including 5 image coordinate key points, two-dimensional image coordinates corresponding to the 5 image coordinate key points and 1 airfoil state;
and after the calibration is completed, obtaining an image training data set of the bionic ornithopter.
4. The visual-based bionic ornithopter space pose estimation method according to claim 1, wherein the method comprises the following steps of: the deep learning detection algorithm adopts a yolov5 target detection algorithm, and an output double channel is added on the basis of the yolov5 target detection algorithm to perform feature training on the key points of the image coordinates and the wing profile state.
5. The visual-based bionic ornithopter space pose estimation method according to claim 4, wherein the method comprises the following steps: introducing two additional branches into a network architecture of a deep learning-based one-stage yolov5 target detection algorithm;
one branch is used for processing an image coordinate key point detection task of the bionic ornithopter unmanned aerial vehicle and generating image coordinates of the image coordinate key point on the bionic ornithopter unmanned aerial vehicle;
the other branch is used for classifying the wing section states of the bionic ornithopter unmanned aerial vehicle, outputting the probability that the bionic ornithopter unmanned aerial vehicle is in different wing section states, and selecting the motion state with the highest probability as the wing section state of the bionic ornithopter unmanned aerial vehicle based on the output probability value.
6. The visual-based bionic ornithopter space pose estimation method according to claim 1, wherein the method comprises the following steps of: the method comprises the steps of calibrating internal and external parameters of an image acquisition device to obtain internal and external parameter information of the image acquisition device, and specifically comprises the following steps: adopting a Zhang Zhengyou checkerboard calibration method to calibrate squares with the checkerboard size of L1 and black and white squares with the size of L2 to form alternately, taking the checkerboard as a calibration reference, and capturing images by changing the orientation of the checkerboard for a plurality of times so as to obtain rich coordinate information;
and then extracting angular points for calibration calculation, deleting images with average errors higher than a threshold value from the calibration result through an average error bar graph, and finally performing Export Camera Parameters calculation to obtain the internal and external parameter information of the image acquisition device.
7. The visual-based bionic ornithopter space pose estimation method according to claim 1, wherein the method comprises the following steps of: and (3) resolving by using a spatial position and posture estimation algorithm to obtain a spatial position and posture result of the bionic ornithopter, specifically calculating three-dimensional coordinates of the image coordinate key points under a world coordinate system, and representing a posture cosine matrix and a position matrix of the position information of the bionic ornithopter through a perspective projection model Jie Suanchu.
8. The visual-based bionic ornithopter space pose estimation method of claim 7, wherein the method comprises the following steps of: the three-dimensional coordinates of the image coordinate key points under the world coordinate system are calculated according to the following formula:
wherein P is the world three-dimensional coordinates of the image coordinate key points of the bionic ornithopter unmanned plane to be solved; p' is the corresponding image coordinate, K is the internal parameter matrix of the image acquisition device; alpha and beta are scaled focal lengths; s is a skew parameter; (u) 0 ,v 0 ) Is an optical center; w is the scale factor of the image point; r andis the external parameter information of the image acquisition device, and represents the required 3D rotation and 3D translation.
9. A computer system comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized by: the processor, when executing the computer program, performs the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program, when executed by a processor, performs the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310990249.0A CN117115252A (en) | 2023-08-07 | 2023-08-07 | Bionic ornithopter space pose estimation method based on vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310990249.0A CN117115252A (en) | 2023-08-07 | 2023-08-07 | Bionic ornithopter space pose estimation method based on vision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117115252A true CN117115252A (en) | 2023-11-24 |
Family
ID=88811961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310990249.0A Pending CN117115252A (en) | 2023-08-07 | 2023-08-07 | Bionic ornithopter space pose estimation method based on vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117115252A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315792A (en) * | 2023-11-28 | 2023-12-29 | 湘潭荣耀智能科技有限公司 | Real-time regulation and control system based on prone position human body measurement |
-
2023
- 2023-08-07 CN CN202310990249.0A patent/CN117115252A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315792A (en) * | 2023-11-28 | 2023-12-29 | 湘潭荣耀智能科技有限公司 | Real-time regulation and control system based on prone position human body measurement |
| CN117315792B (en) * | 2023-11-28 | 2024-03-05 | 湘潭荣耀智能科技有限公司 | Real-time regulation and control system based on prone position human body measurement |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110070615B (en) | Multi-camera cooperation-based panoramic vision SLAM method | |
| CN112634451B (en) | Outdoor large-scene three-dimensional mapping method integrating multiple sensors | |
| CN106679648B (en) | Visual inertia combination SLAM method based on genetic algorithm | |
| CN107341814B (en) | Four-rotor unmanned aerial vehicle monocular vision range measurement method based on sparse direct method | |
| CN106803271B (en) | Camera calibration method and device for visual navigation unmanned aerial vehicle | |
| Panahandeh et al. | Vision-aided inertial navigation based on ground plane feature detection | |
| CN106529538A (en) | Method and device for positioning aircraft | |
| CN107516326B (en) | Robot positioning method and system fusing monocular vision and encoder information | |
| CN111862201B (en) | Deep learning-based spatial non-cooperative target relative pose estimation method | |
| Li et al. | A 4-point algorithm for relative pose estimation of a calibrated camera with a known relative rotation angle | |
| CN110047108B (en) | Unmanned aerial vehicle pose determination method and device, computer equipment and storage medium | |
| CN107481292A (en) | The attitude error method of estimation and device of vehicle-mounted camera | |
| CN108759826B (en) | Unmanned aerial vehicle motion tracking method based on multi-sensing parameter fusion of mobile phone and unmanned aerial vehicle | |
| CN110211151B (en) | Method and device for tracking moving object | |
| CN109858437B (en) | Automatic luggage volume classification method based on generation query network | |
| CN112509036B (en) | Pose estimation network training and positioning method, device, equipment and storage medium | |
| CN113269840A (en) | Combined calibration method for camera and multi-laser radar and electronic equipment | |
| CN111998862B (en) | BNN-based dense binocular SLAM method | |
| CN109214254B (en) | Method and device for determining displacement of robot | |
| CN117115252A (en) | Bionic ornithopter space pose estimation method based on vision | |
| CN111623773B (en) | Target positioning method and device based on fisheye vision and inertial measurement | |
| CN115272596A (en) | Multi-sensor fusion SLAM method oriented to monotonous texture-free large scene | |
| CN111812978B (en) | Cooperative SLAM method and system for multiple unmanned aerial vehicles | |
| CN114812558B (en) | Monocular vision unmanned aerial vehicle autonomous positioning method combining laser ranging | |
| CN115588144A (en) | Real-time attitude capturing method, device and equipment based on Gaussian dynamic threshold screening |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |