CN116360492A - Object tracking method and system for flapping wing flying robot - Google Patents
Object tracking method and system for flapping wing flying robot Download PDFInfo
- Publication number
- CN116360492A CN116360492A CN202310345807.8A CN202310345807A CN116360492A CN 116360492 A CN116360492 A CN 116360492A CN 202310345807 A CN202310345807 A CN 202310345807A CN 116360492 A CN116360492 A CN 116360492A
- Authority
- CN
- China
- Prior art keywords
- target
- airborne
- camera
- tracked
- processing module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 75
- 238000013507 mapping Methods 0.000 claims abstract description 12
- 238000013528 artificial neural network Methods 0.000 claims description 28
- 238000001514 detection method Methods 0.000 claims description 19
- 238000001914 filtration Methods 0.000 claims description 12
- 230000004927 fusion Effects 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 abstract description 8
- 210000003128 head Anatomy 0.000 description 42
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000015654 memory Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 3
- 238000013178 mathematical model Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013016 damping Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000006641 stabilisation Effects 0.000 description 2
- 238000011105 stabilization Methods 0.000 description 2
- 241000238631 Hexapoda Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000004146 energy storage Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/10—Simultaneous control of position or course in three dimensions
- G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
- G05D1/106—Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
Landscapes
- Engineering & Computer Science (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Studio Devices (AREA)
Abstract
The invention relates to the technical field of visual tracking of flapping-wing flying robots, in particular to a target tracking method and a target tracking system of the flapping-wing flying robot. Comprising the following steps: initializing an airborne vision processing module and an airborne camera cradle head module; the method comprises the steps that an airborne camera holder module obtains a long-focus camera aerial image and a short-focus camera aerial image of the ornithopter; the airborne vision processing module selects a target to be tracked based on the short-focus camera aerial image and a target tracking algorithm; the method comprises the steps that an airborne vision processing module obtains the pixel position of a target to be tracked through a target tracking algorithm; the airborne vision processing module controls the rotation of the airborne camera pan-tilt module through the pan-tilt controller according to the pixel position of the target to be tracked; the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through the camera mapping relation, and achieves real-time tracking of the target to be tracked. By adopting the method and the device, the target matching range can be reduced, and the stability of the object tracking of the ornithopter robot can be improved.
Description
Technical Field
The invention relates to the technical field of visual tracking of flapping-wing flying robots, in particular to a target tracking method and a target tracking system of the flapping-wing flying robot.
Background
The wing classes of unmanned aircraft are roughly three types: fixed wings, rotor wings and flapping wings. The flapping wing flying robot is a novel unmanned aerial vehicle designed based on the flying modes of birds and insects, and the biggest difference between the unmanned aerial vehicle and a fixed wing aerial vehicle and a rotor wing aerial vehicle is that the sources of lift force and thrust force are the flapping of wings. The flapping wing flying robot has the advantages of light weight, small volume, low energy consumption and the like, has wide application prospect in the military and civil fields, such as low-altitude reconnaissance, accurate striking and the like in the military field, and has natural disaster monitoring and supporting, environment and pollution monitoring and the like in the civil field. The object tracking function of a ornithopter robot is indispensable for the above-mentioned applications.
The flapping-wing flying robot cannot hover and needs to continuously fly by flapping wings, so that the low-frequency high-amplitude shake, small target and the like of the aerial video of the flapping-wing flying robot exist. Common image stabilization methods comprise mechanical shake elimination of a tripod head, and shake of aerial video of the flapping-wing flying robot cannot be completely eliminated by an electronic image stabilization algorithm. The common target tracking methods such as filtering tracking and IOU matching have poor video jitter resistance capability because the target is required to be displaced in a large section in a short time, and the application effect is poor in aerial video of the ornithopter.
Therefore, the research on the object tracking system of the ornithopter robot is focused on developing a method suitable for the characteristics of aerial videos of the ornithopter robot. However, related researches are less at present, and the invention patent 202210917866.3 discloses a video stream target tracking method, a system and a storage medium for an unmanned aerial vehicle, wherein the method adopts a related filtering algorithm to track a target in a video stream, but a related filtering series algorithm is greatly influenced by video jitter, and the invention patent has poor applicability in the field of ornithopter robots.
Disclosure of Invention
The invention provides a method and a system for tracking a target of a flapping-wing flying robot, which solve the problem that the common method for tracking the target such as filtering tracking and IOU matching has poor video jitter resistance capability and poor application effect in aerial video of the flapping-wing flying robot because the target cannot be displaced in a large scale in a short time.
In order to solve the above-mentioned purpose, the technical scheme provided by the invention is as follows: the method is realized by a flapping-wing flying robot target tracking system, and the flapping-wing flying robot target tracking system comprises a flapping-wing flying robot, an airborne camera holder module and an airborne vision processing module;
the method comprises the following steps:
s1, initializing an airborne vision processing module;
s2, initializing an airborne camera holder module;
s3, an airborne camera holder module acquires an aerial video stream of the ornithopter flying robot, wherein the aerial video stream comprises an aerial image of a long-focus camera and an aerial image of a short-focus camera;
s4, selecting a target to be tracked by the airborne vision processing module based on the short-focus camera aerial image and a target tracking algorithm;
s5, the airborne vision processing module acquires the pixel position of the target to be tracked from the aerial video stream of the ornithopter through a target tracking algorithm;
s6, the airborne vision processing module controls the rotation of the airborne camera cradle head module through the cradle head controller according to the pixel position of the target to be tracked, so as to track the target to be tracked;
s7, the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through a camera mapping relation;
s8, circularly executing the steps S5 to S7 until the real-time tracking of the target to be tracked is completed.
Preferably, in S1, initializing an on-board vision processing module includes:
s11, initializing a maximum range of a clustering target cluster of the airborne vision processing module according to a preset range;
s12, initializing filter parameters of the airborne vision processing module according to preset parameters;
s13, loading a twin network model of the airborne vision processing module.
Preferably, in S4, the target tracking algorithm includes a target detection algorithm, a clustering algorithm, and a twin neural network.
Preferably, in S4, the selecting, by the on-board vision processing module, the target to be tracked based on the aerial image of the short-focus camera and the target tracking algorithm includes:
s41, performing target detection on the aerial image of the short-focus camera based on a target detection algorithm to obtain a plurality of targets;
s42, clustering a plurality of targets by taking the distance as a reference based on a clustering algorithm to obtain a target cluster, and determining the weighted average sum of the confidence coefficients of the plurality of targets as the confidence coefficient of the target cluster;
s43, determining a preselected target cluster where a target to be tracked is located in the target clusters through an IOU matching and filtering track tracking mode respectively;
s44, extracting a local image where the preselected target cluster is located;
s45, inputting the local image and the aerial image of the long-focus camera into a twin neural network for feature matching to obtain a plurality of feature matching targets;
s46, determining an optimal feature matching target from the feature matching targets, and determining the optimal feature matching target as a target to be tracked.
Preferably, the twin neural network adopts Vision-transducer as a backbone network, and an attention module is added to improve network performance; data enhancement and HOG+color feature fusion are used to inhibit image blurring and deformation caused by aerial video shake of the ornithopter.
Preferably, a clustering algorithm in the target tracking algorithm adopts a k-means algorithm to modify the IOU and the track tracked object from a small target to a target cluster.
Preferably, in S6, the on-board vision processing module controls the on-board camera pan-tilt module to rotate through the pan-tilt controller according to the pixel position of the target to be tracked, including:
the pixel position of the target to be tracked is converted into the rotation angle of the airborne camera cradle head module through the following formula:
wherein (θ) x ,θ y ) Indicating the rotation angle theta of the airborne camera cradle head module x And represents the abscissa, θ, of the rotation angle y An ordinate representing the rotation angle; (X) T ,Y T ) Representing the pixel position of the object to be tracked, (X) W ,Y W ) Representing image resolution, f representing focal length of the short-focus camera, a×b representing image plane size;
and controlling the rotation of the airborne camera tripod head module through the tripod head controller according to the rotation angle of the airborne camera tripod head module.
The system is used for the object tracking method of the flapping wing flying robot, and comprises the following steps:
the flapping wing flying robot target tracking system comprises a flapping wing flying robot, an airborne camera holder module and an airborne vision processing module, wherein the airborne camera holder module is a holder which is carried on the flapping wing flying robot and is provided with a long-focus camera and a short-focus camera, the airborne camera holder module controls rotation through a holder controller, and the airborne vision processing module comprises a target tracking algorithm based on clustering and a twin neural network;
the airborne camera holder module is used for acquiring aerial video streams of the ornithopter flying robot, wherein the aerial video streams comprise long-focus camera aerial images and short-focus camera aerial images;
the airborne vision processing module is used for selecting a target to be tracked based on the short-focus camera aerial image and a target tracking algorithm; the airborne vision processing module acquires the pixel position of a target to be tracked from an aerial video stream of the ornithopter robot through a target tracking algorithm; the airborne vision processing module controls the rotation of the airborne camera pan-tilt module through the pan-tilt controller according to the pixel position of the target to be tracked, so as to track the target to be tracked; and the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through the camera mapping relation.
Preferably, the target tracking algorithm includes a target detection algorithm, a clustering algorithm, and a twin neural network.
Preferably, the on-board vision processing module is further configured to:
s41, performing target detection on the aerial image of the short-focus camera based on a target detection algorithm to obtain a plurality of targets;
s42, clustering a plurality of targets by taking the distance as a reference based on a clustering algorithm to obtain a target cluster, and determining the weighted average sum of the confidence coefficients of the plurality of targets as the confidence coefficient of the target cluster;
s43, determining a preselected target cluster where a target to be tracked is located in the target clusters through an IOU matching and filtering track tracking mode respectively;
s44, extracting a local image where the preselected target cluster is located;
s45, inputting the local image and the aerial image of the long-focus camera into a twin neural network for feature matching to obtain a plurality of feature matching targets;
s46, determining an optimal feature matching target from the feature matching targets, and determining the optimal feature matching target as a target to be tracked.
In one aspect, an electronic device is provided that includes a processor and a memory having at least one instruction stored therein that is loaded and executed by the processor to implement the ornithopter robot target tracking method described above.
In one aspect, a computer readable storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement the ornithopter flying robot target tracking method described above is provided.
Compared with the prior art, the technical scheme has at least the following beneficial effects:
according to the scheme, the object tracking scheme of the flapping-wing flying robot based on the long-short-focus double cameras is provided, and two methods are adopted to improve aiming at the characteristics of small aerial video objects and few characteristics of the flapping-wing flying robot: the clustering algorithm is adopted to reduce the size to be large, the target cluster is tracked, and the target matching range is further narrowed; the hawk eyes double-fovea structure is used for reference, and long and short focal cameras are cooperatively used to extract the characteristics of more targets. Aiming at the characteristic of aerial video shake of the ornithopter flying robot, a twin neural network with stronger robustness is adopted to carry out target matching, and a scheme for controlling a cradle head after image target tracking is completed is provided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for tracking a target of a ornithopter flying robot, which is provided by an embodiment of the invention;
FIG. 2 is a schematic diagram of a twin neural network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a cluster target cluster tracking algorithm according to an embodiment of the present invention;
FIG. 4 is a flow chart of a target tracking method of a flapping-wing flying robot based on an eagle eye imitating long-short focus double camera provided by the embodiment of the invention;
FIG. 5 is a block diagram of a target tracking system for a ornithopter flying robot according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an onboard camera pan-tilt module with a long-short-focus dual camera according to an embodiment of the present invention.
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. It will be apparent that the described embodiments are some, but not all, embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without creative efforts, based on the described embodiments of the present invention fall within the protection scope of the present invention.
The invention provides a real-time capturing object tracking method of a flapping wing flying robot, which aims at solving the problems that the existing object tracking methods such as filter tracking and IOU matching are poor in video jitter resistance and poor in application effect in aerial videos of the flapping wing flying robot because the object is required to be capable of displacing in a large section in a short time.
As shown in fig. 1, the embodiment of the invention provides a method for tracking an object of a ornithopter robot, which can be realized by electronic equipment. As shown in fig. 1, the object tracking method flow chart of the ornithopter flying robot, the processing flow of the method can comprise the following steps:
s101, initializing an airborne vision processing module;
in a possible embodiment, initializing an on-board vision processing module includes:
s111, initializing a maximum range of a clustering target cluster of the airborne vision processing module according to a preset range;
s112, initializing filter parameters of the airborne vision processing module according to preset parameters;
s113, loading a twin network model of the airborne vision processing module.
S102, initializing an airborne camera holder module;
s103, an airborne camera holder module acquires an aerial video stream of the ornithopter flying robot, wherein the aerial video stream comprises an aerial image of a long-focus camera and an aerial image of a short-focus camera;
s104, selecting a target to be tracked by the on-board vision processing module based on the aerial image of the short-focus camera and a target tracking algorithm.
In one possible implementation, the target tracking algorithm includes a target detection algorithm, a clustering algorithm, and a twin neural network.
In a possible implementation, the target detection algorithm may use Yolo v5, input 1920×1080 resolution size image returned by the short-focus camera, and output the type, image coordinates and confidence of each target.
In a possible implementation, the twin neural network may employ Vision-transducer as a backbone network, and add an attention module to improve network performance; data enhancement and HOG+color feature fusion are used to inhibit image blurring and deformation caused by aerial video shake of the ornithopter.
The twin neural network is a coupling framework constructed based on a plurality of neural networks, and the main purpose of the twin neural network is to measure the matching degree of a plurality of inputs. It usually contains two inputs, which are mapped to a high-dimensional feature space through a specific transformation, resulting in a corresponding representation. By means of a similarity evaluation function, such as the Euclidean distance, the similarity between the two characterizations can be calculated, thereby obtaining the degree of similarity between the two inputs. As shown in fig. 2, an image x is input 1 、x 2 Each sub-network uses the same network extraction characteristics to generate G respectively w (x 1 )、G w (x 2 ) W is a vector of learnable shared parameters. FinallyCalculating the function E using the similarity W The degree of similarity of the two features is measured.
E W ＝||G w (x 1 )-G w (x 2 )||
The distance between the two paths of features can be calculated by using the similarity calculation function, so that the similarity between the image pairs is obtained, and the judgment is performed. In the target tracking application, the similarity matching is usually carried out on the target features and the search area features, a response score chart is output, and the position with the highest score is the target. The invention adopts Vision-transducer as a backbone network, and adds an attention module to improve the network performance; data enhancement and HOG+color feature fusion are used to inhibit image blurring and deformation caused by aerial video shake of the ornithopter.
In a feasible implementation manner, a clustering algorithm in the target tracking algorithm can adopt a k-means algorithm, as shown in fig. 3, the objects tracked by the IOU and the track are modified from small targets to target clusters, the defect that the small targets are difficult to track under the aerial video shake of the flapping-wing flying robot is overcome, and meanwhile, the feature matching range of the twin neural network is reduced to a certain target cluster, so that the matching success rate is improved. The image coordinate center position of each target is input, and the algorithm steps are as follows: (1) randomly selecting an initial target cluster center point, and distributing the rest objects to the nearest cluster according to the distance between the rest objects and the representative object; (2) inverse multiplexing non-representative objects to replace representative objects to improve the quality of clustering; (3) using cost functions
The average dissimilarity between the subject and the reference subject is assessed.
In a possible implementation manner, the on-board vision processing module selects an object to be tracked based on the aerial image of the short-focus camera and the object tracking algorithm, and the method comprises the following steps:
s141, performing target detection on the aerial image of the short-focus camera based on a target detection algorithm to obtain a plurality of targets;
s142, clustering a plurality of targets by taking the distance as a reference based on a clustering algorithm to obtain a target cluster, and determining the weighted average sum of the confidence coefficients of the plurality of targets as the confidence coefficient of the target cluster;
s143, determining a preselected target cluster where the target to be tracked is located in the target clusters through an IOU matching and filtering track tracking mode.
In a possible implementation manner, the obtained target cluster is subjected to data association in an IOU+filtering mode, and the IOU measurement mode is as follows:
and distributing associated data for each target cluster by adopting a local greedy mode, and considering that matching is successful as long as the IOU metric value of a certain target cluster and the IOU metric value of the previous frame are greater than a threshold value. The Kalman filtering is used for generating an estimated value, and performing data fusion with the observed value, so that the robustness of the tracker is improved.
S144, extracting a local image where the preselected target cluster is located;
s145, inputting the local image and the aerial image of the long-focus camera into a twin neural network for feature matching to obtain a plurality of feature matching targets;
s146, determining an optimal feature matching target from the feature matching targets, and determining the optimal feature matching target as a target to be tracked. As shown in figure 4 of the drawings,
in a possible implementation mode, the twin neural network adopts Vision-transducer as a backbone network, and an attention module is added to improve network performance; data enhancement and HOG+color feature fusion are used to inhibit image blurring and deformation caused by aerial video shake of the ornithopter.
In a possible implementation, the clustering algorithm in the target tracking algorithm adopts a k-means algorithm to modify the objects tracked by the IOU and the track from small targets to target clusters.
S105, the airborne vision processing module acquires the pixel position of the target to be tracked from the aerial video stream of the ornithopter through a target tracking algorithm.
S51, obtaining the left upper-corner abscissa, width and height of the target to be tracked: x, y, w, h.
S52, calculating to obtain the central coordinates of the binding box:
x center ＝x+0.5*w
y center ＝y+0.5*h.
s106, the airborne vision processing module controls the rotation of the airborne camera cradle head module through the cradle head controller according to the pixel position of the target to be tracked, so as to track the target to be tracked;
in a possible implementation manner, the on-board vision processing module controls the on-board camera pan-tilt module to rotate through the pan-tilt controller according to the pixel position of the target to be tracked, and the on-board vision processing module comprises:
the pixel position of the target to be tracked is converted into the rotation angle of the airborne camera cradle head module through the following formula:
wherein (θ) x ,θ y ) Indicating the rotation angle theta of the airborne camera cradle head module x And represents the abscissa, θ, of the rotation angle y An ordinate representing the rotation angle; (X) T ,Y T ) Representing the pixel position of the object to be tracked, (X) W ,Y W ) Representing image resolution, f representing focal length of the short-focus camera, a×b representing image plane size;
and controlling the rotation of the airborne camera tripod head module through the tripod head controller according to the rotation angle of the airborne camera tripod head module.
In a possible implementation, after the position of the target in the image is obtained, the on-board camera pan-tilt module needs to be controlled to aim at the target, and the rotation of the on-board camera pan-tilt module needs to rely on a visual servo control algorithm. And the visual servo control algorithm of the airborne camera tripod head module is used for generating a control signal according to the position of the target in the image, controlling the tripod head to rotate through the tripod head controller, and locking the target in the center of the field of view of the short-focus camera.
The visual servo control algorithm comprises two parts of a mathematical modeling of the airborne camera tripod head module and a cascade double-closed-loop controller of the airborne camera tripod head module. The mathematical modeling of the airborne camera tripod head module is used for establishing a mathematical model of the airborne camera tripod head module; the cascade double closed-loop controller of the on-board camera holder module is used for receiving the visual servo signals and outputting control signals of the on-board camera holder module.
Firstly, a mathematical model of an airborne camera cradle head module needs to be obtained, and the specific processing mode is as follows:
the relation between the torque and the current obtained from the characteristics of the brushless DC motor is:
T m is the electromagnetic torque of the motor; i d Is a current; k (K) E Is a constant related to the motor structure; k (K) T Is the torque coefficient of the motor. The mass center of the cradle head is positioned on the axis, so that the load torque T can be considered to be ignored L And without energy storage elements, the equation of motion for a single axis rotation can be obtained as follows, taking into account the low damping that may be present:
j is the rotational inertia of the system; omega is the rotation speed; t is time; c is damping. The link can be considered as a linear first-order inertial link, and the transfer function is as follows:
wherein: k is a proportionality constant; t is the transition time. Finally, the mathematical model of the cradle head can be obtained through parameter identification.
The pan-tilt controller adopts cascade PID double closed-loop control, after a target angle is set, the target angle is converted into corresponding current through a correction link, the corresponding current is input into a driver, a motor is driven by the driver to output a certain angular velocity, an angle is output through an integration link, and the output angle is fed back to a comparison element.
In a possible implementation manner, the position of the target in the image has a mapping relationship with the rotation angle of the pan-tilt, so that the position of the target can be converted into the rotation angle of the pan-tilt, specifically as follows:
the pixel position of the target to be tracked is converted into the rotation angle of the airborne camera cradle head module through the following formula:
wherein (θ) x ,θ y ) Indicating the rotation angle of the airborne camera cradle head module, (X) T ,Y T ) Representing the pixel position of the object to be tracked, (X) W ,Y W ) Representing image resolution, f representing focal length of the short-focus camera, a×b representing image plane size;
finally, the rotation angle of the airborne camera tripod head module is transmitted to the tripod head through serial port communication, and the airborne camera tripod head module is controlled to rotate through the tripod head controller, so that the target tracking of the fully-autonomous flapping-wing flying robot can be realized.
S107, the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through the camera mapping relation.
In one possible embodiment, the depth value z of the image is due to the fact that the two cameras are not very different in mounting angle and mounting position c Can be considered equal.
The mapping relation of the short-focus camera coordinate system to the pixel coordinate system is as follows:
the mapping relation of the tele camera coordinate system to the pixel coordinate system is as follows:
the coordinate system relationship between the two cameras is as follows:
the mapping relation between the short-focus camera and the long-focus camera can be obtained as follows:
in the above formula, the superscript short represents the short-focus camera parameter, and the superscript long represents the long-focus camera parameter; z c Is the image depth; x, Y, Z is the coordinate of a point in the world coordinate system, and x and y are the pixel coordinates of the point; f (f) x 、f y The focal length reference of the camera, u and v are the optical center reference of the camera.
S108, circularly executing the steps S105 to S107 until the real-time tracking of the target to be tracked is completed.
The invention provides a target tracking scheme of a flapping-wing flying robot based on long and short focal length double cameras for the first time, and adopts two methods to improve aiming at the characteristics of small aerial video targets and few characteristics of the flapping-wing flying robot: the clustering algorithm is adopted to reduce the size to be large, the target cluster is tracked, and the target matching range is further narrowed; the hawk eyes double-fovea structure is used for reference, and long and short focal cameras are cooperatively used to extract the characteristics of more targets. Aiming at the characteristic of aerial video shake of the ornithopter flying robot, a twin neural network with stronger robustness is adopted for target matching. And a scheme of cloud deck control after image target tracking is completed is provided.
Fig. 5 is a schematic diagram of an object tracking system for a ornithopter flying robot of the present invention, the system 200 being used for the object tracking of the ornithopter flying robot, the system 200 comprising: a flapping wing flying robot 210, an onboard camera pan-tilt module 220, and an onboard vision processing module 230;
the on-board vision processing module 220 is a Jetson development board loaded with a Linux system and configured with a TensorRT deep learning environment, and is used for receiving images of a long-focus camera and a short-focus camera, running a target tracking method and sending a control signal to a cradle head. The camera is connected with a camera through an RJ45 port, an aerial image of the flapping wing flying robot is transmitted through an RTSP video stream with a fixed IP address, the aerial image is connected with an onboard camera holder module 220 through a serial port, and the holder is controlled to rotate by sending the postures of a pitching shaft and a yawing shaft. The on-board vision processing module comprises a target tracking algorithm based on clustering and a twin neural network.
The airborne vision processing module 230 is used for selecting a target to be tracked based on the short-focus camera aerial image and a target tracking algorithm; the airborne vision processing module 230 acquires the pixel position of the target to be tracked from the aerial video stream of the ornithopter robot 210 through a target tracking algorithm; the on-board vision processing module 230 controls the on-board camera pan-tilt module 220 to rotate through the pan-tilt controller according to the pixel position of the target to be tracked, so as to track the target to be tracked; the on-board vision processing module 230 obtains the position of the target to be tracked in the aerial image of the tele camera through the camera mapping relation.
The airborne camera cradle head module 220 is a cradle head mounted on the ornithopter flying robot 210 and provided with a long-focus camera and a short-focus camera, the cradle head carrying a pair of long-focus camera and short-focus camera is shown in fig. 6, the airborne camera cradle head module 220 is controlled to rotate by a cradle head controller, and is used for acquiring aerial videos of the ornithopter flying robot 210, and has two degrees of freedom of a pitching axis and a yawing axis, and the received signals of the airborne visual processing module 230 are rotated to enable a target to be located in the center of the visual fields of the short-focus camera and the long-focus camera.
The on-board camera pan-tilt module 220 is configured to obtain an aerial video stream of the ornithopter robot 210, where the aerial video stream includes an aerial image of a tele camera and an aerial image of a short-focus camera.
Optionally, initializing the on-board vision processing module 230230 includes:
s11, initializing a maximum range of clustering target clusters according to a preset range;
s12, initializing filter parameters according to preset parameters;
s13, loading the twin network model.
Optionally, the target tracking algorithm includes a target detection algorithm, a clustering algorithm, and a twin neural network;
the on-board vision processing module 230 is further configured to:
s41, performing target detection on the aerial image of the short-focus camera based on a target detection algorithm to obtain a plurality of targets;
s42, clustering a plurality of targets by taking the distance as a reference based on a clustering algorithm to obtain a target cluster, and determining the weighted average sum of the confidence coefficients of the plurality of targets as the confidence coefficient of the target cluster;
s43, determining a preselected target cluster where a target to be tracked is located in the target clusters through an IOU matching and filtering track tracking mode respectively;
s44, extracting a local image where the preselected target cluster is located;
s45, inputting the local image and the aerial image of the long-focus camera into a twin neural network for feature matching to obtain a plurality of feature matching targets;
s46, determining an optimal feature matching target from the feature matching targets, and determining the optimal feature matching target as a target to be tracked.
Optionally, the twin neural network module adopts Vision-transducer as a backbone network, and adds an attention module to improve network performance; data enhancement and hog+color feature fusion are used to suppress image blurring and distortion due to aerial video shake of the ornithopter robot 210.
Optionally, a clustering algorithm in the target tracking algorithm adopts a k-means algorithm to modify the IOU and the track tracked object from a small target to a target cluster.
Optionally, the on-board vision processing module 230 in S6 controls the on-board camera pan-tilt module 220 to rotate through the pan-tilt controller according to the pixel position of the target to be tracked, including:
the pixel position of the target to be tracked is converted into the rotation angle of the on-board camera pan-tilt module 220 by the following formula:
wherein (θ) x ,θ y ) Indicating the rotation angle of the on-board camera pan-tilt module 220, (X) T ,Y T ) Representing the pixel position of the object to be tracked, (X) W ,Y W ) Representing image resolution, f representing focal length of the short-focus camera, representing image plane size;
according to the rotation angle of the airborne camera tripod head module 220, the rotation of the airborne camera tripod head module is controlled by the tripod head controller.
The invention provides a target tracking scheme of a flapping-wing flying robot based on long and short focal length double cameras for the first time, and adopts two methods to improve aiming at the characteristics of small aerial video targets and few characteristics of the flapping-wing flying robot: the clustering algorithm is adopted to reduce the size to be large, the target cluster is tracked, and the target matching range is further narrowed; the hawk eyes double-fovea structure is used for reference, and long and short focal cameras are cooperatively used to extract the characteristics of more targets. Aiming at the characteristic of aerial video shake of the ornithopter flying robot, a twin neural network with stronger robustness is adopted for target matching. And a scheme of cloud deck control after image target tracking is completed is provided.
Fig. 7 is a schematic structural diagram of an electronic device 300 according to an embodiment of the present invention, where the electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 301 and one or more memories 302, where at least one instruction is stored in the memories 302, and the at least one instruction is loaded and executed by the processors 301 to implement the following steps of a method for modifying a dynamic structural interval model of a multi-cell overlap ratio:
s1, initializing an airborne vision processing module;
s2, initializing an airborne camera holder module;
s3, an airborne camera holder module acquires an aerial video stream of the ornithopter flying robot, wherein the aerial video stream comprises an aerial image of a long-focus camera and an aerial image of a short-focus camera;
s4, selecting a target to be tracked by the airborne vision processing module based on the short-focus camera aerial image and a target tracking algorithm;
s5, the airborne vision processing module acquires the pixel position of the target to be tracked from the aerial video stream of the ornithopter through a target tracking algorithm;
s6, the airborne vision processing module controls the rotation of the airborne camera cradle head module through the cradle head controller according to the pixel position of the target to be tracked, so as to track the target to be tracked;
s7, the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through a camera mapping relation;
s8, circularly executing the steps S5 to S7 until the real-time tracking of the target to be tracked is completed.
In an exemplary embodiment, a computer readable storage medium, such as a memory comprising instructions executable by a processor in a terminal to perform the above-described ornithopter flying robot target tracking method, is also provided. For example, the computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Claims (10)
1. The method is characterized by being realized by a flapping-wing flying robot target tracking system, wherein the flapping-wing flying robot target tracking system comprises a flapping-wing flying robot, an airborne camera holder module and an airborne vision processing module;
the method comprises the following steps:
s1, initializing the airborne vision processing module;
s2, initializing the airborne camera holder module;
s3, the airborne camera holder module acquires an aerial video stream of the ornithopter flying robot, wherein the aerial video stream comprises an aerial image of a long-focus camera and an aerial image of a short-focus camera;
s4, the airborne vision processing module selects a target to be tracked based on the short-focus camera aerial image and a target tracking algorithm;
s5, the airborne vision processing module acquires the pixel position of the target to be tracked from the aerial video stream of the ornithopter through the target tracking algorithm;
s6, the airborne vision processing module controls the rotation of the airborne camera pan-tilt module through the pan-tilt controller according to the pixel position of the target to be tracked, so as to track the target to be tracked;
s7, the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through a camera mapping relation;
s8, circularly executing the steps S5 to S7 until the real-time tracking of the target to be tracked is completed.
2. The method of claim 1, wherein initializing the on-board vision processing module in S1 comprises:
s11, initializing a maximum range of a clustering target cluster of the airborne vision processing module according to a preset range;
s12, initializing filter parameters of the airborne vision processing module according to preset parameters;
s13, loading a twin network model of the airborne vision processing module.
3. The method of claim 1, wherein in S4, the target tracking algorithm comprises a target detection algorithm, a clustering algorithm, and a twin neural network.
4. The method of claim 3, wherein in S4, the on-board vision processing module selects an object to be tracked based on the short-focal camera aerial image and the object tracking algorithm, comprising:
s41, performing target detection on the aerial image of the short-focus camera based on the target detection algorithm to obtain a plurality of targets;
s42, clustering the plurality of targets by taking the distance as a reference based on the clustering algorithm to obtain a target cluster, and determining the weighted average sum of the confidence coefficients of the plurality of targets as the confidence coefficient of the target cluster;
s43, determining a preselected target cluster where a target to be tracked is located in the target clusters through an IOU matching and filtering track tracking mode respectively;
s44, extracting a local image of the preselected target cluster;
s45, inputting the local image and the aerial image of the tele camera into the twin neural network for feature matching to obtain a plurality of feature matching targets;
s46, determining an optimal feature matching target from the feature matching targets, and determining the optimal feature matching target as a target to be tracked.
5. A method according to claim 3, wherein the twin neural network employs Vision-transducer as a backbone network, and adds an attention module to improve network performance; data enhancement and HOG+color feature fusion are used to inhibit image blurring and deformation caused by aerial video shake of the ornithopter.
6. A method according to claim 3, wherein the clustering algorithm in the target tracking algorithm employs a k-means algorithm to modify the objects tracked by the IOU and the track from small targets to target clusters.
7. The method according to claim 1, wherein in S6, the on-board vision processing module controls rotation of the on-board camera pan-tilt module by the pan-tilt controller according to the pixel position of the target to be tracked, including:
the pixel position of the target to be tracked is converted into the rotation angle of the airborne camera cradle head module through the following formula:
wherein (θ) x ,θ y ) Indicating the rotation angle theta of the airborne camera cradle head module x And represents the abscissa, θ, of the rotation angle y An ordinate representing the rotation angle; (X) T ,Y T ) Representing the pixel position of the object to be tracked, (X) W ,Y W ) Representing image resolution, f representing focal length of the short-focus camera, a×b representing image plane size;
and controlling the rotation of the airborne camera tripod head module through the tripod head controller according to the rotation angle of the airborne camera tripod head module.
8. A ornithopter flying robot target tracking system for implementing a method of ornithopter flying robot target tracking, the system comprising: the system comprises a flapping wing flying robot, an airborne camera holder module and an airborne vision processing module;
the flapping wing flying robot target tracking system comprises a flapping wing flying robot, an airborne camera holder module and an airborne vision processing module, wherein the airborne camera holder module is a holder which is carried on the flapping wing flying robot and is provided with a long-focus camera and a short-focus camera, the airborne camera holder module controls rotation through a holder controller, and the airborne vision processing module comprises a target tracking algorithm based on clustering and twin neural network;
the airborne camera holder module is used for acquiring an aerial video stream of the ornithopter flying robot, wherein the aerial video stream comprises an aerial image of a long-focus camera and an aerial image of a short-focus camera;
the airborne vision processing module is used for selecting a target to be tracked based on the short-focus camera aerial image and the target tracking algorithm; the airborne vision processing module acquires the pixel position of the target to be tracked from an aerial video stream of the ornithopter through the target tracking algorithm; the airborne vision processing module controls the rotation of the airborne camera tripod head module through the tripod head controller according to the pixel position of the target to be tracked, so as to track the target to be tracked; and the airborne vision processing module obtains the position of the target to be tracked in the aerial image of the tele camera through a camera mapping relation.
9. The system of claim 8, wherein the target tracking algorithm comprises a target detection algorithm, a clustering algorithm, and a twin neural network.
10. The system of claim 9, wherein the on-board vision processing module is further configured to:
s41, performing target detection on the aerial image of the short-focus camera based on the target detection algorithm to obtain a plurality of targets;
s42, clustering the plurality of targets by taking the distance as a reference based on the clustering algorithm to obtain a target cluster, and determining the weighted average sum of the confidence coefficients of the plurality of targets as the confidence coefficient of the target cluster;
s43, determining a preselected target cluster where a target to be tracked is located in the target clusters through an IOU matching and filtering track tracking mode respectively;
s44, extracting a local image of the preselected target cluster;
s45, inputting the local image and the aerial image of the tele camera into the twin neural network for feature matching to obtain a plurality of feature matching targets;
s46, determining an optimal feature matching target from the feature matching targets, and determining the optimal feature matching target as a target to be tracked.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310345807.8A CN116360492B (en) | 2023-04-03 | 2023-04-03 | Object tracking method and system for flapping wing flying robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310345807.8A CN116360492B (en) | 2023-04-03 | 2023-04-03 | Object tracking method and system for flapping wing flying robot |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116360492A true CN116360492A (en) | 2023-06-30 |
| CN116360492B CN116360492B (en) | 2024-01-30 |
Family
ID=86931689
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310345807.8A Active CN116360492B (en) | 2023-04-03 | 2023-04-03 | Object tracking method and system for flapping wing flying robot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116360492B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117676077A (en) * | 2023-10-31 | 2024-03-08 | 中国人民解放军32806部队 | Unmanned aerial vehicle visual target tracking system and method based on finite state machine |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101720027A (en) * | 2009-11-27 | 2010-06-02 | 西安电子科技大学 | Method for cooperative acquisition of multi-target videos under different resolutions by variable-focus array camera |
| US20110079703A1 (en) * | 2009-10-02 | 2011-04-07 | Teledyne Scientific & Imaging, Llc | Object tracking system |
| US20140372073A1 (en) * | 2011-05-04 | 2014-12-18 | Jacques Georgy | Two-stage filtering based method for multiple target tracking |
| CN106184746A (en) * | 2016-09-09 | 2016-12-07 | 北京航空航天大学 | A kind of Fructus Forsythiae bionical butterfly flapping wing aircraft |
| CN109828610A (en) * | 2019-01-30 | 2019-05-31 | 华通科技有限公司 | Long Range Target Tracking system and method based on long short focus camera group |
| US20200182995A1 (en) * | 2015-07-17 | 2020-06-11 | Origin Wireless, Inc. | Method, apparatus, and system for outdoor target tracking |
| CN111899283A (en) * | 2020-07-30 | 2020-11-06 | 北京科技大学 | Video target tracking method |
| CN113139620A (en) * | 2021-05-14 | 2021-07-20 | 重庆理工大学 | End-to-end multi-target detection and tracking joint method based on target association learning |
| CN113808166A (en) * | 2021-09-15 | 2021-12-17 | 西安电子科技大学 | Single-target tracking method based on clustering difference and depth twin convolutional neural network |
| CN113850848A (en) * | 2021-09-26 | 2021-12-28 | 大连海事大学 | Unmanned boat-mounted marine radar and visual image cooperative marine multi-target long-term detection and tracking method |
| CN114972434A (en) * | 2022-06-06 | 2022-08-30 | 电子科技大学 | End-to-end multi-target tracking system for cascade detection and matching |
| CN115272393A (en) * | 2022-08-01 | 2022-11-01 | 中国电子科技集团公司第五十四研究所 | Video stream target tracking method and device for unmanned aerial vehicle and storage medium |
-
2023
- 2023-04-03 CN CN202310345807.8A patent/CN116360492B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110079703A1 (en) * | 2009-10-02 | 2011-04-07 | Teledyne Scientific & Imaging, Llc | Object tracking system |
| CN101720027A (en) * | 2009-11-27 | 2010-06-02 | 西安电子科技大学 | Method for cooperative acquisition of multi-target videos under different resolutions by variable-focus array camera |
| US20140372073A1 (en) * | 2011-05-04 | 2014-12-18 | Jacques Georgy | Two-stage filtering based method for multiple target tracking |
| US20200182995A1 (en) * | 2015-07-17 | 2020-06-11 | Origin Wireless, Inc. | Method, apparatus, and system for outdoor target tracking |
| CN106184746A (en) * | 2016-09-09 | 2016-12-07 | 北京航空航天大学 | A kind of Fructus Forsythiae bionical butterfly flapping wing aircraft |
| CN109828610A (en) * | 2019-01-30 | 2019-05-31 | 华通科技有限公司 | Long Range Target Tracking system and method based on long short focus camera group |
| CN111899283A (en) * | 2020-07-30 | 2020-11-06 | 北京科技大学 | Video target tracking method |
| CN113139620A (en) * | 2021-05-14 | 2021-07-20 | 重庆理工大学 | End-to-end multi-target detection and tracking joint method based on target association learning |
| CN113808166A (en) * | 2021-09-15 | 2021-12-17 | 西安电子科技大学 | Single-target tracking method based on clustering difference and depth twin convolutional neural network |
| CN113850848A (en) * | 2021-09-26 | 2021-12-28 | 大连海事大学 | Unmanned boat-mounted marine radar and visual image cooperative marine multi-target long-term detection and tracking method |
| CN114972434A (en) * | 2022-06-06 | 2022-08-30 | 电子科技大学 | End-to-end multi-target tracking system for cascade detection and matching |
| CN115272393A (en) * | 2022-08-01 | 2022-11-01 | 中国电子科技集团公司第五十四研究所 | Video stream target tracking method and device for unmanned aerial vehicle and storage medium |
Non-Patent Citations (7)
| Title |
|---|
| HONGWEI SHE等: "Design and implementation of a target tracking and ranging system based on binocular vision", 《2021 IEEE INTERNATIONAL CONFERENCE ON RECENT ADVANCES IN SYSTEMS SCIENCE AND ENGINEERING (RASSE)》 * |
| NAJMADDIN ABO MOSALI等: "Twin Delayed Deep Deterministic Policy Gradient-Based Target Tracking for Unmanned Aerial Vehicle With Achievement Rewarding and Multistage Training", 《IEEE ACCESS》 * |
| 付强等: "仿生扑翼飞行器的视觉感知系统研究进展", 工程科学学报, no. 12 * |
| 余志超等: "结合深度轮廓特征的改进孪生网络跟踪算法", 西安电子科技大学学报, no. 03 * |
| 沈旭等: "机载平台下基于深度检测网络的目标跟踪重捕算法", 红外技术, no. 07 * |
| 靳佳婷: "基于深度学习的多目标跟踪算法研究", 《中国优秀硕士学位论文电子全文库 信息科技辑》 * |
| 黄一凡等: "基于二自由度转台的双目视觉跟踪技术研究", 电子设计工程, no. 14 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117676077A (en) * | 2023-10-31 | 2024-03-08 | 中国人民解放军32806部队 | Unmanned aerial vehicle visual target tracking system and method based on finite state machine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116360492B (en) | 2024-01-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111932588B (en) | Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning | |
| CN107862704B (en) | Target tracking method and system and holder camera used by same | |
| CN109857144B (en) | Unmanned aerial vehicle, unmanned aerial vehicle control system and control method | |
| JP5990453B2 (en) | Autonomous mobile robot | |
| CN105045279A (en) | System and method for automatically generating panorama photographs through aerial photography of unmanned aerial aircraft | |
| US12007794B2 (en) | Method and apparatus for tracking moving target and unmanned aerial vehicle | |
| CN109398688A (en) | A kind of rotor flying double mechanical arms target positioning grasping system and method | |
| CN109816698B (en) | Unmanned aerial vehicle visual target tracking method based on scale self-adaptive kernel correlation filtering | |
| CN106973221B (en) | Unmanned aerial vehicle camera shooting method and system based on aesthetic evaluation | |
| CN116360492B (en) | Object tracking method and system for flapping wing flying robot | |
| CN110498039A (en) | A kind of intelligent monitor system based on bionic flapping-wing flying vehicle | |
| Fu et al. | Vision-based obstacle avoidance for flapping-wing aerial vehicles | |
| CN115291536B (en) | Visual unmanned aerial vehicle tracking ground target semi-physical simulation platform verification method | |
| CN110568860A (en) | Unmanned aerial vehicle return method and device and unmanned aerial vehicle | |
| Valenti et al. | An autonomous flyer photographer | |
| CN116149193B (en) | Anti-disturbance control method and system for rotor unmanned aerial vehicle based on vision | |
| Koo et al. | A jellyfish distribution management system using an unmanned aerial vehicle and unmanned surface vehicles | |
| US11869236B1 (en) | Generating data for training vision-based algorithms to detect airborne objects | |
| CN114089779B (en) | Autonomous control method and system for vision of aerial robot in GPS refusing environment | |
| Qin et al. | Visual-based tracking and control algorithm design for quadcopter UAV | |
| Nair et al. | Autonomous Precision Landing with UAV and Auto charging | |
| CN113206951B (en) | Real-time electronic image stabilization method based on flapping wing flight system | |
| CN116339321A (en) | Global information driven distributed multi-robot reinforcement learning formation surrounding method based on 5G communication | |
| WO2023097769A1 (en) | Unmanned ground vehicle-unmanned aerial vehicle collaborative autonomous tracking and landing method | |
| Sriram et al. | Autonomous drone for defence machinery maintenance and surveillance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |