CN115933718A - Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition - Google Patents
Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition Download PDFInfo
- Publication number
- CN115933718A CN115933718A CN202211385768.6A CN202211385768A CN115933718A CN 115933718 A CN115933718 A CN 115933718A CN 202211385768 A CN202211385768 A CN 202211385768A CN 115933718 A CN115933718 A CN 115933718A
- Authority
- CN
- China
- Prior art keywords
- unmanned aerial
- aerial vehicle
- panoramic
- image
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention discloses an unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target identification, which takes a quad-rotor unmanned aerial vehicle carrying a panoramic camera as a carrier, integrates a panoramic SLAM algorithm with a target identification and obstacle avoidance algorithm, and realizes autonomous flight of the unmanned aerial vehicle in a position environment. The unmanned aerial vehicle autonomous flight technology firstly depends on the panoramic SLAM to complete the construction of an environment map in an unknown environment and simultaneously performs self-positioning, utilizes a YOLOv5 algorithm to identify an object target in a visual field, judges the position information of an obstacle and then combines positioning information, and performs autonomous obstacle avoidance and path planning in real time by a D-Lite algorithm. The system has the following advantages: the device has the advantages of small volume, low cost, wide applicable scenes, accurate positioning, automatic and intelligent completion of the identification and obstacle avoidance process, reduction of manual intervention and improvement of the operation efficiency.
Description
Technical Field
The invention belongs to the field of remote sensing mapping, and particularly relates to an unmanned aerial vehicle autonomous flight technology integrating panoramic SLAM and target identification.
Background
With the development of unmanned aerial vehicle platforms and sensor technologies, unmanned aerial vehicle systems are continuously developed in the directions of identification, autonomy, perception and behavior, so that the development of autonomous flying unmanned aerial vehicles is gradually developed. In the autonomous flight unmanned aerial vehicle field, unmanned aerial vehicle receives the command of taking off the back, and under the condition of not further controlling it, the airborne sensor can help unmanned aerial vehicle perception surrounding environment, fixes a position self motion and state to reach the purpose that unmanned aerial vehicle independently controlled and fly. In the traditional industrial routing inspection, the manual visual routing inspection is required, the labor intensity is high, the severe environment can bring greater challenges to the routing inspection, and the autonomous flying unmanned aerial vehicle technology can realize the refined and automatic routing inspection of equipment such as a wind power tower cylinder and a power transmission line in the environments such as high altitude, forest and the like; in the field of emergency rescue, an environment map can be rapidly constructed by the autonomous flight unmanned aerial vehicle technology, a specified target can be searched, and rescue is assisted.
The airborne sensor of the unmanned aerial vehicle is the key for completing autonomous control and flight. Onboard sensors include vision cameras, lidar, IMU, GNSS, and the like. Usually, in order to complete accurate and reliable autonomous flight, a multi-source sensor needs to be integrated, which puts higher requirements on the loading capacity and the endurance of the unmanned aerial vehicle. Meanwhile, the hardware cost, the integration difficulty and the safety consideration of the multi-source sensor also restrict the application of the unmanned aerial vehicle in autonomous flight.
The characteristics of different sensors play a decisive role in the degree of autonomy of unmanned aerial vehicle flight inspection. GNSS and IMU belong to carrier state sensors and cannot sense the surrounding environment. Under the promotion of the development of SLAM technology, a vision camera and a laser radar can not only estimate the carrier state, but also sense and establish a map of the surrounding environment. Compared with a visual camera and a laser radar, the laser radar can complete positioning and mapping in an absolute scale, but the reliability and accuracy of target identification are insufficient, and the laser SLAM at a higher frame rate is very easily influenced by high-speed motion and large-rotation-angle motion. The vision technology can complete SLAM positioning and mapping, has the advantage of real-time target identification, and is very suitable for target obstacle avoidance and target inspection. In addition, the camera is small in size, light in weight, small in power load and low in cost, and is very suitable for the quad-rotor unmanned aerial vehicle with a small wheelbase. Visual SLAM includes branches of monocular, binocular, and RGB-D. The mainstream sensor that present unmanned aerial vehicle carried on is monocular, binocular and degree of depth camera, and corresponding vision SLAM algorithm all has certain limitation. The monocular SLAM has a small field angle, the amount of acquired environment information is limited, 360-degree perception of the surrounding environment cannot be realized, and the monocular camera scale estimation precision suddenly drops along with the increase of accumulated errors; the field angle of the binocular SLAM is not increased on the basis of solving the scale drift, and the characteristic points can be lost in the case of violent movement. The field angle can be increased through the mode of multi-camera combination, but can bring the increase of unmanned aerial vehicle load simultaneously, be unfavorable for continuation of the journey work, and the demarcation between the camera can further introduce the error. Visual SLAM is not widely used in large scene mobile mapping. Aiming at the problems, the panoramic vision has the advantages of 360-degree environment perception and rapid and complete information acquisition, and the omnidirectional visual angle can improve the reliability and the orientation precision of image inter-frame matching and is not influenced by the motion state. However, the current algorithms based on panoramic vision SLAM are applied less frequently, most algorithms are used for data acquisition, and a complete set of SLAM solution is lacked and applied to industrial application.
In conclusion, in the autonomous unmanned aerial vehicle, the panoramic vision sensor can provide the maximum flight inspection autonomy for the unmanned aerial vehicle with the least hardware and integration cost, and meanwhile, the functions of environment perception, target identification and obstacle avoidance are realized, so that how to provide the autonomous flight technology of the unmanned aerial vehicle integrating the panoramic SLAM and the target identification is a key problem to be solved in the field.
Disclosure of Invention
The invention provides an unmanned aerial vehicle autonomous flight technology integrating panoramic SLAM and target identification, a panoramic vision camera is used as a unique unmanned aerial vehicle platform sensor, and real-time path planning and obstacle avoidance are carried out on the basis of solving the problems of uncertain scale and scale drift of the visual SLAM by integrating the panoramic SLAM technology and the target identification technology, so that autonomous flight of the unmanned aerial vehicle in a position environment is realized.
The autonomous flight unmanned aerial vehicle system built by the invention is suitable for map building and self-positioning in an unknown environment, and simultaneously carries out target identification on real objects in the environment, real-time autonomous obstacle avoidance and path planning. To realize the above functions, the technical problems to be solved by the present invention are: building a light and small panoramic SLAM unmanned aerial vehicle platform; panoramic SLAM techniques; and recognizing and avoiding the obstacle by the panoramic visual target.
The invention provides an unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target identification, which comprises the following steps:
step 1, building a light and small unmanned aerial vehicle platform; selecting a small-wheelbase quad-rotor unmanned aerial vehicle as a flight platform, carrying a panoramic camera at the front end of the unmanned aerial vehicle platform, and effectively acquiring 360-degree visual angle information; transmitting the video information to an airborne computer in real time, and calculating in real time to obtain the flight attitude of the unmanned aerial vehicle and scene interesting target information; then, the flight attitude data is transmitted into a flight control system to autonomously control the unmanned aerial vehicle to fly;
step 2, acquiring an environment image by using a panoramic camera, operating a real-time panoramic SLAM, and completing the positioning and autonomous flight of the unmanned aerial vehicle;
and 3, step 3: and detecting the object in the real-time image based on a target identification technology, planning an optimal path by taking the target object as a terminal point by combining positioning information output by the SLAM, and finishing autonomous obstacle avoidance flight.
Further, the specific implementation of step 2 includes the following sub-steps;
step 2.1, constructing an imaging model of the panoramic image, including a single-lens imaging model and a multi-lens imaging model, realizing high-efficiency splicing of the panoramic image, and completing spherical mapping of image data;
2.2, performing feature extraction and matching splicing on the spherical image mapped in the step 2.1 by using a SPHORB algorithm;
and 2.3, solving a pose transformation relation between adjacent frames by epipolar constraint according to the feature point pairs extracted in the step 2.2, optimizing the three-dimensional coordinates of the feature points and the camera pose by adopting nonlinear optimization to minimize a reprojection error, outputting the optimal camera pose to finish positioning, and transforming and splicing the feature points by the pose to finish image construction.
Further, the specific implementation manner of step 2.1 is as follows;
firstly, establishing a panoramic camera coordinate system, abstracting a panoramic camera carried on an unmanned aerial vehicle platform into a dome camera model, considering the single-lens condition, enabling the spherical center to coincide with the optical center of a camera, setting O-xyz as a camera coordinate system, mapping an object point P at a point P on an image plane, and setting coordinates in the corresponding image coordinate system as (u, v); meanwhile, the ray OP intersects with the spherical surface with O as the center of the sphere at the point P s ；
Defining spherical mapping as function mapping relation
An arbitrary point p (u, v) on the image is mapped onto a spherical surface with a certain radius and expressed as ^ based on the ball coordinate>
If the right hand is used for determination, the thumb rotates towards the direction of the y-axis finger, and the x-axis and the z-axis rotate towards the direction of other fingers by an angle theta; />
For right-handed determination, the thumb rotates in the direction of the z-axis, the x-axis and the y-axis toward the other fingers>
An angular value;
let alpha be a vector
The included angle between the projection on the plane O-yz and the axis is larger and smaller, and beta is a vector->
The size of an included angle between the plane O-yz and the plane; for an actual image, the coordinates in the directions of the u axis and the v axis are both finite values, so that the value ranges of alpha and beta are (-pi/2, pi/2), and according to the space geometric relationship, the following formula (2.1) is provided: />
Wherein f is the focal length of the camera and u 0 ,v 0 ) Biasing for the principal point; in effect, the above equation provides a mapping from the image coordinate system to the local angle in the sphere
Namely, the one-to-one corresponding relation between the points on the image plane and the partial spherical surface points;
in the multi-lens panoramic camera, each sub-camera has an independent camera coordinate system, the multi-lens panoramic camera coordinate system is based on a single-lens spherical coordinate system, but the optical centers of all the lenses are not overlapped due to the process and the volume of the lenses; therefore, when performing spherical mapping on the acquired image data, it is necessary to unify the sub-cameras of the multi-lens panoramic camera into an overall coordinate system, and then convert the pixel coordinates in the image acquired by each lens into the overall coordinate system through formula 2.1 to perform spherical mapping uniformly.
Further, the specific implementation manner of step 3 is as follows;
step 3.1, a YOLOv5 algorithm is used as a target recognition algorithm, a target data set of the panoramic image is input to train a neural network model, the trained model is used for carrying out real-time target recognition on the panoramic image transmitted in the step 2, and a recognition result is output;
step 3.2, calculating the speed, the acceleration and the angular velocity of the aircraft by using the time difference and the pose transformation between adjacent frames obtained in the step 2.3, and dynamically tracking the pose of the aircraft by a Kalman filtering navigation algorithm; and 3, performing three-dimensional flight path planning by using a D × Lite search algorithm according to the obstacle position information and the real-time positioning information of the aircraft identified in the step 3.1 in the flight process, and realizing real-time autonomous obstacle avoidance.
Further, the YOLOv5 network model in step 3.1 includes an input end, a reference network, a Neck network and a Head output end; the input end represents an input picture, the size of the input image is 608 × 608, and the input image can be scaled to the input size of the network at this stage and normalized and the like are performed; in the network training stage, YOLOv5 uses the Mosaic data to enhance the training speed of the operation promotion model and the accuracy of the network; the reference network represents a classifier network with excellent performance and is used for extracting general feature representation, and the CSPDarknet53 and Focus structures are used as the reference network; the CSPDarknet53 structure is designed to improve feature representation dimensionality in a backbone network by taking the design idea of CSPNet as reference, the Focus structure cuts an input picture through Slice operation, the size of the original input picture is 608 × 3, and a feature map of 304 × 12 is output after the Slice and Concat operation; then, a feature map with the size of 304 × 32 is output through a Conv layer with the number of channels being 32; the neutral network is positioned in the middle of the reference network and the Head output end, and the SPP module and the FPN + PAN module are improved by using the CSP2 structure designed by CSPnet for reference, so that the network feature fusion capability is enhanced; the Head output end is used for finishing the output of the target recognition result; the Yolov5 adopts a GIOU _ Loss function, and the function solves the problem when the boundary frames are not overlapped on the basis of the IOU, so that the detection precision of the algorithm is further improved.
Further, the specific implementation manner of step 3.2 is as follows;
firstly, the time difference and the pose change between two adjacent frames obtained in the step 2.3 are utilized to calculate the speed, the acceleration and the angular velocity of the aircraft, and the pose of the aircraft is dynamically tracked by utilizing a Kalman filtering navigation algorithm, so that autonomous navigation is realized; in the flying process, the Yolov5 algorithm in the step 3.1 is used for identifying and detecting the obstacles in the panoramic image, and according to the position information of the obstacles and the real-time positioning information of the aircraft, a D-Lite algorithm is used for carrying out three-dimensional flight path rapid planning on the unmanned aircraft when the target moves in the uncertain environment by adopting a D-Lite search algorithm, so that the autonomous obstacle avoidance in the flying process is realized.
The invention makes the following improvement on the traditional unmanned aerial vehicle flight technology, carries a panoramic vision sensor, instantly acquires rich large-range environment information, and combines with the SLAM technology to improve the positioning precision; the target identification technology is applied to the panoramic image, real-time target identification in a large-range environment is achieved, and high-robustness autonomous obstacle avoidance flight is achieved through a path planning technology according to self-positioning information output by the SLAM and identified obstacle and target information.
The invention has the following advantages: 1. the unmanned aerial vehicle platform adopts four rotor unmanned aerial vehicle of little wheel base, and the core sensor is panoramic camera, and the size is little, and is with low costs, and flexible high, suitable scene is extensive, is one set of light and small portable, can iterate the autonomic unmanned aerial vehicle platform system that updates. 2. The combination of the panoramic camera and the SLAM technology can acquire complete information of the environment in all directions in real time and realize high-precision self-positioning. 3. The panoramic image is used for target identification, so that the real-time performance and accuracy of identification are effectively improved, and the information dimension is increased. 4. And automatically avoiding the obstacle in real time according to the positioning information and the target identification information, so as to quickly avoid the obstacle and plan an optimal path. 5. The whole environment sensing and autonomous obstacle avoidance process is automatic and intelligent, manual intervention is not needed, and the working efficiency is improved.
Drawings
FIG. 1: the overall method of the invention is a flow chart.
FIG. 2 is a schematic diagram: unmanned aerial vehicle platform schematic diagram.
FIG. 3: panoramic SLAM technique flow diagram.
FIG. 4: and (4) a spherical imaging schematic diagram.
FIG. 5: and identifying and avoiding the whole flow chart.
FIG. 6: a flow diagram of a target identification technique.
FIG. 7: and (4) an autonomous obstacle avoidance technology flow chart.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to specific examples. It should be understood that the specific examples described herein are intended merely to illustrate the application and are not intended to limit the application.
The following describes a specific embodiment of the present invention with reference to fig. 1 to 7, which is an autonomous flight technology of an unmanned aerial vehicle integrating panoramic SLAM and target identification, and an overall method flowchart of the present invention is shown in fig. 1, and includes the following steps:
step 1: light small-size unmanned aerial vehicle platform design according to load weight such as the panoramic camera that carries on and size, designs rack construction, accomplishes the hardware and builds. As shown in fig. 2, the specific design of the hardware platform in this example is divided into the following three parts:
(1) The machine frame is integrally designed by adopting a 3K carbon fiber material, common external workpieces (such as a blade protector, a lifting foot rest and the like) are integrated with a horizontal structure of the machine body, the thickness of an integrated plate of the machine arm framework is 3mm, the thickness of the blade protector is 2mm, the lifting foot rest is perpendicular to the horizontal direction and forms an angle of 30 degrees, the machine arm framework is processed by adopting aluminum alloy, and a cushioning rubber cushion is arranged at the ground contact end.
(2) The vertical direction structure of frame divide into the three-layer, uses copper single-through post and aluminium system cavity screw thread post zonulae occludens that the quality is light, intensity is high between the layer, and the interval of a second floor is 25mm, and the interval of two three-layer is 30mm. From bottom to top, the first layer is only expanded on the arm keel integrated plate layer, the PixRacer flight controller is convexly installed, wiring of peripheral equipment is facilitated, and an ESP-8266 wireless network module is externally connected to the top flight control layer and used for wireless flight controller parameter detection and debugging; the lower layer of the blade protector and the second layer are positioned on the same plane, and a telemetering signal receiver and a BeneWake TFmini-S laser TOF radar are installed on a panel, close to the ground, of the second layer. The TOF radar is connected with the flight controller through a UART interface, the electronic speed regulator is connected into an AUX interface corresponding to the flight controller after being separated through a channel, the telemetering signal receiver is connected into an SBUS interface of the flight controller, and the camera is connected with the onboard computer through a USB interface; the third layer is similar to the first layer, only extends on the arm keel integrated plate, and is mainly provided with an onboard microcomputer, a USB expansion module and a serial port communication module.
(3) Compared with the common unmanned aerial vehicle flight platform with the wheelbase of 450mm or more, the wheelbase is reduced to 290mm in the aspect of chord length and load ratio, and meanwhile, the load-carrying performance is still kept. After all necessary equipment is arranged, the flying platform has the self weight of about 1.1kg, and the extra load can be still about 0.5kg. The extra load can be 45.4% in value.
(4) In the aspect of power arrangement, in consideration of the characteristics of high load capacity and low wheelbase of the motor, a motor capable of providing enough torque needs to be selected, and a combination of a 26.8 mm 1750KV brushless motor and a 6-inch three-blade propeller is adopted. In the aspect of system power supply, a 4S 3300mAh 25CLi-PO battery is used as a power supply, and a PM02V3 power distribution ammeter and a double-circuit UBEC are used for separated power supply, so that the influence of power consumption change of an electric appliance on a flight controller or an onboard computer is effectively avoided.
The small-wheelbase quad-rotor unmanned aerial vehicle is selected as a flight platform, has the characteristics of light weight, convenience, stable flight and the like, and can effectively acquire 360-degree visual angle information by carrying a panoramic camera at the front end of the unmanned aerial vehicle platform; transmitting the video information to an airborne computer in real time, and calculating in real time to obtain the flight attitude of the unmanned aerial vehicle and scene interesting target information; then, the flight attitude data is transmitted into a flight control system to autonomously control the unmanned aerial vehicle to fly; the whole set of system provides stable electric energy by 14.8V/16000mAh to ensure the normal flight of the unmanned aerial vehicle. After the system is built, the environment image can be collected, the real-time panoramic SLAM is operated, and the positioning and autonomous flight of the unmanned aerial vehicle are completed.
Step 2: the panoramic SLAM technology shown in the figure 3 firstly constructs an imaging model of a panoramic image, realizes the high-efficiency splicing of the panoramic image, then extracts and matches image features, and finally completes positioning through pose estimation and optimization. The specific process is as follows:
step 2.1: firstly, a panoramic camera coordinate system is established, a panoramic camera (the panoramic camera is used for acquiring a 360-degree panoramic image of the surrounding environment, a subsequent algorithm processes the image and positioning is completed) carried on an unmanned aerial vehicle platform is abstracted into a dome camera model, and the single-lens condition is considered to enable the sphere center to coincide with the camera optical center. As shown in fig. 4, O-xyz is the camera coordinate system, which is mapped to a point P on the image plane for an object point P, with the corresponding coordinates (u, v) in the image coordinate system. Meanwhile, the ray OP intersects with the spherical surface with O as the center of the sphere at the point P s 。
Defining spherical mapping as a functional mapping relation
An arbitrary point p (u, v) on the image is mapped onto a spherical surface with a certain radius and expressed as ^ based on the ball coordinate>
If the right hand is used for determination, the thumb rotates towards the direction of the y-axis finger, and the x-axis and the z-axis rotate towards the direction of other fingers by an angle theta; />
For right-hand rule, the thumb rotates in the direction of the z-axis, the x-axis and the y-axis toward the other fingers>
The angular value.
Let alpha be the vector
The included angle between the projection on the plane O-yz and the axis is larger and smaller, and beta is a vector->
The included angle with the plane O-yz is large. For the actual image, its u-axis and v-axisThe coordinates of the directions are all finite values, so that the value ranges of alpha and beta are (-pi/2, pi/2). From the spatial geometry, there is the following equation (2.1):
wherein f is the focal length of the camera and u 0 ,v 0 ) Is biased to the dominant point. In effect, the above equation provides a mapping from the image coordinate system to the local angle in the sphere
I.e. a one-to-one correspondence of points on the image plane and points of the part-sphere.
In a multi-lens panoramic camera, each sub-camera has an independent camera coordinate system. The coordinate system of the multi-lens panoramic camera is based on a single-lens spherical coordinate system, but the optical centers of all the lenses are not overlapped due to the process and the volume of the lenses. Therefore, when performing spherical mapping on the acquired image data, it is necessary to unify the sub-cameras of the multi-lens panoramic camera into an overall coordinate system, and then convert the pixel coordinates in the image acquired by each lens into the overall coordinate system through formula 2.1 to perform spherical mapping uniformly.
Step 2.2: on the basis of mapping the panoramic image to the spherical model, feature extraction and matching splicing are carried out on the spherical image based on the SPHORB algorithm, after feature points are extracted from the spherical image, homonymous feature point pairs exist in the two images (namely one feature point appears in the two images), the corresponding relation of homonymous points is determined by iteratively calculating the distance between the point pairs, matching is completed at the moment, a relative transformation matrix between the homonymous point pairs is calculated according to the coordinate relation between the homonymous point pairs, and the relative transformation matrix is transformed to a unified coordinate system, so that splicing is completed. The idea of the algorithm is that the spherical image is firstly processed approximately to obtain a hexagonal spherical grid similar to a football, and then a fine-grained pyramid and a robust feature are directly constructed on the hexagonal spherical grid, so that the time-consuming calculation of spherical harmonics and related bandwidth limitation is avoided, and the algorithm has the scale and rotation invariance of the spherical features.
Step 2.3: on the basis of obtaining a large number of feature points after feature extraction and matching, solving the position and posture transformation relation (the posture transformation relation refers to t) between adjacent frames by an epipolar constraint method k At a time (x) k ,y k ,z k ) To (roll) k ,pitch k ,yaw k ) The attitude angle of (a) obtains one frame image at t k+1 At a time (x) k+1 ,y k+1 ,z k+1 ) To (roll) k+1 ,pitch k+1 ,yaw k+1 ) The pose angle of the camera obtains a frame of image, the position and pose transformation existing between two moments is called pose transformation), a nonlinear optimization mode is adopted, and the three-dimensional coordinates of the feature points and the pose of the camera are optimized at the same time, so that the reprojection error is minimum, the optimal camera pose is output, and the position of the camera is determined.
And 3, step 3: in step 2, the unmanned aerial vehicle completes positioning and obtains a real-time panoramic image, real-time detects obstacles and target objects in the surrounding environment through target identification, and plans an optimal feasible path, so that autonomous obstacle avoidance flight is completed. The overall scheme is shown in fig. 5, and the specific steps are as follows:
step 3.1: the technical process of target identification is shown in fig. 6, and mainly comprises the steps of making a target sample data set, building a neural network model, training the model and identifying in real time. And considering the limited capability and the operation efficiency of a computing unit on the light and small unmanned aerial vehicle, a YOLOv5 algorithm is selected as a target recognition algorithm. The network framework is divided into 4 general modules, including: the device comprises an input end, a reference network, a Neck network and a Head output end. The input terminal represents an input picture. The size of the input image represented by the network is 608 × 608, and at this stage, the input image may be scaled to the input size of the network and normalized. In the network training stage, YOLOv5 uses the Mosaic data to enhance the training speed of the operation promotion model and the accuracy of the network; and provides a self-adaptive anchor frame calculation and self-adaptive picture scaling method. The reference network represents a classifier network with excellent performance, the module is used for extracting general feature representation, and CSPDarknet53 and Focus structures are used as the reference network. The CSPDarknet53 structure is designed to improve feature representation dimensionality in a backbone network by taking the design idea of CSPNet as a reference. The Focus structure mainly cuts an input picture through slice operation. The original input picture size is 608 × 3, and a feature map of 304 × 12 is output after Slice and Concat operations; then, a feature map of size 304 × 32 is output through a Conv layer with 32 channels. The neutral network is positioned in the middle of the reference network and the head network, and the SPP module and the FPN + PAN module are improved by using the CSP2 structure designed by CSPnet for reference, so that the network feature fusion capability is enhanced. The Head output section is used for finishing the output of the target recognition result. The Yolov5 adopts a GIOU _ Loss function, and the function solves the problem when the boundary frames are not overlapped on the basis of the IOU, so that the detection precision of the algorithm is further improved.
Step 3.2: on the basis of detecting surrounding environment obstacles in real time in the step 3.1, the unmanned aerial vehicle carries out path planning, navigates and flies according to an optimal obstacle avoidance path, and completes autonomous obstacle avoidance, wherein the overall obstacle avoidance scheme is as shown in fig. 7, firstly, the time difference and the pose change between two adjacent frames are obtained in the step 2.3, so that the speed, the acceleration and the angular speed of the aircraft are calculated, and the pose of the aircraft is dynamically tracked by using a Kalman filtering navigation algorithm, so that autonomous navigation is realized. In the flying process, the panoramic image is identified and detected by using the Yolov5 algorithm in the step 3.1, and according to the position information of the obstacle and the real-time positioning information of the aircraft, a D x Lite algorithm is utilized, and aiming at the problem of planning the three-dimensional flight path of the unmanned aircraft when the target moves in an uncertain environment, the D x Lite search algorithm is adopted to rapidly plan the three-dimensional flight path, so that the autonomous obstacle avoidance in the flying process is realized.
It should be understood that parts of the specification not set forth in detail are well within the prior art.
It should be understood that the foregoing description of the preferred embodiment is specific, and not intended to limit the scope of the present invention, and those skilled in the art can independently construct various unmanned aerial vehicle platforms for different use scenarios without departing from the scope of the present invention as defined in the appended claims.
Claims (6)
1. An unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target identification is characterized by comprising the following steps:
step 1, building a light and small unmanned aerial vehicle platform; selecting a small-wheelbase quad-rotor unmanned aerial vehicle as a flight platform, carrying a panoramic camera at the front end of the unmanned aerial vehicle platform, and effectively acquiring 360-degree visual angle information; transmitting the video information to an airborne computer in real time, and calculating in real time to obtain the flight attitude of the unmanned aerial vehicle and scene interesting target information; then, the flight attitude data is transmitted into a flight control system to autonomously control the unmanned aerial vehicle to fly;
step 2, acquiring an environment image by using a panoramic camera, operating a real-time panoramic SLAM, and completing the positioning and autonomous flight of the unmanned aerial vehicle;
and step 3: and detecting the object in the real-time image based on a target identification technology, planning an optimal path by taking the target object as a terminal point by combining positioning information output by the SLAM, and finishing autonomous obstacle avoidance flight.
2. The unmanned aerial vehicle autonomous flight technical method fusing panoramic SLAM and target recognition as claimed in claim 1, wherein: the specific implementation of the step 2 comprises the following substeps;
step 2.1, constructing an imaging model of the panoramic image, wherein the imaging model comprises a single-lens imaging model and a multi-lens imaging model, realizing high-efficiency splicing of the panoramic image and completing spherical mapping of image data;
2.2, carrying out feature extraction and matching splicing on the spherical image mapped in the step 2.1 by using a SPHORB algorithm;
and 2.3, solving a pose transformation relation between adjacent frames by epipolar constraint according to the feature point pairs extracted in the step 2.2, optimizing the three-dimensional coordinates of the feature points and the camera pose by adopting nonlinear optimization to minimize a reprojection error, outputting the optimal camera pose to finish positioning, and transforming and splicing the feature points by the pose to finish image construction.
3. The unmanned aerial vehicle autonomous flight technical method fusing the panoramic SLAM and the target recognition, as claimed in claim 2, wherein: the specific implementation of step 2.1 is as follows;
firstly, establishing a panoramic camera coordinate system, abstracting a panoramic camera carried on an unmanned aerial vehicle platform into a dome camera model, taking the single-lens condition into consideration, enabling the spherical center to coincide with the optical center of the camera, setting O-xyz as a camera coordinate system, mapping an object space point P at a point P on an image plane, and setting the coordinate in the corresponding image coordinate system as (u, v); meanwhile, the ray OP intersects with the spherical surface with O as the center of the sphere at the point P s ；
Defining spherical surface mapping as function mapping relation f s :
An arbitrary point p (u, v) on the image is mapped onto a spherical surface with a certain radius and expressed as ^ based on the ball coordinate>
If the right hand is used for determination, the thumb rotates towards the direction of the y-axis finger, and the x-axis and the z-axis rotate towards the direction of other fingers by an angle theta; />
For right-handed determination, the thumb rotates in the direction of the z-axis, the x-axis and the y-axis toward the other fingers>
An angular value;
let alpha be the vector
The included angle between the projection on the plane O-yz and the axis is larger and smaller, and beta is a vector->
The size of an included angle between the plane O-yz and the plane; for an actual image, the coordinates in the directions of the u axis and the v axis are both finite values, so that the value ranges of alpha and beta are (-pi/2, pi/2), and according to the space geometric relationship, the following formula (2.1) is provided:
wherein f is the focal length of the camera and u 0 ,v 0 ) Biasing for the principal point; in effect, the above equation provides a mapping from the image coordinate system to the local angle in the sphere
{[u v]′}→{[α β]' }, namely the one-to-one correspondence of points on the image plane and part of the spherical surface; />
In the multi-lens panoramic camera, each sub-camera has an independent camera coordinate system, the multi-lens panoramic camera coordinate system is based on a single-lens spherical coordinate system, but the optical centers of all the lenses are not overlapped due to the process and the volume of the lenses; therefore, when performing spherical mapping on the acquired image data, it is necessary to unify all sub-cameras of the multi-lens panoramic camera into a whole coordinate system, and then convert the pixel coordinates in the image acquired by each lens into the whole coordinate system through formula 2.1, so as to perform spherical mapping uniformly.
4. The unmanned aerial vehicle autonomous flight technical method fusing the panoramic SLAM and the target recognition, as claimed in claim 2, wherein: the specific implementation manner of the step 3 is as follows;
step 3.1, a YOLOv5 algorithm is used as a target recognition algorithm, a target data set of the panoramic image is input to train a neural network model, the trained model is used for carrying out real-time target recognition on the panoramic image transmitted in the step 2, and a recognition result is output;
3.2, calculating the speed, acceleration and angular speed of the aircraft by using the time difference and pose change between adjacent frames obtained in the step 2.3, and dynamically tracking the pose of the aircraft by a kalman filtering navigation algorithm; and 3, performing three-dimensional flight path planning by using a D × Lite search algorithm according to the obstacle position information and the real-time positioning information of the aircraft identified in the step 3.1 in the flight process, and realizing real-time autonomous obstacle avoidance.
5. The unmanned aerial vehicle autonomous flight technical method fusing panoramic SLAM and target recognition as claimed in claim 4, wherein: the YOLOv5 network model in the step 3.1 comprises an input end, a reference network, a Neck network and a Head output end; the input end represents an input picture, the size of the input image is 608 × 608, and the input image can be scaled to the input size of the network at this stage and normalized and the like are carried out; in the network training stage, YOLOv5 uses Mosaic data to enhance the training speed of the operation promotion model and the precision of the network; the reference network represents a classifier network with excellent performance and is used for extracting general feature representation, and the CSPDarknet53 and Focus structures are used as the reference network; the CSPDarknet53 structure is designed to improve feature representation dimensionality in a main network by taking the design thought of CSPNet as reference, the Focus structure cuts an input picture through Slice operation, the size of the original input picture is 608 × 3, and a feature map of 304 × 304 is output after the Slice and Concat operation; then, a feature map with the size of 304 × 32 is output through a Conv layer with the number of channels being 32; the neutral network is positioned in the middle of the reference network and the Head output end, and the SPP module and the FPN + PAN module are improved by using the CSP2 structure designed by CSPnet for reference, so that the network feature fusion capability is enhanced; the Head output end is used for finishing the output of the target recognition result; the Yolov5 adopts a GIOU _ Loss function, and the function solves the problem when the boundary frames are not overlapped on the basis of the IOU, so that the detection precision of the algorithm is further improved.
6. The unmanned aerial vehicle autonomous flight technical method fusing panoramic SLAM and target recognition, as claimed in claim 5, wherein: the specific implementation of step 3.2 is as follows;
firstly, the time difference and the pose change between two adjacent frames obtained in the step 2.3 are utilized to calculate the speed, the acceleration and the angular velocity of the aircraft, and the pose of the aircraft is dynamically tracked by utilizing a Kalman filtering navigation algorithm, so that autonomous navigation is realized; in the flying process, the panoramic image is identified and detected by using the Yolov5 algorithm in the step 3.1, and according to the position information of the obstacle and the real-time positioning information of the aircraft, a D x Lite algorithm is utilized, and aiming at the problem of planning the three-dimensional flight path of the unmanned aircraft when the target moves in an uncertain environment, the D x Lite search algorithm is adopted to rapidly plan the three-dimensional flight path, so that the autonomous obstacle avoidance in the flying process is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211385768.6A CN115933718A (en) | 2022-11-07 | 2022-11-07 | Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211385768.6A CN115933718A (en) | 2022-11-07 | 2022-11-07 | Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115933718A true CN115933718A (en) | 2023-04-07 |
Family
ID=86651769
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211385768.6A Pending CN115933718A (en) | 2022-11-07 | 2022-11-07 | Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115933718A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116929308A (en) * | 2023-09-18 | 2023-10-24 | 众芯汉创(江苏)科技有限公司 | Unmanned aerial vehicle around-flight photographing method and system based on arbitrary point position information of tower |
| CN117111639A (en) * | 2023-10-19 | 2023-11-24 | 浙江容祺科技有限公司 | Unmanned aerial vehicle flight optimal route optimizing method in complex environment |
| CN118135526A (en) * | 2024-05-06 | 2024-06-04 | 南京信息工程大学 | Visual target recognition and positioning method for four-rotor unmanned aerial vehicle based on binocular camera |
-
2022
- 2022-11-07 CN CN202211385768.6A patent/CN115933718A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116929308A (en) * | 2023-09-18 | 2023-10-24 | 众芯汉创(江苏)科技有限公司 | Unmanned aerial vehicle around-flight photographing method and system based on arbitrary point position information of tower |
| CN116929308B (en) * | 2023-09-18 | 2023-11-28 | 众芯汉创(江苏)科技有限公司 | Unmanned aerial vehicle around-flight photographing method and system based on arbitrary point position information of tower |
| CN117111639A (en) * | 2023-10-19 | 2023-11-24 | 浙江容祺科技有限公司 | Unmanned aerial vehicle flight optimal route optimizing method in complex environment |
| CN117111639B (en) * | 2023-10-19 | 2024-01-26 | 浙江容祺科技有限公司 | Unmanned aerial vehicle flight optimal route optimizing method in complex environment |
| CN118135526A (en) * | 2024-05-06 | 2024-06-04 | 南京信息工程大学 | Visual target recognition and positioning method for four-rotor unmanned aerial vehicle based on binocular camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11914370B2 (en) | System and method for providing easy-to-use release and auto-positioning for drone applications | |
| US11649052B2 (en) | System and method for providing autonomous photography and videography | |
| CN110068335B (en) | Unmanned aerial vehicle cluster real-time positioning method and system under GPS rejection environment | |
| CN115933718A (en) | Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition | |
| CN106774386B (en) | Unmanned plane vision guided navigation landing system based on multiple dimensioned marker | |
| CN102190081B (en) | Vision-based fixed point robust control method for airship | |
| US20190243376A1 (en) | Actively Complementing Exposure Settings for Autonomous Navigation | |
| CN106125747A (en) | Based on the servo-actuated Towed bird system in unmanned aerial vehicle onboard the first visual angle mutual for VR | |
| CN112734765A (en) | Mobile robot positioning method, system and medium based on example segmentation and multi-sensor fusion | |
| CN112789672A (en) | Control and navigation system, attitude optimization, mapping and positioning technology | |
| WO2021081774A1 (en) | Parameter optimization method and apparatus, control device, and aircraft | |
| CN114240769A (en) | Image processing method and device | |
| CN109002059A (en) | A kind of multi-rotor unmanned aerial vehicle object real-time tracking camera system and method | |
| Xiang et al. | UAV based target tracking and recognition | |
| US20220019222A1 (en) | Unmanned Aerial Vehicle, Unmanned Aerial Vehicle Flight Control Device, Unmanned Aerial Vehicle Flight Control Method and Program | |
| CN112198903A (en) | Modular multifunctional onboard computer system | |
| Stuckey et al. | An optical spatial localization system for tracking unmanned aerial vehicles using a single dynamic vision sensor | |
| CN214409706U (en) | Indoor unmanned aerial vehicle positioning system based on machine vision | |
| CN113282025A (en) | Unmanned carrier is patrolled and examined in security protection | |
| CN115237158A (en) | Multi-rotor unmanned aerial vehicle autonomous tracking and landing control system and control method | |
| CN115144879A (en) | Multi-machine multi-target dynamic positioning system and method | |
| CN117873159B (en) | Indoor target visual positioning method of multi-rotor unmanned aerial vehicle | |
| Wang et al. | Real-time visual odometry for autonomous MAV navigation using RGB-D camera | |
| WO2022188174A1 (en) | Movable platform, control method of movable platform, and storage medium | |
| Min | Binocular stereo vision control method for landing position of four rotor UAV |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |