CN111736622B - Unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU - Google Patents
Unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU Download PDFInfo
- Publication number
- CN111736622B CN111736622B CN201910225407.7A CN201910225407A CN111736622B CN 111736622 B CN111736622 B CN 111736622B CN 201910225407 A CN201910225407 A CN 201910225407A CN 111736622 B CN111736622 B CN 111736622B
- Authority
- CN
- China
- Prior art keywords
- unmanned aerial
- aerial vehicle
- obstacle avoidance
- area
- binocular camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000004888 barrier function Effects 0.000 claims description 23
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 238000009792 diffusion process Methods 0.000 claims description 8
- 238000005259 measurement Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000002567 autonomic effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/10—Simultaneous control of position or course in three dimensions
- G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
- G05D1/104—Simultaneous control of position or course in three dimensions specially adapted for aircraft involving a plurality of aircrafts, e.g. formation flying
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention provides an unmanned aerial vehicle obstacle avoidance method based on combination of binocular vision and IMU, which comprises the following steps: firstly, acquiring real-time position information of an obstacle in front of an unmanned aerial vehicle; secondly, acquiring the maximum flyable area of the unmanned aerial vehicle in the current direction; thirdly, judging the maximum flyable area, and if the maximum flyable area belongs to a safe flying area, adjusting the course of the unmanned aerial vehicle until obstacle avoidance is finished; if the unmanned aerial vehicle does not belong to the safe flight area, the holder rotates to adjust the position of the binocular camera, the steps from one step to three are repeated until the maximum flyable area belongs to the safe flight area, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance; and fourthly, judging whether the holder rotates, if the holder rotates, controlling the unmanned aerial vehicle to recover to the initial course according to the camera deflection angle recorded by the IMU module, and rotating the holder to the initial position to complete obstacle avoidance flight of the unmanned aerial vehicle. By applying the technical scheme of the invention, the technical problems of low obstacle avoidance efficiency and large limitation of the unmanned aerial vehicle in the prior art are solved.
Description
Technical Field
The invention relates to the technical field of unmanned aerial vehicle obstacle avoidance, in particular to an unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU.
Background
The flying development of prior art has made things convenient for people's various activities, unmanned aerial vehicle is because convenient to use, use extensively, carry easily, there is very big development, but at unmanned aerial vehicle flight in-process, because the barrier that influences the flight in low air is more, and there are various uncontrollable factors, make unmanned aerial vehicle when meetting the barrier, it is too late for the control personnel to react, this just needs unmanned aerial vehicle to possess the autonomic obstacle avoidance function, in order to guarantee flight safety, and reduce the control degree of difficulty of control personnel. Therefore, the obstacle avoidance of the small unmanned aerial vehicle is an important aspect of intelligent flight of the small unmanned aerial vehicle.
Because the small unmanned aerial vehicle has limited carrying capacity, the small unmanned aerial vehicle has few available sensors, and therefore obstacle avoidance strategies of the small unmanned aerial vehicle are limited. The existing commonly used sensor in unmanned aerial vehicle obstacle avoidance has schemes based on infrared rays, ultrasonic waves, visible light cameras, toF cameras, laser radars and the like. The infrared sensors and the ultrasonic sensors can only obtain distance information in a specific direction, and are active sensors, so that requirements on reflected obstacles are met, and the sensors are easy to interfere with each other; although the ToF camera can obtain three-dimensional information, the current ToF camera is high in price and power consumption and may be interfered due to reflection; the laser radar has high power consumption and large volume, and is generally applied to ground robots; the binocular camera is low in price and low in power consumption, the three-dimensional position information of an object can be obtained, and the distance measurement and precision meet the obstacle avoidance requirement of the small unmanned aerial vehicle. What use is more in the current unmanned aerial vehicle keeps away the barrier is two mesh cameras.
Due to the advantages of binocular cameras, various small unmanned aerial vehicle obstacle avoidance schemes based on binocular vision are formed at present, specifically, hrabar detects an obstacle in front of an unmanned aerial vehicle by using stereo matching, and detects obstacles on two sides of the unmanned aerial vehicle by using an optical flow method (Hrabar S, sukhaltm G S, corner P, et al. Combined optical-flow and stereo-based navigation of unmanned aerial vehicles for a UAV [ C ]. Interactive robots and systems,2005 3309-3316; richard performs obstacle detection on drones with binocular cameras and assists drones in landing (Moore R J, thurrowgood S, bland D, et al. A stereo vision system for UAV guidances [ C ]. Intelligent robots and systems, 2009; abraham uses binocular cameras, IMU and laser rangefinders to complete unmanned aerial vehicle positioning and obstacle avoidance on an unmanned aerial vehicle (Bachrach a g. Autonomus flight in unstructured and unworn arrows environments [ D ]. Massachusetts Institute of Technology, 2009.); su Dong implements an obstacle avoidance scheme for unmanned aerial vehicles (sudong. Navigation and obstacle avoidance for small unmanned aerial vehicles based on binocular vision [ D ] electronics university, 2014) using a semi-global binocular matching algorithm. In the binocular vision obstacle avoidance, an important technology is binocular stereo matching, which can be classified into sparse stereo matching and dense stereo matching according to the density of a matching disparity map (kan, research on binocular vision positioning method of marine targets in shipborne video [ D ]. University of maritime, 2016); the binocular Matching can be classified into local Matching, semi-Global Matching, and Global Matching according to the optimization manner of the cumulative Matching cost (Hirschmuller h. Stereo Vision in Structured Environments by configuration sensitive Semi-Global Matching [ C ]. Computer Vision and pattern recognition,2006 2386-2393).
However, for the common obstacle avoidance scheme of the small unmanned aerial vehicle in the prior art, the following problems exist.
First, current some unmanned aerial vehicle can only obtain the distance information of barrier in the fixed direction, not only is not known to the big or small condition of barrier, but also can not obtain the positional information of barrier, can not provide effectual information for unmanned aerial vehicle next step motion.
Secondly, although some existing small unmanned aerial vehicles obtain three-dimensional information of the obstacle by using sensing means such as a binocular camera, no obstacle avoidance scheme for the obstacle by the unmanned aerial vehicle is specifically provided, and the unmanned aerial vehicle can only hover or cannot collide with the obstacle, so that the obstacle cannot be intelligently avoided.
Thirdly, although some existing small unmanned aerial vehicles not only obtain three-dimensional information of obstacles, but also can realize path planning according to the information of the obstacles, the path planning of the obstacles is established on an electronic map of a known flight area or a map is established in advance, and the unmanned aerial vehicles can be subjected to subsequent path planning after encountering the obstacles by combining with the known route information, and although the existing small unmanned aerial vehicles are optimal, the limitation is obvious.
Fourthly, the existing partial binocular vision obstacle avoidance scheme is to perform global path planning by using a disparity map or a depth map obtained by a binocular matching algorithm, so that the calculation is complex, a lot of information is redundant, and the efficiency is low.
Disclosure of Invention
The invention provides an unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and an IMU (inertial measurement Unit), which can solve the technical problems of low obstacle avoidance efficiency and large limitation of an unmanned aerial vehicle in the prior art.
According to one aspect of the invention, an unmanned aerial vehicle obstacle avoidance method based on combination of binocular vision and an IMU is provided, and comprises the following steps: the method comprises the following steps that firstly, a barrier is subjected to stereo matching through a binocular camera so as to obtain real-time position information of the barrier in front of the unmanned aerial vehicle; acquiring the maximum flyable area of the unmanned aerial vehicle in the current direction according to the real-time position information of the obstacle in front of the unmanned aerial vehicle; step three, judging the maximum flying area of the unmanned aerial vehicle, and if the maximum flying area belongs to a safe flying area, adjusting the course of the unmanned aerial vehicle until the unmanned aerial vehicle finishes obstacle avoidance; if the maximum flyable area does not belong to the safe flying area, the cradle head rotates to adjust the position of the binocular camera, the first step, the second step and the third step are repeated until the maximum flyable area of the unmanned aerial vehicle belongs to the safe flying area under the adjusted position of the binocular camera, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance; and step four, judging whether the holder rotates, if so, controlling the unmanned aerial vehicle to recover to the initial course according to the camera deflection angle recorded by the IMU module, and rotating the holder to the initial position to complete obstacle avoidance flight of the unmanned aerial vehicle.
Further, the first step specifically comprises: and (4) carrying out three-dimensional matching on the obstacle by adopting an ELAS three-dimensional matching algorithm and utilizing a binocular camera so as to obtain the real-time position information of the obstacle in front of the unmanned aerial vehicle.
Further, the second step specifically includes: and segmenting an obstacle area according to the real-time position information of the obstacle in front of the unmanned aerial vehicle, and acquiring the maximum flying area of the unmanned aerial vehicle in the current direction according to the segmented obstacle area by using a central diffusion method.
Further, in step three, if the maximum flyable area of the unmanned aerial vehicle under the current binocular camera angle belongs to a safe flying area, adjusting the heading of the unmanned aerial vehicle until the unmanned aerial vehicle finishes obstacle avoidance specifically comprises: if the maximum flying area of the unmanned aerial vehicle under the current binocular camera angle is larger than or equal to the safe area, the maximum flying area of the unmanned aerial vehicle belongs to the safe flying area, the center of the maximum flying area is used as the flying target direction of the unmanned aerial vehicle, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance.
Further, in step three, if the maximum flyable area of the unmanned aerial vehicle under the current binocular camera angle is smaller than the safe area, the maximum flyable area of the unmanned aerial vehicle does not belong to the safe flying area.
Further, in step three, the rotation of the pan-tilt to adjust the position of the binocular camera specifically includes: the holder rotates around the first direction in order to adjust the binocular camera position or the holder rotates around the second direction in order to adjust the binocular camera position, and the first direction is perpendicular to the second direction.
According to another aspect of the invention, an unmanned aerial vehicle obstacle avoidance system based on binocular vision and IMU combination is provided, and the unmanned aerial vehicle obstacle avoidance system uses the unmanned aerial vehicle obstacle avoidance method.
Further, unmanned aerial vehicle keeps away barrier system includes: an unmanned aerial vehicle; the cloud deck is rotatably arranged on the unmanned aerial vehicle; the system comprises a binocular camera, a first control module, a second control module and a control module, wherein the binocular camera is fixedly arranged on a cloud deck, the cloud deck can drive the binocular camera to rotate around a first direction and a second direction, and the binocular camera is used for acquiring front information of the unmanned aerial vehicle so as to judge the real-time position of an obstacle in front of the binocular camera; the IMU module is arranged on the binocular camera and is used for recording the rotation angle of the binocular camera; and the control unit is used for controlling the unmanned aerial vehicle to finish obstacle avoidance flight according to the image acquired by the binocular camera and the angle information acquired by the IMU module.
The technical scheme of the invention is applied, and the unmanned aerial vehicle obstacle avoidance method based on the combination of binocular vision and IMU is provided. Aiming at the low efficiency that the existing unmanned aerial vehicle obstacle avoidance strategy only enables the unmanned aerial vehicle to hover for a next instruction when meeting an obstacle or the limitation of establishing a map in advance, the obstacle avoidance method can not only avoid the harm of most obstacles to the unmanned aerial vehicle when the small unmanned aerial vehicle sails, but also further give the next course of the unmanned aerial vehicle, and can approach the original course, so that the influence of the obstacles on the small unmanned aerial vehicle is reduced, and therefore, the local path planning caused by the obstacles is realized.
Drawings
The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 shows a flowchart of an unmanned aerial vehicle obstacle avoidance method based on binocular vision combined with an IMU according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram illustrating an unmanned aerial vehicle obstacle avoidance system based on binocular vision combined with an IMU according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a flyable region obtained in a depth map provided in accordance with an embodiment of the present invention.
Wherein the figures include the following reference numerals:
10. an unmanned aerial vehicle; 20. a holder; 30. a binocular camera.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
The relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise. Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description. Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate. In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
As shown in fig. 1, according to a specific embodiment of the present invention, an unmanned aerial vehicle obstacle avoidance method based on a combination of binocular vision and an IMU is provided, and the unmanned aerial vehicle obstacle avoidance method includes: the method comprises the following steps that firstly, a barrier is subjected to stereo matching through a binocular camera so as to obtain real-time position information of the barrier in front of the unmanned aerial vehicle; acquiring the maximum flyable area of the unmanned aerial vehicle in the current direction according to the real-time position information of the obstacle in front of the unmanned aerial vehicle; step three, judging the maximum flying area of the unmanned aerial vehicle, and if the maximum flying area belongs to a safe flying area, adjusting the course of the unmanned aerial vehicle until the unmanned aerial vehicle finishes obstacle avoidance; if the maximum flyable area does not belong to the safe flying area, the cradle head rotates to adjust the position of the binocular camera, the first step, the second step and the third step are repeated until the maximum flyable area of the unmanned aerial vehicle belongs to the safe flying area under the adjusted position of the binocular camera, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance; and step four, judging whether the holder rotates, if so, controlling the unmanned aerial vehicle to recover to the initial course according to a camera deflection angle recorded by an Inertial Measurement Unit (IMU) module, and rotating the holder to the initial position to complete obstacle avoidance flight of the unmanned aerial vehicle.
By applying the configuration mode, the unmanned aerial vehicle obstacle avoidance method based on the combination of binocular vision and the IMU is provided, the method obtains information of an obstacle in front of the unmanned aerial vehicle through stereoscopic matching of the binocular camera, obtains a flying course of the unmanned aerial vehicle by combining matching of the tripod head driving the binocular camera to rotate to the obstacle in the surrounding environment, and recovers the original course after the unmanned aerial vehicle avoids the obstacle by utilizing the IMU module, so that the unmanned aerial vehicle cannot deviate from the original course in a large range, and the obstacle avoidance of the small unmanned aerial vehicle is completed. Aiming at the low efficiency that the existing unmanned aerial vehicle obstacle avoidance strategy only enables the unmanned aerial vehicle to hover when meeting an obstacle and wait for a next step of instruction, or the limitation of establishing a map in advance, the obstacle avoidance method not only can avoid the harm of most obstacles encountered in the navigation process of the small unmanned aerial vehicle to the unmanned aerial vehicle, but also can further give the next course of the unmanned aerial vehicle, and can approach to the original route, so that the influence of the obstacles on the small unmanned aerial vehicle is reduced, and therefore, the local path planning caused by the obstacles is realized.
Further, in the present invention, in order to complete obstacle avoidance of the unmanned aerial vehicle, three-dimensional matching of an obstacle needs to be completed first to obtain real-time position information of the obstacle in front of the unmanned aerial vehicle. Specifically, in the invention, the unmanned aerial vehicle carries out stereo matching on the front of the camera by using the binocular camera to obtain the position information of the obstacle. The unmanned aerial vehicle flies in real time, so real-time matching can be completed on an embedded system by the binocular matching algorithm, and the algorithm is required to be dense by using matched information. In addition, in order to reduce the complexity of the operation, the invention divides the obstacles according to the distance range for the matched depth map.
In addition, in the present invention, in order to obtain the maximum flyable area of the drone in the current direction, the second step may be configured to specifically include: and segmenting an obstacle area according to the real-time position information of the obstacle in front of the unmanned aerial vehicle, and acquiring the maximum flying area of the unmanned aerial vehicle in the current direction according to the segmented obstacle area by using a central diffusion method.
After the maximum flying area of the unmanned aerial vehicle in the current direction is obtained, obstacle avoidance of the unmanned aerial vehicle can be carried out. Specifically, in the third step, if the maximum flyable area of the unmanned aerial vehicle under the current binocular camera angle is larger than or equal to the safe area, the maximum flyable area of the unmanned aerial vehicle belongs to the safe flying area, the center of the maximum flyable area is used as the flying target direction of the unmanned aerial vehicle, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance.
If the maximum flying area of the unmanned aerial vehicle under the current binocular camera angle is smaller than the safe area, the maximum flying area of the unmanned aerial vehicle is not considered to belong to the safe flying area, the holder rotates to adjust the position of the binocular camera, the first step to the third step are repeated until the position of the adjusted binocular camera is down, the maximum flying area of the unmanned aerial vehicle belongs to the safe flying area, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance. Wherein, the cloud platform rotates specifically to include with the adjustment binocular camera position: the cloud platform rotates in order to drive the binocular camera around the motion of first direction or the cloud platform rotates in order to drive the binocular camera around the rotation of second direction, and first direction is perpendicular with the second direction mutually.
As a specific embodiment of the present invention, as shown in fig. 1 and fig. 2, the first direction is a vertical axis as shown in fig. 2, the second direction is a horizontal axis as shown in fig. 2, the rotation of the pan-tilt can realize the rotation from left to right and the rotation around the horizontal axis, and the binocular camera is fixedly arranged on the pan-tilt, so that the rotation of the pan-tilt can drive the binocular camera to realize the movement around the first direction or along the first direction.
Further, as shown in fig. 3, as an embodiment of the present invention, for the depth map with the segmented obstacles, the flyable area is obtained by the center diffusion method, and the IMU records the initial heading of the drone. In the normal forward flight process of the unmanned aerial vehicle, the motion direction of the unmanned aerial vehicle is consistent with the direction right ahead of the camera, and if the maximum flying area of the unmanned aerial vehicle under the current binocular camera angle is larger than or equal to the safe area, the center of the maximum flying area is used as the flying target direction of the unmanned aerial vehicle.
If the maximum flying area of the unmanned aerial vehicle is smaller than the safe area under the current angle of the binocular camera, the unmanned aerial vehicle hovers first, then the holder is rotated, the binocular camera is firstly deviated 10 degrees to the left in the horizontal direction (the specific deviation degree is related to the parameters of the binocular camera and the size of the unmanned aerial vehicle), the binocular camera is matched, and if the safe area cannot be obtained, the horizontal direction of the binocular camera is restored to the original direction. And then, the height of the binocular camera is increased by 10 degrees in the vertical direction, the binocular camera is used for carrying out visual processing, and if a safe area cannot be obtained, the vertical direction of the binocular camera is restored to the original direction. And then, the binocular camera is shifted to the left by 15 degrees in the horizontal direction (the specific accumulated degree of each shift is related to the parameters of the binocular camera and the size of the unmanned aerial vehicle), according to the sequence, whether the maximum flyable area obtained by the binocular camera is larger than a safe area is searched, if the safe flyable area is obtained in the searching process, the searching is stopped, and if the safe flyable area is not found, the continuous cyclic searching is carried out by adding 5 degrees to the angle of the shifted direction angle of the binocular camera until the safe flyable area is obtained. Then the center of the flyable area is used as the flying target direction of the unmanned aerial vehicle, the unmanned aerial vehicle is controlled to carry out obstacle avoidance flying, in the obstacle avoidance flying of the unmanned aerial vehicle, through the deviation angle of the binocular camera recorded by the IMU, the holder is gradually controlled to enable the binocular camera to rotate towards the direction of recovering the initial course, meanwhile, the visual system is combined with the control system, the unmanned aerial vehicle is enabled to be in the safe flyable area, when the camera recovers the original course, and after the unmanned aerial vehicle is matched with the obtained depth map, the unmanned aerial vehicle does not find the separated obstacles, and the unmanned aerial vehicle finishes obstacle avoidance.
In order to reduce the influence of the obstacle on the air route of the unmanned aerial vehicle, after the unmanned aerial vehicle finishes obstacle avoidance, whether the cradle head rotates or not is judged, if the cradle head rotates, the unmanned aerial vehicle is controlled to recover to the initial course according to the camera deflection angle recorded by the IMU module, and the cradle head rotates to the initial position to finish obstacle avoidance flight of the unmanned aerial vehicle.
According to another aspect of the invention, an unmanned aerial vehicle obstacle avoidance system based on combination of binocular vision and an IMU is provided, and the unmanned aerial vehicle obstacle avoidance system uses the unmanned aerial vehicle obstacle avoidance method. The system works by taking the avoidance of the unmanned aerial vehicle to the barrier as a guide by adopting a strategy of obtaining three-dimensional information of the barrier through binocular stereo matching.
Further, in order to realize the obstacle avoidance flight of unmanned aerial vehicle, can be configured into including unmanned aerial vehicle 10 with the unmanned aerial vehicle obstacle avoidance system, cloud platform 20, two mesh cameras 30, IMU module and the control unit, cloud platform 20 rotationally sets up on unmanned aerial vehicle 10, two mesh cameras 30 are fixed to be set up on cloud platform 20, cloud platform 20 can drive two mesh cameras 30 and rotate around first direction and second direction, two mesh cameras 30 are used for acquireing the real-time position of unmanned aerial vehicle the place ahead information in order to judge camera the place ahead barrier, the IMU module sets up on two mesh cameras 30, the IMU module is used for the turned angle of two mesh cameras 30 of record, the control unit is used for accomplishing the obstacle avoidance flight with control unmanned aerial vehicle according to the image of two mesh cameras 30 collection and the angle information that the IMU module was gathered.
As a specific embodiment of the present invention, the obstacle avoidance system provided by the present invention uses a small quad-rotor drone as a platform, and the drone can realize functions of forward movement, hovering, lifting, etc. The binocular camera is placed on the cloud platform with two degrees of freedom, and the cloud platform is placed under unmanned aerial vehicle, and the motion of direction and vertical direction about the cloud platform can be realized. In addition, an IMU module is fixed on the binocular camera, and the IMU module can obtain the angles of the binocular camera in the X, Y, Z three directions in real time through calculation. As another embodiment of the present invention, with the development of the technology, if the ToF camera is reduced in price, reduced in power consumption and improved in accuracy, the binocular camera in the present solution may be replaced by the ToF camera.
For further understanding of the present invention, the unmanned aerial vehicle obstacle avoidance method and system based on binocular vision and IMU combination of the present invention are described in detail below with reference to fig. 1 and 3.
As shown in fig. 1 and 3, according to an embodiment of the present invention, an unmanned aerial vehicle obstacle avoidance system based on binocular vision and IMU combination is provided, the system includes an unmanned aerial vehicle 10, a pan/tilt head 20, a binocular camera 30, an IMU module, and a control unit, the pan/tilt head 20 is rotatably disposed right below the unmanned aerial vehicle 10, and the pan/tilt head 20 is rotatable around a first direction and a second direction. Binocular camera 30 is fixed to be set up on cloud platform 20, and cloud platform 20 can drive binocular camera 30 and rotate around first direction and second direction, and binocular camera 30 is used for acquireing the real-time position of unmanned aerial vehicle the place ahead information in order to judge the barrier in camera the place ahead, and the IMU module sets up on binocular camera 30, and the IMU module is used for recording binocular camera 30's turned angle, and the IMU module can obtain the angle of binocular camera in X, Y, Z three directions in real time through resolving. The control unit is used for controlling the unmanned aerial vehicle to finish obstacle avoidance flight according to the image acquired by the binocular camera 30 and the angle information acquired by the IMU module. How to utilize the unmanned plane obstacle avoidance system to complete obstacle avoidance flight of the unmanned plane is described in detail below.
Step one, carrying out three-dimensional matching on the obstacle through a binocular camera to obtain real-time position information of the obstacle in front of the unmanned aerial vehicle. The unmanned aerial vehicle flies in real time, so real-time matching can be completed on an embedded system by the binocular matching algorithm, and the algorithm is required to be dense by using matched information. In addition, in order to reduce the complexity of the operation, the invention divides the obstacles according to the distance range for the matched depth map.
And step two, acquiring the maximum flyable area of the unmanned aerial vehicle in the current direction according to the real-time position information of the obstacle in front of the unmanned aerial vehicle. In this embodiment, the obstacle district is cut apart according to the real-time position information of unmanned aerial vehicle place ahead obstacle, according to the obstacle district that cuts apart and utilize the central diffusion method to acquire the unmanned aerial vehicle in the biggest area that can fly in the current direction.
Step three, judging the maximum flyable area of the unmanned aerial vehicle, if the maximum flyable area belongs to a safe flight area, namely the maximum flyable area of the unmanned aerial vehicle at the current binocular camera angle is larger than or equal to the safe area, taking the center of the maximum flyable area as the flight target direction of the unmanned aerial vehicle, and adjusting the course of the unmanned aerial vehicle until the unmanned aerial vehicle finishes obstacle avoidance; if the maximum flying area does not belong to the safe flying area, namely the maximum flying area of the unmanned aerial vehicle under the current binocular camera angle is smaller than the safe flying area, the holder rotates to adjust the position of the binocular camera, the first step to the third step are repeated until the adjusted position of the binocular camera, the maximum flying area of the unmanned aerial vehicle belongs to the safe flying area, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance.
And step four, after the unmanned aerial vehicle finishes obstacle avoidance, judging whether the cradle head rotates or not, if the cradle head rotates, controlling the unmanned aerial vehicle to recover to the initial course according to the camera deflection angle recorded by the IMU module, and enabling the cradle head to rotate to the initial position to finish obstacle avoidance flight of the unmanned aerial vehicle.
In this embodiment, the obstacle avoidance method for the unmanned aerial vehicle based on the combination of binocular vision and the IMU provided by the invention has the following advantages.
Firstly, an obstacle avoidance strategy of a small unmanned aerial vehicle based on combination of binocular vision and an IMU is as follows: the small unmanned aerial vehicle in the invention does not simply depend on a binocular vision system in formation, but realizes the obstacle avoidance strategy of the small unmanned aerial vehicle by combining binocular vision, IMU and a holder, the method is simple, the operability is strong, the obstacle avoidance strategy cost is low, various expensive sensors are not needed, and the obstacle avoidance effect is good.
Second, the strategy of segmenting obstacles from the depth map by distance range: the method for obtaining the barrier through binocular vision in the invention is to divide the barrier area from the depth map by using the distance range threatening the safe flight of the unmanned aerial vehicle, eliminate useless information and improve the processing efficiency of the depth map.
Third, a strategy for segmenting the flyable region from the depth map: the maximum flyable area in the wide angle of the current camera is obtained by utilizing a center diffusion method, and the maximum flyable area is compared with the safe flyable area, so that whether the course of the small unmanned aerial vehicle can be planned in the current direction or not is judged, the complexity of a traditional path planning algorithm is reduced, the planning efficiency is improved, the real-time effect can be achieved in a visual system, the method can be applied to the visual obstacle avoidance of an unmanned aerial vehicle system, the principle is clear, and the method is easy to understand.
In conclusion, the invention provides the obstacle avoidance method of the small unmanned aerial vehicle based on the combination of binocular vision and the IMU, so as to realize the obstacle avoidance of the small unmanned aerial vehicle. The method is characterized in that a small unmanned aerial vehicle is used as a platform, stereoscopic matching of a barrier in front of the camera is achieved through a binocular camera placed on an unmanned aerial vehicle holder, a safe area of the unmanned aerial vehicle for the barrier is explored through rotation of the holder, the area of the barrier is divided according to a distance range in a depth map, and the maximum safe rectangular area of a current depth map is obtained through a central diffusion method, so that the complexity of operation in the process of planning the path of the barrier is reduced. Compared with the prior art, the obstacle avoidance method of the small unmanned aerial vehicle has the following advantages.
Firstly, compared with the similar design of the existing obstacle avoidance scheme of the small unmanned aerial vehicle, the obstacle avoidance strategy of the small unmanned aerial vehicle based on the combination of binocular vision and IMU can utilize the platform to search for the obstacle in a rotating way, so that the limitation of the SLAM scheme or the known electronic map scheme which needs to be mapped in advance is avoided, most of obstacle avoidance behaviors can be completed in an unknown environment, and the method has the characteristics of clear structure, clear thought and feasibility.
Secondly, the binocular matching separates the obstacle information by using the distance information, and only the information required by the unmanned aerial vehicle for obstacle avoidance can be extracted, so that the complexity of a subsequent image processing algorithm is reduced.
Thirdly, the flyable area is obtained through subsequent processing of the depth map by using a center diffusion method, and whether the flyable area of the unmanned aerial vehicle is a safe flyable area or not can be obtained without a complex algorithm.
Fourthly, the method is low in cost, clear in principle and strong in operability, is basically suitable for obstacle avoidance of the small unmanned aerial vehicle under various conditions, has certain concealment and certain universal applicability, provides a new scheme for obstacle avoidance of the small unmanned aerial vehicle, improves research and development efficiency of the obstacle avoidance technology of the small unmanned aerial vehicle, and has certain inspiring effect on research on obstacle avoidance of the small underwater robot.
For ease of description, spatially relative terms such as "over … …", "over … …", "over … …", "over", etc. may be used herein to describe the spatial positional relationship of one device or feature to another device or feature as shown in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the figures is turned over, devices described as "above" or "on" other devices or configurations would then be oriented "below" or "under" the other devices or configurations. Thus, the exemplary term "above … …" may include both orientations of "above … …" and "below … …". The device may be otherwise variously oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
It should be noted that the terms "first", "second", and the like are used to define the components, and are only used for convenience of distinguishing the corresponding components, and the terms have no special meanings unless otherwise stated, and therefore, the scope of the present invention should not be construed as being limited.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. An unmanned aerial vehicle obstacle avoidance method based on combination of binocular vision and an IMU is characterized by comprising the following steps:
the method comprises the following steps that firstly, a barrier is subjected to stereo matching through a binocular camera so as to obtain real-time position information of the barrier in front of the unmanned aerial vehicle;
acquiring the maximum flyable area of the unmanned aerial vehicle in the current direction according to the real-time position information of the obstacle in front of the unmanned aerial vehicle;
step three, judging the maximum flyable area of the unmanned aerial vehicle, and if the maximum flyable area belongs to a safe flying area, adjusting the course of the unmanned aerial vehicle until the unmanned aerial vehicle finishes obstacle avoidance; if the maximum flyable area does not belong to the safe flying area, the cradle head rotates to adjust the position of the binocular camera, the first step, the second step and the third step are repeated until the maximum flyable area of the unmanned aerial vehicle belongs to the safe flying area under the adjusted position of the binocular camera, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance;
and step four, judging whether the cradle head rotates or not, if the cradle head rotates, controlling the unmanned aerial vehicle to recover to the initial course according to the camera deflection angle recorded by the IMU module, and enabling the cradle head to rotate to the initial position to finish obstacle avoidance flight of the unmanned aerial vehicle.
2. The unmanned aerial vehicle obstacle avoidance method based on the combination of binocular vision and IMU according to claim 1, wherein the first step specifically comprises: and (3) carrying out three-dimensional matching on the obstacle by adopting an ELAS three-dimensional matching algorithm and utilizing a binocular camera to obtain the real-time position information of the obstacle in front of the unmanned aerial vehicle.
3. The unmanned aerial vehicle obstacle avoidance method based on the combination of binocular vision and IMU according to claim 2, wherein the second step specifically comprises: and segmenting an obstacle area according to the real-time position information of the obstacle in front of the unmanned aerial vehicle, and acquiring the maximum flying area of the unmanned aerial vehicle in the current direction according to the segmented obstacle area by using a central diffusion method.
4. The unmanned aerial vehicle obstacle avoidance method based on the combination of binocular vision and the IMU according to any one of claims 1 to 3, wherein in the third step, if the maximum flyable area of the unmanned aerial vehicle under the current binocular camera angle belongs to a safe flying area, adjusting the heading of the unmanned aerial vehicle until the unmanned aerial vehicle finishes obstacle avoidance specifically comprises: if the maximum flying area of the unmanned aerial vehicle is larger than or equal to the safe area under the current binocular camera angle, the maximum flying area of the unmanned aerial vehicle belongs to the safe flying area, the center of the maximum flying area is used as the flying target direction of the unmanned aerial vehicle, and the course of the unmanned aerial vehicle is adjusted until the unmanned aerial vehicle finishes obstacle avoidance.
5. The unmanned aerial vehicle obstacle avoidance method based on the combination of binocular vision and IMU according to any one of claims 1 to 3, wherein in the third step, if the maximum flyable area of the unmanned aerial vehicle under the current binocular camera angle is smaller than a safe area, the maximum flyable area of the unmanned aerial vehicle does not belong to the safe flying area.
6. The unmanned aerial vehicle obstacle avoidance method based on combination of binocular vision and IMU according to claim 5, wherein in the third step, the rotation of the pan-tilt to adjust the positions of the binocular cameras specifically comprises: the holder rotates around a first direction to adjust the position of the binocular camera or rotates around a second direction to adjust the position of the binocular camera, and the first direction is perpendicular to the second direction.
7. An unmanned aerial vehicle obstacle avoidance system based on combination of binocular vision and an IMU (inertial measurement Unit), which is characterized in that the unmanned aerial vehicle obstacle avoidance system uses the unmanned aerial vehicle obstacle avoidance method according to any one of claims 1 to 6.
8. The binocular vision and IMU combined unmanned aerial vehicle obstacle avoidance system of claim 7, wherein the unmanned aerial vehicle obstacle avoidance system comprises:
an unmanned aerial vehicle (10);
a cradle head (20), the cradle head (20) being rotatably arranged on the unmanned aerial vehicle (10);
the binocular camera (30) is fixedly arranged on the cloud deck (20), the cloud deck (20) can drive the binocular camera (30) to rotate around a first direction and a second direction, and the binocular camera (30) is used for acquiring front information of the unmanned aerial vehicle so as to judge the real-time position of an obstacle in front of the binocular camera (30);
an IMU module disposed on the binocular camera (30), the IMU module for recording a rotation angle of the binocular camera (30);
and the control unit is used for controlling the unmanned aerial vehicle to complete obstacle avoidance flight according to the image acquired by the binocular camera (30) and the angle information acquired by the IMU module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910225407.7A CN111736622B (en) | 2019-03-25 | 2019-03-25 | Unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910225407.7A CN111736622B (en) | 2019-03-25 | 2019-03-25 | Unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111736622A CN111736622A (en) | 2020-10-02 |
| CN111736622B true CN111736622B (en) | 2023-04-07 |
Family
ID=72645726
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910225407.7A Active CN111736622B (en) | 2019-03-25 | 2019-03-25 | Unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111736622B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115562348A (en) * | 2022-11-03 | 2023-01-03 | 国网福建省电力有限公司漳州供电公司 | Unmanned aerial vehicle image technology method based on transformer substation |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106292704A (en) * | 2016-09-07 | 2017-01-04 | 四川天辰智创科技有限公司 | The method and device of avoiding barrier |
| CN107077152A (en) * | 2016-11-30 | 2017-08-18 | 深圳市大疆创新科技有限公司 | Control method, equipment, system, unmanned plane and moveable platform |
| CN106843282B (en) * | 2017-03-17 | 2020-11-03 | 东南大学 | M100 development platform-based area complete search and obstacle avoidance system and method |
| CN106931961B (en) * | 2017-03-20 | 2020-06-23 | 成都通甲优博科技有限责任公司 | Automatic navigation method and device |
| CN207123277U (en) * | 2017-07-24 | 2018-03-20 | 国网湖北省电力公司黄石供电公司 | A kind of UAV system binocular aerial survey device |
| CN108140245B (en) * | 2017-12-25 | 2022-08-23 | 深圳市道通智能航空技术股份有限公司 | Distance measurement method and device and unmanned aerial vehicle |
-
2019
- 2019-03-25 CN CN201910225407.7A patent/CN111736622B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111736622A (en) | 2020-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11557210B2 (en) | Trajectory planner for a vehicle | |
| US11237572B2 (en) | Collision avoidance system, depth imaging system, vehicle, map generator and methods thereof | |
| US11029157B2 (en) | Autonomous vehicle navigation system and method | |
| Barry et al. | High‐speed autonomous obstacle avoidance with pushbroom stereo | |
| EP3428766B1 (en) | Multi-sensor environmental mapping | |
| KR101642828B1 (en) | Obstacle avoidance system and method based on multiple images | |
| EP3674657A1 (en) | Construction and update of elevation maps | |
| US20240086862A1 (en) | System, devices and methods for tele-operated robotics | |
| CN105759829A (en) | Laser radar-based mini-sized unmanned plane control method and system | |
| CN113467500B (en) | Unmanned aerial vehicle non-cooperative target tracking system based on binocular vision | |
| WO2023102911A1 (en) | Data collection method, data presentation method, data processing method, aircraft landing method, data presentation system and storage medium | |
| CN110244762A (en) | Based on the flight equipment avoiding collision and equipment, security system for listening sound to distinguish position | |
| CN110997488A (en) | System and method for dynamically controlling parameters for processing sensor output data | |
| CN111736622B (en) | Unmanned aerial vehicle obstacle avoidance method and system based on combination of binocular vision and IMU | |
| Ngo et al. | UAV Platforms for Autonomous Navigation in GPS-Denied Environments for Search and Rescue Missions | |
| Marlow et al. | Local terrain mapping for obstacle avoidance using monocular vision | |
| Kang | Development of a Peripheral-Central Vision System to Detect and Characterize Airborne Threats | |
| CN113093806B (en) | Omnidirectional obstacle avoidance method and system for aircraft space | |
| Radesjö | Collision Avoidance for a Fence Inspecting Drone Operating at an Airport | |
| Naveenkumar et al. | Autonomous Drone Using Time-of-Flight Sensor for Collision Avoidance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |