WO2022048541A1 - 基于双目视觉的环境感知方法、装置及无人飞行器 - Google Patents

基于双目视觉的环境感知方法、装置及无人飞行器 Download PDF

Info

Publication number
WO2022048541A1
WO2022048541A1 PCT/CN2021/115734 CN2021115734W WO2022048541A1 WO 2022048541 A1 WO2022048541 A1 WO 2022048541A1 CN 2021115734 W CN2021115734 W CN 2021115734W WO 2022048541 A1 WO2022048541 A1 WO 2022048541A1
Authority
WO
WIPO (PCT)
Prior art keywords
binocular
binocular camera
viewing angle
aerial vehicle
unmanned aerial
Prior art date
Application number
PCT/CN2021/115734
Other languages
English (en)
French (fr)
Inventor
郑欣
Original Assignee
深圳市道通智能航空技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市道通智能航空技术股份有限公司 filed Critical 深圳市道通智能航空技术股份有限公司
Publication of WO2022048541A1 publication Critical patent/WO2022048541A1/zh
Priority to US18/178,220 priority Critical patent/US20240153122A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G06T7/596—Depth or shape recovery from multiple images from stereo images from three or more stereo images
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64D—EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
    • B64D47/00—Equipment not otherwise provided for
    • B64D47/08—Arrangements of cameras
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U20/00—Constructional aspects of UAVs
    • B64U20/70—Constructional aspects of the UAV body
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/24—Arrangements for determining position or orientation
    • G05D1/243—Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals
    • G05D1/2435—Extracting 3D information
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2109/00—Types of controlled vehicles
    • G05D2109/20—Aircraft, e.g. drones
    • G05D2109/25—Rotorcrafts
    • G05D2109/254—Flying platforms, e.g. multicopters
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2111/00—Details of signals used for control of position, course, altitude or attitude of land, water, air or space vehicles
    • G05D2111/10—Optical signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Definitions

  • Embodiments of the present invention relate to the technical field of aircraft, and in particular, to a method and device for environment perception based on binocular vision, and an unmanned aerial vehicle.
  • UAV Unmanned Aerial Vehicle
  • UAV Unmanned Aerial Vehicle
  • UAVs are equipped with sensors for environmental perception.
  • Typical sensors include ultrasonic sensors, infrared sensors, TOF sensors and visual sensors.
  • ultrasonic sensors, infrared sensors, and TOF sensors cannot acquire flight images, and are usually used as auxiliary sensing means; although monocular vision sensors (ie, monocular cameras) have rich output information and low hardware costs, the acquired flight images are two-dimensional
  • monocular vision sensors ie, monocular cameras
  • the binocular vision sensor ie, binocular camera
  • the binocular vision sensor is based on the principle of parallax, because the stereo vision information generated can directly restore the three-dimensional coordinates of the measured point, and it has become a popular perception sensor.
  • the UAV is only equipped with binocular cameras in some directions, and cannot achieve omnidirectional perception; even if the UAV is equipped with 6 sets of binocular cameras, some binocular vision sensors are easy to be detected by the drone.
  • the arm is blocked, the viewing angle is narrow, and there is also a certain blind spot for perception.
  • the main technical problem solved by the embodiments of the present invention is to provide an environment perception method, device and unmanned aerial vehicle based on binocular vision, which can simplify the omnidirectional perception system and reduce the perception blind spot.
  • a technical solution adopted in the embodiment of the present invention is: firstly, a method for environment perception based on binocular vision is provided, the method is applied to an unmanned aerial vehicle, and the unmanned aerial vehicle is provided with five groups of Binocular cameras, the five groups of binocular cameras include a first binocular camera, a second binocular camera, a third binocular camera, a fourth binocular camera and a fifth binocular camera, the first binocular camera is set At the front of the fuselage of the unmanned aerial vehicle, the second binocular camera is arranged obliquely upward between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle, and the third binocular camera is arranged obliquely upward Between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle, the fourth binocular camera is arranged on the lower part of the fuselage of the unmanned aerial vehicle, and the fifth binocular camera is arranged on the unmanned
  • the method includes:
  • a three-dimensional map of the target scene is constructed according to the five sets of three-dimensional point cloud data in the world coordinate system.
  • the line connecting the optical centers of any binocular camera is parallel to the horizontal plane, and the two optical axes of any binocular camera are parallel to each other.
  • the included angles between the optical axis of the second binocular camera and the optical axis of the third binocular camera and the horizontal plane are all ⁇ .
  • the vertical viewing angle H 2 of the second binocular camera, the vertical viewing angle H 3 of the third binocular camera, and the vertical viewing angle H 4 of the fourth binocular camera satisfy: H 2 +H 3 +H 4 >360°;
  • the vertical viewing angle of the second binocular camera partially overlaps with the vertical viewing angle of the third binocular camera
  • the vertical viewing angle of the third binocular camera partially overlaps the vertical viewing angle of the fourth binocular camera, so The vertical viewing angle of the fourth binocular camera partially overlaps the vertical viewing angle of the second binocular camera.
  • the vertical viewing angle H 2 of the second binocular camera, the vertical viewing angle H 3 of the third binocular camera, and the vertical viewing angle H 4 of the fourth binocular camera also satisfy the following conditions:
  • the horizontal viewing angle V 1 of the first binocular camera, the horizontal viewing angle V 2 of the second binocular camera, the horizontal viewing angle V 3 of the third binocular camera, and the fifth binocular camera satisfies: V 1 +V 2 +V 3 +V 5 >360°;
  • the horizontal viewing angle of the first binocular camera partially overlaps with the horizontal viewing angle of the second binocular camera, and the horizontal viewing angle of the second binocular camera partially overlaps the horizontal viewing angle of the fifth binocular camera.
  • the horizontal viewing angle of the fifth binocular camera partially overlaps with the horizontal viewing angle of the third binocular camera, and the horizontal viewing angle of the third binocular camera partially overlaps the horizontal viewing angle of the first binocular camera.
  • the horizontal viewing angle V 1 of the first binocular camera, the horizontal viewing angle V 2 of the second binocular camera, the horizontal viewing angle V 3 of the third binocular camera, and the horizontal viewing angle of the fifth binocular camera also satisfies the following conditions:
  • the first binocular vertical viewing angle H 1 , the second binocular horizontal viewing angle V 2 , the fourth binocular vertical viewing angle H 4 , and the fifth binocular horizontal viewing angle V 5 satisfies: H 1 +V 2 +H 4 +V 5 >360°;
  • first binocular vertical viewing angle partially overlaps with the second binocular horizontal viewing angle
  • second binocular horizontal viewing angle partially overlaps the fourth binocular vertical viewing angle
  • fourth binocular vertical viewing angle partially overlaps.
  • the target vertical viewing angle partially overlaps the fifth binocular horizontal viewing angle.
  • the first binocular vertical viewing angle H1, the second binocular horizontal viewing angle V2, the fourth binocular vertical viewing angle H4, and the fifth binocular horizontal viewing angle V5 are also met:
  • the third binocular horizontal viewing angle is equal to the second binocular horizontal viewing angle.
  • the method before acquiring a set of binocular views through each of the five sets of binocular cameras, and generating a disparity map according to the binocular views, the method further includes:
  • a left mask and a right mask are respectively constructed according to the occlusion area.
  • the right mask generates a mask view corresponding to the binocular camera, and stores the mask view;
  • the initial three-dimensional point cloud data is generated, which specifically includes:
  • initial 3D point cloud data is generated.
  • the shielding area formed by the body body of the UAV includes at least one of the following:
  • an environment perception device based on binocular vision is provided.
  • the device is applied to an unmanned aerial vehicle.
  • the unmanned aerial vehicle is provided with five sets of binocular cameras, and the five sets of binocular cameras include a first binocular camera.
  • the first binocular camera is arranged at the front of the fuselage of the unmanned aerial vehicle
  • the second binocular camera is The binocular camera is arranged obliquely upward between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle
  • the third binocular camera is arranged obliquely upward between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle.
  • the fourth binocular camera is arranged on the lower part of the fuselage of the unmanned aerial vehicle
  • the fifth binocular camera is arranged on the rear part of the fuselage of the unmanned aerial vehicle
  • the device includes:
  • a disparity map generation module configured to obtain a set of binocular views through each of the five sets of binocular cameras, and generate a disparity map according to the binocular views;
  • an initial point cloud data generation module configured to generate initial three-dimensional point cloud data according to the disparity map and the internal parameters of the corresponding binocular camera respectively;
  • a point cloud data generation module for converting the coordinate system of the initial three-dimensional point cloud data into a world coordinate system, respectively, to obtain five sets of three-dimensional point cloud data in the world coordinate system;
  • a three-dimensional map construction module is used to construct a three-dimensional map of the target scene according to the three-dimensional point cloud data in the five groups of world coordinate systems.
  • the device further includes:
  • a contrasting binocular view acquisition module configured to obtain a set of contrasting binocular views through each set of binocular cameras of the five sets of binocular cameras under a preset environment
  • a mask view generation module configured to respectively construct a left mask and a right mask according to the occlusion region if it is determined that the left and right views in the comparison binocular view have occlusion areas formed by the body of the unmanned aerial vehicle, generating a mask view corresponding to the binocular camera according to the left mask and the right mask, and storing the mask view;
  • the initial point cloud data generation module is further configured to fuse the disparity map and the mask view corresponding to the binocular camera to generate a fused disparity map; according to the fused disparity map and the interior of the binocular camera parameters to generate the initial 3D point cloud data.
  • a third aspect provides an unmanned aerial vehicle, comprising:
  • a first binocular camera arranged at the front of the fuselage of the unmanned aerial vehicle
  • the second binocular camera is disposed obliquely upward between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle;
  • the third binocular camera is disposed obliquely upward between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle;
  • a fourth binocular camera arranged at the lower part of the fuselage of the unmanned aerial vehicle
  • a fifth binocular camera arranged at the rear of the fuselage of the unmanned aerial vehicle
  • a controller respectively connected to the first binocular camera, the second binocular camera, the third binocular camera, the fourth binocular camera and the fifth binocular camera, the controller includes: at least one processor, and
  • a memory in communication with the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor
  • the processor is capable of executing the method as described above.
  • another unmanned aerial vehicle comprising:
  • a first binocular camera arranged at the front of the fuselage of the unmanned aerial vehicle
  • the second binocular camera is disposed obliquely upward between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle;
  • the third binocular camera is disposed obliquely upward between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle;
  • a fourth binocular camera arranged at the lower part of the fuselage of the unmanned aerial vehicle
  • a fifth binocular camera arranged at the rear of the fuselage of the unmanned aerial vehicle.
  • a controller which is respectively connected to the first binocular camera, the second binocular camera, the third binocular camera, the fourth binocular camera and the fifth binocular camera;
  • the first binocular camera, the second binocular camera, the third binocular camera, the fourth binocular camera, and the fifth binocular camera are all used to obtain a set of binocular views , and generate a disparity map according to the binocular view; generate initial three-dimensional point cloud data according to the disparity map and its internal parameters; and convert the coordinate system of the initial three-dimensional point cloud data into a world coordinate system to obtain the world coordinate system 3D point cloud data in the coordinate system;
  • the controller is configured to obtain three-dimensional point cloud data according to the first binocular camera, the second binocular camera, the third binocular camera, the fourth binocular camera, and the fifth binocular camera Build a 3D map of the target scene.
  • the first binocular camera, the second binocular camera, the third binocular camera, the fourth binocular camera, and the fifth binocular camera are also used for preset Under the circumstance, obtain a set of comparative binocular views; if it is determined that the left and right views in the comparative binocular view have an occlusion area formed by the body of the unmanned aerial vehicle, respectively construct a left mask and a right mask according to the occlusion area.
  • embodiments of the present invention provide a non-volatile computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to be executed by an unmanned aerial vehicle to Implement the method described above.
  • an embodiment of the present invention provides a computer program product, where the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and the program instructions are used to be executed by an unmanned aerial vehicle , to implement the method described above.
  • the UAV of the embodiments of the present invention is provided with five sets of binocular cameras, and the five sets of binocular cameras include a first binocular camera and a second binocular camera , the third binocular camera, the fourth binocular camera and the fifth binocular camera, the first binocular camera is arranged on the front of the fuselage of the unmanned aerial vehicle, and the second binocular camera is tilted upward and arranged on the fuselage of the unmanned aerial vehicle Between the left side and the upper part of the fuselage, the third binocular camera is arranged obliquely upward between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle, the fourth binocular camera is arranged at the lower part of the fuselage of the unmanned aerial vehicle, the fifth The binocular camera is installed at the rear of the UAV; five sets of binocular cameras are used to perceive the environmental information of the front, rear, left, right
  • FIG. 1 is a schematic structural diagram of an implementation environment involved in an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of the principle of binocular ranging provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a hardware structure of an implementation environment involved in an embodiment of the present invention.
  • FIG. 4 is a flowchart of a binocular vision-based environment perception method provided by an embodiment of the present invention.
  • FIG. 5 is a flowchart of an environment perception method based on binocular vision provided by another embodiment of the present invention.
  • FIG. 6 is a schematic diagram of generating a mask view based on an occlusion area formed by a body body of an unmanned aerial vehicle according to an embodiment of the present invention
  • FIG. 7 is a schematic diagram of the installation of a second binocular camera and a third binocular camera provided by an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of a binocular perception area of an unmanned aerial vehicle under one viewing angle provided by an embodiment of the present invention
  • FIG. 9 is a schematic diagram of a binocular perception area of an unmanned aerial vehicle from another perspective provided by an embodiment of the present invention.
  • FIG. 10 is a schematic diagram of a binocular perception area of an unmanned aerial vehicle under another viewing angle provided by an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of an environment perception device based on binocular vision provided by an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present invention.
  • FIG. 13 is a schematic structural diagram of an unmanned aerial vehicle provided by another embodiment of the present invention.
  • FIG. 1 is a schematic diagram of an implementation environment involved in various embodiments of the present invention.
  • the implementation environment includes an unmanned aerial vehicle 10 and multiple sets of binocular cameras, and the multiple sets of binocular cameras are respectively arranged on the unmanned aerial vehicle 10 . in different directions to obtain the flight environment around the UAV 10 .
  • the unmanned aerial vehicle 10 includes a body body, and the body body includes a fuselage 110, arms 120 connected to the fuselage 110, and power units 130 located on each of the arms 120.
  • the power units 130 are used to provide the unmanned aerial vehicle 10 with flight
  • the power mainly includes a motor (eg, a brushless motor) and a propeller connected to the motor.
  • a quad-rotor unmanned aerial vehicle is used as an example.
  • the unmanned aerial vehicle 10 may also be a tri-rotor unmanned aerial vehicle, a six-rotor unmanned aerial vehicle, a fixed-wing unmanned aerial vehicle, or the like.
  • the arm 120 can also be folded relative to the body 110 .
  • the UAV 10 further includes a landing gear 140 connected to the bottom of the fuselage 110 or the arm 120 .
  • the multiple sets of binocular cameras include a first binocular camera 21 disposed at the front of the fuselage of the unmanned aerial vehicle 10, and a second binocular camera slanted upwardly disposed between the left side of the fuselage and the upper part of the unmanned aerial vehicle 10. 22, the 3rd binocular camera 23 arranged obliquely upward between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle 10, and the fourth binocular camera 24 (not shown in FIG. shown), and a fifth binocular camera 25 (not shown in FIG. 1 ) disposed at the rear of the fuselage of the UAV 10 .
  • Each binocular camera includes left and right cameras and an image processor.
  • the left and right cameras are used to obtain two images of the measured object from different positions.
  • the image processor calculates the distance between corresponding points of the images based on the principle of parallax. position deviation to obtain the 3D geometric information of the object.
  • the feature point P is a certain point on the subject
  • O R and O T are the optical centers of the two cameras, respectively
  • the imaging points of the feature point P on the two camera photoreceptors are P and P, respectively '(The imaging plane of the camera is rotated and placed in front of the lens)
  • f is the focal length of the camera
  • B Baseline
  • Z is the depth information to be obtained
  • the distance from point P to point P' is set as dis ,but:
  • the focal length f and the center distance B of the two cameras can be obtained by calibration. Therefore, as long as the value of (X R - X T ) is obtained, that is, the difference between the X coordinates corresponding to the same spatial point in the imaging of the two cameras (disparity) to obtain the depth information of point P. Further, by calibrating the obtained internal parameters and external parameters, the image processor can also obtain the three-dimensional coordinates of the point P in the camera coordinate system and the world coordinate system.
  • the internal parameters of the binocular camera reflect the projection relationship between the camera coordinate system and the image coordinate system
  • the external parameters of the binocular camera reflect the rotation R and translation T relationship between the camera coordinate system and the world coordinate system.
  • the three-dimensional coordinates of the feature point P in the camera coordinate system can be obtained by the following formula:
  • cx, cy, fx, and fy are the internal parameters of the binocular camera; px, py are the pixel coordinates of the feature point P on the disparity map.
  • the binocular camera coordinate system can be converted to the world coordinate system of the UAV through the corresponding external parameter matrix.
  • the left image is used as the benchmark, and the mapping relationship between the left camera and the world coordinates is used to calculate the three-dimensional coordinates of the feature point P in the world coordinate system;
  • the right image is used as the benchmark, and the mapping relationship between the right camera and the world coordinates is used to calculate the three-dimensional coordinates of the feature point P in the world coordinate system.
  • the UAV 10 further includes a controller 30 mounted on the body body, a first binocular camera 21 , a second binocular camera 22 , a third binocular camera 23 , a fourth binocular camera 24 ,
  • the fifth binocular cameras 25 are all connected to the controller 30, and the controller 30 is used to obtain the three-dimensional point cloud data transmitted by the image processor of each binocular camera 25, and construct a three-dimensional map or three-dimensional modeling according to the above-mentioned three-dimensional point cloud data, so as to obtain the three-dimensional point cloud data.
  • the control power unit 130 performs tasks such as obstacle avoidance, braking, and path planning.
  • the above-mentioned construction of a three-dimensional map or three-dimensional modeling based on the three-dimensional point cloud data may also be performed by the upper computer 40 that is connected to the unmanned aerial vehicle in communication.
  • the image processors of the binocular cameras, and the image processor and the controller 30 may or may not be physically separated; for example, the image processors of the binocular cameras and the control
  • the controller 30 may be integrated on the same chip, and/or the function of the image processor of each binocular camera and the function of the controller 30 may also be performed by the same controller.
  • FIG. 4 is a flowchart of a binocular vision-based environment perception method provided by an embodiment of the present invention.
  • the method is applied to an unmanned aerial vehicle, and five sets of binocular cameras are provided on the unmanned aerial vehicle.
  • the method includes: :
  • Step 210 Obtain a set of binocular views through each of the five sets of binocular cameras, and generate a disparity map according to the binocular views.
  • the five sets of binocular cameras include a first binocular camera, a second binocular camera, a third binocular camera, a fourth binocular camera, and a fifth binocular camera, and the five sets of binocular cameras are respectively set in unmanned In different directions of the aircraft, it is used to obtain the environmental information around the UAV in all directions.
  • the first binocular camera is arranged at the front of the fuselage of the unmanned aerial vehicle, and can be installed on its mounting surface horizontally, vertically or obliquely, with the optical center facing forward, and is used to obtain environmental information in front of the unmanned aerial vehicle.
  • the second binocular camera is tilted upward and arranged between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle.
  • the optical center of the second binocular camera is inclined to the left and upward, and is used to obtain the left side and obliquely upper part of the unmanned aerial vehicle. environment information;
  • the third binocular camera is tilted upward and set up between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle, and the optical center of the third binocular camera is inclined to the right and upward to obtain the right side of the unmanned aerial vehicle. side and diagonally above environmental information.
  • the second binocular camera and the third binocular camera By slanting the second binocular camera and the third binocular camera, not only a group of binocular cameras disposed on the upper part of the fuselage of the UAV can be reduced, but also the body body of the UAV can be reduced (or even removed) , especially the occlusion of the arm to reduce the blind spot of perception.
  • the fourth binocular camera is arranged at the lower part of the fuselage of the unmanned aerial vehicle, and can be installed on its installation surface horizontally, vertically or obliquely, with the optical center facing downward, and is used to obtain environmental information under the unmanned aerial vehicle.
  • the fifth binocular camera is arranged at the rear of the fuselage of the unmanned aerial vehicle, and can be installed on its mounting surface horizontally, vertically or obliquely, with the optical center facing backward, and is used to obtain environmental information behind the unmanned aerial vehicle.
  • the step specifically includes: obtaining a first group of binocular views through a first binocular camera, and generating a first disparity map according to the first set of binocular views; obtaining a second set of binocular views through a second binocular camera, and The second set of disparity maps is generated from the second set of binocular views; the third set of binocular views is obtained through the third binocular camera, and the third set of disparity maps is generated according to the third set of binocular views; the fourth set of binocular views is obtained through the fourth binocular camera and generate a fourth disparity map according to the fourth set of binocular views; obtain the fifth set of binocular views through the fifth binocular camera, and generate a fifth disparity map according to the fifth set of binocular views.
  • the corresponding disparity map may be generated according to the binocular view based on the binocular matching algorithm.
  • the disparity map is calculated by using the BM algorithm or the SGBM algorithm.
  • Step 220 Generate initial three-dimensional point cloud data according to the disparity map and the corresponding internal parameters of the binocular camera respectively.
  • the obtained first disparity map corresponds to the first binocular camera
  • the second disparity map corresponds to the second binocular camera
  • the third disparity map corresponds to the third binocular camera
  • the fourth disparity map corresponds to the third binocular camera
  • this step specifically includes: generating a first group of initial three-dimensional point cloud data according to the first disparity map and the internal parameters of the first binocular camera, according to The second parallax map and the internal parameters of the second binocular camera generate a second set of initial 3D point cloud data, and according to the third parallax map and the internal parameters of the third binocular camera, generate a third set of initial 3D point cloud data, according to The fourth disparity map and the internal parameters of the fourth binocular camera generate a fourth set of initial 3D point cloud data, and a fifth set of initial 3D point cloud data is generated according to the fifth parallax map and the internal parameters of the fifth binocular camera.
  • B is the baseline length
  • disparity is the disparity data obtained from the disparity map
  • cx, cy, fx, fy are the internal parameters of the corresponding binocular camera
  • px, py are the pixel coordinates of point P on the disparity map, that is, The pixel coordinates of point P in the left image or the pixel coordinates in the right image.
  • the initial 3D point cloud data can be represented as a matrix Pcam constructed by the 3D coordinates of N feature points,
  • Step 230 Convert the coordinate system of the initial three-dimensional point cloud data to the world coordinate system, respectively, to obtain five sets of three-dimensional point cloud data in the world coordinate system.
  • it includes: converting the coordinate system of the first initial 3D point cloud data into the world coordinate system according to the external parameter matrix of the first binocular camera, so as to obtain the first 3D point cloud data in the world coordinate system;
  • the external parameter matrix of the second initial three-dimensional point cloud data is converted into the world coordinate system to obtain the second three-dimensional point cloud data in the world coordinate system;
  • Step 240 Construct a three-dimensional map of the target scene according to the five sets of three-dimensional point cloud data in the world coordinate system.
  • a three-dimensional map of the target scene is constructed according to the first three-dimensional point cloud data, the second three-dimensional point cloud data, the third three-dimensional point cloud data, the fourth three-dimensional point cloud data and the fifth three-dimensional point cloud data in the world coordinate system.
  • 3D mapping or modeling can use existing methods such as octree, ewok, point cloud map, mesh, etc.
  • the five sets of 3D point cloud data include the environmental information in front of, left, diagonally above, right, diagonally above, below, and behind the UAV
  • building a 3D map based on the five sets of 3D point cloud data can reconstruct the The real scene of the front, rear, left, right, up and down six directions of the UAV. If the five sets of binocular cameras cannot perform omnidirectional perception at the same time, you can control the UAV to rotate at a certain angle, obtain five sets of 3D point cloud data again, and construct a 3D point cloud based on the 3D point cloud data acquired twice or more times. Complete 3D map.
  • the first binocular camera is arranged on the front of the fuselage of the unmanned aerial vehicle
  • the second binocular camera is arranged obliquely upward between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle
  • the third binocular camera is arranged at the lower part of the fuselage of the unmanned aerial vehicle
  • the fifth binocular camera is arranged at the rear of the fuselage of the unmanned aerial vehicle
  • FIG. 5 is a flowchart of another binocular vision-based environment perception method provided by an embodiment of the present invention. The method is applied to an unmanned aerial vehicle. The structure of the human aircraft is the same, and the difference is that the method of this embodiment includes:
  • Step 310 In a preset environment, obtain a set of contrasting binocular views through each set of binocular cameras of the five sets of binocular cameras.
  • the first set of comparison binocular views is obtained through the first binocular camera
  • the second set of comparison binocular views is obtained through the second binocular camera
  • the third set of comparison binocular views is obtained through the third binocular camera.
  • a fourth group of contrasting binocular views is obtained through the fourth binocular camera
  • a fifth group of contrasting binocular views is obtained through the fifth binocular camera.
  • the environmental background of the preset environment is a single background, for example, a white background or a green background, so as to accurately identify the occlusion area of the body body of the UAV in the comparison binocular view.
  • Step 320 Determine whether there is an occlusion area formed by the body of the unmanned aerial vehicle on the left and right views in the comparison binocular view.
  • the shielding area formed by the body body of the unmanned aerial vehicle may include: the shielding area formed by the arm of the unmanned aerial vehicle, the shielding area formed by the fuselage of the unmanned aerial vehicle The shielded area, the shielded area formed by the power device (eg, the propeller) of the UAV, and/or the shielded area formed by the protection device (eg, the landing gear) of the UAV.
  • the power device eg, the propeller
  • the protection device eg, the landing gear
  • the shielding area formed by the body body of the unmanned aerial vehicle may also be different.
  • the occlusion area formed by the body of the UAV appears in the contrasting binocular view of the second binocular camera and the third binocular camera, while in another implementation environment, the UAV The occlusion area formed by the body of the fuselage also appears in the control binocular view of the fourth binocular machine.
  • this step specifically includes: determining whether there is an occlusion area formed by the body of the unmanned aerial vehicle on the left and right views in the first group of comparison binocular views, and determining whether there is an unmanned person on the left and right views in the second group of comparison binocular views
  • the occlusion area formed by the body body of the aircraft determine whether there is an occlusion area formed by the body body of the unmanned aerial vehicle in the left and right views in the third group of comparison binocular views, and determine whether there is any occlusion area in the left and right views in the fourth group of comparison binocular views
  • For the occlusion area formed by the body of the unmanned aerial vehicle determine whether there is an occlusion area formed by the body of the unmanned aerial vehicle on the left and right views in the fifth group of comparative binocular views.
  • Step 330 If so, construct a left mask and a right mask according to the occlusion area, respectively, and generate a mask view corresponding to the binocular camera according to the left mask and the right mask, and store the mask view.
  • the mask is a black and white binary image, the black area is covered, and the white area is reserved, and the mask is constructed, that is, the black area and the white area of the mask are determined.
  • a first left mask and a first right mask are respectively constructed according to the occlusion areas.
  • the mask and the first right mask generate the first mask view corresponding to the first binocular camera, and store the first mask view.
  • a second left mask and a second right mask are respectively constructed according to the occlusion area, and according to the second left mask and The second right mask generates a second mask view corresponding to the second binocular camera, and stores the second mask view.
  • the third left mask and the third right mask are respectively constructed according to the occlusion area.
  • the third right mask generates a third mask view corresponding to the third binocular camera, and stores the third mask view;
  • a fourth left mask and a fourth right mask are respectively constructed according to the occlusion area.
  • the fourth right mask generates the fourth mask view corresponding to the fourth binocular camera, and stores the fourth mask view.
  • the fifth left mask and the fifth right mask are respectively constructed according to the occlusion area.
  • the fifth right mask generates a fifth mask view corresponding to the fifth binocular camera, and stores the fifth mask view.
  • FIG. 6 shows a schematic diagram of generating a mask view corresponding to the binocular camera based on the occlusion area formed by the body body of the unmanned aerial vehicle. Understandably, since the binocular algorithm matches adjacent pixels and the camera lens has mounting tolerances, the black masked area should be slightly larger than the actual area. Since the relative position of the binocular camera and the body of the UAV is fixed, the mask view can be used as a fixed configuration and subsequently applied to the disparity map output by the binocular camera.
  • Step 340 Obtain a set of binocular views through each of the five sets of binocular cameras, and generate a disparity map according to the binocular views.
  • step 210 in Embodiment 1 which is easily understood by those skilled in the art, and will not be repeated here.
  • Step 350 Generate initial 3D point cloud data according to the disparity map and the corresponding internal parameters of the binocular camera.
  • the obtained first disparity map corresponds to the first binocular camera
  • the second disparity map corresponds to the second binocular camera
  • the third disparity map corresponds to the third binocular camera
  • the fourth disparity map corresponds to the third binocular camera.
  • the four binocular cameras correspond
  • the fifth disparity map corresponds to the fifth binocular camera.
  • the steps specifically include: fusing the disparity map and the mask view corresponding to the binocular camera to generate a fusion disparity map;
  • the disparity map and the internal parameters of the binocular camera are fused to generate the initial 3D point cloud data.
  • the disparity map and the mask view corresponding to the binocular camera are fused, that is, the mask view is overlaid on the disparity map, and the adverse effects caused by the occlusion of the field of view are further removed through the mask, and the stability of the binocular matching algorithm is improved.
  • Step 360 Convert the coordinate system of the initial three-dimensional point cloud data to the world coordinate system, respectively, to obtain five sets of three-dimensional point cloud data in the world coordinate system.
  • Step 370 Construct a three-dimensional map of the target scene according to the five sets of three-dimensional point cloud data in the world coordinate system.
  • step 360 and step 370 reference may be made to step 230 and step 240 in Embodiment 1, and details are not repeated here.
  • a set of comparison binocular views is obtained through each of the five sets of binocular cameras.
  • a mask view corresponding to the binocular camera is generated based on the occlusion area, and the mask view is stored. influence and improve the stability of the binocular matching algorithm.
  • the embodiment of the present invention provides an unmanned aerial vehicle, and the structure of the unmanned aerial vehicle is different from the structure of the unmanned aerial vehicle involved in the above-mentioned embodiments in that the optical center of any binocular camera of the unmanned aerial vehicle of this embodiment
  • the connecting line is parallel to the horizontal plane, and the two optical axes of any binocular camera are parallel to each other.
  • the angle between the optical axis of the second binocular camera and the optical axis of the third binocular camera and the horizontal plane are all ⁇ .
  • the binocular cameras are symmetrically arranged on the left and right sides of the UAV.
  • the vertical viewing angle H 2 of the second binocular camera and the vertical viewing angle of the third binocular camera H 3 and the vertical viewing angle H 4 of the fourth binocular camera satisfy: H 2 +H 3 +H 4 >360°; and, the vertical viewing angle of the second binocular camera partially overlaps with the vertical viewing angle of the third binocular camera, the third The vertical angle of view of the triple binocular camera partially overlaps with the vertical angle of view of the fourth binocular camera, and the vertical angle of view of the fourth binocular camera partially overlaps the vertical angle of view of the second binocular camera.
  • the vertical viewing angle H2 of the second binocular camera, the vertical viewing angle H3 of the third binocular camera, and the vertical viewing angle H4 of the fourth binocular camera also satisfy the following conditions:
  • the horizontal viewing angle V 3 of the third binocular camera and the horizontal viewing angle V 5 of the fifth binocular camera satisfy: V 1 +V 2 +V 3 +V 5 >360°; and, the horizontal viewing angle of the first binocular camera is the same as
  • the horizontal viewing angle of the second binocular camera partially overlaps, the horizontal viewing angle of the second binocular camera partially overlaps the horizontal viewing angle of the fifth binocular camera, and the horizontal viewing angle of the fifth binocular camera partially overlaps the horizontal viewing angle of the third binocular camera.
  • the horizontal viewing angle of the triple binocular camera partially overlaps with the horizontal viewing angle of the first binocular camera.
  • first binocular horizontal viewing angle V 1 the second binocular horizontal viewing angle V 2 , the third binocular horizontal viewing angle V 3 , and the fifth binocular horizontal viewing angle V 5 also satisfy the following conditions:
  • the binocular vertical viewing angle H 4 and the fifth binocular horizontal viewing angle V 5 satisfy: H 1 +V 2 +H 4 +V 5 >360°; and, the first binocular vertical viewing angle and the second binocular horizontal viewing angle Partially overlapping, the second binocular horizontal viewing angle partially overlaps the fourth binocular vertical viewing angle, and the fourth binocular vertical viewing angle partially overlaps the fifth binocular horizontal viewing angle.
  • first binocular vertical viewing angle H1, the second binocular horizontal viewing angle V2, the fourth binocular vertical viewing angle H4, and the fifth binocular horizontal viewing angle V5 also satisfy the following conditions:
  • the first binocular vertical viewing angle, the third binocular horizontal viewing angle, the fourth binocular vertical viewing angle, the third binocular vertical viewing angle, the The five binocular horizontal viewing angles should also satisfy the above constraints.
  • the UAV of the embodiment of the present invention can achieve omnidirectional coverage of the binocular perception area by setting the specific installation of each binocular camera and defining the relationship between the horizontal and vertical viewing angles of each binocular camera.
  • FIG. 11 is a schematic diagram of an environment perception device based on binocular vision according to an embodiment of the present invention.
  • the device 400 is applied to the above-mentioned unmanned aerial vehicle.
  • Apparatus 400 includes:
  • a disparity map generation module 410 configured to obtain a set of binocular views through each of the five sets of binocular cameras, and generate a disparity map according to the binocular views;
  • the initial point cloud data generation module 420 is used for generating initial three-dimensional point cloud data according to the disparity map and the internal parameters of the corresponding binocular camera respectively;
  • the point cloud data generation module 430 is used to respectively convert the coordinate system of the initial three-dimensional point cloud data into the world coordinate system, so as to obtain five sets of three-dimensional point cloud data in the world coordinate system;
  • the three-dimensional map construction module 440 is configured to construct a three-dimensional map of the target scene according to the three-dimensional point cloud data in the five groups of world coordinate systems.
  • the apparatus 400 further includes:
  • a contrasting binocular view obtaining module 450 configured to obtain a set of contrasting binocular views through each set of binocular cameras of the five sets of binocular cameras under a preset environment;
  • the mask view generation module 460 is configured to construct a left mask and a right mask respectively according to the occlusion area if it is determined that there is an occlusion area formed by the body body of the unmanned aerial vehicle on the left and right views in the comparison binocular view, according to the left and right masks.
  • the right mask generates a mask view corresponding to the binocular camera, and stores the mask view;
  • the initial point cloud data generation module 420 is also used to fuse the disparity map and the mask view corresponding to the binocular camera to generate a fused disparity map; and generate an initial three-dimensional point cloud according to the fused disparity map and the internal parameters of the binocular camera data.
  • the apparatus 400 can respectively execute the binocular vision-based environment perception method provided by Embodiment 1 and Embodiment 2 of the present invention, and has corresponding functional modules and beneficial effects of the execution method,
  • the method for environment perception based on binocular vision provided in Embodiments 1 and 2 of the present invention For technical details that are not described in detail in the embodiments of the device, reference may be made to the method for environment perception based on binocular vision provided in Embodiments 1 and 2 of the present invention.
  • FIG. 12 is an unmanned aerial vehicle provided by an embodiment of the present invention, and the unmanned aerial vehicle 500 includes:
  • the first binocular camera 510 is arranged at the front of the fuselage of the unmanned aerial vehicle;
  • the second binocular camera 520 is disposed obliquely upward between the left side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle;
  • the third binocular camera 530 is disposed obliquely upward between the right side of the fuselage and the upper part of the fuselage of the unmanned aerial vehicle;
  • the fourth binocular camera 540 is arranged at the lower part of the fuselage of the unmanned aerial vehicle;
  • the fifth binocular camera 550 is arranged at the rear of the fuselage of the unmanned aerial vehicle;
  • the controller 560 is respectively connected with the first binocular camera 510, the second binocular camera 520, the third binocular camera 530, the fourth binocular camera 540 and the fifth binocular camera 550, and the controller 560 includes:
  • the processor 561 and the memory 562 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 12 .
  • the memory 562 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the binocular vision-based environmental perception in the embodiment of the present invention
  • the program instructions/modules corresponding to the method for example, the disparity map generation module 410, the initial point cloud data generation module 420, the point cloud data generation module 430, the three-dimensional map construction module 440, the contrast binocular view acquisition module 450 and mask view generation module 460).
  • the processor 561 executes the non-volatile software programs, instructions and modules stored in the memory 562, thereby implementing the binocular vision-based environment perception method of the method embodiment.
  • the memory 562 may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required by at least one function; the storage data area may store data created according to the usage of the PTZ, and the like. Additionally, memory 562 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 562 may optionally include memory located remotely relative to processor 561, which may be connected to the pan/tilt via a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • the one or more modules are stored in the memory 562, and when executed by the one or more processors 561, execute the binocular vision-based environment perception method in the method embodiment, for example, execute the above
  • the described method steps in FIG. 4 and FIG. 5 implement the functions of each module in FIG. 11 .
  • the unmanned aerial vehicle 500 can execute the binocular vision-based environment perception method provided by the embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
  • the method for environment perception based on binocular vision provided by the embodiment of the present invention For technical details not described in detail in this embodiment, reference may be made to the method for environment perception based on binocular vision provided by the embodiment of the present invention.
  • FIG. 13 is an unmanned aerial vehicle provided by another embodiment of the present invention, and the unmanned aerial vehicle 600 includes:
  • the first binocular camera 610 is arranged at the front of the fuselage of the unmanned aerial vehicle;
  • the second binocular camera 620 is disposed obliquely upward between the left side of the fuselage and the upper part of the fuselage of the UAV;
  • the third binocular camera 630 is disposed obliquely upward between the right side of the fuselage and the upper part of the fuselage of the UAV;
  • the fourth binocular camera 640 is arranged at the lower part of the fuselage of the unmanned aerial vehicle;
  • the fifth binocular camera 650 is arranged at the rear of the fuselage of the unmanned aerial vehicle;
  • the controller 660 is connected to the first binocular camera 610 , the second binocular camera 620 , the third binocular camera 630 , the fourth binocular camera 640 and the fifth binocular camera 650 respectively.
  • the first binocular camera 610, the second binocular camera 620, the third binocular camera 630, the fourth binocular camera 640 and the fifth binocular camera 650 all include:
  • the first binocular camera 610 and the one processor 611 are used as examples.
  • the processor 611 and the memory 612 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 13 .
  • the memory 612 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the generation of the disparity map shown in FIG. 11 in the embodiment of the present invention.
  • Module 410 initial point cloud data generation module 420 , point cloud data generation module 430 , contrast binocular view acquisition module 450 and mask view generation module 460 .
  • the processor 611 executes the following method steps by executing non-volatile software programs, instructions and modules stored in the memory 612:
  • the processor 611 also performs the following method steps by executing non-volatile software programs, instructions and modules stored in the memory 612:
  • the left mask and the right mask are respectively constructed according to the occlusion area, and the binocular image is generated according to the left mask and the right mask.
  • the mask view corresponding to the camera, and store the mask view;
  • initial 3D point cloud data is generated.
  • Controller 660 includes:
  • the processor 661 and the memory 662 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 13 .
  • the memory 662 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the three-dimensional map construction shown in FIG. 11 in the embodiment of the present invention.
  • the processor 661 executes the following method steps by executing non-volatile software programs, instructions and modules stored in the memory 662:
  • a three-dimensional map of the target scene is constructed according to the three-dimensional point cloud data of the first binocular camera 610 , the second binocular camera 620 , the third binocular camera 630 , the fourth binocular camera 640 , and the fifth binocular camera 650 .
  • each embodiment can be implemented by means of software plus a general hardware platform, and certainly can also be implemented by hardware.
  • Those of ordinary skill in the art can understand that all or part of the processes in the method of the embodiments can be implemented by computer program instructions related to hardware, and the program can be stored in a computer-readable storage medium, and when the program is executed , the flow of each method embodiment as described may be included.
  • the storage medium may be a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM) or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Remote Sensing (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Mechanical Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Automation & Control Theory (AREA)
  • Astronomy & Astrophysics (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Analysis (AREA)

Abstract

一种基于双目视觉的环境感知方法、装置及无人飞行器（10）。无人飞行器（10）设置有五组双目相机，五组双目相机包括第一双目相机（21）、第二双目相机（22）、第三双目相机（23）、第四双目相机（24）以及第五双目相机（25），第一双目相机（21）设置于无人飞行器（10）的机身前部，第二双目相机（22）倾斜向上设置于无人飞行器（10）的机身左侧与机身上部之间，第三双目相机（23）倾斜向上设置于无人飞行器（10）的机身右侧与机身上部之间，第四双目相机（24）设置于无人飞行器（10）的机身下部，第五双目相机（25）设置于无人飞行器（10）的机身后部。上述无人飞行器（10）能够简化全向感知系统，同时减小感知盲区。

Description

基于双目视觉的环境感知方法、装置及无人飞行器
本申请要求于2020年9月3日提交中国专利局、申请号为2020109157583、申请名称为“基于双目视觉的环境感知方法、装置及无人飞行器”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明实施例涉及飞行器技术领域，特别是涉及一种基于双目视觉的环境感知方法、装置及无人飞行器。
背景技术
随着飞行器相关技术的不断发展，无人飞行器(Unmanned Aerial Vehicle，UAV)，也称无人机成为诸多领域发展的热点；例如，近年来，无人机在灾情调查和救援、空中监控、输电线路巡检、航拍、航测以及军事领域均得到了广泛的应用。
为实现避障、刹车、路径规划等，无人机配备有传感器进行环境感知，典型的传感器包括超声波传感器、红外传感器、TOF传感器以及视觉传感器。其中，超声波传感器、红外传感器、TOF传感器无法获取飞行图像，通常作为辅助感知手段；单目视觉传感器(即，单目相机)虽然输出信息丰富并且硬件成本低，但所获取的飞行图像是二维的，难以满足复杂化的应用场景；而双目视觉传感器(即，双目相机)基于视差原理，因其产生的立体视觉信息可以直接恢复被测量点的三维坐标，成为感知传感器中的热门。
然而，受限于体积或者成本，无人机仅部分方向上配备有双目相机，无法实现全向感知；即使在无人机上配备有6组双目相机，因部分双目视觉传感器易被机臂遮挡，视角较窄，也存在一定感知盲区。
发明内容
本发明实施例主要解决的技术问题是提供一种基于双目视觉的环境感知方法、装置及无人飞行器，能够简化全向感知系统，同时减小感知盲区。
为实现上述目的，本发明实施例采用的一个技术方案是：第一方面，提供一种基于双目视觉的环境感知方法，所述方法应用于无人飞行器，所述无人飞行器设置有五组双目相机，所述五组双目相机包括第一双目相机、第二双目相机、第三双目相机、第四双目相机以及第五双目相机，所述第一双目相机设置于所述无人飞行器的机身前部，所述第二双目相机倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间，所述第三双目相机倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间，所述第四双目相机设置于所述无人飞行器的机身下部，所述第五双目相机设置于所述无人飞行器的机身后部；
所述方法包括：
通过所述五组双目相机的每一组双目相机获取一组双目视图，并根据所述双目视图生成视差图；
分别根据所述视差图及其对应的双目相机的内部参数，生成初始三维点云数据；
分别将所述初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据；
根据所述五组世界坐标系下的三维点云数据构建目标场景的三维地图。
可选地，任一双目相机的光心连线平行于水平面，且任一双目相机的两光轴相互平行。
可选地，所述第二双目相机的光轴、所述第三双目相机的光轴与水平面的夹角均为α。
在一些实施例中，所述第二双目相机的垂直视角H 2、所述第三双目相机的垂直视角H 3以及所述第四双目相机的垂直视角H 4满足： H 2+H 3+H 4>360°；
且，所述第二双目相机的垂直视角与所述第三双目相机的垂直视角部分重叠，所述第三双目相机的垂直与所述第四双目相机的垂直视角部分重叠，所述第四双目相机的垂直视角与所述第二双目相机的垂直视角部分重叠。
可选地，所述第二双目相机的垂直视角H 2、所述第三双目相机的垂直视角H 3、所述第四双目相机的垂直视角H 4还满足以下条件：
H 2+H 4-2α>180°；
H 3+H 4-2α>180°；
H 2+H 3>180°。
在一些实施例中，所述第一双目的水平视角V 1、所述第二双目相机的水平视角V 2、所述第三双目相机的水平视角V 3以及所述第五双目的水平视角V 5满足：V 1+V 2+V 3+V 5>360°；
且，所述第一双目相机的水平视角与所述第二双目相机的水平视角部分重叠，所述第二双目相机的水平视角与所述第五双目的水平视角部分重叠，所述第五双目的水平视角与所述第三双目相机的水平视角部分重叠，所述第三双目相机的水平视角与所述第一双目相机的水平视角部分重叠。
可选地，所述第一双目的水平视角V 1、所述第二双目相机的水平视角V 2、所述第三双目相机的水平视角V 3以及所述第五双目的水平视角V 5还满足以下条件：
V 1+V 2>180°；
V 2+V 5>180°；
V 5+V 3>180°；
V 3+V 1>180°。
在一些实施例中，所述第一双目的垂直视角H 1、所述第二双目的水平视角V 2、所述第四双目的垂直视角H 4、所述第五双目的水平视角V 5满足：H 1+V 2+H 4+V 5>360°；
且，所述第一双目的垂直视角与所述第二双目的水平视角部分重 叠，所述第二双目的水平视角与所述第四双目的垂直视角部分重叠，所述第四双目的垂直视角与所述第五双目的水平视角部分重叠。
可选地，所述第一双目的垂直视角H 1、所述第二双目的水平视角V 2、所述第四双目的垂直视角H 4、所述第五双目的水平视角V 5还满足以下条件：
H 1+V 2>180°；
V 2+H 4>180°；
H 4+V 5>180°；
V 5+H 1>180°。
可选地，所述第三双目的水平视角与所述第二双目的水平视角相等。
在一些实施例中，在所述通过所述五组双目相机的每一组双目相机获取一组双目视图，并根据所述双目视图生成视差图之前，所述方法还包括：
在预设环境下，通过所述五组双目相机的每一组双目相机获取一组对照双目视图；
若确定所述对照双目视图中的左右视图上有所述无人飞行器的机体本体形成的遮挡区域，根据所述遮挡区域分别构建左蒙版和右蒙版，根据所述左蒙版和所述右蒙版生成与所述双目相机对应的蒙版视图，并存储所述蒙版视图；
则所述根据所述视差图及其对应的双目相机的内部参数，生成初始三维点云数据，具体包括：
将所述视差图和所述双目相机对应的蒙版视图进行融合，生成融合视差图；
根据所述融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
可选地，所述无人飞行器的机体本体形成的遮挡区域包括以下至少一种：
所述无人飞行器的机臂形成的遮挡区域，所述无人飞行器的机身形 成的遮挡区域，所述无人飞行器的动力装置形成的遮挡区域，以及所述无人飞行器的保护装置形成的遮挡区域。
第二方面，提供一种基于双目视觉的环境感知装置，所述装置应用于无人飞行器，所述无人飞行器设置有五组双目相机，所述五组双目相机包括第一双目相机、第二双目相机、第三双目相机、第四双目相机以及第五双目相机，所述第一双目相机设置于所述无人飞行器的机身前部，所述第二双目相机倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间，所述第三双目相机倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间，所述第四双目相机设置于所述无人飞行器的机身下部，所述第五双目相机设置于所述无人飞行器的机身后部，所述装置包括：
视差图生成模块，用于通过所述五组双目相机的每一组双目相机获取一组双目视图，并根据所述双目视图生成视差图；
初始点云数据生成模块，用于分别根据所述视差图及其对应的双目相机的内部参数，生成初始三维点云数据；
点云数据生成模块，用于分别将所述初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据；
三维地图构建模块，用于根据所述五组世界坐标系下的三维点云数据构建目标场景的三维地图。
可选地，所述装置还包括：
对照双目视图获取模块，用于在预设环境下，通过所述五组双目相机的每一组双目相机获取一组对照双目视图；
蒙版视图生成模块，用于若确定所述对照双目视图中的左右视图上有所述无人飞行器的机体本体形成的遮挡区域，根据所述遮挡区域分别构建左蒙版和右蒙版，根据所述左蒙版和所述右蒙版生成与所述双目相机对应的蒙版视图，并存储所述蒙版视图；
所述初始点云数据生成模块，还用于将所述视差图和所述双目相机对应的蒙版视图进行融合，生成融合视差图；根据所述融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
第三方面，提供一种无人飞行器，包括：
第一双目相机，设置于所述无人飞行器的机身前部；
第二双目相机，倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间；
第三双目相机，倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间；
第四双目相机，设置于所述无人飞行器的机身下部；
第五双目相机，设置于所述无人飞行器的机身后部；
控制器，分别与所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机和所述第五双目相机连接，所述控制器包括：至少一个处理器，以及
存储器，所述存储器与所述至少一个处理器通信连接，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如上所述的方法。
第四方面，提供另一种无人飞行器，包括：
第一双目相机，设置于所述无人飞行器的机身前部；
第二双目相机，倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间；
第三双目相机，倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间；
第四双目相机，设置于所述无人飞行器的机身下部；
第五双目相机，设置于所述无人飞行器的机身后部；以及
控制器，分别与所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机和所述第五双目相机连接；
其中，所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机、所述第五双目相机均用于获取一组双目视图，并根据所述双目视图生成视差图；根据所述视差图及其内部参数，生成初始三维点云数据；以及将所述初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的三维点云数据；
所述控制器用于根据所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机、所述第五双目相机的三维点云数据构建目标场景的三维地图。
可选地，所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机、所述第五双目相机还均用于在预设环境下，获取一组对照双目视图；若确定所述对照双目视图中的左右视图上有所述无人飞行器的机体本体形成的遮挡区域，根据所述遮挡区域分别构建左蒙版和右蒙版，根据所述左蒙版和所述右蒙版生成与所述双目相机对应的蒙版视图，并存储所述蒙版视图；将所述视差图和所述双目相机对应的蒙版视图进行融合，生成融合视差图；根据所述融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
第五方面，本发明实施例提供一种非易失性计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于被无人飞行器执行，以实现如上所述的方法。
第六方面，本发明实施例提供一种计算机程序产品，计算机程序产品包括存储在非易失性计算机可读存储介质上的计算机程序，计算机程序包括程序指令，程序指令用于被无人飞行器执行，以实现如上所述的方法。
本发明实施例的有益效果是：区别于现有技术的情况，本发明实施例的无人飞行器设置有五组双目相机，五组双目相机包括第一双目相机、第二双目相机、第三双目相机、第四双目相机以及第五双目相机，第一双目相机设置于无人飞行器的机身前部，第二双目相机倾斜向上设置于无人飞行器的机身左侧与机身上部之间，第三双目相机倾斜向上设置于无人飞行器的机身右侧与机身上部之间，第四双目相机设置于无人飞行器的机身下部，第五双目相机设置于无人飞行器的机身后部；通过五组双目相机感知无人飞行器前、后、左、右、上、下六方位的环境信息，能够减少一组双目相机的设置，简化全向感知系统，同时，左右两侧的双目相机倾斜设置，还能够减小无人飞行器的机体本体的遮挡，减小感知盲区。
附图说明
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定，附图中具有相同参考数字标号的元件表示为类似的元件，除非有特别申明，附图中的图不构成比例限制。
图1是本发明实施例涉及的实施环境的结构示意图；
图2是本发明实施例提供的双目测距原理示意图；
图3是本发明实施例涉及的实施环境的硬件结构示意图；
图4是本发明实施例提供的基于双目视觉的环境感知方法的流程图；
图5是本发明另一实施例提供的基于双目视觉的环境感知方法的流程图；
图6是本发明实施例提供的基于无人飞行器的机体本体形成的遮挡区域生成蒙版视图的示意图；
图7是本发明实施例提供的第二双目相机和第三双目相机的安装示意图；
图8是本发明实施例提供的在一个视角下无人飞行器的双目感知区域的示意图；
图9是本发明实施例提供的在另一个视角下无人飞行器的双目感知区域的示意图；
图10是本发明实施例提供的在又一个视角下无人飞行器的双目感知区域的示意图；
图11是本发明实施例提供的基于双目视觉的环境感知装置的示意图；
图12是本发明实施例提供的无人飞行器的结构示意图；
图13是本发明另一实施例提供的无人飞行器的结构示意图。
具体实施方式
下面将结合附图对本发明实施例中的技术方案进行描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发 明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
需要说明的是，当元件被表述“固定于”另一个元件，它可以直接在另一个元件上、或者其间可以存在一个或多个居中的元件。当一个元件被表述“连接”另一个元件，它可以是直接连接到另一个元件、或者其间可以存在一个或多个居中的元件。本说明书所使用的术语“垂直的”、“水平的”、“左”、“右”以及类似的表述只是为了说明的目的。
此外，下面所描述的本发明不同实施方式中所涉及的技术特征只要彼此之间未构成冲突就可以相互结合。
图1是本发明各个实施例涉及的一种实施环境的示意图，如图1所示，该实施环境包括无人飞行器10和多组双目相机，多组双目相机分别设置在无人飞行器10的不同方位上，以获取无人飞行器10周边的飞行环境。
无人飞行器10包括机体本体，机体本体包括机身110、与机身110相连的机臂120、位于每个机臂120上的动力装置130，动力装置130用于给无人飞行器10提供飞行的动力，主要包括电机(如，无刷电机)以及与电机连接的螺旋桨。
图中以四旋翼无人飞行器为例，在其他可能的实施例中，无人飞行器10还可以为三旋翼无人飞行器、六旋翼无人飞行器、固定翼无人飞行器等。为了便于收纳和携带，机臂120还可相对于机身110折叠。
可选地，该无人飞行器10还包括与机身110底部或者机臂120连接的起落架140。
多组双目相机包括设置于无人飞行器10的机身前部的第一双目相机21，倾斜向上设置于无人飞行器10的机身左侧与机身上部之间的第二双目相机22，倾斜向上设置于无人飞行器10的机身右侧与机身上部之间的第三双目相机23，设置于无人飞行器10的机身下部的第四双目相机24(图1未示出)，以及设置于无人飞行器10的机身后部的第五双目相机25(图1未示出)。
每一双目相机均包括左、右两个相机和图像处理器，通过左、右两 个相机从不同的位置获取被测物体的两幅图像，图像处理器基于视差原理通过计算图像对应点间的位置偏差，以获取物体的三维几何信息。
如图2所示，特征点P是被摄物体上的某一点，O R与O T分别是两个相机的光心，特征点P在两个相机感光器上的成像点分别为P和P’(相机的成像平面经过旋转后放在了镜头前方)，f为相机焦距，B(Baseline)为两相机中心距，Z为待求的深度信息，设点P到点P’的距离为dis，则：
dis＝B-(X R-X T)；
根据相似三角形原理：
Figure PCTCN2021115734-appb-000001
可得：
Figure PCTCN2021115734-appb-000002
公式中，焦距f和两相机中心距B可通过标定得到，因此，只要获得了(X R－X T)的值，即，同一个空间点在两个相机成像中对应的X坐标的差值(disparity)即可求得点P的深度信息。进一步地，通过标定得到的内部参数和外部参数，图像处理器还可以通过求出点P在相机坐标系下和世界坐标系下的三维坐标。
其中，双目相机的内部参数反映的是相机坐标系到图像坐标系之间的投影关系，双目相机的外部参数反映的是相机坐标系和世界坐标系之间的旋转R和平移T关系。通过在双目相机前放置特制的标定参照物(通常为棋盘纸)，以使双目相机获取该物体的图像，可计算得出双目相机的内部参数和外部参数。
具体地，可通过以下公式求出特征点P在相机坐标系下的三维坐标：
Figure PCTCN2021115734-appb-000003
Figure PCTCN2021115734-appb-000004
Figure PCTCN2021115734-appb-000005
其中，cx、cy、fx、fy为双目相机的内部参数；px、py为特征点P在视差图上的像素坐标。
根据相机坐标系和世界坐标系的关系，通过相应的外部参数矩阵即可将该双目相机坐标系转换至无人飞行器的世界坐标系。在实际应用中，若px、py采用左图像中的像素坐标，则以左图像为基准，采用左相机和世界坐标的映射关系计算特征点P在世界坐标系下的三维坐标；如果px、py采用右图像中的像素坐标，则以右图像为基准，采用右相机和世界坐标的映射关系计算特征点P在世界坐标系下的三维坐标。
如图3所示，无人飞行器10还包括安装在机体本体上的控制器30，第一双目相机21、第二双目相机22、第三双目相机23、第四双目相机24、第五双目相机25均与控制器30连接，控制器30用于获取各双目相机25的图像处理器传输的三维点云数据，根据上述三维点云数据构建三维地图或三维建模，以控制动力装置130执行避障、刹车、路径规划等任务。
可以理解的是，上述根据三维点云数据构建三维地图或三维建模也可以由与无人飞行器通信连接的上位机40执行。还可以理解的是，各双目相机的图像处理器之间，和图像处理器和控制器30之间可以是或者也可以不是物理上分开的；例如，各双目相机的图像处理器和控制器30可以集成在同一芯片上，和/或各双目相机的图像处理器的功能和控制器30的功能也可由同一控制器执行。
基于上述描述，下面结合附图对本发明实施例作进一步阐述。
实施例1
请参阅图4，图4为本发明实施例提供的一种基于双目视觉的环境感知方法的流程图，该方法应用于无人飞行器，无人飞行器上设置有五组双目相机，方法包括：
步骤210：通过五组双目相机的每一组双目相机获取一组双目视图， 并根据双目视图生成视差图。
示例性地，五组双目相机包括第一双目相机、第二双目相机、第三双目相机、第四双目相机以及第五双目相机，五组双目相机分别设置在无人飞行器的不同方位上，用于全方位获取无人飞行器周边的环境信息。
其中，第一双目相机设置于无人飞行器的机身前部，可以横向、纵向或斜向安装在其安装面上，光心朝前，用于获取无人飞行器前方的环境信息。
第二双目相机倾斜向上设置于无人飞行器的机身左侧与机身上部之间，第二双目相机的光心朝左并向上倾斜，用于获取无人飞行器左侧方及斜上方的环境信息；第三双目相机倾斜向上设置于无人飞行器的机身右侧与机身上部之间，第三双目相机的光心朝右并向上倾斜，用于获取无人飞行器右侧方及斜上方的环境信息。通过将第二双目相机和第三双目相机倾斜设置，不仅可以减少一组设置在无人飞行器的机身上部的双目相机，而且，可以减小(甚至去除)无人飞行器的机体本体，尤其是机臂的遮挡，减少感知盲区。
第四双目相机设置于无人飞行器的机身下部，可以横向、纵向或斜向安装在其安装面上，光心朝下，用于获取无人飞行器下方的环境信息。
第五双目相机设置于无人飞行器的机身后部，可以横向、纵向或斜向安装在其安装面上，光心朝后，用于获取无人飞行器后方的环境信息。
该步骤具体包括：通过第一双目相机获取第一组双目视图，并根据第一组双目视图生成第一视差图；通过第二双目相机获取第二组双目视图，并根据第二组双目视图生成第二视差图；通过第三双目相机获取第三组双目视图，并根据第三组双目视图生成第三视差图；通过第四双目相机获取第四组双目视图，并根据第四组双目视图生成第四视差图；通过第五双目相机获取第五组双目视图，并根据第五组双目视图生成第五视差图。
可基于双目匹配算法根据双目视图生成对应的视差图。在一实施方式中，在对双目视图的两副图像立体校正后，使用BM算法或者SGBM 算法计算视差图。
步骤220：分别根据视差图及其对应的双目相机的内部参数，生成初始三维点云数据。
在上一步骤中，获取的第一视差图与第一双目相机对应，第二视差图与第二双目相机对应，第三视差图与第三双目相机对应，第四视差图与第四双目相机对应，第五视差图与第五双目相机对应；则该步骤具体包括：根据第一视差图及第一双目相机的内部参数，生成第一组初始三维点云数据，根据第二视差图及第二双目相机的内部参数，生成第二组初始三维点云数据，根据第三视差图及第三双目相机的内部参数，生成第三组初始三维点云数据，根据第四视差图及第四双目相机的内部参数，生成第四组初始三维点云数据，根据第五视差图及第五双目相机的内部参数，生成第五组初始三维点云数据。
可采用利用以下公式，获取各视差图中每一特征点的三维坐标：
Figure PCTCN2021115734-appb-000006
Figure PCTCN2021115734-appb-000007
Figure PCTCN2021115734-appb-000008
其中，B为基线长度，disparity为由视差图获取的视差数据，cx、cy、fx、fy为对应的双目相机的内部参数，px、py为点P在视差图上的像素坐标，也即点P在左图像中的像素坐标或者右图像中的像素坐标。
初始三维点云数据，可表示为由N个特征点的三维坐标构建的矩阵Pcam，
Figure PCTCN2021115734-appb-000009
步骤230：分别将初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据。
具体包括：根据第一双目相机的外部参数矩阵将第一初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的第一三维点云 数据；根据第二双目相机的外部参数矩阵将第二初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的第二三维点云数据；根据第三双目相机的外部参数矩阵将第三初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的第三三维点云数据；根据第四双目相机的外部参数矩阵将第四初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的第四三维点云数据；根据第五双目相机的外部参数矩阵将第五初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的第五三维点云数据。
根据双目相机的外部参数矩阵将初始三维点云数据的坐标系转换为世界坐标系，可通过公式p world＝T×p cam T得出，其中，T为对应的双目相机的外部参数矩阵。
步骤240：根据五组世界坐标系下的三维点云数据构建目标场景的三维地图。
即，根据世界坐标系下的第一三维点云数据、第二三维点云数据、第三三维点云数据、第四三维点云数据和第五三维点云数据构建目标场景的三维地图。三维建图或建模可使用现有的八叉树、ewok、点云地图、mesh等方法。
因五组三维点云数据包括无人飞行器前方、左侧方及斜上方、右侧方及斜上方、下方、后方的环境信息，因此，根据五组三维点云数据构建三维地图能够重构出无人飞行器前、后、左、右、上、下六方位的真实场景。若五组双目相机不能在同一时刻进行全向感知，则可以通过控制无人飞行器旋转一定的角度，再次获取五组三维点云数据，基于两次或者多次获取的三维点云数据构建出完整的三维地图。
本发明实施例将第一双目相机设置于无人飞行器的机身前部，第二双目相机倾斜向上设置于无人飞行器的机身左侧与机身上部之间，第三双目相机倾斜向上设置于无人飞行器的机身右侧与机身上部之间，第四双目相机设置于无人飞行器的机身下部，第五双目相机设置于无人飞行器的机身后部，通过五组双目相机感知无人飞行器前、后、左、右、上、下六方位的环境信息，能够减少一组双目相机的设置，简化全向感知系 统，同时，左右两侧的双目相机倾斜设置，还能够减小无人飞行器的机体本体的遮挡，减小感知盲区。
实施例2
请参阅图5，图5为本发明实施例提供的另一种基于双目视觉的环境感知方法的流程图，该方法应用于无人飞行器，该无人飞行器的结构可与上述实施例的无人飞行器的结构相同，不同之处在于，本实施例的方法包括：
步骤310：在预设环境下，通过五组双目相机的每一组双目相机获取一组对照双目视图。
即，在预设环境下，通过第一双目相机获取第一组对照双目视图，通过第二双目相机获取第二组对照双目视图，通过第三双目相机获取第三组对照双目视图，通过第四双目相机获取第四组对照双目视图，通过第五双目相机获取第五组对照双目视图。
可选地，预设环境的环境背景为单一背景，如，为白色背景或者绿色背景，以便于在对照双目视图中精确识别无人飞行器的机体本体的遮挡区域。
步骤320：分别确定对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域。
在实际应用中，根据无人飞机器的机体本体的结构，无人飞机器的机体本体形成的遮挡区域，可以包括：无人飞行器的机臂形成的遮挡区域、无人飞行器的机身形成的遮挡区域、无人飞行器的动力装置(如，螺旋桨)形成的遮挡区域和/或无人飞行器的保护装置(如，起落架)形成的遮挡区域。
根据无人飞机器的机体本体的结构以及双目相机的安装位置，在不同的实施方式中，无人飞行器的机体本体形成的遮挡区域也可能不一样。例如，在一种实施环境中，无人飞行器的机体本体形成的遮挡区域出现在第二双目相机和第三双目相机的对照双目视图中，而在另一实施环境中，无人飞行器的机体本体形成的遮挡区域还出现在第四双目机的 对照双目视图中。
因此，该步骤具体包括：确定第一组对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域，确定第二组对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域，确定第三组对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域，确定第四组对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域，确定第五组对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域。
示例性地，可以通过图像识别、轮廓检测、图像二值化等方式确定每一对照双目视图中的左右视图上是否有无人飞行器的机体本体形成的遮挡区域。
步骤330：若有，根据遮挡区域分别构建左蒙版和右蒙版，以及根据左蒙版和右蒙版生成与该双目相机对应的蒙版视图，并存储该蒙版视图。
蒙版即为一种黑白二值图，黑色区域为遮盖，白色区域为保留，构建蒙版，即，确定蒙版的黑色区域和白色区域。
具体地，若确定第一组对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，根据遮挡区域分别构建第一左蒙版和第一右蒙版，根据第一左蒙版和第一右蒙版生成与第一双目相机对应的第一蒙版视图，并存储第一蒙版视图。
若确定第二组对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，根据遮挡区域分别构建第二左蒙版和第二右蒙版，根据第二左蒙版和第二右蒙版生成与第二双目相机对应的第二蒙版视图，并存储第二蒙版视图。
若确定第三组对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，根据遮挡区域分别构建第三左蒙版和第三右蒙版，根据第三左蒙版和第三右蒙版生成与第三双目相机对应的第三蒙版视图，并存储第三蒙版视图；
若确定第四组对照双目视图中的左右视图上有无人飞行器的机体 本体形成的遮挡区域，根据遮挡区域分别构建第四左蒙版和第四右蒙版，根据第四左蒙版和第四右蒙版生成与第四双目相机对应的第四蒙版视图，并存储第四蒙版视图。
若确定第五组对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，根据遮挡区域分别构建第五左蒙版和第五右蒙版，根据第五左蒙版和第五右蒙版生成与第五双目相机对应的第五蒙版视图，并存储第五蒙版视图。
请参阅图6，图示出了基于无人飞行器的机体本体形成的遮挡区域生成与该双目相机对应的蒙版视图的示意图。可以理解的是，由于双目算法会匹配相邻像素，且相机镜头存在安装公差，黑色遮盖区域应比实际区域略扩大。因双目相机与无人飞行器的机体本体的相对位置是固定的，该蒙版视图可作为固定配置，后续运用于双目相机输出的视差图上。
步骤340：通过五组双目相机的每一组双目相机获取一组双目视图，并根据双目视图生成视差图。
该步骤可参考实施例1中的步骤210，其在本领域技术人员容易理解的范围内，此处不再赘述。
步骤350：分别根据视差图及其对应的双目相机的内部参数，生成初始三维点云数据。
与实施例1类似，获取的第一视差图与第一双目相机对应，第二视差图与第二双目相机对应，第三视差图与第三双目相机对应，第四视差图与第四双目相机对应，第五视差图与第五双目相机对应。然而，在本实施例中，若视差图对应的双目相机还对应有蒙版视图，则其步骤具体包括：将视差图和双目相机对应的蒙版视图进行融合，生成融合视差图；根据融合视差图及双目相机的内部参数，生成初始三维点云数据。
将视差图和双目相机对应的蒙版视图进行融合，即将蒙版视图覆盖到视差图上，通过蒙版进一步去除视野遮挡产生的不良影响，并提高双目匹配算法的稳定性。
步骤360：分别将初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据。
步骤370：根据五组世界坐标系下的三维点云数据构建目标场景的三维地图。
步骤360和步骤370可参考实施例1中的步骤230和步骤240，此处不再赘述。
本发明实施例首先在在预设环境下，通过五组双目相机的每一组双目相机获取一组对照双目视图，若确定对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，基于遮挡区域生成与该双目相机对应的蒙版视图，并存储该蒙版视图，通过将蒙版视图与后续获取的视差图进行融合，能够进一步去除视野遮挡产生的不良影响，并提高双目匹配算法的稳定性。
实施例3
本发明实施例提供一种无人飞行器，该无人飞行器的结构与上述实施例涉及的无人飞行器的结构的不同之处在于，本实施例的无人飞行器的任一双目相机的光心连线平行于水平面，且任一双目相机的两光轴相互平行。
可选地，如图7所示，第二双目相机的光轴、第三双目相机的光轴与水平面的夹角均为α，通过上述设置，可将第二双目相机和第三双目相机对称设置于无人飞行器的左右两侧。
如图8所示，在无人机的后/前视图上，为保证双目感知区域(阴影部分)全方向覆盖，第二双目相机的垂直视角H 2、第三双目相机的垂直视角H 3以及第四双目相机的垂直视角H 4满足：H 2+H 3+H 4>360°；且，第二双目相机的垂直视角与第三双目相机的垂直视角部分重叠，第三双目相机的垂直与第四双目相机的垂直视角部分重叠，第四双目相机的垂直视角与第二双目相机的垂直视角部分重叠。
进一步地，为了平衡各双目相机的感知区域，第二双目相机的垂直视角H 2、第三双目相机的垂直视角H 3、第四双目相机的垂直视角H 4还满足以下条件：
H 2+H 4-2α>180°；
H 3+H 4-2α>180°；
H 2+H 3>180°。
如图9所示，在无人飞行器的俯(仰)视图上，为保证双目感知区域(阴影部分)全方向覆盖，第一双目的水平视角V 1、第二双目相机的水平视角V 2、第三双目相机的水平视角V 3以及第五双目的水平视角V 5满足：V 1+V 2+V 3+V 5>360°；且，第一双目相机的水平视角与第二双目相机的水平视角部分重叠，第二双目相机的水平视角与第五双目的水平视角部分重叠，第五双目的水平视角与第三双目相机的水平视角部分重叠，第三双目相机的水平视角与第一双目相机的水平视角部分重叠。
进一步地，第一双目的水平视角V 1、第二双目相机的水平视角V 2、第三双目相机的水平视角V 3以及第五双目的水平视角V 5还满足以下条件：
V 1+V 2>180°；
V 2+V 5>180°；
V 5+V 3>180°；
V 3+V 1>180°。
如图10所示，在无人飞行器的左视图上，为保证双目感知区域(阴影部分)全方向覆盖，第一双目的垂直视角H 1、第二双目的水平视角V 2、第四双目的垂直视角H 4、第五双目的水平视角V 5满足：H 1+V 2+H 4+V 5>360°；且，第一双目的垂直视角与第二双目的水平视角部分重叠，第二双目的水平视角与第四双目的垂直视角部分重叠，第四双目的垂直视角与第五双目的水平视角部分重叠。
进一步地，第一双目的垂直视角H 1、第二双目的水平视角V 2、第四双目的垂直视角H 4、第五双目的水平视角V 5还满足以下条件：
H 1+V 2>180°；
V 2+H 4>180°；
H 4+V 5>180°；
V 5+H 1>180°。
在无人飞行器的右视图上，同样地，为保证双目感知区域(阴影部 分)全方向覆盖，第一双目的垂直视角、第三双目的水平视角、第四双目的垂直视角、第五双目的水平视角也应该满足上述约束条件。
本发明实施例的无人飞行器通过设置各双目相机的具体安装，以及限定各双目相机的水平视角、垂直视角的相互关系，能够实现双目感知区域的全方向覆盖。
实施例4
请参阅图11，图11为本发明实施例提供的一种基于双目视觉的环境感知装置的示意图，该装置400应用于如上所述的无人飞行器。
装置400包括：
视差图生成模块410，用于通过五组双目相机的每一组双目相机获取一组双目视图，并根据双目视图生成视差图；
初始点云数据生成模块420，用于分别根据视差图及其对应的双目相机的内部参数，生成初始三维点云数据；
点云数据生成模块430，用于分别将初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据；
三维地图构建模块440，用于根据五组世界坐标系下的三维点云数据构建目标场景的三维地图。
在另一实施方式中，装置400还包括：
对照双目视图获取模块450，用于在预设环境下，通过五组双目相机的每一组双目相机获取一组对照双目视图；
蒙版视图生成模块460，用于若确定对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，根据遮挡区域分别构建左蒙版和右蒙版，根据左蒙版和右蒙版生成与双目相机对应的蒙版视图，并存储蒙版视图；
初始点云数据生成模块420，还用于将视差图和双目相机对应的蒙版视图进行融合，生成融合视差图；根据融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
需要说明的是，在本发明实施例4中，装置400可分别执行本发明 实施例1和实施例2所提供的基于双目视觉的环境感知方法，具备执行方法相应的功能模块和有益效果，未在装置的实施例中详尽描述的技术细节，可参见本发明实施例1、2所提供的基于双目视觉的环境感知方法。未在本实施例中详尽描述的结构细节，可参见本发明实施例3所提供的无人飞行器的结构。
实施例5
图12为本发明实施例提供的一种无人飞行器，无人飞行器500包括：
第一双目相机510，设置于无人飞行器的机身前部；
第二双目相机520，倾斜向上设置于无人飞行器的机身左侧与机身上部之间；
第三双目相机530，倾斜向上设置于无人飞行器的机身右侧与机身上部之间；
第四双目相机540，设置于无人飞行器的机身下部；
第五双目相机550，设置于无人飞行器的机身后部；
控制器560，分别与第一双目相机510、第二双目相机520、第三双目相机530、第四双目相机540和第五双目相机550连接，控制器560包括：
至少一个处理器561以及与至少一个处理器561通信连接的存储器562，图12中以一个处理器561为例。
处理器561和存储器562可以通过总线或者其他方式连接，图12中以通过总线连接为例。
存储器562作为一种非易失性计算机可读存储介质，可用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，如本发明实施例中的基于双目视觉的环境感知方法对应的程序指令/模块(例如，图11所示的视差图生成模块410、初始点云数据生成模块420、点云数据生成模块430、三维地图构建模块440、对照双目视图获取模块450和蒙版视图生成模块460)。处理器561通过运行存储在存储器562中的非 易失性软件程序、指令以及模块，从而实现所述方法实施例的基于双目视觉的环境感知方法。
存储器562可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储根据云台使用所创建的数据等。此外，存储器562可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实施例中，存储器562可选包括相对于处理器561远程设置的存储器，这些远程存储器可以通过网络连接至云台。所述网络的实施例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
所述一个或者多个模块存储在所述存储器562中，当被所述一个或者多个处理器561执行时，执行所述方法实施例中的基于双目视觉的环境感知方法，例如，执行以上描述的图4、图5中的方法步骤，实现图11中各模块的功能。
无人飞行器500可执行本发明实施例所提供的基于双目视觉的环境感知方法，具备执行方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节，可参见本发明实施例所提供的基于双目视觉的环境感知方法。未在本实施例中详尽描述的结构细节，可参见本发明实施例3所提供的无人飞行器的结构。
实施例6
图13为本发明另一实施例提供的一种无人飞行器，无人飞行器600包括：
第一双目相机610，设置于无人飞行器的机身前部；
第二双目相机620，倾斜向上设置于无人飞行器的机身左侧与机身上部之间；
第三双目相机630，倾斜向上设置于无人飞行器的机身右侧与机身上部之间；
第四双目相机640，设置于无人飞行器的机身下部；
第五双目相机650，设置于无人飞行器的机身后部；
控制器660，分别与第一双目相机610、第二双目相机620、第三双目相机630、第四双目相机640和第五双目相机650连接。
其中，第一双目相机610、第二双目相机620、第三双目相机630、第四双目相机640和第五双目相机650均包括：
至少一个处理器611以及与至少一个处理器611通信连接的存储器612，图13中以第一双目相机610以及以一个处理器611为例。
处理器611和存储器612可以通过总线或者其他方式连接，图13中以通过总线连接为例。
存储器612作为一种非易失性计算机可读存储介质，可用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，如本发明实施例中图11所示的视差图生成模块410、初始点云数据生成模块420、点云数据生成模块430、对照双目视图获取模块450和蒙版视图生成模块460。处理器611通过运行存储在存储器612中的非易失性软件程序、指令以及模块，从而执行以下方法步骤：
获取一组双目视图，并根据双目视图生成视差图；
根据视差图及其内部参数，生成初始三维点云数据；
以及将初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的三维点云数据。
在另一实施方式中，处理器611通过运行存储在存储器612中的非易失性软件程序、指令以及模块，还执行以下方法步骤：
在预设环境下，获取一组对照双目视图；
若确定对照双目视图中的左右视图上有无人飞行器的机体本体形成的遮挡区域，根据遮挡区域分别构建左蒙版和右蒙版，根据左蒙版和所述右蒙版生成与双目相机对应的蒙版视图，并存储蒙版视图；
将视差图和双目相机对应的蒙版视图进行融合，生成融合视差图；
根据融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
控制器660包括：
至少一个处理器661以及与至少一个处理器661通信连接的存储器662，图13中以一个处理器661为例。
处理器661和存储器662可以通过总线或者其他方式连接，图13中以通过总线连接为例。
存储器662作为一种非易失性计算机可读存储介质，可用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，如本发明实施例中图11所示的三维地图构建模块440。处理器661通过运行存储在存储器662中的非易失性软件程序、指令以及模块，从而执行以下方法步骤：
根据第一双目相机610、第二双目相机620、第三双目相机630、第四双目相机640、第五双目相机650的三维点云数据构建目标场景的三维地图。
进一步地，上述存储器612、存储器662的具体结构，以及存储器612与处理器611的具体设置、存储器662与处理器661的具体设置，可参考实施例5中的存储器562的具体结构，和存储器562与处理器561的具体设置，不再赘述。
需要说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
通过以上的实施例的描述，本领域普通技术人员可以清楚地了解到各实施例可借助软件加通用硬件平台的方式来实现，当然也可以通过硬件。本领域普通技术人员可以理解实现所述实施例方法中的全部或部分流程是可以通过计算机程序指令相关的硬件来完成，所述的程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如所述各方法的实施例的流程。其中，所述的存储介质可为只读存储记忆体(Read-Only Memory,ROM)或随机存储记忆体(RandomAccessMemory,RAM)等。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；在本发明的思路下，以上实施例或者不同实施例中的技术特征之间也可以进行组合，步骤可以以任意顺序实现，并存在如上所述的本发明的不同方面的许多其它变化，为了简明，它们没有在细节中提供；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (17)

  1. 一种基于双目视觉的环境感知方法，其特征在于，所述方法应用于无人飞行器，所述无人飞行器设置有五组双目相机，所述五组双目相机包括第一双目相机、第二双目相机、第三双目相机、第四双目相机以及第五双目相机，所述第一双目相机设置于所述无人飞行器的机身前部，所述第二双目相机倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间，所述第三双目相机倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间，所述第四双目相机设置于所述无人飞行器的机身下部，所述第五双目相机设置于所述无人飞行器的机身后部；
    所述方法包括：
    通过所述五组双目相机的每一组双目相机获取一组双目视图，并根据所述双目视图生成视差图；
    分别根据所述视差图及其对应的双目相机的内部参数，生成初始三维点云数据；
    分别将所述初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据；
    根据所述五组世界坐标系下的三维点云数据构建目标场景的三维地图。
  2. 根据权利要求1所述的方法，其特征在于，任一双目相机的光心连线平行于水平面，且任一双目相机的两光轴相互平行。
  3. 根据权利要求2所述的方法，其特征在于，所述第二双目相机的光轴、所述第三双目相机的光轴与水平面的夹角均为α。
  4. 根据权利要求3所述的方法，其特征在于，所述第二双目相机的垂直视角H 2、所述第三双目相机的垂直视角H 3以及所述第四双目相机的垂直视角H 4满足：H 2+H 3+H 4>360°；
    且，所述第二双目相机的垂直视角与所述第三双目相机的垂直视角部分重叠，所述第三双目相机的垂直与所述第四双目相机的垂直视角部分重叠，所述第四双目相机的垂直视角与所述第二双目相机的垂直视角部分重叠。
  5. 根据权利要求4所述的方法，其特征在于，所述第二双目相机的垂直视角H 2、所述第三双目相机的垂直视角H 3、所述第四双目相机的垂直视角H 4还满足以下条件：
    H 2+H 4-2α>180°；
    H 3+H 4-2α>180°；
    H 2+H 3>180°。
  6. 根据权利要求3所述的方法，其特征在于，所述第一双目的水平视角V 1、所述第二双目相机的水平视角V 2、所述第三双目相机的水平视角V 3以及所述第五双目的水平视角V 5满足：V 1+V 2+V 3+V 5>360°；
    且，所述第一双目相机的水平视角与所述第二双目相机的水平视角部分重叠，所述第二双目相机的水平视角与所述第五双目的水平视角部分重叠，所述第五双目的水平视角与所述第三双目相机的水平视角部分重叠，所述第三双目相机的水平视角与所述第一双目相机的水平视角部分重叠。
  7. 根据权利要求6所述的方法，其特征在于，所述第一双目的水平视角V 1、所述第二双目相机的水平视角V 2、所述第三双目相机的水平视角V 3以及所述第五双目的水平视角V 5还满足以下条件：
    V 1+V 2>180°；
    V 2+V 5>180°；
    V 5+V 3>180°；
    V 3+V 1>180°。
  8. 根据权利要求3所述的方法，其特征在于，所述第一双目的垂直 视角H 1、所述第二双目的水平视角V 2、所述第四双目的垂直视角H 4、所述第五双目的水平视角V 5满足：H 1+V 2+H 4+V 5>360°；
    且，所述第一双目的垂直视角与所述第二双目的水平视角部分重叠，所述第二双目的水平视角与所述第四双目的垂直视角部分重叠，所述第四双目的垂直视角与所述第五双目的水平视角部分重叠。
  9. 根据权利要求8所述的方法，其特征在于，所述第一双目的垂直视角H 1、所述第二双目的水平视角V 2、所述第四双目的垂直视角H 4、所述第五双目的水平视角V 5还满足以下条件：
    H 1+V 2>180°；
    V 2+H 4>180°；
    H 4+V 5>180°；
    V 5+H 1>180°。
  10. 根据权利要求9所述的方法，其特征在于，所述第三双目的水平视角与所述第二双目的水平视角相等。
  11. 根据权利要求1-10任一项所述的方法，其特征在于，在所述通过所述五组双目相机的每一组双目相机获取一组双目视图，并根据所述双目视图生成视差图之前，所述方法还包括：
    在预设环境下，通过所述五组双目相机的每一组双目相机获取一组对照双目视图；
    若确定所述对照双目视图中的左右视图上有所述无人飞行器的机体本体形成的遮挡区域，根据所述遮挡区域分别构建左蒙版和右蒙版，根据所述左蒙版和所述右蒙版生成与所述双目相机对应的蒙版视图，并存储所述蒙版视图；
    则所述根据所述视差图及其对应的双目相机的内部参数，生成初始三维点云数据，具体包括：
    将所述视差图和所述双目相机对应的蒙版视图进行融合，生成融合视差图；
    根据所述融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
  12. 根据权利要求11所述的方法，其特征在于，所述无人飞行器的机体本体形成的遮挡区域包括以下至少一种：
    所述无人飞行器的机臂形成的遮挡区域，所述无人飞行器的机身形成的遮挡区域，所述无人飞行器的动力装置形成的遮挡区域，以及所述无人飞行器的保护装置形成的遮挡区域。
  13. 一种基于双目视觉的环境感知装置，其特征在于，所述装置应用于无人飞行器，所述无人飞行器设置有五组双目相机，所述五组双目相机包括第一双目相机、第二双目相机、第三双目相机、第四双目相机以及第五双目相机，所述第一双目相机设置于所述无人飞行器的机身前部，所述第二双目相机倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间，所述第三双目相机倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间，所述第四双目相机设置于所述无人飞行器的机身下部，所述第五双目相机设置于所述无人飞行器的机身后部，所述装置包括：
    视差图生成模块，用于通过所述五组双目相机的每一组双目相机获取一组双目视图，并根据所述双目视图生成视差图；
    初始点云数据生成模块，用于分别根据所述视差图及其对应的双目相机的内部参数，生成初始三维点云数据；
    点云数据生成模块，用于分别将所述初始三维点云数据的坐标系转换为世界坐标系，以获得五组世界坐标系下的三维点云数据；
    三维地图构建模块，用于根据所述五组世界坐标系下的三维点云数据构建目标场景的三维地图。
  14. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    对照双目视图获取模块，用于在预设环境下，通过所述五组双目相机的每一组双目相机获取一组对照双目视图；
    蒙版视图生成模块，用于若确定所述对照双目视图中的左右视图上有所述无人飞行器的机体本体形成的遮挡区域，根据所述遮挡区域分别构建左蒙版和右蒙版，根据所述左蒙版和所述右蒙版生成与所述双目相机对应的蒙版视图，并存储所述蒙版视图；
    所述初始点云数据生成模块，还用于将所述视差图和所述双目相机对应的蒙版视图进行融合，生成融合视差图；根据所述融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
  15. 一种无人飞行器，其特征在于，包括：
    第一双目相机，设置于所述无人飞行器的机身前部；
    第二双目相机，倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间；
    第三双目相机，倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间；
    第四双目相机，设置于所述无人飞行器的机身下部；
    第五双目相机，设置于所述无人飞行器的机身后部；
    控制器，分别与所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机和所述第五双目相机连接，所述控制器包括：至少一个处理器，以及
    存储器，所述存储器与所述至少一个处理器通信连接，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求1-12任一项所述的方法。
  16. 一种无人飞行器，其特征在于，包括：
    第一双目相机，设置于所述无人飞行器的机身前部；
    第二双目相机，倾斜向上设置于所述无人飞行器的机身左侧与机身上部之间；
    第三双目相机，倾斜向上设置于所述无人飞行器的机身右侧与机身上部之间；
    第四双目相机，设置于所述无人飞行器的机身下部；
    第五双目相机，设置于所述无人飞行器的机身后部；以及
    控制器，分别与所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机和所述第五双目相机连接；
    其中，所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机、所述第五双目相机均用于获取一组双目视图，并根据所述双目视图生成视差图；根据所述视差图及其内部参数，生成初始三维点云数据；以及将所述初始三维点云数据的坐标系转换为世界坐标系，以获得世界坐标系下的三维点云数据；
    所述控制器用于根据所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机、所述第五双目相机的三维点云数据构建目标场景的三维地图。
  17. 根据权利要求16所述的无人飞行器，其特征在于，所述第一双目相机、所述第二双目相机、所述第三双目相机、所述第四双目相机、所述第五双目相机还均用于在预设环境下，获取一组对照双目视图；若确定所述对照双目视图中的左右视图上有所述无人飞行器的机体本体形成的遮挡区域，根据所述遮挡区域分别构建左蒙版和右蒙版，根据所述左蒙版和所述右蒙版生成与所述双目相机对应的蒙版视图，并存储所述蒙版视图；将所述视差图和所述双目相机对应的蒙版视图进行融合，生成融合视差图；根据所述融合视差图及所述双目相机的内部参数，生成初始三维点云数据。
PCT/CN2021/115734 2020-09-03 2021-08-31 基于双目视觉的环境感知方法、装置及无人飞行器 WO2022048541A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/178,220 US20240153122A1 (en) 2020-09-03 2023-03-03 Binocular vision-based environment sensing method and apparatus, and unmanned aerial vehicle

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010915758.3A CN112052788B (zh) 2020-09-03 2020-09-03 基于双目视觉的环境感知方法、装置及无人飞行器
CN202010915758.3 2020-09-03

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/178,220 Continuation US20240153122A1 (en) 2020-09-03 2023-03-03 Binocular vision-based environment sensing method and apparatus, and unmanned aerial vehicle

Publications (1)

Publication Number Publication Date
WO2022048541A1 true WO2022048541A1 (zh) 2022-03-10

Family

ID=73607369

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/115734 WO2022048541A1 (zh) 2020-09-03 2021-08-31 基于双目视觉的环境感知方法、装置及无人飞行器

Country Status (3)

Country Link
US (1) US20240153122A1 (zh)
CN (1) CN112052788B (zh)
WO (1) WO2022048541A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117437563A (zh) * 2023-12-13 2024-01-23 黑龙江惠达科技股份有限公司 一种基于双目视觉的植保无人机打点方法、装置及设备

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112052788B (zh) * 2020-09-03 2024-04-02 深圳市道通智能航空技术股份有限公司 基于双目视觉的环境感知方法、装置及无人飞行器
CN215972078U (zh) * 2021-02-22 2022-03-08 深圳市道通智能航空技术股份有限公司 一种四旋翼无人机
CN116724279A (zh) * 2021-03-12 2023-09-08 深圳市大疆创新科技有限公司 可移动平台、可移动平台的控制方法及存储介质
CN113093806B (zh) * 2021-04-07 2022-06-14 涵涡智航科技(玉溪)有限公司 一种用于飞行器空间全向避障方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107483911A (zh) * 2017-08-25 2017-12-15 秦山 一种基于多目图像传感器的信号处理方法及系统
CN109883398A (zh) * 2019-03-07 2019-06-14 景遥（上海）信息技术有限公司 基于无人机倾斜摄影的植株绿量提取的系统及方法
US20190273909A1 (en) * 2016-11-14 2019-09-05 SZ DJI Technology Co., Ltd. Methods and systems for selective sensor fusion
CN110799853A (zh) * 2018-10-26 2020-02-14 深圳市大疆创新科技有限公司 一种环境感知系统及移动平台
CN110880202A (zh) * 2019-12-02 2020-03-13 中电科特种飞机系统工程有限公司 一种三维地形模型创建方法、装置、设备及存储介质
CN112052788A (zh) * 2020-09-03 2020-12-08 深圳市道通智能航空技术有限公司 基于双目视觉的环境感知方法、装置及无人飞行器

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106384382A (zh) * 2016-09-05 2017-02-08 山东省科学院海洋仪器仪表研究所 一种基于双目立体视觉的三维重建系统及其方法
CN108489454A (zh) * 2018-03-22 2018-09-04 沈阳上博智像科技有限公司 深度距离测量方法、装置、计算机可读存储介质及电子设备
CN109029417B (zh) * 2018-05-21 2021-08-10 南京航空航天大学 基于混合视觉里程计和多尺度地图的无人机slam方法
CN110225249B (zh) * 2019-05-30 2021-04-06 深圳市道通智能航空技术有限公司 一种对焦方法、装置、航拍相机以及无人飞行器

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190273909A1 (en) * 2016-11-14 2019-09-05 SZ DJI Technology Co., Ltd. Methods and systems for selective sensor fusion
CN107483911A (zh) * 2017-08-25 2017-12-15 秦山 一种基于多目图像传感器的信号处理方法及系统
CN110799853A (zh) * 2018-10-26 2020-02-14 深圳市大疆创新科技有限公司 一种环境感知系统及移动平台
CN109883398A (zh) * 2019-03-07 2019-06-14 景遥（上海）信息技术有限公司 基于无人机倾斜摄影的植株绿量提取的系统及方法
CN110880202A (zh) * 2019-12-02 2020-03-13 中电科特种飞机系统工程有限公司 一种三维地形模型创建方法、装置、设备及存储介质
CN112052788A (zh) * 2020-09-03 2020-12-08 深圳市道通智能航空技术有限公司 基于双目视觉的环境感知方法、装置及无人飞行器

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117437563A (zh) * 2023-12-13 2024-01-23 黑龙江惠达科技股份有限公司 一种基于双目视觉的植保无人机打点方法、装置及设备
CN117437563B (zh) * 2023-12-13 2024-03-15 黑龙江惠达科技股份有限公司 一种基于双目视觉的植保无人机打点方法、装置及设备

Also Published As

Publication number Publication date
CN112052788A (zh) 2020-12-08
US20240153122A1 (en) 2024-05-09
CN112052788B (zh) 2024-04-02

Similar Documents

Publication Publication Date Title
WO2022048541A1 (zh) 基于双目视觉的环境感知方法、装置及无人飞行器
JP5455124B2 (ja) カメラ姿勢パラメータ推定装置
US20170134713A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
WO2018210078A1 (zh) 无人机的距离测量方法以及无人机
US20170127045A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
CN110244282B (zh) 一种多相机系统和激光雷达联合系统及其联合标定方法
CN108474658B (zh) 地面形态检测方法及系统、无人机降落方法和无人机
WO2016070318A1 (en) Camera calibration
WO2019113966A1 (zh) 一种避障方法、装置和无人机
CN110225249B (zh) 一种对焦方法、装置、航拍相机以及无人飞行器
WO2019015158A1 (zh) 无人机避障方法及无人机
WO2019061064A1 (zh) 图像处理方法和设备
JP2022089269A (ja) キャリブレーション装置およびキャリブレーション方法
CN110675436A (zh) 基于3d特征点的激光雷达与立体视觉配准方法
CN112837207A (zh) 全景深度测量方法、四目鱼眼相机及双目鱼眼相机
WO2020114433A1 (zh) 一种深度感知方法，装置和深度感知设备
CN115880368A (zh) 一种电网巡检无人机障碍物检测方法、系统及存储介质
CN108195359B (zh) 空间数据的采集方法及系统
CN210986289U (zh) 四目鱼眼相机及双目鱼眼相机
CN117309856A (zh) 烟幕效果监测方法、装置、电子设备和存储介质
TWM594322U (zh) 全向立體視覺的相機配置系統
CN113093176B (zh) 线状障碍物检测方法、装置、电子设备和存储介质
WO2020107487A1 (zh) 图像处理方法和无人机
CN112804515A (zh) 全向立体视觉的摄像机配置系统及摄像机配置方法
CN215972078U (zh) 一种四旋翼无人机

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21863605

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21863605

Country of ref document: EP

Kind code of ref document: A1