CN105222760A - The autonomous obstacle detection system of a kind of unmanned plane based on binocular vision and method - Google Patents

Abstract

The present invention relates to the autonomous obstacle detection system of a kind of unmanned plane based on binocular vision and method, its technical characteristics is: this system is included on unmanned plane and is mounted with binocular vision system, other sensor assemblies and flight control system; The method comprises the visual information that binocular vision system obtains unmanned plane during flying environment, and treatedly obtains obstacle information; Other sensor units obtain the status information of unmanned plane; Flight control system receives obstacle information and unmanned plane status information, sets up flight path, generates flight steering order and sends to unmanned plane; Unmanned plane is according to the flight of flight steering order avoiding barrier.Visual information and other sensor informations merge by the present invention mutually, perception flight environment of vehicle information, carry out flight path control and path planning to evade barrier, efficiently solve the problem that unmanned plane vision keeps away barrier, make it possess to utilize Airborne camera to complete the ability that vision keeps away barrier.

Description

The autonomous obstacle detection system of a kind of unmanned plane based on binocular vision and method

Technical field

The invention belongs to unmanned air vehicle technique field, the autonomous obstacle detection system of especially a kind of unmanned plane based on binocular vision and method.

Background technology

Along with the development of aircraft correlation technique and the complicated of application scenarios thereof, its environment sensing ability is had higher requirement.The airmanship of view-based access control model has the advantages such as investigative range is wide, information capacity is large, and it has the features such as rapid to flight environment of vehicle change capture, reaction is sharp in addition, therefore in aircraft guiding navigation is studied, obtains increasing concern.

The environment sensing of view-based access control model belongs to passive measurement mode, compared with laser, radar and the Active measuring mode such as ultrasonic, the mutual interference of multiple measurement mechanism in testing process can be reduced, what is more important can be reduced in the probability be found when some specific environment (as battlefield) uses, and has stronger disguise.

In the flight environment of vehicle of view-based access control model, barrier aware application can use monocular or binocular vision, and wherein binocular vision can obtain the precision higher compared with monocular vision, and application is comparatively extensive.Monocular vision uses an Airborne camera to obtain flight map picture, but the three-dimensional information of flight environment of vehicle can be lost in image projection process, though the depth information of the method reducing environment of multiple image or off-line training can be utilized, but complex disposal process, cause airborne flush bonding processor to be difficult to realize real-time process, the mode can only accomplishing to return again after image being passed back land station resolves unmanned plane is at present carried out barrier and is evaded control.And binocular vision is based on principle of parallax, its Stereo Vision produced directly can recover the three-dimensional coordinate of targeted environment, and then the depth information of environment can be obtained, the detection for the barrier in unknown flight environment of vehicle and potential collision has important Practical significance.

Binocular vision is an important branch of computer vision, and binocular vision can the process of apish eyes and human stereoscopic vision's perception, is one of core subject of computer vision research.In recent years, binocular vision technology is widely used in fields such as detection of obstacles, industrial automation production, intelligent safety and defence systems.But the problems of the method for barrier perception existence of existing view-based access control model, the application of computer vision methods in aircraft mainly concentrates on independent landing, scene matching aided navigation and target identification and vision inertia combined navigation etc.Visible sensation method in independent landing is conceived to the aircraft lands stage, and needs known landing field information, cannot be applied to the tasks execution phases of aircraft; Scene matching aided navigation and target identification, need the on-board data base setting up scene matching aided navigation to search known target information and to adopt visible sensation method to obtain the relative position of target, but physical environment residing in aircraft flight process is helpless; And air navigation aid calculated amount computer vision technique and airborne ins data combined is comparatively large, the requirement of real-time navigation cannot be met when flight environment of vehicle is complicated.Therefore, though there is research visible sensation method being applied to aircraft navigation, but these methods need known target information or manually arrange reference information on the one hand, there is potential real-time defect on the other hand, the navigation application demand in residing physical environment when aircraft is executed the task cannot be met.

As everyone knows, visual pattern process needed to process mass data within the unit interval, need data operation ability fast, but data operation process was but relatively simple.The computing platform that can be used for visual pattern process at present mainly contains CPU, GPU, ASIC, DSP, FPGA etc.

The computing power of embedded type CPU is limited, and for the vision algorithm that some computation complexities are very high, its processing speed is usually difficult to the real-time needs meeting system.

GPU has the computation capability of height, the problem of computing velocity can be solved preferably, but the computer vision system based on GPU also exists the shortcoming that power consumption is higher, volume is larger, be difficult to meet self institute's charged pool of dependence and power and the needs of the UAS worked long hours.

Utilize application-specific integrated circuit ASIC to realize vision processing algorithm, can solve the contradiction between vision system performance and volume, power consumption, be the effective solution of one of high-performance embedded vision system.But the ASIC construction cycle is long, amendment property and versatility poor.

FPGA can revise the logic function of its inside easily by programming, thus the hardware calculating realized at a high speed and concurrent operation, be a kind of convenient solution of high-performance embedded vision system.Based on the power consumption of the embedded vision system of FPGA well below the vision system based on CPU and GPU, the power consumption of FPGA is usually less than 1W, and the power consumption of high-performance CPU and GPU is usually all at more than 100W.Along with the continuous progress of technology, the integrated level of FPGA is more and more higher, and the design scale that can realize is increasing, and power consumption is then more and more lower.Therefore, the important development direction of computer vision system is become based on the embedded vision system of FPGA.

The programmable system on chip technology on a single chip integrated processor core that comprises is located at interior logic function with mainly outer, these logic functions can reconfigure along with the change of application purpose, this makes system can subtract sanction at any time, expands or upgrading, make FPGA can configure the kernel of flush bonding processor in sheet, make FPGA also can to have in sheet storage unit at a high speed, logical resource on abundant IP kernel resource and enough sheets.

Compared with multi-purpose computer, embedded system has incomparable advantage on power consumption, volume and cost.Therefore embedded system based on ARM is widely used in the field such as industry, civilian even military affairs.(SuSE) Linux OS has very high performance, and when calculating identical data volume, minimum to the power amount of asking for of embedded system, this also makes Linux be able to there is very strong competitive power in built-in field.

In sum, although many scholars in unmanned plane field have carried out large quantity research for unmanned plane obstacle avoidance system both at home and abroad, for the complete airborne process of environmental information, and under the prerequisite ensureing measuring accuracy, current obstacle avoidance system also cannot reach the demand of small size, low-power and low weight, and then realizes the function of the complete local autonomous flight of unmanned plane.

Summary of the invention

The object of the invention is to overcome the deficiencies in the prior art, the autonomous obstacle detection system of a kind of unmanned plane based on binocular vision and method are provided, this detection system can meet unmanned plane under the demand prerequisite of power consumption and load-carrying, obtain the flight environment of vehicle three-dimensional information of real-time degree of precision, this detection method can realize the three-dimensional environment information reconstruction algorithm of the unmanned plane obstacle avoidance system based on embedded real-time binocular vision, makes algorithm have degree of precision and real-time performance.

The present invention solves its technical matters and takes following technical scheme to realize:

The autonomous obstacle detection system of unmanned plane based on binocular vision, comprises unmanned plane, and unmanned plane is mounted with binocular vision system, other sensor assemblies and flight control system;

Described binocular vision system is made up of two Airborne cameras, vision collecting processing unit, described two Airborne cameras are for obtaining the visual information of unmanned plane, described vision collecting processing unit processes vision data sets up three-dimensional flight environment of vehicle information, this vision collecting processing unit by complete image parallel algorithm on-site programmable gate array FPGA module and complete the embedded processors ARM module composition of the serial algorithm of image and the issue of result;

Other sensor assemblies described comprise Inertial Measurement Unit IMU, global position system GPS, magnetic compass and barometer;

Described flight control system receives the visually-perceptible information and the unmanned plane during flying status data that come from binocular vision system and other sensor units, generates flight steering order, for controlling unmanned plane during flying by fusion treatment;

Described unmanned plane flies according to the flight steering order of flight control system, realizes evading flight function to barrier.

Further, the concrete structure of described vision collecting processing unit is: described embedded processors ARM module is connected with exterior I O unit by AMBA, realizes the control to static memory cells by AMBA; On-site programmable gate array FPGA module comprises standard I O interface and is connected with external unit, carries out synchro control to Airborne camera, is realized the video acquisition of GigE video camera by GigE module, carries out internal data transfer by PCIe interface; Described embedded processors ARM module and on-site programmable gate array FPGA module carry out information interaction by AXI interface.

Further, described binocular vision system and adopt the mode of CAN to communicate between other sensor assembly with flight control system.

Further, described system also comprises telepilot and land station, and described telepilot, land station communicate with the mode with wireless link between flight control system and unmanned plane.

The autonomous obstacle detection method of unmanned plane based on binocular vision, comprises the following steps:

Step 1, binocular vision system obtain the visual information of unmanned plane during flying environment, and treatedly obtain obstacle information;

Step 2, other sensor units obtain the status information of unmanned plane;

Step 3, flight control system receive obstacle information and unmanned plane status information, set up flight path, generate flight steering order and send to unmanned plane;

Step 4, unmanned plane fly according to the flight steering order avoiding barrier of flight control system.

Further, the concrete methods of realizing of described step 1 is:

Step (1), two Airborne cameras obtain the visual information of unmanned planes;

Step (2), vision collecting processing unit obtain synchronous images;

Step (3), vision collecting processing unit corrects two-way image information;

Step (4), vision collecting processing unit processes obtain anaglyph;

Step is (5), vision collecting processing unit utilizes disparity map to carry out the three-dimensional reconstruction of environmental information, thus acquired disturbance thing information.

Further, described step (2) vision collecting processing unit obtains the concrete grammar of synchronous images and is: on-site programmable gate array FPGA module produces synchronizing signal by control synchronization trigger element, control Airborne camera synchronous acquisition image, and by image transmitting to embedded processors ARM module, utilize timestamp identification image, for subsequent treatment analysis; Described step (3) vision collecting processing unit to the concrete grammar that two-way image information corrects is: after embedded processors ARM module obtains the image with timestamp mark, calibration result according to video camera carries out distortion correction and Stereo matching rectification to image, and result passes to on-site programmable gate array FPGA module.

Further, described step (4) vision collecting processing unit processes obtains the concrete grammar of anaglyph and is: on-site programmable gate array FPGA module utilizes parallel processing technique, first utilize mean shift algorithm to Image Segmentation Using process, then the Matching power flow between pixel is calculated, combining image carve information constructs new global energy function again, carry out the polymerization of Matching power flow afterwards on multi-direction, finally choosing the disparity map making global energy function minimum is anaglyph;

Described step (5) vision collecting processing unit utilizes disparity map to carry out the three-dimensional reconstruction of environmental information, thus the concrete grammar of acquired disturbance thing information is: combining image feature is resolved by interest points matching and corresponding point of interest distance with anaglyph, 3D unique point is rebuild, image associates, wild point detects, relative pose resolves and position and attitude error is estimated and based on the motion decision-making of least error, thus acquired disturbance thing information.

Further, described step 3 flight control system set up flight path and flight steering order method be: according to obstacle information, flight control system judges whether the former flight path of unmanned plane exists barrier; And then generate new flight path according to obstacle information and unmanned plane status information; Flight steering order is generated according to other sensing datas and new flight path.

Further, the obstacle information that described binocular vision system obtains comprises: the relative pose between barrier and unmanned plane, the kind of barrier, size, shape and movement state information; The status information that other sensor units described obtain described unmanned plane comprises: flying height, flight course, flying speed, flight attitude information.

Advantage of the present invention and good effect are:

1, the present invention's utilize described binocular vision system to achieve unmanned plane effectively keeps away barrier function to the barrier existed in the flight environment of vehicle of place, and feasibility of the present invention is good, real-time is high and algorithm complex is low, there is good robustness simultaneously, effectively can prevent the impact of noise, and then the information such as the kind of Obtaining Accurate barrier, size, shape and motion state, realize the effective barrier avoiding function of unmanned plane to barrier.

2, visual information and other sensor informations merge by the present invention mutually, perception flight environment of vehicle information, carry out flight path control and path planning to evade barrier, efficiently solve the problem that unmanned plane vision keeps away barrier, make it possess to utilize Airborne camera to complete the ability that vision keeps away barrier.

3, vision collecting processing unit of the present invention adopts FPGA and ARM scheme designed in conjunction, have processing speed soon, low-power consumption, small size and low weight feature, therefore, it is possible to be applicable to unmanned plane field.

4, adopt the mode of wireless link to carry out radio communication between flight control system of the present invention, telepilot, land station and unmanned plane, achieve controlling functions more easily.

Accompanying drawing explanation

Fig. 1 is the structural representation of the autonomous obstacle detection system of unmanned plane of the present invention;

Fig. 2 is the embedded system structure figure based on FPGA and ARM of the present invention;

Fig. 3 is binocular vision system Organization Chart of the present invention;

Fig. 4 is the detection of obstacles process algorithm schematic flow sheet of real-time binocular vision system of the present invention;

Fig. 5 is half overall Stereo Matching Algorithm schematic flow sheet of the present invention;

Fig. 6 is that end point of the present invention forms schematic diagram.

Embodiment

Below in conjunction with accompanying drawing, the embodiment of the present invention is further described:

The autonomous obstacle detection system of unmanned plane based on binocular vision, as shown in Figure 1, comprising:

Binocular vision system (101): comprise two Airborne cameras (1011) and vision collecting processing unit (1012), described two Airborne cameras (1011) obtain the environmental visual information of unmanned plane (106), and these data are sent to vision collecting processing unit (1012), described vision collecting processing unit (1012) is carried out understanding to visual information and is analyzed, final acquisition environmental information, judge simultaneously and provide obstacle information, described obstacle information sends to flight control system (103) by CAN (107).

Other sensor units (102): comprise Inertial Measurement Unit IMU (1021), global position system GPS (1022), magnetic compass (1023) and barometer (1024), for obtaining the state of flight information such as flying height, flight course, flying speed, flight attitude of unmanned plane (106); The sensor information that other sensor units (102) obtain sends to flight control system (103) by CAN (107).

Flight control system (103): flight control system (103) receives the visually-perceptible information and the unmanned plane during flying status data that come from binocular vision system (101) and other sensor units (102) by CAN (107), after merging, result is sent to flight controller (1032), to obtain the obstacle information in the flight environment of vehicle at unmanned plane (106) place according to visually-perceptible information, and the flight path of unmanned plane (106) is planned according to obstacle information and unmanned plane status information, and generate flight steering order according to flight path, barrier for controlling unmanned plane (106) evades flight.The controlling functions that described flight control system (103) realizes moving to unmanned plane by controlling the airborne steering wheel group of unmanned plane (106).Described obstacle information includes but not limited to: the kind of the relative pose between barrier and described unmanned plane, described barrier, size, the information such as shape and motion state.

Flight control system (103) comprises data fusion module (1031) and flight controller (1032) and data transmission module (not marking in exemplary plot).There is in described flight control system (103) prolongable Modular Data fusion treatment function, namely unmanned aerial vehicle onboard binocular vision system (101) is utilized to carry out visually-perceptible, obtain inertial guidance data by IMU unit (1021) in other sensor assemblies of unmanned aerial vehicle onboard (102) simultaneously, course angle information is obtained by three axle magnetic compass unit (1022), flight position is obtained by GPS unit (1023), speed, course and altitude information, unmanned plane altitude information is obtained by barometer unit (1024), and in flight control system, utilize Fusion Module (1031) by each sensing data and visually-perceptible phase fusion treatment, so that to unmanned plane send flight path control and path planning instruction to evade barrier.

Whether flight control system (103) is also for being positioned at the former flight path of unmanned plane (106) according to visually-perceptible information disturbance in judgement thing, and set up new flight path according to obstacle information and unmanned plane status information time on the former flight path that disturbance in judgement thing is positioned at unmanned plane (106), and generate flight steering order according to new flight path.

Telepilot (104): directly control unmanned plane (106) by wireless link (108) and run, or through flight control system (103), unmanned plane (106) is controlled by wireless link (108).

Land station (105): through flight control system (103), unmanned plane (106) is controlled by wireless link (108), and the state of flight information of unmanned plane (106) can be received.

Unmanned plane (106): the visual information being obtained unmanned plane by binocular vision system (101), and fly according to the flight steering order of flight control system (103), realize evading barrier.

In the present embodiment, radio communication is carried out in the mode of wireless link (108) between flight control system (103), telepilot (104), land station (105) and unmanned plane (106).Particularly, the data of unmanned plane (106) are passed to land station by wireless link show, simultaneously when being set as manual or Long-distance Control, steering order is sent to flight controller (103) and unmanned plane (106) by wireless link (108) by telepilot (104) and land station (105).

It should be noted that, unmanned plane is different from other ground machine robot systems, it needs to consider the constraints such as load-carrying in flight course and power supply, therefore, visual information and other sensor informations merge by the present invention mutually, perception flight environment of vehicle information, carry out flight path control and path planning to evade barrier.Can solve the problem that unmanned plane vision keeps away barrier, make it possess to utilize Airborne camera to complete the ability that vision keeps away barrier, feasibility is good, load-carrying is little, and power consumption is lower simultaneously.

The inner vision collecting processing unit (1012) of binocular vision system (101) adopts FPGA (202) and ARM (201) designed in conjunction, is made up of the on-site programmable gate array FPGA module (202) of parallel algorithm and the embedded processors ARM module (201) that completes the serial algorithm of image and the issue of result completing image.Fig. 2 gives the embedded system structure figure of FPGA and ARM.Wherein, ARM embedded system is connected with exterior I O unit (205) by AMBA (2012), comprise SPI (2051), IIC (2052), CAN (2053), UART (2054), GPIO (2055), SDIO (2056), USB (2057) and GigE (2058) interface, realize the control to static memory cells (203) by AMBA (2012) simultaneously, support that the ability of multiple interfaces makes embedded system of the present invention expand and connects multiple Airborne camera equipment.Containing standard I O interface (2021) in FPGA (202) module, carry out with external unit, the synchro control of Airborne camera of the present invention (1011) can be realized, by GigE module (2022), also can realize the video acquisition of GigE video camera, and carry out internal data transfer by PCIe interface (2025).In the present embodiment, ARM (201) and FPGA (202) carry out information interaction by AXI interface (2011), by AXI interface (2011), video flowing can be transferred to FPGA (202) module and process by ARM (201) fast.

It should be noted that, to in the operation of view data, in unit interval, data volume is very large, require that data operation speed is fast, and in order to obtain processing speed faster, parallel mode need be adopted to carry out analysis and understanding to the image that binocular camera obtains, therefore, in the present embodiment ARM module (101) after having gathered video data by AXI interface (2011) at a high speed pass to FPGA module (202), and in FPGA module, parallel processing is carried out to the two-way view data acquired, and result is returned to by AXI interface (2025) the further process that ARM module carries out view data.

Fig. 3 gives the frame diagram of binocular vision system (101).ARM module (201) of the present invention gathers the view data of two-way Airborne camera (302) by USB interface, by AXI interface (2011) at a high speed by video streaming to FPGA module (202), carry out the parallel processing of two-way visual information, and result is fed back to ARM module (201), carry out subsequent treatment, result sends to flight control system (103) by transmission interfaces such as CAN.It should be noted that, the synchronous control signal of two-way Airborne camera (302), produced by FPGA control synchronization trigger element (301).

The autonomous obstacle detection method of unmanned plane based on binocular vision, comprises the following steps:

Step 1, binocular vision system obtain the visual information of unmanned plane during flying environment, and treatedly obtain obstacle information;

Step 2, other sensor units obtain the status information of unmanned plane;

Step 3, flight control system receive obstacle information and unmanned plane status information, set up flight path, generate flight steering order and send to unmanned plane;

Step 4, unmanned plane fly according to the flight steering order avoiding barrier of flight control system.

In above-mentioned steps, step 1 binocular vision system obtains the visual information of unmanned plane during flying environment, and the treated obstacle information that obtains is key of the present invention.As shown in Figure 4, ARM (201) module creation four threads carry out Image Acquisition (401), image flame detection (402), environmental information three-dimensional reconstruction (405) and evading decision (407) respectively; FPGA (202) module is carried out Stereo matching to the image after solid rectification and is generated difference view.Specifically comprise the following steps:

(1), vision collecting processing unit (1012) obtains synchronous images to step.Concrete grammar is: FPGA (202) module in vision collecting processing unit (1012) produces synchronizing signal by control synchronization trigger element (301), control Airborne camera (302) synchronous acquisition image, and by USB interface by image transmitting to the ARM of vision collecting processing unit (1012) (201) module, utilize timestamp identification image, for subsequent treatment analysis simultaneously.

(2), vision collecting processing unit (1012) corrects two-way image information step.Concrete grammar is: after ARM (201) module of vision collecting processing unit (1012) obtains the image with timestamp mark, calibration result according to video camera carries out distortion correction and Stereo matching rectification to image, and result passes to FPGA (202) module by AXI interface (2011).

(3), vision collecting processing unit (1012) process obtains anaglyph to step.Concrete grammar is: FPGA (202) module in vision collecting processing unit (1012), utilizes parallel processing technique, carries out half universe calculation respectively and analyzes, and generate disparity map to the image after rectification.

(4) step, utilizes disparity map to carry out the three-dimensional reconstruction of environmental information.Concrete grammar is: combining image feature (4051) and anaglyph carry out the reconstruction of 3D unique point, according to feature operator association two-way image, and utilize RANSAC method to remove wild point, utilize computer vision Models computed relative pose and carry out on-line optimization, obtain the relative pose result closer to true value, thus acquired disturbance thing information.Further, obstacle information includes but not limited to: the kind of the relative pose between described barrier and described unmanned plane, described barrier, size, the information such as shape and motion state.

Receive obstacle information and unmanned plane status information in above-mentioned steps 3, flight control system, set up flight path, generate flight steering order and send to unmanned plane.In this step, flight control system plans the flight path of unmanned plane according to obstacle information and unmanned plane positional information, and generates flight steering order according to other sensing datas and flight path.More specifically, whether the former flight path that first flight control system judges described unmanned plane according to described visual information exists described barrier; And then generate new flight path according to described obstacle information and described unmanned plane status information; Flight steering order is generated with according to other sensing datas described and new flight path.

In the present embodiment, described Image Acquisition (401) module, comprises the two-path video information obtained from Airborne camera (302), utilizes timestamp (402) to mark video information simultaneously.

In the present embodiment, described image flame detection (401) module, comprise and elimination distortion, binocular correction and image cropping are carried out to two-path video information, the corner areas correcting rear left right view is irregular often, and the parallax of correspondence is invalid, parallax can be asked for Stereo matching like this to have an impact, so need to carry out cutting to image after calibration.Once after the structure of binocular camera determines, this mapping table also just secures, and does not need real-time calculating.Only need call mapping function correcting image when therefore processing binocular image online, can follow-up Stereo matching be carried out, reduce the calculated amount of subsequent treatment.

In the present embodiment, described stereoscopic parallax figure generation module (404) module, comprises image after obtaining correction from image flame detection module (403), utilizes half overall stereo matching method to generate disparity map.By treating multiple one dimension path coequally, then the result in each one dimension path is merged (505), be similar to the situation of two dimension, in the global energy function (504) of half overall Stereo Matching Algorithm, by to the change of the difference of the degree of depth in addition different punishment ensure that smoothness constraint, in the aftertreatment part of algorithm, make use of left and right consistency desired result to detect and block a little and Mismatching point, ensure that unique constraints.The impact of algorithm on illumination variation is insensitive, has stronger robustness to noise.Detailed process is as follows: first utilize mean shift algorithm to Image Segmentation Using process (503), then the Matching power flow (502) between pixel is calculated, combining image carve information constructs new global energy function (504) again, carry out the polymerization (505) of Matching power flow afterwards on multi-direction, finally choosing the disparity map making global energy function minimum is coupling parallax result.

Fig. 5 gives half overall Stereo Matching Algorithm flow process, and in generation stereoscopic parallax figure process, the cost function of definition a direction is:

$L_r(p,d)=C(p,d)+\min \left(\begin{array}{c}{L}_r(p-r,d)\\ L_r(p-r,d-1)+P_1\\ L_r(p-r,d+1)+P_1\\ \underset{i}{\min }{L}_r(p-r,i)+P_2\end{array}\right)-\underset{k}{\min }{L}_r(p-r,k)$

What then half overall Stereo Matching Algorithm requirement will be tried to achieve is the minimum value that all directions cost is polymerized: namely minimum value, this wherein has a process of being polymerized.Half global registration algorithm can obtain the matching result compared favourably with other algorithms, and advantage is very high efficiency.

In the present embodiment, described environmental information three-dimensional reconstruction (405) module, comprise resolve (4051) by interest points matching and corresponding point of interest distance, 3D unique point rebuilds (4052), image association (4054), wild point detects (4055), relative pose resolves and position and attitude error estimates (4056) and the motion decision-making (4057) based on least error.

Described interest points matching (4051) process is: utilize the depth image obtained in thread 2 to calculate for keeping away the three-dimensional geometric information hindered in thread 3.First, the point of interest in left figure has Harris Corner Detection device to detect to provide, then before interesting measure setting up 19 × 19, utilize window size be 15 × 15 order filters filtering is carried out to angle point, to remove some high frequency noises and distortion.Do not adopt the reason of traditional SIFT or SURF feature operator in the present embodiment, above two kinds of feature operators require higher to the computing power of processor.

Described 3D unique point rebuilds (4052) process: according to the depth image obtained, can complete the three-dimensional reconstruction to angle point information under camera coordinate system.The information utilizing three reference points in consecutive image in theory just can the motion of computer memory six-freedom degree.

Described image association (4054) process is: the relevance principle of image is that all points of interest of present image can all match with point of interest in image before.On this basis by the consistency detection of detected image relevance in all directions as associated images.Can tackle the acute variation of feature in image space like this, and in actual use, video camera needs the situation moving fast or rotate.

Described open country point detects (4055) process: the testing process of wild point utilizes the fixing principle of the distance in static scene in two viewpoints under camera coordinate system between two three-dimensional coordinate points to detect.

Described relative pose resolves and position and attitude error estimates that the process of (4056) and optimization part is: after open country point is removed, relativeness between three-dimensional point cloud obtains by svd, is further optimized by minimizing oval error.

In the present embodiment, described evading decision (407) module, comprises and carries out the coordinate transform (4071) of relative pose between barrier and unmanned plane to binocular result and evade track (4074) by generating in conjunction with the current state (4072) of unmanned plane and flight model (4073).

Especially, it should be noted that, online self-calibration module (408) in Fig. 4, in the flight course of unmanned plane, due to focal length or the baseline of video camera environmentally can be adjusted in the process of operation, and need to re-start demarcation, and loaded down with trivial details calibration process, seriously limit the application of vision system in unmanned plane field, online self-calibration module (408) of the present invention can realize the Fast Calibration of binocular vision system.Scaling method utilizes end point geometrical property to demarcate binocular vision system, and Fig. 6 is the forming process of end point, and the detailed process of demarcation is as follows:

Definition video camera meets pin-hole model:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{k}_x& k_s& u_0\\ 0& k_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_c/z_c\\ y_c/z_c\\ 1\end{array}\right],$

Wherein, (u, v) is point (x under camera coordinate system C _c, y _c, z _c) coordinate under image coordinate system I, (u ₀, v ₀) be the coordinate of video camera principal point under image coordinate system, k _xand k _ybe respectively normalized focal length, k _sfor distortion parameter.

Two groups of parallel lines can meet at two end points in the picture, and the coordinate of end point in camera coordinate system C is mutually orthogonal, is shown below:

$x_{vh}^T{x}_{vv}=\left[\begin{array}{ccc}{x}_{vh}& y_{vh}& 1\end{array}\right]\left[\begin{array}{c}{x}_{vv}\\ y_{vv}\\ 1\end{array}\right]=0,$

Wherein, x _vh=[x _vhy _vh1] ^tand x _vv=[x _vvy _vv1] ^tbe respectively the expression of two end points under camera coordinate system C, x thus _vi, i=h, v also can write:

$\left[\begin{array}{c}{x}_{vi}\\ y_{vi}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{K}_{11}& K_{12}& K_{13}\\ 0& K_{22}& K_{23}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{u}_i\\ v_i\\ 1\end{array}\right],$

Wherein, (u _h, v _h) and (u _v, v _v) be respectively the coordinate of two end points under image coordinate system I, K ₁₁=1/k _x, K ₁₂=-k _s/ (k _xk _y), K ₁₃=k _sv ₀/ (k _xk _y)-u ₀/ k _x, K ₂₂=1/k _yand K ₂₃=-v ₀/ k _y.Can obtain further:

$\left[\begin{array}{ccc}{u}_h& v_h& 1\end{array}\right]H\left[\begin{array}{c}{u}_v\\ v_v\\ 1\end{array}\right]=0,$

Wherein,

$H={\left(H_{ij}\right)}_{i,j=1,2,3}=\left[\begin{array}{ccc}{K}_{11}^2& K_{11}{K}_{12}& K_{11}{K}_{13}\\ K_{11}{K}_{12}& K_{12}^2+K_{22}^2& K_{12}{K}_{13}+K_{22}{K}_{23}\\ K_{11}{K}_{13}& K_{12}{K}_{13}+K_{22}{K}_{23}& K_{13}^2+K_{23}^2+1\end{array}\right].$

Definition h ₁=H ₁₂/ H ₁₁, h ₁=H ₁₃/ H ₁₁, h ₁=H ₂₂/ H ₁₁, h ₁=H ₂₃/ H ₁₁and h ₁=H ₃₃/ H ₁₁, can obtain:

$h_1\underset{i=h,v}{\Σ}{u}_i{v}_i+h_2\underset{i=h,v}{\Σ}{u}_i+h_3{v}_h{v}_v+h_4\underset{i=h,v}{\Σ}{v}_i+h_5=h^{T}U=-u_h{v}_v,$

Wherein, h=[h ₁h ₂h ₃h ₄h ₅] ^tand $U=\left[\begin{array}{ccccc}\underset{i=h,v}{\Σ}{u}_i{v}_i& \underset{i=h,v}{\Σ}{u}_i& u_h{v}_h& \underset{i=h,v}{\Σ}{v}_i& 1\end{array}\right].$

Design Kalman filter is estimated h, and consider that h is normal vector, the kinetic model of system is:

h(k+1)＝h(k)+η(k)，

And output equation is:

y _c(k)＝C _c(k)h+υ(k)，

Wherein, η (k) and υ (k) is respectively and meets the state variable of Gaussian distribution and the noise of output.

Definition:

$C_c\left(k\right)={\left[\begin{array}{cccc}{U}_1^T& U_2^T& \mathrm{...}& U_n^T\end{array}\right]}^T.$

After can obtaining n >=2 end point.The measured value that system kth is clapped is:

z(k)＝[-u _h1h _v1-u _h2h _v2…-u _hnh _vn] ^T

Utilize Kalman filtering can obtain the optimum results of camera parameters thus, the result that can obtain concrete intrinsic parameter is further:

$\left\{\begin{array}{c}{k}_x=1/K_{11},k_y=1/K_{22},k_s=-K_{12}/\left(K_{11}{K}_{22}\right),\\ u_0=K_{12}{K}_{23}/\left(K_{11}{K}_{22}\right)-K_{13}/K_{11},v_0=-K_{23}/K_{22}.\end{array}\right.$

Wherein,

$K_{11}=\sqrt{(h_3-h_1^2)/(h_3{h}_5-h_2^2{h}_3-h_1^2{h}_5-h_4^2+2h_1{h}_2{h}_4)},$

$K_{12}=h_1{K}_{11},K_{13}=h_2{K}_{11},K_{12}=\sqrt{h_3+h_1^2}{K}_{11},$

$K_{23}=(h_4-h_1{h}_2)K_{11}\sqrt{h_3-h_1^2}.$

After realizing the online self-calibration result of single camera, then determine that the relative position of two video cameras can complete the integral calibrating of binocular camera according to the result of demarcating.

It is emphasized that; embodiment of the present invention is illustrative; instead of it is determinate; therefore the present invention includes the embodiment be not limited to described in embodiment; every other embodiments drawn by those skilled in the art's technical scheme according to the present invention, belong to the scope of protection of the invention equally.

Claims (10)

1. based on the autonomous obstacle detection system of unmanned plane of binocular vision, comprise unmanned plane, it is characterized in that: on unmanned plane, be mounted with binocular vision system, other sensor assemblies and flight control system;

Described binocular vision system is made up of two Airborne cameras, vision collecting processing unit, described two Airborne cameras are for obtaining the visual information of unmanned plane, described vision collecting processing unit processes vision data sets up three-dimensional flight environment of vehicle information, the embedded processors ARM module composition that this vision collecting processing unit is issued with the serial algorithm and result that complete image by the on-site programmable gate array FPGA module of the parallel algorithm completing image;

Other sensor assemblies described comprise Inertial Measurement Unit IMU, global position system GPS, magnetic compass and barometer;

Described flight control system receives the visual information and the unmanned plane during flying status data that come from binocular vision system and other sensor units, generates flight steering order, for controlling unmanned plane during flying by fusion treatment;

Described unmanned plane flies according to the flight steering order of flight control system, realizes evading flight function to barrier.

2. the autonomous obstacle detection system of a kind of unmanned plane based on binocular vision according to claim 1, it is characterized in that: the concrete structure of described vision collecting processing unit is: described embedded processors ARM module is connected with exterior I O unit by AMBA, realize the control to static memory cells by AMBA; On-site programmable gate array FPGA module comprises standard I O interface and is connected with external unit, carries out synchro control to Airborne camera, is realized the video acquisition of GigE video camera by GigE module, carries out internal data transfer by PCIe interface; Described embedded processors ARM module and on-site programmable gate array FPGA module carry out information interaction by AXI interface.

3. the autonomous obstacle detection system of a kind of unmanned plane based on binocular vision according to claim 1 and 2, is characterized in that: described binocular vision system and adopt the mode of CAN to communicate between other sensor assembly with flight control system.

4. the autonomous obstacle detection system of a kind of unmanned plane based on binocular vision according to claim 1 and 2, it is characterized in that: described system also comprises telepilot and land station, described telepilot, land station communicate with the mode with wireless link between flight control system and unmanned plane.

5. the detection method of the autonomous obstacle detection system of unmanned plane as described in any one of Claims 1-4, is characterized in that comprising the following steps:

Step 1, binocular vision system obtain the visual information of unmanned plane during flying environment, and treatedly obtain obstacle information;

Step 2, other sensor units obtain the status information of unmanned plane;

Step 3, flight control system receive obstacle information and unmanned plane status information, set up flight path, generate flight steering order and send to unmanned plane;

Step 4, unmanned plane fly according to the flight steering order avoiding barrier of flight control system.

6. the autonomous obstacle detection method of a kind of unmanned plane based on binocular vision according to claim 5, is characterized in that: the concrete methods of realizing of described step 1 is:

Step (1), two Airborne cameras obtain the visual information of unmanned planes;

Step (2), vision collecting processing unit obtain synchronous images;

Step (3), vision collecting processing unit corrects two-way image information;

Step (4), vision collecting processing unit processes obtain anaglyph;

Step is (5), vision collecting processing unit utilizes disparity map to carry out the three-dimensional reconstruction of environmental information, thus acquired disturbance thing information.

7. the autonomous obstacle detection method of a kind of unmanned plane based on binocular vision according to claim 6, it is characterized in that: described step (2) vision collecting processing unit obtains the concrete grammar of synchronous images and is: on-site programmable gate array FPGA module produces synchronizing signal by control synchronization trigger element, control Airborne camera synchronous acquisition image, and by image transmitting to embedded processors ARM module, utilize timestamp identification image, for subsequent treatment analysis; Described step (3) vision collecting processing unit to the concrete grammar that two-way image information corrects is: after embedded processors ARM module obtains the image with timestamp mark, calibration result according to video camera carries out distortion correction and Stereo matching rectification to image, and result passes to on-site programmable gate array FPGA module.

8. the autonomous obstacle detection method of a kind of unmanned plane based on binocular vision according to claim 6, it is characterized in that: described step (4) vision collecting processing unit processes obtains the concrete grammar of anaglyph and is: on-site programmable gate array FPGA module utilizes parallel processing technique, first utilize mean shift algorithm to Image Segmentation Using process, then the Matching power flow between pixel is calculated, combining image carve information constructs new global energy function again, carry out the polymerization of Matching power flow afterwards on multi-direction, finally choosing the disparity map making global energy function minimum is anaglyph,

Described step (5) vision collecting processing unit utilizes disparity map to carry out the three-dimensional reconstruction of environmental information, thus the concrete grammar of acquired disturbance thing information is: combining image feature is resolved by interest points matching and corresponding point of interest distance with anaglyph, 3D unique point is rebuild, image associates, wild point detects, relative pose resolves and position and attitude error is estimated and based on the motion decision-making of least error, thus acquired disturbance thing information.

9. the autonomous obstacle detection method of a kind of unmanned plane based on binocular vision according to claim 5, is characterized in that: the method that described step 3 flight control system sets up flight path and flight steering order is: according to obstacle information, flight control system judges whether the former flight path of unmanned plane exists barrier; And then generate new flight path according to obstacle information and unmanned plane status information; Flight steering order is generated according to other sensing datas and new flight path.

10. the autonomous obstacle detection method of a kind of unmanned plane based on binocular vision according to any one of claim 5 to 9, is characterized in that: the obstacle information that described binocular vision system obtains comprises: the relative pose between barrier and unmanned plane, the kind of barrier, size, shape and movement state information; The status information that other sensor units described obtain described unmanned plane comprises: flying height, flight course, flying speed, flight attitude information.