JP2010239190A - Obstacle detection device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compose an obstacle detection device capable of accurately extracting a three-dimensional object. <P>SOLUTION: Conversion images are generated by individually subjecting a first shot image and a second shot image shot by a camera Cam to projection conversion to a first virtual plane at a road surface level and a second virtual plane at a high level relative to it; a logical product of the conversion images is obtained to extract three-dimensional candidate regions, and a three-dimensional candidate region having an attitude facing an optical center is extracted as a stereoscopic image from among the extracted three-dimensional candidate regions. The lower end of the stereoscopic image is identified from a plurality of gradient lines passing through the lower ends of individually-extracted three-dimensional object images and the optical center. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an obstacle detection device, and particularly relates to a technique for projectively converting an image around a vehicle photographed by an in-vehicle camera into an image looking down from an upper viewpoint.

  As a technique related to the obstacle detection apparatus configured as described above, Patent Document 1 discloses a road surface projection image by projecting and converting an image around a vehicle photographed at different positions on a road surface. A processing form is described in which a region of an object outside the road surface is extracted by extracting a region of an object other than the above and superimposing it on a road surface projection image to obtain a difference. Furthermore, in this patent document 1, the outline of the area of the object outside the road surface is classified into one derived from the object and one due to projection distortion, and the outline of the area is added to the image converted from the projection image. This makes it easier to understand the situation around the vehicle.

  In Patent Document 1, a solid object having a height component such as a vehicle or a person has a disadvantage that the higher the height, the greater the distortion of an image projected from the object. Since it is difficult to deal with a real object, the three-dimensional object is displayed in an easy-to-see manner.

  In addition, as a related technique, Non-Patent Document 1 uses images before and after movement and predicts and compares the image after movement from the image before movement. Matches the image. It is disclosed that since a point on a three-dimensional object does not match and a parallax occurs, the distance and height related to the point can be calculated.

JP-A-2001-114047 (paragraph numbers [0015] to [0018], [0032] to [0050], FIGS. 1 to 9)

Co-authored by Eisaku Tonami, Shigeki Ishikawa, Minami Miyauchi, Shinji Ozawa “A Method for Obstacle Detection for Visual Guidance of Automated Carriers” IEICE Transactions D Vol.J68-D No.10 pp.1789-1791 Date: 1985/10/20

  As a display form when photographing the periphery of the vehicle with the in-vehicle camera and displaying it on the monitor, a projection image is generated by projecting the image photographed with the in-vehicle camera onto the road surface as described in Patent Document 1, and this projection image May be displayed on a monitor. The image displayed in this way is often described as a bird's-eye view image or a top view, and since the image is taken from a viewpoint above the vehicle, it is easy to grasp the situation around the vehicle. There is also a good aspect.

  In this way, a bird's-eye view image is generated by converting the image of the in-vehicle camera, and the three-dimensional object is also displayed on the road surface. Therefore, as described in Patent Document 1, the three-dimensional object is extracted. By displaying, the position of the three-dimensional object can be grasped.

  Measuring the position of a three-dimensional object based on a captured image captured by an in-vehicle camera can be realized from the difference between two captured images that generate parallax by using a plurality of in-vehicle cameras. In order to improve the accuracy of detecting the position of a three-dimensional object, it is realized by providing three or more in-vehicle cameras. For example, when three or more in-vehicle cameras are used, the cameras are provided. Not only is the cost increased, but it is not preferable in that the processing takes time because three captured images are acquired and a difference between the captured images is obtained.

  Especially when there are obstacles floating on the road surface (there is a part that is not in contact with the road surface), there is a desire to detect the lifted state with high accuracy, but the cost of hardware is increased and the processing time is long. There is also an aspect that wants to avoid time.

  Also, as in Non-Patent Document 1, there may be cases where edge association is not necessary in a scene that assumes a simple building world, but in many cases it cannot be applied in an actual real world. In the real world, there are various patterns on the road surface, and there are also various patterns on the three-dimensional object. In such a scene, it cannot be determined that the point is on the object by the disclosed method. For all the points appearing in (1), it is necessary to calculate the distance and height by associating the predicted image with the post-movement image, and there is a problem that the amount of calculation increases.

  An object of the present invention is to rationally configure an obstacle detection apparatus that can accurately extract a three-dimensional object.

A feature of the present invention is a camera for photographing a vehicle periphery,
Vehicle position detection means for detecting vehicle position coordinates and vehicle posture;
Shooting control means for shooting the first shot image and the second shot image in which a part of the shooting area is shared by the camera at a timing at which the vehicle is at a predetermined position and a timing at which the vehicle is moved from the predetermined position. ,
A converted image in which the first photographed image is looked down from an upper viewpoint with respect to at least two virtual planes among a virtual plane set on the road surface and a plurality of virtual projection planes different in height from the road surface in a posture parallel to the road surface A first projective transformation unit for projective transformation into,
A transformed image obtained by looking down at least two virtual planes of the second photographed image from an upper viewpoint among a virtual plane set on the road surface and a plurality of virtual projection planes different in height from the road surface in a posture parallel to the road surface. A second projective transformation unit for projective transformation into,
A first three-dimensional object candidate region extracting means for extracting an image region existing in an overlapping region in a state where a plurality of converted images generated from the first photographed image are overlapped by the first projective conversion unit;
Second solid object candidate area extracting means for extracting image areas existing in overlapping areas in a state where a plurality of converted images generated from the second photographed image are overlapped by the second projective conversion unit;
A plurality of first azimuth straight lines passing through a plurality of feature points of the image region extracted by the first three-dimensional object candidate region extraction means and the optical center of the first photographed image are set, and the plurality of first azimuth straight lines First gradient straight line generating means for generating a first gradient straight line that passes from the optical center of the first captured image to the lower end of the image area in a virtual vertical plane including
A plurality of second azimuth lines passing through a plurality of feature points of the image region extracted by the second three-dimensional object candidate region extraction means and the optical center of the second photographed image are set, and the plurality of second azimuth lines A second gradient straight line generating means for generating a second gradient straight line passing through the lower end of the image area from the optical center of the second photographed image in a virtual vertical plane including
A plurality of intersections where the plurality of first gradient lines and the plurality of second gradient lines intersect, in a state in which the first gradient line and the second gradient line are coordinate-converted to reference coordinates set on the road surface, or A straight line passing through the lower end of the image area among the intersections of the first virtual straight line having a gradient proportional to the plurality of first gradient straight lines and the second virtual straight line having a gradient proportional to the plurality of second gradient straight lines Front end detection means for detecting the position of the intersection of each other;
And a three-dimensional object region specifying unit that specifies a region where a three-dimensional object is present from the front end of the image region detected by the front end detecting unit.

According to this configuration, the first projective conversion unit generates two converted images from the first captured image. Similarly, the second projective transformation unit generates at least two converted images. The first three-dimensional object candidate area extracting unit extracts the first three-dimensional object candidate area from the corresponding plurality of conversion screens, and the second three-dimensional object region extracting unit extracts the second three-dimensional object candidate area from the corresponding plurality of conversion screens. . With these processes, a three-dimensional object candidate region can be set as a region where a three-dimensional object is expected to exist in the first captured image and the second captured image. Next, the first gradient straight line generation unit includes a plurality of first azimuth straight lines that pass through the plurality of feature points of the image region extracted by the first three-dimensional object candidate region extraction unit and the optical center of the first photographed image. A first gradient straight line passing through the lower end of the stereoscopic image from the optical center is generated on the virtual vertical plane in the vertical posture. A second gradient straight line is generated for the image region extracted by the second three-dimensional object candidate region extracting unit by performing the same processing as the second gradient straight line generating unit. Then, the front end detection means converts any one of the first gradient line and the second gradient line into a plurality of coordinates after the coordinate conversion based on either the first captured image or the captured position of the second captured image. Among a plurality of intersections where the first gradient straight line and the plurality of second gradient straight lines intersect, or among the intersections where the first virtual straight line and the second virtual straight line intersect, intersections between the straight lines passing through the lower end of the image area Is detected as the front end of the image area. Then, the three-dimensional object region specifying means specifies the three-dimensional object region based on the front end information.
As a result, an obstacle detection apparatus that can extract a three-dimensional object with high accuracy is configured.

The feature of the present invention is that two cameras that shoot in the state where the shooting area is shared around the vehicle,
Photographing position information storage means for storing a photographing position relationship between a first photographed image photographed by one of the two cameras and a second photographed image photographed by the other;
The first photographed image is a converted image obtained by looking down at least two virtual planes from an upper viewpoint among a virtual plane set on the road surface and a plurality of virtual projection planes different in height from the road surface in a posture parallel to the road surface. A first projective transformation unit for projective transformation;
The second photographed image is a converted image obtained by looking down at least two virtual planes from an upper viewpoint among a virtual plane set on the road surface and a plurality of virtual projection surfaces different in height from the road surface in a posture parallel to the road surface. A second projective transformation unit for projective transformation;
A first three-dimensional object candidate region extracting means for extracting an image region existing in an overlapping region in a state where a plurality of converted images generated from the first photographed image are overlapped by the first projective conversion unit;
Second solid object candidate area extracting means for extracting image areas existing in overlapping areas in a state where a plurality of converted images generated from the second photographed image are overlapped by the second projective conversion unit;
A plurality of first azimuth straight lines passing through a plurality of feature points of the image region extracted by the first three-dimensional object candidate region extraction means and the optical center of the first photographed image are set, and the plurality of first azimuth straight lines First gradient straight line generating means for generating a first gradient straight line that passes from the optical center of the first captured image to the lower end of the image area in a virtual vertical plane including
A plurality of second azimuth lines passing through a plurality of feature points of the image region extracted by the second three-dimensional object candidate region extraction means and the optical center of the second photographed image are set, and the plurality of second azimuth lines A second gradient straight line generating means for generating a second gradient straight line passing through the lower end of the image area from the optical center of the second photographed image in a virtual vertical plane including
A plurality of intersections where the plurality of first gradient lines and the plurality of second gradient lines intersect, in a state in which the first gradient line and the second gradient line are coordinate-converted to reference coordinates set on the road surface, or A straight line passing through the lower end of the image area among the intersections of the first virtual straight line having a gradient proportional to the plurality of first gradient straight lines and the second virtual straight line having a gradient proportional to the plurality of second gradient straight lines Front end detection means for detecting the position of the intersection of each other;
And a three-dimensional object region specifying unit that specifies a region where a three-dimensional object is present from the front end of the image region detected by the front end detecting unit.

According to this configuration, the first projective conversion unit generates two converted images from the first captured image. Similarly to this, the second projective transformation unit and at least two converted images are generated. The first three-dimensional object candidate area extracting unit extracts the first three-dimensional object candidate area from the corresponding plurality of conversion screens, and the second three-dimensional object region extracting unit extracts the second three-dimensional object candidate area from the corresponding plurality of conversion screens. . With these processes, a three-dimensional object candidate region can be set as a region where a three-dimensional object is expected to exist in the first captured image and the second captured image. Next, the first gradient straight line generation unit includes a plurality of first azimuth straight lines that pass through the plurality of feature points of the image region extracted by the first three-dimensional object candidate region extraction unit and the optical center of the first photographed image. A first gradient straight line passing through the lower end of the stereoscopic image from the optical center is generated on the virtual vertical plane in the vertical posture. A second gradient straight line is generated for the image region extracted by the second three-dimensional object candidate region extracting unit by performing the same processing as the second gradient straight line generating unit. Then, the front end detection means converts any one of the first gradient line and the second gradient line into a plurality of coordinates after the coordinate conversion based on either the first captured image or the captured position of the second captured image. Among a plurality of intersections where the first gradient straight line and the plurality of second gradient straight lines intersect, or among the intersections where the first virtual straight line and the second virtual straight line intersect, intersections between the straight lines passing through the lower end of the image area Is detected as the front end of the image area. Then, the three-dimensional object region specifying means specifies the three-dimensional object region based on the front end information.
As a result, an obstacle detection apparatus that can extract a three-dimensional object with high accuracy is configured.

  According to the present invention, the first three-dimensional object candidate region extracting unit and the second three-dimensional object candidate region extracting unit extract an image region from the two converted images to be processed by contour extraction using a contour extraction filter. After the area extraction process, the image area may be extracted by a process of specifying an image area existing in an overlapping area by superimposing two projected images and accumulating the image area.

  According to this, the image area extraction process extracts the image area surrounded by the contours extracted in the plurality of projection images, and the three-dimensional object candidate is defined as the three-dimensional object candidate area by overlapping the plurality of converted images in a superimposed state. The area extracting means extracts.

It is a figure which shows the driver's seat vicinity of the vehicle of 1st Embodiment. It is a top view which shows typically the vehicle structure of 1st Embodiment. It is explanatory drawing which shows an example of viewpoint conversion. It is a figure which shows the image image | photographed with the camera. It is a figure which shows the image converted into the image of the viewpoint from a substantially perpendicular direction. It is a block circuit diagram which shows the control structure of 1st Embodiment. It is a flowchart which shows the outline | summary of control of 1st Embodiment. It is a perspective view which shows the relationship between a solid object, an optical center, and two virtual planes. It is a side view which shows the relationship between a solid object, an optical center, and two virtual planes. It is a side view which shows the relationship between a solid object, an optical center, and several virtual planes. It is a figure which shows the conversion image produced | generated by the 1st virtual plane and the 2nd virtual plane. It is a figure which shows two conversion images after binarization. It is a figure which shows the image by which the solid object area | region was specified. It is explanatory drawing of the principle which sets an azimuth | straight line. It is explanatory drawing of an azimuth | direction straight line, a virtual vertical surface, and a gradient straight line. It is explanatory drawing of the gradient straight line which passes along two optical centers. It is explanatory drawing of the gradient straight line and cumulative line which are formed in the solid object which a lower end touches a road surface. It is explanatory drawing of the gradient straight line and cumulative line which are formed in the solid object which the lower end separated from the road surface. It is explanatory drawing which specifies the lower end of a solid object from the intersection of a 1st cumulative straight line and a 2nd cumulative straight line. It is a top view which shows typically the vehicle structure of 2nd Embodiment. It is a block circuit diagram which shows the control structure of 2nd Embodiment. It is a flowchart which shows the outline | summary of control of 2nd Embodiment.

A first embodiment of the present invention will be described below with reference to the drawings.
[First Embodiment / Overall Configuration]
The obstacle detection device of the present invention is used for, for example, a parking assistance device or a driving assistance device. 1 and 2 show a basic configuration of a vehicle 30 on which an obstacle detection device is mounted. The steering wheel 24 provided in the driver's seat is interlocked with the power steering unit 33 to transmit the rotational operation force to the front wheels 28f to steer the vehicle 30. An engine 32 and a speed change mechanism 34 having a torque converter, CVT, and the like that shift the power from the engine 32 and transmit the power to the front wheels 28f and the rear wheels 28r are disposed at the front of the vehicle. Power is transmitted to the front wheel 28f and / or the rear wheel 28r according to the driving method of the vehicle 30 (front wheel drive, rear wheel drive, four wheel drive). In the vicinity of the driver's seat, an accelerator pedal 26 as an accelerator operating means for controlling the traveling speed, and a brake pedal 27 for applying a braking force to the front wheels 28f and the rear wheels 28r via the brake devices 31 for the front wheels 28f and the rear wheels 28r, Are arranged in parallel.

  A monitor 20 (display device) is provided at the upper position of the console near the driver's seat. The monitor 20 is a liquid crystal type equipped with a backlight. A pressure-sensitive or electrostatic touch panel is formed on the display surface of the monitor 20, and an instruction input by a user is received by outputting a contact position of a finger or the like as location data. The touch panel is used, for example, as an instruction input unit for starting parking assistance. The monitor 20 is also provided with a speaker, which can emit various guidance messages and sound effects. When the navigation system is mounted on the vehicle 30, it is preferable that the monitor 20 is also used as a display device for the navigation system. The monitor 20 may be of a plasma display type or a CRT type, and the speaker may be provided in another place such as the inside of the door.

  A steering sensor 14 is provided in the operation system of the steering wheel 24, and the steering operation direction and the operation amount are measured. The operation system of the shift lever 25 is provided with a shift position sensor 15 to determine the shift position. The operation system of the accelerator pedal 26 is provided with an accelerator sensor 16 to measure an operation amount. The operation system of the brake pedal 27 is provided with a brake sensor 17, which detects the presence or absence of operation.

  Further, a wheel rotation sensor 18 that measures the amount of rotation of at least one of the front wheel 28f and the rear wheel 28r is provided as a movement distance sensor. In this embodiment, the case where the wheel rotation sensor 18 is provided in the rear wheel 28r is illustrated. As for the movement distance, the transmission mechanism 34 may measure the movement amount of the vehicle 30 from the rotation amount of the drive system. The vehicle 30 is provided with an electronic control unit (ECU) 10 that functions as an obstacle detection device of the present invention.

  A camera Cam that captures the rear of the vehicle 30 is provided at the rear of the vehicle 30. The camera Cam is a digital camera that incorporates an image sensor such as a CCD (charge coupled device) or a CIS (CMOS image sensor) and outputs information captured by the image sensor in real time as moving image information. The camera Cam includes a wide-angle lens and has a field angle of about 140 degrees on the left and right, for example. The camera Cam has a substantially horizontal viewpoint and is installed so that a scene behind the vehicle 30 can be photographed. The camera Cam is installed with a depression angle of, for example, about 30 degrees toward the rear of the vehicle 30, and generally captures an area up to about 8m behind. The captured image is input to the ECU 10 that functions as an obstacle detection device.

[Control structure of ECU]
ECU10 has a microprocessor and implement | achieves the process required by running a program. In this ECU 10, as shown in FIG. 6, the vehicle position calculation means 1 for obtaining signals from the steering sensor 14, the shift position sensor 15, the accelerator sensor 16, the brake sensor 17, and the wheel rotation sensor 18, and the vehicle A shooting control unit 2 for controlling the camera Cam based on information from the position calculation unit 1; a first frame memory M1 for storing a first shot image previously shot by the camera Cam via the shooting control unit 2; Thereafter, a second frame memory M2 for storing a second photographed image photographed by the camera Cam is provided.

  The shooting control means 2 is configured to take a shooting timing so that a part of the shooting area of the first shot image shot first based on the information from the vehicle position calculation means 1 and the second shot image shot after that is shared. And the relative position of the vehicle 30 and the relative attitude of the vehicle 30 at each photographing timing are acquired based on the information from the vehicle position calculation means 1.

  The ECU 10 includes a first projective conversion unit N1 that generates two types of converted images looking down from an upper viewpoint from a captured image of one frame (one frame) from the first frame memory M1. First solid object region extraction means U1 is provided for extracting a three-dimensional object region based on the converted image from the first projective transformation unit N1. In addition, the ECU 10 includes first gradient straight line generating means V1 that generates the first gradient straight line 1 based on the solid object area generated by the first three-dimensional object area extracting means U1. For example, as shown in FIG. 14A, the first gradient straight line generating unit V1 corresponds to the four feature points X, Y, Z, and T at the lower end of the three-dimensional object region. , DZ1 and DT1 are set. These first azimuth lines DX1, DY1, DZ1, and DT1 are examples of the first azimuth line D1.

  As a specific example of the process of generating the first gradient straight line E1 corresponding to the first azimuth straight line DX1, the virtual vertical plane W in the vertical posture including the first directional straight line DX1 corresponding to the feature point X as shown in FIG. A first gradient line EX1 (an example of the first gradient line E1) that passes through the feature point X at the lower end of the image region from the optical center C1 of one captured image is generated. This feature point X is located on the near side (the side close to the optical center C1) of the lower end of the three-dimensional object, and is referred to as the near end. Although not shown in the drawing, the first gradient straight line generating means V1 generates a first gradient straight line for the first azimuth straight lines DY1, DZ1, and DT1. The second projection having the first projection conversion means T1 and the second projection conversion means T2 so that the same processing is performed in parallel on the captured image of one frame (one frame) from the second frame memory M2. The conversion part N2 and the 2nd gradient straight line production | generation means V2 are provided.

  The first three-dimensional object region extraction unit U1 and the second three-dimensional object region extraction unit U2 perform an image extraction process, a three-dimensional object candidate region extraction process, and a three-dimensional object region extraction process, which will be described later. In particular, the three-dimensional object candidate area extracted by the three-dimensional object candidate area extraction process and the three-dimensional object area extracted by the three-dimensional object area extraction process are included in the concept of the image area of the present invention. Both the region and the three-dimensional object region can be processed by the first and second gradient straight line generating means V1 and V2. Similarly, both the three-dimensional object candidate area and the three-dimensional object area can be processed by the three-dimensional object area specifying unit 5.

  The first projective conversion unit N1 and the second projective conversion unit N2 include first projective conversion means T1 and second projective conversion means T2. The first projective transformation means T1 forms a transformed image by projective transformation on the road surface level first virtual plane S1 (see FIGS. 8 and 9), and the second projective transformation means T2 has a level higher than the road surface level. A converted image is generated for the second virtual plane S2. As described above, the first gradient straight line generating unit V1 sets the first azimuth straight line D1 and generates the first gradient straight line E1. The second gradient straight line generating means V2 sets a second azimuth straight line D2 and generates a second gradient straight line E2. Details of these control modes will be described later.

Further, the ECU 10 has a plurality of intersections at which the plurality of first gradient straight lines E1 and the plurality of second gradient straight lines E2 intersect, or a first cumulative straight line F1 having a gradient proportional to the plurality of first gradient straight lines E1 (first The lower end of the image area is defined as the lower end of the image area at the intersection of one example of one imaginary straight line and the second cumulative line F2 (an example of the second imaginary straight line) having a gradient proportional to the plurality of second gradient lines E2. Front end detection means 3 is provided for detecting the position of the intersection point d between the straight lines passing through the point as the front end position. In the case of a three-dimensional object extending upward from the road surface, the front end detected by the front end detection means 3 indicates the side close to the optical centers C1 and C2 among the lower ends in contact with the road surface, and upward from the road surface. In the case of a three-dimensional object that is separated and overhangs, it indicates the side close to the optical centers C1 and C2 among the lower ends of the three-dimensional object that is separated from the road surface. This front end will be described in detail. In the case of a three-dimensional object that extends upward from the position touched on the road surface when observed in the top view images shown in FIGS. 12 to 14, the lower end position in contact with the road surface is the position of the camera. The front end detection means 3 detects this position as the front end when viewed from the position. In the case of an overhanging three-dimensional object that is spaced upward from the road surface, the lower end position of the three-dimensional object that is separated from the road surface appears shifted to the far side from the actual position, but the intersection of the straight lines that pass through the lower end of the image area. Since the front end detecting means 3 is configured to detect the position of the overhanging three-dimensional object spaced upward from the road surface, the actual position, that is, the road surface position immediately below the feature point on the near side of the overhanging three-dimensional object Can be detected as the front end.
Image position correction information generation means 4 is provided for generating image position correction information based on the relative position information and the relative posture information acquired from the imaging control means 2. Further, the information on the front end detected by the front end detection means 3, the information on the three-dimensional object area extracted by the first and second three-dimensional object area extraction means U1 and U2, and the image position correction information generation means 4 A solid object region specifying unit 5 for specifying a three-dimensional object region in the converted image based on the information and the converted image generated by the first projective conversion unit T1 of the first projective conversion unit N1 is provided. Then, the projection distortion correcting unit 6 that corrects the distortion of the image of the three-dimensional object specified by the three-dimensional object region specifying unit 5 and outputs it to the monitor 20, and the image of the three-dimensional object specified by the three-dimensional object region specifying unit 5 And a superimposing unit 7 that superimposes the specific information and outputs it to the monitor 20.

  Own vehicle position calculation means 1, photographing control means 2, first and second projective transformation units N1 and N2, first and second three-dimensional object region extraction means U1 and U2, and first and second gradient straight line generations The means V1, V2, the front end detecting means 3, the image position correction information generating means 4, the three-dimensional object region specifying means 5, the projective distortion correcting section 6, and the superimposing section 7 are all programs (software). It is composed. However, these may be configured individually by hardware, and each part may be configured by hardware and processing may be realized by combination with a program, or each may be configured by hardware. .

  Further, the first and second projective transformation units N1 and N2 are configured by one module or the like, and the processing of the first captured image acquired first and the second captured image acquired later is performed by one module. It may be combined. In the same manner, the first and second three-dimensional object region extracting means U1 and U2 are configured as one module so as to be combined, and the first and second gradient straight line generating means V1 and V2 are also combined. You may comprise by one module etc.

  The ECU 10 captures two frames (two frames) at a timing at which a part of the photographing area is shared by the camera Cam when the vehicle 30 moves, extracts a three-dimensional image from the acquired two-frame captured image, and extracts the three-dimensional image. Allows location.

[Outline of processing by projective transformation unit]
The first projection conversion means T1 and the second projection conversion means T2 constituting the first and second projection conversion units N1 and N2 are a road surface in a posture parallel to the first virtual plane S1 set on the road surface and the road surface. For the second virtual plane S2 at a higher level, a converted image obtained by looking down the captured image of one frame from the upper viewpoint is generated by projective transformation. Each of the first and second projective transformation units N1 and N2 may generate a converted image for three or more virtual planes.

The processing by the first and second projective transformation units N1 and N2 may be referred to as homography, and the first and second projective transformation units N1 and N2 are virtual planes S (generic name for a plurality of virtual planes). In contrast, a GPT (ground plane transformation) image as a kind of bird's-eye view image is generated. As an outline of the processing, one frame of a captured image obtained by capturing a road surface at a predetermined angle is acquired by a camera Cam, and the image is generated based on a unique projective transformation relationship between an image plane on which the captured image exists and a virtual plane S. The photographed image is projectively converted to a virtual plane according to the conversion formula, and necessary calibration processing is performed. As a specific example of processing, as shown in Japanese Patent Laid-Open No. 2006-148745, the relationship between the image plane and the road surface is obtained by using homography and calibrating the camera in a factory or the like (before the user uses it). It is preferable to obtain.

  For example, as shown in FIG. 3, when the shooting area a is shot by the camera Cam, the shot image is as shown in FIG. On the other hand, the converted image after the projective transformation by the first and second projective transformation units N1 and N2 approximates the image photographed by the virtual camera CamA having a viewpoint looking down on the photographing region a from the substantially vertical direction. However, as shown in FIG. 5, the converted image is a so-called bird's-eye view image, and blank areas are generated in the lower left and lower right due to the absence of image data. Since it is obvious that it is necessary to perform distortion correction processing using a wide-angle lens, detailed description thereof is omitted.

  As a specific example of the conversion image in the first and second projective conversion units N1 and N2, the conversion image generated in the first virtual plane S1 can be the one shown in FIG. 11A. In addition, the conversion image formed on the second virtual plane S2 at a higher level than this can be the one shown in FIG. 11B. In these figures, the point indicated by C (superior concept of C1 and C2) is the optical center.

  As shown in FIGS. 8 and 9, when projective transformation is performed on the first virtual plane S <b> 1 by taking a photographed image of a cuboid three-dimensional object 40 whose bottom end is in contact with the road surface, a solid existing in the converted image. A side (vertical line) connecting the lower end position P and the upper end position Q of the object 40 is formed in a straight line having a length extending from the point P1 to the point Q1 in the first virtual plane S1. Further, when projective transformation is performed on the captured image on the second virtual plane S2, the side connecting the lower end position P and the upper end position Q of the three-dimensional object 40 existing in the converted image is a point on the second virtual plane S2. It is formed in a straight line having a length extending from P2 to point Q2.

  In this case, assuming a lower end side virtual line Lp connecting the lower end position P and the optical center C, and assuming an upper end side virtual line Lq connecting the upper end position Q and the optical center C, the lower end side virtual line Lp and the second virtual line Lp are assumed. The position where the plane S2 intersects is the point P2, and the position where the upper end side virtual line Lq and the second virtual plane S2 intersect is the point Q2.

  Furthermore, as shown in FIG. 10, in the projective transformation unit N (a superordinate concept of the first and second projective transformation units N1 and N2), four virtual planes S (a plurality of virtual planes S1 to S4) are arranged. In the case where a converted image is formed with respect to the superordinate concept of the virtual plane, the lower end position P of the three-dimensional object 40 appears as points P1 to P4 corresponding to the first virtual plane S1 to the fourth virtual plane S4, and has a high level. The image corresponding to the lower end position P of the three-dimensional object 40 is displaced in the direction approaching the optical center C as the virtual plane S increases. In the drawing, the upper end position Q of the three-dimensional object 40 appears as points Q1 to Q4 corresponding to the first virtual plane S1 to the fourth virtual plane S4, and the higher level virtual plane S corresponds to the upper end position Q. Although the image to be displaced is displaced in the direction of the optical center C, the upper end position Q is often not included in the photographed image and is not subject to processing.

  In addition, a solid object that is in a vertically upward direction from the road surface is an image in a posture that is directed in the direction of the optical center C when projective transformation is performed on the virtual plane S.

[Outline of processing by front end detection means]
In the case where the region surrounded by the region R in FIG. 13 is the three-dimensional object region extracted by the first three-dimensional object region extracting means U1 and the second three-dimensional object region extracting means U2, for example, as shown in FIG. A first azimuth straight line D1 (radial straight line similar to the edge line EL in the figure) passing through the optical center C1 and the feature point of the three-dimensional object region in the three-dimensional object region extracting means U1 is assumed. Then, as shown in FIG. 15, a first gradient straight line E1 passing through the optical center C1 and the front end (feature point X) of the three-dimensional object region is obtained on the virtual vertical plane W in the vertical posture including the first azimuth straight line D1. be able to.

  A plurality of second gradient straight lines E2 can be obtained by performing the same process on the solid object region extracted by the second solid object region extraction unit U2. Since the three-dimensional object regions extracted by the first three-dimensional object region extracting means U1 and the second three-dimensional object region extracting means U2 are the same object, the feature points substantially coincide with each other. Therefore, the first gradient line E1 and the second gradient line E2 Are considered to be the same number.

  Further, when the first and second gradient straight lines E1 and E2 are generated, if a three-dimensional object region (three-dimensional object 40) exists at a position in contact with the road surface, as shown in FIG. Even in the case where the converted image is formed in (S1 to S8), the first and second gradient straight lines E1 and E2 intersect on the road surface. On the other hand, in the case where the lower end of the three-dimensional object region (three-dimensional object 40) is overhanging away from the road surface, the converted image is displayed on a large number of virtual planes (S1 to S8) as shown in FIG. Even in the formed one, the first gradient line and the second gradient line intersect at the lower end position of the three-dimensional object region (three-dimensional object 40), but the intersecting positions with the road surface are separated from each other.

  Based on such a principle, a plurality of first gradient straight lines E1 generated based on the first three-dimensional object region extraction means U1 and a plurality of second gradient straight lines E2 generated based on the second three-dimensional object region extraction means U2. As a result, the distance between the optical center C and the lower end position (front end) is detected with high accuracy so that the position of the intersection can be identified.

  In the present invention, the calculation method for obtaining the gradient straight line E can be obtained from converted images formed on a plurality of virtual planes S in addition to the method described with reference to FIG. That is, as shown in FIGS. 17 and 18, the lower end position of the converted image formed on a large number of virtual planes (S1 to S8) is set as the camera side end point, and a straight line passing through the camera side end point is acquired as a gradient straight line. It is. Specifically, camera-side end points formed on a number of virtual planes (S1 to S8) with reference to the optical center C1 in the first three-dimensional object region extraction means U1 are shown as J1 to J8. A straight line passing through at least two of the side end points (J1 to J8) can be acquired as the first gradient straight line E1. Similarly, the camera side end points formed on the virtual plane (S1 to S8) are shown as K1 to K8 with reference to the optical center C2 in the second three-dimensional object region extracting means U2, and these camera side end points are shown. A straight line passing through at least two of (K1 to K8) can be acquired as the second gradient straight line E2.

  Of these figures, for the uppermost virtual plane S8, a converted image is formed in the area JA when the optical center C1 is used as a reference, and a converted image is formed in the area KA when the optical center C2 is used as a reference. The areas JA and KA can be similarly described for the other virtual planes S1 to S7. Therefore, the end portion close to the optical center in the areas JA and KA formed in this way is regarded as a camera side end point.

  In FIG. 17 and FIG. 18 described above, a large number of virtual planes S (S1 to S8) are assumed and existed on the virtual plane S existing on the first gradient line E1 and the second gradient line E2 in plan view. A first cumulative straight line F1 (an example of a first imaginary straight line) and a second cumulative straight line F2 (an example of a second imaginary straight line) representing the number of areas JA and KA as cumulative values are shown. That is, in a region where the first gradient straight line E1 is one layer below the uppermost virtual plane S (S8), since there is one virtual plane S above this region, it corresponds in the first cumulative straight line F1. The value of the area indicates 1. Similarly, in the region where the first gradient straight line E1 exists in the lower layer of the second virtual plane S (S7) from the top, there are two virtual planes S above this region. The value of the corresponding area in the cumulative line F1 is 2. Incidentally, it can be expressed as stepped graphs G1 and G2 corresponding to such numerical values.

  The first cumulative straight line F1 has a slope proportional to the slope of the first slope straight line E1, and the second cumulative straight line F2 has a slope proportional to the slope of the second slope straight line E2. As shown in the drawing, since the first cumulative line F1 and the second cumulative line F2 intersect at the lower end position of the three-dimensional object region, the first cumulative line F1 and the second cumulative line F2 are detected in order to detect the lower end of the three-dimensional object region. Two cumulative straight lines F2 may be used, and FIG. 19 shows a specific processing form for detecting the front end (lower end) of the three-dimensional object region based on the first cumulative straight line F1 and the second cumulative straight line F2.

  That is, a plurality of azimuth lines are assumed from the optical center C1 with respect to a plurality of feature points of the three-dimensional object region extracted by the first three-dimensional object region extraction unit U1, and the optical center C1 is assumed on a virtual vertical plane including the plurality of azimuth lines. And a first gradient straight line E1 passing through the front end (lower end) of the three-dimensional object region is generated. In order to identify the accumulated straight line F1 corresponding to the plurality of first gradient straight lines E1 generated in this way, symbols FX1, FY1, FZ1, and FT1 are given in a predetermined order.

  Similarly, a plurality of azimuth lines are assumed from the optical center C2 for a plurality of feature points of the three-dimensional object region extracted by the second three-dimensional object region extraction means U2, and a virtual vertical plane including the plurality of azimuth lines is assumed. A second gradient straight line E2 that passes through the optical center C2 and the front end (lower end) of the three-dimensional object region is generated. In order to identify the accumulated straight line F2 corresponding to the plurality of second gradient straight lines E2 generated in this way, reference numerals FX2, FY2, FZ2, and FT2 are given in a predetermined order. As described above, the number of azimuth lines passing through the plurality of feature points of the three-dimensional object region extracted by the first three-dimensional object region extracting unit U1 and the optical center C1, and the three-dimensional object region extracted by the second three-dimensional object region extracting unit U2. The number of azimuth lines that pass through the plurality of feature points and the optical center C2 coincide with each other. Accordingly, the numbers of the first cumulative straight line F1 and the second cumulative straight line F2 also coincide.

  Since the first cumulative straight lines FX1, FY1, FZ1, and FT1 generated in this way and the second cumulative straight lines FX2, FY2, FZ2, and FT2 have different optical centers C, the first photographing is performed so as to match the same world coordinate system. By performing coordinate conversion of the image and the second photographed image to the reference coordinates set on the road surface, as shown in FIG. 19, the first cumulative straight line FX1, FY1, FZ1, FT1, and the second cumulative straight line FX2, FY2, FZ2, A plurality of intersections where FT2 intersects are obtained. Thus, it is possible to obtain an intersection d between straight lines generated based on the same feature point among the intersecting intersections, and detect a line connecting these as the front end (lower end) of the three-dimensional object region.

  In particular, the first cumulative straight lines FX1, FY1, FZ1, and FT1 and the second cumulative straight lines FX2, FY2, FZ2, and FT2 are generated by designating feature points in the same set order. A process of obtaining the coordinates of the intersection point d where the generated order of the first cumulative straight line F1 and the second cumulative straight line F2 intersect is performed. As a specific processing form, in the first and second gradient straight line generating means V1 and V2, the order of selecting feature points is set in advance, and a plurality of azimuth straight lines are set according to this order, and each directional straight line is selected. A numerical value is given to indicate the order. Since the numerical value indicating this order is inherited by the gradient straight line, it is set when obtaining the intersection where the first cumulative straight line FX1, FY1, FZ1, FT1 and the second cumulative straight line FX2, FY2, FZ2, FT2 intersect. The first cumulative line F1 and the second cumulative line F2 are compared according to the order. Since the coordinate value of the intersection group where the straight lines set in advance intersect in this way is calculated, for example, the coordinate value of the intersection group where all the straight lines intersect is obtained, and the target lower end (front end) is obtained from this coordinate value. Compared with a process for performing identification processing, it is possible to suppress useless calculation and reduce the amount of calculation, thereby enabling high-speed processing.

  Further, in order to detect the front end (lower end) of the three-dimensional object region, an intersection where the plurality of first gradient straight lines E1 and the plurality of second gradient straight lines E2 intersect may be obtained. Also in this process, it is possible to calculate the coordinate value of the intersection point group where the preset straight lines intersect. For example, the coordinate value of the intersection point group where all the straight lines intersect is obtained, and the target value is obtained from this coordinate value. Compared with the processing for specifying the front end (lower end), the amount of calculation can be reduced, and the processing speed can be increased.

  From such a principle, the front end detection means 3 performs processing for obtaining intersections at which the first cumulative straight lines FX1, FY1, FZ1, FT1 and the second cumulative straight lines FX2, FY2, FZ2, FT2 intersect. The front end detection means 3 obtains an intersection where all the accumulated lines intersect when the number of the first accumulated straight lines F1 and the number of the second accumulated straight lines F2 do not match due to noise or the like. The front end is specified by a statistical method.

  In this way, the front end detection means 3 sets the position where the intersections are accumulated most in the image after movement as the front end of the object. When there are many feature points, the intersections are concentrated at the true front end, and it can be expected that the position is Gaussian distributed at the rear end. In addition, when there are few patterns on the three-dimensional object, many feature points cannot be set, and it is expected that the number of intersections will be reduced. In any case, it is conceivable to estimate a likely intersection concentration position by a robust estimation method, and the estimated position can be specified as a position representing the front end of the object.

[Processing form of obstacle detection device]
In the obstacle detection device of the present invention, an image in which the position of the three-dimensional object is specified, including the processing by the projection conversion unit described above and the processing by the front end detection means 3 from the photographing by the camera Cam, is displayed on the monitor 20. 7 is shown in the flowchart of FIG. 7 and will be described together with the information flow in the control configuration of the ECU 10 shown in FIG.

  When the vehicle 30 moves by a set distance based on the information from the own vehicle position calculation means 1, the photographing control means 2 acquires a photographed image from the camera Cam, and one frame (one frame) photographed earlier is obtained. The first captured image is stored in the first frame memory M1, and the second captured image of one frame (one frame) captured next is stored in the second frame memory M2. In addition, the relative position information of the vehicle 30 and the attitude information of the vehicle 30 at the timing when the first captured image is acquired and the timing when the second captured image is captured are acquired by the imaging control means 2 at the time of shooting. (Step # 101).

  In the first projective conversion unit N1, the first projective conversion unit T1 generates a converted image on the first virtual plane S1 from the first photographed image, and the second projective conversion unit T2 has a second level higher than the first virtual plane S1. A converted image is generated on the virtual plane S2. Similarly, the second projective conversion unit N2 generates a converted image from the second captured image on the first virtual plane S1, and generates a converted image on the second virtual plane S2 (step # 102).

  A converted image generated by the first projective conversion means T1 is shown in FIG. 11A, and a converted image generated by the second projective conversion means T2 is shown in FIG. 11B. As shown in the figure, when attention is paid to the image of the columnar object V (an example of the three-dimensional object 40) at the center of the two types of converted images, the lower end position (indicated by P1) of the image of each columnar object V is the second projection. It is understood that the one generated by the conversion means T2 is closer to the optical center C than the lower end position (indicated by P2) of the one generated by the first projective conversion means T1, and the respective lower end positions are offset from each other. it can.

  The first three-dimensional object region extracting unit U1 and the second three-dimensional object region extracting unit U2 obtain the converted images of the first projective conversion unit T1 and the second projective conversion unit T2 from the projective conversion unit T, and perform these two types of conversion. Based on the image, an image area extraction process, a three-dimensional object candidate area extraction process, and a three-dimensional object area extraction process are performed in this order (step # 103).

  In the image region extraction processing, a differential filter that enhances the contour is applied to each converted image to extract the contour by edge enhancement, and noise is removed by binarization processing and processing that extracts an image that exceeds a predetermined threshold. Extract clear images. By performing this image region extraction process, a three-dimensional object region (image region) shown in FIGS. 12A and 12B is generated.

  In the three-dimensional object candidate area extraction process, two converted images after the image area extraction process are overlapped to perform a logical product process. By this processing, overlapping image areas are extracted as three-dimensional object candidate areas. In the three-dimensional object candidate region extraction processing of the present invention, arithmetic logic such as arithmetic addition (ADD) or arithmetic product (MUL) is used as processing other than processing that takes logical product (AND) as processing for extracting overlapping image regions. Arithmetic processing may be performed.

  In the three-dimensional object region extraction process, the three-dimensional object candidate image region is radial from the optical center C based on the characteristic that the vertical contour line of the three-dimensional object is converted radially from the optical center C on the image converted by the projective transformation unit. It is discriminated whether or not it exists in the region formed at the same time. Specifically, as shown in FIG. 13, it is determined whether or not an extension obtained by extracting the edge line EL of the three-dimensional object candidate image region is directed to the optical center C, and the edge line EL is set to the optical center C. A heading object is extracted as a three-dimensional object area where a three-dimensional object image exists (in FIG. 2, an object surrounded by a region R). Note that the three-dimensional object region can be understood as a region that matches the contour of the three-dimensional object image.

  When performing this extraction, the edge line EL does not necessarily cross the optical center C, and may reach the vicinity of the optical center C. Further, as this three-dimensional object region extraction process, a large number of virtual lines are generated radially from the optical center C, and the three-dimensional object candidate image region whose vertical line posture approximates to the virtual line is extracted as a three-dimensional object region. The processing form may be set to

  By performing these processes, even when a road surface pattern is extracted as a three-dimensional object candidate area by the three-dimensional object candidate area extraction process, the road surface pattern is, for example, a form in which simple geometric patterns are continuous. However, since this geometric pattern is not included in the region formed radially from the optical center C, the road surface pattern is not extracted as a three-dimensional object region.

  When the three-dimensional object region is extracted by the three-dimensional object region extraction process, the first gradient straight line generating unit V1 specifies the feature point of the image included in the three-dimensional object region in the setting order, and thereby the feature point from the optical center C1. A plurality of azimuth straight lines are generated, and a virtual vertical plane including each azimuth straight line is generated, and a first gradient straight line E1 passing through the optical center C1 and the front end of the three-dimensional object region is generated on the virtual vertical plane. To do.

  As a specific calculation method for generating the first gradient straight line E1, as shown in FIG. 16, for example, a three-dimensional object region (three-dimensional object 40) whose lower end is spaced upward from the road surface is exemplified. When the first azimuth straight line DX1 passing through the feature point X at the lower end (front end) of the three-dimensional object region (three-dimensional object 40) extracted by the one-dimensional object region extracting means U1 and the optical center C1 is set, this first It is possible to draw the first gradient straight line E1 passing through the feature point X and the optical center C1 on the virtual vertical plane W (attitude orthogonal to the virtual plane S) including the one-direction straight line DX1. The first gradient straight line E1 is a straight line connecting the optical center C1 with the point SX1 where the lower end of the three-dimensional object region appears in the first virtual plane S1, the point SX2 where the lower end of the three-dimensional object region appears in the second virtual plane S2. Therefore, it is possible to obtain a straight line expression from these points. As described above, as a process of generating the gradient straight line, a plurality of virtual planes S are generated as described above, and the gradient straight lines are generated by straight lines connecting the end points on the camera side of the converted image formed on each virtual plane S. Also good.

  A similar calculation method is used to generate the second gradient straight line E2 in the second gradient straight line generation means V2, and the second gradient straight line E2 includes a second vertical straight line DX2 passing through the feature point X and the optical center C1. A two-gradient straight line E2 is drawn, and an expression of the second gradient straight line E2 is obtained. Since the lower end of the three-dimensional object region is spaced upward from the road surface, as shown in FIG. 19, the intersection of the first gradient straight line E1 and the second gradient straight line E2 is the lower end (front end) of the solid object. It will coincide with the bottom of the region. Then, the front end detection means 3 determines the position of the intersection, thereby determining the distance from the optical center C to the three-dimensional object region (three-dimensional object 40) (Step # 104).

  In particular, when the position of the intersection is determined by the front end detection means 3, the first gradient straight line E1 generated by the first gradient straight line generation means V1 and the second gradient straight line generated by the second gradient straight line generation means V2 are used. One coordinate conversion with E2 is performed, and the process of obtaining the position of the intersection is performed in a state where the optical center C1 coincides with the optical center C2.

  As a specific processing form, as shown in FIG. 14, a plurality of feature points X, Y, Z, T at the lower end (front end) of the three-dimensional object region (three-dimensional object 40) and the optical center C1 are passed. First azimuth straight lines DX1, DY1, DZ1, and DT1 can be set. In addition, a first gradient straight line EX1 (an example of E1) that passes through the three-dimensional object region (three-dimensional object 40) and the optical center C1 on the virtual vertical plane W including the first azimuth straight line DX1 may be generated by calculation. Is possible. The first gradient straight lines EY1, EZ1, ET1 can be generated from the same processing for the first azimuth straight lines DY1, DZ1, DT1 other than this.

  By such processing, the second azimuth lines DX2, DY2, DZ2, and DT2 passing through the optical center C2 can be set, and the second gradient lines EX2, EY2, EZ2, and ET2 can be generated. In FIG. 19, a first cumulative straight line F1 (FX1, FY1, FZ1, FT1) corresponding to a plurality of first gradient straight lines E1 (DX1, DY1, DZ1, DT1) and a second gradient straight line E2 (EX2, EY2, A diagram is shown in which two optical centers C1 and C2 are made to coincide by performing one coordinate transformation with a second cumulative straight line F2 (FX2, FY2, FZ2, FT2) corresponding to EZ2, ET2). As shown in the figure, the process of detecting the front end of the three-dimensional object region (three-dimensional object) from the intersection d of straight lines passing through the same feature points can be described.

  As described above as the process for detecting the front end of this three-dimensional object region, a plurality of first gradient straight lines E1 (EX1, EY1, EZ1, ET1) and a plurality of second gradient straight lines E2 (X2, EY2, EZ2, ET2) It is also possible to obtain an intersection where and intersect, and obtaining the intersection in this way obtains the same detection result as the processing described above.

  By performing these processes, even if the three-dimensional object is lifted from the road surface (even if it is overhanging), this front end coincides with the lower end position in plan view. The distance between the vehicle position can be acquired as the distance between the three-dimensional object and the vehicle 30. In the process of obtaining the coordinates of the intersections of the plurality of first cumulative straight lines F1 and the plurality of second cumulative straight lines F2, the intersections at which the first cumulative straight line F1 and the second cumulative straight line F2 corresponding to the same feature point intersect are obtained. Therefore, it is possible to reduce the amount of calculation by suppressing useless calculation, thereby realizing high-speed processing.

  In the process of detecting the front end, when a plurality of three-dimensional object areas are extracted by the first and second three-dimensional object area extracting units U1 and U2, the process is performed for all of the extracted three-dimensional object areas. For example, it may be performed only on the three-dimensional object region at a position close to the vehicle 30.

  Next, the image position correction information generating means 4 sets correction information for superimposing the converted image from the relative vehicle position and vehicle posture information at the time of shooting the first and second shot images. Then, in the three-dimensional object area specifying means 5, information on the three-dimensional object area extracted by the first and second three-dimensional object area extracting means U1, U2, information on the front end detected by the front end detecting means 3, and an image The position of the three-dimensional object area is determined based on the position correction information. And as a picture which can specify a solid thing corresponding to a shape and size of a solid thing field, a frame-like picture shown as field R in Drawing 13, a specific picture, such as a mesh which covers a solid thing, and this specific picture Is generated (step # 105).

  Then, the projection transformation image generated by the first projection transformation means T1 is corrected by the projection distortion correction unit 6 and displayed on the monitor 20, and the above-mentioned specific image is displayed on the superimposition unit as shown in FIG. 7 is displayed on the monitor 20 (step # 106). In FIG. 14, the frame-shaped region R is shown as a specific image for specifying a three-dimensional object, but the hue of the region of the three-dimensional object is not frame-shaped, marking is displayed in the vicinity, It may not be the form shown in the drawings.

  Since the columnar three-dimensional object in the converted image generated by the projective transformation is converted into a form in which the upper end side extends extremely, the projective distortion correction unit 6 performs correction so as to compress this extension. The superimposing unit 7 performs a process of superimposing and displaying the specific image generated by the three-dimensional object region specifying unit 5 on the projection conversion image generated by the first projective conversion unit T1 based on the locate information.

  As a result, a converted image obtained by projective conversion is displayed on the monitor 20 based on the captured image captured by the camera, and an area corresponding to the three-dimensional object (three-dimensional object area) in the converted image. Since a frame-like image or the like that clearly indicates a three-dimensional object is superimposed and displayed, the three-dimensional object can be recognized on the monitor 20.

[Second Embodiment / Overall Configuration]
In the obstacle detection device according to the second embodiment, as shown in FIG. 20, the configuration of the vehicle 30 is the same except that the first camera Cam1 and the second camera Cam2 are provided in a positional relationship sharing a shooting area. This is common with the first embodiment. Further, the ECU 10 performs processing of the captured image acquired from the first camera Cam1 and the second camera Cam2, but the control mode is basically the same as that of the first embodiment. For these reasons, in the drawings referred to in the second embodiment, the same configurations as those in the first embodiment are denoted by the same reference numerals and symbols as those in the first embodiment.

[Control structure of ECU]
ECU10 has a microprocessor and implement | achieves the process required by running a program. Further, as shown in FIG. 21, the ECU 10 inputs the first photographed image photographed by the first camera Cam1 to the photographing control means 2 and the second photographed image photographed by the second camera Cam2. Although the input point is different from that of the first embodiment, other configurations are the same as those of the first embodiment.

  The imaging control means 2 performs imaging simultaneously with the first camera Cam1 and the second camera Cam2, acquires a first captured image from the first camera Cam1, and acquires a second captured image from the second camera Cam2. This photographing control means 2 holds information indicating the relative positional relationship between the first photographed image and the second photographed image photographed by the first camera Cam1 and the second camera Cam2.

  The ECU 10 includes first and second frame memories M1 and M2, first and second projective transformation units N1 and N2, first and second three-dimensional object region extracting means U1 and U2, and first and second gradients. Straight line generating means V1, V2, front end detecting means 3, image position correction information generating means 4, three-dimensional object region specifying means 5, projective distortion correcting section 6, and superimposing section 7 are provided. It has the same function as the first embodiment.

  The ECU 10 takes a picture with the first camera Cam1 and the second camera Cam2 every time the vehicle 30 moves by a predetermined distance, extracts a three-dimensional image from two frames of captured images acquired at the same timing, and positions the three-dimensional image. Allows identification.

  In the second embodiment, the outline of the process by the projection conversion unit and the outline of the process by the front end specifying means 3 are also common to the first embodiment, and thus the description thereof is omitted.

[Processing form of obstacle detection device]
In the obstacle detection device of the present invention, photographing is performed with the first camera Cam1 and the second camera Cam2, the processing by the projection conversion unit described above and the processing by the front end specifying means 3 are performed, and further Processing is performed until an image in which the position of the object is specified is displayed on the monitor 20. The outline of these processes is shown in the flowchart of FIG. 22, and the flow of information in the control configuration of the ECU 10 is shown in FIG.

  When the vehicle 30 reaches the set position, the imaging control unit 2 acquires the first captured image from the first camera Cam1, and the second camera Cam2 acquires the second captured image. The first captured image of one frame (one frame) is stored in the first frame memory M1, and the second captured image of one frame (one frame) is stored in the second frame memory M2 (Step # 201). Note that the imaging control unit 2 functions as an imaging position information storage unit that stores information indicating the relative positional relationship between the first camera Cam1 and the second camera Cam2, as described above.

  In the flowchart, only the processing of step # 201 is different from step # 101 of the first embodiment, and steps # 202 to # 206 are common to steps # 102 to # 107 of the first embodiment. This processing is omitted.

  As a result, a converted image obtained by projective conversion is displayed on the monitor 20 based on the captured image captured by the camera, and an area corresponding to the three-dimensional object (three-dimensional object area) in the converted image. Since a frame-like image or the like that clearly indicates a three-dimensional object is superimposed and displayed, the three-dimensional object can be recognized on the monitor 20.

  As described in the first embodiment and the second embodiment, in the present invention, the first photographed image and the second photographed image photographed at different timings, or the first photographed image photographed simultaneously by two cameras, A second captured image is acquired. Next, a converted image is generated separately from the first captured image and the second captured image into the first virtual plane S1 and the second virtual plane S2, and the contour of the three-dimensional object or the like is extracted by performing differential processing on the converted image. Noise is removed by binarization processing, and the converted image with the first and second virtual planes after binarization processing is overlapped to obtain a logical product, thereby clarifying the range in which the three-dimensional object etc. exists. A one-dimensional object candidate area and a second three-dimensional object candidate area are extracted.

  When the three-dimensional object is projectively transformed to the virtual plane S at a different level, the lower end position of the three-dimensional object region included in the converted image is different (offset), and the three-dimensional image is centered on the optical center C. It has the characteristic that it is formed along the radial region formed as. From such characteristics, the first three-dimensional object candidate area and the second three-dimensional object candidate area that are formed along the radial area formed around the optical center C can be specified as the three-dimensional object area.

  The first azimuth straight line D1 is set for the three-dimensional object region (image region) specified from the first photographed image, the first gradient straight line E1 is generated, and the three-dimensional object region (image region) specified from the second photographed image is used. The second azimuth straight line D2 is set, the second gradient straight line E2 is generated, and the lower end (front end) of the three-dimensional object is specified by obtaining the intersection d between the straight lines passing through the same feature points among these straight lines. . When this process is performed, the three-dimensional object region is extracted with relatively high accuracy, and in the process of obtaining the coordinates of the intersections of the plurality of first cumulative lines F1 and the plurality of second cumulative lines F2, the same feature point is obtained. Since the intersection where the corresponding first cumulative straight line F1 and second cumulative straight line F2 intersect is obtained, wasteful calculation can be suppressed and the amount of calculation can be reduced, and the processing speed can be increased.

  And even if it is a solid object area | region corresponding to the solid object of the state which overhangs with respect to the road surface, the solid object area | region specifying means 5 is clarified by clarifying a lower end position. The accuracy of the position of the three-dimensional object region specified by (2) is increased, and the accuracy of the distance between the three-dimensional object region (image of the three-dimensional object) specified by the region R and the vehicle 30 with respect to the monitor 20 is increased. Therefore, based on the three-dimensional object region displayed on the monitor 20, it is possible to avoid contact between the three-dimensional object existing in the three-dimensional object region and the vehicle 30, and to visually recognize the state of the three-dimensional object well. .

[Another embodiment]
The present invention may be configured as follows in addition to the embodiment described above.

(A) The projective transformation unit N is configured to form a transformed image with respect to three or more virtual planes S. Even when converted images are formed for three or more virtual planes S, the first and second three-dimensional object region extracting units U1 and U2 perform image region extraction processing and three-dimensional object candidate extraction processing as a plurality of converted images. Will be done. Then, all the converted images from which the three-dimensional object candidate areas are extracted may be overlapped to obtain a logical product. Thus, the detection accuracy can be further increased by taking the logical product of three or more converted images.

(B) The first three-dimensional object region extracting unit U1 and the second three-dimensional object region extracting unit U2 do not perform the three-dimensional object region extraction process, but extract the three-dimensional object candidate region (image region) extracted by the three-dimensional object candidate region extraction process. The control form may be set so as to give to the three-dimensional object region specifying means 5.
By performing such processing, the processing load of the first three-dimensional object region extracting unit U1 and the second three-dimensional object region extracting unit U2 can be reduced, and the processing speed can be improved.

(C) Since the extracted three-dimensional object 40 can be regarded as an obstacle existing on the road surface, the distance from the vehicle 30 to the obstacle (three-dimensional object 40) is acquired by image processing, and the acquired distance is determined from the set value. When the value reaches a small value, a process for operating a buzzer and a process for outputting information approaching an obstacle from a speaker in a synthesized person's words are performed.

  INDUSTRIAL APPLICABILITY The present invention can be used for an image display system that displays a top view around a vehicle based on one frame of captured images acquired by a plurality of cameras provided in the vehicle.

DESCRIPTION OF SYMBOLS 1 Own vehicle position detection means 2 Imaging | photography control means and imaging | photography position information storage means 3 Front end detection means 5 Three-dimensional object area | region identification means Cam Camera Cam1 / Cam2 Camera S1 / S2 Virtual plane (1st virtual plane / 2nd virtual plane)
E1 First azimuth straight line E2 Second azimuth straight line F1 First virtual straight line (first cumulative straight line)
F2 Second virtual straight line (second cumulative straight line)
T1 1st projection conversion means T2 2nd projection conversion means N1 1st projection conversion part N2 2nd projection conversion part U1 1st solid object candidate area extraction means U2 2nd solid object candidate area extraction means V1 1st gradient straight line generation means V2 Second gradient straight line generating means W virtual vertical plane

Claims (3)

A camera that captures the area around the vehicle,
Vehicle position detection means for detecting vehicle position coordinates and vehicle posture;
Shooting control means for shooting the first shot image and the second shot image in which a part of the shooting area is shared by the camera at a timing at which the vehicle is at a predetermined position and a timing at which the vehicle is moved from the predetermined position. ,
A converted image in which the first photographed image is looked down from an upper viewpoint with respect to at least two virtual planes among a virtual plane set on the road surface and a plurality of virtual projection planes different in height from the road surface in a posture parallel to the road surface A first projective transformation unit for projective transformation into,
A transformed image obtained by looking down at least two virtual planes of the second photographed image from an upper viewpoint among a virtual plane set on the road surface and a plurality of virtual projection planes different in height from the road surface in a posture parallel to the road surface. A second projective transformation unit for projective transformation into,
A first three-dimensional object candidate region extracting means for extracting an image region existing in an overlapping region in a state where a plurality of converted images generated from the first photographed image are overlapped by the first projective conversion unit;
Second solid object candidate area extracting means for extracting image areas existing in overlapping areas in a state where a plurality of converted images generated from the second photographed image are overlapped by the second projective conversion unit;
A plurality of first azimuth straight lines passing through a plurality of feature points of the image region extracted by the first three-dimensional object candidate region extraction means and the optical center of the first photographed image are set, and the plurality of first azimuth straight lines First gradient straight line generating means for generating a first gradient straight line that passes from the optical center of the first captured image to the lower end of the image area in a virtual vertical plane including
A plurality of second azimuth lines passing through a plurality of feature points of the image region extracted by the second three-dimensional object candidate region extraction means and the optical center of the second photographed image are set, and the plurality of second azimuth lines A second gradient straight line generating means for generating a second gradient straight line passing through the lower end of the image area from the optical center of the second photographed image in a virtual vertical plane including
A plurality of intersections where the plurality of first gradient lines and the plurality of second gradient lines intersect, in a state in which the first gradient line and the second gradient line are coordinate-converted to reference coordinates set on the road surface, or A straight line passing through the lower end of the image area among the intersections of the first virtual straight line having a gradient proportional to the plurality of first gradient straight lines and the second virtual straight line having a gradient proportional to the plurality of second gradient straight lines Front end detection means for detecting the position of the intersection of each other;
An obstacle detection apparatus comprising: a three-dimensional object region specifying unit that specifies a region where a three-dimensional object is present from the front end of the image region detected by the front end detection unit. Two cameras that shoot around the vehicle with the shooting area shared,
Photographing position information storage means for storing a photographing position relationship between a first photographed image photographed by one of the two cameras and a second photographed image photographed by the other;
The first photographed image is a converted image obtained by looking down at least two virtual planes from an upper viewpoint among a virtual plane set on the road surface and a plurality of virtual projection planes different in height from the road surface in a posture parallel to the road surface. A first projective transformation unit for projective transformation;
The second photographed image is a converted image obtained by looking down at least two virtual planes from an upper viewpoint among a virtual plane set on the road surface and a plurality of virtual projection surfaces different in height from the road surface in a posture parallel to the road surface. A second projective transformation unit for projective transformation;
A first three-dimensional object candidate region extracting means for extracting an image region existing in an overlapping region in a state where a plurality of converted images generated from the first photographed image are overlapped by the first projective conversion unit;
Second solid object candidate area extracting means for extracting image areas existing in overlapping areas in a state where a plurality of converted images generated from the second photographed image are overlapped by the second projective conversion unit;
A plurality of first azimuth straight lines passing through a plurality of feature points of the image region extracted by the first three-dimensional object candidate region extraction means and the optical center of the first photographed image are set, and the plurality of first azimuth straight lines First gradient straight line generating means for generating a first gradient straight line that passes from the optical center of the first captured image to the lower end of the image area in a virtual vertical plane including
A plurality of second azimuth lines passing through a plurality of feature points of the image region extracted by the second three-dimensional object candidate region extraction means and the optical center of the second photographed image are set, and the plurality of second azimuth lines A second gradient straight line generating means for generating a second gradient straight line passing through the lower end of the image area from the optical center of the second photographed image in a virtual vertical plane including
A plurality of intersections where the plurality of first gradient lines and the plurality of second gradient lines intersect, in a state in which the first gradient line and the second gradient line are coordinate-converted to reference coordinates set on the road surface, or A straight line passing through the lower end of the image area among the intersections of the first virtual straight line having a gradient proportional to the plurality of first gradient straight lines and the second virtual straight line having a gradient proportional to the plurality of second gradient straight lines Front end detection means for detecting the position of the intersection of each other;
An obstacle detection apparatus comprising: a three-dimensional object region specifying unit that specifies a region where a three-dimensional object is present from the front end of the image region detected by the front end detection unit.   The first three-dimensional object candidate region extraction unit and the second three-dimensional object candidate region extraction unit perform image region extraction processing for extracting an image region by contour extraction with a contour extraction filter from the two converted images to be processed. The obstacle detection device according to claim 1, wherein the image area is extracted by a process of specifying an image area existing in an overlapping area by superimposing two projected images and accumulating the image area later.

Images