JP5083443B2 - Driving support device and method, and arithmetic device - Google Patents

Description

  The present invention relates to a driving support device that supports driving by capturing an image of a situation around the vehicle with an imaging unit attached to the vehicle and displaying the situation in the vehicle, from an image captured by the imaging unit. The present invention relates to a driving support device that supports driving by synthesizing an image so that a distance from an obstacle around a vehicle can be easily grasped on a screen and displaying it to a driver.

  In a conventional driving support apparatus using an imaging unit attached to a vehicle, there is a technique for changing the viewpoint position of a composite image as a method for making it easy to grasp the distance between the vehicle and the surrounding area on a screen. This is disclosed, for example, in JP-A-58-110334. In the case of this conventional apparatus, a composite image from a new viewpoint, for example, looking down from the sky, is created on the assumption that a captured image around the vehicle is three-dimensionally capturing the ground. In this case, since the distance between the vehicle and another object is proportional to the distance on the screen, the distance can be easily grasped intuitively. In addition, although such a driving assistance apparatus may be called a mobile body image display system, it will be called a driving assistance apparatus in this specification.

  First, the operation of the conventional example will be described with reference to FIG. FIG. 31A is a schematic diagram for explaining the positional relationship among actual imaging means, a virtual viewpoint, and a road surface. In this figure, an imaging means 1001 attached to the vehicle images mainly the road surface behind the vehicle at a downward angle. Assuming that the direction 1002 of a certain pixel of the image of the image pickup means 1001 is a point 1003 on the three-dimensional road surface, the position when viewed from a virtual viewpoint (virtual camera) 1004 above is the virtual camera 1004. It can be seen that it corresponds to pixels in the direction of 1005 in the screen of the image. FIG. 31 (b) shows the situation around the vehicle, the rear camera, and the parking lot in an actual parking lot. The imaging unit 1001 captures an image of the visual field range 1006 in the parking lot. Here, in particular, the direction 1002 of the certain pixel represents a corner point 1003 of a white line on the road surface. FIG. 31C shows an image actually obtained from the imaging means 1001. The point 1003 at the corner of the white line on the road surface is converted in position and synthesized so as to correspond to the pixel in the direction 1005 viewed from the virtual viewpoint 1004 in the sky. FIG. 31D shows a synthesized image. In this figure, since the point 1003 is located on the road surface, the relative positional relationship from the vehicle is accurately reproduced. In addition, since the white lines of the other parking sections are all located on the road surface, the relative positional relationship from the vehicle is accurately reproduced on the composite image.

  The driver can accurately grasp the relationship between the vehicle and the surroundings by looking at the composite image.

  However, this prior art has the following problems. The problem of the conventional example will be described with reference to FIG.

  FIG. 32A illustrates a case where an object located on a surface other than the road surface, such as a car bumper, is reflected in the case of FIG. 31A described above. In this case, a point imaged in the direction 1002 from the imaging unit 1001 is a point (a vehicle bumper) 2001 positioned three-dimensionally above the road surface. However, since the road surface is assumed when combining from the virtual viewpoint 1004, the image is combined on the point 2002 that extends on the direction 1002 and touches the road surface.

  FIG. 32B shows a point 2001 of the position of the actual vehicle, the rear camera, and the bumper of the rear vehicle. FIG. 32C shows a captured image of the imaging unit 1001. On the captured image, the bumper position point 2001 and the point 2002 extending in the direction 1002 and contacting the road surface naturally appear to overlap the same point. However, in the synthesized image shown in FIG. 32 (d), the point 2002 is synthesized farther than the point 2001. Further, as shown in 2003, other parts of the vehicle behind the vehicle are synthesized so as to exist farther than the actual part except for the part of the tire that is in contact with the road surface.

JP 58-110334 A

  As described above, in the conventional technique, only the ground is assumed for the viewpoint conversion, and therefore, what is actually not three-dimensionally positioned on the ground, for example, another vehicle or an obstacle, is distorted on the composite image.

  In addition, when using this as a driving support device, bumpers of other cars are displayed so that they are located farther than they actually are. Then, the distance is not enough for an actual obstacle, and as a result, it becomes easy to contact. For this reason, it is a big practical problem to remove the above-described positional deviation and image distortion.

  As a countermeasure against such distortion of the viewpoint-converted image from above, there is an example disclosed in Japanese Patent Laid-Open No. 7-188683. In this example, the same color is extracted on the entire screen, the road surface and the non-road surface region are separated by expansion and contraction of the region, and the converted image from above is synthesized for the road surface, Does not perform viewpoint conversion on the input image in that area, but only enlarges / reduces and pastes it on the converted image. For this reason, it is possible to synthesize an image without distorting an obstacle located above the road.

  However, in this example, the problems as described in the following (1) to (3) remain.

(1) Since roads and non-road regions are separated based on color information, separation is inaccurate at portions where the texture changes greatly on the road surface, or portions such as buildings that have colors similar to roads.

(2) An obstacle such as a bumper away from the ground is synthesized so that it is a road farther than it actually is.

(3) Since the road surface is converted into an image from an upper viewpoint and the obstacle remains as the input image, the composite image is unnatural, and it is not always easy for the driver to intuitively grasp the surrounding information.

  Regarding the problem (1) of this example, Japanese Patent Laid-Open No. 7-334679 discloses a technique for separating a road surface and a non-road surface using a stereo image. In this example, the images from the left and right cameras are related so that they match at the position projected on the road surface, and the two image signals that are similar within the threshold are defined as road areas, and the rest are separated as non-road areas. It is to do.

However, even with this example, the problems described in the following (1) to (3) remain.
(1) Obstacles such as bumpers that are far from the ground are recognized as roads farther than they actually are.

(2) The vertical edge portion is easy to recognize because of the left and right stereo cameras, but the portion without the edge and the horizontal edge portion cannot be recognized. In particular, a boundary with an obstacle road away from the ground such as a bumper tends to be a horizontal edge on the screen.

(3) Although viewpoint conversion from above is not described in this example, there is no effect on distortion correction on the converted image of the obstacle.

The present invention has been made in order to solve the problems of such a conventional driving support device, and an object of the present invention is to provide an obstacle away from the ground by separating a road surface and a non- road surface. However, an object is to provide a driving support device and method for calculating an actual distance value , and an arithmetic device .

A driving support apparatus according to an aspect of the present invention is a driving support apparatus that supports driving using distance data from the imaging unit to an obstacle obtained based on an image captured by an imaging unit, the imaging unit A predetermined position in the image including the road surface based on the parallax data calculated from the plurality of images including the road surface captured by the imaging unit, the parallax detected from the input image of the imaging unit and the parallax data; Extraction to determine whether or not the vehicle exists above the road surface, obtain a distance value from the imaging means for a position above the road surface, and extract a region sandwiched between positions above the road surface And setting means for calculating first distance data from the imaging means to the area by linearly interpolating a distance value of a position existing above the road surface across the area. Characterized in that that. With this configuration, the road surface and the non- road surface are separated, and an actual distance value is calculated even for an obstacle away from the ground .

The driving support device according to an aspect of the present invention is an image pickup unit installed on a moving body and one or more images picked up by the image pickup unit above the position of the image pickup unit based on a road surface model. An obstacle region for detecting, as an obstacle region, a region where a parallax between images captured by the virtual viewpoint or an image orthogonally projected from above and a parallax between captured images of the imaging unit does not match a parallax on a road surface model And detecting means. With this configuration, a region other than the road surface is detected as an obstacle region using the parallax between the captured images of the imaging unit and the parallax of the road surface model.

Furthermore, the driving support apparatus according to an aspect of the present invention includes an imaging unit installed on a moving body and one or more images captured by the imaging unit, above the position of the imaging unit based on a road surface model. Conversion means for converting an image viewed from the virtual viewpoint or an image orthogonally projected from above, obstacle area detection means for detecting an area that does not match between the converted images as an obstacle area, and the converted image It is characterized by comprising overlay means for overlaying and synthesizing the obstacle area, and display means for displaying the synthesized image. With this configuration, an area that does not match between the converted images is detected as an obstacle area, and a signal indicating the obstacle area is overlay-synthesized in the converted image and displayed.

The driving support apparatus according to one aspect of the present invention includes an imaging unit installed in a moving body, and one or more images captured by the imaging unit, and a distance from the imaging unit based on a road surface model. An intermediate image converting means for converting into an intermediate image with coordinates as an angle, an area that does not match between the converted images is detected, and two images in the area are compared to estimate an actual distance An obstacle area detecting means for correcting the distance position of the area in the converted image using the estimated distance and outputting the corrected area as an obstacle area; An overlay unit that overlays the obstacle region, a conversion unit that converts the combined image into a normal coordinate image of a road surface, and a display unit that displays the converted image are provided. With this configuration, the actual distance of the non-matching region between the intermediate images with the distance and angle from the imaging means as coordinates is estimated, the position is corrected, the corrected region is detected as an obstacle region, A signal indicating the obstacle area is synthesized in the intermediate image, and the synthesized image is converted into a normal coordinate image and displayed.

The driving support device according to an aspect of the present invention is an image capturing unit installed on a moving body and an image captured by the image capturing unit, based on a road surface model, at a virtual viewpoint above the position of the image capturing unit. A driving support apparatus comprising: a conversion unit that converts a viewed image or an image that is orthogonally projected from above; and a display unit that displays the converted image. In the conversion unit, each of the screens of the imaging unit The intensity is obtained based on the area of the pixel on the road surface model and the angle to the road surface, and the brightness and coloring of the pixel are changed based on the intensity. With this configuration, the brightness and coloring of the pixels are changed based on the area on the road surface model of each pixel on the screen of the imaging unit and the angle to the road surface.

Furthermore, the driving support apparatus according to an aspect of the present invention provides an imaging unit installed on a moving body and one or more images captured by the imaging unit from the imaging unit based on a road surface model and a cylindrical model. An intermediate image conversion means for converting into an intermediate image with coordinates of the distance or height and angle of the image, an area that does not match between the converted images is detected, and two images in the area are compared, Obstacle region detecting means for correcting the distance position of the region in the transformed image using the estimated distance and outputting the corrected region as an obstacle region; Overlay means for overlaying the obstacle area on the converted image, conversion means for converting the synthesized image into a normal coordinate image of a road surface, and display means for displaying the converted image Characterized in that was. With this configuration, the position or position is corrected by estimating the actual distance of the non-matching area in the intermediate image with the distance or height and angle from the imaging means as coordinates, and the corrected area is detected as an obstacle area. A signal indicating the obstacle region is synthesized in the intermediate image, and the synthesized image is converted into a normal coordinate image and displayed.

A driving support method according to an aspect of the present invention is a driving support method that supports driving using distance data from the imaging unit to an obstacle that is obtained based on an image captured by the imaging unit. The means calculates parallax data from a plurality of images including the road surface imaged by the imaging means, and the extracting means includes the road surface based on the parallax detected from the input image of the imaging means and the parallax data. It is determined whether a predetermined position in the image exists on the road surface or above the road surface, and a distance value from the imaging means is obtained for the position existing above the road surface, and is sandwiched between the positions existing above the road surface. extracting a region, setting means, this to the distance value of the position that is present above the road surface sandwiching the region by linear interpolation, and calculates the distance data from the imaging means to said region The features. With this configuration, an actual distance value is calculated even for an obstacle separated from the ground by separating the road surface and the non- road surface .

An arithmetic device according to one aspect of the present invention is a arithmetic device that calculates distance data from the imaging unit to an obstacle that is obtained based on an image captured by the imaging unit, and is captured by the imaging unit. Whether a predetermined position in the image including the road surface exists on the road surface based on the calculation unit that calculates the parallax data from a plurality of images including the road surface, and the parallax detected from the input image of the imaging unit and the parallax data Determining whether there is above the road surface, obtaining a distance value from the imaging means for a position above the road surface, and extracting means for extracting a region sandwiched between positions above the road surface; and the distance value of the position that is present above the road surface sandwiching the region by linear interpolation, characterized in that it comprises a setting means for calculating distance data to the region from the image pickup means With this configuration, the road surface and the non- road surface are separated, and an actual distance value is calculated even for an obstacle away from the ground .

As described above, the present invention calculates parallax data from a plurality of images including a road surface imaged by an imaging unit, and includes an image including a road surface based on the parallax and parallax data detected from an input image of the imaging unit. It is determined whether the predetermined position on the road surface is above the road surface, the distance value from the imaging means is obtained for the position above the road surface, and the region sandwiched between the positions above the road surface is determined By extracting and linearly interpolating the distance value of the position existing above the road surface across the area and calculating the distance data from the imaging means to the area , the road surface and the non- road surface are separated and separated from the ground Therefore, it is possible to provide a driving support device having an excellent effect that an actual distance value can be calculated even with an obstacle .

  Further, the present invention converts an image captured by the imaging unit into an image viewed from an upper virtual viewpoint or an orthogonally projected image from above, and uses parallax between a plurality of captured images and parallax of a road surface model. An area other than the road surface is detected as an obstacle area, and a signal indicating the obstacle area is combined with the converted image signal and displayed, so that the distance and direction to the obstacle can be presented more easily and accurately. It is possible to provide a driving support device having an excellent effect of being able to.

  Furthermore, the present invention converts an image captured by the imaging means into an image viewed from the upper virtual viewpoint or an orthogonally projected image from above, and detects a region that does not match between the converted images as an obstacle region. Then, by combining the signal indicating the obstacle region with the converted image signal and displaying it, the driving support device having an excellent effect that the distance and direction to the obstacle can be presented more easily and accurately. Can be provided.

  Then, the driving support device of the present invention converts the image captured by the imaging unit into an intermediate image using the captured image as a coordinate with a distance and an angle from the imaging unit, and the actual region of the mismatch between the intermediate images is actually converted. The position is corrected by estimating the distance of the detected area, the corrected area is detected as an obstacle area, a signal indicating the obstacle area is synthesized in the intermediate image, and the synthesized image is converted into a normal coordinate image. By displaying it, it is possible to provide a driving support device having an excellent effect that the distance and direction to the obstacle can be presented more easily and accurately.

  Further, the present invention converts the captured image into an image viewed from the upper virtual viewpoint or an image orthogonally projected from above, and presents an area of an obstacle with a large distance or distortion on the converted image in an easily understandable manner. It is possible to provide a driving assistance device having an excellent effect of being able to present a highly reliable part and a low part in an image.

  Furthermore, the present invention converts the image captured by the imaging unit into an intermediate image with the captured image as a coordinate or distance and height and angle from the imaging unit, and the actual distance of the non-matching region between the intermediate images , The position is corrected, the corrected area is detected as an obstacle area, a signal indicating the obstacle area is synthesized in the intermediate image, and the synthesized image is converted into a normal coordinate image and displayed. Thus, it is possible to provide a driving assistance device having an excellent effect that the distance and direction to the obstacle can be presented more easily and accurately.

  Then, the present invention converts an image captured by the image capturing means into an intermediate image having an axially symmetric surface about the straight line connecting the image capturing means as a projection plane, and actuality of non-coincident areas between the intermediate images. The position is corrected by estimating the distance of the detected area, the corrected area is detected as an obstacle area, a signal indicating the obstacle area is synthesized in the intermediate image, and the synthesized image is converted into a normal coordinate image. By displaying it, it is possible to provide a driving support device having an excellent effect that the distance and direction to the obstacle can be presented more easily and accurately.

  In addition, the present invention captures a plurality of images having a predetermined parallax, corrects the viewpoint conversion image based on the parallax of the plurality of captured images, and displays it, thereby making it easier to understand the distance and direction to the obstacle. It is possible to provide a driving support device having an excellent effect of being able to present accurately.

  As described above, the present invention can provide a driving support device that has an excellent effect of reducing the burden on the driver and promoting accurate and safe driving.

The block diagram which shows the structure of the driving assistance device of the 1st Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 1st Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 1st Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 1st Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 1st Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 1st Embodiment of this invention, The block diagram which shows the structure of the driving assistance device of the 2nd Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 2nd Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 2nd Embodiment of this invention, The figure which shows the example of the effect of the driving assistance device of the 2nd Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 2nd Embodiment of this invention, The block diagram which shows the structure of the modification of the driving assistance device of the 2nd Embodiment of this invention, The block diagram which shows the structure of the driving assistance device of the 3rd Embodiment of this invention, The block diagram which shows the structure of the modification of the driving assistance apparatus of the 3rd Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance device of the 3rd Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance apparatus of the 3rd Embodiment of this invention and its modification, Schematic for demonstrating operation | movement of the modification of the driving assistance apparatus of the 3rd Embodiment of this invention, The block diagram which shows the structure of the driving assistance device of the 4th Embodiment of this invention, The block diagram which shows the structure of the modification of the driving assistance device by the 4th Embodiment of this invention, Schematic for demonstrating operation | movement of the driving assistance apparatus of the 4th Embodiment of this invention, and its modification, The block diagram which shows the structure of the driving assistance device of the 5th Embodiment of this invention, Schematic which shows operation | movement of the driving assistance apparatus of the 5th Embodiment of this invention, Schematic which shows operation | movement of the driving assistance apparatus of the 5th Embodiment of this invention, Schematic which shows operation | movement of the driving assistance apparatus of the 5th Embodiment of this invention, Schematic which shows operation | movement of the driving assistance apparatus of the 5th Embodiment of this invention, Schematic which shows operation | movement of the driving assistance apparatus of the 5th Embodiment of this invention, The flowchart for demonstrating the search process in the driving assistance device of the 5th Embodiment of this invention, Schematic showing a modification of the driving support device of the fifth embodiment of the present invention, Schematic which shows another modification of the driving assistance apparatus of the 5th Embodiment of this invention, The figure which shows the modification of the structure method of the stereo camera of the 1st-5th embodiment of this invention, Schematic showing the operation of a conventional driving support device, It is the schematic which shows the subject of the conventional driving assistance device.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
The driving support apparatus according to the first embodiment of the present invention converts an image captured by an imaging unit installed in a moving body into an image from an upper virtual viewpoint, and images captured by a plurality of imaging units. By detecting three-dimensional information other than the road surface based on the parallax between them, correcting the distortion of the converted image based on the three-dimensional information, and displaying the corrected image, the driver can It is designed to intuitively and accurately confirm the positional relationship and surrounding conditions of obstacles.

  FIG. 1 is a block diagram of the driving support apparatus according to the first embodiment of the present invention, and FIGS. 2 to 6 are schematic diagrams for explaining the operation of the driving support apparatus.

  As shown in FIG. 1, the driving support apparatus according to the first embodiment of the present invention extracts two imaging means 1001 and 3001, and horizontal edges of images taken by the imaging means 1001 and 3001, respectively. Horizontal edge extraction means 3002, 3003, block setting means 3004 for setting a block for parallax search using the output of the horizontal edge extraction means 3002, output of the horizontal edge extraction means 3003, and block setting means 3004 The search means 3005 for searching for a block based on the output and default disparity data 3010, which will be described later, and the output of the search means 3005, sub-pixel estimation for outputting the sub-pixel accuracy parallax and the reliability determination result for each block On the image screen from the imaging unit 3001 based on the output of the reliability determination unit 3006, the output of the subpixel estimation / reliability determination unit 3006, the default parallax data 3010, and road surface data 3011 described later. 3D map creation means 3007 for creating a D map, 3D image composition means 3008 for composing an image from the upper virtual viewpoint based on the image from the image pickup means 3001 and the 3D map on the screen, and 3D image composition And display means 3009 for displaying the output of the means 3008.

  The driving support apparatus according to the first embodiment of the present invention includes default parallax data means 3010, road surface data means 3011, and vehicle attitude information means 3012.

  The two image pickup means 1001 and the image pickup means 3001 are installed at positions as shown in FIG. Here, they are installed on the straight line 4001 in the direction perpendicular to the ground at positions spaced by 10 cm, and the heights from the ground are 100 cm and 110 cm, respectively. Each line-of-sight direction 4003 is downward at an angle of 40 degrees from the horizontal to the ground. The lines of sight of the two imaging means are parallel. The field of view range 4002 of the two image pickup means is 90 degrees in the vertical direction (elevation direction).

  As shown in FIGS. 2 (b) and 2 (c), the input images from the two imaging means 1001 and 3001 shown in FIG. 1 are almost the same in the horizontal direction and the vertical direction. An image whose position has been changed is obtained. Since the image pickup means 3001 is arranged at a position higher than the image pickup means 1001, the input image from the image pickup means 3001 has the same position on the screen of the horizontal line 4004 at infinity but is located closer (for example, A point 4005 with a white line located on the road surface and a point 4006 on the bumper of the car are located below the image of FIG. 2B on the screen. The change in the vertical position becomes vertical parallax 4007 and 4008 as shown in FIG. 2D in which two images are superimposed. Here, 4007 is the parallax of the white line located on the road surface, and 4008 is the parallax of the bumper of the car located above the road surface.

  In the horizontal edge extraction means 3002 and 3003 shown in FIG. 1, the horizontal edges are extracted from the two captured images by the operation shown in the following equation [1]. Here, L is an image luminance signal, and x and y are horizontal and vertical pixel positions.

L ′ (x, y) = 2 * L (x, y) −L (x, y−1) −L (x, y + 1) (1)

  By this operation, for example, as shown in FIG. 3A, the image of FIG. 2B emphasizes the bumper and white line edges (4009, 4010) close to the horizontal direction on the screen and is close to the vertical direction on the screen. The edges (4011, 4012) are weakened.

  In the block setting unit 3004 in FIG. 1, a block for parallax search is set for the image in which the horizontal edge is emphasized by the horizontal edge enhancement unit 3002 or 3003. FIG. 3 (b) is for explaining the block setting. The maximum and minimum points of L ′ (x, y) shown in the equation [1] are shown in accordance with the scanning line 4013 every two pixels in the vertical direction. And a block 4014 of vertical and horizontal 5 × 5 pixels centered on the point is set. As shown in FIG. 3B, a plurality of these blocks are arranged with an overlap on the horizontal edge of the screen.

  Next, the default parallax data 3010 based on the road surface data 3011 in FIG. 1 will be described. As shown in FIG. 3 (c), the parallax that occurs when it is assumed that the two imaging means 1001 and 3001 are both capturing the road surface is calculated in advance. In FIG. 3C, the parallax of the point 4015 on the road surface is calculated to be 4016 in angle. FIG. 3D shows this default parallax data on the screen. The default parallax 4017 at the position of the horizontal line 4004 at infinity is 0, but the default parallax increases to 4018 and 4019 as it approaches (lowers on the screen and down). Further down, this time, the line-of-sight direction of the imaging means 1001 and 3001 is close to the vertical direction, so that the default parallax becomes small as indicated by 4020.

  Since an object located above the road surface has a parallax larger than the default parallax, the search means 3005 in FIG. 1 searches in the direction of the larger parallax using the default parallax as an initial value of the search.

  The upper limit of the search parallax is determined as follows. As shown in FIG. 4A, assuming a wall 4021 that is 50 cm from the vehicle and perpendicular to the ground, and a wall 4022 that has an angle of 60 degrees from directly below the imaging means 1001 and 3001 Approximate the parallax. The value on the screen of the parallax 4023 at this time is set as the upper limit of the search parallax.

  The search means 3005 uses the image from the imaging means 3001 for the signal L ′ (x + j, y + i) of 5 × 5 pixels set by the block setting means 3004 between the initial value and the upper limit of the search. The signal L3 ′ is searched for the parallax DY that maximizes the correlation value F shown in the following equation [2].

F = ΣiΣjL ′ (x + j, y + i) * L3 ′ (x + j, y + i + DY) Equation [2]
: (i = -2 ~ 2, j = -2 ~ 2)

The subpixel estimation / reliability determination means 3006 further analyzes the parallax DY and the correlation value obtained by the search means 3005.

First, the correlation value F and the autocorrelation value SS of the block signal L ′ (x + j, y + i) = ΣiΣjL ′ (x + j, y + i) * L ′ (x + j, y + i). Formula [3]
: When the ratio F / S of (i = -2 to 2, j = -2 to 2) is greater than or equal to 0.75, it is judged as reliable, and when it is less than 0.75, it is judged as unreliable To do.

  For the block determined to be reliable, as shown in FIG. 4B, the maximum point of the curve 4024 interpolated by a quadratic expression using the correlation value F around the parallax DY in units of one pixel. 4025 is obtained, and this position is set as a sub-pixel precision parallax DY ′.

  The subpixel estimation / reliability determination means 3006 outputs the subpixel precision parallax DY 'and the reliability determination result for each block.

  The 3D map creation means 3007 is based on the subpixel accuracy parallax DY ′ and reliability judgment results, road surface data 3001, and default parallax data 3010 for each block, which is the output of the subpixel estimation / reliability judgment means 3006. Then, a 3D map is created on the image screen from the imaging means 3001.

  First, as shown in FIG. 4C, a horizontal edge of an image from the imaging unit 3001 exists at a position 4026 where each block determined to be reliable is moved by the detected parallax DY ′. . At this time, as shown in FIG. 4 (d), it is determined that the horizontal edge 4027 whose detected parallax DY ′ matches the default parallax data 3010 is present on the road surface, and the horizontal edge 4028 that does not match is above the road surface. Is determined to exist. For the edge determined to be present above the road surface, the distance from the imaging means 3001 is obtained from the value of the detected parallax DY ′.

  Based on this determination and the distance, a 3D map is created for the entire screen as shown in FIG. First, as shown in the scanning line 4032, when the screen is scanned vertically from below, there is no horizontal edge as in the area 4029, or there is only an edge determined to exist on the road surface even if it has The region is determined to exist on the road surface, and a 3D distance is given based on the value of the road surface data 3011. On the other hand, in a region 4030 sandwiched between edges determined to exist above the road surface, a value obtained by linear interpolation of the distance between the two edges is given. In an area 4031 above the edge determined to exist above the road surface, data on the distance of the edge is given.

  The 3D image synthesizing unit 3008 synthesizes an image from the upper virtual viewpoint based on the image from the imaging unit 3001 and the 3D map on the screen. Therefore, as shown in FIG. 5 (b), even on a point 4033 existing above the road surface such as a bumper, on the screen viewed from a virtual viewpoint based on an accurate 3D position instead of the position 4034 on the road surface. Since the position can be determined, as shown in FIG. 5C, the image is synthesized at an accurate position 4033 instead of the position 4034 on the road surface. On the other hand, the shaded portion of the vehicle is displayed with diagonal lines as shown in area 4035.

  At this time, as shown in a region 4036 in FIG. 5 (d), it is possible to emphasize a region existing above the road surface with a translucent red curtain or blinking.

  The composite image is displayed on the display means 3009, and the driver can intuitively and accurately grasp the position of the obstacle around the vehicle.

  Note that the vehicle posture information means 3012 shown in FIG. 1 outputs posture change information when a load is loaded on the vehicle or when the vehicle is accelerated. Corresponding to this information, the road surface data means 3011 calculates changed road surface position data 4038 with respect to the normal road surface position data 4037 for the imaging means 1001 and 3001 as shown in FIG. This changed road surface position data is reflected in the processing of the default parallax data means 3010 and the 3D map creation means 3007 thereafter. As a result, an accurate image is synthesized even when a load is loaded on the vehicle or when the posture changes with the acceleration of the vehicle, etc., and the driver always sees the obstacle around the vehicle by looking at this image. The position can be grasped intuitively and accurately.

  Here, in the first embodiment, by disposing the two imaging units 1001 and 3001 at a distance of 10 cm in the vertical direction, the distortion on the composite image of the obstacle can be corrected by the parallax. However, at this time, even if a small amount of error is included in the parallax, the distance between the two imaging means is vertical, so even if the distance to the obstacle is affected by the error, it will be in the direction of the obstacle. Has the advantage that almost no impact is apparent. This is a very important advantage in driving the vehicle.

  In addition, by disposing the two image pickup means 1001 and 3001 at intervals in the vertical direction, the subsequent parallax detection processing can be limited to the edge portion detected by the horizontal edge extraction means, and the amount of processing can be reduced. It can be greatly reduced. Since a portion floating from the road surface such as a bumper of a vehicle has a horizontal edge on the screen, there is an advantage that it can be detected without leaking an obstacle even in the limited processing.

  As described above, according to the first embodiment of the present invention, the image captured by the imaging unit installed on the moving body is converted into an image from the upper virtual viewpoint, and is captured by the plurality of imaging units. By detecting three-dimensional information other than the road surface based on the parallax between the images, correcting the distortion of the converted image based on the three-dimensional information, and displaying the corrected image, the obstacle can be displayed. The distance and direction of the are presented in an easy-to-understand and accurate manner. Therefore, the driver can intuitively and accurately confirm the positional relationship and surrounding conditions of obstacles around the vehicle by looking at the displayed image.

(Second Embodiment)
The driving support apparatus according to the second embodiment of the present invention converts an image captured by an imaging unit installed on a moving body into an image obtained by orthogonal projection from above, and between images captured by a plurality of imaging units. The area other than the road surface is detected as an obstacle area using the parallax of the road surface model and the parallax of the road surface model, and a signal indicating the obstacle area is synthesized and displayed on the converted image signal so that the driver can It is designed to intuitively and accurately confirm the positional relationship of obstacles and the surrounding situation.

  FIG. 7 is a block diagram of the driving support apparatus according to the second embodiment of the present invention, and FIGS. 8 to 11 are schematic diagrams for explaining the operation thereof. FIG. 12 is a block diagram of a modification of the driving support apparatus according to the second embodiment of the present invention. In the block diagrams of FIGS. 7 and 12, the block elements denoted by the same reference numerals as those in FIG. 1 have the same configuration and function as the block elements in FIG.

  The driving support apparatus according to the second embodiment of the present invention includes an image projection unit 3013, an obstacle edge distance unit 3014, an obstacle region unit 3015, and an overlay unit 3016, and includes a 3D map creation unit and a 3D image composition. It differs from the driving support device of the first embodiment in that no means are provided.

  First, the road surface projection unit 3013 differs from the 3D image synthesis unit 3008 in FIG. 1 on the basis of the data of the road surface data unit 3011 and the input image from the imaging unit 3001 is shown in FIG. It is synthesized so that it exists at the position projected on the road surface. At this time, the viewpoint is not a virtual viewpoint as in the first embodiment, but an image obtained by orthogonal projection from directly above is synthesized as indicated by a viewpoint 6001 in FIG. As shown in FIG. 8B, the composite image obtained at this time is such that the bumper point 6002 of the vehicle located above the road surface exists at a far point 6003 as in the conventional example. The image becomes distorted.

  The obstacle edge distance means 3014 is a sub-pixel estimation / reliability determination means for the horizontal edges of the two images shown in FIG. 8C, similarly to the 3D map creation means 3007 of the first embodiment. From the parallax obtained from 3006, an edge present above the road surface is detected as an obstacle edge, and the distance from the imaging means 3001 is calculated. As shown in the scanning line 6006 in FIG. 8C, when the screen is scanned from the bottom in the vertical direction and an obstacle line exists for the vertical line, the minimum value of the distance is the vertical line. Every time it is stored and output.

  As shown in FIG. 8 (d), the obstacle area means 3015 includes a position 6008 obtained by projecting the minimum value of the obstacle line distance for each vertical line on the road surface, which is the output of the obstacle edge distance means 3014. The farther region 6009 is synthesized as an obstacle region.

  The overlay means 3016 writes an instruction to the driver on a layer (layer) different from the composite image, and as shown in FIG. 9A, the composite image of FIG. 8B and FIG. Are combined. At this time, the obstacle region 6009 is synthesized as a semi-transparent red film on the image in the region where the vehicle in FIG. 8B exists.

  The display means 3009 displays the output of the overlay means 3016. By looking at this composite image, the driver can accurately grasp the position of the white line on the road surface, and accurately grasp the distance and direction from the own vehicle to other obstacles such as vehicles. Therefore, the vehicle can be driven much more safely and accurately than the conventional example.

  In the second embodiment, as compared with the first embodiment, the input image from the imaging unit 3001 may not be combined with a 3D map that changes in real time, and a predetermined road surface model is used. Therefore, there is an advantage that realization with an actual circuit is easy.

  Also, the operation of the obstacle edge distance means 3014 and the obstacle area means 3015 is only compared with the 3D map creation means 3007 because it analyzes only the minimum value of the obstacle line distance for each vertical line. There is an advantage of being simple.

  Therefore, the second embodiment has an advantage that it is much easier to implement with an actual circuit than the first embodiment, and is the same as the first embodiment. By seeing the display image, the driver can accurately grasp the white line on the road surface, and can accurately grasp the distance and direction from the own vehicle to obstacles such as other vehicles. As compared with the conventional example, the effect that the vehicle can be driven significantly safely and accurately is obtained.

  As shown in FIG. 9B, not the obstacle area 6009 but only the position 6008 where the minimum distance of the obstacle line is projected on the road surface may be displayed as the boundary line. Even in this case, since the driver can accurately grasp the distance and direction to the obstacle, there is an effect that the vehicle can be driven safely and accurately.

  FIGS. 10A to 10C are examples showing the effect of FIG. 9B on an actual captured image. FIG. 10 (a) is an image taken from the rear camera of the car being parked. Here, a fisheye lens is used to image a wide-angle range. The image shows the space to be parked, the truck on the left and the passenger car on the right. FIG. 10B is an image synthesized by the conventional example using the captured image of FIG. From the center to the right of the combined image, the parking lot situation seen from above and the CG indicating the host vehicle on the left are combined. Here, since the white line on the road surface is synthesized at an accurate position, the driver can accurately grasp the relationship between the vehicle and the parking space. However, anything above the ground is synthesized to be farther than it actually is. Here, the track and the host vehicle are combined so that there is a sufficient margin, but there is actually a risk of contact. FIG. 10 (c) is a synthesis example in the present embodiment, and the distance to the obstacle is indicated by a white line. The white line shows the distance to the actual truck at a glance, and allows the driver to recognize that there is a risk of contact if he goes back further.

  FIG. 9C shows not only the minimum distance value but also the height from the road surface of the obstacle line, and the boundary to be displayed according to the height from the road surface. This is an example of changing the color and thickness of the line. Here, for example, those with a height of 10 cm or less are thinly displayed in yellow, those having a height of 10 to 15 cm are thinly displayed in red, and those having a height of 20 cm or more are displayed in red and thick. By doing so, the white line 6012 detected due to the deviation between the road surface model and the actual model is displayed in a thin yellow color when the road surface is slightly inclined, and the low curbstone 6011 having a height of 15 cm or less is red. The boundary line 6010 indicating an obstacle such as another vehicle is displayed in red and thick. Thus, the driver can pay attention to important obstacles with the highest priority, and can minimize the influence of noise due to rough roads.

  Further, the boundary line 6010 of an area having a certain height (corresponding to an obstacle) in FIG. 9 (c) may be blinked on the composite image display screen to give further attention to the driver. .

  Further, as shown in FIG. 9 (d), a line 6013 perpendicular to the distance direction from the own vehicle passing through the nearest point with respect to the boundary line 6010 of a region having a certain height or more and the distance to the line 6013 The value may be displayed as a numerical value 6014 to allow the driver to recognize the actual distance value.

  Further, as shown in FIG. 11A, the two imaging means do not necessarily have to be positioned on a straight line in the vertical direction. In the case of the arrangement as shown in this figure, when the vehicle behind is considerably approached, the bumper point 7001 is not visible from the upper imaging means 3001 at the blind spot. As shown in the road surface projection unit 7013 in the block diagram, the image from the imaging unit 1001 can be combined by road surface projection as in the conventional case.

  In this case, as in the conventional case, a deviation between the actual bumper position and the composite image occurs. However, since the bumper position is located in a sufficiently lower direction 7002 when viewed from the imaging means 1001, the positional deviation is small.

  Alternatively, a high-resolution color camera of 1024 × 768 pixels may be used as the upper imaging unit 3001 used for the road projection composite image, and a 640 × 480 pixel monochrome camera may be used as the lower imaging unit 1001 for parallax detection. . In this case, the cost of the lower imaging means 1001 can be suppressed, and a high-resolution color image can be obtained as the composite image.

  Further, as shown in FIG. 11B, the upper imaging means 3001 projects the distant image with high resolution, and the lower imaging means 1001 projects the vicinity with high resolution. As shown in the road surface projection unit 7013 in the block diagram, when performing road surface synthesis, an image signal from the imaging unit 3001 may be used for the far side and an image signal from the imaging unit 1001 may be used for the vicinity.

  As described above, according to the second embodiment of the present invention, an image captured by the imaging unit installed on the moving body is converted into an image orthogonally projected from above and captured by a plurality of imaging units. An area other than the road surface is detected as an obstacle area by using the parallax between the images and the parallax of the road surface model, and a signal indicating the obstacle area is synthesized with the converted image signal and displayed. Present distance and direction more clearly and accurately. Therefore, the driver can intuitively and accurately confirm the positional relationship and surrounding conditions of obstacles around the vehicle by looking at the displayed image.

(Third embodiment)
In the driving support apparatus according to the third embodiment of the present invention, images captured by a plurality of imaging units installed on a moving body are converted into images from an upper virtual viewpoint, and the converted images match. By detecting the area that does not pass as an obstacle area and combining the signal indicating the obstacle area with the converted image signal and displaying it, the driver can intuitively and accurately determine the positional relationship and surrounding situation of the obstacle around the vehicle. It can be confirmed to.

  FIG. 13 is a block diagram of a driving support apparatus according to a third embodiment of the present invention, FIG. 14 is a block diagram of a modification thereof, and FIGS. 15 to 17 are schematic diagrams for explaining the operations thereof. . In FIG. 13 and FIG. 14, block elements having the same reference numerals as those in FIG. 7 have the same configuration and function as the block elements in FIG. 7.

  In the present embodiment, as shown in FIG. 15 (a), as in the first and second embodiments, two imaging means 1001, which are installed at a predetermined interval in the vertical direction, Input an image from 3001. The two input images are synthesized from the virtual viewpoint 1004 at the positions projected on the road surface based on the data of the road surface data means 3011 in the road surface projection means 8001 and 8002, respectively.

  At this time, the bumper point 9001 of the rear vehicle located above the road surface is seen in the direction of 9002 when projected onto the road surface from the upper imaging unit 3001, and is 9003 farther from the lower imaging unit 1001. Looks in the direction of. Therefore, in the images to be combined, as shown in FIGS. 15 (b) and 15 (c), the white lines on the road surface are combined at the same position, but the rear lines positioned above the road surface The bumper points 9001 of the vehicle are combined at different positions.

  The obstacle area detecting means 8003 takes the difference between the two composite images and detects an area having a certain difference or more as an obstacle area. As shown in FIG. 15 (d), the regions detected at this time are located above the road surface, and portions having horizontal edges on the original captured image are regions 9004 and 9005. Detected.

  In the overlay unit 8004, as shown in FIG. 16A, the obstacle region shown in FIG. 15D is superimposed on the image shown in FIG. At this time, the obstacle areas 9004 and 9005 are combined with the area where the vehicle exists in FIG.

  The display means 3009 displays the output of the overlay means 8004. The driver can accurately grasp the distinction between the white line on the road surface and the obstacle, unlike the conventional example, by viewing the composite image. In addition, although the distance from the own vehicle to an obstacle such as another vehicle is not accurate, since the direction can be accurately grasped, the vehicle can be driven safely even compared to the conventional example.

  Further, variations of the third embodiment will be described with reference to FIGS. 14, 16 (b) to (d), and FIGS. 17 (a) to 17 (d). First, the picked-up images from the image pickup means 1001 and 3001 are the lens distortion correction / distance / direction image means 8005 and 8006, respectively, and the image from the image pickup means when projected onto the road surface as shown in FIG. It is converted into an image on coordinates developed in the distance R and the direction θ. At this time, if the captured image includes lens distortion or distortion due to the mounting angle, the distortion amount is measured in advance and corrected simultaneously with the conversion. Images on coordinates developed in the distance R and the direction θ are as shown in FIGS. 16C and 16D, respectively. Here, the image from the imaging means 3001 is shown in FIG. 16 (c) and the image from the imaging means 1001 is shown in FIG. 16 (d). The white line on the road surface is synthesized at the same position, but the edge position of the bumper etc. 16 (d) is synthesized farther away.

  The edge comparison means 8007 compares the edge positions according to the scanning line 9008 in FIGS. 16 (c) and 16 (d). FIG. 17A shows the edge of the image on the scanning line 9008. In this figure, the edge signal corresponding to FIG. 16C is 9009 and the edge signal corresponding to FIG. 16D is 9010. Here, when the edge 9011 of the edge signal 9009 is detected according to the scanning line, if the edge 9012 on the edge signal 9010 exists at the same position as this, it is ignored because it is an edge on the road surface. On the other hand, when the edge 9013 of the edge signal 9009 is detected and there is no edge on the edge signal 9010 at the same position, the distance d to the next detected edge 9014 is detected, and the edge 9013 at this time is detected. And the distance d from the edge 9013 to the edge 9014 are output to the distance estimating means 8008.

  The distance estimation means 8008 estimates the actual distance based on the distance R1 to the edge 9013 and the distance d from the edge 9013 to the edge 9014. FIG. 17B shows the relationship. Here, the height of the imaging means 1001 is H1, the difference in height from the imaging means 1001 to the imaging means 3001 is Hd, and the actual point height H and distance R ′ are calculated from R1 and d of the input. The following two relational expressions [4] and [5] are obtained by simple proportionality.

H * R1 = (H1 + Hd) * (R1-R ') Formula [4]

H * (R1 + d) = H1 * (R1 + d-R ') Formula [5]

  From these relational expressions, the height H and the distance R ′ of the actual point are estimated as the following expressions [6] and [7].

R '= R1 * (R1 + d) * Hd / {R1 * Hd + d * (H1 + Hd)} Equation [6]

H = H1 * (H1 + Hd) * d / {Hd * R1 + (H1 + Hd) * d} Expression [7]

  The estimated height H and distance R ′ are output to the obstacle area means 8009. When the distance R ′ when the height H is equal to or greater than the threshold value is input, the obstacle area means 8009 is positioned at the distance R ′ on the scanning line 9008 as shown in FIG. A line is drawn, and the area farther than that is determined as the obstacle area.

  As shown in FIG. 17C, the overlay unit 8004 overlays the obstacle region on FIG. 16C, which is a converted image from the imaging unit 3001.

  The distance / direction / road surface converting means 8010 converts the distance / direction expanded coordinates of the overlay synthesized image into a normal road surface image viewed from above, and outputs it to the display means 3009.

  The image displayed on the display means 3009 is displayed as an obstacle area at an actual distance R1 even if the obstacle is far from the ground, such as a bumper, as shown in FIG. 17 (d). By seeing this, the driver can drive safely.

This embodiment has advantages as described in the following (1) to (4).
(1) Once an image is converted to coordinates developed in the distance / direction, distortion such as camera lens distortion and camera mounting angle can be corrected.

(2) When directly detecting parallax between two input images, it is necessary to consider these distortions separately, but in this embodiment, this can be omitted.

(3) Even when the viewing angles of the two image pickup means are different, the influence of the difference can be absorbed similarly by this operation.

(4) When comparing edges after projecting road surfaces, it is necessary to compare edges in the distance direction. However, since the distance direction is not constant in the road surface projection image, memory access on actual hardware is complicated. In addition, since the distance direction is not constant when the obstacle area is determined, memory access in actual hardware is similarly complicated. However, here, edge comparison and obstacle region determination are performed in a state in which the image is converted into coordinates once developed in the distance and direction. Since the distance direction is used as a coordinate axis on this converted image, the above-described operation can be realized very easily with actual hardware.

  As described above, according to the third embodiment of the present invention, an image captured by a plurality of imaging means installed on a moving body is converted into an image from an upper virtual viewpoint, and the converted images A region that does not match is detected as an obstacle region, and a signal indicating the obstacle region is combined with the converted image signal and displayed, so that the distance and direction to the obstacle can be presented more easily and accurately. Therefore, the driver can intuitively and accurately confirm the positional relationship and surrounding conditions of obstacles around the vehicle by looking at the displayed image.

(Fourth embodiment)
The driving support apparatus according to the fourth embodiment of the present invention converts an image captured by an imaging unit installed on a moving body into an image from an upper virtual viewpoint, and the converted image is accurate and has little distortion. By synthesizing the parts brightly and synthesizing the dark parts with large distortion and inaccurate distance, the driver can intuitively grasp the highly reliable part and the low part in the image.

  FIG. 18 is a block diagram of a driving support apparatus according to the fourth embodiment of the present invention, and FIG. 19 is a block diagram of a modification thereof. FIG. 20 is a schematic diagram for explaining these operations. 18 and 19, the block elements denoted by the same reference numerals as those in FIG. 7 have the same configuration and function as the block elements in FIG.

  In the present embodiment, as shown in FIG. 18, an image is input from one imaging unit 1001. Then, based on this input image, the image projection unit 10001 synthesizes the image at the position projected on the road surface with the data of the road surface data unit 3011. At this time, the intensity calculation unit 10002 uses the camera parameters of the imaging unit 1001. The intensity projected onto the road surface is calculated and determined from the relationship between 10003 and the data of the road surface data means 3011.

  As shown in FIG. 20A, a constant light beam 11001 per pixel is assumed from the imaging unit 10001. Here, the intensity K when the road surface is projected is calculated as the following equation [8] from the area A on which the pixel projects the road surface and the angle θ of the light ray to the road surface.

  To briefly explain equation [8], the strength decreases as the area A increases, and the strength increases as the angle to the road surface is vertical. Also, the strength is clipped at 1.0.

K ′ = α · sin (θ) / S (8)
if (K ′> 1.0) K = 1.0 elseK = K ′ where α is the amplification intensity and constant

  By calculating the intensity, as shown in FIG. 20 (a), the road surface in the vicinity of the imaging unit 1001 has a high intensity, and the distance in the distance is small. As shown in FIG. 20B, the image synthesized according to the intensity value has a constant brightness since the intensity K is 1.0 from the imaging means 1001 to the road surface in the vicinity. It is synthesized and becomes darker as it goes farther away.

  An image obtained by synthesizing the input image from the imaging means 1001 according to this intensity is synthesized with a dark portion at a distant portion having a large distortion and a large distance error, as shown in FIG.

  By looking at this composite image, the driver can clearly see the composite image of the near part with little error and accurate, and the far side with much error and distortion is synthesized darkly. It can be intuitively recognized that the information has only certainty according to the brightness. Therefore, although the distance to the obstacle cannot be accurately grasped, the reliability of the partial portion in the composite image can be intuitively grasped, and the vehicle can be driven safely even compared with the conventional example.

  The modified example shown in FIG. 19 is obtained by extending FIG. 18 to include a plurality of imaging units. Based on the relationship between the respective camera parameters 10003... 10004 and the road surface data 3011, the intensity calculation means 10002... 10005 calculates and determines the intensity to each road surface projection. Images are synthesized by the image projection means 10001 and 10006.

  These road surface projection combined images are further combined into one image by combining means 10007. At this time, composition is performed with weighting according to the intensity of each road surface projection and displayed on the display means 3009. For example, as shown in FIG. 20 (d), the displayed image is obtained by combining images from the three imaging units into one road surface projection image.

  At this time, the imaging means at the center of the rear of the vehicle has high resolution, and the left and right imaging means use auxiliary low-resolution imaging means. Therefore, since the area projected by the left and right imaging means has a large area per pixel, only a very near area is brightly combined. On the other hand, the area projected by the central imaging means has a relatively small area per pixel, so that it is possible to synthesize brightly far away. In the region 11002 where both projection images overlap, the projection image of the central imaging means having a high intensity is synthesized with a large weight, so that a more reliable image is synthesized.

  By viewing this composite image, the driver can intuitively grasp the reliability of the partial portion in the image even if it is a composite image from a plurality of imaging means, and the vehicle can be safely compared with the conventional example Can drive.

  As described above, according to the fourth embodiment of the present invention, the image captured by the imaging means installed on the moving body is converted into an image from the upper virtual viewpoint, and the converted image is accurately and distorted. By combining bright parts with low distortion and combining dark parts with large distortion and inaccurate distances, the driver can intuitively grasp the high and low reliability parts of the image. This prevents dangerous driving such as moving the vehicle quickly in the direction of the low-reliability portion, and as a result, safer driving can be promoted.

  In the fourth embodiment described above, only the brightness of the image is changed according to the calculated intensity, but coloring or the like may be changed in addition to that. It is also possible to produce a foggy effect by mixing gray or white with low intensity parts, and even in this case it is possible to prevent the driver from moving the car in a direction with less intensity so that driving is safe. Can be encouraged.

(Fifth embodiment)
The driving support apparatus according to the fifth embodiment of the present invention converts an image captured by the image capturing unit into an intermediate image with the distance or height from the image capturing unit as a coordinate, and actually performs the non-matching region between the intermediate images. The position is corrected by estimating the distance of the detected area, the corrected area is detected as an obstacle area, a signal indicating the obstacle area is synthesized in the intermediate image, and the synthesized image is converted into a normal coordinate image. By displaying the information, the driver can intuitively and accurately confirm the positional relationship and surrounding conditions of the obstacles around the vehicle.

  FIG. 21 is a block diagram of a driving support apparatus according to a fifth embodiment of the present invention, FIGS. 22 to 26 are diagrams for explaining the operation, and FIG. 27 is a flowchart for explaining a search process. is there. In FIG. 21, the block elements denoted by the same reference numerals as those in FIG. 7 have the same configuration and function as the block elements in FIG.

  In the present embodiment, the imaging means 1001 and 3001 have fisheye lenses, and the captured images are input to the lens distortion correction / distance-height / direction image means 13005 and 13006, respectively. In the lens distortion correction / distance-height / direction image means 13005, 13006, as shown in FIG. 22 (1), the input image is projected onto the road surface up to a predetermined distance Rmax from the image pickup means, and beyond. Is projected onto a cylinder, and as shown in FIG. 22 (b), a converted image of coordinates developed by the distance R on the road surface at this time or the height H on the cylinder and the direction θ is used. Converted.

  At this time, if the captured image includes lens distortion or distortion due to the mounting angle, the distortion amount is measured in advance and corrected simultaneously with the conversion. For example, the captured image diagrams shown in FIGS. 23 (a) and 23 (b) are as shown in FIGS. 23 (c) and (d), respectively. Here, the predetermined distance Rmax = 300 cm on the road surface, and Rmin = 0 cm and Hmax = 200 cm shown in FIG. 22 (a).

  The edge extraction unit 13007 extracts a horizontal edge by an operation of taking a difference of signals separated by 5 pixels in the vertical direction for each of the converted images. FIGS. 24A and 24B show images obtained by extracting the horizontal edges of the converted images output from the lens distortion correction / distance-height / direction image means 13005 and 13006, respectively. Here, FIG. 24A is from the upper imaging means 3001, and FIG. 24B is the other. As shown by the dashed line 14007 in the figure, the near-vertical diagonal edge is weakened and the horizontal edge is emphasized as shown by the thick solid line 14008.

  The horizontal block matching / distance estimating means 13008 scans the edge position in accordance with the scanning line 14001 and, when detecting the edge maximum point 14002, stores the image signal of the block 14003 in the range of 10 pixels in the vertical and horizontal directions centered on the maximum point. To do.

  Next, for the edge image shown in FIG. 24 (b), on the scanning line 14004 at the same horizontal position as the scanning line 14001, in the range according to the search range data means 13012 shown in FIG. A block 14006 having data most similar to the stored block 14003 is detected. At this time, the distance is estimated using the difference between the center point 14005 of the block 14006 and the vertical position of the edge maximum point 14002 as parallax data on the converted image.

  The operations of the search range data means 13012 and the horizontal block matching / distance estimation means 13008 at this time will be described in more detail based on the flowchart of FIG. The search range data means 13012 stores data corresponding to the vertical position of the center position 14002 of the block 14003. As shown by a point 14013 in FIG. 25 (a), when the position of the point 14002 seen from the upper imaging means 3001 corresponds to the road surface (the position of the point 14002 is Rmax in FIG. 24 (a)). Lower case), a vertical position 14015 on the converted image (FIG. 25 (b)) when a point 14013 on the road surface and a point 14014 at a distance of 50 cm from the image pickup means are respectively viewed by the image pickup means 1001 below. 14016 is stored in the search range data means 13012 (however, at this time, the vertical position 14015 when the point 14013 on the road surface is viewed with the imaging means 1001 below is assumed to be a converted image assuming both road images. Since it is synthesized, it is the same as the vertical position of the point 14002).

  In addition, as shown in FIG. 25A, when the position of the point 14002 viewed from the upper imaging means 3001 corresponds to the cylinder (the position of the point 14002 is based on Rmax in FIG. 24A). In the above case), the vertical positions 14020, 14021, 14022 on the converted image when the infinite point 14017, the point 14018 on the cylinder, and the point 14019 at a distance of 50 cm from the imaging unit are viewed by the lower imaging unit 1001, respectively. Is stored in the search range data means 13012 (however, the vertical position 14021 when the point 14018 on the cylinder is viewed by the imaging means 1001 below is assumed to be a converted image assuming that both images are cylinders. Since it is synthesized, it is the same as the vertical position of the point 14002).

  Based on these search range data, a search process is performed according to the flow shown in FIG.

  First, it is determined whether the position of the point 14002 corresponds to a road surface or a cylinder (step S1).

  If it is determined that it corresponds to the road surface (YES in step S1), first, an absolute value summation (SAD) of the block signal difference is obtained at the vertical position 14015 (step S2), and the SAD value is smaller than the threshold value TH. In this case (YES in step S3), it is determined that the block data match. In this case, the determination is made that the horizontal edge at the point 14002 is an edge on the road surface, and the process is terminated (step S4). If the SAD value is larger than the threshold value TH (NO in step S3), a position where the SAD value is minimum in the range of the vertical positions 14015 to 14016 is searched (step S5). When the minimum SAD value is smaller than the threshold value TH (YES in step S6), the distance and the height from the road surface are obtained using the difference between the position and the vertical position of the point 14002 as parallax data, and this is output. The process ends (step S7). If the minimized SAD value is larger than the threshold value TH (NO in step S6), a determination output that there is no matching horizontal edge is output, and the process ends (step S8).

  When it is determined that it corresponds to a cylinder (NO in step S1), first, the minimum value of the SAD value is obtained in the range from the vertical position 14020 to 14021 (step S9), and the minimum value of the SAD value is smaller than the threshold value TH ( If YES in step S10), it is determined that the block data match. In this case, a determination output is made that the horizontal edge of the point 14002 is farther than the distance Rmax to the cylinder, and the process ends (step S11). When the SAD value is larger than the threshold value TH (NO in step S10), a position where the SAD value is minimum in the range of the vertical positions 14021 to 14022 is searched (step S12). If the minimum SAD value is smaller than the threshold value TH (YES in step S13), the distance and the height from the road surface can be obtained using the difference between the position and the vertical position of 14002 as parallax data. The process ends (step S14). If the minimum SAD value is larger than the threshold value TH (NO in step S13), a determination is output that no matching horizontal edge exists, and the process ends (step S15).

  With the above processing flow, road edges and distant horizontal edges that are not related to obstacles can be judged and removed with a small amount of processing, and only a lot of processing related to obstacles is generated, so there is very little overall. In the process, the distance and height of the edge related to the obstacle can be calculated.

  As shown in FIG. 27, the obstacle boundary means 13009 determines that a horizontal edge whose distance and height are detected has a height of 20 cm or more as an edge of the obstacle, and FIG. ), A line is drawn at the location of the distance R ′, so that the set of horizontal edges becomes a line 14023 indicating the obstacle boundary.

  In the image projecting unit 13010, an image viewed directly from the upper virtual viewpoint is synthesized based on the image of the image capturing unit 3001 separately, and in the overlay unit 13004, as shown in FIG. 25 (c), the image projecting unit 13010 Overlay this obstacle on the image.

  This overlay synthesized image is displayed on the display means 3009. As shown in FIG. 25 (c), the displayed image is such that an obstacle boundary line is displayed at the actual distance even if there is an obstacle away from the ground such as a bumper. By seeing this, the driver can drive safely.

This embodiment has advantages as described in the following (1) to (8).
(1) Once an image is converted to coordinates developed in the distance / direction, distortion such as lens distortion of the imaging means and attachment angle of the imaging means can be corrected.

(2) When parallax is directly detected between two input images, it is necessary to consider these distortions separately, but this can be omitted.

(3) Even when the viewing angles of the two imaging means are different, the influence of the difference can be absorbed by this operation as well.

(4) Also, as shown in FIG. 22 (a), by using a converted image assuming a cylinder in addition to the road surface, it does not appear in the road surface projection image because it is at a high position relative to the imaging means. An obstacle can also be detected, and the boundary line of the obstacle can be displayed on the display image.

(5) According to the processing flow shown in FIG. 27, a horizontal edge on the road surface or far away, which is not related to an obstacle, can be judged and removed with a small amount of processing, and a large amount of processing occurs only for the edge related to the obstacle. Therefore, the distance and height of the edge related to the obstacle can be calculated with very little overall processing.

(6) Since only the necessary range can be searched by providing the search range data means 13012, the distance and height of the edge related to the obstacle can be calculated with very little overall processing.

(7) As shown in FIG. 22 (a), by using a converted image assuming a cylinder in addition to the road surface, the search range data is determined only by the vertical position of the horizontal edge 14002. Since it does not depend, the amount of memory can be greatly reduced.

  For example, when performing stereo matching using an image of an imaging unit having a fisheye lens as it is conventionally, the search range is a curve 14024 depending on the vertical position and horizontal position of the screen, as shown in FIG. It becomes 14025. A very large amount of memory is required to store the curve data. In the present embodiment, this memory can be very small and can be realized with a simple configuration.

(8) In the conventional stereo matching, the position of this curve strictly requires the accuracy of the pixel unit or less, but in the calculation of the SAD value in the actual search, it can be obtained only with the position accuracy of the pixel unit. This quantization noise has an adverse effect because it cannot. In this embodiment, the adverse effect can be avoided.

  FIGS. 28A to 28C illustrate variations of the fifth embodiment. Instead of the road surface and the cylindrical projection surface shown in FIGS. 22 (a) and 22 (b), the position of the upper imaging means 3001 is centered as shown in FIGS. 28 (a) and 28 (b). The spherical surface obtained may be used as the projection surface. Further, as shown in FIG. 28 (c), the horizontal axis of the converted image does not necessarily have to be the angle θ, and may be the one compressed by the function F (θ).

  FIG. 29A illustrates another variation of the fifth embodiment. Here, the upper imaging means 3001 has a resolution of 640 × 480 pixels, and the lower imaging means 1001 used only for stereo matching has 320 × 480 pixels. As a result, the composite image for display can be detected with high resolution and the obstacle boundary can be detected with sufficient and sufficient accuracy, and the configuration of the imaging means can be made inexpensive.

  FIG. 29B illustrates still another variation of the fifth embodiment. The image pickup means 18001 is added on the same axis of the image pickup means 3001 and 1001, and the edge detected as an obstacle is further verified with the image of the image pickup means 18001 so that noise can be reduced.

  FIG. 29 (c) explains yet another variation of the fifth embodiment. By adding imaging means 18002 at a position different from the axis of imaging means 3001 and 1001, obstacles can be detected using vertical edges other than horizontal edges.

  FIGS. 30 (a) to 30 (c) show a configuration in which a single camera is used to capture a stereo image that has been configured with two upper and lower cameras in the first to fifth embodiments. In FIG. 30A, a plurality of mirrors 1901 to 1903 are arranged in front of the lens 1904, whereby stereo images with parallax are obtained on the left and right on one imaging plane 1905, respectively. FIG. 30 (b) is an image in which two convex mirrors 1906 and 1907 are picked up by a camera 1908 to obtain an image with parallax substantially in the vertical direction. FIG. 30C shows an image picked up by the camera 1908 in FIG. 30B. On the screen 1909, an image 1910 taken by the convex mirror 1907 positioned above, and a convex mirror positioned below. An image 1911 taken in 1906 is displayed.

  Here, by adjusting the curvatures of the convex mirrors 1906 and 1907, the imaging range and the resolution of the captured image can be adjusted (the vertical direction and the horizontal direction can also be adjusted independently).

  In FIG. 30B, the range of angles captured by the convex mirrors 1906 and 1907 is substantially the same, but the convex mirror 1906 has a larger curvature than the convex mirror 1907 so that the same range can be captured by a small mirror. Therefore, the image 1911 appears smaller than the image 1910 on the screen 1909. As a result, the image 1910 has a high resolution, and the image 1911 has a relatively low resolution. An image 1911 is used for a composite image for viewpoint conversion, and an image 1910 is used only for stereo analysis.

  That is, by this synthesis, the composite image for display is detected with high resolution and the obstacle area is detected with sufficient and sufficient accuracy, as described in the other variation of the fifth embodiment of FIG. The camera and the image processing apparatus can be made inexpensive.

  As described above with reference to FIGS. 30 (a) to 30 (c), in the first to fifth embodiments, a stereo image taken up by two upper and lower cameras is picked up using a convex mirror or a reflecting mirror. A single camera may be used.

  In the first to fifth embodiments described above, the driving support device of the present invention has been described mainly as generating a rear image, but the present invention is not limited to this, and the front and side It is good also as what produces | generates the image in one.

  In the first embodiment described above, an example based on image synthesis using a virtual viewpoint using a road surface model will be described. In the second embodiment, orthogonal projection from above using a road surface model will be described. Although an example based on image synthesis has been described, this may be based on the other, and even in that case, the effect of the driving support device of the present invention can be obtained without change.

  Furthermore, it is realizable using the program which makes a computer perform all or one part of the function of each means of the driving assistance device of this Embodiment.

  The driving support device of the present invention is optimal as a device that supports driving of an automobile.

1001 Imaging means
3001 Imaging means
3002 Horizontal edge extraction means
3003 Horizontal edge extraction means
3004 Block setting method
3005 Search means
3006 Subpixel estimation / reliability judgment means
3007 3D map creation means
3008 3D image composition means
3009 Display means
3010 Default parallax data means
3011 Road surface data means
3012 Vehicle attitude information means
3013 Image projection means
3014 Obstacle edge distance means
3015 Obstacle area means
3016 Overlay means
7013 Image projection means
8001 Image projection means
8002 Image projection means
8003 Obstacle area detection means
8004 Overlay means
8005 Lens distortion correction / distance / direction image means
8006 Lens distortion correction / distance / direction image means
8007 Edge comparison means
8008 Distance estimation means
8009 Obstacle area means
8010 Distance / direction / road surface conversion means
10001 Image projection means
10002 Strength calculation means
10005 Strength calculation means
10006 Image projection means
10007 Imaging means
13004 Overlay means
13005 Lens distortion correction / distance-height / direction image means
13006 Lens distortion correction / distance-height / direction image means
13007 Edge extraction means
13008 Block matching / distance estimation means
13009 Obstacle boundary means
13012 Search range data means

Claims (7)

A driving assistance device that assists driving by using distance data from the imaging means to an obstacle obtained based on an image taken by an imaging means,
Calculation means for calculating the parallax data from a plurality of images including the captured road surface by the imaging means,
Based on the parallax detected from the input image of the imaging means and the parallax data, it is determined whether a predetermined position in the image including the road surface exists on the road surface or above the road surface, and exists above the road surface. An extraction means for obtaining a distance value from the imaging means for a position to be extracted, and extracting an area sandwiched between positions above the road surface;
Setting means for the distance value of the position that is present above the road surface sandwiching the region by linear interpolation, and calculates a first distance data from the imaging means to said region,
A driving support apparatus comprising: The extraction unit determines that the predetermined position exists on the road surface when an edge at the predetermined position in the image including the road surface matches the parallax data, and determines that the predetermined position exists when the edge does not match the parallax data. Determine that the position is above the road surface,
The driving support device according to claim 1. The extraction unit sets second distance data from the imaging unit to a position existing on the road surface based on the parallax data for a position existing on the road surface.
The driving support device according to claim 1. The image processing apparatus further includes synthesis means for generating image data obtained by performing viewpoint conversion on the captured image based on the first distance data and the second distance data .
The driving support device according to claim 3. A display unit for displaying the image data generated by the combining unit;
The driving support device according to claim 4. A driving assistance method for assisting driving using distance data from the imaging means to an obstacle obtained based on an image taken by an imaging means,
The calculating means calculates parallax data from a plurality of images including the road surface imaged by the imaging means,
The extraction unit determines whether a predetermined position in the image including the road surface exists on the road surface or above the road surface based on the parallax detected from the input image of the imaging unit and the parallax data, and the road surface Find the distance value from the imaging means for the position that exists above, extract the region sandwiched by the position that exists above the road surface,
The setting means linearly interpolates a distance value of a position existing above the road surface across the area, and calculates distance data from the imaging means to the area.
Driving support method. An arithmetic device that calculates distance data from the imaging means to an obstacle obtained based on an image taken by an imaging means,
Calculation means for calculating parallax data from a plurality of images including a road surface imaged by the imaging means;
Based on the parallax detected from the input image of the imaging means and the parallax data, it is determined whether a predetermined position in the image including the road surface exists on the road surface or above the road surface, and exists above the road surface. An extraction means for obtaining a distance value from the imaging means for a position to be extracted, and extracting an area sandwiched between positions above the road surface;
A setting means for linearly interpolating a distance value of a position existing above the road surface across the area, and calculating distance data from the imaging means to the area;
An arithmetic device comprising:

Images