JP3301421B2 - Vehicle surrounding situation presentation device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is intended to display a situation around a vehicle in a manner that is easy for a driver to understand, so that the driver can easily grasp the situation around the vehicle and perform an accurate and safe driving operation. To a device for doing so.

[0002]

2. Description of the Related Art A conventional general vehicle surroundings monitoring apparatus captures the surroundings of a vehicle with one or several cameras, converts the images, and displays the surroundings of the vehicle as a single image. To For example, a first conventional example is disclosed in Japanese Patent Laid-Open No. 3-99952.
No. 6,086,045. The contents are as follows. A plurality of cameras are installed in a vehicle, and the images are attached to a plane coordinate system such as a road surface by a perspective transformation. By placing the viewpoint downward above the center of the vehicle, the driver can understand at a glance the relationship between the vehicle and the surrounding conditions.

FIG. 2 is a block diagram showing an embodiment of a conventional vehicle periphery monitoring device. The images from the cameras 1 to N (201) input to the image conversion unit 202 are converted into other coordinates by perspective conversion, and are combined into one image by the image display unit 203. And a TV monitor installed in the driver's seat
Display at 204. In the image display unit 203, it is also possible to deviate the vehicle display position from the center of the screen by using signals according to the gear position, vehicle speed, turn signal operation, etc. It is.

[0004] A second conventional example which further develops this is disclosed in Japanese Patent Application Laid-Open No. Hei 7-186833.
In the above invention, when presenting the surrounding situation to the driver, the road surface portion and the other portion are distinguished in advance, and the road surface portion is an image observed when the viewpoint is positioned downward above the vehicle center by coordinate transformation. And the outer part of the road is
An image from a camera is directly superimposed on the converted image at an appropriate place with an appropriate size. In this way, the state of obstacles around the vehicle, particularly other vehicles approaching from behind the vehicle, is accurately notified.

[0005]

However, in the conventional vehicle periphery monitoring device as described above, it is assumed that all the objects in the image are on the road surface, and the coordinate system based on the road surface is used. For example, if there are no height components such as white lines, arrows drawn on the road, letters drawn on the road, and pedestrian crossings, the state reflected in the camera is exactly the same as the road surface. It is projected to the upper position, but conversely, for an object having a height component above the road surface, such as a vehicle or a person, the height component is converted to a depth component in the camera viewing direction The higher the height of the object, the greater the distortion of the image projected on the object. For this reason, when the image projected on the road surface is viewed downward from above, the object having the height component is greatly distorted and displayed, and it may be difficult to associate the object with a real object. there were.

FIG. 3A shows a state of distortion of a projected image of an object not on the road surface. When an image of the rectangular parallelepiped of FIG. 3A taken from the camera is projected onto a road surface, a projected image that spreads from the rectangular parallelepiped to the side opposite to the camera is generated. This projected image is shown in gray in the figure.

[0007] In the second conventional example for the purpose of avoiding these problems, it is assumed that the road surface has the same color, and a portion of a specific color regarded as the road surface is extracted, so that the road surface and the road other than the road surface are extracted. A method is used in which a region is separated, an object outside the road surface is cut out while being reflected on the original camera, and the object is pasted on a composite image. However, actually, the road surface area is not a single color but a texture containing various patterns. Therefore, it is difficult to separate the road surface from the road surface only by the color.

Further, since the extracted object outside the road surface is an image from the viewpoint position of the original camera, the viewpoint image is different from the image of the road surface synthesized at another viewpoint position such as looking down from above because the viewpoint is different. The directions of the appearance of the images do not match,
It may be difficult to grasp the situation at a glance. This will be described with reference to the drawings. Fig. 3 (b) shows an image of a car on the road taken from the on-board camera.The object is smaller in the distance, so that the car on the left side of the screen can see the right side, and the car on the right side of the screen can see the left side. appear. FIG. 3 (c) is an image obtained by converting this image into an image in which the road surface is looked down from directly above the road surface. Looking straight down, the white lines on the road are straight parallel lines. The area of the car shown in FIG. 3B is cut out on this road surface, and is enlarged and reduced so as to match the position and is synthesized. Just by enlarging or reducing the image, the direction of the car or the like does not match the viewpoint viewed from the sky because one side of the car remains obliquely. Therefore, the direction of the vehicle that should be parallel to the white line of the road or the like looks as if it is the direction of the arrow in FIG.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and an object of the present invention is to present a situation around a vehicle, which is photographed by a camera mounted on the vehicle, to a driver in an area other than a road surface on which distortion is caused. The present invention is to provide a vehicle surrounding situation presentation device that presents the vehicle surroundings easily.

[0010]

Means for Solving the Problems In order to solve the above-mentioned problems, a vehicle surrounding situation presenting apparatus of the present invention has the following configuration. The basic configuration of the present invention according to claim 1 uses image input means for photographing and inputting a situation around a vehicle, and camera parameters indicating a position, a direction, and characteristics of a camera which has taken an image from the image input means. Road surface projection means for creating a road surface projection image projected and converted to a road surface, and an outside road surface area extraction means for extracting an area of an object other than the road surface from the road surface projection image,
It is characterized by comprising an out-of- road area index synthesizing means for generating an outline of the extracted out-of- road area and synthesizing it with the road projection image as an index for distinguishing an object other than the road from the out-of- road area .

In the vehicle surrounding situation presenting apparatus according to the first aspect of the present invention, a road surface projected image projected on a road surface is generated from an image around the vehicle, and an area of an object other than the road surface is extracted. By combining an index with the road projection image so that the extracted off-road area is clearly distinguished from other areas, the driver is presented with an image of the situation around the vehicle from any viewpoint, and It is possible to present an object other than the road surface so that it can be easily judged.

According to a second aspect of the present invention, there is provided an apparatus for presenting a surrounding condition of a vehicle, wherein a projection image is formed by projecting image data photographed at different positions onto a road surface, and a difference between the projection images is detected to detect the difference between the projection images. The region of the object other than is extracted.

[0013] In the vehicle surroundings presentation device according to the invention shown in claim 8, extracted road outside regions is to determine what caused by distortion of the real object area or projection, based on the determination result Presenting the driver with an image of the situation around the vehicle viewed from an arbitrary viewpoint by displaying the contour of the area outside the road surface on the road surface projected image, and presenting the driver with an object other than the road surface so as to be easily determined. Is possible.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a basic configuration example of a vehicle surroundings monitoring device according to the first aspect of the present invention. The vehicle surrounding situation presentation device according to the present invention has, as a basic configuration, an image input unit 101 for photographing the periphery of the vehicle using a camera installed in the vehicle, and a position, a direction, an angle of view, and the like at the time of photographing the input image. Camera parameters 102 which are information;
An image processing unit 103 that executes predetermined image processing on the input image and a display unit 104 that displays an image of the surrounding situation that has been subjected to the image processing are mainly configured.

The image processing means 103 comprises the following functional blocks. A road surface projection unit 105 that creates a road surface projection image by projecting the input image onto a road surface, an outside road surface region unit 106 that extracts an area of an object other than the road surface from the road surface projection image, A road surface index synthesizing means 107 for generating an index for distinguishing objects other than the road surface and synthesizing the index with the road surface projected image.

Next, details of the processing of the present invention will be described in the order of each component.

The image input means 101 takes a picture of the surroundings from a camera mounted on the vehicle. When shooting at different positions, install multiple cameras at different positions of the vehicle, move the vehicle with one camera and shoot at different times, or perform both of them to shoot at different positions. To get the image.

The camera parameters 102 are data such as the focal length of the camera, the position and direction of the camera at the time of shooting, corresponding to the image shot by the image input means 101.
The camera parameters will be described with reference to FIG. FIG.
FIG. 3 is a conceptual diagram illustrating a positional relationship between a vehicle, a road surface, and a camera.

There is a camera coordinate system Xe-Ye-Ze with the center of the lens of the camera as the origin Oe and the optical axis direction as the Z axis, and the origin Ow on the road surface plane on which the car is mounted with respect to the car, and the road surface plane Xw- Y
World coordinate system Xw-Yw-Zw taken to match the w plane
Is expressed as (Equation 1) using the rotation matrix R and the translation vector T.

Further, a point (xe, ye, ze) in the camera coordinate system;
The coordinates (u, v) in the corresponding image are the focal length of the camera
It is expressed as (Equation 2) using f.

[0022]

(Equation 1)

[0023]

(Equation 2)

When the car does not move, the origin Ow of the world coordinate system does not move, so that the relationship between a plurality of cameras can be expressed by R and T for each camera. On the other hand, when the car moves and the same camera shoots from a different position, the origin of the world coordinate system moves according to the movement of the car. It is represented by the relationship between two world coordinate systems due to movement. The point in the world coordinate system before the movement is Pw (Xw, Yw, Zw), and the corresponding point after the movement is Pw '(Xw', Y
w ', Zw'), the relationship is the translation vector T
m and rotation matrix Rm. Now, assuming that all points are on the road surface and that the movement of the car also moves on the same road surface, Zw and Zw 'are 0, and the rotation matrix Tm is Zw
2-by-2 matrix representing rotation about an axis, translation vector
Tm may also be a two-dimensional vector that only moves in the Xw and Yw directions. This relationship is shown in (Equation 3).

[0025]

(Equation 3)

The Rm and Tm can be obtained by obtaining information on the movement of the vehicle, such as the vehicle speed, the steering angle, and the rotational speeds of the left and right tires, from a sensor. Also, a position measurement system (GPS) using a satellite can be used.

As described above, the focal length f, the rotation matrix R of the camera coordinate system, the translation vector T, and the rotation matrix between the world coordinate systems
Rm and the translation vector Tm are camera parameters 102 representing the position, direction, angle of view, and the like of the camera.

Next, the projection of the image data on the road surface projection image by the road surface projection means 105 will be described with reference to FIG. FIG. 5A is a schematic diagram illustrating a state where a rectangular parallelepiped on a road surface is photographed by a camera mounted on a vehicle.

Assuming that all objects shown in the image taken by the camera are on the road surface (XwYw plane), a point Pw (Xw, Yw, 0) on the object and a corresponding point Pv ( u, v)
Is expressed as (Equation 4) using (Equation 1) and (Equation 2).

[0030]

(Equation 4)

Each pixel (u, v) of the image photographed by the camera shown in FIG. 5A is projected onto the road surface Xw-Yw plane by using (Equation 4). The Xw-Yw plane is also expressed as a two-dimensional array, and the pixels of the camera image corresponding to the elements of each array are applied using (Equation 4) to obtain a projected image as an image. FIG. 5B shows the projection image converted in this manner.
In FIG. 5B, the upper shaded portion is a portion outside the field of view of the camera, and is a portion having no data in the camera image. Such a portion is filled with data indicating the outside of the area. The black dots indicate the position of the camera. The rectangular parallelepiped that is not on the road surface is distortedly projected by the projection assuming that it is all on the road surface.

As one method for extracting an area of an object other than the road surface such as the rectangular parallelepiped, an off-road area extraction means 1 is used.
At 06, an area where a difference occurs between road surface projected images of images captured from different positions is obtained. Details of the processing will be described with reference to the drawings. FIG. 6A shows a projection image obtained by projecting an image taken at the camera position P1 onto a road surface, and FIG. 6B shows a projection image at the camera position P2. Note that the picture of the own vehicle in the figure is added for easy understanding of the positional relationship, and is not included in the actual projected image.

Next, the projected images of FIGS. 6A and 6B are expressed by (Equation 3).
And superimpose them at the same position on the road surface plane. As can be seen from (Equation 3), this conversion is two-dimensional translation and rotation, and positioning can be performed by processing as a two-dimensional image. FIG. 6C illustrates a state in which FIG. 6A is rotated, translated, and superimposed on the basis of FIG. 6B. Images on the road surface, such as white lines, overlap without shifting,
The projected image of an object other than the road surface is displaced because the shape of the distortion differs depending on the camera position. Therefore, if the difference between the two projected images is calculated, the difference becomes 0 since the area where the road surface is captured is the same object in both projected images. Also, the area of the object other than the road surface of the projection image is the area of the road surface in the other projection image, and even in the area other than the road surface, the distortion is different depending on the camera position, so the other parts of the object are different. Because it is projected, it does not become 0 if the difference is taken. For example, if the difference between the luminances of the pixels of the two projected images is calculated, the difference is almost 0 in the case where both are on the road surface. On the other hand, a difference occurs in the rectangular parallelepiped region. Figure 6 shows the state of the difference.
It is shown in (d). The difference between the white line and the like becomes zero and disappears, and only the portion of the rectangular parallelepiped which is not on the road surface has a value. Even in the area of the object other than the road surface, there may be an area where the color of the different part happens to match and the difference is 0, but if the target object does not have a hole inside, it will be a hole. By performing the process of filling the small area, the outline of the area can be extracted. Thus, the area outside the road surface is extracted.

To calculate the difference, in addition to the difference in luminance, a difference in a color space, not a comparison of one pixel, but a difference of a local texture pattern around each pixel may be used. It is possible.

When the relationship between the two camera photographing positions represented by (Formula 3) is obtained from the vehicle speed, the rotation speeds of the left and right wheels of the steering angle, the GPS position information, etc., the accuracy of the positional relationship is sufficient. Therefore, there is a possibility that a deviation occurs in the superposition. In that case, a difference occurs even in the area on the road surface. In this case, after the projected images are superimposed by the rotation matrix Rm and the translation vector Tm of (Equation 3), the overlap is shifted by a small amount of movement and a rotation angle around the position, and the difference is calculated. In the case where Rm and Tm can be obtained only with low accuracy by obtaining the position where the difference in the entire image is minimized,
An area outside the road surface can be extracted. For example, the absolute value of the luminance difference is obtained for each pixel of the image, and the position where the sum of the absolute value of all the pixels of the image is minimum is obtained.

In the description so far, the vehicle is moved,
Although the case of shooting with the same camera has been described, an area other than the road surface can be extracted in a similar procedure when shooting is performed with a plurality of cameras at different positions of the vehicle.

The area extracted by the out-of-road area extraction means 106 in FIG.
In step 07, an index that is easy for the driver to understand is synthesized on the road surface projection image. As an example, the outline of a region other than the road surface in FIG. 6D is obtained by edge processing or the like, and is displayed in red or the like on the road surface projection image in FIG.
It is an image shown in FIG. This road surface projection image is presented to the driver on the display means 104. In FIG. 7A, an illustration of the car is overlaid on the position of the own vehicle so that the positional relationship can be more easily grasped. Further, not only the contour but also a part of the road surface projected image corresponding to the area outside the road surface can be synthesized by applying a specific color, shading, or the like.

By displaying the contour of an object other than the road surface, an area where a collision may occur during driving is emphasized, and the area can be easily recognized at a glance. Also, as shown in FIG. 7 (b), displaying only the contour within a certain distance from the host vehicle makes it easier to grasp nearby objects that require more attention. It is also possible to change the color of the contour according to the distance, blink the contour of the vicinity, and so on.

FIG. 7A shows an area outside the road including the distortion due to projection. On the other hand, if a portion having a high possibility of the true contour of the object is known from among them, it will be more useful in a judgment such as driving. Therefore, next, a method of extracting a portion having a high possibility of the true contour of the object by assuming that the object outside the road surface is constituted by a plane perpendicular to the road surface will be described.

FIG. 8A shows a projected image obtained by projecting a rectangular parallelepiped on a road surface. From the viewpoint 802, a dotted rectangular parallelepiped 801
Is projected on the road surface, and the projected image 803 has a shape distorted due to the distance, height, and the like of each point of the rectangular parallelepiped from the viewpoint. The area occupied by the cuboid on the road surface is
This is a region 801 indicated by a dotted line. Similarly, FIG. 8B shows a projection image 805 obtained by projecting an image of the same cuboid from a viewpoint 804 onto a road surface. FIG. 8C shows an area outside the road surface extracted by the difference between FIGS. 8A and 8B. The outline of this area is classified on the front and back from the viewpoints 802 and 804. A set of points that can be seen directly from the viewpoint, that is, a line connecting the viewpoint and a point on the outline does not cross another outline is defined as a front outline, and the rest is defined as a back outline. For example, for the viewpoint 802, the partial outlines 806 and 807 separated by white circles are the outlines of the front, and the partial outlines 808 and 809 are the outlines of the back.
In addition, the contour 810 is classified as a circumscribed contour as a contour circumscribing a straight line from the viewpoint to the contour line. Viewpoint 80
For 4, the front contours are 806, 809, and 810, the back contour is 808, and the circumscribed contour is 807.

Here, the contour of the true object is included in the contour of the table. For example, looking at FIG.
8, and 806 in FIG.
In FIG. 8C, two parts are hidden in the area. This is because a plane perpendicular to the road surface is always distorted in a direction extending in a direction opposite to the viewpoint, so that a contour visible from the viewpoint matches a true contour. 8A and 8B, it is highly likely that the circumscribed contour is an image in which a vertical edge extending from an end point near the viewpoint to the lower end is extended. Therefore, the height of the circumscribed contour can be calculated from the length of the circumscribed contour.

8 (a) and 8 (b), and FIG. 8 (c), which is the sum of those areas, is obtained as a region. The region where the image from the viewpoint is distorted may be the region of the table. The contour 809 is for the viewpoint 804. So 2
The part that becomes the outline of the table from both viewpoints is extracted as the true outline of the object. In FIG. 8C, 806 is extracted as a true contour. Also, regarding the circumscribed contours 807 and 810, since there is a high possibility of a vertical edge, the length at that point is calculated from the length.

FIG. 9 (a) shows only a portion which is considered to be a true outline extracted and displayed from the outlines shown by the thick line in FIG. 7 (a) by the above method. Areas other than the road surface,
By displaying on the image from the desired viewpoint excluding the contour due to distortion, the part to be careful when driving is clarified, and by looking at the displayed contour and the position of the own vehicle, the distance to the object is reduced. It becomes easy to grasp. If the shape of the target object is known in advance, the position can be overwritten with the shape of the object at that position by comparing it with the true contour part.

By calculating the height from the circumscribed contour, the approximate height of the object can be shown in the projected image. FIG. 9B is an example in which the height of the region is displayed in a portion where the height is likely to be obtained.

FIG. 10 is a projection image of a pole object such as a pole. Also in this case, by performing the above-described classification of the contour, the root portion where the pillar stands can be extracted as the contour of the table, and the height information can be displayed more than the circumscribed contour.

It should be noted that the above-described method of obtaining a true outline portion from the outline of the off-road area can also be applied to an off-road area obtained from a single image, such as in Conventional Example 2.

[0047]

As described above, according to the first aspect of the present invention, when an image from an arbitrary viewpoint is synthesized using an image from a camera installed in a car, a road surface index is synthesized. Since the index for discriminating objects other than the road surface is combined with the road surface projected image by the means, it is possible to present the driver with the object other than the road surface so that it can be easily determined.

[0048] Also by using the method for extracting the road outside the region of the present invention according to claim 8, since the method using a conventional flat, thus the height component is converted into the depth component of the camera viewing direction, the road surface The problem is that objects that have a height component above are greatly distorted when projected on a road surface.However, from the projected image of the distorted object, extract the outline of the true object and display it in a way that is easy to understand. Became possible.

Further, according to the present invention according to claim 1 0,
In some parts, it has become possible to calculate the height of the object and combine it with the road surface projected image.

Therefore, the driver can more easily recognize the surrounding situation such as the position of the object outside the road surface by presenting the information on the area outside the road surface as compared with the case where only the composite image of the road surface projection is presented. It can be expected that a proper driving operation can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a basic configuration example of a vehicle surrounding situation presentation device according to claim 1 of the present invention.

FIG. 2 is a block diagram showing a configuration example of a conventional vehicle surrounding monitoring device;

FIGS. 3A and 3B are explanatory diagrams relating to distortion of a projected image of an object outside a road surface; FIGS. 3B and 3C are explanatory diagrams illustrating pasting and combining of an object outside a road surface by a conventional vehicle periphery monitoring device;

FIG. 4 is a diagram showing a relationship between a camera coordinate system, a world coordinate system, a road surface, and camera parameters.

FIG. 5 is a diagram showing an example of road surface projection of a rectangular parallelepiped;

FIG. 6 is a diagram showing a flow of processing of an off-road area extraction unit 106;

FIG. 7 is a diagram showing a method of displaying an area outside the road surface;

FIG. 8 is a diagram showing a process of classifying the contour of the area outside the road surface;

FIG. 9 is a diagram showing an example in which a true outline is displayed by using outline classification.

FIG. 10 is a diagram showing an example in which a columnar object is displayed by using contour classification.

[Explanation of symbols]

 101 image input means 102 camera parameters 103 image processing means 104 display means 105 road surface projection means 106 out-of-road area extraction means 107 out-of-road area index combining means

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI B60R 21/00 628 B60R 21/00 628C 628H 628Z G06T 1/00 280 G06T 1/00 280 (72) Inventor Jun Morimura Kadoma, Osaka Prefecture 1006 Kadoma, Ichidai, Matsushita Electric Industrial Co., Ltd. (56) References JP-A-7-186833 (JP, A) JP-A 8-305999 (JP, A) JP-A 8-242443 (JP, A) JP-A-3-99952 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N 7/18 B60R 1/00 B60R 21/00 G06T 1/00 G08G 1/16

Claims (10)

(57) [Claims] 1. An image input means for photographing and inputting a situation around a vehicle, a position of a camera for photographing an image from the image input means,
A road surface projection unit that creates a road surface projected image that is projected and converted onto the road surface using camera parameters indicating the direction and characteristics; an outside road surface region extraction unit that extracts a region of an object other than the road surface from the road surface projected image; An outside road surface area index synthesizing means for generating an outline of the off- road area and synthesizing the road surface projected image as an index for distinguishing an object other than the road surface from the off-road area, apparatus. 2. The vehicle surrounding situation according to claim 1, wherein the off-road surface area extracting means superimposes a road surface projected image obtained by road-projecting images photographed at different positions to extract an off-road surface area. Presentation device. 3. The vehicle surrounding situation presentation device according to claim 2, wherein said outside road surface area extracting means superimposes a road surface projected image on the basis of vehicle position information. 4. The off-road area extracting means includes a vehicle speed sensor, a steering angle sensor, a tire rotation speed,
4. The apparatus according to claim 3, wherein the position is obtained by at least one of position measurement by a satellite. 5. The method according to claim 2, wherein the off-road area extraction means performs two-dimensional rotation and translation of the road surface projection image and superimposes the road projection image on a position where the difference is minimized. Vehicle surrounding situation presentation device. 6. The road surface area extracting means superimposes the road surface projected image on the basis of information on the position of the vehicle, and then performs two-dimensional rotation and translation of the projected image to correct the position to minimize the difference. The vehicle surrounding situation presentation device according to claim 2, wherein 7. The out-of-road area index synthesizing means, which expresses the out-of-road area in different expressions according to a distance from a shooting position,
The vehicle surrounding situation presentation device according to claim 1, wherein the vehicle surrounding situation presentation device is combined with the road surface projection image. 8. The road surface area index synthesizing means classifies the contour generated by the distortion and the contour of the object, and synthesizes the outline of the road surface area with the road surface projected image based on the classification result. The vehicle surrounding situation presentation device according to claim 1, wherein 9. The out-of-road area index synthesizing means classifies the outline of the out-of-road area into a portion caused by distortion and an outline of an object based on a relationship with a camera position, and synthesizes the contour with the road surface projected image. The vehicle surrounding situation presentation device according to claim 8, wherein: Wherein said road area outside the index combining unit obtains the height of the portion of the object from the contour of the road outside the region, the vehicle surroundings according to claim 1, wherein the combining into the road-surface projection image Presentation device.