JP2002087160A - Vehicular surrounding image processing device and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicular surrounding image processing device and a recording medium capable of depicting by assuming a part out of a visual field of a camera. SOLUTION: In a step 100, a shift position is judged whether in back or not. In a step 110, a bird's-eye view image is produced by transforming the coordinate of a taken image. In a step 120, a coincident area is extracted of two successive images. In a step 140, before depicting, a part previously depicted on a screen B is shifted only by a moving area, thereby ensuring a depiction area. In a step 140, an image of the moving area is depicted in the depiction are ensured in a step 150. In a step 160, the depiction indicating own vehicle and visual field angle is conducted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle peripheral image processing apparatus and a recording medium capable of photographing the rear of a vehicle and displaying the photographed image on a monitor.

[0002]

2. Description of the Related Art Conventionally, as a device for displaying a state behind a vehicle, an image of a camera attached to a rear portion of the vehicle has been used.
There is known a display device that directly outputs a signal to a monitor. In this display device, the state behind the vehicle can be known to a certain extent,
There is a problem that it is difficult to understand the relative positional relationship between the image of the subject vehicle (for example, a parking frame) displayed on the monitor and the host vehicle.

[0003] Apart from this, Japanese Patent Application No. Hei 1-2118
No. 49 proposes a technique in which an image captured by a rear camera (rear camera image) is converted into a bird's-eye view, and the host vehicle is displayed therein.

[0004]

In the case of this technique,
Since the own vehicle is shown in the bird's-eye view, the positional relationship between the target object displayed in the image and the own vehicle is easier to understand than displaying the captured image as it is on the monitor. There was another problem that some of the items could not be displayed.

[0005] Therefore, for example, when the vehicle is moved backward into the parking frame, the parking frame that is out of the field of view of the camera cannot be displayed, so that the operation of inserting the vehicle into the parking frame is still not easy. was there. The present invention has been made to solve the above problems, and has as its object to provide, for example, a vehicle peripheral image processing apparatus and a recording medium that can estimate and draw a part that is out of the field of view of a camera. And

[0006]

Means for Solving the Problems and Effects of the Invention (1) According to the first aspect of the present invention, a photographing means (for example, a camera) for photographing an image around the vehicle and a display means (for example, a monitor) for displaying the image are provided. The present invention relates to a vehicle peripheral image processing apparatus that processes an image photographed by photographing means for a vehicle provided and displays the processed image on a display means.

In the present invention, a bird's-eye view image is generated by converting an image photographed by the photographing means into data of a ground plane coordinate system projected from the photographing means as a viewpoint. Thus, the image is not a distorted image captured by a camera, for example, but an easy-to-view bird's-eye view image as viewed from above the ground. Next, the first bird's-eye view image generated in this manner is compared with a second bird's-eye view image generated after the first bird's-eye view image, and a matching area and a non-matching area are determined. Then, a non-coincidence area obtained by this determination is added to the second bird's-eye view image to create a composite image, the composite image is processed so as to be displayed on the display unit, and displayed on the display unit.

That is, as the vehicle moves, for example, the field of view of the camera changes. Therefore, the first bird's-eye view image and the second bird's-eye view image (during different photographing times) naturally have a mismatched area (first A portion which is displayed in the bird's-eye view image but disappears in the second bird's-eye view image) occurs.

Therefore, in the present invention, a shift between the image of the first bird's-eye view image and the image of the second bird's-eye view image is detected, and the image of the shifted portion (the portion displayed in the past) is added to the second bird's-eye view image. Thereby, the situation around the vehicle can be grasped more clearly. As described above, according to the present invention, for example, an area that is out of the current field of view of the camera can be displayed, so that, for example, when parking in the back, the operation is extremely easy.

(2) According to the second aspect of the invention, the image photographed by the photographing means is, for example, an image behind the vehicle photographed by a camera arranged at the rear of the vehicle. Therefore, for example, when the vehicle is put in the parking frame in the back, the positional relationship between the parking frame and the vehicle can be easily understood, so that there is an effect that parking can be easily performed.

When the side of the vehicle can be photographed by the photographing means, the side can be easily understood because the situation of the side can be easily understood when the side is moved by the back.
Further, the image photographed by the photographing means includes, in addition to the rear of the vehicle, the front of the vehicle. In this case, if the side of the vehicle can be photographed,
Since the situation on the side can be understood well, the width can be easily adjusted.

(3) According to the third aspect of the invention, the bird's-eye view image generating means extracts an image indicating a lane (for example, a parking frame) to generate a bird's-eye view image. That is, when generating a bird's-eye view image, the entire captured image may be converted to a bird's-eye view image, but only the lanes (for example, indicated by white lines)
It may be extracted and displayed by performing a value conversion process. In this case, since only the lane needs to be processed, the load of image processing is reduced.

For example, when a vehicle is put in a parking frame surrounded by lanes, only the position of the parking frame needs to be recognized, and it is difficult to see if there is another image. Therefore, only the lanes are extracted. There is an advantage that the operation of the vehicle is easier. (4) In the invention of claim 4, the first bird's-eye view image and the second bird's-eye view image are temporally continuous bird's-eye view images.

In this case, since the difference between the image of the first bird's-eye view image and the image of the second bird's-eye view image is slight, it is easy to detect the coincident area and the non-coincidence area. Further, even when a non-coincidence area is added to the second bird's-eye view image, the connection between the images is preferable because there is no sense of incongruity. (5) According to the fifth aspect of the invention, the display processing means performs coordinate transformation for displaying the image data of the composite image on the display means.

That is, in order to display a bird's-eye view image on, for example, a monitor, it is necessary to adjust the size of the image. Therefore, necessary processing is performed so that the image can be suitably displayed on the monitor. (6) According to the sixth aspect of the present invention, when displaying the composite image on the display means, an image indicating the own vehicle is additionally displayed.

According to the present invention, by displaying an image showing the own vehicle in addition to a composite image showing the situation around the vehicle,
Since the positional relationship between your vehicle and the parking frame, for example, becomes clear,
Operation of the vehicle becomes easy. Since the camera is fixed at a predetermined position of the vehicle, for example, the position of the host vehicle on the screen of the monitor is always constant. Therefore, the host vehicle can be displayed at a fixed position on the monitor screen, and the vehicle may be described in advance by printing or the like on the monitor screen.

Since the viewing angle of the camera is usually constant, the viewing angle (left and right ranges) of the camera may be displayed not only on the own vehicle but also on the monitor screen. (7) A recording medium according to a seventh aspect of the present invention is a recording medium storing means for executing the processing by the above-described vehicle peripheral image processing device.

That is, there is no particular limitation as long as it stores means such as a program capable of executing the processing of the above-described vehicle peripheral image processing apparatus. For example, examples of the recording medium include various recording media such as an electronic control unit configured as a microcomputer, a microchip, a floppy disk, a hard disk, and an optical disk.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments (embodiments) of a vehicle peripheral image processing apparatus and a recording medium according to the present invention will be described. First Embodiment a) A basic system configuration of the present embodiment will be described with reference to FIGS.

As shown in FIG. 1, a vehicle peripheral image processing apparatus according to this embodiment includes a camera (for example, a CCD camera) 1 disposed at the rear of an automobile and a vehicle-mounted monitor (for example, a liquid crystal display) disposed on a dashboard. 3 and an image processing unit 5 for performing image processing.

The image processing unit 5 is, as shown in FIG. 2, an electronic device for processing image data mainly including a microcomputer. The image processing unit 5 functionally performs coordinate conversion of image data captured by the camera 1. And a matching unit 13 that takes in two temporally continuous bird's-eye view images and compares the two bird's-eye view images, and deviates from the field of view of the camera 1 from a mismatched portion between the two bird's-eye view images. An area estimating unit 15 for estimating an area and a drawing unit 17 for drawing an image to be displayed on the monitor 3 are provided.

Incidentally, the image processing unit 3 may be integrated with the camera 1. b) Next, the image processing procedure and the like in the present embodiment will be described with reference to FIG.
This will be described with reference to FIG. Here, a case where the vehicle is moved backward (back) and put into the parking frame will be described as an example.

First, an image (output image) output from the camera 1 is shown in the left diagram of FIG. Since this image is an image of the parking frame drawn on the ground and its surroundings taken by the camera 1 arranged at the upper part of the rear part of the vehicle, the position of each line of the own vehicle (therefore, the camera 1) and the parking frame is determined. The originally rectangular parking frame is displayed distorted in accordance with the distance of the vehicle.

Next, a bird's-eye view image is generated by performing coordinate transformation on the image shown on the left side of FIG.
Further, an image of the position of the own vehicle and the viewing angle of the camera 1 is added to the bird's-eye view image and synthesized, and the synthesized image is formed as shown in FIG.
(B) Display on the screen of the monitor 5 as shown in the right figure.

That is, since the mounting height, mounting angle, and viewing angle of the camera 1 are known, the output image of the camera 1 is subjected to coordinate conversion by the coordinate conversion unit 11 (known bird's-eye view conversion).
Then, a bird's-eye view image as shown in the right diagram of FIG. The process of converting the image captured by the camera 1 into a bird's-eye view image and displaying the image on the monitor 5 can use a conventional technique (for example, Japanese Patent Application Laid-Open No. H10-211849) as described below.

Here, an image (for example, a parking frame) on the ground plane is obtained by performing the reverse process of the normal perspective transformation.
Is obtained as a bird's eye view. Specifically, as shown in FIG. 4, a perspective transformation is performed when projecting the position data of the ground image onto a screen plane T at a focal distance f from the camera position R.

More specifically, the camera 1 has a point R on the Z axis.
Located at (0,0, H), and looking down on the ground surface (xy
It is assumed that an image on the coordinate plane) is being monitored. Therefore, here, the two-dimensional coordinates (α, β) on the screen plane T can be converted into coordinates on the ground surface (bird's eye view coordinates) (reverse perspective transformation) as shown in the following equation (1).

[0028]

(Equation 1)

That is, by using the above equation (1), the projection image data can be converted into image data corresponding to the screen of the monitor 5 (showing a bird's-eye view image) and displayed on the monitor. Next, when the vehicle comes back and approaches the parking frame, FIG.
(B) As shown in the left diagram and the left diagram in FIG. 3 (c), the portion close to the vehicle gradually deviates from the field of view of the camera 1, but at this time, the matching unit 13 continues in time. Two bird's-eye view images are captured and compared.

For example, FIGS. 5A and 5B show continuous bird's-eye view images (changed from image A to image B).
As will be described later, by matching the image A and the image B, it is possible to extract an area (matching area) that completely matches the two images.

Since the image B is an image in which the vehicle is closer to the parking frame than the image A, a part of the parking frame is cut. Therefore, the region estimating unit 15 estimates the V-shaped region on the camera 1 side among the regions where the images do not match (mismatch region) as the region where the image has moved due to the movement of the vehicle (moving region).

Then, by synthesizing the moving area of the image A with the image B, as shown in FIG. 5C, a synthesized image in which a vehicle frame (synthesized area) outside the field of view of the camera 1 is also displayed. Can be created. Here, a method of matching the image A and the image B will be described, but various conventional methods in image processing can be adopted.

For example, a predetermined (matching) area of the images A and B is divided into a plurality of small areas (pixels) as shown in FIG. A tall object can be detected, and the positional relationship between the image A and the image B (that is, how much the image A has moved) can be obtained from the positional relationship between the matching pixels.

Specifically, as shown in FIG. 6, consider the case where the degree of brightness (or color) of a figure at each pixel is indicated by a number. And the brightness of the upper three rows of pixels of the image B match. Therefore, in this case, the image A is
It can be considered that it moves in the vertical direction of the figure by the width of one column of pixels and matches the image B. In this case, the figure of the image B appears to move upward due to the vehicle moving downward in the figure.

That is, when comparing the image A and the image B, the two images are divided into small regions, and when the two regions match each other in many regions (when the degree of coincidence is high). Indicates that the matching part is the same image. Therefore, if the coincident portion is shifted between the image A and the image B, it can be considered that the image is shifted due to the movement, and the movement amount and the movement direction can be obtained.

Similarly, the amount of movement in the left-right direction and the diagonal direction can be obtained. Next, when displaying this composite image on the monitor 3,
As shown in the right diagram of FIG. 3B and the right diagram of FIG. 3C, the drawing unit 17 draws the above-described composite image, and also combines and draws the image of the vehicle position and the viewing angle of the camera 1. I do.

In this case, since the camera 1 is fixed to the vehicle and is fixed, it may be displayed at the same position on the screen of the monitor 3 (for example, at the upper center of the display screen). c) Next, the processing performed by the image processing unit 5 and the like will be described with reference to the flowchart of FIG.

As shown in FIG. 7, first, at step 100, it is determined whether or not the shift position is the reverse position. If an affirmative determination is made here, the process proceeds to step 110, while if a negative determination is made, the process proceeds to step 170. The shift position is
It can be detected by a signal from a shift position detection sensor (not shown) or a signal from another electronic control unit.

In step 170, the image of the area outside the field of view of the camera which was previously drawn is erased, that is, only the current camera image (bird's eye view image) is displayed.
Return to 0. On the other hand, in step 110, a coordinate conversion of the image captured by the camera 1 (the bird's-eye view conversion) is performed to obtain a bird's-eye view image.

In the following step 120, a coincident region is extracted from two successive bird's-eye view images. For example, a matching area is extracted from the image A and the image B in FIG.
In the following step 130, a moving area that deviates from the current field of view of the camera 1 is extracted.

In the following step 140, the moving area of the screen A is drawn later out of the field of view of the camera on the screen B, but before drawing, the part already drawn on the screen B is limited to only the moving area (screen B). Down)
Allocate a drawing area. In the subsequent step 150, as shown in FIG. 5C, for example, an image of the moving area is drawn in the drawing area secured in the step 130 (a composite image is created).

In the following step 160, for example, FIG.
(C) As shown in the right figure, a V-shaped line indicating the host vehicle and the viewing angle is drawn, and the process is temporarily terminated. d) As described above, in the present embodiment, the image captured by the camera is converted into a bird's-eye view image, the coincidence region between the temporally continuous bird's-eye view images is extracted, and the moving region which is a shift portion between the bird's-eye view images is extracted. Is detected.

When the moving area is combined with the bird's-eye view image showing the current image to create a combined image, and when the combined image is displayed on the screen of the monitor 5, the image of the own vehicle and the viewing angle are added. I have. That is, in the present embodiment, when the vehicle moves, the region outside the camera's field of view is estimated and synthesized, so that it is possible to display a part that cannot be displayed normally outside the camera's field of view. The positional relationship between the car and the parking frame can be clearly understood. for that reason,
There is a remarkable effect that the operation of parking at the back becomes extremely easy. (Embodiment 2) Next, Embodiment 2 will be described, but description of the same parts as in Embodiment 1 will be omitted.

In this embodiment, only the parking frame and the vehicle stop are extracted and drawn, and the background image is omitted. FIG.
As shown in the flowchart of FIG. 7, first, at step 200, it is determined whether or not the shift position is the reverse position. If the determination is affirmative, the process proceeds to step 210, while if the determination is negative, the process proceeds to step 280.

In step 280, the previously drawn image in the area outside the field of view of the camera is erased. On the other hand, in step 210, the image of the parking frame and the car stop is extracted from the image taken by the camera by binarization processing.

Here, it is assumed that the parking frame and the car stop are painted in white and can be easily separated from the black ground of asphalt. In the following step 220, the binarized image is converted into a bird's-eye view to generate a bird's-eye view image of a parking frame and a car stop. Here, the background is a single color.

In the following step 230, a coincident area is extracted from two consecutive bird's-eye view images. In the following step 240, a moving area that is out of the current field of view of the camera 1 is extracted.

In the following step 250, before drawing,
By shifting the part already drawn on the screen B only by the moving area (below the screen B), a drawing area is secured. In the following step 260, an image of the moving area is drawn in the drawing area secured in step 250 (a composite image is created).

In the following step 270, a V-shaped line indicating the own vehicle and the viewing angle is drawn, and the present process is ended once. In this embodiment, the same effects as those of the first embodiment can be obtained. However, in this embodiment, in particular, the entire captured image is not converted into a bird's-eye view and displayed. Only those that are necessary for driving, such as a car or a car stop, are displayed.

As a result, there is an advantage that the burden of the image processing is reduced, and the image display on the monitor 3 is easy to see, so that the operation of the vehicle is facilitated. It should be noted that the present invention is not limited to the above-described embodiment at all, and it goes without saying that the present invention can be implemented in various modes without departing from the present invention.

(1) For example, in the first and second embodiments, the vehicle peripheral image processing apparatus has been described. However, a recording medium storing means for executing processing by this apparatus is also within the scope of the present invention. For example, examples of the recording medium include various recording media such as an electronic control unit configured as a microcomputer, a microchip, a floppy disk, a hard disk, and an optical disk.

That is, there is no particular limitation as long as it stores means such as a program which can execute the processing of the above-described vehicle peripheral image processing apparatus. (2) In the first embodiment, the image of the parking frame or the parking lot is not simply displayed by transforming the coordinates of the photographed image, but also the parking frame serving as a mark of the operation of the vehicle from the image photographed by the camera. It is desirable to extract an image of a car or a car stop and to enhance the image by the color or shade of the parking frame or the car stop.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram illustrating a main configuration of a vehicle peripheral image processing device according to a first embodiment.

FIG. 2 is an explanatory diagram illustrating an electrical configuration of the vehicle peripheral image processing apparatus according to the first embodiment.

FIG. 3 is an explanatory diagram illustrating a procedure of a process performed by the vehicle peripheral image processing apparatus according to the first embodiment.

FIG. 4 is an explanatory diagram showing a positional relationship at the time of coordinate conversion by the vehicle peripheral image processing device of the first embodiment.

FIG. 5 is an explanatory diagram illustrating a procedure of a process performed by the vehicle peripheral image processing device according to the first embodiment;

FIG. 6 is an explanatory diagram illustrating a procedure for obtaining an image movement amount of the vehicle peripheral image processing device according to the first embodiment;

FIG. 7 is a flowchart illustrating a process of the vehicle peripheral image processing device according to the first embodiment.

FIG. 8 is a flowchart illustrating a process performed by the vehicle peripheral image processing device according to the second embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Camera 3 ... In-vehicle monitor (monitor) 5 ... Image processing unit 11 ... Coordinate conversion part 13 ... Matching part 15 ... Area estimation part 17 ... Drawing part

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B050 AA10 BA04 BA09 BA11 EA18 EA19 EA27 FA02 5B057 AA06 AA16 BA02 CA13 CB13 CD01 CE08 DA16 DC32

Claims (7)

[Claims] A photographing means for photographing an image around a vehicle;
For a vehicle having a display unit for displaying an image, a vehicle peripheral image processing device that processes an image captured by the imaging unit and displays the processed image on the display unit, wherein the image captured by the imaging unit is A bird's-eye view image generating means for generating a bird's-eye view image by converting the data into data of a ground plane coordinate system projected with the photographing means as a viewpoint; a first bird's-eye view image generated by the bird's-eye view image generating means; Comparing the obtained second bird's-eye view image with the second bird's-eye view image, and comparing the non-coincidence area determined by the area discriminating means with the second bird's-eye view image. And a display processing means for displaying the composite image on the display means. 2. An image photographed by the photographing means,
2. The image according to claim 1, wherein the image is a rear image of the vehicle.
A vehicle peripheral image processing apparatus according to claim 1. 3. The vehicle peripheral image processing device according to claim 1, wherein the bird's-eye view image generating means extracts a picture indicating a lane to generate a bird's-eye view image. 4. The vehicle peripheral image processing apparatus according to claim 1, wherein the first bird's-eye view image and the second bird's-eye view image are temporally continuous bird's-eye view images. 5. The vehicle periphery according to claim 1, wherein the display processing unit performs coordinate transformation for displaying the image data of the composite image on the display unit. Image processing device. 6. The vehicle peripheral image processing according to claim 1, wherein when displaying the composite image on the display unit, an image indicating the own vehicle is added and displayed. apparatus. 7. A recording medium storing means for executing processing by the vehicle peripheral image processing apparatus according to any one of claims 1 to 6.