JP2003143597A - Vehicle surrounding picture processor and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle surrounding picture processor and a recording medium, which enable suitable processing of the picture of the surrounding of a vehicle by using a vehicle signal and using techniques of picture processing. SOLUTION: In a step 110, it is discriminated whether the vehicle speed is lower than prescribed or not. In a step 120, data of a picture photographed by a camera 1 is inputted. In a step 130, a steering angle signal is inputted from a steering angle sensor 6. In a step 140, a yaw rate signal is inputted from a yaw rate sensor 7. In a step 150, an ultrasonic signal is inputted from a ultrasonic sensor 8. In a step 160, a bird's-eye view picture is generated. In a step 200, it is discriminated whether the vehicle is running straight or not. In a step 210, the curving center and the curving angle of the vehicle are obtained by the steering angle and the yaw rate. In a step 220, the bird's-eye view picture is rotated. In a step 280, a new second bird's-eye view picture and the moved bird's-eye view picture are composited to generate a composite bird's-eye view picture.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle periphery image processing apparatus and a recording medium capable of photographing the periphery of a vehicle and displaying it on a monitor.

[0002]

2. Description of the Related Art Conventionally, as an apparatus for displaying the state of the rear of a vehicle, an image of a camera attached to the rear of the vehicle is
2. Description of the Related Art A display device is known which outputs the information to a monitor as it is. With this display, you can see how far behind the vehicle is,
There is a problem that it is difficult to understand the relative positional relationship between the image of the subject vehicle (for example, a parking frame) displayed on the monitor.

Separately from this, Japanese Patent Laid-Open No. 10-111
Japanese Patent No. 849 proposes a technique of converting an image captured by a rear camera (rear camera image) into a bird's-eye view and displaying the own vehicle therein.

[0004]

In the case of this technique,
Since the own vehicle is shown in the bird's-eye view, the positional relationship between the target object and the own vehicle displayed in the image is easier to understand than the one in which the captured image is displayed on the monitor as it is, but outside the current visual field of the camera. There was another problem that I couldn't display what was in.

Therefore, for example, when the vehicle is retracted and put in the parking frame, the parking frame outside the field of view of the camera cannot be displayed, so that it is still not easy to put the vehicle in the parking frame. was there. On the other hand, the inventors of the present application have already taken images of the rear of the vehicle from a camera attached to the rear of the vehicle, converted these into bird's-eye-view images viewed from above by image processing, and temporal changes in the converted images. There is a technology to display a wider range than before by calculating the movement of the vehicle from, calculating the image area that disappears outside the field of view of the camera, and pasting the converted image at the current time (creating a composite image). Being developed.

With this new technique, it is possible to detect the movement of the vehicle in the curve from the motion vector obtained by image processing and use it for the bird's-eye view image. However, if the image pattern on the road surface is small, an error occurs. Could be
In addition, there is a problem that the amount of calculation for calculating the motion vector increases.

As a countermeasure against this, it is conceivable to use a vehicle signal such as a yaw rate signal or a vehicle speed signal without using image processing. However, there is a vehicle that cannot obtain an extremely low speed vehicle speed signal depending on the type of vehicle, which is not always sufficient. Absent. In addition to this, when the above-described image processing is started at the timing of the back gear, for example, only the camera image from the start time can be obtained, so that the amount of image data displayed on the monitor is small. There was also.

The present invention has been made in order to solve the above-mentioned problems, and it is possible to preferably process an image around the vehicle by using a vehicle signal or an image processing technique. An object is to provide a processing device and a recording medium.

[0009]

[Means for Solving the Problems and Effects of the Invention] (1) The invention of claim 1 includes a photographing means (for example, a camera) for photographing an image of the surroundings of a vehicle and a display means (for example, a monitor) for displaying the image. The present invention relates to a vehicle peripheral image processing device that processes an image captured by an image capturing unit and displays the image on a display unit of a provided vehicle.

In the present invention, an image photographed by the photographing means is converted into data of the ground plane coordinate system projected from the photographing means, for example, to sequentially create image data of bird's eye view images. As a result, the image is not a distorted image as it was captured by a camera, but a bird's-eye view image that is easy to see as seen from above the ground.

Then, the first bird's-eye view image (such as the bird's-eye view image obtained last time or the previous synthetic bird's-eye view image) is moved in accordance with the moving state of the vehicle to create image data of the moved bird's-eye view image. The image data of the post-movement bird's-eye view image and the image data of the new second bird's-eye view image (from the first bird's-eye view image) are combined to create the image data of the combined bird's-eye view image.

In other words, as the vehicle moves, for example, the field of view of the camera changes. Therefore, naturally, the first bird's-eye view image and the second bird's-eye view image having different shooting times have image shifts (differences between the shot parts). Occur. Therefore, the second bird's-eye view image disappears from the camera's field of view on the monitor or the like by appropriately adding the moved bird's-eye view image of the shifted portion (that is, the part that was in the field of view in the past but disappears from the current field of view). The part that is displayed is displayed.

In particular, in the present invention, when the vehicle is at a low speed equal to or lower than a predetermined speed, the movement of the vehicle is based on other vehicle signals other than the vehicle speed signal or image data between bird's-eye view images corresponding to different photographing times. Since the state is estimated and the bird's-eye view image after movement is created, an accurate combined bird's-eye view image can always be created.

That is, as described above, there is a vehicle in which an extremely low speed vehicle speed signal cannot be obtained (or a vehicle in which the accuracy of the vehicle speed signal decreases at a low speed) depending on the vehicle type. Therefore, in the present invention, when the vehicle speed is low, Calculates the amount of movement of the vehicle by using other vehicle signals and image processing technology, without using the vehicle speed signal,
After moving, a bird's-eye view image is created. As a result, an accurate synthetic bird's-eye view image can be created and displayed on a monitor or the like even at low speeds.

As a method of creating a post-movement bird's-eye view image without using a vehicle speed signal, as will be described later, the movement of the vehicle is calculated from the temporal change of the image to create the post-movement bird's-eye view image, or the steering wheel. A method of creating a post-movement bird's-eye view image using vehicle signals such as corner and yaw rate can be adopted.

(2) According to the second aspect of the present invention, the moving state of the vehicle is estimated by determining whether the vehicle is in the straight state or the rotating state. When the vehicle is in a straight traveling state or a rotating state, as will be described later,
Since the method of estimating the moving state of the vehicle is slightly different, the moving state of the vehicle is distinguished here between straight traveling and rotation.

(3) In the invention of claim 3, when it is determined that the vehicle is in a straight traveling state, the moving state of the vehicle is estimated based on the image data between the bird's-eye view images corresponding to different photographing times. To create a bird's-eye view image after movement. When the vehicle is in a straight-ahead state, the bird's-eye view images corresponding to different shooting times also move in a fixed direction, so if the moving state of both bird's-eye view images is obtained by image processing (for example, matching processing of both images), the vehicle It is possible to obtain the movement state of the. Therefore, the post-movement bird's-eye view image can be created based on the movement state of the vehicle.

(4) According to the invention of claim 4, when it is determined that the vehicle is in a straight traveling state, the moving state of the vehicle is estimated from data of distances between the vehicle and surrounding objects at different times. To create a bird's-eye view image after movement. When the vehicle is in a straight traveling state, a change in the distance from a surrounding fixed object (for example, another vehicle or a wall) detected by the ultrasonic sensor is the moving distance of the vehicle. Therefore, the post-movement bird's-eye view image can be created based on this movement distance.

(5) In the invention of claim 5, when it is determined that the vehicle is in a rotating state, the moving state of the vehicle is estimated based on the steering wheel angle and the yaw rate, and a post-movement bird's eye view image is created. To do. If the vehicle is spinning,
If you know the steering wheel angle and yaw rate, you can see how much the vehicle has rotated. Therefore, the moved bird's-eye view image can be created by rotationally moving the bird's-eye view image based on the rotational movement amount of the vehicle.

(6) According to the invention of claim 6, a vehicle equipped with a photographing means for photographing an image of the surroundings of the vehicle and a display means for displaying the image is processed by the image photographed by the photographing means. The present invention relates to a vehicle peripheral image processing device displayed on display means. Then, the first bird's-eye view image created based on the image taken by the imaging means is stored, and the first bird's-eye view image is moved according to the moving state of the vehicle to create the moved bird's-eye view image, and the first bird's-eye view image. The second bird's-eye view image photographed and created later and the moved bird's-eye view image are combined to create a combined bird's-eye view image.

Particularly in the present invention, when it is difficult to estimate the moving state of the vehicle based on the image data between the bird's-eye view images corresponding to different photographing times, the distance between the vehicle and the surrounding objects is detected. The moving state of the vehicle is estimated based on the signal from the distance detecting means, and a post-movement bird's-eye view image is created.

That is, for example, when another vehicle or the like is behind and cannot take an image of the road surface (thus, when the moving state is not known from the image of the road surface), for example, the distance to the fixed object detected by the ultrasonic sensor Since the change is considered to be the travel distance of the vehicle, the post-movement bird's-eye view image can be created based on this travel distance. Therefore, an accurate combined bird's-eye view image can be displayed on the monitor or the like.

(7) The invention of claim 7 exemplifies that the distance between the vehicle and the surrounding objects is detected by an ultrasonic sensor. (8) According to the invention of claim 8, for a vehicle provided with a photographing means for photographing an image of the surroundings of the vehicle and a display means for displaying the image, the image photographed by the photographing means is processed and displayed on the display means. The present invention relates to a vehicle peripheral image processing device for displaying. Then, the first bird's-eye view image created based on the image taken by the imaging means is stored, the first bird's-eye view image is moved in accordance with the moving state of the vehicle, and the post-movement bird's-eye view image is created to create the first bird's-eye view image. The second bird's-eye view image taken after the image and created and the moved bird's-eye view image are combined to create a combined bird's-eye view image.

In particular, in the present invention, at least the bird's-eye view image creating means, when the vehicle is at a low speed equal to or lower than a predetermined speed,
After the movement, the processes of the bird's-eye view image creating means and the composite bird's-eye view image creating means are started. In other words, as described above, when the image processing is started at the timing of the back gear, for example, only the camera image from the start time is obtained, so in the present invention, the low speed that may cause the vehicle to stop is low. If, the image processing is started. With this, for example, when starting parking in the back, it is possible to display an image of the surroundings of the vehicle at a time before that,
It is very convenient for parking operation.

(9) In the invention of claim 9, as the first bird's-eye view image, the combined bird's-eye view image created before the current combined bird's-eye view image is used. As a result, a continuous composite bird's-eye view image to which the bird's-eye view images (images moved according to the movement of the vehicle) of images taken in the past are sequentially added can be displayed on the monitor.

(10) According to the tenth aspect of the invention, the combined bird's-eye view image is displayed in a region other than the display area for displaying the second bird's-eye view image.
It is an image to which a bird's-eye view image after movement is added. The present invention exemplifies a method of forming a synthetic bird's-eye view image. Thereby, images around the vehicle can be continuously displayed.

That is, for example, the image currently taken by the camera can be displayed on the monitor as the second bird's-eye view image. Therefore, an image disappearing from the monitor before that is used as the second bird's-eye view image after the movement. By adding to the area other than the display area, it is possible to display the past surrounding situation that has disappeared from the field of view of the camera on the monitor.

The latest bird's-eye view image is usually used as the second bird's-eye view image used for the composite bird's-eye view image. (11) According to the eleventh aspect of the invention, when the combined bird's-eye view image is displayed on the display means, the image showing the own vehicle is additionally displayed.

In the present invention, by displaying the combined bird's-eye view image showing the situation around the vehicle with the image showing the own vehicle, the positional relationship between the own vehicle and, for example, the parking frame becomes clear, so that the operation of the vehicle can be performed. Will be easier. Note that, for example, since the camera is fixed at a predetermined position on the vehicle, the position of the vehicle on the monitor screen is always constant. Therefore, the own vehicle can be displayed at a predetermined position on the screen of the monitor, and the vehicle may be described in advance on the screen of the monitor by printing or the like.

Since the viewing angle of the camera is usually constant, the viewing angle (left and right range) of the camera may be displayed not only on the host vehicle but also on the monitor screen. (12) The invention of claim 12 is a recording medium storing means for executing the process by the vehicle peripheral image processing device described above.

That is, there is no particular limitation as long as it stores a means such as a program that can execute the processing of the vehicle peripheral image processing apparatus described above. Examples of the recording medium include various recording media such as an electronic control device configured as a microcomputer, a microchip, a floppy disk, a hard disk, and an optical disk.

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, an example (embodiment) of an embodiment of a vehicle peripheral image processing apparatus and a recording medium of the present invention will be described. (Example) a) The basic system configuration of the present example will be described with reference to FIGS. 1 and 2.

As shown in FIG. 1, the vehicle peripheral image processing apparatus of this embodiment has a camera (for example, a CCD camera) 1 arranged at the rear part of an automobile and an in-vehicle monitor (for example, a liquid crystal display) arranged on a dashboard. 3, a vehicle speed sensor 5 for detecting a vehicle speed, a steering wheel angle sensor 6 for detecting a steering wheel angle, and a yaw rate sensor 7 for detecting a yaw rate.
And an ultrasonic sensor 8 for detecting obstacles around the vehicle,
An image processing unit 9 for performing image processing is provided.

As shown in FIG. 2, the image processing unit 9 is an electronic device mainly including a microcomputer for processing image data. The image processing unit 9 performs coordinate conversion of image data taken by the camera 1 to obtain a bird's eye view image. A coordinate conversion unit (bird's-eye view conversion circuit) 11 for creating the image, an image memory A for temporarily storing image data of the created bird's-eye view image, and a bird's-eye view image in the image memory A are moved (for example, rotationally moved) to be combined. CPU 13 for performing calculations such as image processing and image memory B for storing bird's-eye view image data displayed on monitor 3.

The image processing unit 9 has a vehicle speed signal input circuit 15 for inputting a steering wheel angle signal, a vehicle speed signal input circuit 17 for inputting a vehicle speed signal, a yaw rate signal input circuit 19 for inputting a yaw rate signal, and an ultrasonic wave. An ultrasonic signal input circuit 21 for inputting an ultrasonic signal from the sensor 8 is connected, and each signal is input to the CPU 13.

Here, the image data of the bird's-eye view image stored in the image memory B is the image data of the synthetic bird's-eye view image after the second processing, as described later. In other words, a bird's-eye view image (post-movement bird's-eye view image) obtained by moving the bird's-eye view image (first bird's-eye view image) captured by the camera 1 last time (for example, at time T) and created based on the image data,
Newly (for example, at time T + 1) taken by camera 1,
Bird's-eye view image created based on the image data (second
And a bird's-eye view image).

The image processing unit 3 may have a structure integrated with the camera 1, and each of the components in the image processing unit 9 may be partially or wholly integrated into an LSI. Yes. b) Next, referring to FIG.
~ It demonstrates based on FIG.

Here, the case where the vehicle is moved backward (back) and put in the parking frame is taken as an example. First, a procedure for converting an image input from the camera 1 into a bird's-eye view image viewed from the sky by the bird's-eye view conversion circuit 11 and storing the image in the image memory A will be described. First, the original image (output image) output from the camera 1 is shown in FIG.
This original image is taken by the camera 1 placed at the upper rear part of the vehicle.
Is an image of the parking frame drawn on the ground and its surroundings, the originally rectangular parking frame is distorted depending on the distance between the vehicle (and therefore camera 1) and the position of each line of the parking frame. Is displayed.

Therefore, as described later in detail, the image data of the original image is subjected to coordinate conversion to create a bird's-eye view image without distortion as shown in FIG. 3B, and the image data of this bird's-eye view image is converted into image data. It is stored in the image memory A. Next, a procedure for converting an original image captured by the camera 1 into a bird's-eye view image will be described. It should be noted that, as will be described below, this procedure is performed by a conventional technique (for example, Japanese Unexamined Patent Publication No. 10-21184).
No. 9) can be used.

Here, an image on the ground plane (for example, a parking frame) is obtained by performing the reverse processing of the normal perspective transformation.
Position is obtained as a bird's-eye view. Specifically, as shown in FIG. 4, perspective transformation is performed when the position data of the image on the ground is projected on the screen plane T at the focal length f from the camera position R.

Specifically, the camera 1 has a point R on the Z axis.
It is located at (0,0, H) and has a top-down angle τ with a ground surface (xy
It is assumed that the image on the coordinate plane) is being monitored. Therefore, here, the two-dimensional coordinates (α, β) on the screen plane T can be converted into the coordinates (bird's-eye view coordinates) on the ground plane (inverse perspective conversion) as shown in the following formula (1).

[0042]

[Equation 1]

That is, by using the above equation (1), the projected image data can be converted into image data corresponding to the screen of the monitor 3 (showing a bird's-eye view image) and displayed on the monitor. Next, a procedure of image processing when the vehicle is on a curve will be described.

Here, image processing is performed using the steering wheel angle and the yaw rate. Handle angle θh [°] and turning radius X
C [m] is inversely proportional, and conversion coefficient K [m · °]
Can be calculated by the following formula (2). In addition, when the steering wheel angle θh [°] is 0 ° when traveling straight, XC [m] becomes infinite. XC [m] = K [m. °] / tan (θh) [°] (2) As shown in FIG. 5, the yaw rate obtained from the yaw rate sensor 7 is the rotation angle θ [°], The center of rotation of the host vehicle during reverse travel is on the rear wheel axle (distance YC [m] from camera 1)
Therefore, the vehicle has moved by the rotation angle θ [°] when the rotation center position in the curve is (XC, YC).

Then, using this rotation center position and rotation angle, the image in the image memory B (the previous synthetic bird's-eye view image) is moved. Note that the image data of the bird's-eye view image similar to the image memory A is stored in the image memory B only for the first time when the composition is not performed yet.

That is, the image data of the previous combined bird's-eye view image stored in the image memory B is transferred to the internal memory of the CPU 13, and the bird's-eye view image is formed using the rotation center position (XC, YC) and the rotation angle θ. Move to rotate. A conversion formula for performing this rotational movement is shown in the following formula (3). The rotation angle θ is determined by the internal processing of the CPU 13 using this conversion formula.
For the coordinates (XB, YB) after movement,
The corresponding point (XA, YA) before movement is calculated, and (XA, YA)
The pixel value of (YA) is stored in the address memory of (XB, YB). XA = XB · COSθ + YB · SINθ (3) YA = YB · COSθ−XB · SINθ However, (XA, YA): coordinates before movement (XB, YB): coordinates after movement And for all coordinates By performing these calculations, the bird's-eye view image after movement (the bird's-eye view image after movement at time T) is created from the bird's-eye view image (composite bird's-eye view image at time T) stored in the image memory B.

At the same time, the bird's-eye view image created based on the image newly captured by the camera 1, that is, the current bird's-eye view image (the second bird's-eye view image at time T + 1),
It is newly transferred from the image memory A to the CPU 13. Therefore, the CPU 13 writes the second bird's-eye view image at a position having a positional relationship with the time T (a position where the image does not deviate from the post-movement bird's-eye view image), whereby the post-movement bird's-eye view image and the second bird's-eye view image are written. The image data of the synthetic bird's-eye view image is synthesized.

The composite bird's-eye view image is stored in the image memory B, and the composite bird's-eye view image is displayed on the monitor 3. When creating the synthetic bird's-eye view image, the image of the position of the vehicle or the viewing angle of the camera 1 is also added and stored in the image memory B, and the synthetic bird's-eye view image is displayed on the screen of the monitor 3. However, the position of the vehicle and the image of the viewing angle of the camera 1 are not stored in the image memory B, but the monitor 3
When the synthetic bird's-eye view image stored in the image memory B is drawn, the image of the position of the vehicle and the view angle of the camera 1 may be added and drawn.

Next, the procedure of processing when the vehicle is traveling straight ahead will be described. When the vehicle is traveling at a steering wheel angle of 0 °, the vehicle is moving forward or backward, and the vehicle moves linearly. When the vehicle speed is extremely low, for example, 5 km / h or less, the vehicle speed signal may not be input or may be inaccurate even if the vehicle speed signal is input, depending on the vehicle type (the vehicle speed sensor 5).

Therefore, here, when the vehicle speed is low, the moving amount of the vehicle is obtained from the motion vector by image processing without using the vehicle speed signal. For example, when the bird's-eye view image at time T is changed to the bird's-eye view image at time T + 1, both bird's-eye view images are matched to obtain the magnitude V of the motion vector of the image.

Here, the matching method will be described, but various conventional methods in image processing can be adopted.
For example, a predetermined (matching) area of images A and B is
As shown in FIG. 6, the image A and the image B are divided into a plurality of fine regions (pixels), and a pixel having the highest degree of coincidence between the pixels of the images A and B is detected. The positional relationship (i.e., how much the image A has moved) can be obtained.

Specifically, as shown in FIG. 6, considering the case where the degree of brightness (or color) of a figure in each pixel is represented by a numeral, here, the pixels in the lower three columns of the image A are considered. And the brightness of the pixels in the top three columns of image B match. Therefore, in this case, the image A is
It can be regarded as coincident with the image B by moving in the vertical direction of the figure by the width of one column of pixels. In this case, the figure in image B appears to have moved upward because the vehicle has moved downward in the figure.

Therefore, as shown in the following equation (4), the bird's-eye view image is moved in parallel by the magnitude V of the motion vector XA = XB (4) YA = YB-V where (XA, YA ): Coordinates before movement (XB, YB): Coordinates after movement And thereafter, similar to the above rotational movement, by performing these calculations for all coordinates, the bird's eye view image stored in the image memory B. From the above, a bird's-eye view image after the movement is created, and at the same time, a bird's-eye view image created based on the image newly captured by the camera 1 is newly transferred from the image memory A to the CPU 13, and these are combined to form a combined bird's-eye view image. Image data is created.

Then, the composite bird's-eye view image is stored in the image memory B, and the composite bird's-eye view image is displayed on the monitor 3. Next, a procedure of processing when there is an obstacle behind is described. If there is an obstacle behind, the movement amount of the host vehicle cannot be calculated using the motion vector of image processing. That is, since the motion vector requires the motion of the image of the road surface, the correct motion vector cannot be obtained when the road surface cannot be seen because another vehicle is parked behind.

In this case, the movement amount can be calculated from the displacement of the detection distance using the signal of the ultrasonic sensor 8. For example, the signal from the ultrasonic sensor 8 is C1 (m) last time,
If C2 (m) is reached this time, the amount of movement can be calculated by the following equation (5) (however, when traveling straight). XA = XB (5) YA = YB- (C1-C2) However, (XA, YA): Coordinates before movement (XB, YB): Coordinates after movement Accordingly, coordinates before and after movement Since the correspondence relationship of is understood, the post-movement bird's-eye view image can be obtained by moving the previous combined bird's-eye view image based on this correspondence relationship.

Next, the timing of starting the image processing will be described. The above-described processing from capturing an image by the camera 1 to creating a composite bird's-eye view image is started when the vehicle is at a predetermined low speed (for example, 10 km / h) or less, such as when parked.

As a result, when the vehicle stops, for example, to shift to the back gear, a composite bird's-eye view image of the history of forward movement just before that is made, so that compared to the case where the camera 1 is just taking a picture at this time, Many images around the vehicle can be displayed. c) Next, the processing performed by the image processing unit 9 and the like will be described with reference to the flowchart of FIG.

As shown in FIG. 7, first, at step 100, the vehicle speed signal from the vehicle speed sensor 7 is read. In the following step 110, the vehicle speed is a predetermined low speed (for example, 10 km /
h) It is determined whether or not the following. If an affirmative judgment is made here, the routine proceeds to step 120, while if a negative judgment is made, the routine returns to step 100.

At step 120, the data of the image taken by the camera 1 is input. In the following step 130, a steering wheel angle signal is input from the steering wheel angle sensor 6. In the following step 140, the yaw rate signal is input from the yaw rate sensor 7.

In the following step 150, the ultrasonic sensor 8
Input the ultrasonic signal from. In the following step 160,
Coordinate conversion of images taken by the camera 1 (the bird's-eye view conversion)
To create a bird's eye view image. Continued Step 170
Then, the bird's-eye view image is stored in the image memory A.

In the following step 180, it is determined whether or not the current process is the first process. If an affirmative judgment is made here, the routine proceeds to step 190, while if a negative judgment is made, the routine proceeds to step 2
Go to 00. In step 190, since it is the first process, the image data stored in the image memory A is also stored in the image memory B, and the process returns to step 100. The bird's-eye view image stored in the image memory B is displayed on the monitor 3 as it is (see FIG. 8A). On the other hand, step 200
Since this is not the first time processing, it is determined whether the vehicle is in a straight traveling state by whether the steering wheel angle is 0 °. If it is determined that the vehicle is traveling straight ahead, the process proceeds to step 230,
On the other hand, if it is determined to be in the rotating state, the process proceeds to step 210.

In step 210, since the vehicle is in a rotating state, the image processing is performed according to the procedure described above. Specifically, the rotation center and the rotation angle of the vehicle are obtained from the steering wheel angle and the yaw rate. Continued Step 220
Then, the bird's-eye view image is rotated based on the equation (4) using the rotation center and the rotation angle obtained in step 210. That is, the bird's-eye view image stored in the image memory B (the composite bird's-eye view image when the previous combined bird's-eye view image is stored) is rotated to create the moved bird's-eye view image.

In the following step 280, the second bird's-eye view image (at time T + 1) newly captured and stored in the image memory A is read from the image memory A, and the second bird's-eye view image and the moved bird's-eye view image are combined. Then, a composite bird's-eye view image is created. Specifically, a part outside the new camera view (corresponding to the second bird's-eye view image) outside the camera view due to the movement of the vehicle this time, that is, a part of the moved bird's-eye view image (see the second view in detail). A composite bird's-eye view image is created by writing (excluding the portion overlapping the bird's-eye view image).

In the following step 290, the composite bird's-eye view image is stored in the image memory B. In the following step 300, the combined bird's-eye view image stored in the image memory B is added to the monitor 3 in consideration of the image showing the vehicle, the image showing the viewing angle, and the like.
Is displayed (see FIG. 8B), and this processing is once terminated.

On the other hand, in step 230 where the affirmative determination is made in step 200, the vehicle is in a straight traveling state, so the distance from the obstacle is 1.5 m or less based on the signal from the ultrasonic sensor 8. To determine. If a negative decision is made here, the routine proceeds to step 240, while if an affirmative decision is made, the routine proceeds to step 250.

At step 240, since there is no obstacle near the rear of the vehicle, the process of obtaining the motion vector from the image data of the image memory A and the image memory B is performed. In the following step 260, the amount of movement of the image (previous synthetic bird's-eye view image) is obtained from the size of the motion vector.

In the following step 270, the image (previous synthetic bird's-eye view image) is translated in parallel by the amount of movement to create a post-movement bird's-eye view image.
In step 0, a new synthetic bird's-eye view image is displayed on the monitor 3 (see FIG. 8C), and this process ends.

On the other hand, in step 250, when the affirmative decision is made in step 230, the distance to the obstacle is 1.5.
Since it is close to m or less, that is, the movement amount cannot be obtained from the bird's-eye view image of the road surface behind the vehicle, the movement amount of the vehicle is obtained from the displacement amount of the obstacle detection distance by the ultrasonic sensor 8.

In the following step 260, the moving amount of the vehicle is determined to have the opposite direction to the moving amount of the image (previous synthetic bird's-eye view image), and the moving amount of the image is determined.
Then, the process proceeds to the process of 00, a new synthetic bird's-eye view image is displayed on the monitor 3, and this process is once terminated.

Thereafter, the same processing is carried out each time an image newly taken by the camera 1 is captured. That is, the synthetic bird's-eye view image is sequentially created and stored, and the synthetic bird's-eye view image is moved in accordance with the movement of the vehicle to create the moved bird's-eye view image, and the moved bird's-eye view image and the new second bird's-eye view image are stored. Is repeated and displayed on the monitor 3 as a new combined bird's-eye view image. As a result, a continuous image as shown in FIG. 8 is displayed on the monitor 3.

D) As described above, in this embodiment, when the vehicle is at a low speed below a predetermined speed (for example, 10 km / h), other vehicle signals (steering wheel angle and yaw rate) other than the vehicle speed signal, The moving state of the vehicle is estimated based on the image data between the bird's-eye view images corresponding to different shooting times, and the bird's-eye view image after movement is created. Even in the case of the following vehicle types, it is possible to always create a highly accurate synthetic bird's-eye view image and display it on the monitor 3.

Further, in the present embodiment, for example, when there is another vehicle or the like behind and it is not possible to take an image of the road surface (hence, the moving state cannot be known from the image of the road surface), the obstacle detected by the ultrasonic sensor 8 is detected. By considering the change in the distance to the object as the moving distance of the vehicle, the post-moving bird's-eye view image can be created based on this moving distance. Therefore, even in such a case, an accurate combined bird's-eye view image can be displayed on the monitor 3.

Furthermore, in the present embodiment, the timing of the start of the image processing for creating the composite bird's-eye view image and the like is set to a low speed (for example, 10 km / h) where there is a possibility that the vehicle will stop. When parking is started with, the image of the surroundings of the vehicle at a time before that can be displayed, which is very convenient for parking operation.

It is needless to say that the present invention is not limited to the above-mentioned embodiments and can be carried out in various modes without departing from the present invention. (1) For example, the vehicle peripheral image processing apparatus has been described in the above embodiments, but a recording medium storing means for executing processing by this apparatus is also within the scope of the present invention.

(2) Examples of the recording medium include various recording media such as an electronic control unit configured as a microcomputer, a microchip, a floppy disk, a hard disk, and an optical disk. That is, as long as it stores a means such as a program that can execute the processing of the vehicle peripheral image processing device described above,
There is no particular limitation.

[Brief description of drawings]

FIG. 1 is an explanatory diagram illustrating a main configuration of a vehicle peripheral image processing device according to an embodiment.

FIG. 2 is an explanatory diagram showing an electrical configuration of a vehicle peripheral image processing device of the embodiment.

3A and 3B show images used by the vehicle peripheral image processing apparatus of the embodiment, FIG. 3A is an explanatory view of an image captured by a camera, and FIG. 3B is an explanatory view showing a bird's-eye view image.

FIG. 4 is an explanatory diagram showing a positional relationship at the time of coordinate conversion by the vehicle peripheral image processing device of the embodiment.

FIG. 5 is an explanatory diagram showing a method for obtaining a movement amount of the vehicle during rotation by the vehicle periphery image processing apparatus according to the embodiment.

FIG. 6 is an explanatory diagram showing a method for obtaining a movement amount of a vehicle when the vehicle is traveling straight ahead by the vehicle periphery image processing apparatus according to the embodiment.

FIG. 7 is a flowchart showing the processing of the vehicle peripheral image processing device of the embodiment.

FIG. 8 shows a monitor image displayed by the vehicle periphery image processing apparatus according to the embodiment.

[Explanation of symbols]

1 ... Camera 3 ... In-vehicle monitor (monitor) 5 ... Vehicle speed sensor 6 ... Handle angle sensor 7 ... Yaw rate sensor 8 ... Ultrasonic sensor 9 ... Image processing unit

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hirohiko Yanagawa             1-1, Showa-cho, Kariya city, Aichi stock market             Inside the company DENSO F-term (reference) 3D020 BA05 BC03 BE03                 5C054 FD01 FE12 FE19 GA01 GA02                       GA04 GB01 GB15 HA30

Claims (12)

[Claims] 1. A photographing means for photographing an image of a periphery of a vehicle,
A vehicle peripheral image processing device for processing an image captured by the image capturing unit and displaying the image on the display unit for a vehicle including a display unit displaying an image, based on the image captured by the image capturing unit. A bird's-eye view image storage unit for storing the first bird's-eye view image created by the above; and a moved bird's-eye view image creation unit for moving the first bird's-eye view image according to the moving state of the vehicle to create a moved bird's-eye view image. , A second image created after the first bird's-eye view image
A bird's-eye view image and a post-moving bird's-eye view image are combined to create a combined bird's-eye view image, and a combined bird's-eye view image creating unit is provided, and when the vehicle is at a low speed equal to or lower than a predetermined speed, other than a vehicle speed signal. A vehicle periphery image processing apparatus, which estimates the moving state of the vehicle based on another vehicle signal or image data between bird's-eye view images corresponding to different photographing times to create the moved bird's-eye view image. . 2. The vehicle peripheral image processing apparatus according to claim 1, wherein the moving state of the vehicle is estimated by determining whether the vehicle is in a straight state or a rotating state. 3. When it is determined that the vehicle is in a straight traveling state, the moving state of the vehicle is estimated based on image data between bird's-eye view images corresponding to different photographing times,
The vehicle peripheral image processing device according to claim 1 or 2, wherein the post-movement bird's-eye view image is created. 4. When it is determined that the vehicle is in a straight traveling state, a moving state of the vehicle is estimated from data of distances between the vehicle and surrounding objects at different times,
The vehicle peripheral image processing device according to claim 1 or 2, wherein the post-movement bird's-eye view image is created. 5. When the vehicle is determined to be in a rotating state, the moving state of the vehicle is estimated based on the steering wheel angle and the yaw rate, and the post-moving bird's-eye view image is created. The vehicle peripheral image processing device according to claim 1 or 2, wherein 6. A photographing means for photographing an image around the vehicle,
A vehicle peripheral image processing device for processing an image captured by the image capturing unit and displaying the image on the display unit for a vehicle including a display unit displaying an image, based on the image captured by the image capturing unit. A bird's-eye view image storage unit for storing the first bird's-eye view image created by the above; and a moved bird's-eye view image creation unit for moving the first bird's-eye view image according to the moving state of the vehicle to create a moved bird's-eye view image. , A second image created after the first bird's-eye view image
The bird's-eye view image and the moved bird's-eye view image are combined to create a composite bird's-eye view image creating means, and the vehicle is based on image data between the bird's-eye view images corresponding to different shooting times. When it is difficult to estimate the moving state of the vehicle, the moving state of the vehicle is estimated based on a signal from a distance detecting unit that detects the distance between the vehicle and an object around the moving bird's-eye view image. An image processing apparatus for a vehicle periphery, which is characterized in that: 7. The ultrasonic sensor detects a distance between the vehicle and a surrounding object.
Alternatively, the vehicle peripheral image processing device according to Item 6. 8. A photographing means for photographing an image around the vehicle,
In a vehicle peripheral image processing device for processing an image captured by the image capturing unit and displaying the image on the display unit for a vehicle including a display unit displaying an image, based on the image captured by the image capturing unit. A bird's-eye view image storage unit that stores the first bird's-eye view image created by the above; , A second image created after the first bird's-eye view image
A bird's-eye view image and a post-moving bird's-eye view image are combined to create a combined bird's-eye view image, and a composite bird's-eye view image creating unit is provided. A vehicle peripheral image processing device, characterized in that the processing of the image creating means, the moved bird's-eye view image creating means, and the combined bird's-eye view image creating means is started. 9. The vehicle peripheral image processing device according to claim 1, wherein a synthetic bird's-eye view image created before the current synthetic bird's-eye view image is used as the first bird's-eye view image. 10. The composite bird's-eye view image is an image to which the post-movement bird's-eye view image is added, in addition to the display area for displaying the second bird's-eye view image.
10. The vehicle peripheral image processing device according to any one of 9 to 10. 11. The vehicle peripheral image according to claim 1, wherein when the composite bird's-eye view image is displayed on the display means, an image showing the own vehicle is additionally displayed. Processing equipment. 12. A recording medium storing means for executing processing by the vehicle peripheral image processing device according to claim 1.