JP2003191810A - Vehicle surroundings monitoring system, and vehicle moving state detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To monitor and display a state outside the visual field of a monitoring camera loaded on a vehicle. <P>SOLUTION: A birds-eye view provided by converting an image photographed by a rear camera 13 to a plane image seen from the sky is temporarily stored in a storage memory 23. A CPU 25 calculates vehicle movement (rotation center position, rotation angle, and the like of the vehicle) based on a handle angle signal, a car speed signal, a yaw rate signal, and an ultrasonic sensor signal. The CPU 25 then reads, rotates, and moves the birds-eye image obtained from a history image storage memory 27 at a previously calculating timing based on the vehicle movement, overlays on the result a latest birds-eye image obtained at the present calculating timing to produce a synthetic birds-eye image, and stores this in the history image storage memory 27 and an image memory 29 for display. A monitor 31 displays the synthetic birds-eye image including a past history read out of the image memory 29. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle surroundings monitoring system, and more particularly, to a vehicle surroundings monitoring system for improving convenience and safety when moving a vehicle to a desired position. The present invention relates to a vehicle movement state detection device effective for realizing a system.

[0002]

2. Description of the Related Art Conventionally, a rear camera is used to photograph the rear of a vehicle, and an image captured by the rear camera is displayed in real time on a monitor mounted on a dashboard, so that it is convenient to park at the back. There is known a rear monitoring system that is improved.

Further, in the rear monitoring system disclosed in Japanese Patent Laid-Open No. 10-211849, an image of the rear of the vehicle taken by a rear camera is converted into a plan view (bird's-eye image) viewed from the sky, and this is used as a monitor. In addition to displaying it, the own vehicle is also displayed on this monitor.

According to the rear monitoring system disclosed in Japanese Unexamined Patent Publication No. 10-211849, the own vehicle and the state of the rear are displayed on the monitor as a bird's-eye view image, so that the positional relationship between the own vehicle and the parking frame can be easily understood. The effect is demonstrated.

[0005]

However, in these prior arts, since the situation outside the field of view of the rear camera cannot be displayed, the convenience when the vehicle is trying to put it in the parking frame with the back is not sufficiently improved.

Therefore, a first object of the present invention is to make it possible to display on the monitor even a situation outside the field of view of a surveillance camera mounted on a vehicle.

Further, the conventional vehicle surroundings monitoring system has been developed and put into practical use by emphasizing only the case of backing such as garage entry. However, in recent vehicles,
In order to improve the front view, many bonnet styles have a lower tip. For this reason, when trying to pass through a narrow space by moving forward, or when trying to park the vehicle forward in the parking lot, especially in the case of a vehicle with a right-hand drive, the situation in front of left cannot be grasped sufficiently,
You may experience the possibility of contacting the wall surface of a parking lot, an adjacent vehicle, or a structure on the road such as a utility pole, or the vehicle may be damaged.

Therefore, a second object of the present invention is to provide the user with correct information about the surrounding situation in front of the vehicle.

Further, when entering the garage, a turning operation may be involved. In such a case, with a conventional monitoring system, you can check the position of obstacles and parking frames behind you by looking at the back monitor when you are moving backward, but keep your eyes on the monitor when moving forward to turn back. , Looking forward through the windshield is repeated. Especially when there is an obstacle ahead, as mentioned above, the visibility is poor at the left front corner of the right-hand drive vehicle.
There is a problem that the switching operation may be difficult.

Therefore, a third object of the present invention is to provide the user with not only the rear situation of the vehicle but also the front situation of the vehicle. The fourth object is to provide the user with correct information by minimizing the blind spot in the positional relationship with.

[0011]

Means for Solving the Problems and Effects of the Invention
The vehicle periphery monitoring system according to claim 1, which has been made in order to achieve the purpose of,
The bird's-eye view conversion unit converts the bird's-eye view image, and the bird's-eye view image storage unit stores the bird's-eye view image. Further, based on the moving state of the vehicle acquired by the vehicle moving state acquiring means, the image rotating / moving means rotates or moves in parallel the past bird's-eye view image stored in the bird's-eye image storage means. Here, when the vehicle is turning, rotational movement is executed, and when the vehicle is linearly moving, parallel movement is executed. Then, the synthetic bird's-eye image generation means synthesizes the past bird's-eye image rotated and moved in this way and the current bird's-eye image obtained by subjecting the current vehicle rear image photographed by the photographing means to the bird's-eye conversion means. Generate a synthetic bird's-eye view image. Here, for example, it is preferable to overlay the past bird's-eye image so that the current bird's-eye image is in the front. Finally, the display control means displays the synthetic bird's-eye view image on the display means so that the relationship with the position of the vehicle can be understood. As a method of displaying so that the relationship with the position of the vehicle can be understood, for example,
An image of the plan view of the vehicle is displayed at a predetermined position of the display means so that it is in the foreground. The display position of the combined bird's-eye view image may be determined so that it is displayed backward from.

As a result, according to the vehicle periphery monitoring system of claim 1, not only the present bird's-eye view image but also the past bird's-eye view images are displayed on the display means so that the positional relationship with the vehicle can be understood. The situation outside the field of view of the photographing means can be displayed on the monitor, and the driver can be more accurately discriminated between the situation of the vehicle and the surroundings. Therefore,
According to the vehicle periphery monitoring system of the first aspect, the effect of improving convenience and operability when parking the vehicle in the back is exhibited.

The vehicle periphery monitoring system according to a second aspect of the present invention, which has been made to achieve the second object, has a photographing means installed in the front part of the vehicle. Then, the bird's-eye conversion unit converts the vehicle front image captured by the image capturing unit into a bird's-eye image. This bird's-eye view image is stored in the bird's-eye view image storage means. Further, the vehicle movement state acquisition means acquires the movement state of the vehicle. Then, based on the movement state of the vehicle acquired by the vehicle movement state acquisition means, the image rotation / moving means,
The past bird's-eye image stored in the bird's-eye image storage means is rotated or moved in parallel. Here, when the vehicle is turning, rotational movement is executed, and when the vehicle is linearly moving, parallel movement is executed. Then, the synthetic bird's-eye image generation means synthesizes the past bird's-eye image rotated and moved in this way and the current bird's-eye image obtained by subjecting the current vehicle rear image photographed by the photographing means to the bird's-eye conversion means. Generate a synthetic bird's-eye view image. Here, for example, it is preferable to overlay the past bird's-eye image so that the current bird's-eye image is in the front. Finally, the display control means displays the synthetic bird's-eye view image on the display means so that the relationship with the position of the vehicle can be understood. As a method of displaying so that the relationship with the position of the vehicle can be understood, for example, an image conceivable in a plan view of the vehicle is displayed at a predetermined position of the display means so as to be the forefront, and a predetermined image is displayed on the front end side of the vehicle. A reference line may be set and the display position of the combined bird's-eye image may be determined so that the current bird's-eye image in the combined bird's-eye image is displayed forward from this reference line.

As a result, according to the vehicle periphery monitoring system of the second aspect, it becomes possible to monitor and display the situation in front of the vehicle around the front corner of the vehicle. Therefore, for example, when trying to pass through a narrow place by moving forward, or when trying to park the vehicle forward in the parking lot, it is possible to sufficiently understand the situation in front of the left, which is likely to be a blind spot in the vehicle with the right steering wheel, It is possible to effectively prevent contact with a wall surface of a parking lot, an adjacent vehicle, or a structure on the road such as a utility pole.

The vehicle periphery monitoring system according to a third aspect of the present invention, which has been made to achieve the third object, is provided with photographing means at the front and the rear of the vehicle, respectively. And
The rear range photographing means is used when the shift range detecting means detects that the shift range is the reverse range, and the front photographing means is used when it is detected that the shift range is the forward range. Therefore, when the driver intends to park while performing the switching operation, an image from the rear photographing means is displayed on the display means when the shift is changed to the R range, and an image from the front photographing means is displayed on the display means when the shift is changed to the D range. Even if the line of sight is not significantly changed, it is possible to smoothly perform the final parking position adjustment particularly when parking.

Further, the vehicle periphery monitoring system according to the fourth aspect, which has been made to achieve the fourth object, exhibits the following action in addition to the action by the vehicle periphery monitoring system according to the third aspect. The rearward image captured by the rearward capturing unit is converted into a bird's-eye view image by the bird's-eye view conversion unit, and the bird's-eye-view converted rearward image is stored in the bird's-eye view image storage unit. Then, based on the movement state of the vehicle acquired by the vehicle movement state acquisition means, the past bird's-eye view image stored in the bird's-eye view image storage means is rotated or moved in parallel. Then, the synthetic bird's-eye image generation means synthesizes the rotated and moved past bird's-eye image and the current bird's-eye image to generate a synthetic bird's-eye image. This composite bird's-eye view image is displayed on the display means. Further, when the shift range is switched, this time the shooting is switched to the shooting by the front shooting means, and the same processing as described above is executed based on the image shot by the front shooting means.

As a result, according to the vehicle periphery monitoring system of the fourth aspect, when moving forward by turning back, the state of the front is displayed on the display means in a form reflecting the history from the beginning of the forward movement. To be done. Therefore, the vehicle can be moved forward while accurately determining the positional relationship between the obstacle in front and the own vehicle. At this time, it is possible to correctly transmit the situation of a region that is difficult to see with a right-hand drive vehicle, such as a left front corner of the vehicle. Similarly, when the vehicle moves backward in the reverse direction, the state of the rear is displayed on the display means in a form reflecting the history from the beginning of the backward movement. Therefore, the vehicle can be moved backward while accurately determining the positional relationship between the rear obstacles and the own vehicle. In this way, as a result of being able to display an appropriate peripheral situation on the display means, in the final phase of the switching operation, it becomes possible to operate the steering wheel without moving the line of sight from the display means to the windshield. The effect that the operability is greatly improved is exhibited.

The vehicle periphery monitoring system according to claim 5 is the vehicle periphery monitoring system according to any one of claims 1 to 4, wherein the past bird's-eye view images stored in the bird's-eye view image storage means are stored. Is provided with a past image updating means for updating based on the synthetic bird's-eye image generated by the synthetic bird's-eye image generating means.

According to the vehicle periphery monitoring system of the fifth aspect, since the past image updating means is provided, the past images are updated and accumulated in the bird's-eye image storage means.
After the vehicle starts to move backward or forward, the surrounding condition of the vehicle can be gradually displayed in a wider range, and not only the rearward or forward condition of the vehicle but also the lateral condition of the side of the vehicle can be transmitted to the driver. Further, by updating the past image, the past bird's-eye view image after being rotated / moved by the image rotation / moving means can be correctly joined to the current bird's-eye view image.

By the way, when the first to fourth objects can be achieved, there are the following problems.

First, since the past image is used to display the situation around the vehicle, there is a problem in that the situation around the vehicle at the present time cannot be completely provided to the user.

For example, if a vehicle is put in a garage or the like while gazing at only the monitor, even if another vehicle enters the area displaying the past image, the image of the other vehicle will not be displayed. , Careless operation of the handle may cause a contact accident. Therefore, in order to put the system developed so far into practical use, the present applicants, of course, do not rely solely on the monitor for driving, and in particular, be sure to visually check the lateral area of the vehicle. It is planned to clarify the points that need to be done in the operation manual etc. and call the user's attention. However, it cannot be said that the user does not perform a turning operation of the vehicle or the like without performing visual confirmation due to getting used to it.

Therefore, a fifth object of the present invention is to prevent a contact accident with a moving object even when the user does not faithfully execute the operation in accordance with the instruction of the operation manual. And

According to a sixth aspect of the invention, there is provided a vehicle surroundings monitoring system according to any one of the first to fifth aspects, wherein:
The composite bird's-eye image generated by the composite bird's-eye image generating means includes image processing means for performing image processing for distinguishing a portion corresponding to a past bird's-eye image and a portion corresponding to the current bird's-eye image, The display control means is configured to display on the display means the composite bird's-eye image after being image-processed by the image processing means.

According to the vehicle periphery monitoring system of the sixth aspect, the image processing means generates a portion corresponding to the past bird's-eye image and the current bird's-eye image with respect to the synthetic bird's-eye image generated by the synthetic bird's-eye image generating means. Image processing is performed to distinguish the corresponding portion. As an example of the image processing, for example, a portion where only the past bird's-eye view image is displayed can be color-distinguished from the current bird's-eye view image by filtering. It is also possible to explicitly indicate the boundary line between the past bird's-eye image and the current bird's-eye image. Further, by performing image processing such as reducing the gradation on the portion where only the past bird's-eye view image is displayed, it is possible to visually distinguish the current bird's-eye view image by the difference in gradation. In addition, various image processing techniques can be applied to bring the past bird's-eye view image and the current bird's-eye view image into a distinguishable state.

Further, the vehicle periphery monitoring system according to claim 7 which is also made to achieve the fifth object,
In the vehicle periphery monitoring system according to any one of claims 6 to 6, predetermined information for calling the driver's attention is displayed in a portion where a past bird's-eye view image is displayed in the combined bird's-eye view image displayed on the display means. It is characterized in that it is provided with a caution information display means for displaying.

According to the vehicle periphery monitoring system of the seventh aspect, the caution information display means calls the driver's attention to the portion where the past bird's-eye view image is displayed in the combined bird's-eye view image displayed on the display means. Predetermined information for that purpose, for example, a caution mark, a moving image display in which a person or a vehicle moves in and out of a display area of a past bird's-eye view image by animation, and a message for urging the driver's eyesight confirmation are displayed. As a result, the driver can easily determine which part in the monitor finally needs visual confirmation, and can improve safety. Note that not only visual attention but also voice attention can be used. Therefore,
As means for achieving the fifth object, as in the vehicle periphery monitoring system according to claim 8, in the vehicle periphery monitoring system according to any one of claims 1 to 7, the combination displayed on the display means. A part of the bird's-eye image displaying the past bird's-eye image may be provided with a notification means for notifying the driver that visual confirmation is necessary.

Further, as described above, when the vehicle is put in the garage or the like, in addition to simply backing up, there are cases in which forward and backward movements are repeated like a turning operation. For example, when the user switches the shift range to the R range, if the image history when moving forward by that time remains,
There is also a problem that the image history at the time of forward movement is combined as an image of an area outside the field of view of the camera, and a correct combined image cannot be obtained. Conversely, the same problem occurs when switching from the R range to the forward shift range.

Therefore, it is a sixth object of the present invention to be able to always display a composite image that correctly reflects the surrounding conditions of the vehicle regardless of whether the vehicle is moving forward or backward.

A vehicle surroundings monitoring system according to a ninth aspect is a vehicle surroundings monitoring system according to any one of the first to eighth aspects, wherein:
Shift range switching detection means for detecting switching of the shift range of the vehicle, and when the shift range switching detection means detects that the shift range has been switched, a past bird's-eye view image stored in the bird's-eye view image storage means And a past image erasing means for erasing.

According to the vehicle periphery monitoring system of the ninth aspect, when the shift range switching detecting means detects that the shift range has been switched, the past image erasing means is activated and stored in the bird's-eye view image storing means. Delete the past bird's-eye image that has been displayed. As a result, when the vehicle makes complicated movements due to turning back, the past image obtained when moving backward is displayed on the monitor when it is moving forward, while the past image obtained when moving forward is displayed when moving backward. It is possible to provide the driver with image information that accurately reflects the surrounding conditions of the vehicle.

In the vehicle periphery monitoring system according to a tenth aspect of the present invention, the vehicle movement state acquisition means includes a rotation movement amount calculation means for calculating the rotation movement amount of the vehicle at the time of turning from the steering wheel angle and the yaw rate. Is characterized by.

According to the vehicle periphery monitoring system of the tenth aspect, the rotational movement amount calculating means calculates the rotational movement amount of the vehicle at the time of turning from the steering wheel angle and the yaw rate, and the image rotation is performed based on the rotational movement amount. -The moving means rotates and moves the past bird's-eye view image. In this case, when the vehicle is backing up, it can be estimated that the center of rotation is on the rear wheel axle, so the center of rotation can be determined by calculating the radius of gyration from the steering wheel angle. Then, the rotation angle is calculated from the yaw rate. If the rotation center and the rotation angle are obtained, it is possible to mathematically calculate in which direction and how much the vehicle has rotationally moved with respect to the rotation center. If the amount of rotation movement can be calculated in this way, then, the pixels constituting the past bird's-eye view image are subjected to coordinate conversion in the direction opposite to the actual rotation direction based on this amount of rotation movement. Then, if a current bird's-eye view image is overlaid on this, a composite bird's-eye view image is generated for a coordinate system in which the rear wheel axle of the vehicle is the reference line in the X direction and the center line of the vehicle is the reference line in the Y direction. be able to.
It should be noted that a synthetic bird's-eye view image can be easily generated by a similar method even when moving forward.

By the way, in the moving state without rotation,
By calculating the linear movement amount of the vehicle based on the vehicle speed, moving the past bird's-eye view image in the direction opposite to the movement direction by this linear movement amount, and overlaying the current bird's-eye view image on this, it is easy to synthesize the bird's-eye image Can be generated.

This is realized by the vehicle surroundings monitoring system according to claim 11, and in the vehicle surroundings monitoring system according to any one of claims 1 to 10, the vehicle moving state acquisition means is a vehicle speed or It is characterized in that it is provided with a linear movement amount calculation means for calculating the linear movement amount of the vehicle when the vehicle does not turn, based on predetermined information capable of calculating the vehicle speed.

By the way, when a vehicle is about to be parked, the vehicle is in a state of moving at an extremely low speed. There is a problem in that the vehicle speed at such an extremely low speed may not be accurate if the detection signal of the normal vehicle speed sensor is used. Alternatively, some sensors cannot detect the vehicle speed when the speed is lower than a predetermined speed.

Therefore, a seventh object of the present invention is to make it possible to detect the vehicle speed or the moving amount of the vehicle at low speed with high accuracy.

A vehicle periphery monitoring system according to a twelfth aspect of the present invention, which is made to achieve the seventh object, is the vehicle periphery monitoring system according to any one of the first to eleventh aspects, wherein the vehicle movement state acquisition means is The present invention is characterized by comprising a linear movement amount calculation means for calculating the linear movement amount of the vehicle by comparing the past bird's-eye image and the current bird's-eye image.

The vehicle periphery monitoring system according to claim 13 which is also made to achieve the seventh object is the vehicle periphery monitoring system according to any one of claims 1 to 12, wherein the vehicle movement state acquisition means is , A linear movement amount calculating means for calculating a linear movement amount of the vehicle based on a change in the distance between the vehicle and any target detected by the sonar.

Similarly, the vehicle movement state detecting device according to claim 14 made to achieve the seventh object converts the photographing means installed in the vehicle and the image photographed by the photographing means into a bird's-eye view image. Bird's-eye view conversion means, and bird's-eye view image storage means for storing the bird's-eye view image converted by the bird's-eye view conversion means,
By comparing the past bird's-eye image stored in the bird's-eye image storage means with the current bird's-eye image obtained by bird's-eye conversion of the current vehicle rear image photographed by the photographing means by the bird's-eye conversion means, And a calculating means for calculating the moving amount or moving speed of the.

Similarly, a vehicle moving state detecting device according to a fifteenth aspect of the present invention, which is made to achieve the seventh object, is installed in a vehicle and detects a sonar for detecting an object in a traveling direction, and a predetermined target detected by the sonar. And a calculating means for calculating the moving amount or moving speed of the vehicle based on the change in the distance between the object and the vehicle.

First, a vehicle movement state detection device according to claim 14 will be described. According to the vehicle movement state detecting device of the fourteenth aspect, the bird's-eye view conversion unit converts the image captured by the image capturing unit into the bird's-eye view image, and the bird's-eye view image storage unit stores the bird's-eye view image storage unit. Then, the calculation unit compares the past bird's-eye image stored in the bird's-eye image storage unit with the current bird's-eye image captured by the image-capturing unit and bird-eye-converted by the bird's-eye conversion unit. Alternatively, the moving speed is calculated. The vehicle periphery monitoring system according to a fourteenth aspect executes processing such as rotation / movement of the bird's-eye view image based on the movement amount or movement speed of the vehicle calculated in this way.

The calculation method by the calculation means is as follows.
For example, the highest brightness pixel (highest brightness pixel) is extracted from the bird's-eye view image at the previous calculation time T, and the highest brightness pixel is also extracted from the bird's-eye view image at the current calculation time T + 1. In the meantime, the vehicle may move linearly to obtain the amount of movement of the highest brightness pixel in the image.

In this case, the highest-brightness pixel divides the bird's-eye view image into several areas of a predetermined size,
It is also possible to extract each area and to calculate the linear movement amount by comparing the positional relationship between the plurality of highest luminance pixels at time T with the positional relationship at time T + 1. This way,
The reliability of the calculation result is improved.

Further, as another method, the degree of brightness of each pixel in the bird's-eye view image at time T is extracted as a numerical value, and each is stored as an attribute of each pixel.
Similarly, the degree of brightness of each pixel in the bird's-eye view image at time T + 1 is extracted as a numerical value and stored as an attribute of each pixel. Then, the linear movement amount of the vehicle from the time T to the time T + 1 can be calculated by calculating the positional relationship between the pixels having the same brightness.

Further, for example, if it is known that the vehicle is moving backward, the brightness and color of the pixels in the image area of a predetermined size at the rear end side edge portion in the bird's-eye view image at the time T. The image characteristic information is extracted as numerical data, and in the bird's-eye view image at time T + 1, a pixel located within an image area of a predetermined size with the rear end being a position linearly separated from the rear end edge by a minute distance Δy. The image characteristic information such as the brightness and color of the image is extracted, and the degree of coincidence between the two is determined. If they do not coincide with each other, the bird's-eye view image at time T + 1 is set at a position 2Δy away from the rear edge. Similar image-like characteristic information is extracted from the pixels in the image area of a predetermined size which is the rear end, and is matched with the characteristic information extracted from the bird's-eye view image at time T. Until the degree of coincidence between them satisfies a predetermined condition, n is changed to 3, 4, ... Then, the same calculation process is executed, and nΔ when the predetermined degree of coincidence is obtained. It is also possible to execute a calculation process such that y is a linear movement amount.

In this way, by comparing the past bird's-eye view image with the current bird's-eye view image, it is possible to easily calculate the moving amount or moving speed of the vehicle at the time of low-speed moving in image processing.

Next, a vehicle moving state detecting device according to a fifteenth aspect will be described. According to the vehicle moving state detecting device of the fifteenth aspect, the moving amount or moving speed of the vehicle can be easily calculated by obtaining the change in the distance between the vehicle and some stationary object in the traveling direction detected by the sonar. . The vehicle periphery monitoring system according to a fifteenth aspect executes processing such as rotation / movement of the bird's-eye view image based on the movement amount or movement speed of the vehicle calculated in this way.

Since the accuracy of the sensor increases with the times, if a vehicle speed sensor that can detect an accurate vehicle speed even at an extremely low speed can be used, the amount of movement or the moving speed of the vehicle can be determined from the detection signal of the vehicle speed sensor. Is calculated, and the bird's-eye view image is rotated / moved based on this, without departing from the gist of the present invention.

[0050]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a vehicle periphery monitoring system 1 of the present embodiment. As shown in the drawing, a front camera 11 and a rear camera 13, which are image input means, and bird's-eye conversion circuits 15 and 17 for executing image processing for converting images captured by these cameras 11 and 13 into bird's-eye images. , Temporary storage memories 21 and 23 for temporarily storing the images converted by the bird's-eye view conversion circuits 15 and 17, and a CPU 25 that controls the center of control processing.
And a history image storage memory 27 that stores an image as a result of executing a predetermined arithmetic process in the CPU 25, and an image stored in the history image storage memory 27 or the temporary storage memories 21 and 23 are combined and obtained. Image memory 29 for storing the displayed image, and display image memory 29
Monitor 3 for reading and displaying images stored in
1 and a steering wheel angle signal input circuit 41 for inputting information on the steering wheel operation of the user to the CPU 25.
Similarly, a vehicle speed signal input circuit 43 for inputting information on the vehicle speed to the CPU 25, a yaw rate signal input circuit 45 for inputting information on the turning condition of the vehicle to the CPU 25, and obstacles existing in the moving direction of the vehicle. Information such as the presence or absence of an object and the distance between the obstacle and the vehicle is detected by C
Ultrasonic sensor signal input circuit 4 for input to PU25
7 and a shift position signal input circuit 49.

The bird's-eye view conversion circuits 15 and 17, the temporary storage memories 21 and 23, the CPU 25, and the history image storage memory 2
7, display image memory 29, steering wheel angle signal input circuit 4
1, vehicle speed signal input circuit 43, yaw rate signal input circuit 4
5, the ultrasonic sensor signal input circuit 47 and the shift position signal input circuit 49 may be wholly or partially integrated into one LSI. Further, with respect to the bird's-eye view conversion circuits 15 and 17, instead of providing a dedicated image processing circuit, the images taken by the cameras 11 and 13 are input to the CPU 25, and the images are converted into bird's-eye view images by software through arithmetic processing. It can also be realized by such a method.

As shown in FIG. 2, the front camera 11 is installed, for example, in the vicinity of the center of the front grill of the vehicle so that an image of the front of the vehicle can be taken.
Further, the rear camera 13 is installed, for example, in the vicinity of the center of the rear wall of the luggage room of the vehicle so that an image of the rear of the vehicle can be taken.

Here, the front camera 11 and the rear camera 1
A plurality of 3 may be installed at different positions in the width direction of the vehicle. However, when a plurality of cameras are installed in the front and / or the rear, the images having different shooting ranges are adjusted for each camera during the image processing for realizing the system of the present embodiment. Since it is necessary to determine the front image and the rear image, the field of view becomes wider and more information can be obtained, but there is also the problem that the arithmetic processing circuit and the program for image processing become complicated, It is necessary to put it into practical use in consideration of cost aspects.

The image taken by the front camera 11 is
It is input to the bird's-eye view conversion circuit 15 for the front image, subjected to image processing to be a front bird's-eye view image, and stored in the temporary storage memory 21. At the start of photographing, the front bird's-eye view image is further stored in the front history image storage area in the history image memory 27. The image stored in the front history image storage area is read by the CPU 25 at the next calculation timing, and the steering wheel angle signal input circuit 41, the vehicle speed signal input circuit 43, the yaw rate signal input circuit 45, and the ultrasonic sensor signal input. Geometrical image processing such as rotation and movement is performed using various information input to the CPU 25 via the circuit 47. The history image after this geometrical image processing is further captured by the front camera 11 at this calculation timing, bird-eye converted by the bird's-eye conversion circuit 15, and overwritten in the temporary storage memory 21. Is overlaid on the history image storage memory 27 again.

Similarly, the image captured by the rear camera 13 is input to the bird's-eye view conversion circuit 17 for the rear image, subjected to image processing to be a rear bird's-eye view image, and temporarily stored in the memory 23.
Memorized in. At the start of shooting, this rear bird's-eye image is
Further, it is also stored in the backward history image storage area in the history image memory 27. The image stored in the backward history image storage area is stored in the CPU 2 at the next calculation timing.
5, the steering wheel angle signal input circuit 41,
Geometrical image processing such as rotation and movement is performed using various information input to the CPU 25 via the vehicle speed signal input circuit 43, the yaw rate signal input circuit 45, and the ultrasonic sensor signal input circuit 47. The history image after the geometrical image processing is further updated with the latest backward bird's-eye image captured by the rear camera 13 at this calculation timing, bird's-eye converted by the bird's-eye conversion circuit 17, and overwritten in the temporary storage memory 23. Is overlaid on the history image storage memory 27 again.

The image to be finally displayed on the monitor 31 after being subjected to predetermined arithmetic processing by the CPU 25 is
It is stored in the display image memory 29. Monitor 31
It is arranged at a position easily visible from the driver's seat (for example, the center of the dashboard), and can also serve as the display of the car navigation device.

Next, the contents of the control processing executed by the vehicle periphery monitoring system 1 of the present embodiment will be described. here,
Regarding the image of the rear of the vehicle taken by the rear camera 13, what kind of processing is executed to achieve the first and second objects of the present invention and what kind of image is displayed on the monitor 31 will be described. To do.

The image captured by the rear camera 13 is subjected to image processing in the bird's-eye view conversion circuit 17 and converted into a bird's-eye view image viewed from the sky.

The bird's-eye view conversion is executed as follows. As a prerequisite, the rear camera 13 has an autofocus function. Therefore, the image captured by the rear camera 13 has the focal length matched with the center of the image. This focal length can be acquired as numerical information indicating the distance based on the information indicating how much the lens has been moved from the origin position based on the autofocus function. Further, the mounting height and mounting angle of the rear camera 13 are known in advance. Further, in the present embodiment, the ground is a plane as a precondition for calculation. Based on these information and conditions, it is possible to calculate the distance and the position in the width direction to each pixel with the vehicle rear end as the reference line when the image captured by the rear camera 13 is decomposed into pixel units. When the position coordinates of each pixel after the bird's-eye view conversion can be calculated in this way, the pixels are rearranged according to the position coordinates. As a result, the image captured by the rear camera 13 can be converted into a bird's-eye view image as a plane image viewed from above.

By executing this bird's-eye view conversion,
In terms of images, it is assumed that there will be gaps or overlaps between pixels, but when the vehicle is parked in the back, the image displayed on the monitor 31 has been thinned out to some extent. Even if it is such an image, the bird's-eye view converted image has better operability than the case of executing the garage while watching the raw image taken by the rear camera 13, and the object of the present invention. There is no problem in achieving

For a more detailed calculation formula regarding the bird's-eye view conversion, refer to Japanese Unexamined Patent Publication No. 10-211849 for details.

The bird's-eye view image thus obtained is stored in the temporary storage memory 23 together with the raw image taken by the rear camera 13. On the other hand, the CPU 25 includes a steering wheel angle signal input from the steering wheel angle signal input circuit 41, a vehicle speed signal input from the vehicle speed signal input circuit 43, a yaw rate signal input from the yaw rate signal input circuit 45, and an ultrasonic sensor signal input circuit. Based on the ultrasonic sensor signal input from 47, the movement of the vehicle (the rotation center position and the rotation angle of the vehicle) is calculated.

Then, the CPU 25 reads the bird's-eye view image obtained from the image taken by the rear camera 13 at the previous calculation timing from the history image storage memory 27 based on the movement of the vehicle, and rotates / moves the bird's-eye view image. On the other hand, after the latest bird's-eye view image stored in the temporary storage memory 23 is overlaid at the calculation timing of this time to obtain a combined image, this combined image is stored in the history image storage memory 2
Store in 7. As the vehicle moves backward, the latest bird's-eye view image is overlaid on the image read out from the history image storage memory 27 and rotated / moved, so that the rear camera 13 is more likely to capture the image than the range currently captured. It is also possible to form an image in a wide range, and it is possible to provide an image of the side of the vehicle that cannot be captured by the rear camera 13 as information.

This state will be described with reference to the drawings. As shown in FIG. 3, it is assumed that the vehicle is slowly moving backward with the back in the curved portion. Then, in the conventional technique, as shown in FIGS. 4A and 4B, only the bird's-eye view image of the range captured by the rear camera 13 from the current position of the vehicle is displayed. On the other hand, according to the present embodiment, a bird's-eye view image in the past is rotated / moved in accordance with the movement of the vehicle, and a current bird's-eye view image at the time of calculation is overlaid to form a combined bird's-eye view image. Since it is displayed on the monitor 31,
In the vehicle traveling as shown in FIG. 3, when the vehicle is about to pass the curve,
As in (C), an image can be displayed even in the blind spot of the rear camera 13.

The CPU 25 also stores this composite image in the display image memory 29. Then the monitor 31
Reads the composite image stored in the display image memory 29 and displays it.

As a result, the monitor 31 can display not only the range currently being photographed by the rear camera 13 but also past information as long as it can be displayed on the monitor 31. Moreover, since this past information is rotated and moved according to the movement of the vehicle, the monitor 31
In this case, an image of a state in which the vicinity of the vehicle is viewed from above is displayed including the blind spot of the rear camera 13 (see FIG. 4C).

Next, more specific contents of the image processing for rotating / moving the history image will be described.

The CPU 25 uses the steering wheel angle signal input circuit 4
The turning radius of the host vehicle is calculated from the steering wheel angle sensor signal input from 1. Handle angle θh and turning radius XC
(M) can be calculated from (Equation 1) using the conversion coefficient K.

[0069]

[Formula 1] XC (m) = L (m) / tan (K · θh) −
W (m) / 2 where L (m) is the wheel base and W (m) is the vehicle width.

As the vehicle rotation angle θC, a yaw rate sensor signal input via the yaw rate signal input circuit 45 can be used. The center of rotation of the vehicle is
It may be on the rear wheel axle (on the distance YC (m) from the camera) (see FIG. 5).

Next, the rear bird's-eye view image stored in the rear image storage area in the history image storage memory 27 is rotated / moved using the rotation center position, the rotation angle, and the linear movement amount.

The rotation conversion is performed based on a basic mathematical formula such as the following formula.

[0073]

[Formula 2] XA = XB · cos θC + YB · sin θC YA = YB · cos θC−XB · sin θC (XA, YA): coordinates before movement, (XB, Y
B): Coordinates after movement.

From the relationship of this expression 2, the rear bird's-eye view image stored in the rear image storage area in the history image storage memory 27 is rotated / moved based on the obtained coordinates after the movement. If the vehicle is not turning, the following equation 3
The coordinate after movement can be calculated by.

[0075]

[Formula 3] XA = XB YA = YB + V However, V is the amount of linear movement.

The conversion according to the equation 3 is executed when the turning radius XC <30 or the yaw rate sensor signal is zero. The linear movement amount V can be calculated from the vehicle speed signal, the calculation interval, and the movement direction such as whether the vehicle is moving forward or backward. For the moving direction of the vehicle, it may be determined whether the shift range is forward or backward. When the vehicle speed signal from the vehicle speed signal input circuit 43 is extremely low speed (for example, 5 km / h or less), the reliability of the normally used vehicle speed sensor is considered to be poor. Therefore, in the present embodiment, the linear movement amount V is calculated by image processing as follows.

In the bird's-eye view image at the previous calculation time T,
The pixel with the highest brightness (highest brightness pixel) is extracted.
As an example, it is assumed that the highest brightness pixel is extracted in the center of the image as shown in FIG. The highest brightness pixel can be easily extracted by calculating the brightness for each pixel and taking the difference. Next, the highest brightness pixel is also extracted from the bird's-eye view image at the current calculation time T + 1. As the vehicle moves linearly from time T to time T + 1, the distance of the highest brightness pixel in the image from the trailing edge of the image changes, as shown in FIG. This distance difference ΔY = the linear movement amount V. The sign of V is (-) in this case. In addition, as shown in FIG. 7, the highest-brightness pixel is divided into several areas of a predetermined size in the bird's-eye view image and extracted for each area. If the linear movement amount V is calculated by comparing the positional relationship at time T with the positional relationship at time T + 1, the reliability of the calculation result increases.
According to the present embodiment, the linear movement amount V can thus be calculated from the image data, and as a result, the vehicle speed at an extremely low speed can be easily calculated.

As another method, as shown in FIG. 8, the degree of brightness of each pixel in the bird's-eye view image at time T is extracted as a numerical value, and the attribute of each pixel is extracted as an internal memory of the CPU 25. Remember. Similarly,
The degree of brightness of each pixel in the bird's-eye view image at time T + 1 is extracted as a numerical value, and each is stored in the internal memory of the CPU 25 as an attribute of each pixel. Then, the linear movement amount V of the vehicle from the time T to the time T + 1 can be calculated by calculating the positional relationship between the pixels having the same brightness. In the illustrated example, time T
The brightness of three columns of pixels at the rear end side edge portion in the bird's-eye view image at 1 is equal to the brightness of pixels for three columns at a distance ΔY from the rear end side edge at the time T + 1. I am doing it. So in this case,
The distance ΔY is calculated as the linear movement amount V. The sign of V is (-) in this case.

If it is known that the vehicle is moving backward, the image characteristic information such as the brightness and the color of the pixel in the image area of the predetermined size of the trailing edge portion in the bird's-eye image at time T is numerically calculated. Extracted as data, time T +
With respect to the bird's-eye view image in No. 1, image characteristic information such as brightness and color of pixels in an image area of a predetermined size is extracted with a rear end at a position linearly separated from the rear end edge by a small distance Δy. , The degree of coincidence between them is determined, and if they do not coincide with each other, the bird's-eye view image at time T + 1 is within an image area of a predetermined size with the rear end at a position 2Δy away from the rear end side edge. Similar image-like characteristic information is extracted from a certain pixel, and the degree of coincidence with the characteristic information extracted from the bird's-eye view image at time T is determined. , 4, ... and execute the same calculation process,
It is also possible to execute a calculation process such that nΔy when the predetermined degree of coincidence is obtained is the linear movement amount V. In addition,
In this case, at time T + 2, it is estimated that the amount of movement is substantially the same as the linear movement amount V = nΔy calculated at time T + 1, and the calculation time is set by selecting the calculation start condition. It is more desirable to shorten it.

Next, the main control processing executed by the CPU 25 in the present embodiment will be described based on the flowcharts of FIGS. 9 and 10. In this control process, the bird's-eye view conversion described as being executed in the bird's-eye view conversion circuits 15 and 17 is assumed to be executed by the CPU 25 as software.

In this processing, the vehicle speed signal input circuit 43
The vehicle speed signal detected by the vehicle speed sensor is input from (S1
0), it is determined whether or not this value is 10 km / h or less (S
20). Until the vehicle speed falls below 10 km / h, S1
The processing of 0 and S20 is repeatedly executed. On the other hand, when the vehicle speed is 10 km / h or less, the shift range is continuously changed to R.
It is determined whether it is in the range (S30). Whether or not the shift range is the R range can be easily determined by using the input signal from the shift position switch.

When it is determined that the shift range is the R range (S30: YES), the D range is further changed to the R range.
It is determined whether or not it is immediately after switching to the range (S
40). Immediately after switching from the D range to the R range (S40: YES), the history image storage memory 27 is reset (S50). When it is determined that the shift range is not the R range (S30: NO), it is further determined whether or not it is immediately after switching from the R range to the D range (S45). Immediately after switching from the R range to the D range (S45: YES), the history image storage memory 27 is reset (S55).

Next, the rear camera 13 is operated (S6
0), the image photographed by the rear camera 13 is input (S70). Then, the steering wheel angle signal detected by the steering wheel angle sensor is input through the steering wheel angle signal input circuit 41 (S80). In addition, the yaw rate signal input circuit 45
The yaw rate signal detected by the yaw rate sensor is input via (S90). Further, the ultrasonic sensor signal is input via the ultrasonic sensor signal input circuit 47 (S1).
00). In addition, as the ultrasonic sensor signal, in a vehicle having a back sonar using ultrasonic waves, detection data by the back sonar can be used.

After inputting the necessary information in this way, the image input in S70 is then converted into a bird's-eye view image using the conversion method as described above (S110). And
The result is stored in the temporary storage memory 23 (S12).
0).

Then, it is determined whether or not the current process is the first process (S130). This determination can be easily realized by using a flag or the like. Specifically, when the vehicle speed is not 10 Km / h or less, the initial determination flag is reset to 0 for returning to S10,
When the initial determination flag is 0, it may be determined that it is the first time. Then, when it is determined to be YES in S130, this initial determination flag is set to 1, and
It suffices not to reset the initial determination flag until the vehicle speed is no longer than 10 km / h.

If it is determined that the process is the first time (S
130: YES), the history image storage memory 27 is reset (S140), and the bird's-eye view image stored in the temporary storage memory 23 is transferred to the history image storage memory 27 (S15).
0). Then, the process returns to S10 again.

On the other hand, when it is determined that it is not the first time (S130: NO), it is determined whether the steering wheel angle is 0 ° (S160). When it is determined that the steering wheel angle is not 0 ° (S160: YES), the rotation center and the rotation angle of the vehicle are calculated from the steering wheel angle and the yaw rate using the above-described arithmetic expression and the like (S170). . Then, based on the calculation result, the bird's-eye view image stored in the history image storage memory 27 is rotated (S180). Next, a synthetic bird's-eye image is created by overlaying the latest bird's-eye image stored in the temporary storage memory 23 on the result of this rotational movement so that the latest bird's-eye image becomes the front (S190). Then, the composite bird's-eye view image is overwritten and stored again in the history image storage memory 27 (S200).

Next, the combined bird's-eye view image created in S190 is subjected to image processing so that the portion corresponding to the past image display section in the display area of the monitor 31 is in a state of being color-filtered (S210). Then, this result is transferred to the display image storage memory 29 (S220). Finally,
With respect to the combined bird's-eye view image after the filter processing read from the display face storage memory 29, a vehicle display image is further overlaid on the foreground and displayed on the monitor 31 (S230). Note that the vehicle display image is a bird's-eye view image of the vehicle in which the present system is mounted as seen from above, and is stored in advance in an EEPROM or the like.

Next, the processing when the steering wheel angle is not 0 ° will be described. When the steering wheel angle is not 0 ° (S160: NO), it is determined whether or not the distance (ultrasonic sensor distance) to the obstacle that is identified from the ultrasonic sensor signal is 1.5 m or less (S240). The ultrasonic sensor distance is 1.
When it is determined that the distance is 5 m or less (S240: YE
S), the relative speed is calculated based on the change state of the ultrasonic sensor distance (S250). On the other hand, when it is determined that the ultrasonic sensor distance is not less than 1.5 m (S240:
NO), using the past image stored in the history image storage memory 27 and the current image stored in the temporary storage memory 23, the image characteristic information of the pixel such as the brightness and the color as described above is used. A motion vector representing the motion of the vehicle is calculated by the predetermined calculation process (S260). Next, the amount of movement of the image is determined based on this motion vector or relative speed (S270). Then, the parallel movement of the image is executed as shown in Expression 3 (S280), and, as a result of the parallel movement, the latest bird's-eye view image stored in the temporary storage memory 23 is changed to the latest bird's-eye view image. A composite bird's-eye image is created by overlaying it so that it is on the front (S29).
0). Then, thereafter, the same processing as that after the rotational movement is performed, and finally the combined bird's-eye view image is displayed on the monitor 31 (S200 to S230).

As a result of executing the above processing, an image as shown in FIG. 11 is displayed on the monitor 31. As shown in the figure, a vehicle graphic 50 is displayed in the monitor 31 so that the center of the upper end of the screen is the tip of the vehicle.
Further, a synthetic bird's-eye view image 60 is displayed behind and on the side of the vehicle graphic 50. This synthetic bird's-eye image is
A real image display unit 61 that displays an image obtained by bird's-eye conversion of an image currently captured by the rear camera 13 and a past image display unit 63 other than the real image display unit 61 are provided. The past image display section 63 is, for example, in a state in which a gray filter is applied, and the real image display section 61 displays a bird's-eye view image in full color display.

As a result, according to the present embodiment, not only the current image but also the bird's-eye view image reflecting the past image is displayed, which enables the driver to drive the vehicle when he or she tries to park the vehicle in the back. The positional relationship between an obstacle and a parking frame is very easy to understand, and operability is greatly improved.

On the monitor 31, the past image is displayed with a gray filter applied so that the image actually photographed by the rear camera 13 and the past image can be easily distinguished. ing. Therefore, the driver can understand the range of the portion of the bird's-eye view image displayed on the monitor 31 that may be invading another vehicle. As a result, it is possible to call the driver's attention so as not to neglect the visual confirmation of a situation change such as whether another vehicle is intruding.

Further, when entering a garage or the like, not only simply backing up, but forward and reverse may be repeated like a switching operation. However, according to the present embodiment, the timing when the shift range is switched Then, since the history image storage memory 27 is once reset, when the driver switches the shift range from the D range to the R range, the image history when the driver was moving forward does not remain and it is erroneous. Problems such as giving information to the driver can be solved. Similarly, even when the vehicle moves forward again by switching back, the history image storage memory 27 is once reset at the timing when the R range is switched to the D range, so the driver switches the shift range from the R range to the D range. Occasionally, the image history of when the vehicle was moving backward will not remain, and this too can solve the problem of giving incorrect information to the driver.

For example, when the memory reset is not executed under such a condition, the image at the time of the forward movement in the previous turning operation remains at the time of the backward movement as shown by the portion surrounded by an ellipse in FIG. It may not be possible to grasp the situation correctly. On the other hand, in the present embodiment, since the memory reset is executed under the above-described conditions,
As shown in FIG. 3, when moving backward, the image when moving forward does not remain and erroneous information is not transmitted to the driver.

In addition, according to the present embodiment, the speed of the vehicle moving at an extremely low speed can be calculated based on the image processing or the ultrasonic sensor signal. Therefore, the vehicle speed sensor which is not sufficiently accurate in the low speed range. Even if the vehicle adopts, it is possible to provide a highly reliable vehicle peripheral image.

Next, the perimeter monitoring system 1 of this embodiment
In, an optional control process when attempting to park the vehicle forward will be described. This optional control process can be executed when a predetermined condition such as operating a camera changeover switch provided on the monitor 31 is satisfied.

The control processing of this option is configured as shown in the flow charts of FIGS. 14 and 15, and the vehicle speed signal detected by the vehicle speed sensor from the vehicle speed signal input circuit 43 is the same as when the vehicle is parked in the back. Is input (S410), and it is determined whether this value is 10 km / h or less (S420). The processes of S410 and S420 are repeatedly executed until the vehicle speed becomes 10 km / h or less.
On the other hand, when the vehicle speed becomes 10 km / h or less, it is subsequently determined whether or not the shift range is the D range (S430).

When it is determined that the shift range is the D range (S430: YES), it is further determined whether or not it is immediately after switching from the R range to the D range (S440). Immediately after switching from the R range to the D range (S440: YES), the history image storage memory 27 is reset (S450). If it is determined that the shift range is not the D range (S430: N
O), and it is further determined whether or not it is immediately after switching from the D range to the R range (S445). Immediately after switching from the D range to the R range (S445: YE
S), the history image storage memory 27 is reset (S45
5).

Next, the front camera 11 is operated (S46
0), the image photographed by the front camera 11 is input (S470). Then, the steering wheel angle signal detected by the steering wheel angle sensor is input through the steering wheel angle signal input circuit 41 (S480). Further, the yaw rate signal detected by the yaw rate sensor is input via the yaw rate signal input circuit 45 (S490). Further, the ultrasonic sensor signal is input via the ultrasonic sensor signal input circuit 47 (S500). In addition, as the ultrasonic sensor signal, in a vehicle having a front sonar for maintaining an inter-vehicle distance using ultrasonic waves and for preventing collision, the detection data by the front sonar can be used.

When the necessary information is input in this way, the image input in S470 is then converted into a bird's-eye view image by using the conversion method as described above (S510). Then, this result is stored in the temporary storage memory 21 (S5).
20).

Then, it is determined whether or not the current process is the first process (S530). If it is determined that the process is the first time (S530: YES), the history image storage memory 27
Is reset (S540), and the bird's-eye view image stored in the temporary storage memory 21 is transferred to the history image storage memory 27 (S550). Then, the process returns to S410 again.

On the other hand, if it is determined that it is not the first time (S530: NO), it is determined whether the steering wheel angle is 0 ° (S560). When it is determined that the steering wheel angle is not 0 ° (S560: YES), the rotation center and the rotation angle of the vehicle are calculated from the steering wheel angle and the yaw rate using the above-described arithmetic expression and the like (S570). . Then, based on this calculation result, the bird's-eye view image stored in the history image storage memory 27 is rotated (S580). Next, a synthetic bird's-eye image is created by overlaying the latest bird's-eye image stored in the temporary storage memory 21 on the result of this rotational movement so that the latest bird's-eye image becomes the front (S590). Then, the composite bird's-eye view image is overwritten and re-stored in the history image storage memory 27 (S600).

Next, the combined bird's-eye view image created in S590 is subjected to image processing so that the portion corresponding to the past image display area in the display area of the monitor 31 is in the state of being subjected to the color filter (S610). Then, the result is transferred to the display image storage memory 29 (S620). Finally,
With respect to the combined bird's-eye view image after the filter processing read from the display face storage memory 29, a vehicle display image is further overlaid on the foreground and displayed on the monitor 31 (S630).

On the other hand, when the steering wheel angle is not 0 ° (S560: NO), it is determined whether the ultrasonic sensor distance is 1.5 m or less (S640). When it is determined that the ultrasonic sensor distance is 1.5 m or less (S640: Y
ES), the relative speed is calculated based on the change state of the ultrasonic sensor distance (S650). On the other hand, if it is determined that the ultrasonic sensor distance is not less than 1.5 m (S640:
NO), using the past image stored in the history image storage memory 27 and the current image stored in the temporary storage memory 21, the image-like characteristic information of the pixel such as the brightness and the color as described above is used. The motion vector representing the motion of the vehicle is calculated by the predetermined calculation process (S660). Next, the amount of movement of the image is determined based on this motion vector or relative speed (S670). Then, the parallel movement of the image is executed as shown in Expression 3 (S680), and, as a result of the parallel movement, the latest bird's-eye view image stored in the temporary storage memory 21 is changed to the latest bird's-eye view image. Create a composite bird's-eye view image by overlaying it so that it is on the front (S69).
0). Then, thereafter, the same processing as after the rotational movement is performed, and finally the combined bird's-eye view image is displayed on the monitor 31 (S600 to S630).

As a result of executing the above processing, an image as shown in FIG. 16 is displayed on the monitor 31. As shown in the figure, a vehicle graphic 50 is displayed in the monitor 31 so that the center of the lower end of the screen is the center of the rear end of the vehicle. Further, a synthetic bird's-eye view image 60 is displayed in front of and on the side of the vehicle graphic 50. This synthetic bird's-eye view image 60
Further, the real image display unit 61 that displays an image obtained by bird's-eye conversion of the image currently captured by the rear camera 13.
And the past image display section 6 other than the real image display section 61.
It is 3. The past image display section 63 is, for example, in a state in which a gray filter is applied, and the real image display section 61 displays a bird's-eye view image in full color display.

As a result, according to the present embodiment, more information can be provided to the driver even when the vehicle is parked forward as in the case of back parking, and the vehicle tries to park the vehicle forward. At this time, the positional relationship between the vehicle and obstacles, parking frames, etc. becomes very easy to understand, and operability is greatly improved.

Further, as in the case of back parking, the image actually taken by the front camera 11 and the past image can be easily distinguished, and the driver can select one of the bird's-eye images displayed on the monitor 31. It is possible to understand the range where there is a possibility that another vehicle is invading. As a result, it is possible to call the driver's attention so as not to neglect the visual confirmation of a situation change such as whether another vehicle is intruding.

Furthermore, even in forward parking, it is possible to solve the problem that incorrect information is given to the driver when the driver performs a turning operation.

In addition, it is possible to provide a highly reliable image of the surroundings of a vehicle even if the vehicle employs a vehicle speed sensor that does not have sufficient accuracy in the low speed range.

Next, the second embodiment will be described. The peripheral monitoring system of the second embodiment has the same hardware configuration as that of the above-described embodiment and is characterized by the contents of the control processing described below.

The control process of the second embodiment is shown in FIG.
7. As shown in the flowchart of FIG. 18, first, the vehicle speed signal detected by the vehicle speed sensor is input from the vehicle speed signal input circuit 43 (S710), and this value is 10 km.
/ H or less is determined (S720). Vehicle speed is 10K
The processes of S410 and S420 are repeatedly executed until m / h or less. On the other hand, when the vehicle speed becomes 10 km / h or less, it is subsequently determined whether or not the shift range is the R range (S730).

When it is determined that the shift range is the R range (S730: YES), it is further determined whether or not it is immediately after switching from the D range to the R range (S740). Immediately after switching from the R range to the D range (S740: YES), the history image storage memory 27 is reset (S750), and the rear camera 1
3 is operated (S760). Also, the shift range is R
If it is determined that the range is not set (S730: N
O), and further, it is determined whether or not it is immediately after switching from the R range to the D range (S745). Immediately after switching from the R range to the D range (S745: YE
S), the history image storage memory 27 is reset (S755), and the front camera 11 is operated (S76).
5).

Next, the front camera 11 (or the rear camera 1)
The image photographed in 3) is input (S770). continue,
The steering wheel angle signal detected by the steering wheel angle sensor is input through the steering wheel angle signal input circuit 41 (S780).
Further, the yaw rate signal detected by the yaw rate sensor is input via the yaw rate signal input circuit 45 (S79).
0). Further, the ultrasonic sensor signal is input via the ultrasonic sensor signal input circuit 47 (S800).

When the necessary information is input in this way, the image input in S770 is then converted into a bird's-eye view image by using the conversion method as described above (S810). Then, this result is stored in the temporary storage memory 21 (or the temporary storage memory 23) (S820).

Then, it is determined whether or not the current process is the first process (S830). If it is determined that the process is the first time (S830: YES), the history image storage memory 27
Is reset (S840), and the bird's-eye view image stored in the temporary storage memory 21 (or the temporary storage memory 23) is transferred to the history image storage memory 27 (S850). And
It returns to S710 again.

On the other hand, if it is determined that it is not the first time (S830: NO), it is determined whether the steering wheel angle is 0 ° (S860). When it is determined that the steering wheel angle is not 0 ° (S860: YES), the rotation center and the rotation angle of the vehicle are calculated from the steering wheel angle and the yaw rate using the above-described arithmetic expression and the like (S870). . Then, based on this calculation result, the bird's-eye view image stored in the history image storage memory 27 is rotated (S880). Next, the result of this rotation movement is overlaid with the latest bird's-eye view image stored in the temporary storage memory 21 (or the temporary storage memory 23) so that the latest bird's-eye view image is on the front side to form a combined bird's-eye view image. Create (S890). Then, the composite bird's-eye view image is overwritten and re-stored in the history image storage memory 27 (S900).

Next, the composite bird's-eye view image created in S890 is subjected to image processing so that the portion corresponding to the past image display portion in the display area of the monitor 31 is in a state of being color-filtered (S910). Then, this result is transferred to the display image storage memory 29 (S920). Finally,
The vehicle image for vehicle display is further overlaid on the foreground of the combined bird's-eye view image after the filter processing read from the display face storage memory 29 and then displayed on the monitor 31 (S930).

On the other hand, when the steering wheel angle is not 0 ° (S860: NO), it is determined whether the ultrasonic sensor distance is 1.5 m or less (S940). When it is determined that the ultrasonic sensor distance is 1.5 m or less (S940: Y
ES), the relative speed is calculated based on the change state of the ultrasonic sensor distance (S950). On the other hand, when it is determined that the ultrasonic sensor distance is not less than 1.5 m (S940:
NO), using the past image stored in the history image storage memory 27 and the current image stored in the temporary storage memory 21, the image-like characteristic information of the pixel such as the brightness and the color as described above is used. A motion vector representing the motion of the vehicle is calculated by the predetermined calculation process (S960). Next, the amount of movement of the image is determined based on this motion vector or the relative speed (S970). Then, the parallel movement of the image is executed as shown in Expression 3 (S980), and thereafter, as a result of the parallel movement, the latest bird's-eye view image stored in the temporary storage memory 21 is changed to the latest bird's-eye view image. Create a composite bird's-eye view image by overlaying it so that it is on the front (S99).
0). Then, thereafter, the same processing as that after the rotational movement is performed to finally display the composite bird's-eye view image on the monitor 31 (S900 to S930).

As a result of executing the above-described processing, the monitor 31 will be shown in FIG. 19 when the shift range is the R range.
An image as shown in (A) is displayed. R as shown
When the range is displayed, the vehicle graphic 50 is displayed in the monitor 31 so that the center of the upper end of the screen is the center of the front end of the vehicle. Further, a synthetic bird's-eye view image 60 is displayed behind and on the side of the vehicle graphic 50. This synthetic bird's-eye view image 60
Further, the real image display unit 61 that displays an image obtained by bird's-eye conversion of the image currently captured by the rear camera 13.
And the past image display section 6 other than the real image display section 61.
It is 3. Then, the past image display section 63 is, for example, in a state where a gray filter is applied, and the bird's-eye view image in full color display is displayed on the real image display section 61. On the other hand, when the shift range is the D range, as shown in FIG. 19B, the vehicle graphic 50 is displayed in the monitor 31 such that the center of the lower end of the screen is the center of the rear end of the vehicle. Further, a synthetic bird's-eye view image 60 is displayed in front of and on the side of the vehicle graphic 50. The synthetic bird's-eye view image 60 further includes a real image display unit 61 displaying an image obtained by bird's-eye conversion of the image currently captured by the front camera 11, and a past image display unit 63 other than the real image display unit 61. It is supposed to be. The past image display unit 63 is, for example, in a state where a gray filter is applied, and the real image display unit 6 is used.
1, a bird's-eye view image in full color display is displayed.

As a result, according to the present embodiment, when the vehicle is parked and the vehicle is turned back to switch from reverse to forward, the monitor 31 displays a composite image based on the image taken by the front camera 11. A bird's-eye view image is displayed so that the positional relationship with the own vehicle can be understood. If you are going backwards,
A synthetic bird's-eye image generated from the image captured by the rear camera 13 is displayed on the monitor 31. Therefore, when trying to park while repeating the switching operation,
The driver can check the situation around the vehicle centering on the traveling direction by the monitor 31. In addition, the monitor 31 displays a composite bird's-eye view image including the past history of forward movement when moving forward, and displays a composite bird's-eye view image including past history of backward traveling when updating, so that the driver switches back. In the final phase of the operation, the effect of being able to park the vehicle accurately while observing the monitor 31 is exhibited.

Further, as in the case of the main control process and the optional control process of the first embodiment, the current image actually photographed by the front camera 11 or the rear camera 13 and the past image are recorded. Can be easily distinguished, and the driver can understand the range of the portion of the bird's-eye view image displayed on the monitor 31 that may be invading another vehicle. As a result, do not neglect to visually check the situation change such as whether another vehicle is intruding,
It can call the driver's attention.

Although the embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and it is needless to say that various embodiments can be adopted without departing from the scope of the invention. Is.

For example, as shown in FIG. 20, the boundary line 81 may be simply overwritten and displayed. Further, as shown by the dotted line in FIG. 21, regarding the past image display unit,
The gradation of the image may be reduced. Further, as shown in FIG. 22, a caution mark 83 and a comment 85 may be overwritten on the past image display section. These modifications are also applicable to the image taken by the front camera 11. It should be noted that the filter processing, the border line display, the gradation processing, and the overwriting of the caution mark may be combined with each other, and when the past image is displayed on the monitor 31, a voice message “sideways” is displayed. For the above, please check visually. "

Further, as shown in FIG. 23, the one in which the composite bird's-eye view image is simply displayed on the monitor without displaying the boundary line and the like achieves the first object of the present invention. Such a simple configuration can also be adopted.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a vehicle surroundings monitoring system as an embodiment.

FIG. 2 is a block diagram showing an arrangement example of cameras and the like of the vehicle periphery monitoring system as the embodiment.

FIG. 3 is an explanatory diagram showing preconditions for explaining monitor display contents in the embodiment.

FIG. 4 is an explanatory diagram showing the contents displayed on the monitor in the embodiment in comparison with the case of the conventional technique in the traveling state as shown in FIG.

FIG. 5 is an explanatory diagram showing a method of determining a center of rotation and the like when rotating and moving a bird's-eye view image in the embodiment.

FIG. 6 is an explanatory diagram showing a method of calculating a movement amount of a vehicle from a bird's-eye view image in the embodiment.

FIG. 7 is an explanatory diagram showing another method of calculating the movement amount of the vehicle from the bird's-eye view image in the embodiment.

FIG. 8 is an explanatory diagram showing still another method for calculating the movement amount of the vehicle from the bird's-eye view image in the embodiment.

FIG. 9 is a flowchart showing the contents of main control processing executed by software in the embodiment.

FIG. 10 is a flowchart showing the contents of main control processing executed by software in the embodiment.

FIG. 11 is an explanatory diagram showing a monitor display situation in the embodiment.

FIG. 12 is an explanatory diagram showing a new problem to be solved by the embodiment.

FIG. 13 is an explanatory diagram showing a state of a bird's-eye view image displayed on the monitor in a state in which the new problem shown in FIG. 12 is solved in the embodiment.

FIG. 14 is a flowchart showing the content of optional control processing executed by software in the embodiment.

FIG. 15 is a flowchart showing the contents of optional control processing executed by software in the embodiment.

FIG. 16 is an explanatory diagram showing a monitor display situation in the embodiment.

FIG. 17 is a flowchart showing the contents of optional control processing executed by software in the second embodiment.

FIG. 18 is a flowchart showing the contents of optional control processing executed by software in the second embodiment.

FIG. 19 is an explanatory diagram showing a monitor display situation according to the second embodiment.

FIG. 20 is an explanatory diagram showing the contents of a monitor display in another example that can be implemented to achieve the object of the present invention.

FIG. 21 is an explanatory diagram showing the contents of a monitor display in another example that can be implemented to achieve the object of the present invention.

FIG. 22 is an explanatory diagram showing the contents of a monitor display in another example that can be implemented to achieve the object of the present invention.

FIG. 23 is an explanatory diagram showing the contents of a monitor display in another example that can be implemented to achieve the object of the present invention.

[Explanation of symbols]

1 ... Vehicle surroundings monitoring system 11 ... Front camera 13 ... rear camera 15 ... Bird's-eye conversion circuit 17 ... Bird's-eye conversion circuit 21 ... Temporary storage memory 23 ... Temporary storage memory 25 ... CPU 27 ... History image storage memory 29 ... Image memory for display 31 ... Monitor 41 ... Handle angle signal input circuit 43 ... Vehicle speed signal input circuit 45 ... Yaw rate signal input circuit 47 ... Ultrasonic sensor signal input circuit 49: Shift position signal input circuit 50 ... Vehicle figure 60: Synthetic bird's-eye image 61: Real image display section 63 ... Past image display section 81 ... Boundary 83 ... Caution mark 85 ... Comment

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) B60R 21/00 624 B60R 21/00 624C 624E 626 626G G06T 1/00 330 G06T 1/00 330B 3/00 100 3/00 100 300 300 3/20 3/20 3/60 3/60 G08G 1/16 G08G 1/16 C H04N 7/18 H04N 7/18 J (72) Inventor Katsuyuki Imanishi 14 Shimohakaku Iwatani, Nishio City, Aichi Prefecture Address: Japan Auto Parts Research Institute, Inc. (72) Inventor: Makoto Takeichi, 1-1, Showa-cho, Kariya-shi, Aichi Prefecture Denso Co., Ltd. (72) Inventor: Yoshihisa Sato, 1-1, Showa-cho, Kariya-shi, Aichi Shares F term in the company DENSO (reference) 5B057 AA16 CA19 CD02 CD03 CD11 CE08 CH01 CH11 5C054 AA01 CH02 FC11 FD03 FE12 GA01 GA04 GB 11 HA30 5H180 AA01 CC04 CC11 FF33 LL02 LL15 LL17

Claims (15)

[Claims] 1. A photographing means installed at a rear portion of a vehicle, a bird's-eye converting means for converting a vehicle rear image photographed by the photographing means into a bird's-eye image, and a bird's-eye image converted by the bird's-eye converting means are displayed. In a vehicle periphery monitoring system comprising a display means, a bird's-eye view image storage means for storing the bird's-eye view image converted by the bird's-eye view conversion means, a vehicle movement state acquisition means for obtaining a movement state of the vehicle, and the vehicle movement state acquisition means. Image rotation / moving means for rotationally moving or translating the past bird's-eye image stored in the bird's-eye image storage means based on the acquired moving state of the vehicle, and rotation / movement by the image rotating / moving means. A past bird's-eye view image and a current bird's-eye view image obtained by bird-eye-view converting the current vehicle rear image captured by the image-capturing means by the bird-eye-view converting means. A display for displaying a synthetic bird's-eye image generating means for synthesizing to generate a synthetic bird's-eye image and a synthetic bird's-eye image generated by the synthetic bird's-eye image generating means on the display means together with a vehicle figure so that the relationship with the position of the vehicle can be understood. A vehicle periphery monitoring system comprising: a control means. 2. A photographing means installed at a front part of a vehicle, a bird's-eye converting means for converting a vehicle front image photographed by the photographing means into a bird's-eye image, and a bird's-eye image converted by the bird's-eye converting means are displayed. Display means, a bird's-eye view image storage means for storing the bird's-eye view image converted by the bird's-eye view conversion means, a vehicle movement state acquisition means for obtaining the movement state of the vehicle, and a movement of the vehicle obtained by the vehicle movement state acquisition means. An image rotating / moving means for rotating / moving the past bird's-eye image stored in the bird's-eye image storing means based on the state; and a past bird's-eye image rotated / moved by the image rotating / moving means. Generating a synthetic bird's-eye image by synthesizing a current bird's-eye image obtained by the bird's-eye conversion of the current vehicle rear image photographed by the photographing means with the bird's-eye conversion means Vehicle and display control means for displaying the combined bird's-eye view image generated by the combined bird's-eye view image generation means on the display means together with a vehicle figure so that the relationship with the position of the vehicle can be understood. Perimeter monitoring system. 3. A front photographing means installed in a front part of the vehicle, a rear photographing means installed in a rear part of the vehicle, a display means for displaying an image photographed by the front photographing means or the rear photographing means, and a shift. A shift range detecting means for detecting a range, and when the shift range detecting means detects that the shift range is a reverse range, the backward photographing means is used as the photographing means, and it is detected that the range is a forward range. In this case, the vehicle periphery monitoring system is provided with an image capturing direction switching unit that switches the image capturing direction so that the front image capturing unit is used. 4. The vehicle periphery monitoring system according to claim 3, wherein a bird's-eye view conversion unit that converts the image captured by the image capturing unit into a bird's-eye view image, and a bird's-eye image that stores the bird's-eye image converted by the bird's-eye conversion unit. Storage means, vehicle movement state acquisition means for obtaining the movement state of the vehicle, and past bird's-eye view images stored in the bird's-eye view image storage means based on the movement state of the vehicle acquired by the vehicle movement state acquisition means An image rotating / moving means for rotating or moving in parallel, a past bird's-eye image rotated / moved by the image rotating / moving means, and a current image photographed by the photographing means by the bird's-eye converting means. A synthetic bird's-eye image generating unit that synthesizes the converted current bird's-eye image to generate a synthetic bird's-eye image, and a synthetic bird's-eye image generated by the synthetic bird's-eye image generating unit. A vehicle periphery monitoring system comprising: display control means for displaying on the display means so that the relationship between the two positions can be seen. 5. The vehicle periphery monitoring system according to claim 1, wherein a past bird's-eye image stored in the bird's-eye image storage means is a synthetic bird's-eye view generated by the synthetic bird's-eye image generating means. A perimeter monitoring system characterized by comprising a past image updating means for updating based on an image. 6. The vehicle periphery monitoring system according to claim 1, wherein, with respect to the combined bird's-eye image generated by the combined bird's-eye image generation means, a portion corresponding to a past bird's-eye image and a current position. An image processing unit that performs image processing for distinguishing from a portion corresponding to the bird's-eye image is provided, and the display control unit displays the combined bird's-eye image after the image processing by the image processing unit on the display unit. Surrounding monitoring system characterized by being configured in. 7. The vehicle periphery monitoring system according to claim 1, wherein a driver's attention is given to a portion displaying a past bird's-eye image in the combined bird's-eye image displayed on the display unit. A surroundings monitoring system comprising: a caution information display means for displaying predetermined information for prompting the user. 8. The vehicle periphery monitoring system according to claim 1, wherein a portion of the combined bird's-eye image displayed on the display unit that displays a past bird's-eye image is visually checked. An environment monitoring system characterized by comprising a notification means for notifying a driver that it is necessary. 9. The vehicle periphery monitoring system according to claim 1, wherein shift range switching detection means for detecting switching of the shift range of the vehicle, and shift range switching by the shift range switching detection means. And a past image erasing means for erasing a past bird's-eye view image stored in the bird's-eye view image storage means when it is detected that the surroundings monitoring system is provided. 10. The vehicle periphery monitoring system according to claim 1, wherein the vehicle movement state acquisition means calculates a rotation movement amount of the vehicle at the time of turning from a steering wheel angle and a yaw rate. A vehicle surroundings monitoring system comprising a calculating means. 11. The vehicle surroundings monitoring system according to claim 1, wherein the vehicle movement state acquisition means includes a vehicle speed or predetermined information from which vehicle speed can be calculated, and the vehicle movement status acquisition means can detect the vehicle speed when the vehicle does not turn. A vehicle periphery monitoring system comprising a linear movement amount calculation means for calculating a linear movement amount. 12. The vehicle periphery monitoring system according to claim 1, wherein the vehicle movement state acquisition unit compares the past bird's-eye image with the current bird's-eye image to linearly move the vehicle. A vehicle periphery monitoring system comprising a linear movement amount calculation means for calculating an amount. 13. The vehicle periphery monitoring system according to claim 1, wherein the vehicle movement state acquisition means is a vehicle based on a change in a distance between a vehicle and any target detected by a sonar. A vehicle periphery monitoring system comprising: a linear movement amount calculating means for calculating the linear movement amount of 14. A photographing means installed in a vehicle, a bird's-eye converting means for converting an image photographed by the photographing means into a bird's-eye image, and a bird's-eye image storing means for storing the bird's-eye image converted by the bird's-eye converting means. And comparing the past bird's-eye image stored in the bird's-eye image storage means with the current bird's-eye image obtained by bird's-eye conversion of the current vehicle rear image photographed by the photographing means by the bird's-eye conversion means. ,
A vehicle moving state detecting device comprising: a calculating unit that calculates a moving amount or a moving speed of the vehicle. 15. A moving amount or moving speed of a vehicle is calculated based on a sonar installed in the vehicle for detecting an object in the traveling direction and a change in the distance between the vehicle and a predetermined target detected by the sonar. A vehicle movement state detection device comprising a calculation means.