JP2006186840A - Driving support system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a driving support system tending to readily recognize the traveling direction of a vehicle on a display image easily in a driving support system for displaying a photographed image or a plurality of composited images of the photographed images. <P>SOLUTION: The system is provided with a display unit mounted on a vehicle; an imaging apparatus mounted on the vehicle; an image-fetching means for fetching an image photographed by the imaging apparatus with specified timing; a movement detection means for detecting the movement of each part in the imaging apparatus, on the basis of the photographed image fetched by the image-fetching means; a movement direction estimation means for estimating the traveling direction of the vehicle on the photographed image; and a means of combining the indices showing the traveling direction of the vehicle in the photographed image, on the basis of the traveling direction of the vehicle estimated by the movement direction estimating means and displaying the obtained composed image to the display unit. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a driving support system.

  When a driver of an automobile or the like goes out from an alley or a garage, a blind spot is generated by surrounding fences and buildings, so it is difficult to check the surroundings. Also, when moving backward, a blind spot is generated by the vehicle body, making it difficult to check the surroundings. In view of this, a driving support system has been developed in which the vicinity of the vehicle body, which is a blind spot of the driver, is imaged by an in-vehicle camera and the obtained image is displayed on a display such as a car navigation system.

  As a driving support system, there is a system in which a camera that images a front diagonal left direction and a camera that images a front diagonal right direction are installed on the front side of a vehicle body, and the captured images of these cameras are displayed as two screens, respectively. In such a system, the images captured by the two cameras are displayed on separate screens, making it difficult to recognize the space. In view of this, a system has been developed in which panoramic images are generated by combining images captured by these two cameras and the obtained panoramic images are displayed on one screen. Such a system makes it easy to understand the positional relationship between people and objects.

By the way, in a driving support system that displays a panoramic image on one screen, if the traveling direction of the vehicle matches the center of the panoramic image in the left-right direction, the traveling direction of the vehicle can be grasped on the panoramic image. Easy. However, when the direction of travel of the vehicle changes, such as when turning left or right, there is a difference between the direction of the vehicle and the direction of the tire, so the center of the panoramic image in the left-right direction is determined by the tire direction. It will not match. For this reason, it is difficult to grasp the traveling direction of the vehicle on the panoramic image.
Japanese Patent No. 3298851 Japanese Patent No. 3297040 Japanese Patent No. 3435084

  An object of the present invention is to provide a driving support system that makes it easy to recognize the traveling direction of a vehicle on a display image in a driving support system that displays a captured image or a composite image of a plurality of captured images.

  According to a first aspect of the present invention, there is provided a display mounted on a vehicle, an imaging device mounted on the vehicle, an image capturing unit that captures a captured image captured by the imaging device at a given timing, and the image capturing unit. Based on the captured image, motion detection means for detecting the motion of each part in the captured image, and based on the motion of each part in the captured image detected by the motion detection means, the vehicle on the captured image Based on the traveling direction estimation means for estimating the traveling direction, and the traveling direction of the vehicle estimated by the traveling direction estimation means, an index indicating the traveling direction of the vehicle is combined with the captured image, and the obtained combined image is displayed on the display It is characterized by having a means to display on a vessel.

  According to a second aspect of the present invention, in the first aspect of the present invention, it is determined whether or not the vehicle has moved a predetermined amount or more based on the movement of each part in the captured image detected by the movement detecting means. And a reference image storage means for storing the captured image captured this time as a reference image for motion detection when the determination means determines that the vehicle has moved a predetermined amount or more. The detecting means detects movement of each part in the picked-up image based on the picked-up image captured this time by the image pick-up means and the reference image stored most recently by the reference image storing means. The movement direction estimation means is a movement of each part in the captured image detected by the movement detection means when the determination means determines that the vehicle has moved a predetermined amount or more. Based on, and characterized in that for estimating the traveling direction of the vehicle on the captured image.

  The invention described in claim 3 is a display mounted on a vehicle, a plurality of imaging devices mounted on the vehicle, an image capturing unit that captures captured images captured by the plurality of imaging devices at a given timing, and A panoramic image generating unit that generates a panoramic image by combining a plurality of captured images captured by the image capturing unit, and detects movement of each part in the panoramic image based on the panoramic image generated by the panoramic image generating unit Based on the movement of each part in the panoramic image detected by the motion detecting means, the moving direction estimating means for estimating the traveling direction of the vehicle on the panoramic image, and the moving direction estimating means. Based on the traveling direction of the vehicle, an index indicating the traveling direction of the vehicle is combined with the panoramic image, and the resulting combined image is Characterized in that it comprises means for displaying on the display device.

  According to a fourth aspect of the invention, in the third aspect of the invention, it is determined whether or not the vehicle has moved a predetermined amount or more based on the movement of each part in the panoramic image detected by the movement detecting means. A discriminating unit, and a reference image storage unit that stores the panoramic image generated this time as a reference image for motion detection when the discriminating unit determines that the vehicle has moved a predetermined amount or more. The detection means detects movement of each part in the captured image based on the panorama image generated this time by the panorama image generation means and the reference image stored latest by the reference image storage means. The moving direction estimating means is a panorama detected by the motion detecting means when the determining means determines that the vehicle has moved a predetermined amount or more. Based on the movement of each part in the image, and characterized in that for estimating the traveling direction of the vehicle on the panoramic image.

  According to the present invention, in the driving support system that displays a captured image or a composite image of a plurality of captured images, the traveling direction of the vehicle can be easily recognized on the display image.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments in the case where the present invention is applied to a driving support system that images and displays the front side of a vehicle will be described below with reference to the drawings.

  FIG. 1 shows two cameras mounted on an automobile.

  A camera 1 that images the front diagonal left direction and a camera 2 that images the front diagonal right direction are installed on the front side of the vehicle body 10. The viewing angles θ1 and θ2 of the cameras 1 and 2 are about 90 degrees, respectively, and the cameras 1 and 2 are arranged so that the imaging ranges of the cameras 1 and 2 overlap.

  FIG. 2 shows the configuration of the driving support system.

  The driving support system is a panoramic image based on the captured images stored in the frame memories 3 and 4 and the frame memories 3 and 4 that store the captured images of the two cameras 1 and 2 and the cameras 1 and 2 for each frame. A presentation image generating device 5 that generates a presentation image by synthesizing a panoramic image and an index (marker image) indicating the traveling direction of the vehicle, estimating a traveling direction of the vehicle on the panoramic image, and generating a presentation image A frame memory 6 for storing the presentation image generated by the device 5 and a monitor 7 for displaying the presentation image stored in the frame memory 6 are provided. The presentation image generation device 5 is composed of, for example, a microcomputer.

  FIG. 3 shows a processing procedure by the presentation image generating device 5.

  An image captured by the camera 1 is referred to as a first image, and an image captured by the camera 2 is referred to as a second image.

  First, images taken by the cameras 1 and 2 are taken from the frame memories 3 and 4 (step S1). Then, the obtained first image and second image are combined to generate a panoramic image with a viewing angle of approximately 180 degrees (step S2). As is well known, in this image synthesis, a projection transformation matrix for obtaining a position on the other image corresponding to a point on one image is obtained in advance, and one projection transformation matrix is used by using this projection transformation matrix. This is done by converting the coordinates of each point on the image into coordinates on the other image.

  It is determined whether or not the panoramic image generated in step S2 is the first panoramic image generated after the present processing by the presentation image generating device 5 is started (step S3). If the panorama image generated in step S2 is the first panorama image generated after the present processing by the presentation image generator 5 is started, the panorama image generated this time is used for optical flow calculation. After recording as a reference image (step S4), the process returns to step S1.

  If it is determined in step S3 that the panoramic image generated in step S2 is not the first panoramic image generated after the present processing by the presentation image generating device 5 is started, optical flow calculation is performed. Based on the latest recorded panorama image (the reference image recorded in step S4 or step S7 described later and recorded most recently) and the panorama image generated this time The optical flow is calculated (step S5). The optical flow refers to the velocity field of each point on the image and represents the amount of movement. As a method for calculating the optical flow, for example, the method described in Japanese Patent No. 3435084 is used.

Next, based on the calculated optical flow, it is determined whether or not the moving amount of the vehicle from the time when the reference image is generated is greater than or equal to a predetermined value (step S6). Specifically, the horizontal component (left component or right component) of each optical flow calculated this time is extracted. The cumulative value of the absolute value of the horizontal component of each optical flow is calculated. Then, when the cumulative value of the absolute value of the horizontal component of each optical flow is equal to or greater than a predetermined threshold value, it is determined that the amount of movement of the vehicle is equal to or greater than a predetermined value.

  If the moving amount of the vehicle is less than the predetermined value, the process returns to step S1. If the moving amount of the vehicle is greater than or equal to a predetermined value, the panoramic image generated this time is recorded as a reference image for optical flow calculation (step S7).

  Next, a process for estimating the traveling method of the vehicle on the panoramic image generated this time is performed (step S8).

  In a panoramic image, there is little change in the image in the vicinity of the vehicle traveling direction (the vehicle traveling direction determined by the tire direction), and the velocity field (optical flow) is small. On the other hand, the velocity field (optical flow) increases in the direction orthogonal to the traveling direction of the vehicle, that is, in the left and right sides of the traveling direction of the vehicle. Therefore, the direction with the smallest velocity field (optical flow) is estimated as the estimated direction of the vehicle.

  Specifically, the horizontal direction component (left direction component or right direction component) of each optical flow calculated in step S5 is extracted. For each azimuth of the panoramic image (each azimuth between the leftmost azimuth and the rightmost azimuth), a cumulative value of the left direction component of the optical flow (hereinafter referred to as a left direction component cumulative value) is calculated. Further, a cumulative value of the right direction component of the optical flow (hereinafter referred to as a right direction component cumulative value) is calculated for each direction of the panoramic image. Further, for each azimuth of the panoramic image, a value (horizontal component cumulative value) obtained by adding the left component cumulative value (absolute value) and the right component cumulative value (absolute value) is calculated. Then, the direction having the smallest horizontal component accumulation value for each direction of the panoramic image is determined to be the traveling direction of the vehicle.

  FIG. 4 shows the left component cumulative value (curve L) calculated for each azimuth of the panoramic image and the right component cumulative value (curve R) calculated for each azimuth of the panoramic image. The direction corresponding to the intersection of the curve L and the curve R is determined as the traveling direction of the vehicle.

  When the traveling direction of the vehicle is determined in this way, a marker image indicating the traveling direction of the vehicle is synthesized with the panoramic image recorded in step S7, and the obtained synthesized image is stored in the frame memory 6 ( Step S9). The image stored in the frame memory 6 is displayed on the monitor 4. Then, the process returns to step S1. FIG. 5 shows an example of the marker image M displayed in the panoramic image G. The composite image displayed on the monitor 4 is subsequently displayed in step S6 until YES, and a new composite image is generated and stored in the frame memory 6 in step S9.

  In the above embodiment, a panoramic image is generated by combining images captured by two cameras. However, a panoramic image may be generated by combining images captured by three or more cameras. . In addition, a wide-angle lens may be used to capture a wide-angle image with a single camera. In this case, since the image captured from one camera corresponds to the panoramic image, step S2 in FIG. 3 is omitted.

  The present invention can also be applied to a driving support system that images and displays the rear side of the vehicle.

It is a schematic diagram which shows two cameras mounted in the motor vehicle. It is a block diagram which shows the structure of a driving assistance system. 4 is a flowchart showing a processing procedure by a presentation image generation device 3. It is a graph which shows an example of the left direction component cumulative value (curve L) calculated for every azimuth | direction of a panoramic image, and the right direction component cumulative value (curve R) calculated for every azimuth | direction of a panoramic image. 4 is a schematic diagram illustrating an example of a marker image M displayed in a panoramic image G. FIG.

Explanation of symbols

1, 2 Cameras 3, 4 Frame memory 5 Presentation image generator 6 Frame memory 7 Monitor

Claims (4)

Indicator mounted on the vehicle,
An imaging device mounted on a vehicle,
Image capturing means for capturing a captured image captured by the imaging apparatus at a given timing;
Motion detection means for detecting the movement of each part in the captured image based on the captured image captured by the image capture means;
Based on the movement of each part in the captured image detected by the motion detecting means, a moving direction estimating means for estimating the traveling direction of the vehicle on the captured image, and the traveling direction of the vehicle estimated by the moving direction estimating means Based on the above, a means for synthesizing an index indicating the traveling direction of the vehicle with the captured image, and displaying the obtained synthesized image on the display unit,
A driving support system characterized by comprising: Based on the movement of each part in the captured image detected by the motion detection means, it is determined that the vehicle has moved a predetermined amount or more, and the determination means determines that the vehicle has moved a predetermined amount or more. A reference image storage means for storing the captured image captured this time as a reference image for motion detection,
The motion detection means detects the movement of each part in the captured image based on the captured image captured this time by the image capture means and the reference image stored latest by the reference image storage means. The movement direction estimation means is configured to detect the movement direction on the captured image based on the movement of each part in the captured image detected by the motion detection means when the determination means determines that the vehicle has moved a predetermined amount or more. The driving support system according to claim 1, wherein the driving direction of the vehicle is estimated. Indicator mounted on the vehicle,
A plurality of imaging devices mounted on a vehicle;
Image capturing means for capturing captured images captured by the plurality of imaging devices at a given timing;
Panoramic image generation means for generating a panoramic image by combining a plurality of captured images captured by the image capture means;
Motion detection means for detecting the motion of each part in the panoramic image based on the panoramic image generated by the panoramic image generating means;
Based on the movement of each part in the panoramic image detected by the motion detecting means, the moving direction estimating means for estimating the traveling direction of the vehicle on the panoramic image, and the traveling direction of the vehicle estimated by the moving direction estimating means A driving support system comprising: means for synthesizing an indicator indicating a traveling direction of the vehicle on the basis of the panoramic image, and displaying the obtained synthesized image on the display device. Based on the movement of each part in the panoramic image detected by the motion detecting means, it is determined that the vehicle has moved a predetermined amount or more, and the determining means determines that the vehicle has moved a predetermined amount or more. And a reference image storage means for storing the panorama image generated this time as a reference image for motion detection,
The motion detection means detects the movement of each part in the captured image based on the panorama image generated this time by the panorama image generation means and the reference image stored latest by the reference image storage means. And the movement direction estimation means is arranged on the panorama image based on the movement of each part in the panorama image detected by the movement detection means when the determination means determines that the vehicle has moved a predetermined amount or more. The driving support system according to claim 3, wherein the traveling direction of the vehicle is estimated.

Images