JP2010191793A - Alarm display and alarm display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an alarm display and an alarm display method, capable of alarming a driver about the existence of an object having a risk of colliding with a vehicle in which the driver drives, and capable of releasing the alarm after confirming that the driver recognizes the object. <P>SOLUTION: The alarm display includes: a first imaging part (2) for acquiring a surrounding image obtained by photographing the surroundings of the vehicle (10); a dangerous object detecting part (531) for detecting a dangerous object having a risk of colliding with the vehicle (10) from the surrounding image; an alarm image generating part (532) for generating an alarm image which highlights the dangerous object detected on the surrounding image by the dangerous object detecting part (531); a displaying part (3) for displaying the alarm image; and a confirmation determining part (533) for determining whether or not the driver of the vehicle (10) confirms the dangerous object while the alarm image is displayed on the displaying part (3). The displaying part (3) stops the display of the alarm image after the confirmation determining part (533) determines that the driver confirms the dangerous object. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a warning display device and a warning display method, and more particularly to a warning display device and a warning display method that warn a vehicle driver of the presence of an object that may collide with a vehicle.

  In recent years, in order to prevent the occurrence of a collision accident, a warning display device that detects an object that may collide with a vehicle and warns the driver of the vehicle based on the detection result has been studied. (For example, see Patent Document 1). When such an alarm display device detects an object that may collide with a vehicle, it is important to reliably notify the driver of the presence of the object. Therefore, the information display device disclosed in Patent Document 1 as one of the warning display devices identifies an object that may come into contact with the vehicle, and displays two types of warnings according to the degree of danger. When the degree of risk is lower than a predetermined threshold, the information display device warns the driver by recognizing a two-dimensional arrow. On the other hand, when the degree of danger is higher than a predetermined threshold, the information display device issues a warning by causing the driver to recognize a three-dimensional arrow that faces the driver.

JP 2006-250934 A

  On the other hand, if the warning display device continues to issue a warning regarding an object once recognized by the driver, the warning becomes troublesome for the driver, which may cause the driver's ability to concentrate on driving. In view of this, there has been a demand for a technique for canceling the warning at an appropriate timing, while giving a warning that surely informs the driver of the existence of an object that may collide.

  Therefore, the present invention provides a warning display device and a warning display method that warn a driver of the presence of an object that may collide with a vehicle driven by the driver and cancel the warning at an appropriate timing. For the purpose.

According to the first aspect of the present invention, a warning display device is provided as one aspect of the present invention. The warning display device detects a dangerous object that may collide with the vehicle (10) from the first imaging unit (2) that acquires a peripheral image obtained by photographing the periphery of the vehicle (10) and the peripheral image. A dangerous object detection unit (531), a warning image generation unit (532) for generating a first warning image highlighting the dangerous object detected by the dangerous object detection unit (531) on the peripheral image, and a first Display unit (3) for displaying a warning image, and confirmation determination for determining whether or not the driver of the vehicle (10) has confirmed a dangerous object while the first warning image is displayed on the display unit (3). Part (533). The display unit (3) stops displaying the warning image after it is determined by the confirmation determination unit (533) that the driver has confirmed the dangerous object.
The warning display device according to the present invention has the above-described configuration, warns the driver of the presence of an object that may collide with the vehicle that the driver drives, and cancels the warning at an appropriate timing. it can. Therefore, this warning display device can prevent the occurrence of a collision accident and can prevent the driver's ability to concentrate on driving from decreasing.

According to a second aspect of the present invention, the warning display device further includes a second imaging unit (4) installed so as to photograph the driver's face, and the confirmation determination unit (533) While the first warning image is displayed on the part (3), the direction of the driver's face is determined from the driver image obtained by photographing the driver's face acquired from the second imaging unit (4), When it is determined that the face is facing the display unit (3), it is preferable to determine that the driver has confirmed the dangerous object.
By having such a configuration, the warning display device can automatically cancel the warning for the dangerous object recognized by the driver. For this reason, this warning display device does not require the driver to perform an operation for canceling the warning, so the operation burden on the driver can be reduced.

Alternatively, according to the third aspect of the present invention, the warning display device further includes a sound collection unit installed in the vehicle interior, and the confirmation determination unit (533) displays the first warning image on the display unit (3). While being displayed, it is preferable to determine that the driver has confirmed the dangerous object when the sound collected by the sound collecting unit is analyzed and a predetermined word set in advance is detected.
By having such a configuration, the warning display device can automatically cancel the warning for the dangerous object only by making a statement indicating that the driver has recognized the dangerous object. For this reason, this warning display device does not require the driver to perform an operation for canceling the warning, so the operation burden on the driver can be reduced.

According to the fourth aspect of the present invention, the warning image generation unit (532) determines that the confirmation determination unit (533) determines that the driver of the vehicle (10) has confirmed the dangerous object until a predetermined period elapses. During this time, a second warning image is generated that has a less dangerous object highlighting degree than the first warning image, and the display (3) has a confirmation / determination unit (533) that makes the driver of the vehicle (10) dangerous. It is preferable to display the second warning image from when it is determined that the object is confirmed until the predetermined period elapses.
By having such a configuration, the warning display device can reduce a visual load caused by a sudden change in the screen displayed on the display unit after the driver confirms the dangerous object.

According to the description of claim 5, a warning display method is provided as another embodiment of the present invention. According to the warning display method, there is a step of acquiring a peripheral image captured by the first imaging unit (2) and capturing the periphery of the vehicle (10), and there is a risk of collision with the vehicle (10) from the peripheral image. A step of detecting a dangerous object, a step of generating a warning image highlighting the dangerous object on the peripheral image, a step of displaying the warning image on the display unit (3), and a warning image on the display unit (3). While being displayed, the step of determining whether or not the driver of the vehicle (10) has confirmed the dangerous object, and the warning on the display unit (3) after determining that the driver has confirmed the dangerous object Stopping the display of the image.
The warning display method according to the present invention has the steps described above to warn the driver of the presence of an object that may collide with the vehicle that the driver drives, and cancel the warning at an appropriate timing. it can. For this reason, this warning display method can prevent the occurrence of a collision accident and also prevent the driver's ability to concentrate on driving from decreasing.

  The reference numerals in parentheses attached to the above-described parts are examples that show the correspondence with specific means described in the embodiments described later.

1 is a schematic configuration diagram of a warning display device according to an embodiment of the present invention. It is a functional block diagram of the control part of the warning display apparatus which concerns on one Embodiment of this invention. It is the schematic which shows the relationship between the normal vector showing a driver | operator's face direction, and a display. It is an operation | movement flowchart of a warning display and warning cancellation process controlled by the control part of the warning display apparatus which concerns on one Embodiment of this invention.

Hereinafter, a warning display device according to one embodiment will be described with reference to the drawings.
This warning display device detects a dangerous object that is estimated to possibly collide with a vehicle from an image acquired from a camera that captures the periphery of the vehicle. When the dangerous object is detected, the warning display device highlights the dangerous object on a display installed in the vehicle to warn the driver that there is a risk of collision. Furthermore, the warning display device cancels the warning when it detects from the driver's reaction to the warning that the driver has confirmed the presence of the object.

  FIG. 1 is a schematic configuration diagram of a warning display device according to one embodiment. As shown in FIG. 1, the warning display device 1 includes a vehicle exterior camera 2, a display 3, a vehicle interior camera 4, and a controller 5. The vehicle exterior camera 2, the display 3, the vehicle interior camera 4, and the controller 5 are connected to each other by a control area network (hereinafter referred to as CAN) 6. In FIG. 1, for convenience of explanation, each component of the warning display device 1 and the shape and size of the vehicle 10 are different from actual ones.

The vehicle exterior camera 2 is a CCD or C-MOS 2D detector composed of a photoelectric converter sensitive to visible light, and an image of structures around the vehicle 10 is connected to the 2D detector. An imaging optical system for imaging is included. And the vehicle exterior camera 2 is installed so that the front area | region of a vehicle may be image | photographed. In addition, the camera 2 outside the camera performs photographing at regular time intervals (for example, 1/30 seconds). The outside camera 2 is a color image in which a peripheral image obtained by photographing the front area of the vehicle 10 has, for example, horizontal 640 pixels × vertical 480 pixels, and the luminance of each pixel is represented by 256 gradations of red, green, and blue, respectively. Generate as The vehicle camera 2 may generate the peripheral image as a gray image corresponding to the illuminance of near infrared light within the imaging range. The warning display device 1 may have a plurality of outside cameras. For example, as will be described later, when the control unit 53 detects a dangerous object that may collide with the vehicle 10 using stereo matching, one unit is provided in the vicinity of the left and right ends of the front window and the ceiling in the vehicle interior. In addition, a camera outside the vehicle that captures the front of the vehicle 10 may be installed. In addition, in order to photograph the rear of the vehicle 10, a camera outside the vehicle may be installed so as to photograph the rear region of the vehicle 10 in the vicinity of the rear window and the ceiling in the vehicle interior.
The vehicle exterior camera 2 sequentially transmits a peripheral image obtained by photographing to the controller 5. In addition, when the vehicle outside camera which image | photographs the front area of a vehicle and the vehicle outside camera which image | photographs the back area of a vehicle are attached, the controller 5 selects only the image from the vehicle outside camera which image | photographs the direction which the vehicle is advancing To get. For this purpose, the controller 5 acquires a shift position signal representing the position of the shift lever from an electronic control unit (not shown) of the vehicle 10 via the CAN 6. Then, when the shift position signal is a drive position indicating that the vehicle 10 moves forward, the controller 5 acquires a peripheral image from an outside camera that captures a front area of the vehicle. On the other hand, when the shift position signal is a reverse position indicating that the vehicle 10 is moving backward, the controller 5 acquires a peripheral image from an outside camera that captures the rear area of the vehicle.

  The display 3 is composed of, for example, a liquid crystal display or an organic EL display, and is arranged in the instrument panel so that the display screen faces the driver. The display 3 may be arranged independently of the instrument panel. Alternatively, the display 3 may be a display of another device such as a navigation device. The display 3 is controlled by the controller 5 and displays an image received from the controller 5.

The in-vehicle camera 4 captures the driver's face. For this purpose, the in-vehicle camera 4 forms a two-dimensional detector composed of a photoelectric converter sensitive to visible light, such as CCD or C-MOS, and an image of the driver's face on the two-dimensional detector. An imaging optical system. The in-vehicle camera 4 is attached to the top of the display 3 so that the driver's face is photographed from the front when the driver looks at the display 3. The in-vehicle camera 4 takes images at regular time intervals (for example, 1/30 seconds). The in-vehicle camera 4 is a color image in which the driver image obtained by photographing the driver's face has, for example, horizontal 640 pixels × vertical 480 pixels, and the luminance of each pixel is represented by 256 gradations of red, green, and blue. Generate as
The in-vehicle camera 4 sequentially transmits a driver image obtained by photographing to the controller 5.

The controller 5 includes a storage unit 51, a communication unit 52, and a control unit 53. The storage unit 51 includes, for example, a semiconductor memory such as an electrically rewritable nonvolatile memory and a volatile memory. The storage unit 51 stores various programs and parameters for controlling the warning display device 1, temporary calculation results by the control unit 53, and the like.
Furthermore, the storage unit 51 associates a recognized flag, which is detected by the control unit 53 and indicates that a dangerous object that may collide with the vehicle 10 has already been recognized by the driver, with information related to each dangerous object. Remember.
The communication unit 52 includes a communication interface that communicates with the outside camera 2, the display 3, the in-vehicle camera 4 and the CAN 6 and a control circuit thereof.

The control unit 53 includes one or a plurality of processors (not shown) and their peripheral circuits. And the control part 53 controls the warning display apparatus 1 whole.
FIG. 2 shows a functional block diagram of the control unit 53. As illustrated in FIG. 2, the control unit 53 includes a dangerous object detection unit 531, a warning image generation unit 532, and a confirmation determination unit 533. Each unit included in the control unit 53 is implemented as a functional module realized by a computer program executed on a microprocessor included in the control unit 53, for example.

  The dangerous object detection unit 531 detects a dangerous object that may collide with the vehicle 10 from the peripheral image obtained from the outside camera 2. Therefore, the dangerous object detection unit 531 performs pattern matching while changing the relative positional relationship between the template corresponding to the object to be detected and the latest peripheral image, for example, and thereby the template and its latest periphery. A correlation value with the image is calculated. Then, when the correlation value between the template and the surrounding image satisfies a predetermined condition, the dangerous object detection unit 531 selects an area corresponding to the template on the surrounding image as a candidate area where the dangerous object may exist. Detect as. Note that a plurality of templates may be prepared for each object to be detected, such as a pedestrian, a vehicle, and a bicycle. These templates are stored in advance in the storage unit 51, and the dangerous object detection unit 531 reads the templates from the storage unit 51 and uses them when pattern matching is executed or when the warning display device 1 is activated. The correlation value is calculated, for example, by dividing the sum of absolute values of luminance value differences between corresponding pixels of the template and the surrounding image by the total number of pixels where the template and the surrounding image overlap. Furthermore, it is preferable that the predetermined condition is set as a range of correlation values that can detect a dangerous object with high accuracy and has a low probability of erroneously detecting an object that is not a dangerous object as a candidate region. For example, the predetermined condition may be that the correlation value calculated as described above is not more than a value obtained by multiplying the maximum value of the correlation value by 0.1 or 0.2. In addition, when calculating the correlation value between the template and the surrounding image, the dangerous object detection unit 531 may use any of various known methods for calculating the correlation value. For example, when the dangerous object detection unit 531 calculates the normalized correlation value between the template and the surrounding image, the predetermined condition is that the calculated normalized correlation value is equal to the maximum value that the normalized correlation value can take. It can be set to be a value obtained by multiplying 0.9 or 0.8.

Further, the dangerous object detection unit 531 may detect the candidate area using another method. For example, the dangerous object detection unit 531 obtains the distance from the vehicle 10 to each point in the peripheral image by performing stereo matching on the latest peripheral image. Based on the distance, the dangerous object detection unit 531 may detect a candidate area using a discriminator that detects an object having a predetermined shape corresponding to a pedestrian, a vehicle, a bicycle, or the like as a candidate area. . Such discriminators are disclosed in, for example, three people outside Soga, “Pedestrian Detection with Stereo Vision”, Proc. Of the ICDE '05. Further, the warning display device 1 has two outside cameras installed at different positions as described with respect to the outside camera 2, and the dangerous object detection unit 531 has two images taken almost simultaneously by these outside cameras. Stereo matching can be performed based on surrounding images. Alternatively, the dangerous object detection unit 531 may calculate the distance from the vehicle 10 to each point in the peripheral image using a motion stereo technique from a plurality of peripheral images acquired in time series from the one outside camera 2. Good. Such motion stereo technology is disclosed in, for example, Japanese Patent Application Laid-Open No. 2007-256029 or K. Fintzel et al., “3D Parking Assistant System”, 2004 IEEE Intelligent Vehicles Symposium, 2004.
Furthermore, the vehicle exterior camera 2 can be a camera having a function of measuring the distance to the object to be photographed for each pixel, such as a light wave ranging image sensor. In this case, the dangerous object detection unit 531 acquires a distance measurement signal corresponding to each pixel in the peripheral image from the outside camera 2 through the CAN 6, and determines a distance to each point in the peripheral image based on the distance measurement signal. May be.

  Further, the dangerous object detection unit 531 divides the latest peripheral image into a plurality of small regions, detects HOG (Histograms of Oriented Gradients) feature values or Haar-like features from the small regions, and extracts these feature values from the Adaboost. By inputting to the discriminator, it may be determined whether or not the small region is a candidate region. For details of the Haar-like feature and the Adaboost classifier, see, for example, Paul Viola and Michael Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", IEEE CVPR, vol.1, pp.511-518, 2001. Is disclosed.

Next, the dangerous object detection unit 531 determines whether the object shown in the candidate area detected from the latest peripheral image is the same as the object shown in any candidate area detected from the past peripheral image. To determine. Therefore, the dangerous object detection unit 531 can use various tracking techniques for tracking the same object on a plurality of images acquired in time series.
For example, the dangerous object detection unit 531 determines the moving direction and moving speed of the object based on the positional deviation amount of the candidate area corresponding to the same object respectively detected from a plurality of time-series peripheral images acquired in the past. presume. Then, the dangerous object detection unit 531 estimates the position of the candidate area in the latest peripheral image from the moving direction and moving speed thereof and the position of the candidate area before the predetermined number of frames. When the candidate area is detected at the estimated position and the vicinity thereof, the dangerous object detection unit 531 has the same object moved to the candidate area as the object reflected in the candidate area before the predetermined number of frames. Is determined.

  For example, the dangerous object detection unit 531 is acquired by the dangerous object detection unit 531 before (m + 1) frames and the position of the candidate area detected from the surrounding image acquired m frames before (where m is a natural number). From the position difference of the candidate area detected from the surrounding image, the estimated movement direction and the estimated movement speed at the time point m frames before the object shown in the candidate area are calculated. Then, the dangerous object detection unit 531 calculates the estimated position of the object in the latest peripheral image, assuming that the object shown in the candidate area advances according to the estimated movement direction and estimated movement speed at the time point m frames before. Therefore, the dangerous object detection unit 531 moves to the position of the candidate area before m frames along the estimated movement direction at that time, and obtains the latest peripheral image at the estimated movement speed at that time and the surrounding image before m frames. By adding a distance obtained by multiplying the time difference at the time of acquisition, the estimated position of the object in the latest peripheral image is calculated. Similarly, the dangerous object detection unit 531 calculates the estimated position of the object in the latest peripheral image of the object shown in each candidate area from a plurality of frames acquired in the past. Then, the dangerous object detection unit 531 multiplies each estimated position obtained by a weighting factor that becomes larger as the estimated position obtained from the new surrounding image, and performs weighted averaging, thereby finalizing the object in the latest surrounding image. The estimated position can be determined. As a result, the dangerous object detection unit 531 can accurately estimate the latest position of the candidate object even if an occlusion occurs where a candidate area corresponding to the same candidate object cannot be detected temporarily.

  However, among candidate areas detected from a peripheral image acquired one frame before, a candidate area that does not correspond to any of candidate areas detected from a peripheral image acquired before that (hereinafter referred to as an initial detection candidate area) The dangerous object detection unit 531 sets a predetermined range centered on the initial detection candidate region as a search range. Then, when a candidate area is detected within the search range in the latest peripheral image, the dangerous object detection unit 531 detects the first detection candidate area detected from the object shown in the candidate area and the peripheral image one frame before. It is determined that the objects shown in are identical. The search range is determined according to the assumed moving speed of the object to be detected. For example, if the object to be detected is a person, the search range is a range that can move at a speed of 10 km / h. If the object to be detected is a vehicle, the search range is set as a range that can be moved at a speed of 60 km / h.

The dangerous object detection unit 531 was able to estimate the position of the candidate area for a predetermined number of frames (for example, the number of frames included in one second) and to detect the candidate area continuously in the vicinity of the estimated position. In this case, it may be determined that the same object appears in the candidate area detected over the plurality of peripheral images. Such a tracking method is disclosed, for example, in B. Lucas and T. Kanade, “An iterative image registration technique with an Application to Stereo Vision”, 1981, Proc. Of the DARPA IU Workshop, pp. 121-130. ing.
The dangerous object detection unit 531 may use a Kalman filter or an αβ filter in order to determine the estimated position. The αβ filter is disclosed in, for example, the URL of the Internet: http://en.wikipedia.org/wiki/Alpha_beta_filter (searched on February 3, 2009).

  When the dangerous object detection unit 531 can detect the candidate area from the latest peripheral image, the object shown in the candidate area is reflected in any of the candidate areas detected from the past peripheral image as described above. It is determined whether or not the object is the same. If it is determined that they are the same, the dangerous object detection unit 531 attaches the same identification number as the candidate area detected from the corresponding past peripheral image to the candidate area detected from the latest peripheral image. On the other hand, when the candidate area detected from the latest peripheral image is the first detection candidate area, the dangerous object detection unit 531 attaches an identification number different from any of the other candidate areas to the candidate area. Then, the dangerous object detection unit 531 associates the identification number with the position and size of the candidate area on the peripheral image and the detection time, and stores them in the storage unit 51.

  On the other hand, when the dangerous object detection unit 531 cannot detect the candidate area at the estimated position obtained from the candidate area detected from the past peripheral image and the vicinity thereof over the peripheral images of a plurality of frames, The tracking of the object corresponding to the candidate area detected from the peripheral image is stopped. Then, the dangerous object detection unit 531 deletes the identification number of each candidate area corresponding to the object, the position, size, and detection time of the candidate area on the peripheral image from the storage unit 51.

The dangerous object detection unit 531 determines whether there is a possibility that the object collides with the vehicle 10 based on the moving direction and moving speed of the object represented in the candidate area being tracked. For example, when the moving direction of the object is toward the center of the peripheral image, it is considered that the object is approaching the front of the vehicle 10. Therefore, the dangerous object detection unit 531 detects the object as a dangerous object. Alternatively, the dangerous object detection unit 531 estimates the distance to the object from the measurement result of the size of the candidate area or the distance from the vehicle 10 by stereo matching, and whether or not the object collides with the vehicle 10 based on the estimated distance. It may be used for the determination. In this case, the dangerous object detection unit 531 estimates the position of the vehicle 10 after a predetermined time from the speed and the steering angle of the vehicle 10 when the latest peripheral image is acquired. Also, the dangerous object detection unit 531 estimates the position of the object after a predetermined time from the moving speed and moving direction of the object represented in the candidate area. If the distance between the estimated position of the vehicle 10 and the estimated position of the object falls within a predetermined range where there is a risk of collision, the dangerous object detection unit 531 detects the object as a dangerous object. The predetermined range can be set to 1 m, for example. The predetermined range may be set so as to increase as the speed of the vehicle 10 increases.
The speed and steering angle of the vehicle 10 can be acquired from the vehicle speed sensor (not shown) and the steering angle sensor (not shown) mounted on the vehicle 10 through the CAN 6.

  When the dangerous object detection unit 531 detects the dangerous object, information indicating the position and size of the area corresponding to the dangerous object (for example, the coordinates of the upper left end point of the candidate area and the horizontal and vertical lengths of the candidate area) ) To the warning image generation unit 532.

  When a dangerous object is detected and the driver has not confirmed the dangerous object, the warning image generation unit 532 generates a warning image in which the detected dangerous object is highlighted on the peripheral image. On the other hand, the warning image generation unit 532 does not perform highlighting for dangerous objects that have already been confirmed by the driver. As a result, the warning display device 1 prevents the driver from feeling bothered by continuing to be warned about the dangerous object even though the driver has already recognized the dangerous object, and driving the driver. It is possible to prevent a decrease in concentration on the subject. When the recognized flag is stored in the storage unit 51 in association with the identification number of the candidate area corresponding to the dangerous object, the warning image generation unit 532 indicates that the dangerous object has already been confirmed by the driver. judge.

  For example, the warning image generation unit 532 uses the brightness value of the pixel located on the contour of the region corresponding to the dangerous object detected by the dangerous object detection unit 531 in the latest peripheral image acquired from the outside camera 2 as the contour. A value obtained by inverting the average luminance value of neighboring pixels. Alternatively, the warning image generation unit 532 may change the color of the pixel located on the outline of the region corresponding to the dangerous object to a color that is conspicuous for the driver, for example, red. Further, the warning image generation unit 532 changes the color or luminance value of the pixel on the contour alternately as described above for each frame, and makes the contour blink for the driver. May be. Alternatively, the warning image generation unit 532 may enlarge or reduce the outline for each frame.

Further, the warning image generation unit 532 displays the degree of highlighting of the dangerous object until the predetermined period elapses from the final confirmation time when the dangerous object is detected and confirmed by the driver. Generates a warning image weaker than when the dangerous object is not confirmed. By displaying such a warning image on the display 3 during a predetermined period, the warning display device 1 prevents the image displayed on the display 3 from changing suddenly, and the visual burden on the driver is reduced. Can be reduced. Note that the predetermined time can be, for example, 3 seconds.
In order to weaken the degree of highlighting, for example, the warning image generation unit 532 changes the luminance value and color of the outline of the region corresponding to the dangerous object to the luminance value and color of the corresponding pixel of the original peripheral image as time passes. You may make it approach. In this way, by changing the luminance value and color of the highlighted contour pixel in time, the warning image generation unit 532 fades out the frame attached to the contour of the dangerous object to the driver. You can make it look like

The warning image generation unit 532 passes the generated warning image to the control unit 53.
The control unit 53 displays the warning image generated by the warning image generation unit 532 on the display 3 while there is a dangerous object that has not been confirmed by the driver and until a predetermined period has elapsed from the final confirmation time. On the other hand, when the dangerous object is not detected, or when the dangerous object is detected but the driver confirms the presence of the dangerous object, the control unit 53 does not highlight the dangerous object. An image is displayed on the display 3.

The confirmation / determination unit 533 allows the driver to view the display 3 from the driver image obtained from the in-vehicle camera 4 when there is a dangerous object that the driver has not confirmed and a warning image is displayed on the display 3. It is determined whether or not. Then, when it is determined that the driver has seen the display 3, the confirmation determination unit 534 determines that the driver has confirmed the dangerous object highlighted in the warning image.
Specifically, the confirmation determination unit 533 checks the direction of the driver's face from the driver image. Therefore, the confirmation determination unit 533 extracts a skin color pixel having a skin color in the driver image in order to extract an area where the driver's face is estimated to be captured. In order to extract the skin color pixel, the confirmation determination unit 533 converts the color system of the driver image into a color system that can be separated into color information and luminance, for example, a YCrCb color system or an HSV color system. . And the confirmation determination part 533 makes the pixel in which the value regarding the axis | shaft showing color information is contained in the predetermined range equivalent to skin color in the color system after conversion as a skin color pixel. The predetermined range can be, for example, a range of 6 to 38 with respect to the H axis in the HSV color system, and 80 to 130 with respect to the Cr axis and 130 with respect to the Cb axis in the case of the YCrCb color system. Can range from 170 to 170.
When the skin color pixel is extracted, the confirmation determination unit 533 performs a morphological closing operation to obtain a region where the skin color pixel and the pixel surrounded by the skin color pixel are connected. And the confirmation determination part 533 detects the area | region as a driver | operator's face area.

Next, the confirmation determination unit 533 detects the cornea and nostril which are characteristic points from the face region. Here, the cornea and the nostril have a shape that is relatively close to a circle on the driver image. Therefore, in the present embodiment, the confirmation determination unit 533 detects the cornea and the nostril from the face area using a method of detecting a circle. In order to detect a circle, the confirmation determination unit 533 can use, for example, a Hough transform or a separability filter.
When the Hough transform is used, the confirmation determination unit 533 first performs a difference calculation between neighboring pixels in the face area, and the absolute value of the difference becomes equal to or greater than a predetermined value (for example, the average value of the absolute differences in the face area). Edge pixels are extracted. Then, the confirmation determination unit 533 performs Hough transform using the center coordinates and radius of the circle as variables, and if there is a point where the number of edge pixels is more than a predetermined number that is recognized as a circle, a circular region represented by the point Are extracted as nostril or cornea candidate regions.
The separability filter is a difference value obtained by subtracting the average luminance value in the circular area corresponding to the size of the cornea or nostril on the driver image from the average luminance value in the annular area set around the circular area. Is a filter for calculating as a degree of separation. Then, the confirmation determination unit 533 calculates the degree of separation by changing the position of the degree of separation filter in the face region, and the region corresponding to the circular region of the degree of separation filter when the degree of separation is equal to or greater than a predetermined threshold. Are extracted as nostril or cornea candidate regions.

  The confirmation determination unit 533 is a candidate region that satisfies the positional relationship corresponding to the positional relationship between the cornea and the nostril of both eyes, that is, the distance between the upper two points between the lower two points. Candidate regions located at the apexes of a substantially inverted trapezoid that is larger than the distance are defined as nostrils or corneas, respectively. Of course, the confirmation determination unit 533 uses the two candidate regions positioned at the upper vertex of the substantially inverted trapezoid as the cornea, respectively, while the two candidate regions positioned at the lower vertex of the approximately inverted trapezoid as the nostrils, respectively. And

Next, the confirmation determination unit 533 obtains a plane corresponding to the front of the face from the middle point of the two nostrils on the driver image and the center point of the left and right corneas, and calculates a normal vector of the plane. Face orientation.
FIG. 3 shows the relationship between the normal vector representing the driver's face direction and the display 3. In FIG. 3, A and B are the center points of the left and right corneas, respectively, and C is the midpoint of the two nostrils. N represents a normal vector. Here, in the three-dimensional real coordinate system, the positional relationship between the center points A and B of the left and right corneas and the midpoint C of the two nostrils varies among individuals but can be expressed as follows. it can.
d (A, C) = d (B, C)
d (A, B) = 1.0833d (A, C) (1)
d (A, B) = 6.5cm
Here, d (X, Y) is the Euclidean distance between the point X and the point Y. Further, it is assumed that the points A, B, and C are mapped to points a, b, and c on the driver image according to the following formula.
a = (1 / α) KA
b = (1 / β) KB (2)
c = (1 / γ) KC
Α, β, and γ are unknown scaling factors, K is a camera internal parameter matrix, and the value of each element of K is obtained in advance by the focal length, the angle of view, and the like of the imaging optical system of the in-vehicle camera 4. It is done.

The confirmation determination unit 533 obtains the coordinates of A, B, and C from the above expressions (1) and (2). Then, the confirmation determination unit 533 calculates a normal vector n representing the face direction according to the following equation.
n = AC × AB / | AC × AB | (3)
In the equation (3), “×” represents an outer product of vectors, and AB and AC are a vector between points A and B and a vector between points A and C, respectively.
The technique for estimating the face orientation described above is disclosed in, for example, Kaminski et al., “Head orientation and gaze detection from a single image”, 2006, International Conference of Computer Vision Theory and Applications.

The confirmation / determination unit 533 obtains an angle θ between the normal vector n obtained by the expression (3) and the straight line L connecting the midpoints A and B of the driver's corneas and the center of the display 3. In the present embodiment, the in-vehicle camera 4 is attached to the upper part of the display 3 so that the driver's face is photographed from the front when the driver looks at the display 3, so the normal vector n And the angle formed by the optical axis of the photographing optical system of the in-vehicle camera 4 may be θ. Since the center position of the display 3 and the direction of the optical axis of the photographing optical system in the three-dimensional real coordinate system are known when the display 3 and the in-vehicle camera 4 are installed, the coordinates of the center position of the display 3 or the photographing optical system The direction vector representing the direction of the optical axis is stored in the storage unit 51 in advance, and the confirmation determination unit 533 reads them from the storage unit 51 and uses them when calculating θ.
And the confirmation determination part 533 determines with the driver _| operator having looked at the display 3, when the angle (theta) is less than predetermined _| prescribed angle (theta) _th considered that the person is gazing at the display 3. FIG. The predetermined angle _θth can be optimized experimentally, and is set to 5 °, for example.

  Further, the confirmation determination unit 533 may determine whether or not the driver is watching the display 3 by another method. For example, the confirmation determination unit 533 is an isosceles side in which a triangle whose apexes are the center points A and B of the corneas of both eyes and the midpoint C of the nostril has the side AB as a base, and the length of the side AB is substantially equal to the length of the side AC. If it is a triangle, it may be determined that the driver has seen the display 3.

  Note that the confirmation determination unit 533 selects a dangerous object highlighted in the warning image when it is determined that the driver has viewed the display 3 in each of the plurality of driver images acquired during a certain period. The driver may determine that the driver has confirmed. In this case, the certain period can be set to, for example, a minimum period necessary for recognizing a moving object or a period slightly longer than that period, for example, 1 second.

When it is determined that the driver has confirmed the dangerous object, the confirmation determination unit 533 confirms that the driver has confirmed the dangerous object for all the dangerous objects detected in the latest peripheral image at that time. Set the recognized flag to indicate. Then, the confirmation determination unit 533 stores the recognized flag in the storage unit 51 in association with the identification number of the candidate area corresponding to the dangerous object. Thereafter, the control unit 53 checks whether the recognized flag associated with the identification number of the candidate area corresponding to the dangerous object stored in the storage unit 51 is stored, so that the dangerous object is driven. It can be determined whether or not it has already been confirmed by the person.
Further, the confirmation determination unit 533 acquires the current time from the processor constituting the control unit 53 and stores the current time in the storage unit 51 as the final confirmation time.

FIG. 4 shows a flowchart of the warning display and warning release operation of the warning display device 1 controlled by the control unit 53. In addition, whenever the control part 53 acquires a new periphery image from the camera 2 outside a vehicle, the operation | movement shown in this flowchart is repeated.
First, the dangerous object detection unit 531 of the control unit 53 detects a candidate area that may be a dangerous object that may collide with the vehicle 10 from the latest peripheral image acquired from the outside camera 2 (step S101). . Next, when one or more candidate areas are detected, the dangerous object detection unit 531 tracks the candidate areas detected in the peripheral image acquired in the past and the candidate areas detected in the latest peripheral image. Processing is executed (step S102). The dangerous object detection unit 531 associates a candidate area detected in the past acquired peripheral image with a candidate area estimated to correspond to the same object among the candidate areas detected in the latest peripheral image. The dangerous object detection unit 531 obtains the moving direction and moving speed of each candidate area based on the positional shift of each candidate area in a plurality of time-series surrounding images corresponding to the same object. Based on the moving direction or moving speed of the object, the dangerous object detection unit 531 determines whether a dangerous object that may collide with the vehicle 10 is detected (step S103).

When the dangerous object is not detected in the latest peripheral image, the control unit 53 causes the display unit 3 to display the peripheral image acquired from the vehicle exterior camera 2 (step S104). Then, the control unit 53 ends the process.
On the other hand, when a dangerous object is detected in the latest peripheral image, the control unit 53 determines whether or not the driver has confirmed all the detected dangerous objects (step S105). If the recognized flag is stored in the storage unit 51 in association with the identification number of the candidate area corresponding to the dangerous object, the control unit 53 can determine that the dangerous object has been confirmed by the driver. On the other hand, if there is no recognized flag associated with the identification number of the candidate area corresponding to the dangerous object in the storage unit 51, the control unit 53 can determine that the dangerous object has not been confirmed by the driver.

When there is a dangerous object that has not been confirmed by the driver, the warning image generation unit 532 of the control unit 53 generates a warning image that highlights the dangerous object that has not been confirmed by the driver, and passes the warning image to the control unit 53. Then, the control unit 53 displays the warning image on the display unit 3 (step S106). Thereafter, the confirmation determination unit 533 of the control unit 53 determines whether or not the driver has confirmed an unconfirmed dangerous object (step S107). As described above, the confirmation determination unit 533 checks the direction of the driver's face based on the driver image acquired from the in-vehicle camera 4 and determines that the driver has viewed the display 3 from the direction of the face. The driver determines that an unidentified dangerous object has been confirmed.
When it is determined that the driver has confirmed an unconfirmed dangerous object, the confirmation determination unit 533 sets a recognition flag for all the dangerous objects detected in the latest peripheral image (step S108). Then, the confirmation determination unit 533 stores the recognition flag in the storage unit 51 in association with the identification number of the candidate area corresponding to the corresponding dangerous object. Further, the confirmation determination unit 533 acquires the current time from the processor constituting the control unit 53 and stores the current time in the storage unit 51 as the final confirmation time.
After step S108 or when the confirmation determination unit 533 determines in step S107 that the driver has not confirmed an unconfirmed dangerous object, the control unit 53 ends the process.

On the other hand, when it is determined in step S105 that all dangerous objects have been confirmed by the driver, the control unit 53 determines whether or not the current time is within a predetermined period from the last confirmation time stored in the storage unit 51. (Step S109). If the current time has passed the predetermined period from the final confirmation time, the control unit 53 proceeds to step S104.
On the other hand, if the current time is within a predetermined period from the final confirmation time, the warning image generation unit 532 generates a warning image in which the degree of emphasis on the dangerous object is weaker than when the driver has not confirmed the dangerous object. Then, it is passed to the control unit 53. And the control part 53 displays the warning image on the display part 3 (step S110). Thereafter, the control unit 53 ends the process.

  As described above, the warning display device according to an embodiment of the present invention detects a dangerous object estimated to possibly collide with a vehicle from an image acquired from a camera that captures the surroundings of the vehicle. To do. This warning display device displays a warning image highlighting the dangerous object on the display installed in the vehicle when a dangerous object is detected, thereby indicating that there is a risk of collision with the driver. Can be warned. Further, the warning display device can confirm that the warning image has been seen from an image obtained by photographing the driver's face, so that the driver can cancel the warning without performing a special operation for the warning. Therefore, this warning display device can reliably warn the driver that there is a risk that the vehicle will collide with something, and can automatically cancel the warning once confirmed by the driver. Can prevent a decrease in concentration.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, the confirmation determination unit of the control unit may determine whether or not the dangerous object has been confirmed by analyzing the driver's voice. In this case, the warning display device according to an embodiment of the present invention includes a microphone installed in the vehicle interior, for example, the steering, instead of the in-vehicle camera in the above embodiment. The driver's voice collected by the microphone is sent to the controller. Further, the confirmation determination unit analyzes the sound collected by the microphone using, for example, a hidden Markov model while there is a dangerous object that has not been confirmed by the driver and the warning image is displayed on the display. It is determined whether or not a predetermined word (for example, “confirmation”, “seen”, etc.) that indicates that the dangerous object has been confirmed can be detected. Then, the confirmation determination unit determines that the driver has confirmed the presence of the dangerous object when the predetermined word can be detected. The predetermined words may be set freely by the driver by inputting the predetermined words to the controller through a microphone in advance and storing them in the storage unit of the controller.

Moreover, the warning display apparatus which concerns on one Embodiment of this invention may have the microphone installed in the vehicle interior with the vehicle camera in said embodiment. The confirmation determination unit of the control unit confirms that the driver has viewed the display based on the driver image while there is a dangerous object that the driver has not confirmed and the warning image is displayed on the display. If the predetermined word indicating that the dangerous object has been confirmed can be detected from the sound collected by the microphone, the driver may determine that the presence of the dangerous object has been confirmed. Alternatively, while there is a dangerous object that has not been confirmed by the driver and the warning image is displayed on the display, the confirmation determination unit confirms that the driver has viewed the display based on the driver image. Alternatively, if a predetermined word indicating that the dangerous object has been confirmed can be detected from the sound collected by the microphone, the driver may determine that the presence of the dangerous object has been confirmed.
In addition, the warning display device can be used to display a display when no dangerous object is detected, or when there is no dangerous object that has not been confirmed by the driver and a predetermined period has elapsed since the last confirmation time of the dangerous object. You may turn off the screen. In this case, the warning display device may gradually darken the screen of the display until a predetermined period elapses from the final confirmation time. Furthermore, the warning display device may stop displaying the warning image immediately after the confirmation determination unit determines that the driver has confirmed the dangerous object.
As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Warning display apparatus 2 Camera outside vehicle 3 Display 4 Camera in vehicle 5 Controller 6 Control area network 51 Memory | storage part 52 Communication part 53 Control part 531 Dangerous object detection part 532 Warning image generation part 533 Confirmation determination part

Claims (5)

A first imaging unit (2) for acquiring a peripheral image obtained by photographing the periphery of the vehicle (10);
A dangerous object detection unit (531) for detecting a dangerous object that may collide with the vehicle (10) from the surrounding image;
A warning image generation unit (532) for generating a first warning image highlighting the dangerous object detected by the dangerous object detection unit (531) on the peripheral image;
A display unit (3) for displaying the first warning image;
A confirmation determination unit (533) for determining whether or not the driver of the vehicle (10) has confirmed the dangerous object while the first warning image is displayed on the display unit (3);
Have
The display unit (3) stops displaying the warning image after it is determined by the confirmation determination unit (533) that the driver has confirmed the dangerous object.
A warning display device characterized by that. A second imaging unit (4) installed so as to photograph the driver's face;
The confirmation determination unit (533) captures the driver's face acquired from the second imaging unit (4) while the first warning image is displayed on the display unit (3). The direction of the driver's face is determined from the driver image, and when it is determined that the face direction is facing the display unit (3), it is determined that the driver has confirmed the dangerous object. The warning display device described. It further has a sound collection part installed in the passenger compartment,
The confirmation determination unit (533) analyzes the sound collected by the sound collection unit while the first warning image is displayed on the display unit (3), and sets predetermined words. The warning display device according to claim 1, wherein when it is detected, the driver determines that the dangerous object has been confirmed. The warning image generation unit (532) includes the first determination unit (533) until the predetermined period elapses after the confirmation determination unit (533) determines that the driver of the vehicle (10) has confirmed the dangerous object. Generating a second warning image with a lower degree of highlighting of the dangerous object than the warning image;
The display unit (3) includes the second warning until the predetermined period elapses after the confirmation determination unit (533) determines that the driver of the vehicle (10) has confirmed the dangerous object. The warning display device according to claim 1, which displays an image. Obtaining a peripheral image captured by the first imaging unit (2) and capturing the periphery of the vehicle (10);
Detecting a dangerous object that may collide with the vehicle (10) from the surrounding image;
Generating a warning image highlighting the dangerous object on the peripheral image;
Displaying the warning image on the display unit (3);
Determining whether the driver of the vehicle (10) has confirmed the dangerous object while the warning image is displayed on the display unit (3);
Stopping the display of the warning image on the display unit (3) after it is determined that the driver has confirmed the dangerous object;
The warning display method characterized by including.

Images