JP4910802B2 - Monitoring device and method, recording medium, and program - Google Patents

Description

  The present invention relates to a monitoring apparatus and method, a recording medium, and a program, and more particularly, to a monitoring apparatus and method, a recording medium, and a program that can accurately detect a dazzling state for a driver.

  When driving a vehicle, it may be dazzled by the light incident on the inside of the vehicle, deteriorating the field of view and adversely affecting driving.

  Therefore, conventionally, in the image of the driver's face, the shading range by the shading means provided on the windshield of the vehicle is changed according to the brightness of the area around the driver's eyes to reduce dazzling by direct sunlight It has been proposed (see, for example, Patent Document 1).

JP 2002-331835 A

  By the way, in the light that dazzles the driver while driving, first of all, sunlight, headlights of oncoming cars, brake lamps of cars ahead, lighting of buildings such as shops, street lights, electric bulletin boards There is light from a primary light source that emits light, such as light. In addition, the specularly reflected light of the light from the primary light source, that is, the light reflected from the primary light source at the same angle as the incident angle by the reflecting surface, such as the window glass, water surface, road surface or reflecting plate The light is specularly reflected, the headlight of the following vehicle is specularly reflected by the interior mirror in the vehicle, the light is specularly reflected by the road surface on which the headlight of the oncoming vehicle is wet, and the like.

  However, the invention described in Patent Document 1 only considers mitigating dazzling due to direct sunlight, and does not consider detecting a dazzling state caused by light other than direct sunlight.

  The present invention has been made in view of such a situation, and makes it possible to accurately detect a dazzling state for a driver.

One aspect of the monitoring device of the present invention, based on the sight line orientation of the driver, the field image is an image of a region of a predetermined range including the center of the field of view from the front image driver is an image of the captured forward direction of the vehicle A field-of-view image extracting means for extracting and a first state detecting means for detecting whether or not strong light is emitted to the eyes of the driver based on the luminance of the field-of-view image.

In the monitoring device of one aspect of the present invention, based on the sight line orientation of the driver, view field image is an image of a region of a predetermined range including the center of the field of view from the front image driver is an image of the captured forward direction of the vehicle Is extracted, and based on the luminance of the field-of-view image, a state in which strong light is applied to the eyes of the driver is detected.

  Therefore, a dazzling state for the driver can be accurately detected.

  The visual field image extraction unit and the first state detection unit are constituted by, for example, a CPU (Central Processing Unit).

  The first state detection means is configured to detect whether or not the driver's eyes are irradiated with strong light based on a ratio of pixels having a luminance equal to or higher than a predetermined threshold in the view field image. Can do.

  Thereby, a dazzling state for the driver can be detected more accurately.

  This monitoring apparatus includes a sensitivity estimation unit that estimates the sensitivity of a driver's eyes based on the brightness of a field-of-view image from a predetermined time to the present, and a threshold setting that sets a threshold based on the estimated sensitivity of the driver Means.

  Thereby, a dazzling state for the driver can be detected more accurately.

  This threshold value setting means is constituted by a CPU, for example.

The monitoring device includes a position detection unit that detects a change position that is a position where the luminance changes greatly in a front image, and a region that is divided by the change position and an end of the front image, compared to an adjacent region. An area detection unit that detects an area composed of pixels included in the high luminance area as a light source area is further included , and the first state detection unit includes the light source area among the pixels of the view field image. Based on the luminance of the pixel, it can be detected whether or not the driver's eyes are irradiated with strong light.

  Thereby, a dazzling state for the driver can be detected more accurately.

  The position detection means and the area setting means are constituted by a CPU, for example.

  The position detecting means may detect a position where a change in luminance between adjacent pixels is a predetermined magnitude or more as a change position for each horizontal row and vertical column of the front image. it can.

  Accordingly, the light source region can be detected with one pixel as a minimum unit, and a dazzling state for the driver can be detected more accurately.

The monitoring apparatus detects a second state of detecting whether or not strong light is emitted to the eyes of the driver based on the luminance of the area around the eyes of the driver in the face image that is an image of the driver's face. It can further comprise a detection means and a selection means for selecting which one of the first state detection means and the second state detection means to use based on the luminance of the front image.

  As a result, for example, depending on whether or not sunlight is being applied, it is possible to detect a state in which strong light is applied to the eyes of the driver using a more appropriate detection method.

  This state detection means and selection means are constituted by a CPU, for example.

  The monitoring apparatus may further include a line-of-sight direction detecting unit that detects the direction of the driver's line of sight based on a face image that is an image of the driver's face.

  Thereby, a visual field image can be extracted based on the direction of the line of sight detected in the monitoring apparatus.

  This visual field direction detection means is constituted by a CPU, for example.

  This monitoring device controls the in-vehicle device provided in the vehicle so as to perform an operation corresponding to the deterioration of the visual field of the driver when a state in which strong light is irradiated to the eyes of the driver is detected. Means may further be provided.

  Thereby, the fall of the safety | security and convenience accompanying the deterioration of the driver | operator's visual field by glare is suppressed.

  This in-vehicle device control means is constituted by a CPU, for example.

  The in-vehicle device control means can control the display device as the in-vehicle device so as to display a front image that is an image of the front of the vehicle driven by the driver.

  As a result, the driver can surely confirm the situation ahead of the vehicle, which is dazzling and difficult to see, by looking at the display device.

  This in-vehicle device control means can control the display device as the in-vehicle device so as to increase the brightness of the screen.

  Thereby, the driver can reliably recognize the information displayed on the screen of the display device even in a dazzling state.

  The vehicle-mounted device control means can control the head-up display as the vehicle-mounted device so as to change the position where the image is displayed.

  As a result, the driver can reliably recognize the image displayed by the head-up display.

  The in-vehicle device control means can control a front monitoring device that is an in-vehicle device that monitors the front of the vehicle so as to extend a distance for monitoring the front of the vehicle.

  As a result, the driver's attention is reduced due to the deterioration of the field of view due to glare, and the safety of the occupant can be prevented from being impaired even if the confirmation ahead of the vehicle is slower than normal.

  This in-vehicle device control means controls the pre-crash safety system so that the pre-crash safety system that is an in-vehicle device that avoids accidents in advance or performs actions to reduce the damage caused by a collision is started earlier. can do.

  As a result, the driver's attention is reduced due to the deterioration of the field of view due to glare, and the safety of the occupant can be prevented from being impaired even if the confirmation ahead of the vehicle is slower than normal.

  This vehicle-mounted device control means can be controlled by a power control device that is a vehicle-mounted device that controls the operation of the vehicle engine so as to suppress acceleration of the vehicle.

  Thereby, even if the driver's attention is reduced due to the deterioration of the field of view due to glare and the confirmation of the front of the vehicle is later than usual, it is possible to prevent a rear-end collision or a collision.

A monitoring method, a program, and a program recorded on a recording medium according to one aspect of the present invention are based on the driver's line of sight and the center of the driver's field of view from a front image that is an image of the front of the vehicle. A visual field image extraction step for extracting a visual field image that is an image in a predetermined range including a state, and a state detection step for detecting whether or not strong light is emitted to the eyes of the driver based on the luminance of the visual field image. Including.

In the monitoring method, the program, and the program recorded on the recording medium according to one aspect of the present invention, the center of the driver's visual field from the front image that is an image of the front of the vehicle based on the direction of the driver's line of sight A field-of-view image that is an image of an area in a predetermined range including is extracted, and based on the brightness of the field-of-view image, a state in which strong light is emitted to the eyes of the driver is detected.

  Therefore, a dazzling state for the driver can be accurately detected.

This view image extraction step is configured by, for example, a view image extraction step in which the CPU extracts a view image that is an image in a predetermined range including the center of the driver's field of view from a front image that is an image of the front of the vehicle. Then, this state detection step includes, for example, a state detection step in which the CPU detects whether or not strong light is emitted to the eyes of the driver based on the luminance of the view field image.

  As described above, according to one aspect of the present invention, a dazzling state for a driver can be detected. In particular, according to one aspect of the present invention, a dazzling state for the driver can be accurately detected.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of a monitoring system 101 to which the present invention is applied. The monitoring system 101 monitors the driver driving the vehicle provided with the monitoring system 101 and the state inside the vehicle, and is in a state where the driver's eyes are irradiated with strong light, that is, a dazzling state for the driver (hereinafter simply dazzling state). In other words, the operation of another in-vehicle device provided in the vehicle is controlled so as to perform an operation corresponding to the deterioration of the driver's field of view due to glare.

  The monitoring system 101 is configured to include an indoor monitoring camera 111, a front monitoring camera 112, a glare detection unit 113, a display control unit 114, and a vehicle control unit 115. Further, the glare detection unit 113 is configured to include a tracking unit 121, a gaze direction detection unit 122, a detection method selection unit 123, a sunlight glare detection unit 124, and an illumination light glare detection unit 125.

The indoor monitoring camera 111 and the front monitoring camera 112 capture an image of a subject with a very wide dynamic range (for example, about 170 dB) using a logarithmic conversion type imaging device, for example, as described later with reference to FIG. Consists of devices. The images captured by the indoor monitoring camera 111 and the front monitoring camera 112 are, for example, 14-bit unsigned binary digital data, and luminance values of 16384 gradations ranging from the darkest 0 to the brightest 2 ¹⁴ −1. Is assigned.

  FIG. 2 is a diagram schematically illustrating an example of installation positions of the indoor monitoring camera 111 and the front monitoring camera 112. The upper diagram in FIG. 2 is a diagram when the vehicle 141 provided with the monitoring system 101 is viewed from the right direction, and the lower diagram is a diagram when the vehicle 141 is viewed from above.

  The indoor monitoring camera 111 is, for example, in the vicinity of the steering wheel 151 of the vehicle 141 and a position where the steering wheel 151 does not enter within the angle of view of the indoor monitoring camera 111, that is, the driver 142 for driving the vehicle 141 by the steering wheel 151. It is installed at a position that does not interfere with the imaging of the face of the driver 142 and images the face of the driver 142 and its periphery. The indoor monitoring camera 111 supplies the captured image (hereinafter referred to as a face image) to the tracking unit 121.

  The front monitoring camera 112 is installed, for example, in the lower center of the front end of the body of the vehicle 141 and images the front of the vehicle 141. The front monitoring camera 112 supplies the captured image (hereinafter referred to as a front image) to the car navigation device 102, the front monitoring device 104, the detection method selection unit 123, and the illumination glare detection unit 125.

  As described later with reference to FIG. 9 and the like, the glare detection unit 113 detects a dazzling state for the driver.

  The tracking unit 121 tracks the driver's face in the face image using a predetermined method. That is, the tracking unit 121 detects the driver's face and facial features (for example, the position and shape of the face and each organ of the face). The tracking unit 121 supplies the face image and information indicating the tracking result of the face image to the gaze direction detection unit 122 and the sunlight glare detection unit 124. Note that the method of tracking the human face by the tracking unit 121 is not limited to a specific method, and a method that can more quickly and accurately track the human face is desirable.

  The gaze direction detection unit 122 detects the gaze direction of the driver based on the face image and the tracking result using a predetermined method. The line-of-sight direction detection unit 122 supplies information indicating the detection result to the illumination light glare detection unit 125. Note that the method of detecting the direction of the line of sight by the line-of-sight direction detection unit 122 is not limited to a specific method, and a method capable of detecting the direction of the line of sight more quickly and accurately is desirable.

  As will be described later with reference to FIG. 9 or FIG. 12, the detection method selection unit 123 selects a method for detecting a dazzling state based on the front image. The detection method selection unit 123 supplies information indicating the selection result to the sunlight glare detection unit 124 and the illumination light glare detection unit 125.

  As will be described later with reference to FIG. 13, the sunlight glare detection unit 124 detects a dazzling state mainly by sunlight and reflected light from which sunlight is reflected. The sunlight glare detection unit 124 supplies information indicating the detection result to the display control unit 114 and the vehicle control unit 115.

  As will be described later with reference to FIG. 15, the illumination light glare detection unit 125 detects a dazzling state mainly by illumination light other than sunlight and reflected light from which the illumination light is reflected. The illumination light glare detection unit 125 supplies information indicating the detection result to the display control unit 114 and the vehicle control unit 115.

  As will be described later with reference to FIGS. 16 and 17, the display control unit 114 performs an operation corresponding to deterioration of the driver's field of view due to glare when a dazzling state for the driver is detected. The operation of the display device such as the car navigation device 102 and the head-up display 103 provided in the vehicle provided with the control is controlled.

  As will be described later with reference to FIGS. 16 and 17, the vehicle control unit 115 performs an operation corresponding to deterioration of the driver's field of view due to glare when a dazzling state for the driver is detected. The operation of the in-vehicle device that performs control and processing related to the traveling of the vehicle, such as the front monitoring device 104, the pre-safety crash system 105, and the power control device 106, which are provided in the vehicle provided with the control unit, is controlled.

  The car navigation device 102 displays a navigation function for guiding a travel route to a destination, and a front image captured by the front monitoring camera 112 or a rear image of the vehicle captured by a camera (not shown). Provide a vehicle occupant with a monitor function that assists in checking the situation around the vehicle.

  The head-up display 103 projects an image formed at a point at infinity onto the windshield of the vehicle, so that the driver can keep the line of sight while driving while looking at the foreground of the vehicle. The image is displayed so that it can be seen as it is. For example, a vehicle speed meter or the like is displayed on the head-up display 103.

  The front monitoring device 104 monitors the situation in front of the vehicle based on the front image, detects the presence of a thing in front of the vehicle, for example, a person, an animal, an obstacle, etc., and uses the detected information as a pre-crash safety system. It supplies to 105.

  The pre-crash safety system 105 is a system that performs an operation for avoiding an accident in advance or reducing damage caused by a collision in order to ensure the safety of an occupant. For example, if the pre-crash safety system 105 determines that there is a possibility of a collision with a person or an obstacle ahead of the vehicle based on information from the front monitoring device 104, the pre-crash safety system 105 may issue an alarm to avoid the collision. When it is determined that the collision is unavoidable, the seat belt is automatically wound to increase the restraint of the occupant, or the brake is automatically operated to reduce the impact at the time of the collision.

  The power control device 106 controls the speed and acceleration characteristics of the vehicle by controlling the operation of the engine of the vehicle.

  FIG. 3 is a block diagram illustrating functional configurations of the indoor monitoring camera 111 and the front monitoring camera 112 of FIG. The indoor monitoring camera 111 and the front monitoring camera 112 are configured to include a lens 201 and a logarithmic conversion type imaging device 202. The logarithmic conversion type image pickup device 202 is a logarithmic conversion type image pickup device such as HDRC (High Dynamic Range CMOS (Complementary Metal Oxide Semiconductor) (registered trademark)), for example, and includes a light detection unit 211, logarithmic conversion unit 212, A / A D conversion unit 213 and an imaging timing control unit 214 are included.

  The light emitted from the subject imaged by the indoor surveillance camera 111 or the front surveillance camera 112 (or the light reflected by the subject) enters the lens 201 and is a diagram of the light detection unit 211 of the logarithmic conversion type image sensor 202. An image is formed on a light detection surface (not shown).

  The light detection unit 211 is configured by, for example, a light receiving element including a plurality of photodiodes. The light detection unit 211 converts the light of the subject imaged by the lens 201 into a charge corresponding to the brightness (illuminance) of the incident light, and accumulates the converted charge. The light detection unit 211 supplies the accumulated charges to the logarithmic conversion unit 212 in synchronization with the control signal supplied from the imaging timing control unit 214.

  The logarithmic conversion unit 212 includes, for example, a plurality of MOSFETs (Metal Oxide Semiconductor Field Effect Transistors). The logarithmic conversion unit 212 uses the sub-threshold characteristic of the MOSFET to convert the charge supplied from the photodetection unit 211 to the logarithm of the number of charges (the intensity of current) for each pixel, that is, the light of the subject. An analog electric signal converted into a voltage value substantially proportional to the logarithm of the light amount is generated. The logarithmic conversion unit 212 supplies the generated analog electrical signal to the A / D conversion unit 213.

The A / D conversion unit 213 A / D converts an analog electrical signal into digital image data (hereinafter also simply referred to as an image) in synchronization with a control signal supplied from the imaging timing control unit 214. For example, in the case of conversion to 14-bit unsigned binary digital image data, the luminance value (or pixel value) of the image data takes a value in the range of 0 to the brightest 2 ¹⁴ −1. The A / D conversion unit 213 supplies the converted digital image data to the glare detection unit 113.

  As described above, the indoor monitoring camera 111 or the front monitoring camera 112 is a digital image having a brightness value (or a pixel value) proportional to the logarithm of the incident light quantity, that is, the brightness of the light incident on the light detection unit 211. Output data. The details of the logarithmic conversion type image pickup device are disclosed in, for example, JP-T-7-506932.

  In the light detection unit 211 of the logarithmic conversion type image pickup device 202, the converted charge can be supplied to the logarithmic conversion unit 212 as it is without accumulating.

  FIG. 4 is a graph showing sensitivity characteristics of the logarithmic conversion type image pickup device 202, CCD image pickup device, silver salt film, and human eye. The horizontal axis in FIG. 4 represents the logarithmic value of the illuminance (unit is lux) of the incident light, and the vertical axis represents the sensitivity to the illuminance of the incident light. A line 231 indicates the sensitivity characteristic of the logarithmic conversion type image sensor 202, a line 232 indicates the sensitivity characteristic of the CCD image sensor, a line 233 indicates the sensitivity characteristic of the silver salt film, and a line 234 indicates the sensitivity characteristic of the human eye. ing.

  As described above, the logarithmic conversion type imaging device 202 outputs image data composed of luminance values (or pixel values) that are substantially proportional to the logarithm of the incident light quantity. Therefore, even when the incident light quantity becomes large, the logarithmic conversion type imaging element 202 is used. The capacitance of elements such as photodiodes and MOSFETs constituting the element 202 is not saturated, and the current flowing through each element and the applied voltage do not exceed the range in which output corresponding to the input of each element can be performed. Accordingly, it is possible to obtain a luminance value (or pixel value) according to the change in the amount of incident light almost accurately within the range of luminance that can be imaged. That is, the intensity of the incident light from the subject with a dynamic range of about 170 dB, which is wider than the human eye, for example, from about 1 millilux to about 500 kilolux higher than the brightness of sunlight. It is possible to capture an image consisting of a luminance value (or pixel value) that reflects the above accurately. Note that the dynamic range of the logarithmic conversion type image pickup device 202 used for the indoor monitoring camera 111 or the front monitoring camera 112 is not limited to 170 dB as described above, and a required dynamic such as about 100 dB or 200 dB depending on the purpose of use. What corresponds to the range may be used.

  Therefore, the indoor monitoring camera 111 or the front monitoring camera 112 using the logarithmic conversion type image sensor 202 does not generate luminance clipping in a luminance range that can be visually recognized by humans. Therefore, the amount of incident light is adjusted by adjusting the aperture, shutter speed, and the like. There is no need to do. That is, the indoor monitoring camera 111 or the front monitoring camera 112 can accurately capture the detailed luminance distribution of the subject without adjusting the amount of incident light.

  For example, when imaging the front of a vehicle from inside the vehicle during the day, the indoor monitoring camera 111 or the front monitoring camera 112 does not adjust the amount of incident light, even if the sun is in the field of view. An image in which the luminance distribution is faithfully reproduced can be taken. In addition, when imaging the front of the vehicle from inside the vehicle at night, the indoor monitoring camera 111 or the front monitoring camera 112 does not adjust the amount of incident light even if the headlight of the oncoming vehicle is illuminated from the front. An image that faithfully reproduces the luminance distribution from the light of the headlight to the area not illuminated by the headlight of the host vehicle can be taken.

  Further, in the CCD image pickup device and the silver salt film, as shown by the lines 232 and 233, the logarithmic conversion type image pickup device has a sensitivity characteristic that is not proportional to the logarithm of the illuminance of incident light due to factors such as a gamma characteristic. At 202, the sensitivity characteristic is approximately proportional to the logarithm of the illuminance of the incident light.

  As described above, the indoor monitoring camera 111 and the front monitoring camera 112 using the logarithmic conversion type image sensor 202 are not affected by the occurrence of luminance clipping, adjustment of incident light quantity, and gamma characteristics. Accordingly, the luminance value (or pixel value) of the image data captured by the indoor monitoring camera 111 or the front monitoring camera 112 changes so as to reflect the change in the luminance of the subject and the movement of the subject almost faithfully. That is, the difference value of each pixel of the difference data obtained by taking the difference of the image data between frames is a value in which the luminance variation of the subject and the movement of the subject are reflected almost faithfully.

  FIG. 5 is a block diagram illustrating a functional configuration of the detection method selection unit 123 of FIG. The detection method selection unit 123 is configured to include a statistics unit 301 and a selection unit 302.

  As will be described later with reference to FIG. 9 or FIG. 12, the statistical unit 301 acquires a front image from the front monitoring camera 112, calculates a statistical value of a luminance value of the acquired front image, and indicates the calculated statistical value Information is supplied to the selection unit 302.

  As will be described later with reference to FIG. 9 or FIG. 12, the selection unit 302 selects a method for detecting a dazzling state based on the statistical value of the luminance value of the front image, and the information indicating the selection result is dazzled by sunlight. This is supplied to the height detection unit 124 and the illumination light glare detection unit 125.

  FIG. 6 is a block diagram illustrating a functional configuration of the sunlight glare detection unit 124 of FIG. The sunlight glare detection unit 124 is configured to include a statistics unit 321 and a state detection unit 322.

  The statistics unit 321 acquires information indicating the face image and the tracking result of the face image from the tracking unit 121, and acquires information indicating the detection method selection result from the detection method selection unit 123. As will be described later with reference to FIG. 13, the statistical unit 321 calculates the statistical value of the luminance value of the area around the eyes of the driver in the face image based on the acquired tracking result, and indicates the calculated statistical value Information is supplied to the state detection unit 322.

  As will be described later with reference to FIG. 13, the state detection unit 322 detects a dazzling state for the driver based on the statistical value calculated by the statistical unit 321, and displays information indicating the detection result as the display control unit 114 and the vehicle. This is supplied to the control unit 115.

  FIG. 7 is a block diagram illustrating a functional configuration of the illumination light glare detection unit 125 of FIG. The illumination light glare detection unit 125 is configured to include a view image extraction unit 341, a statistics unit 342, a sensitivity estimation unit 343, a threshold setting unit 344, and a state detection unit 345.

  The field-of-view image extraction unit 341 acquires a front image from the front monitoring camera 112, acquires information indicating a detection result of the driver's line-of-sight direction from the line-of-sight direction detection unit 122, and selects a detection method from the detection method selection unit 123 Get information indicating The field-of-view image extraction unit 341 has a predetermined range (for example, a range with high brightness sensitivity for a general person) centered on the center of the driver's visual field based on the detected direction of the driver's line of sight. A captured image (hereinafter referred to as a view field image) is extracted from the front image. That is, it can be said that the field-of-view image is an image in which a region having a large influence on the glare felt by the driver is shown. The view image extraction unit 341 supplies the extracted view image to the statistics unit 342 and the state detection unit 345.

  As will be described later with reference to FIG. 15, the statistical unit 342 calculates a statistical value of the luminance value of each pixel of the field-of-view image, and supplies information indicating the calculated statistical value to the sensitivity estimation unit 343.

  As will be described later with reference to FIG. 15, the sensitivity estimation unit 343 estimates the driver's eye sensitivity based on the statistical value calculated by the statistical unit 342, and sends information indicating the estimation result to the threshold setting unit 344. Supply.

  As described later with reference to FIG. 15, the threshold setting unit 344 is a luminance value for extracting an area estimated to be dazzling for the driver based on the estimated eye sensitivity of the driver from the view image. A certain luminance threshold is set, and information indicating the luminance threshold is supplied to the state detection unit 345.

  As will be described later with reference to FIG. 15, the state detection unit 345 detects a dazzling state for the driver based on the luminance value and the luminance threshold of the view field image, and displays information indicating the detection result as the display control unit 114 and the vehicle control. To the unit 115.

  Next, processing executed by the monitoring system 101 will be described with reference to FIGS.

  First, the in-vehicle device control process executed by the monitoring system 101 will be described with reference to FIG. This process is started, for example, when an engine of a vehicle provided with the monitoring system 101 is started.

  In step S1, the indoor monitoring camera 111 starts imaging the face of the driver. The indoor monitoring camera 111 supplies a face image that is a captured image to the tracking unit 121.

  In step S2, the front monitoring camera 112 starts imaging the front of the vehicle. The front monitoring camera 112 supplies a front image, which is a captured image, to the front monitoring device 104, the statistical unit 301 of the detection method selection unit 123, and the view image extraction unit 341 of the illumination glare detection unit 125.

  In step S3, the glare detection unit 113 executes a glare detection process. Although details of the glare detection process will be described later with reference to FIG. 9 or FIG. 12, a dazzling state for the driver is detected by this process.

  In step S4, the display control unit 114 and the vehicle control unit 115 determine whether or not a transition to a dazzling state has occurred. Specifically, the display control unit 114 and the vehicle control unit 115 change from a non-glare state to a dazzling state for the driver based on the detection result notified from the sunlight glare detection unit 124 or the illumination light glare detection unit 125. Determine if the transition has occurred. If it is determined that the screen has transitioned to a dazzling state, the process proceeds to step S5.

  In step S5, the display control unit 114 and the vehicle control unit 115 execute a process when a dazzling state is detected. Details of the process at the time of detecting the dazzling state will be described later with reference to FIG. 16. By this process, the in-vehicle device such as the car navigation device 102 is controlled so as to start the operation corresponding to the deterioration of the visual field of the driver. The Thereafter, the process proceeds to step S8.

  In step S4, when it is determined that the screen has not transitioned to the dazzling state, the process proceeds to step S6.

  In step S <b> 6, the display control unit 114 and the vehicle control unit 115 determine whether or not the state has transitioned to a non-glare state. Specifically, the display control unit 114 and the vehicle control unit 115 change from a dazzling state to a non-dazzling state for the driver based on the detection result notified from the sunlight glare detection unit 124 or the illumination light glare detection unit 125. Determine if the transition has occurred. When it determines with having changed to the state which is not dazzling, a process progresses to step S7.

  In step S7, the display control unit 114 and the vehicle control unit 115 execute a process at the end of the dazzling state. The details of the process at the end of the dazzling state will be described later with reference to FIG. 17. By this process, the in-vehicle device such as the car navigation device 102 is controlled so as to stop the operation corresponding to the deterioration of the driver's visual field. The Thereafter, the process proceeds to step S8.

  If it is determined in step S6 that the state has not transitioned to the non-dazzling state, that is, if it is determined that either the dazzling state or the non-dazzling state continues, the process of step S7 is skipped, and the process Advances to step S8.

  In step S8, the monitoring system 101 determines whether to end the process. If it is determined not to end the process, the process returns to step S3, and thereafter, the processes in steps S3 to S8 are repeatedly executed until it is determined in step S8 that the process is to be ended.

  In step S8, for example, when the engine of the vehicle is stopped, the monitoring system 101 determines to end the process, and the in-vehicle device control process ends.

  Next, details of the glare detection process in step S3 of FIG. 8 will be described with reference to the flowchart of FIG.

  In step S31, the statistics unit 301 detects the median value of the luminance values of the front image. Specifically, the statistical unit 301 generates data in which the luminance values of the pixels of the front image are arranged in ascending order or descending order. The statistic unit 301 detects the value at the center of the generated data, that is, the median value of the luminance values of the front image.

  In step S <b> 32, the statistical unit 301 calculates an average value of the luminance values of pixels whose luminance value is lower than the median value. Specifically, the statistical unit 301 calculates an average value of luminance values of pixels whose luminance values are lower than the detected median value among the pixels of the front pixels, that is, an average of luminance values of pixels darker than the median value in the image. Calculate the value. The statistics unit 301 supplies information indicating the calculated average value to the selection unit 302.

  In step S33, the selection unit 302 determines whether the average value is equal to or greater than a predetermined threshold value.

  An image 401 in FIG. 10 schematically shows an example of a front image taken from a vehicle traveling in the suburbs during the day on a sunny day, and a histogram 402 shows a distribution of luminance values of the image 401. An image 421 in FIG. 11 schematically shows an example of a front image taken from a vehicle traveling in the city at night, and a histogram 422 shows a distribution of luminance values of the image 421. Note that the horizontal axes of the histogram 402 and the histogram 422 represent luminance values, and the vertical axis represents the number of pixels. In the horizontal axis direction, the luminance value decreases as it goes to the left, and the luminance value increases as it goes to the right.

  In the image 401, there is almost no shadow portion, and there is no subject that emits light by itself. Therefore, there are no very bright and very dark subjects compared to the surroundings. Accordingly, the luminance values of most pixels are relatively high and are concentrated in a narrow range, so that a peak 411-1 with a narrow width and a very large peak value appears in the histogram 402. Moreover, since the sky on a clear day is brighter than the surroundings, a mountain 411-2 having a small peak appears at a relatively close position on the right side of the mountain 411-1.

  On the other hand, in the image 421, the luminance of most subjects is concentrated in a dark range. Accordingly, since the luminance values of most pixels are low and concentrate in a narrow range, in the histogram 422, the width is narrow and the peak value is large on the left side of the peak 411-1 of the histogram 402, that is, in the dark luminance range. A mountain 431-1 appears. In addition, since the road surface image illuminated by the headlight of the own vehicle has a relatively large proportion in the angle of view and is brighter than the surroundings, the peak value is at a position on the right side of the mountain 431-1. The second largest mountain 431-2 appears. Furthermore, the image of a subject that emits light such as a vehicle headlight, tail lamp, street light, or signal light occupies a very small ratio in the angle of view and is very bright compared to the surroundings. A mountain 431-3 having a very small peak appears at a position further to the right from.

  As shown in the examples of FIGS. 10 and 11, in general, when the distribution of luminance values in the front image captured in the daytime is compared with the distribution of luminance values in the front image captured at night, the image was captured in the daytime. In the forward image, the luminance values are concentrated in a bright range and distributed in a narrow range, compared to the forward image captured at night. This is because, in the daytime outdoors, there is a wide range in which sunlight is irradiated, and there is a tendency that there are few areas with extremely different luminance compared to the surroundings. On the contrary, in the front image captured at night, the luminance values are concentrated in a dark range and distributed over a wider range than the front image captured in the daytime. This is because, in the outdoors at night, there is a tendency that there is a wide range of brightness from an extremely bright area to a dark area not illuminated by the light source, such as various illumination lights.

  Further, the difference in the luminance value in the dark region is larger than the difference in the luminance value in the bright region between the front image captured in the daytime and the forward image captured in the nighttime. This is because the difference between the brightness of the bright area of the front image captured at night (for example, the brightness of various illumination lights) and the brightness of the bright area of the front image captured at daytime (for example, the brightness of the sky) The brightness of the dark area of the front image captured in the day (for example, the brightness of the area not illuminated by light) and the brightness of the dark area of the front image captured in the daytime (for example, brightness not illuminated by sunlight) This is because it tends to be smaller than the difference.

  Therefore, by comparing the brightness values of the dark areas in the front image, it is an image taken in the daytime or an image taken at night, in other words, taken in the state of being irradiated with sunlight. It is possible to more accurately determine whether the image is a captured image or an image captured without being irradiated with sunlight. That is, by comparing the average value of the luminance values of pixels darker than the median value in the front image calculated in step S32 with a predetermined threshold value, the front image is captured in a state in which sunlight is irradiated. It is possible to more accurately determine whether the image is an image captured in a state where sunlight is not irradiated.

  If it is determined that the average value calculated in step S32 is equal to or greater than a predetermined threshold value, that is, if it is determined that a front image has been captured in the state of being irradiated with sunlight, the process proceeds to step S34.

  In step S34, the selection unit 302 selects a detection method mainly for sunlight and reflected light of sunlight. The selection unit 302 supplies information indicating the selection result to the sunlight glare detection unit 124 and the illumination light glare detection unit 125.

  In step S35, the sunlight glare detection unit 124 executes sunlight glare detection processing, and the glare detection processing ends. Details of the sunlight glare detection process will be described later with reference to FIG. 13. By this process, it is detected whether the current state is a dazzling state for the driver, and information indicating the detection result is displayed on the display control unit 114. And supplied to the vehicle control unit 115.

  In Step S33, when it is determined that the average value is less than the predetermined threshold, that is, when it is determined that the front image is captured in a state where sunlight is not irradiated, the process proceeds to Step S36.

  In step S36, the selection unit 302 selects a detection method mainly for illumination light and reflected light of illumination light. The selection unit 302 supplies information indicating the selection result to the sunlight glare detection unit 124 and the illumination light glare detection unit 125.

  In step S37, the illumination light glare detection unit 125 executes an illumination light glare detection process, and the glare detection process ends. Details of the illumination light glare detection process will be described later with reference to FIG. 15. With this process, it is detected whether the current state is dazzling for the driver, and information indicating the detection result is displayed on the display control unit 114. And supplied to the vehicle control unit 115.

  Next, a second embodiment of the glare detection process will be described with reference to FIG.

  In step S61, the median value of the luminance values of the front image is detected in the same manner as in step S31 of FIG.

  In step S62, the average value of the luminance values of the pixels whose luminance value is lower than the median value is calculated in the same manner as the processing in step S32 in FIG.

  In step S63, the statistical unit 301 calculates an average value of the luminance values of pixels having a luminance value higher than the median value. Specifically, the statistical unit 301 calculates an average value of the luminance values of pixels having a luminance value higher than the detected median value among the pixels of the front pixels. The statistics unit 301 supplies information indicating the calculated average value to the selection unit 302.

  In step S64, the selection unit 302 determines whether the difference between the two average values is greater than or equal to a predetermined threshold value.

  As described above with reference to FIGS. 10 and 11, in general, the luminance value is distributed over a wider range in the front image captured at night than in the image captured in the daytime. That is, the difference between the two average values calculated in steps S62 and S63 is greater in the forward image captured at night than in the forward image captured in the daytime. Therefore, by appropriately setting the threshold, based on the difference between the two average values, the front image is an image captured in a state where sunlight is irradiated, or a state where sunlight is not irradiated It is possible to accurately determine whether the image is captured with the.

  If it is determined that the difference between the two average values is equal to or greater than the predetermined threshold value, that is, if it is determined that the front image has been captured in a state where sunlight is irradiated, the process proceeds to step S65.

  In step S65, similar to the processing in step S34 of FIG. 9 described above, a detection method mainly for sunlight and reflected light of sunlight is selected.

  In step S66, similarly to the process of step S35 of FIG. 9 described above, the sunlight glare detection process is executed, and the glare detection process ends.

  In step S64, when it is determined that the difference between the two average values is less than the predetermined threshold value, that is, when it is determined that the front image has been captured in a state where sunlight is not irradiated, the process proceeds to step S67. move on.

  In step S67, similar to the processing in step S36 of FIG. 9 described above, a detection method mainly for illumination light and reflected light of illumination light is selected.

  In step S68, the illumination light glare detection process is executed similarly to the process of step S37 of FIG. 9 described above, and the glare detection process ends.

  In both the first and second embodiments of the glare detection process, by adjusting the threshold value when the amount of sunlight irradiation is small despite the daytime due to cloudy weather or rainy weather. The determination result in step 33 or step S64 can be adjusted. That is, by adjusting the threshold value, it is possible to adjust which of the sunlight glare detection process and the illumination light glare detection process is selected.

  Next, details of the sunlight glare detection process in step S35 in FIG. 9 and step S66 in FIG. 12 will be described with reference to the flowchart in FIG.

  In step S91, the tracking unit 121 tracks the driver's face in the face image using a predetermined method. Accordingly, the position and shape of the driver's face in the face image, the position and shape of the eyes, and the like are detected. The tracking unit 121 supplies a face image and information indicating the tracking result of the face image to the statistics unit 321.

  In step S92, the statistical unit 321 calculates the average value of the luminance values of the images around the driver's eyes, and in step S93, the average value of the luminance values of the images around the driver's eyes is equal to or greater than a predetermined threshold value. Determine whether.

  Specifically, based on the tracking result by the tracking unit 121, for example, the statistical unit 321 generates a region 441-1 around the right eye of the driver and a region around the left eye as shown in FIG. The region 441-2 is extracted. The statistics unit 321 calculates the average value of the luminance values of the pixels excluding the right eye portion in the region 441-1 and the average value of the luminance values of the pixels excluding the left eye portion in the region 441-2. If the statistical unit 321 determines that at least one of the two calculated average values is equal to or greater than a predetermined threshold, that is, if the periphery of the driver's eyes is brighter than the predetermined luminance, the process proceeds to step S94.

  The light that the driver feels dazzling in the daytime when sunlight is radiated is either direct sunlight or reflected light in which the incident sunlight is reflected in almost the same direction by the reflecting surface, that is, regular reflection light of sunlight. One is almost. In addition, when sunlight or sunlight specularly reflected light enters human eyes, most people feel dazzled regardless of the surrounding brightness. That is, when the driver feels dazzling in the daytime when sunlight is radiated, it can be said that light of a certain level or higher is radiated to the driver's eyes and the vicinity thereof. Therefore, in the daytime when sunlight is radiated, it is possible to detect a dazzling state for the driver almost accurately based on the luminance value of the image around the driver's eyes, that is, the brightness around the driver's eyes. it can.

  In step S94, the state detection unit 322 determines that the driver is in a dazzling state, and the sunlight glare detection process ends. Specifically, the state detection unit 322 irradiates the driver's eyes with direct sunlight or regular specular light of sunlight when the average value of the luminance values of the images around the driver's eyes is equal to or greater than a predetermined threshold value. It is determined that the current state is a dazzling state for the driver. The state detection unit 322 supplies information indicating that a dazzling state for the driver has been detected to the display control unit 114 and the vehicle control unit 115.

  If it is determined in step S93 that the average luminance value of the image around the driver's eyes is less than the predetermined threshold value, that is, if the periphery of the driver's eyes is darker than the predetermined luminance, the process proceeds to step S95. move on.

  In step S95, the state detection unit 322 determines that the driver is not in a dazzling state, and the sunlight glare detection process ends. Specifically, the state detection unit 322 irradiates the driver's eyes with direct sunlight or regular specular light of sunlight when the average value of the luminance values of the images around the driver's eyes is less than a predetermined threshold value. It is determined that the current state is not dazzling for the driver. The state detection unit 322 supplies information indicating that a dazzling state for the driver has not been detected to the display control unit 114 and the vehicle control unit 115.

  The area used for calculating the average value in step S92 is an example, and another area including both eyes of the driver may be used. In the above description, the dazzling state for the driver is detected using the average value of the luminance values of the pixels in the region 441-1 and the region 441-2. Other statistical values representing the brightness of the image in the region, such as a value, may be used.

  Next, details of the illumination light glare detection process in step S37 in FIG. 9 and step S68 in FIG. 12 will be described with reference to the flowchart in FIG.

  In step S121, the driver's face in the face image is tracked in the same manner as in step S91 in FIG. The tracking unit 121 supplies the face image and information indicating the tracking result of the face image to the gaze direction detection unit 122.

  In step S122, the line-of-sight direction detection unit 122 detects the direction of the driver's line of sight. Specifically, the line-of-sight direction detection unit 122 detects the direction of the line of sight of the driver based on the face image and the tracking result by the tracking unit 121 using a predetermined method. The line-of-sight direction detection unit 122 supplies information indicating the detection result to the visual field image extraction unit 341.

  In step S123, the visual field image extraction unit 341 extracts a visual field image. Specifically, the visual field image extraction unit 341 extracts a visual field image from the front image based on the direction of the driver's line of sight. The view image extraction unit 341 supplies the extracted view image to the statistics unit 342 and the state detection unit 345.

  For example, the field-of-view image extraction unit 341 corrects parallax based on the relative relationship between the position of the front monitoring camera 112 and the position of the driver's face, in other words, the center point of the line-of-sight direction in the front image, in other words. Find the center of the field of view. The field-of-view image extraction unit 341 extracts, as a field-of-view image, an image of a circular region included in a predetermined range around the center of the field of view (for example, a field of view with an angle of 15 degrees centered on the center of the field of view). Note that the shape of the field-of-view image is not limited to a circle, and may be, for example, an ellipse or a rectangle.

  In the front image, it is desirable to exclude from the view image an area that is actually blocked by the shield in the visual field of the driver, that is, an area that does not significantly affect the glare of the driver. Examples of such a shield include a dashboard, a door, and a pillar around the driver's seat. It is also possible to detect the use of the shade and exclude the area blocked by the shade from the view image. Furthermore, based on the face image, it is possible to detect that the driver is wearing a hat with a brim, and to exclude an area blocked by the brim of the hat from the view image.

  In step S124, the statistics unit 342 calculates an average value of the luminance values of the field-of-view images from a predetermined time before to the present. The statistics unit 342 supplies information indicating the calculated average value to the sensitivity estimation unit 343.

  In step S125, the sensitivity estimation unit 343 estimates the sensitivity of the driver's eyes.

  The human eye changes to one of three states, photopic vision, twilight vision, and scotopic vision, roughly divided with respect to ambient brightness. Photopic vision is a state in which pyramidal cells mainly work in the retina under an environment of about 10 lux to 100,000 lux, and the sensitivity and shape of the eye is lower than that of scotopic vision. Is clearly understood. Dark vision is a condition in which rod cells are mainly working in the retina in an environment of about 0.001 lux to 0.01 lux, and the sensitivity of the eyes is higher than that of photopic vision, and only the lightness and darkness of the object is blurred. It is visible. Mesopic vision is a state where both pyramidal cells and rod cells work in the retina under an environment of about 0.01 lux to 10 lux, and is an intermediate state between photopic vision and scotopic vision. That is, the sensitivity of the human eye changes according to the surrounding brightness.

  In addition, the human eye cannot readily adapt to changes in ambient brightness. Specifically, even if the human eye transitions from a dark state to a bright state, the number of times that the human eye changes from a dark place to a bright place, that is, until it adapts to a bright state (light adaptation). It takes about 10 seconds to several minutes, and even if the surroundings transition from a bright state to a dark state, it takes about 30 until it changes from a photopic vision to a scotopic vision, that is, until it adapts to a dark state (dark adaptation). It takes 2 hours from a minute. That is, even under the same brightness environment, the sensitivity of the human eye varies depending on the surrounding brightness transition.

  Furthermore, the human eye can adjust the amount of light incident on the eye by adjusting the degree of opening of the pupil in a very short time with respect to the incident light.

  The sensitivity estimation unit 343 estimates the sensitivity of the driver's eyes using the average value of the luminance values calculated by the statistics unit 342 based on the characteristics of the human eye described above. For example, the sensitivity estimation unit 343 is based on the transition of the average value of the luminance value of the visual field image from a predetermined time before (for example, one minute before), that is, based on the transition of the brightness of the driver's visual field. The driver's eye sensitivity is estimated by estimating whether the driver's eye state is photopic vision, twilight vision, or scotopic vision. Further, as described above, since dark adaptation takes a long time, for example, when the average value of the luminance value of the field-of-view image transitions to a low state, from a longer time (for example, one hour ago) to the present The sensitivity of the driver's eyes may be estimated more finely based on the average value of the luminance values of the visual field images and the dark adaptation curve (Aubert function curve). Furthermore, since the sensitivity characteristics of the human eye have individual differences due to differences such as age, the sensitivity of the driver may be estimated in consideration of individual differences.

  The sensitivity estimation unit 343 supplies information indicating the estimated driver eye sensitivity to the threshold setting unit 344.

  In step S126, the threshold setting unit 344 sets a luminance threshold.

  By the way, humans may or may not feel dazzling even for light of the same luminance depending on the sensitivity of the eyes at that time. That is, humans can easily feel dazzling when the eye sensitivity is high, even when the light has the same luminance, and human eyes are less likely to feel dazzling when the eye sensitivity is low.

  Based on the eye sensitivity estimated by the sensitivity estimation unit 343, the threshold setting unit 344 uses a conversion equation or a conversion table set in advance to determine an area estimated to be dazzling for the driver from the view image. A luminance threshold value for extraction is set. That is, when the estimated eye sensitivity is high, the brightness threshold is set to a low value, and when the eye sensitivity is low, the brightness threshold is set to a high value. The threshold setting unit 344 supplies information indicating the set threshold to the state detection unit 345.

  In step S127, the state detection unit 345 detects the ratio of pixels in the view field image whose luminance value is equal to or higher than the luminance threshold. Specifically, the state detection unit 345 extracts a pixel whose luminance value is equal to or higher than the luminance threshold among the pixels of the view field image. The state detection unit 345 detects the ratio of the number of extracted pixels to the total number of pixels in the view field image.

  In step S128, the state detection unit 345 determines whether the detected ratio is equal to or greater than a predetermined threshold. If the state detection unit 345 determines that the ratio detected in step S127 is greater than or equal to the predetermined threshold, the process proceeds to step S129.

  In step S129, the state detection unit 345 determines that the driver is dazzling, and the illumination light glare detection process ends. Specifically, the state detection unit 345 determines that the ratio of the bright area in the predetermined range centered on the center of the driver's field of view is the predetermined ratio when the ratio detected in step S127 is equal to or greater than the predetermined threshold. It is determined that the current state is dazzling for the driver. The state detection unit 345 supplies information indicating that a dazzling state for the driver has been detected to the display control unit 114 and the vehicle control unit 115.

  If it is determined in step S128 that the ratio detected in step S127 is less than the predetermined threshold, the process proceeds to step S129.

  In step S130, the state detection unit 345 determines that the driver is not in a dazzling state, and the illumination light glare detection process ends. Specifically, the state detection unit 345 determines that the ratio of the bright area in the predetermined range centered on the center of the driver's visual field is the predetermined ratio when the ratio detected in step S127 is less than the predetermined threshold. It is determined that the current state is not dazzling for the driver. The state detection unit 345 supplies information indicating that a dazzling state for the driver has not been detected to the display control unit 114 and the vehicle control unit 115.

  Next, with reference to the flowchart of FIG. 16, the detail of the process at the time of the dazzling state detection of FIG.8 S5 is demonstrated.

  In step S151, the display control unit 114 causes the car navigation device 102 to display a front image. Specifically, the display control unit 114 supplies information for instructing display of the front image to the car navigation device 102. The car navigation device 102 starts acquiring the front image from the front monitoring camera 112 and displaying the front image. Accordingly, the driver can surely confirm the situation in front of the vehicle that is dazzling and difficult to see.

  In step S152, the display control unit 114 adjusts the display setting of the car navigation device 102. Specifically, the display control unit 114 supplies information for instructing to increase the brightness of the screen to the car navigation device 102, for example. The car navigation device 102 increases the brightness of the screen based on the instruction. Thereby, even in a dazzling state, a decrease in the visibility of the screen of the car navigation device 102 is suppressed, and the driver can reliably recognize the information displayed on the screen.

  In step S153, the display control unit 114 adjusts the display position of the head-up display 103. Specifically, the display control unit 114 supplies the head-up display 103 with information instructing to display an image at a position that is not dazzling for the driver. The head-up display 103 changes the image display position to the instructed position. Thereby, the driver can surely recognize the image displayed by the head-up display 103.

  Here, the position that is not dazzling for the driver is a position where dazzling light is not incident on the axis connecting the image display position of the head-up display 103 and the eyes of the driver. For example, the display control unit 114 presets a position where the possibility that dazzling light is incident on the axis connecting the display position of the image of the head-up display 103 and the eyes of the driver is low, such as the lower end of the windshield. The image display position is moved to the set position. For example, the display control unit 114 connects the display position of the image of the head-up display 103 and the driver's eyes based on the position of the light source in the front image detected by the light source detection process of FIG. The display position of the image is moved to a position where the light emitted from the light source is not incident on the axis.

  In step S154, the vehicle control unit 115 extends the detection distance for forward monitoring. Specifically, the vehicle control unit 115 supplies information for instructing to extend the detection distance of the forward monitoring to the forward monitoring device 104. The forward monitoring device 104 extends the distance for detecting the presence of a person, an animal, an obstacle, or the like ahead of the vehicle based on the instruction. In other words, the driver's attention is reduced due to deterioration of visibility due to glare, and the range for monitoring the front of the vehicle is widened so that the safety of the occupant is not impaired even if the confirmation of the front of the vehicle is slower than normal. .

  In step S155, the vehicle control unit 115 advances the timing at which the pre-crash safety system 105 is activated. Specifically, the vehicle control unit 115 supplies the pre-crash safety system 105 with information instructing to accelerate the start timing. For example, the pre-crash safety system 105 sets itself so that the timing for issuing an alarm for avoiding a collision, the timing for automatically winding the seat belt, the timing for operating the brake, and the like are made earlier than usual. In other words, even if the driver's attention is reduced due to the deterioration of visibility due to glare, and the confirmation of the front of the vehicle is slower than normal, accidents can be avoided in advance so as not to impair the safety of passengers, Actions to reduce damage will be performed at an earlier timing.

  In step S156, the vehicle control unit 115 suppresses the acceleration of the vehicle, and the process at the time of detecting the dazzling state ends. Specifically, the vehicle control unit 115 supplies information that instructs suppression of acceleration of the vehicle to the power control device 106. The power control device 106 controls the operation of the vehicle engine so as to suppress acceleration from normal. In other words, even if the driver's attention is reduced due to deterioration of visibility due to glare, the vehicle acceleration is suppressed so that rear-end collisions and collisions can be prevented even if the forward confirmation of the vehicle is slower than normal. Is done.

  Next, the details of the process at the end of the dazzling state in step S7 of FIG. 8 will be described with reference to the flowchart of FIG.

  In step S171, the display control unit 114 causes the car navigation device 102 to stop displaying the front image. Specifically, the display control unit 114 supplies information for instructing to stop displaying the front image to the car navigation device 102. The car navigation apparatus 102 stops displaying the front image and resumes normal display, for example, display of route guidance to the destination.

  In step S172, the display control unit 114 restores the display setting of the car navigation device 102. Specifically, the display control unit 114 supplies information that instructs the car navigation apparatus 102 to return the screen brightness, contrast, and the like to normal settings. Based on the instruction, the car navigation device 102 returns the screen brightness, contrast, and the like to normal settings.

  In step S173, the display control unit 114 restores the display position of the head-up display 103. Specifically, the display control unit 114 supplies the head-up display 103 with information instructing to return the image display position to the normal position. The head-up display 103 returns the image display position to the instructed position, that is, the normal position.

  In step S174, the vehicle control unit 115 restores the detection distance for forward monitoring. Specifically, the vehicle control unit 115 supplies information for instructing to return the detection distance of the front monitoring to the normal distance to the front monitoring device 104. The forward monitoring device 104 returns the forward monitoring detection distance to the normal distance based on the instruction.

  In step S175, the vehicle control unit 115 restores the timing at which the pre-crash safety system 105 is activated. Specifically, the vehicle control unit 115 supplies the pre-crash safety system 105 with information instructing to return the activation timing to the normal time. Based on the instructions, the pre-crash safety system 105 is configured so that, for example, the timing for issuing an alarm for avoiding a collision, the timing for automatically winding the seat belt, the timing for operating the brake, etc. Set up.

  In step S176, the vehicle control unit 115 cancels the suppression of the acceleration of the vehicle, and the process at the end of the dazzling state ends. Specifically, the vehicle control unit 115 supplies the power control device 106 with information instructing to cancel the suppression of acceleration of the vehicle. The power control device 106 controls the operation of the engine of the vehicle so as to achieve normal acceleration characteristics.

  As described above, a dazzling state for a driver can be accurately detected regardless of the type of light source. In addition, by changing the luminance threshold based on the sensitivity of the driver's eyes, it is possible to detect a dazzling state for the driver more accurately than in the case where the luminance is based only on the surrounding brightness or the luminance of the driver's face. Further, by controlling an in-vehicle device provided in the vehicle so as to perform an operation corresponding to the deterioration of the driver's field of view, a decrease in safety and convenience due to the deterioration of the driver's field of view is suppressed.

  In addition, it is possible to easily estimate the direction of the light source that causes glare because it is determined whether the driver is dazzling based on the brightness of the extracted view image by extracting the view image. Can do. Furthermore, by analyzing the field-of-view image, for example, the light source can be identified, and the type and position of the identified light source can be used as a countermeasure for reducing glare.

  Next, referring to FIG. 18 to FIG. 21, the second implementation of the illumination light glare detection unit 125 of FIG. 1 and the illumination light glare detection process of step S37 of FIG. 9 and step S68 of FIG. A form is demonstrated.

  FIG. 18 is a block diagram showing a second embodiment of the illumination light glare detection unit 125 of FIG. The illumination light glare detection unit 125 of FIG. 18 is configured to include a view image extraction unit 341, a light source detection unit 501, a determination value calculation unit 502, and a state detection unit 503. The light source detection unit 501 is configured to include a luminance change detection unit 511 and a region detection unit 512. In the figure, parts corresponding to those in FIG. 7 are denoted by the same reference numerals, and description of parts having the same processing will be omitted because it will be repeated.

  As will be described later with reference to FIG. 20, the light source detection unit 501 is based on a change in luminance value with respect to a change in the position of each pixel in the front image (hereinafter referred to as a light source region). Detected). In the light source area, the area where the primary light source emitting light is reflected, and the regular reflection light of the light from the primary light source, that is, the light from the primary light source is reflected by the reflecting surface at substantially the same angle as the incident angle. The area where the reflected light is reflected is included.

  The luminance change detection unit 511 acquires a front image from the front monitoring camera 112. As will be described later with reference to FIG. 20, the luminance change detection unit 511 detects a change in luminance value with respect to a change in the position of each pixel of the front image, and supplies information indicating the detection result to the region detection unit 512.

  As will be described later with reference to FIG. 20, the region detection unit 512 detects the light source region based on the detection result from the luminance change detection unit 511. The area detection unit 512 supplies information indicating the position of the detected light source area to the determination value calculation unit 502.

  The determination value calculation unit 502 acquires a view image from the view image extraction unit 341. As will be described later with reference to FIG. 20, the determination value calculation unit 502 calculates a glare determination value for detecting a dazzling state for the driver, and sends information indicating the calculated glare determination value to the state detection unit 503. Supply.

  As will be described later with reference to FIG. 19, the state detection unit 503 detects a dazzling state for the driver based on the dazzling determination value calculated by the determination value calculation unit 502, and controls display of information indicating the detection result. To the unit 114 and the vehicle control unit 115.

  Next, a second embodiment of the illumination light glare detection process in step S37 of FIG. 9 will be described with reference to the flowchart of FIG.

  In step S201, the light source detection unit 501 performs a light source detection process. The details of the light source detection processing will be described later with reference to FIG. 20, but by this processing, the light source region is detected, and information indicating the position of the light source region is supplied from the region detection unit 512 to the determination value calculation unit 502. .

  In step S202, the direction of the driver's line of sight is detected in the same manner as in step S122 in FIG. 15 described above, and in step S203, a field-of-view image is extracted in the same manner as in step S123 in FIG. The extracted view field image is supplied from the view field image extraction unit 341 to the determination value calculation unit 502.

  In step S204, the determination value calculation unit 502 calculates a dazzling determination value. Specifically, the determination value calculation unit 502 extracts pixels in the light source region detected by the region detection unit 512 in the view field image. The determination value calculation unit 502 calculates a glare determination value by summing up the luminance values of the extracted pixels. That is, the glare determination value becomes larger as the area of the light source region is larger and the luminance value of the pixel in the light source region is higher in the view image.

  Note that, for example, a weight may be set so as to increase as the pixel is closer to the center of the visual field of the driver, and the luminance value may be multiplied by the set weight and then added to obtain the glare determination value.

  In step S205, the state detection unit 503 determines whether the dazzling determination value is greater than or equal to a predetermined threshold value. If it is determined that the dazzling determination value is equal to or greater than the predetermined threshold, the process proceeds to step S206.

  In step S206, the state detection unit 503 determines that the driver is dazzling, and the illumination light glare detection process ends. Specifically, the state detection unit 503 has a large total value of the areas of the light sources existing within a predetermined range centered on the center of the driver's field of view when the glare determination value is equal to or greater than a predetermined threshold. Or it determines with the brightness | luminance of the light source which exists in the predetermined range centering on the center of the visual field of a driver being high, and the present state is a dazzling state for a driver. The state detection unit 503 supplies information indicating that a dazzling state for the driver has been detected to the display control unit 114 and the vehicle control unit 115.

  If it is determined in step S205 that the dazzling determination value is less than the predetermined threshold value, the process proceeds to step S207.

  In step S207, the state detection unit 503 determines that the driver is not in a dazzling state, and the illumination light glare detection process ends. Specifically, the state detection unit 503 has a small glare determination value that is less than a predetermined threshold, and thus the total value of the areas of the light sources existing within a predetermined range centered on the center of the driver's visual field is small. In addition, it is determined that the brightness of the light source is low and the current state is not dazzling for the driver. The state detection unit 503 supplies information indicating that a dazzling state for the driver has not been detected to the display control unit 114 and the vehicle control unit 115.

  Next, details of the light source detection processing in step S201 in FIG. 19 will be described with reference to the flowchart in FIG.

  In step S231, the luminance change detection unit 511 detects a position where the luminance value greatly changes in the horizontal direction of the front image. For example, the luminance change detection unit 511 looks at the luminance value of each row in the horizontal direction of the front image from the left direction to the right direction, and the position where the difference in luminance value between adjacent pixels is equal to or greater than a predetermined size, That is, a position where a change in luminance value between adjacent pixels is a predetermined magnitude or more is detected. The luminance change detection unit 511 supplies information indicating the detected position to the region detection unit 512.

  An image 601 in FIG. 21 schematically shows an example of a front image taken from a vehicle traveling in the suburbs at night, and the graph in FIG. 21 shows the luminance of a horizontal row sandwiched between two arrows A in the image 601. The change in value is shown. The horizontal axis of the graph indicates the position of the pixel, and the vertical axis indicates the luminance value of the pixel at that position.

  As shown in the graph of FIG. 21, in the image 601, the brightness value of the area of the headlights 611-1 and 611-2 of the oncoming vehicle, which is the light source that emits light, protrudes compared to the surrounding brightness value. It is high. Therefore, in the row between the two arrows A of the image 601, in step S231, the position P1 near the left end of the headlight 611-1, the position P2 near the right end, and the vicinity of the left end of the headlight 611-2. The position P3 and the position P4 near the right end are detected as positions where the change in luminance value between adjacent pixels is greater than or equal to a predetermined magnitude. In addition, since the luminance value does not change greatly at positions other than the vicinity of the ends of the headlights 611-1 and 611-2, a position where the change in luminance value between adjacent pixels is not less than a predetermined magnitude is not detected.

  Similarly, in general, in the front image captured at night, the position near the edge of the light source or the specularly reflected light from the light source indicates that the change in luminance value between adjacent pixels is greater than or equal to a predetermined magnitude. It is detected as a certain position. In addition, a position where a change in luminance value between adjacent pixels is not less than a predetermined magnitude is hardly detected except for a position near the edge of the light source or the regular reflection light of the light from the light source.

  In step S232, as in the process of step S231, the luminance change detection unit 511 detects a position where the luminance value greatly changes in the vertical direction of the front image. The luminance change detection unit 511 supplies information indicating the detected position to the region detection unit 512.

  Note that the method of detecting the position where the luminance value greatly changes is not limited to the above-described method. For example, an image obtained by performing filtering on the front image using a linear linear differential filter in the horizontal or vertical direction is used. You may make it detect using.

  In step S233, the region detection unit 512 detects a region where the light source is reflected. Specifically, the area detection unit 512 determines, for each horizontal line of the front image, the position detected in step S231 and the adjacent area among the areas divided by the left and right end portions of the front image. In comparison, pixels included in a region having a high luminance value are extracted. For example, in the row between the two arrows A of the image 601 in FIG. 21, the area between the left end of the image 601 and the position P1, the area between the position P1 and the position P2, and the position between the position P2 and the position P3. Among the region, the region between the position P3 and the position P4, and the region between the position P4 and the right end of the image 601, the region between the position P1 and the position P2 that has a higher luminance value than the adjacent region. And pixels included in the region and the region between the position P3 and the position P4 are extracted.

  In addition, the region detection unit 512 compares the position detected in step S232 for each column in the vertical direction of the front image and the adjacent region among the regions divided by the upper and lower ends of the front image. Thus, pixels included in the region having a high luminance value are extracted.

  The region detection unit 512 detects a region formed by the extracted pixels as a region where a light source is reflected in the front image, that is, a light source region. The area detection unit 512 supplies information indicating the position of the detected light source area to the determination value calculation unit 502.

  As described above, a dazzling state for a driver can be accurately detected regardless of the type of light source. Also, by adding the brightness and area of the light source in the driver's field of view to the detection condition, it is possible to detect a dazzling state for the driver more accurately than in the case where it is based solely on the surrounding brightness or the brightness of the driver's face. it can. Furthermore, the position where the light source is reflected in the front image can be detected easily and accurately.

  In addition, in the process of FIG. 9 described above, either the sunlight glare detection process or the illumination light glare detection process is selected and executed based on the result of the determination process in step S33, but step S35 is performed. In this case, the illumination light glare detection process of the first embodiment or the second embodiment may be performed by using a threshold different from the process of step S37. That is, a dazzling state for the driver may be detected by adjusting the threshold value based on whether or not sunlight is radiated and executing the illumination light glare detection process.

  In the above description, in the illumination light glare detection process of FIGS. 15 and 19, a dazzling state for the driver is detected only for the front of the vehicle. The direction, the rear, etc. may be included in the target. In this case, an imaging device that images the detection direction to be added and an imaging device that images the face of the driver when the driver faces the detection direction to be added are provided, and the detection direction to be added is imaged in the same manner as the above-described processing. A field-of-view image may be extracted from an image captured by the imaging device, or a region where a light source is reflected may be detected to detect a dazzling state for the driver. In this case, for example, an omnidirectional camera that can image all directions around the vehicle may be provided.

  Next, a second embodiment of the present invention will be described with reference to FIGS.

  FIG. 22 is a block diagram showing a configuration of a monitoring system 701 according to the second embodiment of this invention. The monitoring system 701 is configured to include an indoor monitoring camera 111, a glare detection unit 711, a display control unit 114, and a vehicle control unit 115. Further, the glare detection unit 711 is configured to include a tracking unit 121, a statistics unit 721, a reference height detection unit 722, and a state detection unit 723. Note that the front monitoring camera 112 provided in the monitoring system 101 is provided outside the monitoring system 701.

  In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and the description of the portions having the same processing will be omitted because it is repeated.

  The glare detection unit 711 detects a dazzling state for the driver, as will be described later with reference to FIG.

  The tracking unit 121 supplies the face image captured by the indoor monitoring camera 111 and information indicating the tracking result of the face image to the statistics unit 721. The tracking unit 121 supplies information indicating the eye height of the driver detected by tracking to the reference height detection unit 722 and the state detection unit 723.

  FIG. 23 is a diagram illustrating an example of a position where the eye height is detected. For example, the height between the upper eyelid and the lower eyelid at the midpoint P between the eye head 801 and the eye corner 802 indicated by the arrow H in the figure is detected as the eye height of the driver. The position for detecting the eye height is not limited to this example. For example, the height between the highest position at the lower end of the upper eyelid and the lowest position at the upper end of the lower eyelid is set as the eye height. You may make it detect as height.

  Similar to the statistic unit 321 in FIG. 6, the statistic unit 721 calculates the statistic value of the brightness value of the image in the region around the eyes of the driver in the face image based on the acquired tracking result, and calculates the calculated statistic value. Information to be shown is supplied to the reference height detector 722.

  As will be described later with reference to FIG. 25, the reference height detection unit 722 detects a reference height that is the height of the driver's eyes that serves as a reference for detecting a dazzling state for the driver, and detects the detected reference height. Information indicating this is supplied to the state detection unit 723.

  As will be described later with reference to FIG. 26, the state detection unit 723 includes the statistical value of the luminance value of the image in the area around the driver's eyes in the face image, and the height of the driver's eyes detected by the tracking unit 121. Based on the comparison result between the height and the reference height, a dazzling state for the driver is detected, and information indicating the detection result is supplied to the display control unit 114 and the vehicle control unit 115.

  Next, processing of the monitoring system 701 will be described with reference to FIGS.

  First, the in-vehicle device control process executed by the monitoring system 701 will be described with reference to FIG. This process is started, for example, when an engine of a vehicle provided with the monitoring system 701 is started.

  In step S301, imaging of the driver's face is started in the same manner as in step S1 of FIG.

  In step S302, the glare detection unit 711 executes a reference height detection process. Details of the reference height detection process will be described later with reference to FIG. 25, and the reference height of the driver's eyes is detected by this process.

  In step S303, the glare detection unit 711 executes a glare detection process. The details of the glare detection process will be described later with reference to FIG. 26, but this process detects a dazzling state for the driver.

  The processing in steps S304 to S308 is the same as the processing in steps S4 to S8 in FIG.

  Next, details of the reference height detection process in step S302 in FIG. 24 will be described with reference to the flowchart in FIG.

  In step S331, the tracking unit 121 tracks the driver's face in the face image in the same manner as in step S91 of FIG. The tracking unit 121 supplies a face image and information indicating the tracking result of the face image to the statistics unit 721. In addition, the tracking unit 121 supplies data indicating the eye height of the driver detected by performing tracking to the reference height detection unit 722.

  In step S332, the statistical unit 721 calculates the average value of the luminance values of the images around the driver's eyes in the same manner as the processing performed by the statistical unit 321 in step S92 of FIG. The statistics unit 721 supplies information indicating the calculated average value to the reference height detection unit 722.

  In step S333, the reference height detection unit 722 determines whether the average luminance value of the images around the driver's eyes is less than a predetermined threshold. If it is determined that the average value of the luminance values calculated by the statistical unit 721 is greater than or equal to a predetermined threshold, that is, if the periphery of the driver's eyes is brighter than the predetermined luminance, the process returns to step S331. Thereafter, the processing in steps S331 to S333 is repeatedly executed until it is determined in step S333 that the average value of the luminance values of the images around the driver's eyes is less than a predetermined threshold value.

  If it is determined in step S333 that the average luminance value of the image around the driver's eyes is less than the predetermined threshold, the process proceeds to step S334.

  In step S334, the reference height detection unit 722 accumulates data. Specifically, the reference height detection unit 722 determines the height of the driver's eyes detected in the immediately preceding step S331, that is, the height of the driver's eyes when the periphery of the driver's eyes is darker than a predetermined luminance. This data is stored in an internal storage unit (not shown).

  In step S335, the reference height detection unit 722 determines whether a predetermined amount of data has been accumulated. If it is determined that the predetermined amount of data has not been accumulated, the process returns to step S331, and then, in step S335, the processes of steps S331 to S335 are performed until it is determined that the predetermined amount of data has been accumulated. It is executed repeatedly.

  If it is determined in step S335 that a predetermined amount of data has been accumulated, that is, if a predetermined amount of data indicating the eye height of the driver has been accumulated, the process proceeds to step S336.

  In step S336, the reference height detection unit 722 detects the reference height, and the reference height detection process ends. Specifically, the reference height detection unit 722 calculates an average value of the driver's eye height in the accumulated data and sets it as the reference height. That is, the reference height is an average value of eye height in a state where the periphery of the driver's eyes is not very bright, that is, in a state where the driver is not so dazzling.

  At this time, in order to exclude data when the size of the eye is changed due to blinking or the like, the average value may be calculated by excluding data near the maximum value and the minimum value.

  Next, details of the glare detection process in step S303 in FIG. 24 will be described with reference to the flowchart in FIG.

  In step S361, the tracking unit 121 tracks the driver's face in the face image in the same manner as the process in step S91 of FIG. The tracking unit 121 supplies a face image and information indicating the tracking result of the face image to the statistics unit 721. In addition, the tracking unit 121 supplies data indicating the eye height of the driver detected by performing tracking to the state detection unit 723.

  In step S362, the statistical unit 721 calculates the average value of the luminance values of the images around the driver's eyes in the same manner as the processing by the statistical unit 321 in step S92 of FIG. 13 described above. The statistics unit 721 supplies information indicating the calculated average value to the state detection unit 723.

  In step S363, the state detection unit 723 determines whether the driver is dazzling in the previous process. If it is determined in the glare detection process executed immediately before that the driver is not in a dazzling state, the process proceeds to step S364.

  In step S364, the state detection unit 723 determines whether the average luminance value of the image around the driver's eyes has changed in a direction higher than a predetermined threshold. Processing is performed when it is determined that the average value of the luminance values of the images around the driver's eyes has changed in a direction higher than a predetermined threshold, that is, when the luminance around the driver's eyes has changed brightly by a predetermined value or more. Advances to step S365.

  In step S365, the state detection unit 723 determines whether the eye height of the driver is within a predetermined ratio range of the reference height. If it is determined that the driver's eye height is within a predetermined percentage of the reference height (for example, 60%), that is, the driver narrows his eyes to a predetermined percentage of the reference height. If yes, the process proceeds to step S366.

  In step S366, the state detection unit 723 determines whether or not the state in which the eye height of the driver is within a predetermined ratio range of the reference height has continued for a predetermined time. If it is determined that the state in which the driver's eye height is within the predetermined ratio range of the reference height has not yet continued for a predetermined time (for example, 1 second), the process proceeds to step S367.

  In step S367, the tracking unit 121 tracks the driver's face in the face image in the same manner as in step S91 in FIG. In addition, the tracking unit 121 supplies data indicating the eye height of the driver detected by performing tracking to the state detection unit 723.

  Thereafter, the process returns to step S365, and it is determined in step S365 that the driver's eye height exceeds a predetermined range of the reference height, or in step S366, the driver's eye height is The processes in steps S365 to S367 are repeatedly executed until it is determined that the state within the predetermined ratio range of the reference height has continued for a predetermined time.

  If it is determined in step S366 that the eye height of the driver is within a predetermined ratio range of the reference height, it is determined that the state has continued for a predetermined time, that is, up to a predetermined ratio range of the reference height. If the driver is narrowing his eyes for a predetermined time, the process proceeds to step S368.

  In step S368, the state detection unit 723 determines that the state has shifted to a dazzling state for the driver, and the dazzling detection process ends. Specifically, the state detection unit 723 changes the average value of the luminance values of the images around the driver's eyes in a direction higher than a predetermined threshold, and the driver's eye height is a predetermined ratio of the reference height. It is determined that the state in the range has continued for a predetermined time, the driver has narrowed his eyes to avoid glare, and the driver has transitioned from a state not dazzling to a dazzling state. The state detection unit 723 supplies information indicating that the state has changed to a dazzling state for the driver to the display control unit 114 and the vehicle control unit 115.

  In step S365, when it is determined that the eye height of the driver exceeds the predetermined ratio range of the reference height, it is not determined that the driver has shifted to a dazzling state, and the glare detection process ends. To do.

  In step S364, when it is determined that the average value of the luminance values of the images around the driver's eyes has not changed in a higher direction than the predetermined threshold, that is, the luminance around the driver's eyes is equal to or higher than the predetermined value. When it is not changing brightly, it is not determined that the driver has shifted to a dazzling state, and the dazzling detection process ends.

  In step S363, if it is determined in the previous glare detection process that the driver is dazzling, the process proceeds to step S369.

  In step S369, as in the process of step S365, it is determined whether the eye height of the driver is within a predetermined ratio range of the reference height. If it is determined that the eye height of the driver exceeds the range of a predetermined ratio of the reference height, that is, if the driver has not narrowed the eye to the range of the predetermined ratio of the reference height, the process is Proceed to step S370.

  In step S370, the state detection unit 723 determines whether the average luminance value of the image around the driver's eyes is equal to or less than a predetermined threshold value. If it is determined that the average value of the luminance values of the images around the driver's eyes is equal to or less than the predetermined threshold value, that is, if the luminance around the driver's eyes is darker than the predetermined luminance, the process proceeds to step S371.

  In step S371, the state detection unit 723 determines that the state has changed to a state that is not dazzling for the driver, and the dazzling detection processing ends. Specifically, the state detection unit 723 has the driver's eye height exceeding a predetermined ratio range of the reference height, and the luminance around the driver's eye is darker than the predetermined luminance. It is determined that the driver has transitioned from a dazzling state to a non-dazzling state. The state detection unit 723 supplies the display control unit 114 and the vehicle control unit 115 with information indicating a transition to a state that is not dazzling for the driver.

  If it is determined in step S370 that the average value of the luminance values of the images around the driver's eyes is greater than a predetermined threshold, that is, the driver has not narrowed his eyes to a predetermined percentage range of the reference height. However, if the luminance around the driver's eyes is brighter than the predetermined luminance, the process of step S371 is skipped, and it is not determined that the driver has transitioned to a non-glare state, and the glare detection process ends.

  When it is determined in step S369 that the driver's eye height is within a predetermined ratio range of the reference height, that is, the driver is narrowing his eyes to a predetermined ratio range of the reference height. Is continued, the processes of steps S370 and S371 are skipped, and it is not determined that the driver has shifted to a state that is not dazzling, and the glare detection process ends.

  As described above, using only the driver's face image, a dazzling state for the driver can be accurately detected regardless of the type of light source. In addition, by adding the height of the driver's eyes and the brightness around the driver's eyes to the detection condition, the dazzling state for the driver can be more accurately compared to the case based solely on the surrounding brightness and the brightness of the driver's face. Can be detected. Moreover, the dazzling state according to each driver can be detected by detecting the reference height for each driver.

  It is also possible to detect a dazzling state for the driver based on only the eye height of the driver without considering the brightness of the driver's face.

  In the above description, an example is shown in which a front image is displayed on the car navigation device 102 or display settings of the car navigation device are adjusted. However, the display device provided in a vehicle other than the car navigation device 102 A front image may be displayed, or display settings of the display device may be adjusted.

  In addition, by using a wide-angle lens for the front monitoring camera 112 or providing a plurality of front monitoring cameras 112, it becomes possible to extract a field of view image from a wider range of front images, and more accurately detect a dazzling state for the driver. can do.

  Furthermore, taking into account that light reflected by mirrors such as door mirrors and rearview mirrors is applied to the driver's face and affects glare, a rear surveillance camera that captures the rear of the vehicle is provided, and the driver is driven by the mirror. An image in a range that falls within the field of view is extracted from the image captured by the rear monitoring camera. Then, in the front image, the area where the mirror is actually visible to the driver may be replaced with the extracted image, and then the view field image may be extracted. Thereby, the dazzling state for the driver can be detected in consideration of the influence of the mirror.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 27 is a block diagram showing an example of the configuration of a personal computer 900 that executes the above-described series of processing by a program. A CPU (Central Processing Unit) 901 executes various processes according to a program stored in a ROM (Read Only Memory) 902 or a recording unit 908. A RAM (Random Access Memory) 903 appropriately stores programs executed by the CPU 901, data, and the like. The CPU 901, ROM 902, and RAM 903 are connected to each other by a bus 904.

  An input / output interface 905 is also connected to the CPU 901 via the bus 904. The input / output interface 905 is connected to an input unit 906 made up of a keyboard, mouse, microphone, etc., and an output unit 907 made up of a display, a speaker, etc. The CPU 901 executes various processes in response to a command input from the input unit 906. Then, the CPU 901 outputs the processing result to the output unit 907.

  The recording unit 908 connected to the input / output interface 905 includes, for example, a hard disk, and stores programs executed by the CPU 901 and various data. A communication unit 909 communicates with an external device via a network such as the Internet or a local area network.

  Further, the program may be acquired via the communication unit 909 and stored in the recording unit 908.

  A drive 910 connected to the input / output interface 905 drives a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the recording unit 908 as necessary.

  As shown in FIG. 27, a program recording medium for storing a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory, DVD (Digital Versatile Disc), a magneto-optical disk, a removable medium 911 which is a package medium composed of a semiconductor memory, a ROM 902 in which a program is temporarily or permanently stored, and a recording unit 908 It is comprised by the hard disk etc. which comprise. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 909 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram which shows one Embodiment of the monitoring system to which this invention is applied. It is a figure which shows typically the example of the installation position of the indoor surveillance camera of FIG. 1, and a front surveillance camera. It is a block diagram which shows the functional structure of the indoor surveillance camera of FIG. 1, and a front surveillance camera. It is a graph which shows sensitivity characteristics, such as a logarithm conversion type image sensor. It is a block diagram which shows the functional structure of the detection method selection part of FIG. It is a block diagram which shows the functional structure of the sunlight glare detection part of FIG. It is a block diagram which shows the functional structure of the illumination light glare detection part of FIG. It is a flowchart for demonstrating the vehicle equipment control process performed by the monitoring system of FIG. It is a flowchart for demonstrating the detail of the glare detection process of FIG.8 S3. It is a figure which shows the example of the distribution of the front image and luminance value which were imaged in the daytime. It is a figure which shows the example of the front image imaged at night, and the distribution of a luminance value. It is a flowchart for demonstrating 2nd Embodiment of the glare detection process of FIG.8 S3. It is a flowchart for demonstrating the detail of the sunlight glare detection process of FIG.9 S35 and FIG.12 S66. It is a figure which shows the example of the area | region which detects the average value of a brightness | luminance. 13 is a flowchart for explaining details of illumination light glare detection processing in step S37 of FIG. 9 and step S68 of FIG. It is a flowchart for demonstrating the detail of the process at the time of the dazzling state detection of FIG.8 S5. It is a flowchart for demonstrating the detail of the process at the time of the dazzling state completion | finish of step S7 of FIG. It is a block diagram which shows the structure of 2nd Embodiment of the illumination light glare detection part of FIG. It is a flowchart for demonstrating 2nd Embodiment of the illumination light glare detection process of FIG.9 S37 and FIG.12 S68. It is a flowchart for demonstrating the detail of the light source detection process of step S201 of FIG. It is a figure which shows the example of the front image imaged at night, and the distribution of a luminance value. It is a block diagram which shows 2nd Embodiment of the monitoring system to which this invention is applied. It is a figure which shows the example of the position which detects the height of eyes. It is a flowchart for demonstrating the vehicle-mounted apparatus control process performed by the monitoring system of FIG. It is a flowchart for demonstrating the detail of the reference | standard height detection process of step S302 of FIG. It is a flowchart for demonstrating the detail of the glare detection process of step S303 of FIG. And FIG. 11 is a block diagram illustrating an example of a configuration of a personal computer.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Monitoring system 102 Car navigation apparatus 103 Head-up display 104 Front monitoring apparatus 105 Pre-crash safety system 106 Power control part 111 Indoor monitoring camera 112 Front monitoring camera 113 Dazzle detection part 114 Display control part 115 Vehicle control part 121 Tracking part 122 Line of sight Direction detection unit 123 Detection method selection unit 124 Sunlight glare detection unit 125 Illumination light glare detection unit 202 Logarithmic conversion type imaging device 301 Statistics unit 302 Selection unit 321 Statistics unit 322 State detection unit 341 View image extraction unit 342 Statistics unit 343 Sensitivity estimation unit 344 Threshold setting unit 345 State detection unit 501 Light source detection unit 502 Determination value calculation unit 503 State detection unit 511 Brightness change detection unit 512 Area detection unit 701 Monitoring system 711 Dazzle detection 721 Statistics 722 reference height detecting unit 723 state detector

Claims (17)

A visual field image extracting means for extracting a visual field image that is an image in a predetermined range including the center of the visual field of the driver from a front image that is an image of the front of the vehicle based on the direction of the driver's line of sight;
And a first state detection unit that detects whether or not strong light is emitted to the eyes of the driver based on the luminance of the field-of-view image. Second state detecting means for detecting whether or not intense light is emitted to the eyes of the driver based on the brightness of the area around the eyes of the driver in the face image that is an image of the face of the driver When,
The front image based on the luminance of the monitoring device of claim 1, further comprising a selecting means for selecting whether to use either one of the first state detection means and said second state detection means. The first state detection unit detects whether or not strong light is emitted to the eyes of the driver based on a ratio of pixels having luminance equal to or higher than a predetermined threshold in the visual field image. The monitoring device described in 1. Sensitivity estimation means for estimating the sensitivity of the eyes of the driver based on the brightness of the field-of-view image from a predetermined time to the present;
The monitoring apparatus according to claim 3 , further comprising: a threshold setting unit that sets the threshold based on the estimated sensitivity of the driver. In the front image, position detecting means for detecting a change position that is a position where the luminance changes greatly,
A region detecting means for detecting, as a light source region, a region constituted by pixels included in a region having a higher brightness than an adjacent region among regions divided by the change position and an edge of the front image; In addition ,
Said first state detection means, among the pixels of the field image, the claims based on the brightness of the pixels included in the light source region, strong light to the eye of the driver to detect whether it is illuminated The monitoring apparatus according to 1. It said position detecting means, for each column of horizontal rows and vertical direction of the front image, to claim 5 for detecting the position where the change in brightness between adjacent pixels is equal to or greater than the predetermined size as the change positions The monitoring device described. The monitoring apparatus according to claim 1, further comprising a gaze direction detection unit that detects a gaze direction of the driver based on a face image that is an image of the driver's face. On-vehicle device control means for controlling an on-vehicle device provided in the vehicle so as to perform an operation corresponding to a deterioration in the visibility of the driver when a state in which strong light is irradiated to the driver's eyes is detected. The monitoring device according to claim 1, further comprising: The in-vehicle device control unit, the monitoring device according to claim 8 for controlling the display device as the vehicle device so as to display the front image. The monitoring device according to claim 8, wherein the in-vehicle device control unit controls the display device as the in-vehicle device so as to increase a screen brightness. The monitoring device according to claim 8, wherein the in-vehicle device control unit controls a head-up display as the in-vehicle device so as to change a position for displaying an image. The monitoring device according to claim 8, wherein the in-vehicle device control means controls the front monitoring device that is the in-vehicle device that monitors the front of the vehicle so as to extend a distance for monitoring the front of the vehicle. The vehicle-mounted device control means controls the pre-crash safety system so as to advance the timing of starting the pre-crash safety system, which is the vehicle-mounted device that performs an operation for avoiding an accident in advance or reducing damage caused by a collision. Item 9. The monitoring device according to Item 8. The monitoring device according to claim 8, wherein the in-vehicle device control unit controls a power control device that is the in-vehicle device that controls operation of an engine of the vehicle so as to suppress acceleration of the vehicle. A visual field image extraction step for extracting a visual field image that is an image in a predetermined range including the center of the visual field of the driver from a front image that is an image obtained by imaging the front of the vehicle based on the direction of the line of sight of the driver;
A state detecting step of detecting whether or not intense light is emitted to the eyes of the driver based on the luminance of the field-of-view image. A visual field image extraction step for extracting a visual field image that is an image in a predetermined range including the center of the visual field of the driver from a front image that is an image obtained by imaging the front of the vehicle based on the direction of the line of sight of the driver;
A program that causes a computer to execute processing including: a state detection step of detecting whether or not strong light is emitted to the eyes of the driver based on the luminance of the field-of-view image.   A recording medium on which the program according to claim 16 is recorded.

Images