WO2022195836A1

WO2022195836A1 - Backlight determination device, backlight determination method, and backlight determination program

Info

Publication number: WO2022195836A1
Application number: PCT/JP2021/011312
Authority: WO
Inventors: 雅也仁平
Original assignee: 三菱電機株式会社
Priority date: 2021-03-19
Filing date: 2021-03-19
Publication date: 2022-09-22
Also published as: JPWO2022195836A1; JP7270866B2

Abstract

A backlight determination device (10) comprises: an information acquisition unit (11) that acquires event occurrence information (A), vehicle position information (G) indicating the position of a vehicle at the time of occurrence of an event, an image (I) obtained by photographing the outside of the vehicle, and distance information (L) on a distance to a nearby object obtained by detecting the outside of the vehicle at the time of occurrence of the event; a light source position acquisition unit (12) that acquires light source position information indicating the relative position of a light source with respect to the vehicle at the time of occurrence of the event on the basis of the image (I) or calculation using the vehicle position information (G) and a time (t); a shady region acquisition unit (13) that acquires shade information indicating a shady region at the time of occurrence of the event on the basis of map information (M) including road and structure information, the vehicle position information (G), the light source position information (S), and the event occurrence information (A); and a determination unit (14) that determines whether or not the state of the driver of the vehicle at the time of occurrence of the event has been a backlight state in which light is incident from the light source on the eyes of the driver on the basis of the light source position information and the shade information.

Description

Backlight determination device, backlight determination method, and backlight determination program

The present disclosure relates to a backlight determination device, a backlight determination method, and a backlight determination program.

An image recognition unit that detects objects based on image information obtained by imaging the surroundings of the vehicle and acquires information about the objects, and a vehicle surroundings that converts the vehicle surrounding information including the information about the objects into a predetermined code. A vehicle periphery information collection device has been proposed that includes an information generation unit and a storage unit that stores codes converted by the vehicle periphery information generation unit. See, for example, US Pat.

JP 2019-120969 A (for example, abstract, see FIG. 2)

However, the device described in Patent Document 1 cannot accurately determine whether or not the driver who was driving the vehicle was in backlight when some event occurred in the vehicle.

An object of the present disclosure is to make it possible to accurately determine whether or not the driver is in a backlight state when a predetermined event occurs.

The backlight determination device of the present disclosure includes event occurrence information indicating that a predetermined event has occurred in a vehicle, vehicle position information indicating the position of the vehicle at the time of occurrence of the event, and and an information acquisition unit that acquires an image obtained by detecting the outside of the vehicle at the time of occurrence and distance information to a surrounding object obtained by detecting the outside of the vehicle, and a calculation using the vehicle position information and the time Alternatively, a light source position acquisition unit that acquires light source position information indicating the relative position of the light source with respect to the vehicle at the time of occurrence based on the image, map information including road and structure information, the vehicle position information, and the light source a shaded area acquisition unit that acquires shaded area information indicating a shaded area at the time of occurrence based on position information and the event occurrence information; and a determination unit that determines whether or not the driver of the vehicle is in a backlight state in which light from the light source is incident on the eyes of the driver.

Another backlight determination device of the present disclosure includes an acquisition unit that acquires an image output from an in-vehicle camera that captures the face of a driver of a vehicle and event occurrence information that indicates that a predetermined event has occurred in the vehicle. and, using a learned model generated by prior machine learning to classify facial expressions when backlight enters the eye, when the event occurs, the driver's state changes from the light source to the and a determination unit that determines whether or not the vehicle is in a backlight state in which light is incident on the eyes of the driver of the vehicle.

Still another backlight determination device of the present disclosure acquires an image output from an in-vehicle camera that captures the eyes of a driver of a vehicle and event occurrence information indicating that a predetermined event has occurred in the vehicle. and whether a corneal reflection and an opposing reflection occurred sequentially in the driver's eye at the time of the event, the driver's condition is incident on the vehicle's driver's eye from a light source. and a determination unit that determines whether or not there is a backlit state in which light exists.

According to the present disclosure, it is an object of making it possible to accurately determine whether or not the driver was in a backlight state when a predetermined event occurred.

1 is a functional block diagram schematically showing the configuration of a backlight determination device according to Embodiment 1; FIG. 1 is a diagram showing a hardware configuration of a backlight determination device according to Embodiment 1; FIG. (A) is a diagram showing an example when an event occurs (when there is no shade), and (B) is a diagram showing the position of the sun relative to the vehicle. (A) is a diagram showing an example when an event occurs (when there is a shade), and (B) is a diagram showing an example when an event occurs (when there is a shade). 5 is a flow chart showing a shadow region detection operation of the backlight determination device according to Embodiment 1; FIG. 5 is a functional block diagram schematically showing the configuration of a backlight determination device according to Embodiment 2; It is a figure which shows the calculation process of a reflected light vector. 10 is a flow chart showing the reflected light detection operation of the backlight determination device according to the second embodiment. FIG. 11 is a functional block diagram schematically showing the configuration of a backlight determination device according to Embodiment 3; FIG. 10 is a diagram showing a hardware configuration of a backlight determination device according to Embodiment 3; FIG. 10 is a diagram showing a configuration for learning a facial expression determination model and a configuration for utilizing a display determination model; 4 is a flowchart showing facial expression detection processing; FIG. 11 is a functional block diagram schematically showing the configuration of a backlight determination device according to Embodiment 4; It is a figure which shows reflection detection processing. It is a flow chart which shows eyelid detection processing.

A backlight determination device, a backlight determination method, and a backlight determination program according to an embodiment will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be combined as appropriate and each embodiment can be modified as appropriate.

Embodiment 1.
FIG. 1 is a functional block diagram schematically showing the configuration of a backlight determination device 10 according to Embodiment 1. As shown in FIG. The backlight determination device 10 is a device capable of executing the backlight determination method according to the first embodiment. The backlight determination device 10 is, for example, a computer. The backlight determination device 10 can execute the backlight determination method according to the first embodiment by executing the backlight determination program.

As shown in FIG. 1, the backlight determination device 10 has an information acquisition section 11, a light source position acquisition section 12, a shade area acquisition section 13, and a determination section . The backlight determination device 10 determines whether or not the vehicle is in a backlight state in which light is incident on the eyes of the driver based on the information stored in the storage unit 62 of the drive recorder 60 of the vehicle and the map information 63 . The information stored in the storage unit 62 of the drive recorder 60 is obtained from the in-vehicle sensor 50 while the vehicle is running. The in-vehicle sensor 50 includes, for example, an exterior camera 51 that captures the exterior of the vehicle, a range sensor 52 that measures the distance to objects around the vehicle, a GPS 53 that is a position detection unit of a positioning system, and a predetermined event detection. It has an acceleration sensor 54 that is The ranging sensor 52 is, for example, LiDAR, millimeter wave radar, sonar radar, or the like. When a stereo camera is used as the vehicle exterior camera 51, the ranging sensor 52 can be omitted.

The information acquisition unit 11 receives event occurrence information A indicating that a predetermined event E has occurred in the vehicle, vehicle position information G indicating the position of the vehicle at the time the event E occurred, and outside the vehicle at the time of occurrence. A photographed image I and distance information L indicating the distance to a peripheral object obtained by detecting the outside of the vehicle when the event E occurred are acquired.

The light source position acquisition unit 12 acquires the position of the sun as a light source based on calculation using the vehicle position information G and time t. Further, the light source position acquisition unit 12 acquires light source position information indicating the relative position of the light source with respect to the vehicle when the event E occurs, based on the image I captured by the exterior camera 51 . The light source may be the headlights of an oncoming vehicle traveling in the oncoming lane. In this case, the position of the light source is calculated based on the image of the camera outside the vehicle or the detection signal of the ranging sensor.

The shade area acquisition unit 13 obtains shade information indicating the shade area when the event E occurs based on the map information M including the information on the road and the structure, the vehicle position information G, the light source position information S, and the event occurrence information A. to get

Based on the light source position information and the shade information, the determination unit 14 determines whether or not the driver of the vehicle when the event E occurred was in a backlit state in which there was light entering the driver's eyes from the light source. make a judgment. Note that the occurrence time of the event E includes a predetermined time immediately before the moment when the event E occurred. The predetermined time is, for example, 0.5 seconds, 1 second, 2 seconds, or 3 seconds. In addition, determination may be made using a plurality of different times, instead of the predetermined time.

FIG. 2 is a diagram showing the hardware configuration of the backlight determination device 10 according to Embodiment 1. As shown in FIG. As shown in FIG. 2, the backlight determination device 10 includes a processor 101 such as a CPU (Central Processing Unit), a memory 102 that is a volatile storage device, and a non-volatile memory such as a hard disk drive (HDD) or solid state drive (SDD). and a storage device 103 for the data. The memory 102 is, for example, a volatile semiconductor memory such as a RAM (Random Access Memory).

Each function of the backlight determination device 10 is realized by a processing circuit. The processing circuitry may be dedicated hardware or processor 101 executing a program stored in memory 102 . The processor 101 may be any of a processing device, an arithmetic device, a microprocessor, a microcomputer, and a DSP (Digital Signal Processor).

If the processing circuit is dedicated hardware, the processing circuit may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) ), or a combination of any of these.

When the processing circuit is the processor 151, the backlight determination program is realized by software, firmware, or a combination of software and firmware. Software and firmware are written as programs and stored in memory 102 . The processor 101 reads and executes the backlight determination program stored in the memory 102, thereby implementing the functions of the units shown in FIG.

It should be noted that the backlight determination device 10 may be partially realized by dedicated hardware and partially realized by software or firmware. As such, the processing circuitry may implement each of the functions described above in hardware, software, firmware, or any combination thereof.

FIG. 3(A) is a diagram showing an example when an event occurs (when there is no shade), and FIG. 3(B) is a diagram showing the position of the sun relative to the vehicle. As shown in FIG. 3(B), when event E occurs, a first angle .alpha. is within a predetermined first angle range, and the second angle β, which is the vertical angle formed by the central ray with respect to the traveling direction, is within a predetermined second angle range Moreover, if the driver is outside the shaded area, the determination unit 14 determines that the driver was in the backlight state when event E occurred. The first angle α is the azimuth angle. The second angle is the elevation angle.

FIG. 4(A) is a diagram showing an example when an event occurs (when there is a shade), and FIG. 4(B) is a diagram showing an example when an event occurs (when there is a shade). If the driver is in the shadow area when event E occurs, the determining unit determines that the driver was in a non-backlit state when event E occurred. For example, as shown in FIG. 4A, if there is a shaded area at the time of event E caused by a building, which is a structure shown in the map information, the driver's state of the vehicle in this shaded area is non-backlit. state. In addition, as shown in FIG. 4(B), if there is a shadow due to the presence of a large vehicle in front of the camera image when event E occurs, the driver of the vehicle in the shaded area is in a non-backlit state. .

FIG. 5 is a flow chart showing the shadow area detection operation of the backlight determination device 10 according to the first embodiment. The backlight determination device 10 determines whether or not there is an impact based on the output of the acceleration sensor, and determines that an event has occurred when, for example, the impact exceeds a threshold value (step S101). An event in this case is, for example, a situation where the vehicle has hit or been hit by something.

Next, the backlight determination device 10 acquires vehicle position information by GPS (step S102).

Next, the backlight determination device 10 detects the map information, the image of the camera outside the vehicle, the structure (for example, a building) or the surrounding object (for example, a large vehicle ahead on the road) detected by the ranging sensor (step S103).

Next, the backlight determination device 10 estimates (that is, calculates) the position of the sun, which is the light source, from the vehicle position information, the surrounding object information, and the time (step S104).

If there is no surrounding object (NO in step S105), the backlight determination device 10 determines whether or not the vehicle receives light based on the positional relationship (that is, relative position) between the vehicle and the light source (step S106).

If there is a surrounding object (YES in step S105), the backlight determination device 10 determines whether the surrounding object is a structure registered in the map information (step S107).

If the structure is not registered in the map information (NO in step S107), the backlight determination device 10 acquires the position and shape of the object using the camera outside the vehicle or the distance measuring sensor (step S108). The presence or absence of light entering the vehicle is determined from the position information of (step S110).

If the structure is registered in the map information (YES in step S107), the backlight determination apparatus 10 acquires the position and shape of the object from the map information (step S109), It is determined whether or not light enters the vehicle (step S110).

As described above, if the backlight determination device 10, the light shielding determination method, or the light shielding determination program according to the first embodiment is used, it is possible to determine whether the driver is in a backlight state or a non-backlight state when a predetermined event occurs. can be determined accurately.

Embodiment 2.
In Embodiment 1, the case where the light from the light source is directly incident on the eyes of the driver of the vehicle has been described. However, there are cases where a structure has a light reflecting area, such as a building whose walls are made of glass windows. Embodiment 2 describes a backlight determination device 20 that enables determination in consideration of reflected light from a structure having a light reflection area.

FIG. 6 is a functional block diagram schematically showing the configuration of the backlight determination device 20 according to the second embodiment. The backlight determination device 20 is a device capable of executing the backlight determination method according to the second embodiment. The backlight determination device 20 is, for example, a computer. The backlight determination device 20 can execute the backlight determination method according to the second embodiment by executing the backlight determination program.

As shown in FIG. 6, the backlight determination device 20 includes an information acquisition unit 21, a light source position acquisition unit 22, a reflected light vector acquisition unit 23, a shade area acquisition unit 24, and a determination unit 25. there is
The information acquisition unit 21 and the light source position acquisition unit 22 are the same as the information acquisition unit 11 and the light source position acquisition unit 12 in the first embodiment. The backlight determination device 20 differs from the backlight determination device 10 according to the first embodiment in that it has a reflected light vector acquisition unit 23 and a shadow area acquisition unit 24 .

FIG. 7 is a diagram showing the calculation process of the reflected light vector. First, the reflected light vector acquisition unit 23 grasps the reflection state of light from surrounding objects, determines whether or not the light is incident on the vehicle, and records the result. For the reflection area, for example, the presence or absence of reflection is determined from the image of the camera outside the vehicle. After the reflection area is determined by the external camera, the light reflection surface is calculated from the measurement result of the distance measuring sensor or the map information. From the positional relationship between the light and the reflecting surface, the normal vector of the reflecting surface is calculated.

FIG. 8 is a flowchart showing the reflected light detection operation of the backlight determination device 20 according to the second embodiment. The backlight determination device 20 determines whether or not there is an impact based on the output of the acceleration sensor, and determines that an event has occurred when, for example, the impact exceeds a threshold value (step S201).

Next, the backlight determination device 20 acquires vehicle position information by GPS (step S202).

Next, the backlight determination device 20 detects the map information, the image of the camera outside the vehicle, the structure (for example, building) or the surrounding object (for example, a large vehicle ahead on the road) detected by the ranging sensor (step S203).

Next, the backlight determination device 20 estimates (that is, calculates) the position of the sun, which is the light source, from the vehicle position information, the surrounding object information, and the time (step S204).

If there is no surrounding object (NO in step S205), the backlight determination device 10 determines whether or not the vehicle receives light based on the positional relationship (that is, relative position) between the vehicle and the light source (step S106).

If there is a surrounding object (YES in step S205), the backlight determination device 10 determines whether the surrounding object is a structure registered in the map information (step S207).

If the structure is not registered in the map information (NO in step S207), the backlight determination device 20 acquires the position, shape, and surface of the object using the camera outside the vehicle or the distance measuring sensor (step S208), and determines the surface of each object. In step S210, the presence or absence of light incident on the vehicle is determined based on the positional information of the overexposed area by the camera.

If the structure is registered in the map information (YES in step S207), the backlight determination device 20 acquires the position, shape and surface of the object from the map information (step S209), Whether or not there is light entering the vehicle is determined based on the positional information of the kite area (step S210).

If there is a highlighted area (YES in step S211), the backlight determination device 20 determines whether or not the vehicle has entered the port (step S212). An incident light vector to the vehicle is calculated from the light source position information and the surface of the object (step S213).

As described above, if the backlight determination device 20, the light shielding determination method, or the light shielding determination program according to the second embodiment is used, it is possible to determine whether the driver is in a backlight state or a non-backlight state when a predetermined event occurs. can be determined accurately.

Except for the above, the second embodiment is the same as the first embodiment.

Embodiment 3.
Embodiments 1 and 2 above do not use the driver's facial expression (for example, squinting) when the driver is dazzled. In Embodiment 3, an example will be described in which the presence or absence of backlight is determined based on the facial expression of the driver. However, it is possible to combine the configuration of the third embodiment with the backlight determination device according to the first or second embodiment. For example, backlight determination can be performed using the determination result of the backlight determination apparatus according to the first or second embodiment and the determination result of the backlight determination apparatus according to the third embodiment.

FIG. 9 is a functional block diagram schematically showing the configuration of the backlight determination device 30 according to Embodiment 3. As shown in FIG. The backlight determination device 30 is a device capable of executing the backlight determination method according to the third embodiment. The backlight determination device 30 is, for example, a computer. The backlight determination device 30 can execute the backlight determination method according to the third embodiment by executing the backlight determination program.

As shown in FIG. 9, the backlight determination device 30 has an information acquisition section 31 and an expression determination section 32 as a determination section. The facial expression determination unit 32 uses a facial expression determination model, which is a learned model generated in advance by machine learning (for example, deep learning), to determine whether or not the facial expression of the driver is that of a dazzling expression. . The backlight determining device 30 determines whether or not the vehicle is in a backlight state in which light is incident on the eyes of the driver based on the information stored in the storage unit 62 of the drive recorder 60 of the vehicle. The information stored in the storage unit 62 of the drive recorder 60 is obtained from the in-vehicle sensor 50 while the vehicle is running. The in-vehicle sensor 50 has, for example, an in-vehicle camera 55 that captures the face of the driver inside the vehicle, and an acceleration sensor 54 that is used to detect a predetermined event. In addition, the vehicle-mounted sensor 50 may be provided with the ranging sensor 52, GPS53, and the vehicle exterior camera 51 like Embodiment 1 or 2. FIG.

FIG. 10 is a diagram showing the hardware configuration of the backlight determination device 30 according to the third embodiment. As shown in FIG. 10, the backlight determination device 20 has a processor 301 such as a CPU, a memory 302 which is a volatile storage device, and a storage device 303 such as an HDD or SDD. Memory 302 is, for example, a RAM.

Each function of the backlight determination device 30 is implemented by a processing circuit, as in the case of the first embodiment. The processing circuitry may be dedicated hardware or processor 101 executing a program stored in memory 102 .

FIG. 11 is a diagram showing the configuration for learning the facial expression determination model and the configuration for utilizing the display determination model. In the learning of FIG. 11, the facial expression of the driver is detected, and the facial expression in glare is used as a teacher, and the facial expression is classified by a model learned by deep learning, and classified into the face in glare and other faces. A facial expression is recorded for several seconds immediately before event E, and a pre-learned facial expression determination model is used to determine whether or not the subject is feeling glare. It is also possible to determine the glare step by step according to the accuracy rate of the classification.

FIG. 12 is a flowchart showing facial expression detection processing of the backlight determination device 30 according to the third embodiment. The backlight determination device 30 determines whether or not there is an impact based on the output of the acceleration sensor, and determines that an event has occurred when, for example, the impact exceeds a threshold value (step S301).

Next, the backlight determination device 30 acquires a camera image of the driver's expression at the time of the occurrence of the event (including several seconds immediately before) (step S202).

Next, the backlight determination device 30 determines how the driver's eyes are opened or closed (step S303). This is because people tend to make facial expressions such as narrowing their eyes or lowering the corners of their eyes when the light is bright.

Next, the backlight determination device 30 determines whether or not the length of time during which the driver feels glare is equal to or greater than a predetermined threshold (step S304).

If the value is less than the threshold (NO in step S304), the backlight determination device 30 determines that the driver does not have a dazzling facial expression. If the value is equal to or greater than the threshold (YES in step S304), the backlight determination device 30 performs glare determination (value 0 or 1) using the facial expression determination model, which is a learned model, for a plurality of frames immediately before the occurrence of the event. (step S306). At this time, the degree of glare is calculated by assigning a larger weight to the glare determination value for a frame closer to the occurrence of the event (step S307). The weight increases linearly in proportion to time, for example, closer to the event occurrence time.

If there is a surrounding object (YES in step S304), the backlight determination device 30 determines whether the surrounding object is a structure registered in the map information (step S107).

As described above, if the backlight determination device 30, the light shielding determination method, or the light shielding determination program according to the third embodiment is used, a predetermined expression using a learned model of the expression when feeling glare is determined in advance. It is possible to accurately determine whether the driver was in a backlit or non-backlit condition when the event occurred.

The determination of the third embodiment may be performed in addition to the determination of at least one of the first and second embodiments. If a plurality of determination methods are used together, the determination accuracy is improved.

Embodiment 4.
Embodiments 1 to 3 described above do not utilize the change in the pupil that the driver exhibits when the driver is dazzled. In Embodiment 4, an example will be described in which the presence or absence of backlight is determined based on the driver's pupils. However, the configuration of Embodiment 4 can be combined with any one or more of the backlight determination devices of Embodiments 1 to 3. For example, backlight determination can be performed using the determination result of the backlight determination apparatus according to the first or second embodiment and the determination result of the backlight determination apparatus according to the third embodiment.

FIG. 13 is a functional block diagram schematically showing the configuration of the backlight determination device 40 according to the fourth embodiment. The backlight determination device 40 is a device capable of executing the backlight determination method according to the fourth embodiment. The backlight determination device 40 is, for example, a computer. The backlight determination device 40 can execute the backlight determination method according to the fourth embodiment by executing the backlight determination program. As shown in FIG. 13, the backlight determination device 40 includes an information acquisition unit 41, an eyelid opening/closing detection unit 42 that detects the opening and closing of the eyes (eyelids) based on the face image, and a light reflection detection unit that detects the light reflection of the pupil. It has a reflection detection section 43, a corneal reflection detection section 44 for detecting reflection from the cornea, and an eye determination section 45 as a determination section. It is the same as that of 3.

FIG. 14 is a diagram showing reflection detection processing of the backlight determination device 40 according to the fourth embodiment. Humans close their eyes (eyelids) when they feel glare, but it is unknown whether they close their eyes due to the influence of incoming light or by chance. Therefore, it is determined whether or not the eyes are closed due to the backlight by combining with the reflected light vector. In other words, the backlight determination device 40 determines that the backlight state is present when there is reflected light entering the eyes detected by the method of the first or second embodiment and the eyes are closed.

Also, when the eyes are not closed, the corneal reflex is detected first, then the pupil becomes smaller and the light reflex occurs. Therefore, when the corneal reflection is detected by the corneal reflection detection unit 44 and then the opposite reflection is detected by the light reflection detection unit 43, it is determined that the backlight state is present. The order of nerve transmission until the pupil reacts is the following order, which delays the reaction of the pupil. Light input, retina, optic nerve, optic chiasm, optic tract, pretectal olivary nucleus, Edinger-Westphal nucleus, oculomotor nerve, ciliary ganglion, sphincter pupillae.

FIG. 15 is a flowchart showing eyelid detection processing of the backlight determination device 40 according to the fourth embodiment. The backlight determination device 40 determines whether or not there is an impact based on the output of the acceleration sensor, and determines that an event has occurred when, for example, the impact exceeds a threshold value (step S401).

Next, the backlight determination device 40 acquires the camera image of the driver's eyes at the time of the occurrence of the event (including several seconds immediately before) (step S402).

Next, the backlight determination device 40 determines the degree of opening and closing of the driver's eyes (eyelids) (step S403). This is because people tend to have an expression of closing their eyes when it is bright. Information for several seconds before the occurrence of the event is recorded, and if the time until the eyelids in the open state become closed is greater than or equal to the threshold, it is judged to be open.

Next, the backlight determination device 40 determines whether or not the eyelids are open (step S404).

When the eyelids are closed (NO in step S404), the backlight determination device 40 determines whether light is received by the method described in the first or second embodiment.

If the eyelids are open (YES in step S404), the backlight determination device 40 detects the presence or absence of corneal reflection for each frame (step S406). The presence or absence of light reflection is detected (step S408), and the process proceeds to step S409.

If there is corneal reflection (YES in step S407), the backlight determination device 40 weights the eyelid open/close value closer to the moment of event occurrence for each frame (step S407). The weight increases linearly in proportion to time, for example, closer to the event occurrence time.

As described above, if the backlight determination device 40, the light shielding determination method, or the light shielding determination program according to the fourth embodiment is used, the presence or absence of corneal reflection and light reflection can be determined in advance based on the detection results. It can be accurately determined whether the driver was backlit or non-backlit when the event occurred.

In addition to the determination of at least one of Embodiments 1 to 3, the determination of Embodiment 4 may be performed. If a plurality of determination methods are used together, the determination accuracy is improved.

Except for the above, the fourth embodiment is the same as the third embodiment.

Modification.
The judgment results of the backlight judgment devices of Embodiments 1 to 4 can be applied to a vehicle driving support device. For example, among the events whose number of occurrences should be reduced, for events determined to be caused by backlight, a device that warns the driver of a vehicle that is traveling at a time and in a place where backlight is likely to occur. In this case, the driver can know the time and place where backlighting is likely to occur before arriving at that place, and can take countermeasures.

10, 20, 30, 40

Backlight determination device

11, 21, 31, 41

Information acquisition unit

12, 22 Light source position acquisition unit 13 Shade

area acquisition unit

14, 25 Judgment unit 23 Reflected light vector acquisition unit 32 Facial expression determination unit, 42 Eyelid open/close detection unit, 43 Light reflection detection unit, 44 Corneal reflection detection unit, 50 In-vehicle sensor, 51 External camera, 52 Ranging sensor, 53 GPS, 54 Acceleration sensor, 55 In-vehicle camera, 60 Drive recorder , 62 storage unit, 63 map information.

Claims

event occurrence information indicating that a predetermined event has occurred in the vehicle; vehicle position information indicating the position of the vehicle when the event occurred; an image taken outside the vehicle when the event occurred; an information acquisition unit that acquires distance information to peripheral objects obtained by detecting the outside of the vehicle at the time;
a light source position acquisition unit that acquires light source position information indicating the relative position of the light source with respect to the vehicle at the time of occurrence, based on calculation using the vehicle position information and time or based on the image;
a shade area acquisition unit that acquires shade information indicating a shade area at the time of occurrence based on map information including road and structure information, the vehicle position information, the light source position information, and the event occurrence information;
Based on the light source position information and the shade information, it is determined whether or not the state of the driver of the vehicle at the time of occurrence was a backlit state in which light from the light source is incident on the eyes of the driver. a determination unit that performs
A backlight determination device characterized by comprising:
further comprising a reflected light vector acquisition unit for calculating a reflected light vector from the light reflection area of the structure at the time of occurrence based on the map information, the vehicle position information, the light source position information, and the event occurrence information. have
The determining unit determines whether or not the state of the driver of the vehicle at the time of occurrence was a backlit state in which light from the light source enters the driver's eyes via the light reflecting region. 2. The backlight determination device according to claim 1, wherein:
a first angle, which is a horizontal angle formed by the center ray of light incident on the eye of the driver at the time of occurrence, with respect to the running direction of the vehicle is within a predetermined first angle range; Further, when the second angle, which is the vertical angle formed by the central ray with respect to the running direction, is within a predetermined second angle range, the driver is outside the shade area. 3. The backlight determination device according to claim 1, wherein the determination unit determines that the driver's state is the backlight state at the time of occurrence.
4. Any one of claims 1 to 3, characterized in that, if the driver is in the shaded area at the time of occurrence, the determining unit determines that the state of the driver at the time of occurrence was a non-backlit state. 1. A backlight determination device according to claim 1.
The backlight determination apparatus according to any one of claims 1 to 4, wherein the shade area is formed by the structure or the surrounding object indicated by the map information.
The backlight determination device according to any one of claims 1 to 5, wherein the time of occurrence includes a predetermined time immediately before the moment when the event occurred.
7. The information acquisition unit according to any one of claims 1 to 6, wherein the information acquisition unit acquires the event occurrence information, the vehicle position information, the image, and the distance information from a drive recorder of the vehicle. backlight judgment device.
an acquisition unit that acquires an image output from an in-vehicle camera that captures the face of a driver of the vehicle and event occurrence information that indicates that a predetermined event has occurred in the vehicle;
Using a learned model generated by prior machine learning to classify facial expressions when backlight enters the eye, the driver's state changes from a light source to the vehicle at the time of the event. a determination unit that determines whether or not the vehicle is in a backlight state in which light is incident on the eyes of the driver;
A backlight determination device characterized by comprising:
an acquisition unit that acquires an image output from an in-vehicle camera that captures the eyes of a driver of the vehicle and event occurrence information that indicates that a predetermined event has occurred in the vehicle;
Based on whether corneal reflection and counter reflection occurred in sequence in the driver's eye at the time of the event, the driver's condition is determined by the presence of light from a light source entering the vehicle's driver's eye. A determination unit that determines whether or not it was a backlight state,
A backlight determination device characterized by comprising:
10. The backlight determination device according to claim 8 or 9, wherein the time of occurrence includes a predetermined time immediately before the moment when the event occurred.
The judging unit judges whether or not the driver's state is the backlight state by using images of a plurality of frames within a predetermined time period immediately before the moment when the event occurs. The backlight determination device according to claim 10.
The determining unit treats an image of a frame closer to the instant at which the predetermined event occurs as data having a larger weight value, and determines whether the driver's condition is determined by the light incident on the eyes of the driver of the vehicle from a light source. 12. The backlight determination device according to claim 11, wherein it is determined whether or not there was a backlight condition in which a
The backlight determination device according to any one of claims 8 to 12, wherein the event occurrence information and the image are obtained from a drive recorder of the vehicle.
The backlight determination device according to any one of claims 1 to 13, wherein the light source is the sun or headlights of another vehicle traveling in an oncoming lane.
A backlight determination method executed by a backlight determination device, comprising:
event occurrence information indicating that a predetermined event has occurred in the vehicle; vehicle position information indicating the position of the vehicle when the event occurred; an image taken outside the vehicle when the event occurred; acquiring distance information to surrounding objects obtained by detecting the outside of the vehicle at the time;
obtaining light source position information indicative of the position of a light source relative to the vehicle at the time of occurrence, based on calculations using the vehicle position information and the time of day, or based on the image;
acquiring shade information indicating a shaded area at the time of occurrence based on map information including road and structure information, the vehicle position information, the light source position information, and the event occurrence information;
Based on the light source position information and the shade information, it is determined whether or not the state of the driver of the vehicle at the time of occurrence was a backlit state in which light from the light source is incident on the eyes of the driver. the steps to take; and
A backlight determination method, comprising:
A backlight determination method executed by a backlight determination device,
obtaining an image output from an in-vehicle camera that captures the face of a driver of the vehicle and event occurrence information indicating that a predetermined event has occurred in the vehicle;
Using a learned model generated by prior machine learning to classify facial expressions when backlight enters the eye, the driver's state changes from a light source to the vehicle at the time of the event. determining whether or not the vehicle is in a backlight condition in which there is light incident on the eyes of the driver;
A backlight determination method, comprising:
A backlight determination method executed by a backlight determination device,
obtaining an image output from an in-vehicle camera that captures the eyes of a driver of the vehicle and event occurrence information indicating that a predetermined event has occurred in the vehicle;
Based on whether corneal reflection and counter reflection occurred in sequence in the driver's eye at the time of the event, the driver's condition is determined by the presence of light from a light source entering the vehicle's driver's eye. a step of determining whether or not it was a backlight condition;
A backlight determination method, comprising:
A backlight determination program characterized by causing a computer, which is a backlight determination device, to execute the backlight determination method according to any one of claims 15 to 17.