WO2017199709A1 - Face orientation estimation device and face orientation estimation method - Google Patents

Abstract

Provided is a face orientation estimation device, comprising: a face orientation information acquisition unit (71) which acquires a detection result in which the face orientation of a driver (DR) of a mobile body (110) has been detected; a count processing unit (76) which, on the basis of the detection result, counts the number of times that the driver has viewed objects to be viewed which are installed upon the mobile body; and an action assessment unit (77) which assesses whether a viewing action of the driver who views the objects to be viewed has changed due to a change in the position of a vanishing point in the foreground which the driver views. If the viewing action of the driver has changed, the count processing unit changes the description of the count process in accordance with the change in the position of the vanishing point.

Description

Face orientation estimating apparatus and face orientation estimating method Cross-reference of related applications

This application is based on Japanese Patent Application No. 2016-101244 filed on May 20, 2016, the contents of which are incorporated herein by reference.

The present disclosure relates to a face direction estimation device and a face direction estimation method for estimating the face direction of a driver who drives a vehicle or the like.

Conventionally, for example, Patent Document 1 discloses a technique in which a driver's face driving a moving body such as a vehicle is photographed by a camera and the driver's face orientation is recognized based on the photographed image. Specifically, the attention reduction detection system of Patent Document 1 counts the number of times the driver visually recognizes the side mirror installed in the vehicle based on the recognized face orientation. Then, based on the counted number of times the side mirror is visually recognized, a decrease in the driver's attention is determined.

JP 2014-48885 A

Now, the face orientation when the driver is viewing the side mirror differs depending on the driver's physique, sitting position, sitting posture, and the like. Therefore, the inventors of the present disclosure specify the position of the visual recognition object from the visual behavior of the driver, and compare the position of the visual recognition object thus identified with the recognized driver's face direction. We have been developing a technology that counts the number of times a visual target is viewed.

However, with the technology for identifying the position of the visual recognition object from the visual behavior of the driver, the inventors of the present disclosure have a problem that the accuracy of counting the number of visual recognitions is reduced due to road environments such as curves and slopes. Newly discovered. More specifically, the driver usually changes the face direction from the vanishing point to the visual recognition object with the vanishing point in the foreground as the zero point position. Therefore, when traveling on a curve, a slope, or the like, if the position of the vanishing point in the foreground changes, the mode of visual behavior that changes the face direction from the vanishing point to the visual target object also changes. For this reason, if the driver's visual behavior changes due to the position change of the vanishing point, the number of times the driver visually recognizes the visual target object may not be counted correctly.

The present disclosure relates to a face direction estimation device and a face direction that can accurately count the number of times a driver visually recognizes a visual target object even if the driver's visual behavior changes due to a change in the position of the vanishing point. An object is to provide an estimation method.

In the first aspect of the present disclosure, the face direction estimation device includes a face direction information acquisition unit that acquires a detection result of detecting a face direction of a driver who drives a moving body, and the face direction information acquisition unit Based on the detection result, the counting processing unit that counts the number of times the driver visually recognizes the visual target object installed on the moving body, and due to the position change of the vanishing point in the foreground visually recognized by the driver An action determination unit that determines whether or not the driver's visual action for visually recognizing the visual target object has changed. When the behavior determination unit determines that the driver's visual behavior has changed, the counting processing unit changes the content of the counting process in accordance with the position change of the vanishing point.

According to the above face orientation estimation device, when the moving body travels on a curve or the like, and the position change of the vanishing point in the foreground occurs, the driver's visual behavior changes due to such a position change. Can be determined. As a result, the process of counting the number of times the driver visually recognizes the visual target object installed on the moving body is changed to the content corresponding to the change in the driver's visual behavior in accordance with the change in the position of the vanishing point. Therefore, even if the driver's visual behavior changes due to the position change of the vanishing point, the number of times that the driver visually recognizes the visual target object can be accurately counted.

According to a second aspect of the present disclosure, there is provided a face direction estimation method for estimating a face direction of a driver who drives a moving body, wherein at least one processor includes a face direction detection unit that detects the face direction of the driver. Based on the detection result, the driver counts the number of times the driver visually recognizes the visually recognized object installed on the moving body, and the visually recognized object is caused by a change in the position of the vanishing point in the foreground visually recognized by the driver. It is determined whether or not the driver's visual behavior has changed, and when it is determined that the driver's visual behavior has changed, the content of the counting process is changed in accordance with the position change of the vanishing point. To do.

According to the face direction estimation method described above, when the position of the vanishing point in the foreground occurs due to the moving body traveling on a curve or the like, the driver's visual behavior changes due to such a position change. Can be determined. As a result, the process of counting the number of times the driver visually recognizes the visual target object installed on the moving body is changed to the content corresponding to the change in the driver's visual behavior in accordance with the change in the position of the vanishing point. Therefore, even if the driver's visual behavior changes due to the position change of the vanishing point, the number of times that the driver visually recognizes the visual target object can be accurately counted.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. The drawing
FIG. 1 is a block diagram showing the entire face direction estimation system according to the first embodiment. FIG. 2 is a diagram showing a vehicle equipped with a face orientation estimation system, FIG. 3 is a diagram showing a form of a wearable device, FIG. 4 is a diagram showing functional blocks constructed in the in-vehicle controller. 5 (a) and 5 (b) are diagrams showing a mode of a two-dimensional distribution of visual recognition points in a straight section and a histogram thereof. 6 (a) and 6 (b) are diagrams showing a mode of a two-dimensional distribution of visual recognition points in the left curve section and a histogram thereof. FIG. 7 is a diagram illustrating a change in the line-of-sight direction to the yaw direction in a straight section and a threshold value for counting the number of times the left and right side mirrors are viewed. FIG. 8 is a diagram illustrating a change in the line-of-sight direction to the yaw direction in the left curve section and a threshold value for counting the number of times the left and right side mirrors are viewed FIG. 9 is a flowchart showing each process performed in the in-vehicle control unit. FIG. 10 is a diagram comparing the respective false detection and non-detection probabilities when detecting the visual behavior of the left and right side mirrors by the driver. FIG. 11 is a block diagram showing the entire face direction estimation system according to the second embodiment.

Hereinafter, a plurality of embodiments of the present disclosure will be described with reference to the drawings. In addition, the overlapping description may be abbreviate | omitted by attaching | subjecting the same code | symbol to the corresponding component in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other part of the configuration. Moreover, not only the combination of the configurations explicitly described in the description of each embodiment, but also the configuration of a plurality of embodiments can be partially combined even if they are not explicitly described, as long as there is no problem in the combination. And the combination where the structure described in several embodiment and the modification is not specified shall also be disclosed by the following description.

(First embodiment)
The face orientation estimation system 100 according to the first embodiment of the present disclosure includes a wearable device 10 and an in-vehicle device 40 that can communicate with each other, as illustrated in FIGS. 1 and 2. The face orientation estimation system 100 mainly functions in the passenger compartment of a vehicle 110 that is a moving object. The face direction estimation system 100 detects the behavior of the head HD of a driver DR who gets on the vehicle 110 and drives the vehicle 110 by the wearable device 10. The face orientation estimation system 100 uses the in-vehicle device 40 to calculate the face orientation of the driver DR from the detected behavior of the head HD.

The driver DR's face direction information by the face direction estimation system 100 determines, for example, a decrease in the confirmation frequency for visually recognizing each mirror installed in the vehicle 110, a long look aside, terminal operation in the downward direction, and dozing. Used for applications that When an abnormality of the driver DR as described above is detected, an actuation such as a warning for the driver DR is executed.

The wearable device 10 is a glasses-type motion sensor device in which the detection circuit 20 is mounted on the glasses 10a as shown in FIG. The wearable device 10 shown in FIGS. 1 and 2 is mounted on the head HD of the driver DR and sequentially outputs the face orientation detection results toward the in-vehicle device 40. The detection circuit 20 of the wearable device 10 includes a face direction detection unit 11, a communication unit 17, an operation unit 18, a battery 19, and the like.

The face orientation detection unit 11 is a motion sensor mounted on the glasses 10a (see FIG. 3). The face orientation detection unit 11 detects a value related to the face orientation of the driver DR by mounting the wearable device 10 on the head HD of the driver DR. The face orientation detection unit 11 includes an electrooculogram sensor 12, a gyro sensor 13, and an acceleration sensor 14. The face orientation detection unit 11 sequentially provides the detection results acquired using the sensors 12 to 14 to the communication unit 17.

The electrooculogram sensor 12 has a plurality of electrodes 12a (see FIG. 3) formed of a metal material such as stainless steel. Each electrode 12a is provided on the frame of the glasses 10a so as to be spaced apart from each other. When the driver DR as a user wears the wearable device 10, each electrode 12 a comes into contact with the skin of the driver DR in the vicinity of the eyeball of the driver DR. The electrooculogram sensor 12 measures the electrooculogram that changes with the eye movement of the driver DR based on the potential difference between two points of each electrode 12a. Measurement data (hereinafter referred to as “ocular potential data”) from the electrooculogram sensor 12 is sequentially provided to the communication unit 17 as a detection result.

The gyro sensor 13 is a sensor that detects angular velocity as a voltage value. The gyro sensor 13 measures the magnitude of the angular velocity generated around each of the three axes that are orthogonal to each other defined by the face orientation detection unit 11 when the glasses 10a are attached to the head HD. Based on the angular velocity around each axis, there is an operation of shaking the head HD in the horizontal (yaw) direction, an operation of shaking the head HD in the vertical (pitch) direction, and an operation in the direction of tilting the head HD left and right (roll). Detected. Measurement data (hereinafter referred to as “angular velocity data”) by the gyro sensor 13 is sequentially provided to the communication unit 17 as a detection result.

The acceleration sensor 14 is a sensor that detects acceleration as a voltage value. The acceleration sensor 14 measures the magnitude of acceleration acting on the head HD along the respective axial directions of the three axes defined in the face direction detection unit 11 by being worn on the head HD of the glasses 10a. To do. Data measured by the acceleration sensor 14 (hereinafter, “acceleration data”) is sequentially provided to the communication unit 17 as a detection result.

The communication unit 17 can transmit / receive information to / from the in-vehicle device 40 by wireless communication using, for example, Bluetooth (registered trademark) and a wireless LAN. The communication unit 17 has an antenna corresponding to a wireless communication standard. The communication unit 17 is electrically connected to the face orientation detection unit 11 and acquires measurement data related to the face orientation output from the sensors 12 to 14. When a connection by wireless communication is established with the in-vehicle device 40, the communication unit 17 sequentially encodes each input measurement data and transmits it to the in-vehicle device 40.

The operation unit 18 includes a power switch for switching the power of the wearable device 10 between an on state and an off state. The battery 19 is a power source that supplies power for operation to the face direction detection unit 11 and the communication unit 17. The battery 19 may be a primary battery such as a lithium battery or a secondary battery such as a lithium ion battery.

The in-vehicle device 40 is one of a plurality of electronic control units mounted on the vehicle 110. The in-vehicle device 40 operates using electric power supplied from the in-vehicle power source. The in-vehicle device 40 is connected with an HMI control unit 50, a vehicle speed sensor 55, a locator 56, and the like. The in-vehicle device 40 may be configured so that it can be brought into the vehicle 110 by the driver and fixed to the vehicle body later.

The HMI (Human Machine Interface) control unit 50 is one of a plurality of electronic control units mounted on the vehicle 110. The HMI control unit 50 controls the acquisition of the operation information input by the driver DR and the information presentation to the driver DR in an integrated manner. The HMI control unit 50 is connected to, for example, a display 51, a speaker 52, an input device 53, and the like. The HMI control unit 50 presents information to the driver DR by controlling the display on the display 51 and the sound output by the speaker 52.

The vehicle speed sensor 55 is a sensor that measures the traveling speed of the vehicle 110. The vehicle speed sensor 55 outputs a vehicle speed pulse corresponding to the traveling speed of the vehicle 110 to the in-vehicle device 40. The vehicle speed information calculated based on the vehicle speed pulse may be supplied to the in-vehicle device 40 via an in-vehicle communication bus or the like.

The locator 56 includes a GNSS receiver 57, a map database 58, and the like. A GNSS (Global Navigation Satellite System) receiver 57 receives positioning signals from a plurality of artificial satellites. The map database 58 stores map data including road shapes such as road longitudinal gradient and curvature. Locator 56 sequentially measures the current position of vehicle 110 based on the positioning signal received by GNSS receiver 57. The locator 56 reads out map data around the vehicle 110 and the traveling direction from the map database 58 and sequentially provides it to the in-vehicle device 40. As described above, the in-vehicle device 40 can acquire the shape information of the road on which the vehicle 110 is scheduled to travel.

The in-vehicle device 40 connected to each of the above components includes a memory 46, a communication unit 47, and an in-vehicle control unit 45.

The memory 46 is a non-transitional tangible storage medium such as a flash memory. The memory 46 stores an application program and the like necessary for the operation of the in-vehicle device 40. Data in the memory 46 can be read and rewritten by the in-vehicle control unit 45. The memory 46 may be configured to be built in the in-vehicle device 40, or may be configured to be inserted into a card slot provided in the in-vehicle device 40 in the form of a memory card or the like.

In addition, the memory 46 has a first storage area 46a and a second storage area 46b. In each of the first storage area 46a and the second storage area 46b, data related to the detection result of the face orientation classified by the in-vehicle control unit 45 is accumulated, as will be described later.

The communication unit 47 transmits and receives information to and from the wearable device 10 by wireless communication. The communication unit 47 has an antenna corresponding to a wireless communication standard. The communication unit 47 sequentially acquires measurement data from the sensors 12 to 14 by decoding the radio signal received from the communication unit 17. The communication unit 47 outputs each acquired measurement data to the in-vehicle control unit 45.

The in-vehicle control unit 45 is mainly composed of a microcomputer having a processor 45a, a RAM, an input / output interface, and the like. The in-vehicle control unit 45 executes the face orientation estimation program stored in the memory 46 by the processor 45a. As a result, the in-vehicle control unit 45 includes a driver information acquisition unit 71, a travel information acquisition unit 72, a travel section determination unit 73, a stop determination unit 74, a face orientation correction unit 75, a count processing unit 76, an action shown in FIG. Functional blocks such as a determination unit 77 and a warning control unit 78 are constructed. The details of each functional block will be described below with reference to FIGS.

The driver information acquisition unit 71 acquires the detection result by the face direction detection unit 11 received by the communication unit 47 from the communication unit 47. Specifically, the driver information acquisition unit 71 acquires electrooculogram data, angular velocity data, and acceleration data.

The traveling information acquisition unit 72 sequentially acquires road shape information output from the locator 56. In addition, the travel information acquisition unit 72 acquires the current vehicle speed information of the vehicle 110 based on the vehicle speed pulse output from the vehicle speed sensor 55.

Based on the road shape information acquired by the travel information acquisition unit 72, the travel section determination unit 73 determines whether the vehicle 110 is traveling in a straight section and whether the vehicle 110 is traveling in a horizontal section without inclination. It is determined whether or not. Specifically, when the curvature of the road ahead of the vehicle is smaller than a predetermined threshold, the travel section determination unit 73 determines that the vehicle is traveling in a straight section. On the other hand, when the curvature of the road ahead of the vehicle is larger than the predetermined curvature, the traveling section determination unit 73 determines that the vehicle is traveling in a curved section instead of a straight section.

Similarly, when the longitudinal gradient on the road ahead of the vehicle is smaller than a predetermined threshold, the travel section determination unit 73 determines that the vehicle is traveling in a horizontal section. On the other hand, when the longitudinal gradient on the road ahead of the vehicle is larger than a predetermined value, the traveling section determination unit 73 determines that the traveling section is traveling on a slope section that is not a horizontal section but an uphill or downhill.

The stop determination unit 74 determines whether or not the vehicle 110 is in a stopped state. Specifically, the stop determination unit 74 is based on the vehicle speed information acquired by the travel information acquisition unit 72, when the travel speed is substantially zero, or the travel speed is less than a stop threshold (for example, 5 km / h). If it is, it is determined that the vehicle 110 is not in a running state but in a stopped state. On the other hand, when the traveling speed exceeds the above-described stop threshold, the stop determination unit 74 determines that the vehicle 110 is not in a stopped state but in a traveling state.

In addition, the stop determination unit 74 can determine whether or not the vehicle 110 is in a traveling state based on acceleration data from the acceleration sensor 14. Specifically, when the vertical vibration associated with traveling is not detected from the transition of the acceleration data and there is substantially no change in the longitudinal acceleration, the stop determination unit 74 indicates that the vehicle 110 is in the stop state. It is determined that

The face orientation correction unit 75 estimates the line-of-sight direction of the driver DR by combining the electrooculogram data from the electrooculogram sensor 12 and the angular velocity data from the gyro sensor 13. Specifically, the face orientation correcting unit 75 calculates the angle of the head HD in each rotation direction by time-integrating the angular velocity around each axis. As described above, the face orientation correcting unit 75 estimates the face orientation of the driver DR caused by the swing of the driver DR.

In addition, the face orientation correcting unit 75 estimates the eye movement of the driver DR based on the change mode of electrooculogram data using a so-called EOG (Electro-Oculography) method. Specifically, the face orientation correction unit 75 calculates the up / down / left / right directions in which the eyeballs are directed and the amount of change from the front of the eyeballs, based on the front direction of the face of the driver DR. The face direction correction unit 75 specifies the line-of-sight direction of the driver DR with reference to the vehicle 110 by correcting the estimated face direction by the amount of change in the direction of the eyeball.

The counting processing unit 76 counts the number of times the driver DR visually recognizes the visual recognition object, the plotting process that plots the line-of-sight direction of the driver DR, the specific process that specifies the position of the visual recognition object mounted on the vehicle 110. At least a counting process to be performed. The plot process, the specific process, and the counting process are performed when the stop determination unit 74 determines that the vehicle 110 is not in the stop state, and the stop determination unit 74 determines that the vehicle 110 is in the stop state. In case of failure, it can be interrupted.

In the plotting process, the driver's DR visual point VP (FIG. 5 (a) and FIG. 5 (b) is obtained by sampling the line-of-sight direction of the driver DR specified by the face direction correcting unit 75 at predetermined time intervals. ) See). The visual point VP of the driver DR is a geometrical relationship between a virtual curved surface defined to include the interior side surface of the windshield of the vehicle 110 and a virtual line defined to extend in the line-of-sight direction from the eyeball of the driver DR. This is an intersection. By the above plot processing, a large number of visual recognition points VP are plotted on the virtual curved surface. In addition, outside the windshield, the virtual curved surface is defined as a shape that smoothly extends the windshield.

In the specifying process, the positions of a plurality of viewing objects installed in the vehicle 110 are specified based on the distribution of the many viewing points VP plotted in the plotting process. Each visual target object includes left and right side mirrors 111 and 112, a rearview mirror 113, a display 51, and a combination meter that are substantially fixed to the vehicle 110. In the specifying process, it is specified that the visual recognition object exists in a range where the visual recognition points VP are densely gathered (hereinafter, “aggregation part”).

More specifically, as shown in FIGS. 5 (a) and 5 (b), the central portion 61 of the windshield center where the most visible points VP gather is a vanishing point in the foreground of the driver DR. Then, each of the collective portions 62 to 66 in which the number of the visually recognized points VP plotted is the second or later becomes the position of each visually recognized object.

For example, the gathering portion 62 on the left edge of the windshield corresponds to the position of the left side mirror 111 (see FIG. 2). The gathering portion 63 on the right edge of the windshield corresponds to the position of the right side mirror 112 (see FIG. 2). The collecting unit 64 positioned above the collecting unit 61 corresponds to the position of the rearview mirror 113 (see FIG. 2). A set portion 65 located below each set portion 61, 64 corresponds to the position of the display 51 (see FIG. 1). And the gathering part 66 located in the lower front of the driver DR corresponds to the position of the combination meter. 5A, 5B, 6A, and 6B, the outermost edge of the horizontally long rectangular frame corresponds to the outer edge of the windshield.

Here, the position of the vanishing point in the foreground of the driver DR shown in FIG. 2 changes in the left-right direction or the up-down direction with respect to the vehicle 110 in a scene where the vehicle 110 is traveling in a curve section or a slope section. As a result, as shown in FIGS. 6 (a) and 6 (b), the position of the gathering unit 61 is also different in each viewing object during the period in which the vehicle 110 (see FIG. 2) is traveling in a curve section or a slope section. It changes relatively with respect to the other collecting portions 62 to 66 corresponding to the positions of the objects.

Therefore, as shown in FIGS. 4 and 1, the counting processing unit 76 classifies the position information of the plotted visual recognition points VP based on the road environment in which the vehicle 110 is traveling, and specifies the position of each visual recognition object. Is stored in the first storage area 46a or the second storage area 46b. Specifically, the position information of the visual point VP plotted in the period in which it is determined that the traveling section determination unit 73 is traveling in a horizontal and linear section is accumulated in the first storage area 46a.

On the other hand, the position information of the visual recognition point VP plotted in the period determined to be traveling in the curve section or the slope section by the travel section determination unit 73 is excluded from the accumulation target in the first storage area 46a. . Then, the position information of the visual recognition point VP plotted in the curve section is associated with the curvature information indicating the magnitude of the curvature of the curve section and the curve direction of the curve, and is stored in the second storage area 46b. Similarly, the position information of the visual recognition point VP plotted in the slope section is associated with slope information indicating the magnitude of the vertical slope of the slope section and the vertical direction of the slope, and is stored in the second storage area 46b.

In the counting process, the number of visual recognitions of each visual target by the driver DR is counted. In the counting process, the position of the collection target 61 corresponding to the vanishing point is used as a reference (zero point position), based on the transition of the relative change in the line-of-sight direction specified by the face orientation correction unit 75. The number of viewing times is counted (see FIGS. 7 and 8).

Specifically, in the counting process, a threshold is set for counting that each visual target object has been viewed. The threshold values in the vertical direction and the horizontal direction can be individually set for one visual object. The count processing unit 76 counts the number of times that the line-of-sight direction specified by the face orientation correcting unit 75 exceeds each threshold as the number of times of viewing each visual target object.

For example, as shown in FIG. 7, in a straight section, a threshold th (r) (in the right direction from a zero point position based on the position of the gathering unit 61 (see FIGS. 5A and 5B). For example, it is assumed that the line-of-sight direction has moved beyond 15 °. In this case, the counting processing unit 76 counts that the right side mirror 112 (see FIG. 2) is viewed once. Similarly, the line-of-sight direction exceeds the threshold th (l) (for example, 30 °) to the left from the zero point position based on the position of the gathering unit 61 (see FIGS. 5A and 5B). When it moves, the counting process part 76 counts that the right side mirror 112 (refer FIG. 2) was visually recognized once. Specific values of the above threshold values th (r) and th (l) can be set based on the distribution of the position information of the visual recognition point VP accumulated in the first storage area 46a. In FIG. 7, the swing to the right is on the minus side, and the swing to the left is on the plus side.

The behavior determination unit 77 determines whether the visual recognition behavior of the driver DR who visually recognizes each visual target object has changed. As described above, in a scene where the vehicle 110 travels in a curve section or a slope section, the position of the vanishing point changes in the foreground visually recognized by the driver DR (see FIGS. 6A and 6B). ). The behavior determination unit 77 detects the occurrence of a change in the viewing behavior of the driver DR caused by the position change of the vanishing point.

The behavior determination unit 77 can determine whether or not the visual behavior of the driver DR has changed based on the change in the detection result of the face direction detection unit 11. Specifically, the behavior determination unit 77 determines whether the viewing behavior of the driver DR has changed based on the behavior change of the behavior of visually recognizing the side mirror far from the driver DR, that is, the left side mirror 111. Determine whether or not. The behavior determination unit 77 determines whether or not the visual behavior of the driver DR has changed based on the frequency of viewing the left side mirror 111 (number of times per predetermined time) and the change in the pattern of viewing the left side mirror 111. Can be determined.

In addition, the behavior determination unit 77 can detect a change in the visual behavior of the driver DR based on the determination result of the travel section determination unit 73. Specifically, the behavior determination unit 77 determines that the visual behavior of the driver DR has changed when the travel segment determination unit 73 determines that the vehicle travels in a curve section or a slope section. The determination result by the above action determination unit 77 is sequentially provided to the count processing unit 76.

Based on the determination result of the behavior determination unit 77 described above, the counting processing unit 76 changes the content of the counting process so as to correspond to the position change of the vanishing point. Specifically, the counting processing unit 76, when the behavior determining unit 77 determines that the driver DR's visual behavior has changed, the aggregation unit 61 (see FIGS. 6A and 6B). Change the reference zero point position according to the slide position. As a result, as shown in FIGS. 5A to 8, each threshold value used for counting the number of times of visual recognition is changed.

For example, as shown in FIGS. 6A and 6B, when the vehicle 110 is traveling in a left curve section, the collecting unit 61 shifts to the left side with respect to the vehicle 110, and the left side mirror It is close to the gathering portion 62 corresponding to 111 (see FIG. 2). As a result, the zero point position serving as the reference for the swing angle is also shifted to the left. Therefore, as shown in FIG. 8, the absolute value of the threshold th (r) for counting the number of times the right side mirror 112 is viewed is changed to a larger value (for example, 25 °) than when traveling in a straight section. The On the other hand, the absolute value of the threshold th (l) for counting the number of times the left side mirror 111 is visually recognized is changed to a smaller value (for example, 20 °) than when traveling in a straight section. Specific values of the threshold values th (r) and th (l) are distributions of accumulated data corresponding to the road environment currently being traveled among the positional information of the visual recognition points VP accumulated in the second storage area 46b. Is set based on Therefore, the threshold values th (r) and th (l) change stepwise according to the curvature of the curve and the gradient.

4 and 1, the warning control unit 78 acquires, from the counting processing unit 76, the number of times of viewing of each visual target object per predetermined time counted in the counting process. The warning control unit 78 determines whether or not the driver DR's visual behavior is in an abnormal state based on the obtained number of times of visual recognition of each visual target object. For example, when the frequency of visually recognizing the left and right side mirrors 111 and 112 falls below a predetermined frequency threshold, the warning control unit 78 determines that the state is abnormal. When it is determined that the warning control unit 78 is in an abnormal state, the warning control unit 78 outputs a command signal instructing execution of warning to the HMI control unit 50. As a result, a warning for the driver DR is performed by the display 51, the speaker 52, and the like.

Details of each process for estimating the face direction by the in-vehicle controller 45 described above will be described with reference to FIGS. 1 and 2 based on the flowchart of FIG. Each process shown in FIG. 9 is based on the fact that, for example, both the wearable device 10 and the in-vehicle device 40 are powered on, and transmission of the detection result from the wearable device 10 to the in-vehicle device 40 is started. Started by. Each process illustrated in FIG. 9 is repeatedly performed until at least one of the wearable device 10 and the in-vehicle device 40 is turned off.

In S101, identification of the face direction and line-of-sight direction of the driver DR and plotting of the visual recognition point VP are started, and the process proceeds to S102. In S102, each threshold value for counting the number of times each visual object is viewed is set, the number of times the visual object is viewed is started, and the process proceeds to S103. In S102, a preset initial value is set as each threshold value.

In S103, the traveling environment of the vehicle 110 is determined based on the distribution of the visual recognition points VP that started plotting in S101, the road shape information, the latest vehicle speed information, and the like, and the process proceeds to S104. If it is determined in S103 that the vehicle 110 is in a stopped state, the process returns from S104 to S103. Thus, when it is determined that the vehicle 110 is in the stopped state, the plot processing started in S101 may be temporarily interrupted until the vehicle 110 is switched to the traveling state. On the other hand, when it is determined in S103 that the vehicle 110 is in the traveling state, the process proceeds from S104 to S105.

In S105, it is determined whether the visual behavior has changed. If the determination result of the traveling environment is substantially the same as the conventional one in S103, it is determined in S105 that the visual behavior has not changed, and the process proceeds to S107. On the other hand, if a change in the determination result of the driving environment has occurred in S103, it is determined in S105 that the visual behavior has changed, and the process proceeds to S106.

In S106, the threshold value corresponding to the traveling environment determined in the immediately preceding S103 is changed, and the process proceeds to S107. In S107, a counting process is performed to count the number of times each visual target is viewed using the currently set threshold values, and the process proceeds to S108. In S108, the identification of the line-of-sight direction of the driver DR and the plotting of the visual recognition point VP are continued, and the data of the plotted visual recognition point VP is accumulated in the respective storage areas 46a and 46b corresponding to the current driving environment, and in S109. move on.

In S109, it is determined whether or not the driver DR is in an abnormal state based on the result of the counting process performed in the immediately preceding S107. In S109, when it is determined that the driver DR is performing normal visual recognition behavior and is not in an abnormal state, the process returns to S103. On the other hand, when it determines with driver | operator DR being in an abnormal state in S109, it progresses to S110.

In S110, a command signal for instructing the HMI control unit 50 to execute a warning is output, and the series of processes is temporarily terminated. Based on the command signal output in S110, a warning by the display 51 and the speaker 52 is performed. When the warning in S110 ends, the vehicle-mounted control unit 45 starts S101 again.

In the first embodiment described so far, when the vehicle 110 travels in a curve section, a slope section, or the like, and a change occurs in the position of the vanishing point in the foreground, the driver DR It is determined that the viewing behavior has changed. As a result, the counting process for counting the number of times that the driver DR visually recognizes the visual recognition object is changed to the content corresponding to the change in the visual recognition behavior of the driver DR in accordance with the position change of the vanishing point. Therefore, even if the driver DR's visual behavior changes due to the position change of the vanishing point, the number of times the driver visually recognizes the visual target object can be accurately counted.

Also, as described above, when the position of the vanishing point changes, the detection result of the driver DR's face orientation surely changes. As a result, a clear change may occur in the visual recognition frequency of the visual object and the visual pattern of the visual object based on the detection result of the face orientation. Therefore, if it is determined whether or not there is a change in the viewing behavior of the driver DR based on the changes in the viewing frequency and the viewing pattern, the occurrence of the change in the viewing behavior due to the position change of the vanishing point can be detected without omission. Become.

In addition, in the first embodiment, whether or not there is a change in the entire visual behavior due to the change in the position of the vanishing point is determined based on the behavior change of the behavior of visually recognizing the left side mirror 111 far from the driver DR. . For example, when visually recognizing the right side mirror 112, it is possible to move only by eye movement mainly by eye movement, so that the change in face orientation due to swinging becomes small. On the other hand, when visually recognizing the left side mirror 111, the amount of movement is insufficient only by the movement of the line of sight by eye movement, and thus a clear change in the face orientation due to swinging occurs.

For the above reasons, as shown in FIG. 10, erroneous detection and non-detection when detecting the visual behavior of the left side mirror 111 are false detection and non-detection when detecting the visual behavior of the right side mirror 112. Rather than reduced. Thus, the action of visually recognizing the left side mirror 111 is easily detected with high accuracy. Therefore, based on the behavior change of the behavior of visually recognizing the left side mirror 111, the behavior determination unit 77 can accurately determine the occurrence of the change in the behavior of the driver DR as a whole.

Furthermore, the behavior determination unit 77 of the first embodiment can also determine the change in the visual behavior of the driver DR by using the determination result of the travel section determination unit 73. As described above, the vanishing point position change mainly occurs in the curve section and the slope section. Therefore, if the road shape information is used to specify the traveling of the curve section or the slope section, the behavior determination unit 77 accurately determines the occurrence of the change in the visual behavior due to the position change of the vanishing point without delay. be able to.

In addition, in the first embodiment, the position information of the visual recognition point VP on which the face orientation detection result is plotted is accumulated in the first storage area 46a and used for specifying the position of the visual recognition object. Thus, if the position information of the visual recognition point VP is continuously accumulated, the position of the visual recognition object can be specified with higher accuracy. As a result, it is possible to improve both the accuracy of determining the behavior change of the visual recognition behavior and the accuracy of counting the number of times the visual target is viewed.

In the first embodiment, the position information of the visual recognition point VP acquired during the period determined by the traveling section determination unit 73 as not being a straight section is not accumulated in the first storage area 46a. According to such processing, only the position information of the visual recognition point VP in the period in which the position change of the vanishing point has not substantially occurred is accumulated in the first storage area 46a. Therefore, based on the position information accumulated in the first storage area 46a, the position of the visual recognition object can be specified more accurately in the straight section.

Further, in the first embodiment, when it is determined that the visual recognition behavior of the driver DR has changed, each threshold used for counting the number of visual recognitions is changed. According to the above, the counting processing unit 76 can select an appropriate threshold value in accordance with the change in the visual recognition behavior of the driver DR, and can continue to count the number of visual recognitions of the visual target object with high accuracy.

In addition, in the first embodiment, the counting process is interrupted when the vehicle 110 is substantially in a stopped state. As described above, if the vehicle 110 is not in the traveling state, the warning to the driver DR related to the visual recognition behavior is troublesome for the driver DR and is substantially unnecessary. Therefore, if it is a stop state, it is desirable that the counting process is interrupted when the warning is interrupted.

In the first embodiment, the acceleration data detected by the acceleration sensor 14 of the face direction detection unit 11 is used for the stop determination of the stop determination unit 74. As described above, if the existing configuration for detecting the behavior of the head HD is used, it is possible to accurately determine the stop state while avoiding the complexity of the face direction estimation system 100.

Further, in the first embodiment, the face direction of the driver DR is estimated based on the measurement data of the gyro sensor 13 attached to the head HD of the driver DR. As described above, the gyro sensor 13 that moves integrally with the head HD can faithfully detect the movement of the head HD. As a result, the estimated accuracy of the face direction of the driver DR is easily ensured.

Further, as in the first embodiment, if the face direction detection unit 11 is mounted on the glasses 10a, the face direction detection unit 11 is not easily displaced from the head HD, and the movement of the head HD is surely performed. It can be detected. Therefore, the form in which the face direction detection unit 11 is mounted on the glasses 10a can contribute to an improvement in the accuracy of the process of counting the number of times the visual target is viewed.

In the first embodiment, the first storage area 46a corresponds to the “storage area”, the driver information acquisition unit 71 corresponds to the “face direction information acquisition unit”, and the in-vehicle device 40 corresponds to the “face direction estimation device”. The vehicle 110 corresponds to a “moving object”.

(Second embodiment)
A face orientation estimation system 200 according to the second embodiment of the present disclosure shown in FIG. 11 is a modification of the first embodiment. In the face orientation estimation system 200 of the second embodiment, each process for estimating the face orientation of the driver DR (see FIG. 2) is mainly performed by the wearable device 210. The in-vehicle device 240 transmits the acquired vehicle speed information and road shape information from the communication unit 47 to the wearable device 210 by wireless communication.

Wearable device 210 is a glasses-type motion sensor device similar to that of the first embodiment (see FIG. 3). The detection circuit 220 of the wearable device 210 includes a wearable control unit 215 and a vibration notification unit 218 in addition to the face orientation detection unit 11, the communication unit 17, the operation unit 18, and the battery 19 that are substantially the same as those in the first embodiment. Yes.

The wearable control unit 215 includes a processor 215a, a RAM, a microcomputer mainly including an input / output interface, a memory 216, and the like. In the memory 216, a first storage area 216a and a second storage area 216b corresponding to the storage areas 46a and 46b (see FIG. 1) of the first embodiment are secured.

The wearable control unit 215 executes the face orientation estimation program stored in the memory 216 by the processor 215a, similarly to the in-vehicle control unit 45 (see FIG. 1) of the first embodiment. As a result, a plurality of functional blocks (71 to 78, see FIG. 4) substantially the same as those of the first embodiment are constructed in the wearable control unit 215.

The driver information acquisition unit 71 (see FIG. 4) constructed in the wearable control unit 215 can acquire the detection result directly from the face orientation detection unit 11. On the other hand, the travel information acquisition unit 72 (see FIG. 4) constructed in the wearable control unit 215 acquires vehicle speed information and road shape information received by the communication unit 17.

The warning control unit 78 (see FIG. 4) constructed in the wearable control unit 215 outputs a command signal for commanding the warning to the vibration notification unit 218 when it is determined that the state is abnormal. As a result, a warning for the driver DR (see FIG. 2) is performed by the vibration of the vibration notification unit 218.

As in the second embodiment described so far, even when each process for estimating the face orientation is performed by the wearable device 210, the counting process can be changed to a content corresponding to a change in the driver's DR visual behavior. . Therefore, similarly to the first embodiment, even if the driver DR's visual behavior changes due to the position change of the first vanishing point, the number of times the driver visually recognized the visual target can be accurately counted. Become. In the second embodiment, the wearable device 210 corresponds to a “face orientation estimating device”.

(Other embodiments)
Although a plurality of embodiments according to the present disclosure have been described above, the present disclosure is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure. can do.

In the above-described embodiment, each process related to face orientation estimation is performed by either an in-vehicle device or a wearable device. However, each process related to face orientation estimation may be performed in a distributed manner by each processor of the in-vehicle device and the wearable device. In such a form, the face orientation estimation system corresponds to a “face orientation estimation device”. In addition, each function provided by the processor in the above embodiment can be provided by hardware and software different from the above-described configuration, or a combination thereof. Further, the in-vehicle device may be, for example, a mobile terminal such as a smartphone and a tablet brought into the vehicle by the driver.

The wearable device 210 (see FIG. 11) of the second embodiment acquires vehicle speed information and road shape information by wireless communication. However, the wearable device may perform each process related to face orientation estimation without using the vehicle speed information and the road shape information.

If the in-vehicle device of the first embodiment can acquire the planned travel route of the vehicle from the navigation device, for example, even in a scene where a right / left turn is performed, the travel section determination unit is a straight section as in the determination of the curve section. It is possible to determine that it is not. In such a form, it is possible to accurately warn of an oversight of the side mirror in a scene where a right or left turn is performed.

The face orientation detection unit of the above embodiment has a plurality of sensors 12-14. However, the electrooculogram sensor can be omitted from the face orientation detection unit as long as it does not take into account changes in the line-of-sight direction due to eye movements. Further, if the stop determination is performed only by the vehicle speed information, the acceleration sensor can be omitted from the face direction detection unit. Furthermore, other sensors different from the above, for example, a magnetic sensor and a temperature sensor, may be included in the face orientation detection unit.

In the above embodiment, the glasses-type configuration is exemplified as the wearable device. However, the wearable device is not limited to the glasses-type configuration as long as it can be attached to the head above the neck. For example, a badge-type wearable device attached to a hat or the like, or an ear-mounted wearable device that is hooked behind the ear can be employed in the face orientation estimation system. Furthermore, the configuration for detecting the driver's face orientation is not limited to the wearable device. For example, a camera for photographing the driver's face may be provided in the face direction estimation system as a face direction detection unit. In such a form, for example, the in-vehicle control unit performs a process of detecting the face orientation by image analysis from the image captured by the camera.

However, if a wearable device is used, it is not necessary to secure a place to install the camera. In addition, it is possible to avoid a situation in which it is difficult to detect the face orientation using the image of the camera due to normal glasses or sunglasses worn by the driver or strong external light.

The behavior determination unit of the above embodiment determines the change in the driver's visual behavior by detecting the change in the visual frequency and the visual pattern of the left side mirror. However, the behavior determination unit may determine that the driver's visual behavior has changed, for example, when the ratio of the number of times of visual recognition of the side mirrors arranged side by side changes within a predetermined period. Further, when the steering is disposed on the left side of the vehicle, the behavior determination unit detects a change in the viewing frequency and the viewing pattern of the right side mirror as a side mirror far from the driver, thereby driving It is desirable to determine a change in a person's visual behavior.

In the above embodiment, the position of the display, the meter, etc. is specified as the visual recognition object, but the visual recognition object whose position is specified may be only the left and right side mirrors and the rearview mirror. Moreover, the position of the configuration of another vehicle may be specified as the visual recognition object. Furthermore, instead of the left and right side mirrors, the position of the screen of the electronic mirror may be specified as the visual recognition object.

In the above embodiment, when the vehicle is almost stopped, the warning based on the abnormality determination and each processing by the counting processing unit are interrupted. However, for example, the process of plotting the driver's line-of-sight direction may be continued even when the vehicle is stopped. Furthermore, stop determination may be performed based on the transition of the positioning signal received by the locator.

In the above embodiment, the position information of the visual recognition point VP acquired by the plot processing is classified and accumulated in the first storage area and the second storage area. However, the classification and accumulation of position information may not be performed. Further, the storage medium in which the first storage area and the second storage area are provided is not limited to the flash memory, and may be another storage medium such as a hard disk drive.

In the above embodiment, the position of the collection unit 61 (see FIGS. 5A, 5B, 6A, and 6B) corresponding to the vanishing point is defined as a zero point, and the vertical direction In addition, the face direction and the line-of-sight direction in the left-right direction were digitized (see FIGS. 7 and 8). Therefore, each threshold when counting the number of visual recognitions is also set based on the position of the collective part corresponding to the vanishing point. However, for example, as long as the face direction and the line-of-sight direction are digitized based on a specific position (for example, the center position) of the windshield, each threshold value may be set based on the specific position of the windshield.

And the process to which the face direction estimation method of the present disclosure is applied can be performed even on a mobile object different from the above embodiment. Furthermore, an operator who monitors an automatic driving function in a vehicle during automatic driving can also correspond to a driver.

Here, the flowchart described in this application or the process of the flowchart is configured by a plurality of sections (or referred to as steps), and each section is expressed as S101, for example. Further, each section can be divided into a plurality of subsections, while a plurality of sections can be combined into one section. Further, each section configured in this manner can be referred to as a device, module, or means.

Although the present disclosure has been described based on the embodiments, it is understood that the present disclosure is not limited to the embodiments and structures. The present disclosure includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more or less, are within the scope and spirit of the present disclosure.

Claims (13)

1. A face orientation information acquisition unit (71) for acquiring a detection result of detecting the face orientation of the driver (DR) driving the mobile body (110);
   Based on the detection result acquired by the face orientation information acquisition unit, a count processing unit (76) that counts the number of times the driver visually recognizes the visual target object installed on the moving body;
   An action determination unit (77) for determining whether or not the driver's visual behavior that visually recognizes the visual target object has changed due to a change in position of the vanishing point in the foreground visually recognized by the driver; Prepared,
   The counting processing unit is a face direction estimating device that changes the content of the counting process in accordance with a change in position of the vanishing point when the behavior determining unit determines that the driver's visual behavior has changed.
2. The face orientation estimation apparatus according to claim 1, wherein the behavior determination unit determines whether or not the driver's visual behavior has changed based on a change in the detection result acquired by the face orientation information acquisition unit.
3. The behavior determination unit is configured to change the driver's driver when at least one of a frequency of visually recognizing the visual object, a pattern of visually recognizing the visual object, and a ratio of the number of visual recognitions of the visual object arranged side by side. The face direction estimation apparatus according to claim 2, wherein it is determined whether or not the visual behavior of the user has changed.
4. The visual objects include left and right side mirrors (111, 112) installed in a vehicle as the moving body,
   The said action determination part determines whether the visual recognition action by the said driver changed based on the mode change of the action which visually recognizes the said side mirror far from the said driver among right and left. The face direction estimation apparatus according to claim 1.
5. A travel information acquisition unit (72) for acquiring shape information of a road on which the mobile body is scheduled to travel;
   5. The behavior determination unit according to claim 1, wherein the driver's visual behavior is changed when the moving body is traveling in a curve section or a slope section based on the shape information. The face direction estimation apparatus according to one item.
6. A storage area (46a, 216a) for accumulating the detection results acquired by the face orientation information acquisition unit;
   The face orientation estimation apparatus according to any one of claims 1 to 5, wherein the counting processing unit specifies the position of the visual recognition object based on the detection result accumulated in the storage area.
7. A travel section determination unit (73) for determining whether or not the mobile body is traveling in a straight section;
   The face orientation estimation apparatus according to claim 6, wherein the detection result of the face orientation acquired in a period determined not to be a straight section by the travel section determination unit is excluded from a target accumulated in the storage area.
8. The counting unit according to any one of claims 1 to 7, wherein when the behavior determining unit determines that the driver's visual behavior has changed, the counting processing unit changes a threshold for counting the number of times the visual target is viewed. The face orientation estimation device according to any one of the above.
9. A stop determination unit (74) for determining whether or not the moving body is in a stopped state;
   The face orientation estimation device according to any one of claims 1 to 8, wherein the counting processing unit interrupts the counting processing when the stop determining unit determines that the moving body is in a stopped state.
10. The face orientation information acquisition unit acquires measurement data of an acceleration sensor (14) that is mounted on the driver's head (HD) and measures acceleration acting on the head,
    The face orientation estimation apparatus according to claim 9, wherein the stop determination unit determines whether or not the moving body is in a stop state based on measurement data detected by the acceleration sensor.
11. The face orientation information acquisition unit acquires measurement data of a gyro sensor (13) that is mounted on the driver's head and measures an angular velocity generated in the head. The face orientation estimation apparatus described.
12. The face orientation information acquisition unit according to any one of claims 1 to 11, wherein the face orientation information acquisition unit acquires the detection result from a face orientation detection unit (11) mounted on glasses (10a) worn by the driver. Face orientation estimation device.
13. A face orientation estimation method for estimating a face orientation of a driver (DR) driving a mobile body (110),
    At least one processor (45a, 215a)
    Based on the detection result of the face direction detection unit (11) that detects the face direction of the driver, the number of times the driver visually recognizes the visual target object installed on the moving body is counted (S107),
    It is determined whether or not the driver's visual behavior that visually recognizes the visual target object has changed due to a change in the position of the vanishing point in the foreground visually recognized by the driver (S105),
    When it is determined that the driver's visual behavior has changed, the content of the counting process is changed in accordance with the position change of the vanishing point (S106).

Images