JP2018148423A - Direction detection device and operation control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a direction detection device and the like capable of appropriately controlling the exposure in an imaging unit.SOLUTION: A line-of-sight operation device 100 functions as a direction detection device and detects the line-of-sight direction of an operator U who is on a vehicle by using a face image PF of the operator U. The line-of-sight operation device 100 includes an exposure evaluation unit 22, an exposure setting unit 23, an error detection unit 36, and a map generation unit 38. The exposure evaluation unit 22 acquires information on a light measurement amount indicating the brightness around the operator U. The exposure setting unit 23 sets an exposure value based on the light measurement amount and controls the exposure in an imaging unit 10. The error detection unit 36 detects a state of an error in which the line-of-sight direction cannot be detected from the face image PF. The map generation unit 38 records occurrence of an error and learns a permission range of a settable exposure value. The exposure setting unit 23 sets the exposure value within the permission range set in the map generation unit 38 with respect to the light measurement amount acquired by the exposure evaluation unit 22.SELECTED DRAWING: Figure 1

Description

  The disclosure according to this specification relates to a technique for detecting a line of sight or a face direction using a face image.

  Conventionally, for example, Patent Document 1 discloses an imaging apparatus for a vehicle including a camera that captures a face of a passenger boarding the vehicle. In such a vehicular imaging device, the exposure setting in the camera is set according to the brightness of the surroundings of the occupant so that the position of the pupil and the position of the cornea reflected light necessary for detecting the line of sight are detected from the face image. Automatically controlled.

JP-A-2016-49260

  Now, according to the running of the vehicle, for example, when the vehicle enters the tunnel, the brightness around the passenger changes suddenly. It is very difficult to comprehensively assume such scenes with sudden changes in brightness. If the brightness changes abruptly due to factors that are difficult to imagine in advance, the exposure of the imaging unit such as a camera may be out of the appropriate range. As a result, there is a possibility that an error state in which the direction of the line of sight or the like cannot be detected from the face image continues to occur.

  The present disclosure has been made in view of the above problems, and an object thereof is to provide a direction detection device and an operation control device capable of appropriately controlling exposure in an imaging unit.

  In order to achieve the above object, one disclosed aspect is that a passenger uses a face image (PF) imaged by an imaging unit (10) that captures the face of a passenger (U) boarding a moving body. A direction detection device that detects at least one direction of the line of sight and the face, and a photometric quantity acquisition unit (22) that acquires information of the photometric quantity indicating the brightness of the occupant's surroundings, and imaging based on the photometric quantity An exposure setting unit (23) for setting an exposure value used in the image pickup unit and controlling the exposure in the imaging unit, an error detection unit (36) for detecting an error state in which the direction cannot be detected from the face image, and an error occurrence And a setting learning unit (38) that learns an allowable range of exposure values that can be set for the photometric amount based on the recorded error occurrence rate, and the exposure. The setting unit measures the measurement acquired by the photometry acquisition unit. The amount to, there is a direction detecting device for setting an exposure value within the set allowed range by setting learning unit.

  Moreover, one disclosed aspect is an operation control device connected to an operation target device (110) that can be operated by a passenger (U) boarding a moving body, and an imaging unit that captures the face of the passenger Based on the direction detected by the direction detector and the direction detector (35) that detects at least one direction of the line of sight of the passenger and the face using the face image (PF) captured in (10). An operation control unit (40) for controlling the operation of the operation target device, a light measurement amount acquisition unit (22) for acquiring information on a light measurement amount indicating the brightness around the occupant, and an imaging unit based on the light measurement amount An exposure setting unit (23) for setting an exposure value used for controlling the exposure in the imaging unit, an error detection unit (36) for detecting an error state in which the direction cannot be detected from the face image, Record and record in relation to the photometric quantity and exposure value. A setting learning unit (38) that learns a permissible range of exposure values that can be set for photometric light based on the error occurrence rate, and the exposure setting unit performs photometry obtained by the photometric light acquisition unit. The operation control device sets the exposure value within the permitted range set by the setting learning unit with respect to the amount.

  According to these aspects, the occurrence of an error in which the line of sight or the direction of the face cannot be detected from the face image is recorded. Based on the error occurrence rate, a range of exposure values that can be set for the photometric amount is learned. As described above, if the occurrence of an error in the output stage in the direction detection is fed back to the setting of the exposure value, the occurrence of an error due to a factor that cannot be assumed in advance can be gradually reduced through the use of the direction detection device. Based on the above, the direction detection device and the operation control device can control the exposure in the imaging unit within an appropriate range.

  Note that the reference numbers in the parentheses merely show an example of a correspondence relationship with a specific configuration in an embodiment described later, and do not limit the technical scope at all.

It is a block diagram which shows the whole image of the structure mounted in-vehicle including the gaze control apparatus by 1st embodiment. It is a flowchart which shows the detail of the gaze measurement process by a gaze measurement part. It is a figure which compares and shows the influence on the gaze detection resulting from exposure setting. It is a flowchart which shows the detail of the map production | generation process by a map production | generation part. FIG. 5A is a diagram for explaining a relationship between a photometric amount and an appropriate exposure value, and FIG. 5A is a diagram illustrating a change in an error rate according to the photometric amount for each exposure setting, and FIG. 5B is an example of a setting availability map. FIG. 5C is a diagram schematically showing the error limit characteristics based on the exposure setting availability map. It is a time chart which shows transition of exposure control at the time of passage of a tunnel. It is a flowchart which shows the detail of the exposure control process by an exposure control part with FIG. It is a flowchart which shows the detail of the exposure control process by an exposure control part with FIG. It is a flowchart which shows the detail of the sudden change prediction process by the sudden change prediction part. It is a flowchart which shows the detail of the sudden change detection process by an exposure setting part. It is a time chart which shows transition of exposure control by a second embodiment. 10 is a time chart showing a transition of exposure control according to Modification 1.

  Hereinafter, a plurality of embodiments of the present disclosure will be described based on the drawings. In addition, the overlapping description may be abbreviate | omitted by attaching | subjecting the same code | symbol to the corresponding component in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other part of the configuration. Moreover, not only the combination of the configurations explicitly described in the description of each embodiment, but also the configuration of a plurality of embodiments can be partially combined even if they are not explicitly described, as long as there is no problem in the combination. And the combination where the structure described in several embodiment and the modification is not specified shall also be disclosed by the following description.

(First embodiment)
A line-of-sight operating device 100 according to the first embodiment of the present disclosure shown in FIG. 1 is mounted on a vehicle together with, for example, a navigation device 110. The line-of-sight operating device 100 is electrically connected to the navigation device 110 directly or indirectly. The line-of-sight operating device 100 detects the direction of the line of sight of the operator U using the face image PF. The line-of-sight operating device 100 detects the direction of the line of sight of the operator U, and causes the navigation device 110 to execute an action corresponding to the detected direction of the line of sight. The operator U is a passenger boarding the vehicle, and is a driver or the like as an example. With the function of the line-of-sight operating device 100, the operator U can operate the navigation device 110 according to the direction of the line of sight without using a hand during driving.

  The line-of-sight operating device 100 includes an imaging unit 10, an exposure control unit 20, a line-of-sight measurement unit 30, an operation control unit 40, and an external communication unit 50.

  The imaging unit 10 is a camera device that captures the face of the operator U and its surroundings. The imaging unit 10 captures the face image PF of the operator U and sequentially outputs it to the exposure control unit 20. The imaging unit 10 includes a light source 11, an optical system 12, an image sensor 13, and a controller 14.

  The light source 11 is a near infrared light source that radiates near infrared light toward the operator U, for example. The optical system 12 is optically coupled to the image sensor 13. The optical system 12 includes a lens, a band-pass filter (hereinafter “band-pass filter: BPF”), and the like. The image sensor 13 captures a face image PF obtained by capturing the periphery of the face from the light introduced by the optical system 12. The controller 14 is a control unit that is electrically connected to the light source 11 and the image sensor 13. The controller 14 adjusts the light emission time of the light source 11 based on the light emission control value acquired from the exposure control unit 20. The controller 14 adjusts the exposure time, aperture value, ISO sensitivity, and the like in the image sensor 13 based on the exposure value acquired from the exposure control unit 20.

  The exposure control unit 20 repeatedly performs an exposure control process (see FIGS. 7 and 8) for controlling the exposure of the imaging unit 10 so that the image of the operator U is appropriately captured by the imaging unit 10. The exposure control unit 20 includes an image acquisition unit 21, an exposure evaluation unit 22, and an exposure setting unit 23.

  The image acquisition unit 21 sequentially acquires image data of the face image PF captured by the image sensor 13. The image acquisition unit 21 outputs the acquired image data to the line-of-sight measurement unit 30.

  The exposure evaluation unit 22 acquires information on the photometric quantity indicating the brightness around the operator U. Specifically, the exposure evaluation unit 22 calculates an average luminance for the face image PF acquired by the image acquisition unit 21 as information corresponding to the photometric amount. In calculating the average luminance, the exposure evaluation unit 22 may calculate a weighted average in which the importance of a specific area such as the center of the face image PF and the face range is increased.

  The exposure setting unit 23 sets an exposure value used for controlling the image sensor 13 and a light emission control value used for controlling the light source 11 based on the photometric amount. Specifically, the exposure setting unit 23 adjusts the exposure value and the light emission control value so that the average luminance calculated by the exposure evaluation unit 22 approaches a preset target luminance. The exposure setting unit 23 sequentially transmits the set exposure value and light emission control value to the controller 14. The exposure value is set so that the exposure time increases as the external light decreases, and the exposure time decreases as the external light increases.

  The line-of-sight measurement unit 30 detects the direction of the line of sight of the operator U using the face image PF captured by the imaging unit 10. The line-of-sight measurement unit 30 may be capable of detecting not only the line-of-sight direction of the operator U but also the face direction (hereinafter, “face orientation”). The line-of-sight measurement unit 30 repeatedly performs the line-of-sight measurement process (see FIG. 2). The line-of-sight measurement unit 30 includes a frame memory 31, a face detection unit 32, a face part detection unit 33, an eye detection unit 34, a geometric calculation unit 35, an error detection unit 36, a direction measurement memory 37, and a map generation unit 38. Yes.

  The frame memory 31 acquires the face image PF from the image acquisition unit 21. The frame memory 31 is a storage unit that can temporarily hold a face image PF for at least one frame. The face image PF stored in the frame memory 31 can be accessed simultaneously from the face detection unit 32, the face part detection unit 33, and the eye detection unit 34.

  The face detection unit 32 detects the face of the operator U from features such as the density of the outline of the face in the face image PF (see S31). The face part detection unit 33 detects these face parts from the features such as shades and positional relationships around the face parts such as eyes, nose and mouth (see S32). The eye detection unit 34 performs precise patterning of the iris or pupil by performing circular pattern matching from the roughly specified eye location by face detection by the face detection unit 32 or face part detection by the face part detection unit 33. (See S33). The geometric calculation unit 35 calculates the line-of-sight direction and the face direction from the positional relationship between the face parts and the positional relationship between the eyelids and the eyes. The detection results of the gaze direction and the face direction are stored in the measurement memory.

  The error detection unit 36 detects an error state when the line-of-sight direction or the face direction cannot be detected from the face image PF by the line-of-sight measurement process (see FIG. 2). As shown in FIG. 3, if the exposure of the imaging unit 10 is appropriately controlled with respect to the brightness, the detection of the face contour and the extraction of the iris, Purkinje, and the like are both performed normally. On the other hand, in the case of overexposure, the iris of the eye, Purkinje, etc. can be extracted normally, but the detection of the face contour becomes difficult. In the case of underexposure, detection of the facial contour can be performed normally, but extraction of the iris and Purkinje of the eyes becomes difficult. The error detection unit 36 illustrated in FIG. 1 notifies the map generation unit 38 of error occurrence information when detecting an error state due to either overexposure or underexposure.

  The direction measurement memory 37 is used as a storage area for calculation by the face detection unit 32, the face part detection unit 33, and the eye detection unit 34. The direction measurement memory 37 stores information on the line-of-sight direction and the face direction detected by the geometric calculation unit 35. These pieces of information are referred to by the operation control unit 40.

  The map generation unit 38 acquires, from the error detection unit 36, information on occurrence of detection errors for the line-of-sight direction and the face direction. The map generation unit 38 acquires from the exposure evaluation unit 22 and the exposure setting unit 23 information on the exposure value and the photometric quantity at the time of capturing the face image PF in which an error has occurred. The map generation unit 38 records the occurrence of an error in association with the photometric amount and the exposure value. The map generation unit 38 continuously performs the map generation process (see FIG. 4), thereby learning the allowable range of exposure values that can be set for the current photometric amount, and setting the availability map (see FIG. 5B). Generate.

  Specifically, the map generation unit 38 generates an error occurrence rate (hereinafter referred to as “error rate”) based on error occurrence information acquired from the error detection unit 36 every time the line-of-sight detection is output by the geometric calculation unit 35. Is monitored (see S41). The line-of-sight detection error rate is a value obtained by dividing the number of occurrences of line-of-sight detection errors by the number of line-of-sight detection outputs. The error rate is calculated separately for both the numerator and denominator by exposure value and photometric quantity. When the number of eye-gaze detection outputs is small, a correction coefficient corresponding to the number of detections may be multiplied to reduce an error in the error rate.

  The map generation unit 38 determines whether or not the error rate is equal to or higher than a specified value (see S42). As shown in FIG. 5, the shorter the exposure time, the less likely the error rate will increase even if the surroundings of the operator U become brighter. On the contrary, as the exposure time becomes longer, the error rate is less likely to increase even if the surroundings of the operator U become dark.

  The map generation unit 38 shown in FIG. 1 records an exposure value (exposure amount) and a photometric amount as an error log when the error rate is equal to or higher than a specified value (see S43). The map generation unit 38 statistically analyzes the recorded error log, and generates a setting availability map based on the error rate (see S44). As shown in FIG. 5B, the setting availability map defines whether or not each exposure value can be set for the photometric quantity. The exposure value setting availability map is provided to the exposure setting unit 23. The range of “◯” in the setting availability map is a permission range indicating that setting is possible, and the range of “x” is outside the permission range where setting is not possible. In the permitted range, the error rate is less than or equal to the specified value, and outside the permitted range, the error rate exceeds the specified value. The error rate for the face direction can be calculated by the same method as the error rate for the line-of-sight direction.

  The line-of-sight measurement unit 30 illustrated in FIG. 1 records an exposure condition in which the error rate exceeds a specified value for each output of detection results such as a line-of-sight direction and a face direction. The error occurrence information is fed back to the exposure settings of the exposure control unit 20 and the imaging unit 10 so that the exposure is controlled within a range where no error occurs.

  The operation control unit 40 includes an operation instruction unit 41 and a sudden change prediction unit 42. The operation instruction unit 41 acquires information on the line-of-sight direction and the face direction from the direction measurement memory 37. The operation instruction unit 41 sets an operation to instruct the navigation device 110 based on the line-of-sight direction and the face direction. The sudden change prediction unit 42 predicts a sudden change in the photometric quantity accompanying the movement of the vehicle. The sudden change prediction unit 42 predicts a sudden change in the photometric quantity at a timing just before the vehicle enters the tunnel or the like and a timing just before the vehicle leaves the tunnel, for example, based on the map data. The sudden change prediction unit 42 acquires information for predicting a sudden change in brightness around the vehicle through the external communication unit 50.

  The external communication unit 50 is electrically connected to the navigation device 110 directly or indirectly. The external communication unit 50 acquires map data and the like of the traveling direction of the vehicle from the navigation device 110. The external communication unit 50 acquires location information such as a tunnel based on the map data, and provides it to the sudden change prediction unit 42. The external communication unit 50 outputs the operation set by the operation instruction unit 41 to the navigation device 110 as a command signal. Under the control of the operation instruction unit 41 and the external communication unit 50, the operator U can perform scrolling of the display screen, selection of icons, and the like according to the line-of-sight direction.

  The navigation device 110 includes a GNSS (Global Navigation Satellite System) receiver 111, a map database (hereinafter “map DB”) 112, a display 113, and a communication device 114. The GNSS receiver 111 measures the current position of the vehicle. The map DB 112 is a storage device that stores a large amount of map data. The display 113 displays the current position of the vehicle and surrounding map data on the display screen. The communication device 114 acquires traffic regulation information and the like from an information database (hereinafter “information DB”) 121 provided in the information center 120. With the above configuration, the navigation device 110 guides the route to the destination to the operator U who is a driver.

  Next, details of exposure control performed by the exposure control unit 20 will be described with reference to FIG. 1 based on FIG. 6. The time chart shown in FIG. 6 shows the transition of exposure control when the vehicle passes through the tunnel. In FIG. 6, the solid line represents the transition of exposure control including feedback, and the broken line represents the transition of exposure control without feedback for comparison.

  During normal travel before time t1, the exposure time is controlled within a permission range that can be set in the setting availability map (see FIG. 5B) based on the photometric quantity calculated from the face image PF. At time 1, the sudden change prediction unit 42 detects the entrance of the tunnel in the traveling direction based on the map data acquired from the navigation device 110. When the brightness change from light to dark is prefetched by the sudden change prediction unit 42, the exposure setting unit 23 sets an exposure value that shifts the exposure time to the overexposed side slightly before the change in brightness occurs. Make adjustments. Specifically, the exposure time is set to the sum of the normal exposure feedback calculation value and the overshift setting value. The exposure time after the shift is set to a value with a margin with respect to the upper limit value so as not to exceed the upper limit value for the current photometric amount. Note that the upper limit value and the lower limit value of the exposure time can be set based on substantially the same concept as the setting availability map, and are boundary values in a region where the error rate is equal to or less than a specified value.

  When entering the tunnel at time 2, a sudden change in brightness from light to dark occurs. At this time, the follow-up time for the exposure value to follow the photometric amount can be shortened by the exposure time shift performed in advance. As a result, it is possible to prevent an error or return to normal from an error in a very short time.

  Further, at time t2, the exposure control unit 20 sets a control gain for response control that changes the exposure value in accordance with the photometric quantity to be high, with detection of a sudden change in the photometric quantity as a trigger. Thereby, the response delay of exposure time is shortened. Specifically, the control gain is switched from a normal value to a preset acceleration control gain. The acceleration control gain is temporarily set in the control target range from time t2 to time t3. When the exposure value calculated using the acceleration control gain is within the allowable range of the setting availability map, the calculated exposure value is used as it is by the controller 14. On the other hand, when the exposure value calculated using the acceleration control gain is outside the allowable range of the setting availability map, the calculated exposure value is corrected so as to fall within the allowable range. As described above, even within the control target range, only a value learned to be suitable for detection of the line-of-sight direction is used as the exposure value.

  At time 4, the sudden change prediction unit 42 detects the exit of the tunnel in the traveling direction, and pre-reads the brightness change from dark to bright. As a result, the exposure setting unit 23 adjusts the exposure value so as to shift the exposure time to the underexposure side slightly before the change in brightness occurs. Specifically, the exposure time is set to the sum of the normal exposure feedback calculation value and the under shift setting value. The exposure time after the shift is set to a value with a margin with respect to the lower limit value so as not to exceed the lower limit value for the current photometric amount. The over shift setting value is a positive value that increases the exposure time, and the under shift setting value is a negative value that decreases the exposure time.

  At time 5, when a sudden change in brightness from dark to bright occurs due to the approach to the tunnel exit, the control gain for controlling the exposure value triggered by the detection of the sudden change in the photometric quantity is changed from the normal value to the acceleration control gain. Can be switched to. At time t6, the vehicle passes through the tunnel exit. Even in the case of a sudden change in brightness from dark to bright, the acceleration control gain is temporarily set in the control target range from time t5 to time t7. As described above, the exposure time shifts between the upper limit value and the lower limit value even in the transition period in which the brightness changes suddenly.

  Note that the method of pre-reading the brightness change by the sudden change prediction unit 42 may be changed as appropriate. For example, the sudden change prediction unit 42 may predict the future brightness around the vehicle from an image in the traveling direction of a periphery monitoring camera or a drive recorder provided in the vehicle. Further, the navigation device 110 may calculate that the direct sunlight hits the face of the operator U, and the information may be provided to the sudden change prediction unit 42.

  Details of the control performed by the line-of-sight operating device 100 in order to realize the control exemplified in the above time chart will be described based on FIGS. 7 to 9 and with reference to FIGS. Each flow described in FIGS. 7 to 9 is started when the line-of-sight operating device 100 is activated, and is repeatedly performed until the line-of-sight operating device 100 is stopped.

  The exposure control process shown in FIGS. 7 and 8 is performed mainly by the exposure control unit 20, and sets the exposure value using the exposure setting availability map so that the detection error is suppressed to a predetermined value or less. It is. In S51 of the exposure control process, exposure control is performed every time the face image PF is captured, and the process proceeds to S52. In S52, it is determined whether or not the exposure control has been initialized. If the initialization has been completed, the process proceeds to S55. On the other hand, if the initialization has not been completed, the process proceeds to S53.

  In S53, the initial value of the set value is loaded as a temporary value used at the time of activation, and the process proceeds to S54. In S54, an initialized flag is set, and the process proceeds to S55. In S55, it is determined whether or not there is data of a setting availability map newly updated by the map generation processing (see FIG. 4). If it is determined in S55 that there is no update data, the process proceeds to S57. On the other hand, if it is determined in S55 that there is update data, the process proceeds to S56. In S56, the setting availability map is updated, and the process proceeds to S57.

  In S57, a photometric amount is calculated from the face image PF, and a target light amount is calculated. As an example, in S57, the current brightness is estimated from the average brightness calculated from the face image PF, the target brightness of the face image PF is calculated, and the process proceeds to S58.

  In S58, it is determined whether or not the sudden change prediction flag of the photometric quantity is set by a sudden change prediction process (see FIG. 9) described later. When the sudden change prediction flag is not detected in S58, the process proceeds to S60. On the other hand, when the sudden change prediction flag is detected in S58, the process proceeds to S59. In S59, the target light quantity of the photometric quantity is increased by a predetermined value, increased or decreased, and the process proceeds to S60. Specifically, when a sudden change from light to dark is predicted, the target light amount is increased so that the overshift setting value is added to the exposure time (see times t1 to t2 in FIG. 6). On the other hand, when a sudden change from dark to bright is predicted, the target light amount is reduced so that the under shift setting value is added to the exposure time (see times t4 to t5 in FIG. 6).

  In S60, it is determined whether or not the sudden change flag of the photometric quantity is set by a sudden change detection process (see FIG. 10) described later. If the sudden change flag is not detected in S60, the process proceeds to S61. In S61, the exposure amount (exposure value) is calculated so that the photometric light amount (average luminance) of the face image becomes the target light amount (target luminance). In addition, in S61, the control gain for controlling the exposure value is set to a normal value, and the process proceeds to S63.

  On the other hand, when the sudden change flag is detected in S60, the process proceeds to S62. Also in S62, the exposure amount (exposure value) is calculated so that the photometric light amount (average luminance) of the face image becomes the target light amount (target luminance). In addition, in S62, the control gain is increased from the normal value. Specifically, the control gain is switched from the normal value to the acceleration response gain with high responsiveness, and the process proceeds to S63 (see time t2 to t3 and time t5 to t6 in FIG. 6).

  In S63, the exposure value calculated in S61 or S62, that is, the relationship between the exposure time and the photometric quantity is collated with the setting availability map, and the process proceeds to S64. In S64, it is determined whether or not the exposure value calculated in S61 or S62 can be set as it is. If it is determined in S64 that the calculated value can be set, the process proceeds to S66 and the exposure value is transmitted to the controller 14.

  On the other hand, if it is determined in S64 that the calculated value is within the setting rejection range, the process proceeds to S65. In S65, the relationship between the exposure time and the measured amount is corrected. Specifically, in S66, the shift setting value, the acceleration control gain, and the like are adjusted so as not to enter the setting rejection area in the setting availability map, the exposure value including the correction value is recalculated, and the process proceeds to S66. . In S66, the corrected exposure value recalculated in S65 is transmitted to the controller 14.

  The sudden change prediction process shown in FIG. 9 is mainly performed by the sudden change predicting unit 42, detects a forward tunnel or the like based on map data acquired from the navigation device 110 (see times t1 and t4 in FIG. 6), and controls exposure. This is a process for changing the mode. In S71 of the sudden change prediction process, prediction information in which the photometric quantity changes suddenly, that is, information such as a tunnel ahead is received from another device such as the navigation device 110, and the process proceeds to S72.

  In S72, it is determined in S71 whether or not information for which a sudden change in the photometric quantity is predicted is detected. If the sudden change prediction information is not detected in S72, the sudden change prediction process is temporarily terminated. On the other hand, when the sudden change prediction information is detected in S72, the process proceeds to S73. In S73, the abrupt change prediction flag of the photometric quantity is set, and the process proceeds to S74.

  In S74, it is determined whether the set flag of the timer is set. If it is determined in S74 that the timer has been set, the sudden change prediction process is temporarily terminated. On the other hand, if it is determined that the timer has not been set, the process proceeds to S75. In S75, a timer is set, and the process proceeds to S76. In S76, the set flag of the timer is set, and the sudden change prediction process is temporarily ended. Note that the exposure control unit 20 is in a standby state for detecting a sudden change in the photometric quantity by setting the timer in S75. The timer value is also used in the sudden change detection process shown in FIG.

  The sudden change detection process illustrated in FIG. 10 is a process that is mainly performed by the exposure control unit 20 and detects a sudden change in brightness based on a change in the face image PF captured by the imaging unit 10. In S81 of the sudden change detection process, it is monitored from the average luminance of the face image PF whether or not the photometric amount has changed suddenly, and the process proceeds to S82. In S82, it is determined whether or not a sudden change in the photometric quantity is detected by monitoring in S81. If a sudden change in the photometric quantity is detected in S82, the process proceeds to S83. In S83, the sudden change flag of the photometric quantity is set, and the sudden change detection process is temporarily ended.

  On the other hand, if a sudden change in the photometric quantity is not detected in S82, the process proceeds to S84. In S84, based on the timer value set in S75 of the sudden change prediction process, it is determined whether or not the sudden change standby state has timed out. If the timer value is less than the predetermined value, the sudden change detection process is temporarily terminated. On the other hand, if the value of the timer is equal to or greater than the predetermined value, it is determined that a timeout has occurred, and the process proceeds to S85. In S85, the standby for sudden change detection of the photometric quantity is stopped, and the process proceeds to S86. In S86, a timer termination process is performed, and the sudden change detection process is temporarily terminated.

  In the first embodiment described so far, the occurrence of an error in which the line-of-sight direction cannot be detected from the face image PF is recorded. Based on the error occurrence rate, a range of exposure values that can be set for the photometric amount is learned. As described above, if the occurrence of an error in the output stage in the detection of the direction of the line of sight is fed back to the setting of the exposure value, the occurrence of an error due to a difficult factor to be assumed in advance is gradually reduced through the use of the line-of-sight operating device 100. obtain. Based on the above, the line-of-sight operating device 100 can control the exposure in the imaging unit 10 within an appropriate range.

  In addition, according to the first embodiment, when a sudden change in the photometric quantity is predicted, the exposure value is shifted in advance to the side corresponding to such a change in brightness. In addition, the exposure value after the shift is within a range that can be set for the current photometric amount. According to the above, the exposure value can be maintained within a normal range before and after the sudden change in the photometric amount without protruding into a range where errors frequently occur.

  Further, according to the first embodiment, the shifted exposure value has a margin with respect to the upper or lower boundary value of the exposure value range that can be set in the setting availability map (FIG. 6). Therefore, it is possible to suppress a situation in which the exposure value protrudes into a range where errors frequently occur before sudden change.

  Furthermore, in the first embodiment, the control gain for changing the exposure value according to the photometric quantity is temporarily increased from the normal value to the acceleration control gain in the control target range. Therefore, even when the photometric quantity changes suddenly with passage through a tunnel or the like, the exposure value can follow the suddenly changed photometric quantity so that it falls within a range where errors do not occur frequently.

  In addition, according to the first embodiment, the exposure setting unit 23 corrects the exposure value within the permitted range even when the exposure value according to the acceleration control gain temporarily increased exceeds the permitted range. According to the above, even if a situation that cannot be assumed at the time of a sudden change in brightness occurs, frequent occurrence of errors can be prevented.

  In addition, as in the first embodiment, if the learning result is held in the form of a setting availability map, the exposure setting unit 23 quickly selects an exposure value for the current photometric amount with low-load processing. It becomes possible. Therefore, the followability of the exposure value with respect to the photometric quantity can be easily improved, and the frequent occurrence of errors can be further reduced.

  In the first embodiment, the operator U corresponds to a “passenger”, the exposure evaluation unit 22 corresponds to a “photometric amount acquisition unit”, and the geometric calculation unit 35 corresponds to a “direction detection unit”. The map generation unit 38 corresponds to a “setting learning unit”, the navigation device 110 corresponds to an “operation target device”, and the line-of-sight operation device 100 corresponds to a “direction detection device” and an “operation control device”.

(Second embodiment)
The second embodiment of the present disclosure shown in FIG. 11 is a modification of the first embodiment. In the exposure control unit 20 (see FIG. 1) of the second embodiment, control gain increase control based on detection of a sudden change in the photometric quantity is omitted. The exposure control unit 20 shifts the exposure time to the overexposure side immediately after time 1 based on the look-ahead of the brightness change, while using the normal value of the control gain in the control target range from time t2 to t3. Then, the exposure time is extended to the over side so as to follow the increase in the photometric quantity. Similarly, immediately after time 4, the exposure control unit 20 shifts the exposure time to the underexposure side based on the look-ahead of the brightness change, while the control gain range is normal in the control target range from time t5 to t6. Use the value to shorten the exposure time to the underside to follow the photometric quantity.

  In the second embodiment described so far, while the exposure shift is performed, the increase of the control gain is not performed. Also in the second embodiment, as in the first embodiment, occurrence of an error in direction detection is learned as a setting availability map and fed back to the exposure value setting. According to the above, since the exposure setting with a high error tendency is not used, the exposure in the imaging unit 10 (see FIG. 1) can be controlled within a range in which the line-of-sight direction can be detected. In FIG. 11, the solid line represents the transition of exposure control including feedback, and the broken line represents the transition of exposure control without performing feedback for comparison.

(Other embodiments)
Although a plurality of embodiments of the present disclosure have been described above, the present disclosure is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure. can do.

  In the first modification of the second embodiment, as shown in FIG. 12, the control gain increase control is omitted. Even in the first modification, the occurrence of an error in direction detection is learned as a setting availability map and fed back to the exposure value setting. As described above, the exposure control unit can capture a face image suitable for direction detection without causing underexposure and overexposure during the period from time t1 to t2 and time t3 to t4 when the photometric quantity changes suddenly. You can continue to set the exposure time. In FIG. 12, the solid line represents the transition of exposure control including feedback, and the broken line represents the transition of exposure control without feedback for comparison.

  In the embodiment, both the line-of-sight direction and the face direction are detected. However, the direction detected by the line-of-sight operating device may be only the line-of-sight direction or only the face direction. In addition, the map generation unit may generate a setting availability map for each of the gaze direction and the face orientation, or may generate one setting availability map based on detection errors for both the gaze direction and the face orientation. Good. Furthermore, the detection error information of the line-of-sight direction or the face direction may be learned in a form different from the setting availability map. For example, a discriminator (function) that defines a permissible range of exposure values that can be set for the photometric amount may be generated based on a detection error in the gaze direction or the face direction.

  The photometric quantity in the above embodiment is measured as the average luminance of the face image. However, the exposure control unit may sequentially acquire measurement data of the photometric quantity from a measuring instrument that measures the quantity of light, and set the exposure value based on the acquired measurement data of the photometric quantity.

  In the above-described embodiment, an example in which the exposure control according to the present disclosure is applied to the line-of-sight operating device has been described. However, the above-described exposure control may be applied to a direction detection device that does not include an operation control unit. Furthermore, the face image may be captured by an imaging unit separate from the line-of-sight operating device or the direction detecting device, and transmitted to the line-of-sight operating device or the direction detecting device through wired communication or wireless communication. In addition, the operation target device that can be operated by the line-of-sight detection by the line-of-sight operation device is not limited to the navigation device. For example, the operation target device that can be remotely operated by the line-of-sight operation device may be an audio device, an air conditioning control device, and a communication terminal brought into the vehicle.

  In the above embodiment, an example in which the line-of-sight operating device is mounted on a vehicle as a moving body has been described. However, the moving body is not limited to a vehicle, and may be a ship, an aircraft, a transportation device, or the like. The line-of-sight operating device is not limited to the configuration mounted on the moving body, and may be configured to be brought into the moving body by the user.

PF face image, U operator (passenger), 10 imaging unit, 22 exposure evaluation unit (photometric amount acquisition unit), 23 exposure setting unit, 35 geometric calculation unit (direction detection unit), 36 error detection unit, 38 map generation Unit (setting learning unit), 40 operation control unit, 42 sudden change prediction unit, 100 gaze operation device (direction detection device, operation control device), 110 navigation device (operation target device)

Claims (7)

Direction detection for detecting at least one of the line of sight of the passenger and the face using a face image (PF) imaged by an imaging unit (10) that captures the face of the passenger (U) boarding the moving body A device,
A photometric quantity acquisition unit (22) for acquiring photometric quantity information indicating brightness around the passenger;
An exposure setting unit (23) for setting an exposure value used in the imaging unit based on the photometric quantity and controlling the exposure in the imaging unit;
An error detection unit (36) for detecting an error state in which the direction cannot be detected from the face image;
A setting learning unit that records the occurrence of the error in association with the light measurement amount and the exposure value, and learns the allowable range of the exposure value that can be set for the light measurement amount based on the recorded error occurrence rate. (38)
The direction detection device, wherein the exposure setting unit sets the exposure value within the permission range set by the setting learning unit with respect to the photometric amount acquired by the photometric amount acquisition unit. A sudden change prediction unit (42) for predicting a sudden change in the photometric quantity accompanying the movement of the moving body,
The exposure setting unit shifts the exposure value to the side corresponding to a change in brightness before the sudden change in the photometric amount when the sudden change in the photometric amount is predicted by the sudden change prediction unit. ,
The value after the shift of the exposure value is a value within a range that can be set by the setting learning unit with respect to the photometric amount acquired by the photometric amount acquisition unit. Direction detection device.   The direction detection device according to claim 2, wherein the exposure setting unit sets the shifted exposure value to a value having a margin with respect to a boundary value of a range that can be set by the setting learning unit. The exposure setting unit
By monitoring the photometric quantity acquired by the photometric quantity acquisition unit, a sudden change in the photometric quantity is detected,
4. The control gain of response control for changing the exposure value in accordance with the photometric amount during a period in which the sudden change in the photometric amount is higher than before the sudden change in the photometric amount. The direction detection device according to claim 1. The exposure setting unit
It is determined whether or not the exposure value according to the corrected control gain is within the permission range set by the setting learning unit during a period in which the abrupt change in the photometric amount occurs.
The direction detection device according to claim 4, wherein when the exposure value is out of the permitted range, the exposure value is corrected so as to be within the permitted range. The setting learning unit generates a setting availability map that defines whether or not a plurality of the exposure values can be set for each of the light metering amounts having different values.
The said exposure setting part selects the said exposure value prescribed | regulated that it can set with the said setting availability map with respect to the said photometric quantity acquired by the said photometric quantity acquisition part. The direction detection device according to item. An operation control device connected to an operation target device (110) operable by a passenger (U) who rides on a moving body,
A direction detection unit (35) for detecting at least one direction of the line of sight of the passenger and the face, using a face image (PF) imaged by an imaging unit (10) for imaging the face of the passenger;
An operation control unit (40) for controlling the operation of the operation target device based on the direction detected by the direction detection unit;
A photometric quantity acquisition unit (22) for acquiring photometric quantity information indicating brightness around the passenger;
An exposure setting unit (23) for setting an exposure value used in the imaging unit based on the photometric quantity and controlling the exposure in the imaging unit;
An error detection unit (36) for detecting an error state in which the direction cannot be detected from the face image;
A setting learning unit that records the occurrence of the error in association with the light measurement amount and the exposure value, and learns the allowable range of the exposure value that can be set for the light measurement amount based on the recorded error occurrence rate. (38)
The operation setting device, wherein the exposure setting unit sets the exposure value within the permission range set by the setting learning unit with respect to the light measurement obtained by the light measurement acquisition unit.

Images