JP2018101211A - On-vehicle device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an on-vehicle device that can improve the accuracy in detecting the position of the eyeball center and thereby improving the accuracy in measuring the line of sight.SOLUTION: An on-vehicle device calculates the degree of the face directed to the front side and the degree of the line of sight directed to the front side on the basis of a user's face image data picked up with a camera, when the face is directed to the front side and the line of sight is directed to the front side, newly calculates the coordinates of the eyeball center, reflects the coordinates on the center coordinates of a three-dimensional eyeball model, and automatically calibrates the coordinates, when the face is directed to the front side and the line of sight is directed to the front side, calculates the position of the iris center, calculates the angle of the line of sight on the basis of the distance between the position of the iris center and the center line of a face image, and calculates the coordinates of the eyeball center in the three-dimensional eyeball model on the basis of the position of the iris center and the angle of the line of sight, repeats processing to calculate the coordinates of the eyeball center and updates the coordinates of the eyeball center in the three-dimensional model by using an averaged coordinate value, and thereby can reduce an influence of noise.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a vehicle-mounted device that uses a face image photographed to include a face, and more particularly to a calibration technique.

  In a vehicle, it is desired to detect the direction of the driver's line of sight and to use the information of the line of sight for safe driving. As representative techniques for detecting the line of sight, a “corneal reflection method”, a “pupil-corneal reflection method” (for example, Patent Document 1), and a “model-based method” (for example, Patent Document 2) are known. .

  In the case of the “corneal reflection method” and the “pupil-corneal reflection method”, a line of sight is detected based on the positional relationship between the pupils using corneal reflection light (Purkinje image) generated by applying infrared rays to the eyes as a reference point. In the case of the “model-based method”, the line of sight is calculated using a facial feature point group, a three-dimensional face model, and a three-dimensional eyeball model.

  The “corneal reflection method” and the “pupil-corneal reflection method” cannot perform line-of-sight measurement without calibration. On the other hand, the “model-based method” can be measured without calibration. Moreover, since the individual difference can be eliminated by performing the calibration, the detection accuracy of the line of sight can be improved.

  For example, in the technique of Patent Document 3, a target that can be visually recognized by the driver is displayed as a virtual image using a head-up display at the time of calibration, and the line-of-sight direction of the driver who is gazing at the position of the target is detected. Calibration is performed.

  For example, Patent Document 4 shows a calibration technique that does not require any special operation by the driver. For example, when it is considered that a specific state is detected based on the distance between the host vehicle and the other vehicle, the brake depression state, etc., and the driver's line of sight is likely to face the other vehicle, Calibration is automatically performed using the images taken in (including other vehicles). Further, when the ODO meter reset button is operated, calibration is performed using an image photographed by the camera assuming that the driver's line of sight is in a specific direction (see paragraph [0055]).

Japanese Patent Laid-Open No. 2006-49260 JP 2008-194146 A JP 2009-183473 A JP 2012-201232 A

  However, when the above-mentioned “corneal reflection method” or “pupil-corneal reflection method” is used for line-of-sight detection, the scale of the system for measurement increases, making it difficult to apply to the interior space of the vehicle. On the other hand, when the above “model-based method” is adopted, it can be realized with a single camera, which is suitable for application to the interior space of a vehicle.

  However, the “model-based method” has a lower gaze measurement accuracy than the “corneal reflection method”. That is, since the error in parameters used for eye gaze detection, such as the center position of the eyeball, is large, the detection accuracy decreases.

  Therefore, calibration is necessary. However, the method as disclosed in Patent Document 3 is difficult to apply to a vehicle because the burden on the user when performing calibration is very large. In addition, it is difficult to perform calibration with high accuracy by the method disclosed in Patent Document 4.

  The present invention has been made in view of the above-described circumstances, and an object of the present invention is to provide an in-vehicle device that can improve the detection accuracy of the center position of the eyeball and thereby increase the accuracy of eye gaze measurement.

In order to achieve the above-described object, the vehicle-mounted device according to the present invention is characterized by the following (1) to (4).
(1) An in-vehicle device including a data processing unit that calculates an eyeball center coordinate where a center is located in a three-dimensional eyeball model in which an eyeball included in the face image is modeled from a face image photographed so as to include the face.
(2) The data processing unit
A black eye center position where the center of the black eye included in the face image is located on the three-dimensional eyeball model is calculated;
Calculating a line-of-sight angle based on the distance between the center position of the black eye and the center line in the face image;
The vehicle-mounted device according to (1), wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated based on the black eye center position and the line-of-sight angle.
(3) The data processing unit
Determine whether the face included in the face image is facing the front,
Determining whether the line of sight based on the position of the black eye included in the face image is facing the front,
The in-vehicle device according to (1) or (2), wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated when it is determined that the face and the line of sight are facing the front.
(4) The data processing unit
The in-vehicle device according to any one of (1) to (3), wherein the calculation processing of the eyeball center coordinates is repeatedly performed, the calculation result is stored, and the result of averaging the plurality of calculation results is output.

According to the vehicle-mounted device having the configuration (1), since the algorithm for calibrating the center position of the eyeball is provided in the vehicle-mounted device, the accuracy of detection of the eyeball center position by the vehicle-mounted device is improved, and thereby the gaze measurement accuracy is improved. Will increase. In addition, since calibration is performed based on the photographed image, it is not necessary to prepare special calibration equipment, and the user of the vehicle-mounted device does not need to perform a special operation for calibration.
According to the vehicle-mounted device having the configuration (2), the eyeball center coordinates can be calculated by a simple process. Therefore, the processing load of the data processing unit is reduced, and even a computer with low calculation performance can be practically used. A small processing burden on the data processing unit is suitable for in-vehicle devices that are increasingly required to be downsized.
According to the vehicle-mounted device having the configuration (3), the measurement accuracy of the eyeball center coordinates is improved. In other words, since the calculation is performed with the face and black eyes facing the direction of the camera to be photographed, the calculation error due to the effect of the difference in the orientation of the face and the calculation error due to the effect of the difference in the direction of the line of sight can be reduced. . Further, in the case of the configuration in which the above (2) and (3) are combined, the eyeball center coordinates can be measured even when the face center and the image center are not the same. At the time of such calibration, it is desirable to shoot with the face positioned in the direction of the camera shooting direction of 0 degrees as a general technique. However, according to this configuration, the camera shooting direction of 0 degrees is desired. Even if it is not, it can be calibrated by the same processing as in the case where the face is positioned at 0 degrees in the shooting direction.
According to the vehicle-mounted device having the configuration (4), even when an apparent outlier is temporarily calculated due to noise, the influence can be reduced by averaging, so that the detection accuracy is improved. In particular, desirable results can be obtained in applications where the same driver is monitored for a long time.

  According to the vehicle-mounted device of the present invention, it is possible to improve the detection accuracy of the center position of the eyeball, thereby increasing the gaze measurement accuracy. That is, since the on-vehicle device is provided with an algorithm for calibrating the center position of the eyeball, the accuracy of detecting the center position of the eyeball by the on-vehicle device is improved, thereby increasing the accuracy of gaze measurement. In addition, since calibration is performed based on the photographed image, it is not necessary to prepare special calibration equipment, and the user of the vehicle-mounted device does not need to perform a special operation for calibration.

  The present invention has been briefly described above. Further, the details of the present invention will be further clarified by reading through a mode for carrying out the invention described below (hereinafter referred to as “embodiment”) with reference to the accompanying drawings. .

FIG. 1 is a perspective view illustrating a specific example of an arrangement state of the vehicle-mounted device according to the embodiment. FIG. 2 is a flowchart showing an outline of a processing procedure when a line of sight is detected using a three-dimensional eyeball model. FIG. 3 is a schematic diagram illustrating an example of a relationship between a two-dimensional input image and a three-dimensional eyeball model. FIG. 4 is a flowchart showing an outline of a processing procedure for automatically calibrating the eyeball center position. FIG. 5 is a flowchart showing each of the two types of processing procedures for calculating the face front degree. FIG. 6 is a front view showing a specific example of the positional relationship between the face image and the detected feature point group. FIG. 7 is a front view showing a specific example of the positional relationship between the face image and the center of the face. FIG. 8 is a front view showing a specific example of the relationship between the face center line and the left and right black eye positions in the face image. FIG. 9 is a plan view showing an example of the positional relationship between the eyeball center, black eye center, and viewing angle of the three-dimensional eyeball model. FIG. 10 is a front view showing an example of the positional relationship between the center line and the center of the left and right eyes when the center of the face is equal to the center of the image. FIG. 11 is a plan view showing an example of the viewing angle, distance, and positional relationship between the left and right black eye centers when the center of the face is equal to the center of the image. FIG. 12 is a front view showing an example of the positional relationship between the center line and the left and right black eye centers when the center of the face and the center of the image are not the same. FIG. 13 is a plan view illustrating an example of the positional relationship between the viewing angle, the distance, and the left and right black eye centers when the center of the face and the center of the image are not the same. FIG. 14 is a schematic diagram illustrating an example of the relationship between the correction value of the eyeball center position, the three-dimensional eyeball model, and the input image. FIG. 15 is a flowchart illustrating a specific example of processing for reflecting the correction value of the center position of the eyeball in the operation of the vehicle-mounted device.

  Specific embodiments relating to the vehicle-mounted device of the present invention will be described below with reference to the drawings.

<Specific examples of environment to which the present invention is applied>
The specific example of the arrangement | positioning state of the onboard equipment in embodiment of this invention is shown in FIG.
In the example shown in FIG. 1, a camera 12 and a processing unit 13 are installed in a vehicle interior 10 of a vehicle. Specifically, the person 11 sitting in the driver's seat, that is, the face including the driver's eye area 11a can be photographed with the camera 12 in the meter unit in front of the driver's seat or in the column cover. A camera 12 is installed.

  The processing unit 13 processes the two-dimensional image data obtained by the camera 12 capturing the face of the person 11, and estimates information indicating the direction of the line of sight 11b mainly from the data of the eye region 11a. The information on the line of sight 11b estimated by the processing unit 13 can be used for the vehicle-mounted device to recognize whether or not the driver driving the vehicle is moving the line of sight for safety confirmation. Of course, such line-of-sight information can be used for various purposes. Therefore, it is assumed that the vehicle-mounted device of the present invention is configured as, for example, the processing unit 13 in FIG. 1 or a part thereof.

  Note that the vehicle-mounted device of the present embodiment has a function of automatically calibrating the center position of the eyeball based on two-dimensional image data captured by the camera 12, as will be described later. Therefore, in order to calibrate the gaze detection function of the vehicle-mounted device, it is not necessary to prepare special calibration equipment other than the configuration shown in FIG. Also, the person 11 does not need to perform a special adjustment operation for calibration.

<Outline of eye-gaze detection algorithm>
FIG. 2 shows an outline of a processing procedure in the case where a gaze is detected based on a face image using a three-dimensional eyeball model. That is, when the computer of the vehicle-mounted device executes a predetermined program, operations according to the line-of-sight detection algorithm shown in FIG. 2 are sequentially performed.

  The camera 12 constantly shoots an image of an area including the face of the person 11 at a constant cycle, and outputs a video signal. The computer of the vehicle-mounted device captures one frame of the video signal from the camera 12 in step S11. In addition, “down-processing” such as conversion of the image data format including gray scale is actually performed in S11. For example, for each pixel position in one frame, 8-bit data representing the luminance with gradations in the range of “0 to 255” is aligned in the vertical direction (y) and horizontal direction ( Image data of a two-dimensional (2D) array arranged in x) is generated.

In step S12, the computer processes the one-dimensional or plural-frame two-dimensional image data input in step S11 to detect the face of the person 11. As a specific example, face detection is performed using the “Viola-Jones method”, and a region including a face is extracted as a rectangular region from one frame of two-dimensional image data. That is, a facial region is extracted using a detector that is characterized by the shadow difference of the face and created by learning using “Boosting”. Of course, the face may be detected by using a uniquely developed method. The technique of “Viola-Jones method” is shown in the following document.
"P.viola and MJJones," Rapid Object Detection Using a Boosted Cascade of Simple Features, "IEEE CVPR (2001)."

  In the next step S13, the data in the rectangular area of the face extracted in S12 is processed to detect each of the left and right black eyes (iris). For example, black eyes can be detected by using template matching which is a known technique. That is, for example, reference data of a template representing a feature of a black eye such as a black circular pattern is prepared in advance, and this template is sequentially scanned on the processing target image to specify a position where the feature matches. Note that before executing black-eye detection, the positions of various feature points or shading differences are detected for the entire face, and the positions of the regions of the left and right eyes are grasped. Thereby, when the left and right black eyes are detected, the range of data to be searched can be limited, so that the processing time can be shortened. For example, the eye region can be detected by using the “Viola-Jones method”.

  Now, in this embodiment, since it is assumed that the line of sight is detected by a model-based method using a three-dimensional eyeball model, it is necessary to specify the eyeball center position as a reference point for each of the left and right eyeballs. The computer estimates the eyeball center position in step S14. For this processing, it is assumed that a method for determining the center position of the eyeball based on the position of the corner of the eye and the position of the eyeball, or a method for calculating the center position of the eyeball by simultaneously estimating the skeleton from the facial feature points. However, in the embodiment of the present invention, the eyeball center position is specified using a special algorithm as described later.

  In step S15, the computer determines the state (position, angle) of the three-dimensional eyeball model based on the detected black eye position and eyeball center position, and calculates the angle of the line of sight based on the result.

  In step S16, the computer performs processing such as data averaging and noise removal on the line-of-sight angle obtained in step S15 as necessary, and uses other information such as an application program that uses information on the line-of-sight direction. Data representing the line of sight is transmitted to the vehicle-mounted device.

<Description of 3D eyeball model>
An example of the relationship between the two-dimensional input image and the three-dimensional eyeball model is shown in FIG.
In the three-dimensional eyeball model 30 shown in FIG. 3, the eyeball is a sphere represented by an eyeball center 31 and a radius R. The black eye (iris) 32 is regarded as a disk-shaped region having a radius Di attached on the eyeball surface of the three-dimensional eyeball model 30. The center 32a of the black eye (iris) 32 corresponds to the center of the pupil. Note that the dimensions of the eyeball radius R and the black eye radius Di can be determined based on human body dimension data and various parameters of the imaging system of the camera 12.

  In this three-dimensional eyeball model 30, the direction of the line of sight can be specified as the direction from the eyeball center 31 toward the center of the black eye 32, and the rotation angle yaw with respect to the reference direction in the horizontal plane and the rotation angle with respect to the reference direction in the vertical direction It can be represented by a pitch. Further, the coordinates of the black eye center 32a can be expressed by an eyeball radius R, a yaw, and a pitch when the eyeball center 31 is used as a reference.

  On the other hand, the video imaged by the camera 12 represents a specific two-dimensional plane. Therefore, for example, when the two-dimensional image 35 of the eye region shown in FIG. 3 is input as the input image, it is necessary to perform two-dimensional / three-dimensional mutual conversion in order to use the three-dimensional eyeball model 30. .

  Specifically, the two-dimensional coordinates of the position of the black eye center 32 a and the position of the eyeball center 31 are respectively detected in the two-dimensional image 35 of the eye region, and these two-dimensional coordinates are obtained as the third order of the three-dimensional eyeball model 30. Project onto original coordinates. Thereby, the vector of the line of sight 11b can be calculated. The relationship between the vector of the line of sight 11b and the rotation angles yaw and pitch in each direction is expressed by the following equations.

X = −R × cos (pitch) × sin (yaw) (1)
Y = R × sin (pitch) (2)
X: distance in the x direction from the eyeball center 31 on the two-dimensional image plane Y: distance in the y direction from the eyeball center 31 on the two-dimensional image plane

<Description of characteristic processing>
<Outline of processing>
An outline of a processing procedure for automatically calibrating the position of the eyeball center 31 in the three-dimensional eyeball model 30 is shown in FIG. For example, the computer in the processing unit 13 executes the process shown in FIG. 4 as step S14 shown in FIG. 2, whereby the position of the eyeball center 31 is automatically calibrated. Thereby, since the positional accuracy of the eyeball center 31 is improved, the visual line detection accuracy is also improved. There is no need for a user such as a driver to perform a special operation for calibration.

  In step S101 of FIG. 4, the computer calculates the front degree of the face of the person 11 shown in the input image, that is, “the degree that the face is facing the front”. Then, based on the calculation result of S101, if the state can be regarded as “the face is facing the front”, the process proceeds from step S102 to S103.

  In step S103, the computer calculates the front degree of the line of sight of the person 11 shown in the input image, that is, “the degree that the direction of the line of sight is facing the front”. Then, based on the calculation result of S103, if the state can be regarded as “the direction of the line of sight is facing the front”, the process proceeds from step S104 to S105.

  In step S105, the latest eyeball center position is calculated based on the current characteristics of the person 11 shown in the input image, and a correction value necessary for updating the eyeball center position is obtained. In step S106, the correction value is reflected on the coordinates of the eyeball center 31 of the three-dimensional eyeball model 30 to execute calibration.

<Details of “Face Facing” Calculation>
The “face front degree” represents “the degree to which the face is facing the front” on the input image. FIG. 5 shows each of two types of processing procedures for calculating the “face front degree”. Any one of these processing procedures is executed in step S101 shown in FIG. Each of “Face Frontality Calculation 1” and “Face Frontality Calculation 2” shown in FIG. 5 will be described below.

<"Face Frontality Calculation 1">
A specific example of the positional relationship between the face image and the detected feature point group 61 is shown in FIG.
In step S21 of FIG. 5, the image data F of the face area detected in S12 of FIG. 2 is input as a processing target. In step S22, the computer performs predetermined image processing on the image data F of the face area, and executes feature point detection. As a result, for example, as shown in FIG. 6, a feature point group 61 representing an outline or the like is detected for each part such as eyebrows, left and right eyes, nose, lips, and chin.

  In the next step S23, the computer calculates the barycentric positions of these facial feature point groups based on the feature point group 61 obtained in step S22 by the following equation (3). When this calculation is executed, processing is performed on the premise that the condition “the feature point group 61 is set symmetrically with respect to the center of the face” is satisfied.

g (x, y): Center of gravity position represented by coordinates (x, y) P: Total number of face feature points f (p): Position information of p-th face feature point

  Therefore, the x coordinate of the gravity center position g (x, y) calculated in step S23 is affected by the direction of the left and right faces. That is, when the face is front-facing (face orientation is 0), the x-coordinate “g.x” of the gravity center position g (x, y) is the x-coordinate of the face center (facecen.x, facecen.y). If the face is not facing the front, the center of gravity position g (x, y) is shifted from the center of the face.

Note that an x-coordinate value (facecen.x) and a y-coordinate value (facecen.y) representing the center position of the face are respectively the median value in the horizontal (x) direction and the vertical (y ) Can be calculated as the median of the direction.
Therefore, the face front degree Dx can be calculated by the following equation.

Dx: Face front degree (x direction)
facecen. x: x coordinate of the center of the face g. x: x coordinate αx of the center of gravity position: threshold value for identifying whether the face direction is front or not

  When the face front degree Dx calculated by the above equation (4-1) is used, the face front degree Dx is compared with the threshold value αx in step S102 of FIG. 4 to identify whether the face direction is front or not. it can. As for the magnitude of the threshold value αx, it is assumed that an optimum value is adopted in consideration of parameters representing the photographing conditions of the camera 12 and individual differences.

The face front degree can consider not only the horizontal direction but also the vertical direction. However, since the structure of the human face is not symmetrical in the vertical direction, the y coordinate g.g. It is necessary to obtain a deviation amount dy between y and the y coordinate “facecen.y” of the center position of the face. The dy value is determined by the feature point arrangement of the detector, and is obtained as a fixed value. The vertical face front degree Dy is expressed by the following equation (4-2).

<"Face frontal calculation 2">
A specific example of the positional relationship between the face image and the center of the face (facecen.x, facecen.y) is shown in FIG.

  In step S31 of FIG. 5, the image data F of the face area detected in S12 of FIG. 2 is input as a processing target.

  In step S32 of FIG. 5, the image data F of the face region input in S31 is processed, and the computer creates inverted image data FL in which the left and right are inverted with respect to the center (facecen.x) of the x-axis in the face image. . In the next step S33, the computer calculates the correlation value between the image data F and the reverse image data FL as the face front degree D. The face front degree D in this case is calculated by the following equation.

D: Face front degree N: Number of pixels in the x direction of the face image M: Number of pixels in the y direction of the face image F (i, j): Luminance value (gradation) of the pixel at the coordinates (i, j) of the image data F
FL (i, j): luminance value (gradation) of the pixel at the coordinates (i, j) of the inverted image data FL
α: Threshold value for identifying whether the face direction is front or not

  In other words, assuming that the facial features are symmetrical with respect to the center of the face (facecen.x), the features on the left side and the features on the right side from the center of the face are equal. Therefore, the face front degree D representing the reciprocal of the correlation between the image data F and the reverse image data FL becomes very small when the face direction is close to the front, and increases when the face direction deviates from the front.

  When the face front degree D calculated by the expression (5) is used, it is possible to identify whether or not the face direction is the front by comparing the face front degree D with the threshold value α in step S102 of FIG. As for the magnitude of the threshold value α, it is assumed that an optimum value is adopted in consideration of a parameter representing the photographing condition of the camera 12 and individual differences.

<Details of “Gaze frontality” calculation>
A specific example of the relationship between the center line of the face and the left and right black eye positions in the face image is shown in FIG.
The gaze front degree FD is calculated from the degree to which the gaze direction of the person 11 shown in the input image is facing the front, that is, the rotation angle obtained by applying the three-dimensional eyeball model 30 to the left and right eyes. Represents the degree to which the viewing direction θ (yaw, pitch) is 0 degrees.

  In the present embodiment, as shown in FIG. 4, the computer calculates the gaze front degree FD according to the following expression on the premise that the condition that “the face direction is the front” is satisfied.

FD: Gaze frontality irisR. x: x coordinate irisL. x: x-coordinate facecen. x: x-coordinate of the center of the face β: threshold for identifying whether the direction of the line of sight is front or not

  In the present embodiment, the positional relationship between the left and right eyes such as both eyes is described as representing the left and right viewed from the camera 12 side and the left and right on the two-dimensional image obtained by photographing, not the left and right with respect to the person 11. .

  That is, the line-of-sight front degree FD represented by the above expression (6) represents the difference between the center position between the left and right black eye centers and the position of the face center line (facecen.x) in the x-axis direction. ing. Therefore, for example, if the distance from the center line position (facecen.x) to the center position of the right black eye and the distance to the center position of the left black eye are equal, the direction of the line of sight faces the front. Can be determined. Further, if it can be determined that the direction of the line of sight is facing the front based on the positional relationship in the x-axis direction, it can be considered that the direction of the line of sight is also facing the front in the y-axis direction at the same time.

  The line-of-sight front degree FD becomes 0 when the direction of the line of sight faces the front, and increases as the line of sight deviates from the front. Therefore, when using the gaze front degree FD calculated by the above equation (6), the gaze front degree FD is compared with the threshold value β in step S104 in FIG. 4 to determine whether or not the gaze direction is front. Can be identified. As for the magnitude of the threshold value β, it is assumed that an optimum value is adopted in consideration of parameters representing photographing conditions of the camera 12 and individual differences.

<Calculation of eyeball center position>
The process for calculating the latest eyeball center position in step S105 shown in FIG. 4 will be described below.

<When the center of the face in the x-axis direction is equal to the center of the image>
An example of the positional relationship in the horizontal plane (or the plane parallel to the x and z axes) of the eyeball center, black eye center, and viewing angle of the three-dimensional eyeball model is shown in FIG. FIG. 10 shows an example of the positional relationship between the center line and the left and right black eye centers when the center of the face in the x-axis direction is equal to the center of the image. FIG. 11 shows an example of the positional relationship in the horizontal plane of the viewing angle, distance, and left and right black eye centers when the center of the face in the x-axis direction is equal to the center of the image.

Since each of the left and right eyes can be calculated by substantially the same method, the following description assumes that only the right eye is processed as a calculation target.
For example, in the state shown in FIG. 9, the relationship of the following expression (7) is established.

Xeyer = irisR. x-Ro · sin θ Ryaw (7)
Xeyer: x coordinate of right eyeball center irisR. x: x coordinate of the right eye center Ro: radius of the eyeball (eg constant)
θRyaw: Eye angle of the right eye (yaw direction)

  That is, if the position “irisR.x” of the center of the black eye and the line-of-sight angle “θRyaw” are known, the x-coordinate “Xeyer” of the eyeball center can be calculated from the above equation (7).

As shown in FIG. 10, the x-coordinate “facecen.x” of the face center position in the image is equal to the x-coordinate “Icenter.x” of the center of the image taken by the camera, as shown in FIG. Assuming the situation, the line-of-sight angle “θRyaw” can be calculated by the following equation (8).
tan θRyaw = (irisR.x-Icenter.x) / Df (8)
Df: Distance between camera and face

  The distance Df can be treated as known information if it is measured using an appropriate distance sensor, for example. Further, for example, if a technique disclosed in “Japanese Patent Laid-Open No. 2015-206764” is used, a distance sensor becomes unnecessary. Further, since the black eye can be detected from the face image as described above, the position “irisR.x” of the center of the black eye can also be used as known information. Further, for example, a predetermined constant can be used as the x-coordinate “Icenter.x” of the center of the image.

  Therefore, the line-of-sight angle “θRyaw” can be calculated from the equation (8). Further, the x-coordinate Xeyer of the right eyeball center can be calculated by substituting the line-of-sight angle “θRyaw”, the black eye center position “irisR.x”, and the eyeball radius Ro into the expression (7).

Also, the y-coordinate Yeyer at the center of the right eyeball can be similarly calculated by using the following formulas (9) and (10).
Yeyer = irisR. y-Ro · sinθRpitch (9)
Yyer: y-coordinate of the right eyeball center irisR. y: y coordinate of the right eye center Ro: radius of eyeball (for example, constant)
θRpitch: right eye's line-of-sight angle (pitch direction)

tan θRpitch = (irisR.y-Icenter.y) / Df (10)
Icenter. y: y coordinate of the center of the image Df: distance between the camera and the face

<When the center of the face in the x-axis direction is shifted from the center of the image>
An example of the positional relationship between the center line and the center of the left and right eyes when the center of the face and the center of the image are not the same is shown in FIG. Further, FIG. 13 shows an example of the positional relationship in the horizontal plane (plane parallel to the x and z axes) of the viewing angle, distance, and left and right black eyes when the center of the face and the center of the image are not the same.

  For example, as shown in FIG. 13, even if the x-coordinate “facecen.x” of the center line of the face is shifted from the x-coordinate “Icenter.x” of the center of the image as shown in FIG. Is centered on the camera 12 and exists on the circumference whose radius is represented by the distance Df. Further, the face center position “facecen.x” and the image center “Icenter.x” are shifted by a face rotation angle “facerad”. However, if the face is facing the front of the camera 12 and the eyes of both eyes are facing the camera 12, the right-eye gaze angle “θRyaw” is the image as shown in FIG. It can be expressed as the rotation angle of the eye position with respect to the center “Icenter.x”.

  Therefore, even when the center of the face in the x-axis direction is shifted from the center of the image, the line-of-sight angle “θRyaw” can be calculated using the equation (8). Further, the x-coordinate “Xeyer” of the center of the eyeball can be calculated based on the line-of-sight angle “θRyaw” and the expression (7).

On the other hand, even when the coordinates of the center of the eyeball are calculated for the left eye, it can be calculated in the same manner as described above by using the following calculation formulas.
Xeyel = irisL. x-Ro · sin θLyaw (11)
Xeyel: x-coordinate of left eyeball center irisL. x: x coordinate of the center of the black eye on the left side Ro: radius of eyeball (for example, constant)
θLyaw: Eye angle of the left eye (yaw direction)
tan θLyaw = (irisL.x-Icenter.x) / Df (12)
Df: Distance between camera and face

Yeyel = irisL. y-Ro · sinθLpitch (13)
Yeel: y-coordinate of left eyeball center irisL. y: y coordinate of the left eye center Ro: radius of eyeball (for example, constant)
θLpitch: Eye angle of the left eye (pitch direction)
tan θLpitch = (irisL.y-Icenter.y) / Df (14)
Icenter. y: y coordinate of the center of the image Df: distance between the camera and the face

<Correction of eyeball center position>
Even when there is an error in the position coordinates of the eyeball center 31 in the three-dimensional eyeball model 30, the correction can be performed so that the error can be reduced by correcting using the latest eyeball center position obtained by the above calculation. it can. The correction value can be calculated by, for example, the following formula.

dXeye = Xeye-Xeyemodel (15)
dYeye = Yeee-Yeemodel (16)
dXeye: x coordinate correction value (right or left)
dYeye: y coordinate correction value (right or left)
Xeye: The latest x-coordinate of the calculated center position of the eyeball (right or left)
Yeye: the latest y-coordinate of the calculated center position of the eyeball (right or left)
Xeyemodel: eye center x coordinate (right or left) before correction in the three-dimensional eye model
Eyemodel: y-coordinate of the eyeball center before correction in the three-dimensional eyeball model (right or left)

<Eyeball center position correction 1>
An example of the relationship between the correction value dXeye of the eyeball center position, the three-dimensional eyeball model, and the input image is shown in FIG.

  In step S105 shown in FIG. 4, since the face front degree D is less than or equal to the threshold value α and the line-of-sight front degree FD is less than or equal to the threshold value β, the coordinates of the latest eyeball center position (Xeye, Yeye) are determined. It calculates about each of.

  Then, the correction values dXeye and dYeye are calculated for each of the left and right eyes according to the above equations (15) and (16). Further, the position coordinates of the eyeball center 31 in the three-dimensional eyeball model 30 are updated for each of the left and right eyes as shown in FIG. 14 using the correction values dXeye and dYeye so that the error is reduced.

<Eyeball center position correction 2>
A specific example of processing for reflecting the correction value of the center position of the eyeball in the operation of the vehicle-mounted device is shown in FIG.
When the process shown in FIG. 4 is applied to the vehicle-mounted device, step S105 is repeatedly executed each time a state where the person 11 faces the front and the line of sight faces the front is detected. Therefore, the update of the eyeball center position can be repeated using the new correction value after the power is turned on to the vehicle-mounted device and before the operation is completed.

  Therefore, in order to perform a more appropriate update process, for example, the computer executes the process of “eyeball center position correction 2” shown in FIG. In FIG. 15, only the coordinate in the x-axis direction is processed, but the coordinate in the y-axis direction can be similarly processed. With the processing shown in FIG. 15, the coordinates of the eyeball center position can be converged to a certain value.

  In step S41 of FIG. 15, the computer calculates a correction value dXeye for the center position of the eyeball based on the above expressions (7), (8), and (15). In addition, in order to eliminate an apparent outlier that may occur due to the influence of noise, the correction value dXeye is compared with a predetermined threshold value in the next step S42.

  If the correction value dXeye is not an obvious outlier, 1 is added to the update count value N in step S43 (the initial value of N is 0). In the next step S44, the data of the current correction value dXeye is stored in a memory location associated with the update count value N.

  In step S45, the computer reads data of N correction values dXeye stored in the memory, and outputs a result dXeye_T obtained by averaging the data. As a specific method of averaging, calculation methods such as a simple average value, a weighted average value, and a moving average value can be applied.

  By applying the processing as shown in FIG. 15, an apparent outlier due to the influence of noise can be excluded from the update target. Moreover, the rapid change of the eyeball center position can be avoided by averaging. That is, it is robust against noise. Therefore, in an environment where an operation targeting the same user is performed for a long time, such as an in-vehicle device that monitors the driver's line of sight, high eye-gaze detection accuracy can be expected by updating the effective eyeball center as needed. .

<Advantages of OBE>
In the vehicle-mounted device described above, the position of the eyeball center 31 can be automatically calibrated by executing the processing shown in FIG. In addition, it is not necessary for the user to perform a special operation in order to perform this calibration, and it is not necessary to prepare special calibration equipment. Further, by repeatedly executing the processing shown in FIG. 15, it is possible to eliminate the influence of noise and prevent a sudden change, and thus it is possible to avoid malfunction of the vehicle-mounted device.

  Therefore, the above-described technique can be used for driver's line-of-sight detection and side-view detection using an in-vehicle camera. Also, for example, it can be used for visual confirmation of digital signage (electronic signage).

Here, the features of the above-described embodiments of the vehicle-mounted device according to the present invention are summarized and listed in the following [1] to [4], respectively.
[1] An eyeball center where a center (eyeball center 31) in a three-dimensional eyeball model (three-dimensional eyeball model 30) in which an eyeball included in the face image is modeled from a face image photographed so as to include the face is located A vehicle-mounted device including a data processing unit (processing unit 13) that calculates coordinates (S105).
[2] The data processing unit
A black eye center position where the center of the black eye included in the face image is located on the three-dimensional eyeball model is calculated;
Calculating a line-of-sight angle based on the distance between the center position of the black eye and the center line in the face image;
Based on the black eye center position and the line-of-sight angle, the eyeball center coordinates in the three-dimensional eyeball model are calculated (see Expression (7)).
The vehicle-mounted device according to [1] above.
[3] The data processing unit
It is determined whether the face included in the face image is facing the front (S102),
It is determined whether or not the line of sight based on the position of the black eye included in the face image is facing the front (S104),
When it is determined that the face and line of sight are facing the front, the eyeball center coordinates in the three-dimensional eyeball model are calculated (S105).
The vehicle-mounted device according to [1] or [2] above.
[4] The data processing unit
The calculation process of the eyeball center coordinates is repeatedly performed, the calculation result is stored, and the result obtained by averaging a plurality of calculation results is output (S45).
The vehicle-mounted device according to any one of [1] to [3].

DESCRIPTION OF SYMBOLS 10 Vehicle interior 11 Person 11a Eye area 11b Line of sight 12 Camera 13 Processing part 30 3D eyeball model 31 Eyeball center 32 Black eye (iris)
32a Center of black eyes (pupil)
35 Two-dimensional image of eye area

Claims (4)

  An in-vehicle device including a data processing unit that calculates an eyeball center coordinate where a center is located in a three-dimensional eyeball model in which an eyeball included in the face image is modeled from a face image photographed so as to include a face. The data processing unit
A black eye center position where the center of the black eye included in the face image is located on the three-dimensional eyeball model is calculated;
Calculating a line-of-sight angle based on the distance between the center position of the black eye and the center line in the face image;
The in-vehicle device according to claim 1, wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated based on the black eye center position and the line-of-sight angle. The data processing unit
Determine whether the face included in the face image is facing the front,
Determining whether the line of sight based on the position of the black eye included in the face image is facing the front,
The in-vehicle device according to claim 1, wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated when it is determined that the face and the line of sight are facing the front. The data processing unit
The in-vehicle device according to any one of claims 1 to 3, wherein the calculation processing of the eyeball center coordinates is repeatedly performed, the calculation result is stored, and a result obtained by averaging a plurality of calculation results is output.