JP6906943B2 - On-board unit - Google Patents

Description

The present invention relates to an in-vehicle device that uses a face image taken so as to include a face, and particularly relates to a calibration technique.

In a vehicle, it is desired to detect the direction of the driver's line of sight and use the information of this line of sight for safe driving and the like. As typical methods for detecting the line of sight, "corneal reflex method", "pupil-corneal reflex method" (for example, Patent Document 1), and "model-based method" (for example, Patent Document 2) are conventionally known. ..

In the case of the "corneal reflex method" and the "pupil-corneal reflex method", the line of sight is detected based on the positional relationship of the pupils with the corneal reflected light (Pulkinier image) generated by irradiating the eyes with infrared rays as a reference point. In the case of the "model-based method", the line of sight is calculated using a facial feature point cloud, a three-dimensional face model, and a three-dimensional eyeball model.

The "corneal reflex method" and the "pupil-corneal reflex method" cannot measure the line of sight without calibration. On the other hand, the "model-based method" can measure without calibration. In addition, by performing calibration, individual differences can be eliminated, so that the line-of-sight detection accuracy can be improved.

For example, in the technique of Patent Document 3, a head-up display is used at the time of calibration to display a target visible to the driver as a virtual image in front, and to detect the line-of-sight direction of the driver while gazing at the position of the target. Calibration is carried out by this.

Further, for example, Patent Document 4 shows a calibration technique that does not require a special operation by the driver. For example, when a specific state is detected based on the distance between the own vehicle and another vehicle, the state of braking, etc., and it is highly likely that the driver's line of sight is facing the other vehicle, the camera is used. Calibration is automatically performed using the images taken in (including other vehicles). Further, when the ODO meter reset button is operated, calibration is performed using an image taken by the camera assuming that the driver's line of sight is in a specific direction (see paragraph [0055]).

Japanese Unexamined Patent Publication No. 2016-49260 Japanese Unexamined Patent Publication No. 2008-194146 Japanese Unexamined Patent Publication No. 2009-183473 Japanese Unexamined Patent Publication No. 2012-201232

However, when the above-mentioned "corneal reflex method" or "pupil-corneal reflex method" is adopted for line-of-sight detection, it is difficult to apply it to the space inside the vehicle because the scale of the system for measurement becomes large. On the other hand, when the above-mentioned "model-based method" is adopted, it can be realized with a single camera, so that it is suitable for application to the vehicle interior space.

However, the "model-based method" has poorer line-of-sight measurement accuracy than the "corneal reflex method". That is, since the error of the parameter used for detecting the line of sight, for example, the position of the center of the eyeball, is large, the detection accuracy is lowered.

Therefore, calibration is required. However, the method as shown in Patent Document 3 is difficult to apply to a vehicle because the burden on the user when performing calibration is very large. Further, it is difficult to calibrate with high accuracy by the method shown in Patent Document 4.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an in-vehicle device capable of improving the detection accuracy of the center position of the eyeball and thereby improving the eye-gaze measurement accuracy.

In order to achieve the above-mentioned object, the on-board unit according to the present invention is characterized by the following (1) to (3).
(1) A data processing unit for calculating the eyeball center coordinates where the center is located in the three-dimensional eyeball model in which the eyeball included in the face image is modeled from the face image taken so as to include the face is provided .
The data processing unit
The center of the black eye included in the face image is located on the three-dimensional eyeball model, and the center position of the black eye is calculated.
The line-of-sight angle is calculated based on the distance between the center position of the black eye and the center line passing through the center of the image in the face image, and the distance between the camera that captures the face image and the face.
An in-vehicle device that calculates the eyeball center coordinates in the three-dimensional eyeball model based on the black eye center position and the line-of-sight angle.
( 2 ) The data processing unit
It is determined whether or not the face included in the face image is facing the front, and it is determined.
It is determined whether or not the line of sight based on the position of the iris included in the face image is facing the front.
The on-board unit according to (1) above, which calculates the eyeball center coordinates in the three-dimensional eyeball model when it is determined that the face and the line of sight are facing the front.
( 3 ) The data processing unit
The on-board unit according to (1) or (2) above, wherein the calculation process of the eyeball center coordinates is repeated, the calculation results are saved, and the result of averaging a plurality of calculation results is output.

According to the on-board unit having the configuration of (1) above, since the on-board unit is provided with an algorithm for calibrating the eyeball center position, the detection accuracy of the eyeball center position by the on-board unit is improved, thereby improving the eye-gaze measurement accuracy. Will increase. Further, since the calibration is performed based on the captured image, it is not necessary to prepare special calibration equipment, and the user of the on-board unit does not need to perform any special operation for calibration.
Further, according to the in-vehicle device having the above configuration (1 ), the eyeball center coordinates can be calculated by a simple process. Therefore, the processing load of the data processing unit is reduced, and even a computer having low computing performance can withstand practical use. The fact that the processing load of the data processing unit is small is suitable for in-vehicle devices that are increasingly required to be miniaturized.
According to the in-vehicle device having the configuration of (2 ) above, the measurement accuracy of the eyeball center coordinates is improved. That is, since the calculation is executed with the face and the iris facing the direction of the camera to be photographed, it is possible to reduce the calculation error due to the influence of the difference in the direction of the face and the calculation error due to the influence of the difference in the direction of the line of sight. .. Further, in the case of the configuration in which the above (1 ) and ( 2 ) are combined, the eyeball center coordinates can be measured even when the face center and the image center are not the same. In such calibration, as a general method, it is desirable to shoot with the face positioned in the direction of 0 degrees in the shooting direction of the camera, but according to this configuration, the shooting direction is 0 degrees in the camera. Even without it, it can be calibrated by the same process as when the face is positioned at 0 degrees in the shooting direction.
According to the in-vehicle device having the above configuration (3 ), even if a clear outlier is temporarily calculated due to noise, the influence can be reduced by averaging, so that the detection accuracy is improved. In particular, desirable results can be obtained in applications such as monitoring the same driver for a long time.

According to the on-board unit of the present invention, it is possible to improve the detection accuracy of the center position of the eyeball, thereby improving the eye-gaze measurement accuracy. That is, since the algorithm for calibrating the eyeball center position is provided in the on-board unit, the detection accuracy of the eyeball center position by the on-board unit is improved, and the line-of-sight measurement accuracy is improved. Further, since the calibration is performed based on the captured image, it is not necessary to prepare special calibration equipment, and the user of the on-board unit does not need to perform any special operation for calibration.

The present invention has been briefly described above. Further, the details of the present invention will be further clarified by reading through the embodiments described below (hereinafter referred to as "embodiments") with reference to the accompanying drawings. ..

FIG. 1 is a perspective view showing a specific example of the arrangement state of the vehicle-mounted device of the embodiment. FIG. 2 is a flowchart showing an outline of a processing procedure in the case of detecting the line of sight using a three-dimensional eyeball model. FIG. 3 is a schematic diagram showing an example of the relationship between the two-dimensional input image and the three-dimensional eyeball model. FIG. 4 is a flowchart showing an outline of a processing procedure for automatically calibrating the center position of the eyeball. FIG. 5 is a flowchart showing each of the two types of processing procedures for calculating the face front degree. FIG. 6 is a front view showing a specific example of the positional relationship between the face image and the detected feature point cloud. FIG. 7 is a front view showing a specific example of the positional relationship between the face image and the center of the face. FIG. 8 is a front view showing a specific example of the relationship between the center line of the face and the positions of the left and right irises in the face image. FIG. 9 is a plan view showing an example of the positional relationship between the center of the eyeball, the center of the iris, and the viewing angle of the three-dimensional eyeball model. FIG. 10 is a front view showing an example of the positional relationship between the center line and the left and right iris centers when the center of the face and the center of the image are equal to each other. FIG. 11 is a plan view showing an example of the viewing angle, the distance, and the positional relationship between the left and right iris centers when the center of the face and the center of the image are equal. FIG. 12 is a front view showing an example of the positional relationship between the center line and the left and right iris centers when the center of the face and the center of the image are not the same. FIG. 13 is a plan view showing an example of the viewing angle, the distance, and the positional relationship between the left and right iris centers when the center of the face and the center of the image are not the same. FIG. 14 is a schematic diagram showing an example of the relationship between the correction value of the eyeball center position and the three-dimensional eyeball model and the input image. FIG. 15 is a flowchart showing a specific example of processing for reflecting the correction value of the eyeball center position in the operation of the vehicle-mounted device.

A specific embodiment of the vehicle-mounted device of the present invention will be described below with reference to each figure.

<Specific example of environment to which the present invention is applied>
FIG. 1 shows a specific example of the arrangement state of the on-board unit according to the embodiment of the present invention.
In the example shown in FIG. 1, the camera 12 and the processing unit 13 are installed in the vehicle interior 10 of the vehicle. Specifically, in the meter unit in front of the driver's seat or at the column cover so that the person 11 seated in the driver's seat, that is, the face including the driver's eye area 11a can be photographed by the camera 12. A camera 12 is installed.

The processing unit 13 processes the two-dimensional image data obtained by photographing the face of the person 11 by the camera 12, and estimates the information indicating the direction of the line of sight 11b mainly from the data of the eye region 11a. The information of the line of sight 11b estimated by the processing unit 13 can be used as a typical example for the in-vehicle device to recognize whether or not the driver who drives the vehicle is moving the line of sight for safety confirmation. Of course, such line-of-sight information can be used for various purposes. Therefore, it is assumed that the on-board unit of the present invention is configured as, for example, the processing unit 13 in FIG. 1 or a part thereof.

The on-board unit of the present embodiment is equipped with a function of automatically calibrating the center position of the eyeball based on the two-dimensional image data captured by the camera 12, as will be described later. Therefore, in order to calibrate the line-of-sight detection function of the on-board unit, it is not necessary to prepare special calibration equipment other than the configuration shown in FIG. Further, the person 11 does not need to perform a special adjustment operation for calibration.

<Overview of line-of-sight detection algorithm>
FIG. 2 shows an outline of the processing procedure in the case of detecting the line of sight based on the face image using the three-dimensional eyeball model. That is, when the computer of the on-board unit executes a predetermined program, the operations according to the line-of-sight detection algorithm shown in FIG. 2 are sequentially performed.

The camera 12 constantly captures an image of a region including the face of the person 11 at a constant cycle, and outputs an image signal. The computer of the on-board unit captures one frame of the video signal from the camera 12 in step S11. Further, in reality, "preparation" such as conversion of the image data format including grayscale conversion is performed in S11. For example, 8-bit data representing the brightness in gradations in the range of "0 to 255" for each pixel position in one frame is displayed in the vertical direction (y) and the horizontal direction (y) according to the scanning direction in the frame at the time of shooting. Generate image data of a two-dimensional (2D) array arranged in x).

In step S12, the computer processes the two-dimensional image data of one frame or a plurality of frames input in S11 to detect the face of the person 11. As a specific example, face detection is performed using the "Viola-Jones method", and a region including a face is extracted as a rectangular region from one frame of two-dimensional image data. That is, the facial region is extracted using a detector created by learning using "Boosting", which is characterized by the difference in the shading of the face. Of course, the face may be detected by using a method originally developed. The technology of the "Viola-Jones method" is shown in the following documents.
"P. viola and MJ Jones," Rapid Object Detection Using a Boosted Cascade of Simple Features, "IEEE CVPR (2001)."

In the next step S13, the data in the rectangular region of the face extracted in S12 is processed to detect each of the left and right black eyes (iris). For example, black eyes can be detected by using template matching, which is a known technique. That is, reference data of a template representing a feature of black eyes such as a black circular pattern is prepared in advance, and this template is sequentially scanned on the image to be processed to specify a position where the features match. Before executing the detection of the black eye, the positions of various feature points or the shadow difference are detected for the entire face, and the positions of the respective regions of the left and right eyes are grasped. As a result, when detecting the left and right black eyes, the range of data to be searched can be limited, so that the processing time can be shortened. For example, the eye region can be detected by using the "Viola-Jones method".

Since the present embodiment assumes a case where the line of sight is detected by a model-based method using a three-dimensional eyeball model, it is necessary to specify the eyeball center position as a reference point for each of the left and right eyeballs. The computer estimates the center position of the eyeball in step S14. For this process, it is assumed that a method of determining the center of the eyeball based on the positions of the outer and inner corners of the eye and a method of estimating the skeleton from the feature points of the face and simultaneously calculating the center of the eyeball are used. However, in the embodiment of the present invention, the center position of the eyeball is specified by using a special algorithm as described later.

In step S15, the computer determines the state (position, angle) of the three-dimensional eyeball model based on the detected iris position and eyeball center position, and calculates the line-of-sight angle based on the result.

In step S16, the computer performs processing such as data averaging and noise removal on the line-of-sight angle obtained in step S15 as necessary, and uses the information on the line-of-sight direction in an application program or other device. Data representing the line of sight is transmitted to the in-vehicle device.

<Explanation of 3D eyeball model>
An example of the relationship between the two-dimensional input image and the three-dimensional eyeball model is shown in FIG.
In the three-dimensional eyeball model 30 shown in FIG. 3, this eyeball is a sphere represented by an eyeball center 31 and a radius R. The black eye (iris) 32 is regarded as a disk-shaped region having a radius of Di attached to the surface of the eyeball of the three-dimensional eyeball model 30. The center 32a of the black eye (iris) 32 corresponds to the central part of the pupil. The dimensions of the radius R of the eyeball and the radius Di of the black eye can be determined based on the dimensional data of the human body and various parameters of the photographing system of the camera 12.

In this three-dimensional eyeball model 30, the direction of the line of sight can be specified as the direction from the center of the eyeball 31 toward the center of the black eye 32, the rotation angle yaw with respect to the reference direction in the horizontal plane, and the rotation angle with respect to the reference direction in the vertical direction. It can be expressed by pitch. Further, the coordinates of the iris center 32a can be represented by the eyeball radius R, yaw, and pitch when the eyeball center 31 is used as a reference.

On the other hand, the image captured by the camera 12 represents a specific two-dimensional plane. Therefore, for example, when the two-dimensional image 35 of the eye region shown in FIG. 3 is input as the input image, it is necessary to perform two-dimensional / three-dimensional mutual conversion in order to use the three-dimensional eyeball model 30. ..

Specifically, the two-dimensional coordinates of the position of the black eye center 32a and the position of the eyeball center 31 are detected in the two-dimensional image 35 of the eye region, and these two-dimensional coordinates are used as the third order of the three-dimensional eyeball model 30. Project on the original coordinates. This makes it possible to calculate the vector of the line of sight 11b. The relationship between the vector of the line of sight 11b and the rotation angles yaw and pitch in each direction is expressed by the following equation.

X = -R x cos (pitch) x sin (yaw) ... (1)
Y = R × sin (pitch) ・ ・ ・ (2)
X: Distance in the x direction from the center of the eyeball 31 on the two-dimensional image plane Y: Distance in the y direction from the center of the eyeball 31 on the two-dimensional image plane

<Explanation of characteristic processing>
<Outline of processing>
FIG. 4 shows an outline of a processing procedure for automatically calibrating the position of the eyeball center 31 in the three-dimensional eyeball model 30. For example, the position of the eyeball center 31 is automatically calibrated by the computer inside the processing unit 13 executing the process shown in FIG. 4 as step S14 shown in FIG. As a result, the position accuracy of the eyeball center 31 is improved, so that the line-of-sight detection accuracy is also improved. There is no need for a user such as a driver to perform a special operation for calibration.

In step S101 of FIG. 4, the computer calculates the frontal degree of the face of the person 11 shown in the input image, that is, "the degree to which the face is facing the front". Then, if it can be considered that "the face is facing the front" based on the calculation result of S101, the process proceeds from steps S102 to S103.

In step S103, the computer calculates the degree of frontality of the line of sight of the person 11 shown in the input image, that is, "the degree to which the direction of the line of sight is facing the front". Then, if it can be considered that "the direction of the line of sight is facing the front" based on the calculation result of S103, the process proceeds from step S104 to S105.

In step S105, the latest eyeball center position is calculated based on the current characteristics of the person 11 shown in the input image, and the correction value required for updating the eyeball center position is obtained. In the next step S106, this correction value is reflected in the coordinates of the eyeball center 31 of the three-dimensional eyeball model 30 to perform calibration.

<Details of calculation of "face front degree">
The "face front degree" represents "the degree to which the face faces the front" on the input image. Each of the two types of processing procedures for calculating the "face front degree" is shown in FIG. One of these processing procedures is executed in step S101 shown in FIG. Each of the "face front degree calculation 1" and the "face front degree calculation 2" shown in FIG. 5 will be described below.

<"Face front degree calculation 1">
FIG. 6 shows a specific example of the positional relationship between the face image and the detected feature point cloud 61.
In step S21 of FIG. 5, the image data F of the face region detected in S12 of FIG. 2 is input as a processing target. In step S22, the computer performs predetermined image processing on the image data F of the face region to detect the feature points. As a result, as shown in FIG. 6, for example, a feature point group 61 representing contours of each part such as eyebrows, left and right eyes, nose, lips, and chin is detected.

In the next step S23, the computer calculates the position of the center of gravity of these facial feature point groups by the following equation (3) based on the feature point group 61 obtained in step S22. When executing this calculation, it is premised that the condition that "the feature point group 61 is set symmetrically with respect to the center of the face" is satisfied.

ｇ（ｘ，ｙ）：座標（ｘ，ｙ）で表される重心位置
Ｐ：顔特徴点の総数
ｆ（ｐ）：ｐ番目の顔特徴点の位置情報

g (x, y): Center of gravity position represented by coordinates (x, y) P: Total number of facial feature points f (p): Position information of p-th facial feature point

Therefore, the x-coordinate of the center of gravity position g (x, y) calculated in step S23 is affected by the orientation of the face in the left-right direction. That is, when the face is facing forward (the direction of the face is 0), the x-coordinate "g.x" of the center of gravity position g (x, y) is the x-coordinate of the center of the face (facecen.x, facecen.y). If the face is not facing forward, the center of gravity position g (x, y) is deviated from the center of the face.

The x-coordinate value (facecen.x) and the y-coordinate value (facecen.y) representing the center position of the face are the median value in the horizontal (x) direction and the vertical (y) in the image data F of the face region, respectively. ) Can be calculated as the median of the direction.
Therefore, the face front degree Dx can be calculated by the following equation.

Ｄｘ：顔正面度（ｘ方向）
ｆａｃｅｃｅｎ．ｘ：顔の中心のｘ座標
ｇ．ｘ：重心位置のｘ座標
αｘ：顔向きが正面か否かを識別するための閾値

Dx: Face front (x direction)
facecen. x: x coordinate of the center of the face g. x: x-coordinate of the position of the center of gravity αx: Threshold value for identifying whether or not the face is facing the front

When the face front degree Dx calculated by the above equation (4-1) is used, it is identified whether or not the face orientation is front by comparing the face front degree Dx with the threshold value αx in step S102 of FIG. can. Regarding the magnitude of the threshold value αx, it is assumed that an optimum value is adopted in consideration of parameters representing the shooting conditions of the camera 12 and individual differences.

It is possible to consider not only the horizontal orientation but also the vertical orientation for the face front degree. However, since the structure of the human face is not symmetrical in the vertical direction, the y-coordinate g. It is necessary to obtain the amount of deviation dy between y and the y coordinate "facecen.y" of the center position of the face. The dy value is determined by the arrangement of the feature points of the detector, and is obtained as a fixed value. The face front degree Dy in the vertical direction is as shown in the following equation (4-2).

<"Face front degree calculation 2">
FIG. 7 shows a specific example of the positional relationship between the face image and the center of the face (facecen.x, facecen.y).

In step S31 of FIG. 5, the image data F of the face region detected in S12 of FIG. 2 is input as a processing target.

In step S32 of FIG. 5, the image data F of the face region input in S31 is processed, and the computer creates the inverted image data FL in which the left and right sides are inverted with respect to the center (facecen.x) of the x-axis in the face image. .. Then, in the next step S33, the computer calculates the correlation value between the image data F and the inverted image data FL as the face front degree D. The face front degree D in this case is calculated by the following formula.

Ｄ：顔正面度
Ｎ：顔画像のｘ方向画素数
Ｍ：顔画像のｙ方向画素数
Ｆ（ｉ，ｊ）：画像データＦの座標（ｉ，ｊ）の画素の輝度値（階調）
ＦＬ（ｉ，ｊ）：反転画像データＦＬの座標（ｉ，ｊ）の画素の輝度値（階調）
α：顔向きが正面か否かを識別するための閾値

D: Face front degree N: Number of pixels in x-direction of face image M: Number of pixels in y-direction of face image F (i, j): Luminance value (gradation) of pixels at coordinates (i, j) of image data F
FL (i, j): Luminance value (gradation) of pixels at coordinates (i, j) of inverted image data FL
α: Threshold for identifying whether the face is facing the front

That is, assuming that the facial features are symmetrical with respect to the center of the face (facecen.x), the features on the left side from the center of the face and the features on the right side are equal. Therefore, the face front degree D, which represents the reciprocal of the correlation between the image data F and the inverted image data FL, becomes very small when the face orientation is close to the front, and increases when the face orientation deviates from the front.

When the face front degree D calculated by the above equation (5) is used, it is possible to identify whether or not the face orientation is front by comparing the face front degree D with the threshold value α in step S102 of FIG. Regarding the magnitude of the threshold value α, it is assumed that an optimum value is adopted in consideration of parameters representing the shooting conditions of the camera 12 and individual differences.

<Details of calculation of "line-of-sight front">
FIG. 8 shows a specific example of the relationship between the center line of the face and the positions of the left and right irises in the face image.
The line-of-sight front degree FD is calculated from the degree to which the line-of-sight direction of the person 11 shown in the input image is facing the front, that is, the rotation angle obtained by applying the three-dimensional eyeball model 30 to each of the left and right eyes. Indicates the degree to which the line-of-sight direction θ (yaw, pitch) is 0 degrees.

In the present embodiment, on the premise that the condition that "the direction of the face is the front" is satisfied as shown in FIG. 4, the computer calculates the line-of-sight front degree FD by the following formula.

ＦＤ：視線正面度
ｉｒｉｓＲ．ｘ：右側の黒目の中心位置のｘ座標
ｉｒｉｓＬ．ｘ：左側の黒目の中心位置のｘ座標
ｆａｃｅｃｅｎ．ｘ：顔の中心のｘ座標
β：視線の向きが正面か否かを識別するための閾値

FD: Line-of-sight front degree irisR. x: x-coordinate of the center position of the right black eye irisL. x: x-coordinate facecen. Of the center position of the left black eye. x: x-coordinate β of the center of the face: threshold value for identifying whether or not the direction of the line of sight is front

In the present embodiment, the positional relationship between the left and right eyes of both eyes and the like is described as not representing the left and right with respect to the person 11, but the left and right as seen from the camera 12 side and the left and right on the two-dimensional image obtained by photographing. ..

That is, the line-of-sight frontality FD represented by the above equation (6) represents the difference between the center position between the left and right iris centers and the position of the center line of the face (facecen.x) in the x-axis direction. ing. Therefore, for example, if the distance from the position of the center line (facecen.x) to the center position of the black eye on the right side and the distance to the center position of the black eye on the left side are equidistant, the direction of the line of sight is facing the front. Can be determined. Further, if it can be determined that the direction of the line of sight is facing the front based on the positional relationship in the x-axis direction, it can be considered that the direction of the line of sight is also facing the front in the y-axis direction at the same time.

Further, the line-of-sight front degree FD becomes 0 when the line-of-sight direction is facing the front, and increases as the line-of-sight direction deviates from the front. Therefore, when the line-of-sight front degree FD calculated by the above equation (6) is used, whether or not the line-of-sight direction is front is determined by comparing the line-of-sight front degree FD with the threshold value β in step S104 of FIG. Can be identified. Regarding the magnitude of the threshold value β, it is assumed that an optimum value is adopted in consideration of parameters representing the shooting conditions of the camera 12 and individual differences.

<Calculation of eyeball center position>
The process for calculating the latest eyeball center position in step S105 shown in FIG. 4 will be described below.

<When the center of the face in the x-axis direction is equal to the center of the image>
FIG. 9 shows an example of the positional relationship between the center of the eyeball, the center of the iris, and the visual angle in the horizontal plane (or the plane parallel to the x and z axes) of the three-dimensional eyeball model. Further, FIG. 10 shows an example of the positional relationship between the center line and the left and right iris centers when the center of the face in the x-axis direction and the center of the image are equal to each other. Further, FIG. 11 shows an example of the visual angle, the distance, and the positional relationship between the left and right iris centers in the horizontal plane when the center of the face in the x-axis direction is equal to the center of the image.

Since the calculation can be performed for each of the left and right eyes in almost the same way, the following explanation assumes that only the right eye is processed as the calculation target.
For example, in the state shown in FIG. 9, the relationship of the following equation (7) is established.

Xeyer = irisR. x-Ro ・ sinθRyaw ・ ・ ・ (7)
Xeyer: x-coordinate of the center of the right eyeball irisR. x: x-coordinate of the center of the right iris Ro: radius of the eyeball (for example, a constant)
θRyaw: Line-of-sight angle of the right eye (yaw direction)

That is, if the position "irisR.x" at the center of the iris and the line-of-sight angle "θRyaw" are known, the x-coordinate "Xeyer" at the center of the eyeball can be calculated from the above equation (7).

Further, as shown in FIG. 10, the x-coordinate “facecen.x” of the face center position in the image and the x-coordinate “Icenter.x” of the center of the image taken by the camera are equal, and as shown in FIG. Assuming that it is a situation, the line-of-sight angle "θRyaw" can be calculated by the following equation (8).
tanθRyaw = (irisR.x-Icenter.x) / Df ... (8)
Df: Distance between the camera and the face

The distance Df can be treated as known information if it is measured using, for example, an appropriate distance sensor. Further, for example, if the technique disclosed in "Japanese Unexamined Patent Publication No. 2015-206764" is used, the distance sensor becomes unnecessary. Further, since the iris can be detected from the face image as described above, the position "irisR.x" at the center of the iris can also be used as known information. Further, as the x-coordinate "Icenter.x" of the center of the image, for example, a predetermined constant can be used.

Therefore, the line-of-sight angle "θRyaw" can be calculated from the above equation (8). Further, by substituting the line-of-sight angle "θRyaw", the position of the center of the iris "irisR.x", and the radius Ro of the eyeball into the above equation (7), the x-coordinate Xeyer of the center of the right eyeball can be calculated.

Further, the y-coordinate Yeyer at the center of the right eyeball can be similarly calculated by using the following equations (9) and (10).
Yeyer = irisR. y-Ro ・ sinθRpitch ・ ・ ・ (9)
Yeyer: y-coordinate of the center of the right eyeball irisR. y: y coordinate of the center of the right iris Ro: radius of the eyeball (for example, a constant)
θRpitch: Line-of-sight angle of the right eye (pitch direction)

tanθRpitch = (irisR.y-Center.y) / Df ... (10)
Icenter. y: y coordinate of the center of the image Df: Distance between the camera and the face

<When the center of the face in the x-axis direction and the center of the image are out of alignment>
FIG. 12 shows an example of the positional relationship between the center line and the left and right iris centers when the center of the face and the center of the image are not the same. Further, FIG. 13 shows an example of the visual angle, the distance, and the positional relationship in the horizontal plane (the plane parallel to the x and z axes) of the left and right iris centers when the center of the face and the center of the image are not the same.

For example, even if the x-coordinate "facecen.x" of the center line of the face deviates from the x-coordinate "Icenter.x" of the center of the image as shown in FIG. 12, the face is shown in FIG. Is centered on the camera 12 and exists on the circumference whose radius is represented by the distance Df. Further, the center position "facecen.x" of the face and the center "Icenter.x" of the image are deviated by the face rotation angle "facead". However, if the face is facing the front of the camera 12 and the lines of sight of both eyes are facing the camera 12, the line-of-sight angle "θRyaw" of the right eye is as shown in FIG. It can be expressed as the rotation angle of the eye position with respect to the center "Icenter.x".

Therefore, even when the center of the face in the x-axis direction and the center of the image are deviated from each other, the line-of-sight angle “θRyaw” can be calculated using the above equation (8). Further, the x-coordinate "Xeyer" of the center of the eyeball can be calculated based on the line-of-sight angle "θRyaw" and the above equation (7).

On the other hand, even when calculating the coordinates of the center of the eyeball for the left eye, it can be calculated in the same manner as above by using each of the following calculation formulas.
Xeyel = irisL. x-Ro ・ sinθLyaw ・ ・ ・ (11)
Xeyel: x-coordinate of the center of the left eyeball irisL. x: x-coordinate of the center of the left iris Ro: radius of the eyeball (for example, a constant)
θLyaw: Line-of-sight angle of the left eye (yaw direction)
tanθLyaw = (irisL.x-Icenter.x) / Df ... (12)
Df: Distance between the camera and the face

Yeyel = irisL. y-Ro ・ sinθLpitch ・ ・ ・ (13)
Yeyel: y-coordinate of the center of the left eyeball irisL. y: y coordinate of the center of the left iris Ro: radius of the eyeball (for example, a constant)
θLpitch: Line-of-sight angle of the left eye (pitch direction)
tanθLpitch = (irisL.y-Icenter.y) / Df ... (14)
Icenter. y: y coordinate of the center of the image Df: Distance between the camera and the face

<Correction of eyeball center position>
Even if there is an error in the position coordinates of the eyeball center 31 in the three-dimensional eyeball model 30, it is possible to calibrate so that the error is reduced by correcting using the latest eyeball center position obtained by the above calculation. can. The correction value can be calculated by, for example, the following formula.

dXeye = Xeye-Xeyemodel ・ ・ ・ (15)
dYee = YeYe-Yeemodel ... (16)
dXeye: Correction value of x coordinate (right or left)
dYee: Correction value of y coordinate (right or left)
Xeye: x-coordinate of the latest calculated eyeball center position (right or left)
YeYe: y-coordinate of the latest calculated eyeball center position (right or left)
Xeyemodel: Eye center x coordinate (right or left) before correction in 3D eye model
Eyemodel: Y-coordinate of eye center before correction in 3D eye model (right or left)

<Eye center position correction 1>
FIG. 14 shows an example of the relationship between the correction value dXeye of the eyeball center position and the three-dimensional eyeball model and the input image.

In step S105 shown in FIG. 4, since the face front degree D is equal to or less than the threshold value α and the line-of-sight front degree FD is equal to or less than the threshold value β, the latest coordinates (Xeye, Yeye) of the eyeball center position are set to the left and right eyes. Calculate for each of.

Then, the correction values dXeye and dYee are calculated for each of the left and right eyes by the above equations (15) and (16). Further, as shown in FIG. 14, the position coordinates of the eyeball center 31 in the three-dimensional eyeball model 30 are updated for each of the left and right eyes using the correction values dXeye and dYey so that the error becomes small.

<Eye center position correction 2>
FIG. 15 shows a specific example of the process for reflecting the correction value of the eyeball center position in the operation of the on-board unit.
When the process shown in FIG. 4 is applied to the vehicle-mounted device, step S105 is repeatedly executed every time a state in which the person 11 faces the front and the line of sight also faces the front is detected. Therefore, the update of the eyeball center position can be repeated using the new correction value between the time when the power is turned on to the on-board unit and the time when the operation is completed.

Therefore, in order to perform a more appropriate update process, for example, the computer executes the process of "eyeball center position correction 2" shown in FIG. Although only the coordinates in the x-axis direction are processed in FIG. 15, the coordinates in the y-axis direction can be processed in the same manner. By the process shown in FIG. 15, the coordinates of the center position of the eyeball can be converged to a constant value.

In step S41 of FIG. 15, the computer calculates the correction value dXeye of the eyeball center position based on the above equations (7), (8), and (15). Further, in the next step S42, the correction value dXeye is compared with a predetermined threshold value in order to eliminate obvious outliers that may be caused by the influence of noise.

If the correction value dXeye is not an obvious outlier, 1 is added to the update count value N in step S43 (the initial value of N is 0). Further, in the next step S44, the data of the correction value dXeye this time is saved in the memory position associated with the update count value N.

In step S45, the computer reads the data of N correction values dXeye stored in the memory, averages them, and outputs dXeye_T. As a specific method of averaging, a calculation method such as a simple average value, a weighted average value, or a moving average value can be applied.

By applying the processing as shown in FIG. 15, it is possible to exclude obvious outliers due to the influence of noise from the update target. In addition, abrupt changes in the center position of the eyeball can be avoided by averaging. That is, it becomes robust against noise. Therefore, in an environment where an operation targeting the same user is performed for a long time, such as an in-vehicle device that monitors the driver's line of sight, high eye line detection accuracy can be expected by updating the effective eyeball center as needed. ..

<Advantages of on-board equipment>
In the above-mentioned on-board unit, the position of the eyeball center 31 can be automatically calibrated by executing the process shown in FIG. Moreover, the user does not need to perform a special operation for performing this calibration, and there is no need to prepare special calibration equipment. Further, by repeatedly executing the process shown in FIG. 15, the influence of noise can be eliminated and a sudden change can be prevented, so that the malfunction of the on-board unit can be avoided.

Therefore, the above-mentioned technology can be used for the driver's line-of-sight detection, side-view detection, etc. using the in-vehicle camera. It can also be used, for example, for visually confirming digital signage (electronic signboard).

Here, the features of the above-described embodiment of the vehicle-mounted device according to the present invention are briefly summarized and listed below in [1] to [4], respectively.
[1] From a face image taken so as to include a face, the center of the eyeball (eyeball center 31) in the three-dimensional eyeball model (three-dimensional eyeball model 30) in which the eyeball included in the face image is modeled is located. An in-vehicle device including a data processing unit (processing unit 13) for calculating coordinates (S105).
[2] The data processing unit is
The center of the black eye included in the face image is located on the three-dimensional eyeball model, and the center position of the black eye is calculated.
The line-of-sight angle is calculated based on the distance between the center position of the iris and the center line in the face image.
Based on the black eye center position and the line-of-sight angle, the eyeball center coordinates in the three-dimensional eyeball model are calculated (see equation (7)).
The in-vehicle device according to the above [1].
[3] The data processing unit is
It is determined whether or not the face included in the face image is facing the front (S102).
It is determined whether or not the line of sight based on the position of the iris included in the face image is facing the front (S104).
When it is determined that the face and the line of sight are facing the front, the eyeball center coordinates in the three-dimensional eyeball model are calculated (S105).
The in-vehicle device according to the above [1] or [2].
[4] The data processing unit is
The calculation process of the eyeball center coordinates is repeated, the calculation results are saved, and the result of averaging the plurality of calculation results is output (S45).
The vehicle-mounted device according to any one of the above [1] to [3].

10 Inside the vehicle 11 People 11a Eye area 11b Line of sight 12 Camera 13 Processing unit 30 3D eye model 31 Eye center 32 Black eye (iris)
32a center of iris (pupil)
Two-dimensional image of the 35th area D Face front degree

Claims (3)

It is provided with a data processing unit that calculates the eyeball center coordinates where the center is located in the three-dimensional eyeball model in which the eyeball included in the face image is modeled from the face image taken so as to include the face .
The data processing unit
The center of the black eye included in the face image is located on the three-dimensional eyeball model, and the center position of the black eye is calculated.
The line-of-sight angle is calculated based on the distance between the center position of the black eye and the center line passing through the center of the image in the face image, and the distance between the camera that captures the face image and the face.
An in-vehicle device that calculates the eyeball center coordinates in the three-dimensional eyeball model based on the black eye center position and the line-of-sight angle. The data processing unit
It is determined whether or not the face included in the face image is facing the front, and it is determined.
It is determined whether or not the line of sight based on the position of the iris included in the face image is facing the front.
The vehicle-mounted device according to claim 1, wherein the eyeball center coordinates in the three-dimensional eyeball model are calculated when it is determined that the face and the line of sight are facing the front. The data processing unit
The vehicle-mounted device according to claim 1 or 2 , wherein the calculation process of the eyeball center coordinates is repeated, the calculation results are saved, and the result of averaging a plurality of calculation results is output.

Images