JP2023125652A - Pupil detection device and pupil detection method - Google Patents

Abstract

To enhance calculation efficiency and detection accuracy with a simple device configuration.SOLUTION: A sight line detection device 1 comprises: a camera 10 which acquires face images at continuous timing by imaging the face of an object person A; a light source 13 which emits light toward the face of the object person A; and an image processing device 20 which processes the face image acquired by the camera 10 at the irradiation timing of light. The image processing device 20 comprises: a sight line detection unit 23 which detects a position of the pupil of the object person A on the face image; and a pupil position prediction unit 25 which predicts the position of the pupil during eye closure of the object person A by searching for a position of a feature part of the eye in an eye closure state on the face image. The sight line detection unit 23 detects the position of the pupil by tracking the pupil by utilizing the position of the pupil predicted by the pupil position prediction unit 25.SELECTED DRAWING: Figure 4

Description

The present invention relates to a pupil detection device and a pupil detection method for detecting the position of a pupil from an image of a subject.

BACKGROUND ART Conventionally, devices have been used that acquire an image of a subject's face including the eyes and detect the position of the subject's pupils based on the image (for example, see Patent Document 1 below). This device detects the position of the pupil based on a difference image obtained by the difference between a bright pupil image in which the pupil is brightly depicted and a dark pupil image in which the pupil is darkly depicted. At that time, in order to improve calculation efficiency and accuracy, a window is set from the position of the pupil detected on the image of the previous frame to the part predicted to be the position of the pupil on the image of the next frame, and the window is Exploring the pupils inside.

JP2007-268026A

In the device described in Patent Document 1 mentioned above, when a subject closes his or her eyes by blinking or the like (hereinafter also referred to as "closed eyes"), the position of the window set on the image differs from the actual position of the pupil. There was a tendency for the pupil to become undetectable. Although such a situation can be avoided by setting a large window, the calculation efficiency will decrease.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a pupil detection device and a pupil detection method that can improve calculation efficiency and detection accuracy.

In order to solve the above problems, a pupil detection device according to one embodiment of the present invention includes a camera that acquires facial images at consecutive timings by capturing an image of a target person's face, and a camera that irradiates light toward the target person's face. The computing device includes a pupil position detection unit that detects the position of the subject's pupil on the facial image, and a computing device that processes the facial image acquired by the camera at the timing of light irradiation. and a pupil position prediction unit that predicts the position of the pupil of the subject when the eyes are closed by searching for the position of the facial features in the eye closed state, and the pupil position detection unit predicts the position of the pupil of the subject when the eyes are closed. The position of the pupil is detected by tracking the pupil using the predicted pupil position.

Further, a pupil detection method according to another aspect of the present invention includes: a camera that captures facial images of a target person at consecutive timings by imaging the target person's face; a light source that irradiates light toward the target person's face; A pupil detection method using a computing device that processes a face image acquired by a camera at the timing of light irradiation, the computing device detecting a pupil position of a subject on the face image; , a pupil position prediction step in which the arithmetic device predicts the position of the subject's pupil when the subject's eyes are closed by searching for the position of facial features in the eye closed state on the face image, and a pupil position detection step Then, the pupil position is detected by tracking the pupil using the pupil position predicted in the pupil position prediction step.

According to the pupil detection device of one form described above or the pupil detection method of another form described above, the position of the pupil is detected on face images acquired at consecutive timings by a camera, and the position of the pupil is detected from the face image in a state where the eyes are closed. By searching the position of the characteristic part of the face, the position of the pupil when the eyes are closed can be predicted on the face image. When detecting the position of the pupil, the pupil is tracked using the position of the pupil predicted on the face image. Thereby, even when the subject's eyes are closed, the position of the pupil can be detected by tracking the pupil, and highly accurate pupil detection processing can be achieved with high computational efficiency.

Here, the facial feature may be a feature of the subject's eyes. In this case, by searching for the position of the eye feature in the face image and predicting the pupil position from that position, it is possible to improve the prediction accuracy of the pupil position and achieve more accurate pupil detection processing. be done.

Furthermore, the pupil position prediction unit may predict the position of the pupil using a machine learning model using a neural network. In this case, the accuracy of predicting the position of the pupil on the face image can be reliably improved by a simple learning method, and stable pupil detection processing can be achieved.

In addition, the pupil position prediction unit inputs a partial image extracted from the facial image as input data to a machine learning model, and uses the machine learning model to search for the position of the partial image containing the facial features. It can also be used to predict the position of the pupil. In this case, by using a partial image cut out from the face image as input data, the position of the pupil on the face image can be predicted through simple processing. As a result, the calculation efficiency of the pupil detection process can be further improved.

Furthermore, the pupil position prediction unit may cut out the partial image in a size that includes the entire facial feature. In this case, the entire feature of the face can be contained in a partial image cut out from the face image, and the accuracy of predicting the position of the pupil can be improved. As a result, more stable pupil detection processing is realized.

The arithmetic device further includes a model learning unit that learns a machine learning model, and the model learning unit stores a face image obtained when the subject's eyes are closed, which is detected by the pupil position detection unit immediately before the subject's eyes are closed. It is also possible to cut out the face image based on the position of the pupil and use the cut out face image as training data to train the machine learning model. In this way, it is possible to create appropriate training data, and by performing learning using the training data, it is possible to improve the prediction accuracy of the pupil position. As a result, more stable pupil detection processing is realized.

The arithmetic device further includes a pupil distance detection unit that detects the distance from the camera to the pupil, and the pupil position prediction unit changes the size of the cutout of the partial image as input data according to the distance of the pupil. It can also be used to set. In this case, the size of the partial image used as input data can be appropriately set, and the accuracy of predicting the pupil position can be improved. As a result, more stable pupil detection processing is realized.

The computing device further includes a pupil distance detection unit that detects the distance from the camera to the pupil, and the model learning unit variably sets the size of the cutout of the face image as training data according to the distance of the pupil. It can also be said to do. In this way, the size of the cutout of the face image used as training data can be appropriately set, and the accuracy of predicting the position of the pupil can be improved. As a result, more stable pupil detection processing is realized.

Furthermore, the model learning unit may identify a region in which facial features exist from the extracted facial image, and set the size of the facial image to be extracted as training data based on the identified region. . In this way, the size of the cutout of the face image used as training data can be appropriately set according to the region of the facial feature, and the accuracy of predicting the position of the pupil can be improved. As a result, more stable pupil detection processing is realized.

In addition, the model learning unit identifies regions in which facial features are present in the extracted facial images, and sets the location for cropping the facial images as training data so that the identified regions are located in the center of the image. It can also be said to do. In this way, the position of cutting out the face image used as training data can be appropriately set according to the region of the facial feature, and the accuracy of predicting the position of the pupil can be improved. As a result, more stable pupil detection processing is realized.

In addition, the machine learning model predicts the amount of shift in the position of a partial image containing facial features with respect to a face image extracted based on the position of the pupil detected by the pupil position detection unit just before the subject's eyes are closed. The model learning unit may generate a shifted image while shifting the extracted face image, and use the shifted image and the amount of shift of the shifted image as training data to learn the learning model. In this case, the accuracy of predicting the position of the pupil on the face image can be improved regardless of the amount of deviation on the face image between the plurality of partial images used as input data. As a result, more stable pupil detection processing is realized.

Furthermore, when the subject closes their eyes and then opens them again, the pupil position detection unit sets a window on the face image based on the pupil position predicted by the pupil position prediction unit at the timing when the subject closes their eyes. The position of the pupil may be detected by this method. Conventionally, after the eyes are closed and then opened, the pupils tend to be undetectable for several frames of images. According to such a configuration, pupil detection can be stably resumed without delay from the frame immediately after the eye is opened.

In addition, the pupil position detection unit detects the pupil position by tracking the pupil position using the pupil position detected on the face image of consecutive frames, and detects the pupil position on the face image of the immediately previous frame. If the detection of the position of the pupil fails, the pupil position may be tracked using the pupil position predicted by the pupil position prediction unit. According to this configuration, when continuously detecting the position of the pupil, even if the subject closes his eyes due to blinking or the like and the image of the pupil no longer appears on the image, the position of the pupil can be stably tracked; When the subject opens his or her eyes and the image of the pupil appears on the image again, the position of the pupil can be stably detected.

According to the present invention, the calculation efficiency and detection accuracy of pupil detection processing can be improved with a simple device configuration.

FIG. 1 is a perspective view showing a line of sight detection device according to an embodiment. FIG. 3 is a plan view showing a lens portion of the camera. FIG. 1 is a diagram showing a hardware configuration of an image processing device according to an embodiment. FIG. 1 is a block diagram showing a functional configuration of a line of sight detection device according to an embodiment. 5 is a diagram showing an image of a partial image created by the partial image creation unit 24 of FIG. 4. FIG. FIG. 2 is a diagram showing a layer structure of the machine learning model 1 used by the pupil position prediction unit 25. FIG. 2 is a diagram showing a layer structure of a machine learning model 2 used by a pupil position prediction unit 25. FIG. It is a figure which shows an example of closed-eye images GP _L1 and GP _R1 of the left and right pupils created by the learning image creation unit 26 for the face image G _F2 . FIG. 3 is a diagram showing an example of learning data "learning image 3" created by the learning image creating unit 26 for the face image _GF2 . 3 is a flowchart showing the operation procedure of the line of sight detection device 1. FIG.

Hereinafter, preferred embodiments of a pupil detection device and a pupil detection method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description will be omitted.

[Configuration of line of sight detection device]
First, the configuration of a line of sight detection device 1, which is a pupil detection device according to an embodiment, will be explained using FIGS. 1 to 4. The line of sight detection device 1 is a computer system that detects the target's pupil and corneal reflex by capturing an image of the target's face, and uses the detection results to detect the direction of the target's line of sight. A pupil detection method according to this embodiment is implemented. The target person is a person whose gaze direction is to be detected, and can also be called a test subject. The purposes of use of the line of sight detection device 1 and the pupil detection method are not limited in any way, and include, for example, detecting driving while looking away, checking the safety confirmation operation of the driver's side mirrors and room mirrors, detecting the driver's drowsiness, and detecting product interest. The line of sight is used to investigate the severity of the disease, input data into computers used in amusement machines, etc., diagnostic equipment for diagnosing autism in infants, communication systems used between remote locations, observation equipment to observe events remotely, etc. A detection device 1 can be used.

As schematically shown in FIG. 1, the line of sight detection device 1 includes a pair of cameras 10 that function as stereo cameras and an image processing device (arithmetic device) 20. Below, the pair of cameras 10 will be distinguished into a left camera _10L on the left side of the subject A and a right camera _10R on the right side of the subject A, as necessary. In the present embodiment, the line-of-sight detection device 1 further includes a display device 30, which is the object that the subject A looks at. However, since the purpose of use of the line-of-sight detection device 1 is not limited as described above, The object located there is not limited to the display device 30, but may also be a windshield of a car, for example. Therefore, the display device 30 is not an essential element in the line of sight detection device 1. Each camera 10 is connected to an image processing device 20 wirelessly or by wire, and various data or commands are transmitted and received between the camera 10 and the image processing device 20. Camera calibration is performed for each camera 10 in advance.

The camera 10 photographs the facial part of the subject A including the left and right eyes to obtain a facial image. A pair of cameras 10 are arranged at a predetermined interval along the horizontal direction, and are designed to face the face of the subject A for the purpose of preventing reflected light from appearing in the face image when the subject A is wearing glasses. installed at a lower position. The elevation angle of the camera 10 with respect to the horizontal direction is set, for example, in the range of 20 to 35 degrees, taking into consideration both reliable detection of the pupil and avoidance of obstruction of the visual field of the subject A. Camera calibration is performed for each camera 10 in advance.

In this embodiment, the camera 10 is a camera that can acquire multiple frames of facial images at continuous and regular timing. The camera 10 images the subject A in response to a command from the image processing device 20 and outputs a facial image to the image processing device 20.

The lens portion of the camera 10 is schematically shown in FIG. As shown in this figure, in the camera 10, an objective lens 11 is housed in a circular opening 12, and a light source 13 is attached to the outside of the opening 12. The light source 13 is a device for irradiating illumination light toward the face of the subject A, and includes a plurality of light emitting elements 13a and a plurality of light emitting elements 13b. The light emitting elements 13a are semiconductor light emitting elements (LEDs) whose output light has a center wavelength of 850 nm, and are arranged in a ring shape along the edge of the opening 12 at equal intervals. The light emitting element 13b is a semiconductor light emitting element whose output light has a center wavelength of 940 nm, and is arranged in a ring shape at equal intervals outside the light emitting element 13a. Therefore, the distance from the optical axis of the camera 10 to the light emitting element 13b is greater than the distance from the optical axis to the light emitting element 13a. Each of the light emitting elements 13a and 13b is provided so as to emit illumination light along the optical axis of the camera 10. Note that the arrangement of the light source 13 is not limited to the configuration shown in FIG. 2, and may be arranged in other ways as long as the camera can be regarded as a pinhole model. The light source 13 emits illumination light at a timing according to a command from the image processing device 20.

The image processing device 20 is a computer (arithmetic device) that controls the camera 10 and the light source 13, and detects the line of sight direction using the face image of the subject A. The image processing device 20 may be constructed using a stationary or portable personal computer (PC), a workstation, or another type of computer. Alternatively, the image processing device 20 may be constructed by combining a plurality of arbitrary types of computers. When using multiple computers, these computers are connected via a communication network such as the Internet or an intranet.

FIG. 3 shows a general hardware configuration of the image processing device 20. The image processing device 20 includes a CPU (processor) 101 that executes an operating system or an application program, a main storage section 102 made up of ROM and RAM, and an auxiliary storage section 103 made up of a hard disk or flash memory. It includes a communication control unit 104 made up of a network card or a wireless communication module, an input device 105 such as a keyboard or a mouse, and an output device 106 such as a display or a printer.

Each functional element of the image processing device 20, which will be described later, loads predetermined software onto the CPU 101 or the main storage unit 102, and operates the communication control unit 104, input device 105, output device 106, etc. under the control of the CPU 101. This is realized by reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. Data or databases required for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

As shown in FIG. 4, the image processing device 20 includes a lighting control section 21, an image acquisition section 22, a line of sight detection section (pupil position detection section, pupil distance detection section) 23, a partial image creation section 24, and a pupil distance detection section 23 as functional components. It includes a position prediction section 25, a learning image creation section 26, and a model learning section 27. The lighting control unit 21 controls the lighting timing of the light source 13. The image acquisition unit 22 is a functional element that acquires facial image data from the camera 10 at the lighting timing by controlling the lighting timing of the light source 13 to be synchronized with the imaging timing of the camera 10 . The line-of-sight detection unit 23 is a functional element that detects the direction of the visual axis (also referred to as line-of-sight) based on the line-of-sight vector obtained from the face image. The visual axis (line of sight) is a line connecting the center of a subject's pupil and the subject's point of gaze (the point at which the subject is looking). Note that the term "visual axis" includes the meanings (concepts) of starting point, ending point, and direction. In addition, the "line-of-sight vector" is a vector representing the direction of the subject's visual axis, and is one form of expressing the "direction of the visual axis." The output destination of the detection result of the image processing device 20 in the direction of the visual axis is not limited at all. For example, the image processing device 20 may display the determination result as an image, figure, or text on a monitor, store it in a storage device such as a memory or database, or transmit it to another computer system via a communication network. You can also send it.

Here, the basic operation of visual line detection by the image processing device 20 will be explained.

First, the lighting control unit 21 controls the lighting timing of the light emitting elements 13a and the light emitting elements 13b included in the light source 13 so that they are lit alternately in synchronization with the photographing timing of the camera 10. In addition, the image acquisition unit 22 acquires from each camera 10 a bright pupil image (face image) in which the pupil is relatively bright and a dark pupil image (face image) in which the pupil is relatively dark. Next, the line of sight detection unit 23 uses the difference image (or divided image) between the bright pupil image and the dark pupil image from each camera 10 to determine the position of the pupil center and the cornea in the face image of each camera 10. Detect the position of the reflection. Then, the line of sight detection unit 23 calculates the three-dimensional coordinates of the left and right pupil centers of the subject A, and the respective The distance from the camera 10 to the left and right pupils is obtained. Furthermore, the line-of-sight detection unit 23 determines the visual axis (line-of-sight vector) of the left and right eyes based on the calculated positions of the pupil center and corneal reflection in the face image of either camera 10 and the three-dimensional coordinates of the left and right pupils. calculate. Furthermore, the line of sight detection unit 23 may calculate a gaze point on a predetermined visual symmetry plane with reference to the calculated visual axis. The above processing is repeatedly performed on pairs of bright pupil images and dark pupil images that are obtained alternately.

Here, the detection of the position of the pupil center and the position of the corneal reflection for the bright pupil image and the dark pupil image by the line of sight detection unit 23 is performed using a predetermined size set in the bright pupil image and the dark pupil image. Executed on images within the window. The position of this window is based on the position of the pupil on the current frame predicted using at least the three-dimensional position of the pupil detected in the previous frame of the face image to be detected, and is set to include the position of the pupil. (For example, the method described in Japanese Unexamined Patent Publication No. 2007-268026 is adopted.) That is, the position of the window is set so as to track the position of the pupil between consecutive frames. However, if the line of sight detection unit 23 fails to detect the position of the pupil center for the face image of the previous frame, the three-dimensional position of the pupil predicted by the pupil position prediction unit 25, which will be described later, is used to The position of the window on the face image of the current frame is set. Here, the determination of failure in detection of the position of the pupil center by the line of sight detection unit 23 (determination of eye closure due to blinking, etc.) is based on the pupil detected on the difference image (or division image) between the bright pupil image and the dark pupil image. This is performed by recognizing that the pupil does not exist when the area (for example, the number of pixels in the pupil region of a binarized image using a threshold value) approaches zero. Generally, eye closure due to blinking occurs instantaneously, so the pupil suddenly becomes undetectable in a certain frame. Therefore, if the head does not move between immediately before and immediately after closing the eyes, the eyes or pupils will be in the same position immediately before and immediately after closing the eyes. Therefore, it is possible to associate the eye-closed image with the pupil position. The detection of the position of the pupil by the line of sight detection unit 23 and the prediction of the position of the pupil by the pupil position prediction unit 25 may be performed in parallel on continuously acquired facial images. In this case, when the line-of-sight detection unit 23 determines that detection in the previous frame has failed, the prediction result by the pupil position prediction unit 25 is used to set the window position in the current frame. On the other hand, if detection of the pupil position by the line-of-sight detection unit 23 fails in a certain frame, the pupil position prediction unit 25 can perform prediction processing for that frame. In this case, the line of sight detection unit 23 uses the prediction result by the pupil position prediction unit 25 to set the window position in the next frame.

Next, the functions of other components of the image processing device 20 will be explained. Note that the processing by the partial image creation unit 24, pupil position prediction unit 25, learning image creation unit 26, and model learning unit 27, which will be described below, is based on bright pupil images and It is performed separately on the dark pupil image.

The partial image creation unit 24 cuts out a partial image of a rectangular area of a predetermined size from the face image acquired from each camera 10 by the image acquisition unit 22. For example, a face image has a size of 150 pixels horizontally by 80 pixels vertically, and this partial image has a size of 30 pixels horizontally by 20 pixels vertically. The size of this partial image is set to include the entire eyelashes, which are characteristic parts of the eyes of the subject A's face. At this time, the partial image creation unit 24 uses the eye characteristics based on the distance from the camera 10 to the pupil detected or predicted by the line of sight detection unit 23 or the pupil position prediction unit 25 for the frame immediately before the face image to be processed. The cutout size of the partial image may be set variably so that the entire portion is included, and the cutout partial image may be converted into image data of a predetermined size (for example, 8 pixels x 5 pixels). Then, the partial image creation unit 24 repeatedly shifts the cutout regions of the partial images in the face image in a two-dimensional direction (for example, shifts by 10 pixels in the horizontal direction and by 5 pixels in the vertical direction), and mutually shifts the cutout regions in the horizontal direction. and create a plurality of partial images that overlap in the vertical direction.

FIG. 5 shows an image of a partial image created by the partial image creation section 24. For example, the face image G _F1 to be cropped has a size of 150 pixels horizontally x 80 pixels vertically, the cropping size of the partial image is 30 pixels horizontally x 20 pixels vertically, and the horizontal shift amount is 10 pixels, When the vertical shift amount is 5 pixels, the partial image creation unit 24 creates a total of 169 partial images GP _F1 for one frame of the face image G _F1 .

In addition, if the pupil position prediction unit 25 predicts the position of the pupil on the face image in the frame immediately before the frame to be processed, the partial image creation unit 24 creates a window of a rectangular area centered at the position. , and it functions to cut out a partial image from within that window. For example, the partial image creation unit 24 sets a window of 60 pixels horizontally and 40 pixels vertically centering on the predicted position, and shifts the window by 5 pixels vertically and horizontally from within the window to a size of 40 pixels horizontally and 20 pixels vertically. Cut out a partial image while

In addition, when the three-dimensional position of the pupil is predicted by the pupil position prediction unit 25 in the frames one and two frames before the frame to be processed, the partial image creation unit 24 determines that the pupil is The position of the pupil in the frame to be processed is predicted using a uniform velocity model under the assumption that the pupil is moving in a dimensional space. In this case, the position of the pupil is predicted by projecting the position in the three-dimensional space onto the face image, which is a two-dimensional image. Then, the partial image creation unit 24 functions to set a window of a rectangular area centered on the predicted position and cut out a partial image from within the window. For example, the partial image creation unit 24 sets a window of 60 pixels horizontally and 40 pixels vertically centering on the predicted position, and shifts the window by 5 pixels vertically and horizontally from within the window to a size of 40 pixels horizontally and 20 pixels vertically. while cutting out a partial image.

The pupil position prediction unit 25 calculates the characteristics of the eyes in the closed eye state on the face image (in this embodiment By searching for the position of the eyelashes, the position of the pupil on the face image of subject A with his eyes closed is predicted. This prediction of the pupil position is performed for each of the left and right eyes.

That is, the pupil position prediction unit 25 predicts the pupil position using a machine learning model using a CNN (convolutional neural network). First, the pupil position prediction unit 25 uses the machine learning model 1 learned in advance by the model learning unit 27 to select an image (which includes the characteristic part of the eye (eyelash part) in the eye-closed state) from among the plurality of partial images. Hereinafter, a partial image with a high likelihood of being an "eyes closed image" is predicted. FIG. 6 shows an example of the layer configuration of the machine learning model 1 used by the pupil position prediction unit 25. However, the layer configuration shown in FIG. 6 is an example, and other layer configurations may be employed. Machine learning model 1 includes a convolutional layer and a preprocessing unit that has an activation function such as a ReLU (Rectified Linear Unit) function that converts the output of the convolutional layer, and a preprocessing unit that converts the outputs of the smoothing layer, fully connected layer, and fully connected layer. A post-processing unit includes an activation function such as a ReLU function, and a fully connected layer in this order, and processes an input image to calculate a likelihood regarding an eye-closed image. Here, the input partial image is converted into image data of a predetermined size based on the distance from the camera 10 to the pupil so that the entire characteristic part of the eye is included. Since it is difficult for the size of the image to be processed to change during calculation, in machine learning model 1, the size features can be maintained in the feature extraction process without reducing the image, so the pooling layer in the preprocessing section can be omitted. I can do it. In addition, one of the roles of the pooling layer is to reduce the influence of positional shifts of feature parts in the input image by overlapping a plurality of partial images by the partial image creation unit 24 so that the entire feature part is included. This can be achieved through the ability to create while

Furthermore, the pupil position prediction unit 25 inputs the partial image predicted as described above into the machine learning model 2 trained in advance by the model learning unit 27, and based on the output value of the machine learning model 2, Predict the amount of two-dimensional positional deviation of the partial image from the eye-closed image. For example, the pupil position prediction unit 25 calculates the amount of deviation Δx in the x-axis direction and the amount of deviation in the y-axis direction as two-dimensional positional deviation amounts, with the horizontal direction of the image as the x-axis and the vertical direction of the image as the y-axis. Δy is predicted in subpixel units. FIG. 7 shows an example of the layer configuration of the machine learning model 2 used by the pupil position prediction unit 25. However, the layer configuration shown in FIG. 7 is an example, and other layer configurations other than this configuration may be employed. Machine learning model 2 includes a preprocessing unit including a convolution layer and a pooling layer, a smoothing layer, three fully connected layers, an activation function such as a ReLU function that converts the output of the fully connected layer, and the output of the fully connected layer. and a post-processing unit having a linear function that converts the . This machine learning model 2 processes the input image and calculates the likelihood for each two-dimensional shift amount (Δx, Δy) with respect to the eye-closed image.

In addition, the pupil position prediction unit 25 calculates the position of the pupil between the closed-eye image stored in advance in the image processing device 20 based on the amount of deviation (Δx, Δy) with respect to the closed-eye image predicted as described above. With reference to the related data, the position of the pupil in the partial image is calculated so as to offset the amount of deviation. Here, the relational data regarding the position of the pupil, which is stored in advance in the image processing device 20, has different values for each of the left and right pupils. Thereby, the pupil position prediction unit 25 predicts the position of the pupil on the face image while the eyes are closed. Then, the pupil position prediction unit 25 uses the stereo method to predict the positions of the left and right pupils on the face image, which are predicted for the two face images captured simultaneously by the two cameras 10. Calculate 3D position. The pupil position prediction unit 25 repeatedly performs prediction of the positions of the left and right pupils for each successive frame of the face image.

The learning image creation unit 26 creates learning data (training data) for preliminary learning of the machine learning model 1 and the machine learning model 2 described above. Preferably, the learning data creation section 26 creates the learning data immediately after the calibration process for detecting the line-of-sight direction disclosed in Japanese Patent Laid-Open No. 2005-230049 and the like is completed.

That is, the learning image creation unit 26 generates consecutive frames of facial images from each camera 10 before and after the subject A closes his eyes in response to an instruction output to the subject A by the image processing device 20 in the calibration process. is acquired, and the position of the pupil finally detected by the line of sight detection unit 23 is specified based on those facial images. Then, the learning image creation unit 26 acquires a face image of a frame at a timing when the subject A closes his eyes immediately after the frame in which the pupil position is specified, and creates a rectangle of a predetermined size based on the pupil position. The image of the area is cut out as an eye-closed image. For example, if the face image has a size of 150 pixels wide x 80 pixels high, the size of the eye-closed image is 30 pixels wide x 20 pixels high. The size of this eye-closed image is set to include the entire eyelashes, which are characteristic parts of the eyes of the subject A's face. Further, the relationship between the position of the pupil and the cutout position of the eye-closed image is set in advance to be different for the left and right pupils, and these relationships are stored in the image processing device 20, and the position of the pupil is determined by the pupil position prediction unit 25. Referenced when making predictions. Furthermore, the learning image creation unit 26 repeats the same process to create multiple frames of closed-eye images of the left and right pupils, and stores these as positive (correct) learning data "learning image 1" in the image processing device 20. Remember. At this time, the learning image creation unit 26 directs the subject A's face to face in a plurality of directions by outputting the instruction information from the image processing device 20 (for example, outputting it to the display device 30), and makes the subject A look at the subject A. Close the eyes (for example, in the front direction, 30 degrees to the right, and 30 degrees to the left with respect to the display device 30), and create closed-eye images of the left and right pupils in each direction as positive (correct answer) learning data "learning image 1". You can also do it.

FIG. 8 shows an example of closed-eye images GP _L1 and GP _R1 of the left and right pupils created by the learning image creation unit 26 for the face image G _F2 . In this way, an eye-closed image that includes the entire eyelashes is automatically created from the face image G _F2 based on the pupil positions P _L1 and P _R1 detected immediately before the eyes were closed.

Further, the learning image creation unit 26 creates negative (incorrect) learning data "learning image 2" at the same time as creating the positive learning data "learning image 1" for the plurality of frames described above. That is, the learning image creation unit 26 creates a plurality of frames of images shifted by a predetermined width in the two-dimensional direction from the reference position of the eye-closed image, which is positive learning data, in the face image as negative learning data. For example, for one eye-closed image, images are created by shifting 5 pixels in 8 patterns in two-dimensional directions within a range of ±5 pixels in the vertical direction and ±5 pixels in the horizontal direction as negative learning data. . In addition, images shifted by 20 pixels in four patterns in two-dimensional directions within a range of ±10 pixels in the vertical direction and ±10 pixels in the horizontal direction are created as negative learning data. However, the learning image creation unit 26 may create negative learning data in an image size different from that of positive learning data. The learning image creation unit 26 stores the created negative learning data “learning image 2” in the image processing device 20.

Note that, similar to the partial image creation section 24, the learning image creation section 26 calculates the distance between the camera 10 and the pupil based on the distance from the camera 10 to the pupil detected by the line of sight detection section 23 regarding the frame immediately before the face image to be processed. The cropping size of the learning data is set variably so that the entire feature part (eyelashes in this embodiment) is included, and the cropped learning data is converted to image data of a predetermined size (e.g., 8 pixels x 5 pixels). It's okay.

In addition, the learning image creation unit 26 creates the learning data "learning image 3" for the machine learning model 2 for determining the pupil position at the same time as creating the positive learning data "learning image 1" for multiple frames described above. We also create . Specifically, the learning image creation unit 26 generates a plurality of images (shifted images) that are shifted by a predetermined width in a two-dimensional direction from the reference position of the eye-closed image, which is positive learning data, in the face image as "learning images 3". Create frames. However, the shift width of "learning image 3" is set to a smaller value than the shift width of "learning image 2." For example, for one eye-closed image, multiple images shifted by 1 pixel in 2-dimensional directions in 25 patterns within the range of ±2 pixels in the vertical direction and ±2 pixels in the horizontal direction are transferred to the learning data "Learning image". 3". However, the learning image creation unit 26 may create the learning data "learning image 3" in a larger image size than the learning data "learning image 1." The learning image creation unit 26 sends the plurality of created learning data "learning images 3" to the image processing device along with the amount of deviation Δx in the x-axis direction and the amount Δy of deviation in the y-axis direction of each learning image with respect to the eye-closed image. Stored within 20 days.

FIG. 9 shows an example of learning data "learning image 3" created by the learning image creating unit 26 using the face image _GF2 acquired by subject A when his eyes were closed. In this way, the learning images GP _R2 to GP _R4 , in which closed-eye images that include the entire eyelashes are shifted by ±2 pixels in the vertical direction and ±2 pixels in the horizontal direction, are automatically created from the face image G _F2 . Created in

The model learning unit 27 uses the learning data created by the learning image creation unit 26 to train machine learning model 1 and machine learning model 2 used by the pupil position prediction unit 25. That is, the model learning unit 27 performs machine learning using a plurality of positive learning data “learning images 1” and a plurality of negative learning data “learning images 2” stored in the image processing device 20. Train model 1. In addition, the model learning unit 27 uses a combination of the plurality of pieces of learning data “learning image 3” stored in the image processing device 20 and the deviation amounts Δx, Δy of the respective learning images as positive image training data. The machine learning model 2 is trained using the machine learning model 2.

Next, the operating procedure of the line of sight detection device 1 will be explained, and the steps of the pupil detection method according to this embodiment will be explained. FIG. 10 is a flowchart showing the operation procedure of the line of sight detection device 1.

First, when the line-of-sight direction detection processing for the subject A is started, the lighting control unit 21 and the image acquisition unit 22 of the image processing device 20 control the lighting timing of the light source 13 and the image from the camera 10. Acquisition control is started (step S101). Thereafter, the learning image creation unit 26 of the image processing device 20 creates the learning data "learning image 1," "learning image 2," and "learning image 3" during the calibration process for line-of-sight direction detection. , a plurality of images are acquired (step S102). Next, the model learning unit 27 of the image processing device 20 uses the learning data to learn machine learning model 1 and machine learning model 2 (step S103).

Then, the line-of-sight detection unit 23 of the image processing device 20 executes the line-of-sight direction detection process (including the detection of the pupil position) for the facial images of frames continuously acquired from the camera 10 (step S104; Pupil position detection step). In parallel, a plurality of partial images are acquired from the facial image by the partial image creation unit 24 of the image processing device 20 (step S105). Thereafter, the pupil position prediction unit 25 of the image processing device 20 executes pupil position prediction processing based on the plurality of partial images (step S106; pupil position prediction step). Furthermore, in the image processing device 20, it is determined whether or not a face image of the next frame exists (step S107), and if a face image of the next frame exists, the processes of steps S104 to S106 are repeated. Here, in setting the window in the face image in step S104, the three-dimensional position of the pupil predicted by the process in step S106 is set to be tracked in the frame before the face image to be processed.

The effects of the line of sight detection device 1 and the pupil detection method using the same according to the embodiment of the present disclosure will be described.

According to the line of sight detection device 1, the position of the pupil is detected on the face images acquired at consecutive timings by the camera 10, and the position of the characteristic part of the eye in the closed eye state is searched from the face image. The position of the pupil when the eyes are closed is predicted on the face image. When detecting the position of the pupil, the pupil is tracked by setting a window using the position of the pupil predicted on the face image. As a result, the position of the pupil can be detected by tracking the pupil even when subject A closes his eyes, so the window size can be made relatively small, and the pupil detection process can be performed with high calculation efficiency and high precision. realizable. In addition, since a complicated optical system is not required, pupil detection processing can be realized with a simple device configuration.

The line of sight detection device described in the conventional Japanese Patent Application Publication No. 2017-102731 includes a subject detection device that detects the three-dimensional position of the subject's head, a narrow-field camera, and a pan-tilt that adjusts the attitude of the narrow-field camera. It was equipped with a mechanism. In this device, the posture and zoom value of the narrow-field camera are controlled based on the three-dimensional position of the subject's head detected by the subject detection device, and the posture and zoom value are obtained by the controlled narrow-field camera. The position of the subject's pupil is detected using the bright pupil image and the dark pupil image. On the other hand, this conventional device requires two types of detection optical systems, which tends to complicate the device configuration. On the other hand, according to the configuration of the line of sight detection device 1 of this embodiment, a complicated optical system is not required, so that pupil detection processing can be realized with a simple device configuration.

In this embodiment, a machine learning model using a neural network is used to predict the pupil position. In this case, the accuracy of predicting the position of the pupil on the face image can be reliably improved by a simple learning method, and stable pupil detection processing can be achieved.

Furthermore, in this embodiment, a partial image cut out from a face image is input to a machine learning model as input data, and the machine learning model is used to search for the position of the partial image that includes the characteristic part of the eye. location is predicted. In this case, by using a partial image cut out from the face image as input data, the position of the pupil on the face image can be predicted through simple processing. As a result, the calculation efficiency of the pupil detection process can be further improved.

At this time, the partial image used as input data for the machine learning model is cut out to a size that includes the entire characteristic part of the eye. In this case, the entire characteristic part of the eye can be included in the partial image extracted from the face image, and the accuracy of predicting the position of the pupil can be improved. As a result, more stable pupil detection processing is realized.

The image processing device 20 further includes a model learning unit 27 that learns a machine learning model, and the model learning unit 27 detects a face image acquired when the subject A closes his or her eyes immediately before the subject A closes his or her eyes. A machine learning model is trained using the extracted facial images as training data. In this way, it is possible to create training data in which the relationship between the extracted face image and the pupil position is appropriately set, and by using this training data for learning, it is possible to improve the prediction accuracy of the pupil position. can. As a result, more stable pupil detection processing is realized. Furthermore, this embodiment enables automatic annotation in which training data is automatically acquired during calibration processing. This eliminates the need for manual acquisition of training data, reducing the user's workload during learning.

The image processing device 20 also has a function of detecting the distance from the camera 10 to the pupil, and sets the size of the cutout of the partial image as input data variably according to the distance of the pupil. With such a function, it is possible to appropriately set the size of a partial image used as input data, and it is possible to improve the prediction accuracy of the pupil position. As a result, more stable pupil detection processing is realized. Further, in the machine learning model used for predicting the pupil position, a pooling layer provided to enable prediction even if there is a change in image size can be omitted. Therefore, it is possible to speed up calculations during learning and prediction.

The image processing device 20 also has a function of detecting the distance from the camera 10 to the pupil, and sets the size of the cutout of the face image as training data variably according to the distance of the pupil. With such a function, it is possible to appropriately set the size of the cutout of the face image used as training data, and it is possible to improve the prediction accuracy of the pupil position. As a result, more stable pupil detection processing is realized.

Further, the machine learning model 2 used in the image processing device 20 is a model that predicts the two-dimensional shift amount of the position of a partial image that includes the eye feature, and the model learning unit 27 is a model that predicts the amount of two-dimensional shift in the position of a partial image that includes the eye feature part. A shifted image is generated while shifting the image two-dimensionally, and a learning model is trained using the shifted image and the two-dimensional shift amount of the shifted image as training data. In this case, the accuracy of predicting the position of the pupil on the face image can be improved regardless of the amount of deviation on the face image between the plurality of partial images used as input data at the time of prediction. As a result, more stable pupil detection processing is realized.

In addition, the line of sight detection unit 23 of the image processing device 20 detects the position of the pupil by tracking the position of the pupil using the position of the pupil detected on the face image of consecutive frames, and detects the position of the pupil in the previous frame. If detection of the pupil position on the face image fails, the pupil position predicted by the pupil position prediction unit 25 may be used to track the pupil position. According to this configuration, when detecting the position of the pupil continuously, even if the subject A closes his eyes due to blinking or the like and the image of the pupil no longer appears on the image, the position of the pupil can be stably tracked. When subject A opens his eyes and the pupil image appears again on the image, by using the pupil position that was tracked when the eyes were closed for window setting, the pupil image can be stably detected without delay immediately after opening the eyes. The location can be detected.

The present invention is not limited to the embodiments described above. The configuration of the above embodiment may be modified in various ways.

For example, although the image processing device 20 according to the above embodiment includes the learning image creation unit 26 and the model learning unit 27, the functions of either or both of the learning image creation unit 26 and the model learning unit 27 are It may be implemented in a computer separate from the processing device 20, and either or both of the creation of the training data and the learning of the machine learning model may be performed by the separate computer.

Furthermore, in this embodiment, the characteristic parts of the eye that are searched may be eyelids, eyebrows, etc. instead of eyelashes. Furthermore, the search target may be a facial feature such as a nostril instead of an eye feature. When searching for a facial feature such as a nostril, the image processing device 20 stores information on the relative three-dimensional position between the pupil and the facial feature in advance, and further It is preferable to predict the position of the pupil based on information about the rotational state and movement state of the head determined from the pupil and facial features.

Furthermore, the line of sight detection device 1 detects the position of the pupil by tracking the pupil using the predicted pupil position and setting the position of a window to be set on the face image. Similar to the configuration described in the publication, a configuration in which the attitude and zoom value of the narrow-field camera can be controlled may be adopted. In this case, the line of sight detection device 1 is configured to track the pupil by controlling the attitude and zoom value of the narrow-field camera using the predicted three-dimensional position of the pupil. Even with such a modification, the pupil can be tracked even when the subject A closes his eyes, and pupil detection processing with high calculation efficiency and high accuracy can be realized.

In addition, the image processing device 20 of the line of sight detection device 1 used to predict the pupil position using a machine learning model using a neural network, but it operates to predict the pupil position using other image processing such as template matching. Good too. Through such an operation, the position of the characteristic part of the eye can be searched for in the face image, and the position of the pupil can be predicted based on this position.

Furthermore, instead of calculating the three-dimensional position of the pupil using the stereo method, the image processing device 20 calculates the position of the pupil, the position of the nostril, etc. It may also function to determine the three-dimensional position of the pupil using the position of a facial feature, the position of a marker attached to the face, or the like. Furthermore, the image processing device 20 provides a dot pattern to the face, and simultaneously detects the pupil while grasping the shape of the entire face using this dot pattern, thereby determining the three-dimensional position of the pupil with respect to the structure (shape) of the entire face. It may function as such. The image processing device 20 may also function to determine the three-dimensional position of the pupil with respect to the facial structure by determining the distance to the face for each pixel using a TOF (Time Of Flight) camera.

When the learning data is created by the image processing device 20 according to the embodiment described above, an eye-closed image that includes the entire eyelashes, which are the characteristic part of the eye, is generated. If the size of this eye-closed image is too small, the eyelashes will protrude from the image, reducing the accuracy of predicting the pupil position by the machine learning model. On the other hand, if the size of the eyes-closed image is too large, the image will include features other than eyelashes (for example, eyebrows, hair, etc.), and the accuracy of predicting the pupil position by the machine learning model will decrease due to the influence of these features. The image processing device 20 may have a function of automatically optimizing the size of the eye-closed image in order to prevent the prediction accuracy of the pupil position from decreasing due to the size of the eye-closed image. For example, the learning image creation unit 26 of the image processing device 20 first acquires an eye-closed image in a relatively large size, and then identifies a region where eyelashes exist from the acquired eye-closed image by image analysis, and creates a partial image. The size of the eye-closed image is set so that a gap larger than the shift amount is secured around the eyelashes. With such a function, the size of the cutout of the face image used as training data can be appropriately set according to the eyelash area, and the prediction accuracy of the pupil position can be improved. As a result, more stable pupil detection processing is realized.

Additionally, if the position of the pupil just before closing the eyes and the position of the eyelashes are misaligned in the eye-closed image, the position of the eyelashes will be located at the edge of the area of the eye-closed image, reducing the accuracy of predicting the pupil position by the machine learning model. It is also possible to do so. In other words, it is considered preferable for the eyelashes to be located near the center of the eye-closed image in order to improve the prediction accuracy of the pupil position. In this case, the eye-closed image can be set to a relatively small size, and the calculation time for predicting the pupil position can also be shortened. In order to prevent a decrease in the prediction accuracy of the pupil position due to the positional deviation of the eyelashes in the eye-closed image, the learning image creation unit 26 of the image processing device 20 calculates the presence of eyelashes by image analysis from the acquired eye-closed image. A setting is made to change the cutout area of the eye-closed image so that the eyelashes are located at the center of the eye-closed image. In this case as well, the learning image creation unit 26 stores in the image processing device 20 data indicating the relationship between the pupil position and the cutout position of the eye-closed image. With such a function, it is possible to appropriately set the cutting position of the face image used as training data according to the eyelash area, and it is possible to improve the prediction accuracy of the pupil position. As a result, more stable pupil detection processing is realized.

Further, the line of sight detection unit 23 of the image processing device 20 of this embodiment may set a window when detecting the position of the pupil as follows. Specifically, when subject A closes his eyes and opens them again, the line of sight detection unit 23 uses the prediction result of the pupil position when the eyes are closed, which was tracked by the pupil position prediction unit 25. By setting a window of a predetermined size in , it is possible to restart pupil detection based on the difference image between the bright pupil image and the dark pupil image. Conventional pupil detection devices tend to be unable to detect the pupil for several frames after the eyes are opened after the eyes are closed. Due to the function of the line of sight detection unit 23 described above, pupil detection can be restarted stably without delay from the frame immediately after the subject A reopens his eyes after closing his eyes.

Furthermore, the pupil position prediction unit 25 of the image processing device 20 according to the present embodiment predicts the position of the pupil using two networks, machine learning model 1 and machine learning model 2. On the other hand, as a modified example, the image processing device 20 may predict the pupil position using one network that integrates the prediction functions of two networks, or may predict the pupil position using three or more networks. may be predicted. For example, the three networks are "machine learning model 1" that predicts an eye-closed image using multiple partial images cut out from within the window of a face image, and "machine learning model 1" that predicts the amount of deviation of a partial image from the eye-closed image. "Learning Model 2" and "Machine Learning Model 3" which predicts an eye-closed image using a plurality of partial images cut out from the entire face image may be used.

1... Line of sight detection device (pupil detection device), 10... Camera, 13... Light source, 20... Image processing device (computation device), 23... Line of sight detection section (pupil position detection section, pupil distance detection section), 24... Partial image Creation _{unit, 25... Pupil position prediction unit, 27... Model learning unit, A... Subject, GF1, GF2... Face image, GP F1... Partial image, GP L1 , GP R1 ...} Eyes closed image, _PL1 , _PR1 ...pupil position.

Claims (14)

a camera that captures facial images of a subject at consecutive timings by capturing an image of the subject's face;
a light source that emits light toward the subject's face;
a calculation device that processes a face image acquired by the camera at the timing of irradiation of the light;
The arithmetic device is
a pupil position detection unit that detects the position of the subject's pupil on the face image;
a pupil position prediction unit that predicts the position of the pupil of the subject when the subject's eyes are closed by searching for the position of the characteristic part of the face in the eye-closed state on the face image;
The pupil position detection unit detects the position of the pupil by tracking the pupil using the position of the pupil predicted by the pupil position prediction unit.
Pupil detection device. The facial features are eye features of the subject;
The pupil detection device according to claim 1. The pupil position prediction unit predicts the pupil position using a machine learning model using a neural network.
The pupil detection device according to claim 1 or 2. The pupil position prediction unit inputs a partial image extracted from the face image to the machine learning model as input data, and uses the machine learning model to search for a position of the partial image including the facial feature. predicting the position of the pupil by
The pupil detection device according to claim 3. The pupil position prediction unit cuts out the partial image in a size that includes the entire feature of the face.
The pupil detection device according to claim 4. The arithmetic device further includes a model learning unit that learns the machine learning model,
The model learning unit cuts out the face image acquired when the subject's eyes are closed based on the position of the pupil detected by the pupil position detection unit immediately before the subject closes his eyes, and extracts the cut out face image. learning the machine learning model using as training data;
The pupil detection device according to claim 4 or 5. The arithmetic device further includes a pupil distance detection unit that detects a distance from the camera to the pupil,
The pupil position prediction unit variably sets the size of the cutout of the partial image as the input data according to the distance of the pupil.
The pupil detection device according to claim 4 or 5. The arithmetic device further includes a pupil distance detection unit that detects a distance from the camera to the pupil,
The model learning unit variably sets the size of the cutout of the face image as training data according to the distance of the pupil.
The pupil detection device according to claim 6. The model learning unit specifies a region in which the facial feature exists from the cut out face image, and sets a size for cutting out the face image as training data based on the specified region.
The pupil detection device according to claim 6. The model learning unit identifies a region in which the facial features are present in the cut out facial image, and adjusts the cutout of the facial image as training data so that the identified region is located in the center of the image. set the position,
The pupil detection device according to claim 6 or 9. The machine learning model calculates a positional shift of a partial image including the facial features with respect to the face image cut out based on the position of the pupil detected by the pupil position detection unit immediately before the subject's eyes are closed. It is a model that predicts the amount of
The model learning unit generates a shifted image while shifting the cut out face image, and uses the shifted image and a shift amount of the shifted image as training data to learn the learning model.
The pupil detection device according to claim 6 or 8. The pupil position detection unit detects the pupil position by tracking the pupil position using the pupil position detected on the face image in consecutive frames, and detects the pupil position on the face image in the immediately previous frame. If detection of the position of the pupil on the image fails, tracking the position of the pupil using the position of the pupil predicted by the pupil position prediction unit;
The pupil detection device according to any one of claims 1 to 11. The pupil position detection section is configured to detect a window on the face image based on the pupil position predicted by the pupil position prediction section at the timing when the subject's eyes are closed, when the subject closes his eyes and then opens his eyes again. Detecting the position of the pupil by setting
The pupil detection device according to claim 12. A camera that captures facial images of a target person at successive timings by imaging the target person's face, a light source that irradiates light toward the target person's face, and facial images acquired by the camera at the light irradiation timings. A pupil detection method using a calculation device that processes
a pupil position detection step in which the calculation device detects the position of the subject's pupil on the face image;
a pupil position prediction step in which the arithmetic device predicts the position of the pupil of the subject when the subject's eyes are closed by searching for the position of the feature part of the face in the eye-closed state on the face image;
In the pupil position detection step, the pupil position is detected by tracking the pupil using the pupil position predicted in the pupil position prediction step.
Pupil detection method.

Images