JP7263959B2 - Eyeball detection device, sight line detection device, eyeball detection method, and eyeball detection program - Google Patents

Description

The present invention relates to an eyeball detection device, a line of sight detection device, an eyeball detection method, and an eyeball detection program.

As a line-of-sight detection device for detecting the line of sight of a subject, for example, there is a configuration in which the subject's face is imaged with a camera, the left and right eyeball images are detected from the captured face image by searching, and the line of sight of the subject is detected based on the detection result. known (see, for example, Patent Document 1).

JP 2008-125619 A

In the line-of-sight detection device described in Patent Document 1, even if one of the left and right eyeball images is lost, for example, it is necessary to search for both the left and right eyeball images from the face image, resulting in a long processing time. Is required. Similar problems also exist in other devices that search for eyeball images from face images, such as face recognition devices.

The present invention has been made in view of the above, and provides an eyeball detection device, a line of sight detection device, an eyeball detection method, and an eyeball detection program capable of shortening the processing time when detecting the eyeballs of a subject. intended to

An eyeball detection device according to the present invention includes an imaging device that acquires a face image including left and right eyeball images of a subject, and a first distance calculation unit that calculates a first distance between the subject and the imaging device. , searching the left and right eyeball images from the face image, and if one of the left and right eyeball images of the subject person is detected by the search, a search result of the eyeball detection unit for the one eyeball image; an eyeball searching unit that searches for the other eyeball image of the subject from the face image based on the first distance and a second distance between the right and left eyeballs of the subject.

The line-of-sight detection apparatus according to the present invention detects the right and left eyeball images of a subject using the above-described eyeball detection apparatus, and detects the line of sight of the subject based on the detection results.

An eyeball detection method according to the present invention comprises: obtaining a face image including right and left eyeball images of a subject; calculating a first distance between the subject and the imaging device; searching for the left and right eyeball images; if one of the left and right eyeball images of the subject is detected by the search, the search result for the one eyeball image; the first distance; searching the face image for the subject's other eye image based on a second distance between the subject's left and right eyeballs.

An eyeball detection program according to the present invention comprises a process of acquiring a face image including left and right eyeball images of a subject, a process of calculating a first distance between the subject and the imaging device, a process of searching for the left and right eyeball images; when one of the left and right eyeball images of the subject is detected by the search, the search result for the one eyeball image; the first distance; a second distance between the left and right eyeballs of the subject, and a process of searching the face image for the other eyeball image of the subject.

Advantageous Effects of Invention According to the present invention, it is possible to provide an eyeball detection device, a line-of-sight detection device, an eyeball detection method, and an eyeball detection program capable of shortening the processing time when detecting the eyeballs of a subject.

FIG. 1 is a diagram schematically showing an example of a line-of-sight detection device according to this embodiment. FIG. 2 is a functional block diagram showing an example of the line-of-sight detection device. FIG. 3 is a diagram showing an example of a subject's face image acquired by an imaging device. FIG. 4 is a diagram showing an example of an operation of detecting an eyeball image from a face image. FIG. 5 is a diagram showing an example of the operation of predicting the position of the other eyeball image. FIG. 6 is a flow chart showing an example of an eyeball detection method according to this embodiment. FIG. 7 is a diagram showing another example of the operation of predicting the position of the other eyeball image. FIG. 8 is a diagram showing another example of the operation of predicting the position of the other eyeball image. FIG. 9 is a flowchart showing another example of the eyeball detection method according to this embodiment.

EMBODIMENT OF THE INVENTION Hereinafter, embodiment of the eyeball detection apparatus, line-of-sight detection apparatus, eyeball detection method, and eyeball detection program which concern on this invention is described based on drawing. In addition, this invention is not limited by this embodiment. In addition, components in the following embodiments include components that can be easily replaced by those skilled in the art, or components that are substantially the same.

In the following description, a three-dimensional global coordinate system is set to describe the positional relationship of each part. The direction parallel to the first axis of the predetermined plane is defined as the X-axis direction, the direction parallel to the second axis of the predetermined plane orthogonal to the first axis is defined as the Y-axis direction, and the direction is orthogonal to the first axis and the second axis. Let the direction parallel to the third axis be the Z-axis direction. The predetermined plane includes the XY plane.

[Line-of-sight detection device]
FIG. 1 is a diagram schematically showing an example of a line-of-sight detection device (eyeball detection device) 100 according to this embodiment. A line-of-sight detection apparatus 100 according to the present embodiment detects the line of sight of a subject (subject). As the line-of-sight detection device 100, for example, a device that detects the line of sight based on the position of the subject's pupil and the position of the corneal reflection image, or a device that detects the line of sight based on the position of the inner corner of the eye and the position of the iris of the subject, etc. Various devices that can detect the line of sight of the subject can be used.

As shown in FIG. 1 , the line-of-sight detection device 100 includes a display device 10 , an image acquisition device 20 , a computer system 30 , an output device 40 , an input device 50 and an input/output interface device 60 . The display device 10 , the image acquisition device 20 , the computer system 30 , the output device 40 and the input device 50 perform data communication via the input/output interface device 60 . The display device 10 and the image acquisition device 20 each have a drive circuit (not shown).

Display device 10 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). In this embodiment, the display device 10 has a display section 11 . The display unit 11 displays information such as images. The display unit 11 is substantially parallel to the XY plane. The X-axis direction is the horizontal direction of the display unit 11 , the Y-axis direction is the vertical direction of the display unit 11 , and the Z-axis direction is the depth direction orthogonal to the display unit 11 . The display device 10 may be a head-mounted display device. In the case of a head-mounted display, components such as the image capture device 20 would be placed within the head-mounted module.

The image acquisition device 20 acquires image data of a face image including left and right eyeballs EB (left eye EBL, right eye EBR) of the subject, and transmits the acquired image data to the computer system 30 . The image acquisition device 20 has an imaging device 21 . The image pickup device 21 acquires the image data by photographing the face portion including the left and right eyeballs EB of the subject. The imaging device 21 has various cameras according to the method of detecting the line of sight of the subject. For example, in the case of a system that detects the line of sight based on the position of the subject's pupil and the position of the corneal reflection image, the imaging device 21 has an infrared camera and can transmit near-infrared light with a wavelength of 850 [nm], for example. It has an optical system and an imaging device capable of receiving the near-infrared light. In addition, for example, in the case of a system that detects the line of sight based on the position of the inner corner of the eye and the position of the iris of the subject, the imaging device 21 has a visible light camera. The imaging device 21 outputs a frame synchronization signal. The period of the frame synchronization signal can be, for example, 20 [msec], but is not limited to this. In addition, the imaging device 21 can detect the first distance D between the imaging device 21 and the subject's face. For this reason, the imaging device 21 can have, for example, a configuration of a stereo camera having a first camera 21A and a second camera 21B, but is not limited to this.

In addition, for example, in the case of a system that detects the line of sight based on the position of the subject's pupil and the position of the corneal reflection image, the image acquisition device 20 has an illumination device 22 that illuminates the subject's eyeball EB. The illumination device 22 includes an LED (light emitting diode) light source, and can emit near-infrared light with a wavelength of 850 [nm], for example. Note that, for example, in the case of a system that detects a line-of-sight vector based on the position of the inner corner of the eye and the position of the iris of the subject, the illumination device 22 may not be provided. The illumination device 22 emits detection light in synchronization with the frame synchronization signal of the imaging device 21 . The illumination device 22 can have a configuration including, for example, a first light source 22A and a second light source 22B, but is not limited to this.

The computer system 30 centrally controls the operation of the line-of-sight detection device 100 . The computer system 30 includes an arithmetic processing unit 30A and a storage device 30B. The arithmetic processing unit 30A includes a microprocessor such as a CPU (central processing unit). The storage device 30B includes memory or storage such as ROM (read only memory) and RAM (random access memory). The arithmetic processing unit 30A performs arithmetic processing according to a computer program 30C stored in the storage device 30B.

Output device 40 includes a display device such as a flat panel display. Note that the output device 40 may include a printing device. The input device 50 generates input data by being operated. Input device 50 includes a keyboard or mouse for the computer system. Note that the input device 50 may include a touch sensor provided in the display section of the output device 40, which is a display device.

A visual line detection device 100 according to the present embodiment is a device in which the display device 10 and the computer system 30 are separate devices. Note that the display device 10 and the computer system 30 may be integrated. For example, the line-of-sight detection device 100 may include a tablet personal computer. In this case, the tablet personal computer may be equipped with a display device, an image acquisition device, a computer system, an input device, an output device, and the like.

FIG. 2 is a functional block diagram showing an example of the line-of-sight detection device 100. As shown in FIG. As shown in FIG. 2, the computer system 30 includes a display control unit 31, a distance calculation unit (first distance calculation unit, second distance calculation unit) 32, an eyeball search unit 33, a fixation point detection unit 34, and a storage unit 35 . The functions of the computer system 30 are exhibited by an arithmetic processing unit 30A and a storage device 30B (see FIG. 1). A part of the functions of the computer system 30 may be provided outside the line-of-sight detection device 100 .

The display control unit 31 displays, for example, an image, a video, a character, or the like to be watched by the subject on the display unit 11 .

The distance calculator 32 calculates the inter-subject apparatus distance (first distance) D1 and the interocular distance (second distance) L1. The inter-subject device distance D<b>1 is the distance between the subject and the imaging device 21 . The interocular distance L1 is the distance between the subject's left eye EBL and right eye EBR. The distance calculator 32 calculates the inter-subject device distance D1 from the images of the first camera 21A and the second camera 21B of the imaging device 21 .

Further, the interocular distance L1 can be measured in advance and the measurement result can be stored in the storage unit 35 . In this case, the distance calculation unit 32 can acquire the measurement result stored in the storage unit 35 as the interocular distance L1.

Further, even when the measurement result of the interocular distance L1 is not obtained in advance, the distance calculation unit 32 calculates , the interocular distance L1 can be obtained by the following procedure. That is, assuming that the size of the subject image formed on the photodetection area of the imaging device 21 is S [mm] and the focal length of the lens is F [mm], the distance from the imaging device 21 is D [mm]. An actual object size R [mm] is obtained by the following [Equation 1].

Further, if the pixel pitch of the light receiving portion of the imaging device 21 is P [μm/px], the distance L [px] between two points on the image captured by the imaging device 21 is obtained by the following [Equation 2]. be done.

Therefore, the distance between the subject and the imaging device 21 (distance between subject devices) is D1, and the distance between both eyeball images on the face image captured by the imaging device 21 (hereinafter referred to as the distance between eyeball images) is L2. Then, the interocular distance L1 [mm], which is the actual distance between the left and right eyeballs of the subject, is obtained from [Equation 1] and [Equation 2] by the following [Equation 3].

The distance calculation unit 32 can calculate the interocular distance L1 based on the inter-subject apparatus distance D1 and the eyeball image distance L2 by using the above [Equation 3].

The eyeball search unit 33 searches for left and right eyeball images E (EL, ER) from the face image acquired by the imaging device 21 . The eyeball search unit 33 compares, for example, the face image with an eyeball pattern or the like stored in advance in the storage unit 35, and searches for the eyeball image E based on the comparison result. The eyeball search unit 33 can set the search period to be the same period as the period of the frame synchronization signal output from the imaging device 21 (for example, every 20 [msec]).

For example, the eyeball search unit 33 searches for the eyeball image E over the entire face image in the initial search, and stores the eyeball position based on the eyeball image E in the storage unit 35 when the eyeball image E is detected. . Examples of the eyeball position include reference coordinates, pupil coordinates, corneal reflection coordinates, and the like. The reference coordinates are the coordinates of a reference position for the eyeball image E, such as the coordinates of the central position of the eyeball image E. The pupil coordinates are coordinates indicating the position of the center of the pupil in the eyeball image E. The corneal reflection coordinates are coordinates indicating the position of the corneal reflection center in the eyeball image E. FIG. Pupil coordinates or corneal reflection coordinates may be used as the reference coordinates. In subsequent searches, the eyeball search unit 33 searches for an area within a predetermined radius around the eyeball position stored in the storage unit 35 . This shortens the processing time compared to searching the entire face image each time.

The gazing point detection unit 34 detects the position data of the gazing point of the subject. In this embodiment, the point-of-regard detection unit 34 detects the subject's line-of-sight vector defined by the three-dimensional global coordinate system based on the left and right eyeball images in the subject's face image captured by the imaging device 21 . The point-of-regard detection unit 34 detects the position data of the intersection between the detected line-of-sight vector of the subject and the display unit 11 of the display device 10 as the position data of the point of gaze of the subject. That is, in the present embodiment, the position data of the gaze point is the position data of the intersection between the subject's line-of-sight vector defined by the three-dimensional global coordinate system and the display unit 11 of the display device 10 . This makes it possible to detect which position on the display unit 11 the subject is gazing at. The gazing point detection unit 34 detects the position data of the gazing point of the subject for each prescribed sampling period. This sampling period can be, for example, the period of the frame synchronization signal output from the imaging device 21 (for example, every 20 [msec]).

The storage unit 35 stores distance data of the inter-subject apparatus distance D1 and inter-eye distance L1, distance data of the eyeball image distance L2, eyeball image position data, gaze point data, and the like. Further, the storage unit 35 performs a process of acquiring a face image including left and right eyeball images of the subject, a process of calculating the distance D1 between the subject apparatuses, a process of searching the left and right eyeball images from the face image, and a process of searching the left and right eyeball images from the face image. When one of the left and right eyeball images is detected, based on the search result for the one eyeball image, the distance D1 between the subject's devices, and the distance L1 between the eyes of the subject, the other eyeball image of the subject from the face image It stores an eyeball detection program that causes a computer to execute a process of searching for an eyeball image.

[Eye detection method]
Next, an eyeball detection method according to this embodiment will be described. The eyeball detection method according to the present embodiment can be used to detect the subject's eyeball when detecting the subject's line of sight using the line-of-sight detection apparatus 100 described above.

FIG. 3 is a diagram showing an example of a subject's face image acquired by the imaging device 21. As shown in FIG. As shown in FIG. 3 , the eyeball search unit 33 searches for an eyeball image E of the subject H from the face image IM acquired by the imaging device 21 . In FIG. 3, the eyeball searching unit 33 compares the eyeball image pattern stored in advance in the storage unit 35 with the face image IM, and obtains the left eyeball image EL and the right eyeball image ER as the eyeball image E. detectable. When the eyeball images E of both the left eyeball image EL and the right eyeball image ER are successfully detected, the distance calculation unit 32 calculates the eyeball image distance L2 between the left and right eyeball images E, and the calculation result and the above [number 3], the interocular distance L1 is calculated, and the interocular distance L1 is stored in the storage unit 35 . Further, the eyeball search unit 33 calculates the positions of the left and right eyeball images EL in the face image IM based on the detection result, and stores the calculation result in the storage unit 35 .

The point-of-regard detection unit 34 detects the line-of-sight vector of the subject based on the left and right eyeball images E in the face image IM. Then, the point-of-regard detection unit 34 detects the position data of the intersection between the detected line-of-sight vector of the subject and the display unit 11 of the display device 10 as the position data of the point of gaze of the subject.

Here, the eyeball search unit 33 may fail to detect the eyeball image E in each search. One of the causes of the detection failure of the eyeball image E is that the subject blinks. When the subject blinks, the face image IM with the eyelids closed is acquired. Therefore, the eyeball search unit 33 cannot detect the eyeball image E corresponding to the eyeball pattern stored in advance from the face image IM. However, in the case of blinking, although the eyelids are closed, the positions of the left and right eyeball images EL and ER hardly change, and the period during which the eyelids are closed is short. Therefore, if the eyeball image E detection failure is caused by blinking, even if the subject closes his/her eyelids in one search, there is a high possibility that the subject opens his/her eyelids in subsequent searches. Therefore, when blinking is the cause, there is a high possibility that the eyeball image E will be detected by searching the same search range as the previous search in the next and subsequent searches. Therefore, when the eyeball image E fails to be detected, the eyeball search unit 33 searches the same search range as the previous search in the next search.

Next, as an example of the case where the eyeball image E cannot be detected, there is a case where the position of the subject's face changes due to the subject turning his/her face away or looking down. In this case, there is a high possibility that the positions of the left and right eyeballs EB will move as the position of the face changes, and the period during which the eyeball image E cannot be detected is longer than the blink. Therefore, if the eyeball image E cannot be detected due to a change in the position of the face, it is necessary to search for the eyeball image E again from the face image. Therefore, if the detection failure of the eyeball image E continues for a predetermined period of time or longer, the eyeball search unit 33 changes the search method as follows.

FIG. 4 is a diagram showing an example of the operation of detecting the eyeball image E from the face image IM. As shown in FIG. 4, the eyeball searching unit 33 first detects one of the left and right eyeball images E in the acquired face image IM. When detecting one of the left and right eyeball images E (for example, the right eyeball image ER), a search area DA is set for one of the eyeball images E and searched. Therefore, the processing time can be shortened compared to the case where both the right and left eyeball images E are detected. When one eyeball image E is detected, the eyeball search unit 33 extracts the face image IM from the face image IM based on the search result for the one eyeball image E, the inter-subject apparatus distance D1, and the inter-eye distance L1. The other eyeball image E (for example, the left eyeball image EL) of the subject H is searched.

In this case, the eyeball search unit 33 calculates the position of the one eyeball image E in the face image IM. Then, the eyeball search unit 33 predicts the position of the other eyeball image based on the calculated eyeball position based on one eyeball image E, the inter-subject apparatus distance D1, and the inter-eye distance L1, and the prediction result is search based on

Here, from the above [Equation 3], the distance L2 between the eyeball images is represented by [Equation 4].

The eyeball search unit 33 calculates the distance L2 between the eyeball images using the above [Equation 4]. The eyeball search unit 33 predicts the eyeball position of the other eyeball image E using the calculated distance L2.

That is, the first search area is set at a position separated from one of the detected eyeball images E by the inter-eyeball image distance L2 in the horizontal direction. The coordinates (x, y) of the origin of the first search area can be expressed as in [Equation 5] below. In addition, θ indicates an angle in a rotating direction about one eyeball image E as a center.

FIG. 5 is a diagram showing an example of the operation of predicting the eyeball position of the other eyeball image E. As shown in FIG. In the following example, a case where one detected eyeball image E is the right eyeball image ER and the other eyeball image E whose detection has failed is the left eyeball image EL will be described as an example, but this is not limiting. not. For example, the same explanation can be given even if one detected eyeball image E is the left eyeball image EL and the other eyeball image E whose detection fails is the right eyeball image ER.

As shown in FIG. 5, the eyeball search unit 33 first places a first search area in the acquired face image IM at a position on a virtual straight line M parallel to the right eyeball image ER detected in the left-right direction. Set DA1 and DA2. The eyeball search unit 33 sets the first search areas DA1 and DA2 such that the distances between the right eyeball image ER and the first search areas DA1 and DA2 are equal to the inter-eyeball image distance L2. The first search areas DA1 and DA2 are set at equidistant positions on both sides in the left-right direction with the right eyeball image ER as the center.

The eyeball search unit 33 searches the set first search areas DA1 and DA2. The eyeball search unit 33 may simultaneously search the first search area DA1 on the right side of the right eyeball image ER and the first search area DA2 on the left side of the right eyeball image ER. , the search may be performed at different times.

FIG. 6 is a flow chart showing an example of an eyeball detection method according to this embodiment. As shown in FIG. 6, when an instruction to start measurement is input (step S101), the eyeball searching unit 33 resets the counter CNT1 (step S102), and acquires the face image IM captured by the imaging device 21. (Step S103).

Based on the acquired face image IM, the eyeball search unit 33 searches for left and right eyeball images, and the distance calculation unit 32 calculates the inter-subject apparatus distance D1 (step S104). When the eyeball search unit 33 has successfully detected both the left and right eyeball images E (Yes in step S105), the distance calculation unit 32 determines whether or not the distance L2 between the eyeball images has been obtained (step S106), If it is determined that it has not been acquired (No in step S106), the eyeball image distance L2 is acquired (step S107). In step S107, when the inter-eye distance L1 of the subject is stored in advance in the storage unit 35, the distance calculation unit 32 calculates the inter-eyeball image distance L2 based on the inter-eye distance L1. Further, when the inter-eye distance L1 is not stored, the distance calculation unit 32 calculates the inter-eye distance L1 and the inter-eye image distance L2 based on the detection results of both eyeball images E. The distance calculation unit 32 causes the storage unit 35 to store the obtained inter-eyeball image distance L2. If the interocular distance L1 of the subject is not stored in advance in the storage unit 35, the interocular distance L1 is also stored in the storage unit 35 (step S108). After that, the process proceeds to the following step S117.

On the other hand, if both eyeball images E have not been successfully detected (No in step S105), it is determined whether detection has failed for both eyeball images E (step S109). If detection fails for both eyeball images E (Yes in step S109), the eyeball search unit 33 increments the value of the counter CNT1 by +1 (step S110), and determines whether the value of the counter CNT1 is equal to or greater than a specified value. Determine (step S111). When it is determined that the value of the counter CNT1 is equal to or greater than the specified value (Yes in step S111), the eyeball searching unit 33 searches for one of the left and right eyeball images E in the face image IM (step S112). If one of the left and right eyeball images E is detected as a result of the search (Yes in step S113), the process proceeds to the following step S114.

If the value of the counter CNT1 is not equal to or greater than the specified value in step S111 (No in step S111), or if the eyeball image E is not detected in step S113 (No in step S113), the following processing in step S119 is performed. proceed to

When one eyeball image E is detected, the eyeball searching unit 33 predicts the eyeball position of the other undetected eyeball image E based on the inter-subject apparatus distance D1 and the eyeball image distance L2 (step S114). ). In the prediction in step S114, the eyeball searching unit 33 locates the first search areas DA1 and DA2 at positions on a virtual straight line parallel to the left-right direction with respect to one detected eyeball image E in the acquired face image IM. set. The eyeball search unit 33 searches for the other eyeball image E in the predicted first search areas DA1 and DA2 (step S115).

If the other eyeball image E is detected as a result of the search (Yes in step S116), or after step S108 is performed, the eyeball searching unit 33 stores the eyeball positions of the detected left and right eyeball images E. It stores it in the unit 35 (step S117), and resets the counter CNT1 (step S118). On the other hand, if the other eyeball image E is not detected (No in step S116), the process proceeds to the following step S119.

After that, when an instruction to end measurement is input (step S119), the process ends. If no instruction to end measurement is input (No in step S119), the process returns to step S103.

As described above, the line-of-sight detection apparatus 100 according to the present embodiment includes the imaging device 21 that acquires the face image IM including the left and right eyeball images E of the subject, and the inter-subject device distance D1 between the subject and the imaging device 21. and a distance calculation unit 32 that searches for the left and right eyeball images E of the subject from the face image IM, and when one of the left and right eyeball images E of the subject is detected by the search, the search for one eyeball image E An eyeball search unit 33 searches for the other eyeball image E of the subject from the face image IM based on the result, the distance D1 between the subject apparatuses, and the distance L2 between the eyeball images between the left and right eyeballs of the subject. Further, the line-of-sight detection apparatus 100 according to the present embodiment detects the left and right eyeball images E of the subject, and detects the line of sight of the subject based on the detection results.

The eyeball detection method according to the present embodiment includes acquiring a face image IM including left and right eyeball images E of a subject, calculating an inter-subject-apparatus distance D1 between the subject and the imaging apparatus 21, Searching left and right eyeball images E from IM, and if one of the left and right eyeball images E of the subject is detected by the search, the search result for the one eyeball image E, the inter-subject apparatus distance D1, and the subject searching for the other eyeball image E of the subject from the face image IM based on the inter-eyeball image distance L2 between the left and right eyeballs.

The eyeball detection program according to the present embodiment includes a process of acquiring a face image IM including left and right eyeball images E of a subject, a process of calculating a subject-apparatus distance D1 between the subject and the imaging apparatus 21, and a process of calculating a face image A process of searching left and right eyeball images E from the IM, and when one of the left and right eyeball images E of the subject is detected by the search, the search result for the one eyeball image E, the inter-subject apparatus distance D1, and the subject and a process of searching for the other eyeball image E of the subject from the face image IM based on the inter-eyeball image distance L2 between the left and right eyeballs.

In this embodiment, when one of the left and right eyeball images E of the subject is detected, the search result for the one eyeball image E, the inter-subject apparatus distance D1, and the inter-eyeball image distance between the left and right eyeballs of the subject Since the other eyeball image E of the subject is searched from the face image IM based on L2, the processing time can be shortened compared to the case where both the left and right eyeball images E are detected. Therefore, it is possible to shorten the processing time for detecting the eyeball of the subject.

In the line-of-sight detection apparatus 100 according to the present embodiment, the eyeball search unit 33 calculates the eyeball position of one eyeball image E based on one eyeball image E, and calculates the eyeball position of the one eyeball image E and The eyeball position of the other eyeball image E is predicted based on the distance D1 and the inter-eyeball image distance L2, and a search is performed based on the prediction result. Thereby, the search accuracy of the other eyeball image E is improved.

In the line-of-sight detection device 100 according to this embodiment, the distance calculator 32 calculates the inter-eyeball image distance L2 based on the face image IM acquired by the imaging device 21 . This makes it possible to obtain the inter-eyeball image distance L2 even in a state in which the interocular distance L1 of the subject is not obtained in advance.

In the line-of-sight detection device 100 according to the present embodiment, the distance calculation unit 32 calculates the inter-subject device distance D1 based on the face image acquired by the imaging device 21 . Accordingly, it is possible to efficiently calculate the inter-subject apparatus distance D1 without the need to separately provide a means or the like for measuring the inter-subject apparatus distance D1.

The technical scope of the present invention is not limited to the above embodiments, and modifications can be made as appropriate without departing from the scope of the present invention. For example, in the above-described embodiment, if the other eyeball image E is not detected in the first search areas DA1 and DA2 (see FIG. 5), the eyeball search unit 33 searches the first search areas DA1 and DA2 for one A second search area shifted by a predetermined angle in the direction of rotation about the eyeball position of the eyeball image E can be set, and a search can be performed for the second search area.

7 and 8 are diagrams showing another example of the operation of predicting the eyeball position of the other eyeball image E. FIG. As shown in FIG. 7, the eyeball search unit 33 moves the second search areas DA1 and DA2 counterclockwise around the right eyeball image ER of the subject H2 by an angle θ2. Search areas DA3 and DA4 are set, and the second search areas DA3 and DA4 are searched. In this case, the distances between the right eyeball image ER and the second search areas DA3 and DA4 are equal to the inter-eyeball image distance L2.

When the other eyeball image E is not searched in the second search areas DA3 and DA4 shown in FIG. A second search area is set at a position shifted by an angle θ2 in the direction of , and the second search area is searched. After that, if the other eyeball image E is not searched in the newly set second search area, the eyeball searching unit 33 moves the second search area in the counterclockwise direction around the right eyeball image ER. A new second search area is set at a position shifted by an angle of θ2, and the second search area is searched. The eyeball search unit 33 can set the second search areas within the range of a predetermined angle or less in the counterclockwise direction around the right eyeball image ER with respect to the first search areas DA1 and DA2.

Further, even if a second search area is set in a range of a predetermined angle in a counterclockwise direction centered on the right eyeball image ER with respect to the first search areas DA1 and DA2 and a search is performed, the other eyeball image E is not retrieved, the eyeball search unit 33 shifts the first search areas DA1 and DA2 clockwise from the right eyeball image ER by an angle θ3 as shown in FIG. Two search areas DA5 and DA6 are set, and the second search areas DA5 and DA6 are searched. In this case, the distances between the right eyeball image ER and the second search areas DA5 and DA6 are equal to the inter-eyeball image distance L2.

When the other eyeball image E of the subject H3 is not searched in the second search areas DA5 and DA6 shown in FIG. A second search area is set at a position shifted by an angle θ3 in the counterclockwise direction, and the second search area is searched. After that, if the other eyeball image E is not searched in the newly set second search area, the eyeball searching unit 33 moves the second search area in the counterclockwise direction around the right eyeball image ER. A new second search area is set at a position shifted by an angle of θ3, and the second search area is searched. The eyeball searching unit 33 can set second search areas within a range of a predetermined angle or less in the clockwise direction around the right eyeball image ER with respect to the first search areas DA1 and DA2. 7 and 8, the second search area is first set in the counterclockwise direction, and then the second search area is set in the clockwise direction. not. Contrary to the above, the second search area may first be set in the clockwise direction, and then the second search area may be set in the counterclockwise direction.

FIG. 9 is a flowchart showing an eyeball detection method according to a modification. As shown in FIG. 9, steps other than step S116 are the same as in the above embodiment. As a result of the determination in step S116, when it is determined that the other eyeball image E has been detected in the first search areas DA1 and DA2 (Yes in step S116), the operations after step S117 are performed as in the above embodiment.

On the other hand, when it is determined that the other eyeball image E is not detected in the first search areas DA1 and DA2 (No in step S116), the eyeball searching unit 33 searches the first search areas DA1 and DA2 for the right eyeball image A search is performed by setting a second search area within a range of a predetermined angle in the counterclockwise or clockwise direction around the ER (step S120). In step S120, when the second search area is moved by a predetermined angle or more with respect to the first search areas DA1 and DA2 (step S121), the processing from step S119 is performed. Also, in step S120, if the movement of the second search area with respect to the first search areas DA1 and DA2 is less than the predetermined angle (step S121), the process after step S119 is performed.

In this way, when the other eyeball image E is not detected in the first search areas DA1 and DA2, the eyeball searching unit 33 rotates the first search areas DA1 and DA2 in the rotational direction around the one eyeball image E. By setting a second search area shifted by a predetermined angle and performing a search for the second search area, when the subject tilts his or her face, the other eyeball image E is rotated with respect to the other eyeball image E. It is possible to detect the other eyeball image E even when there is a deviation. Thereby, the detection accuracy of the eyeball image E can be improved.

Further, in the above-described embodiment, the configuration in which the eyeball detection device is applied to the line-of-sight detection device 100 has been described as an example, but the present invention is not limited to this. The eyeball detection device can be applied to other devices such as a face image recognition device that acquires a face image of a subject and detects an eyeball image from the face image.

Further, in the above-described embodiment, an example of calculating the inter-subject device distance D1 from the images of the first camera 21A and the second camera 21B of the imaging device 21 has been described, but the method is not limited to this method. may be used to measure the distance.

θ2, θ3... Angle, D... First distance, D1... Distance between test subject devices, DA1, DA2... First search area, DA3, DA4, DA5, DA6... Second search area, E... Eyeball image, EB... Eyeball, EL... left eye image, H, H2, H3... subject, L1... distance between eyes, L2... distance between eye images, M... virtual straight line, IM... face image, ER... right eye image, EBL... left eye, EBR... Right eye CNT1 Counter 10 Display device 11 Display unit 20 Image acquisition device 21 Imaging device 21A First camera 21B Second camera 22 Lighting device 22A First light source , 22B... second light source, 30... computer system, 30A... arithmetic processing unit, 30B... storage device, 30C... computer program, 31... display control unit, 32... distance calculation unit, 33... eye search unit, 34... gaze point Detector 35 Storage unit 40 Output device 50 Input device 60 Input/output interface device 100 Line of sight detection device

Claims (8)

an imaging device that acquires a face image including left and right eyeball images of a subject;
a first distance calculation unit that calculates a first distance between the subject and the imaging device;
the left and right eyeball images are searched from the face image, and when one of the left and right eyeball images of the subject person is detected by the search, a search result for the one eyeball image and the first distance; and an eyeball searching unit that searches for the other eyeball image of the subject from the face image based on a second distance between the left and right eyeballs of the subject. The eyeball search unit calculates an eyeball position based on the one eyeball image, and calculates the eyeball position of the other eyeball image based on the eyeball position of the one eyeball image, the first distance, and the second distance. The eyeball detection device according to claim 1, wherein an eyeball position is predicted, and the search is performed based on the prediction result. The eyeball search unit sets a first search area at a position on a virtual straight line parallel to the left-right direction with respect to the one eyeball image, performs the search for the first search area, and performs the search in the first search area. When the other eyeball image is not detected, a second search area shifted by a predetermined angle in a rotational direction around the position of the one eyeball image is set with respect to the first search area, and the second search area is The eyeball detection device according to claim 2, wherein the search is performed. The eyeball detection device according to any one of claims 1 to 3, further comprising a second distance calculator that calculates the second distance based on the face image acquired by the imaging device. The eyeball detection device according to any one of claims 1 to 4, wherein the first distance calculator calculates the first distance based on the face image acquired by the imaging device. A line-of-sight detecting apparatus for detecting left and right eyeball images of a subject by the eyeball detecting apparatus according to any one of claims 1 to 5, and detecting a line of sight of the subject based on the detection results. Acquiring a face image including left and right eyeball images of a subject by an imaging device ;
calculating a first distance between the subject and the imaging device;
searching the left and right eyeball images from the face image;
When one of the left and right eyeball images of the subject is detected by the search, the search result for the one eyeball image, the first distance, and a second distance between the left and right eyeballs of the subject. and searching the subject's other eyeball image from the face image based on. A process of acquiring a face image including the left and right eyeball images of the subject by an imaging device ;
a process of calculating a first distance between the subject and the imaging device;
a process of searching the left and right eyeball images from the face image;
When one of the left and right eyeball images of the subject is detected by the search, the search result for the one eyeball image, the first distance, and a second distance between the left and right eyeballs of the subject. and a process of searching for the other eyeball image of the subject person from the face image based on the above.

Images