JP2018205819A - Gazing position detection computer program, gazing position detection device, and gazing position detection method - Google Patents

Abstract

To provide a gazing position detection computer program capable of estimating an interpupillary distance of a user.SOLUTION: A gazing position detection computer program includes a command for causing the computer to execute: determining whether a user face is straightly directed to an imaging unit on the basis of an image generated by the imaging unit, detecting pupils of a left eye and a right eye from an image where the user face is straightly directed to the imaging unit, detecting directions of visual lines of the left eye and the right eye on the image, calculating positions of the pupil of the left eye and the pupil of the right eye in a real space on the basis of a positional relation between the imaging unit according to the positions of the pupil of the left eye and the pupil of the right eye on the image where the user face is straightly directed to the imaging unit, and the left eye and the right eye, directions of the visual lines of the left eye and the right eye, and a positional relation between the imaging unit and a gazing object plane, and calculating the interpupillary distance of the left eye and the right eye in the real space on the basis of the positions of the pupil of the left eye and the pupil of the right eye in the real space.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a gaze position detection computer program, a gaze position detection apparatus, and a gaze position detection method for detecting a gaze position based on, for example, an image obtained by photographing an eye.

  By detecting the direction of the person's line of sight, it is considered to obtain information about what the person is interested in, or to automatically perform operations related to the direction of the line of sight or the position where the user is gazing. ing. Hereinafter, for the sake of convenience, a position where a person is gazing is simply referred to as a gazing position.

  A person's gaze position or line-of-sight direction is closely related to the direction of the person's pupil. Therefore, a technique is known in which a pupil is detected from an image obtained by photographing a human eye with a camera, and the gaze direction or gaze position of the person is specified based on the positional relationship between the pupil and a reference point.

  Even if the line-of-sight direction is constant, the gaze position varies depending on the distance from the camera to the person. Therefore, in order to accurately detect the gaze position, it is preferable to accurately obtain the distance from the camera to the person. Therefore, based on the distance between the pupils of both eyes (hereinafter simply referred to as the interpupillary distance), the distance between the pupils in the real space, the focal length of the camera, etc. There has been proposed a technique for calculating a viewpoint distance at (see, for example, Patent Document 1).

JP 2014-89304 A

  However, the interpupillary distance varies from person to person. Therefore, when the user whose gaze position is to be detected is an unspecified person, the distance between the pupils in the real space is unknown, and therefore the distance from the camera to the person cannot be calculated accurately. As a result, the gaze position detection accuracy may be insufficient.

  In one aspect, an object of the present invention is to provide a computer program for gaze position detection that can estimate a user's interpupillary distance.

  According to one embodiment, a computer program for gaze position detection is provided. This computer program for gaze position detection is based on an image generated by the imaging unit and showing at least a part of the user's face including the left eye and the right eye. The left eye and the right eye pupil are detected from the image when the user's face is directly facing the imaging unit, and the line of sight of the left eye is determined based on the left eye on the image. And detecting the gaze direction of the right eye based on the right eye on the image, and the position of the left eye pupil and the right eye pupil on the image when the user's face is facing the imaging unit Based on the positional relationship between the imaging unit and the left eye and the right eye according to the position of the left eye, the gaze direction of the left eye and the gaze direction of the right eye, and the positional relationship between the imaging unit and the surface to be watched by the user Calculate the left eye pupil position and right eye pupil position in real space, From the positions of the pupil and the position of the pupil of the right eye, and calculates the interpupillary distance between the left and right eyes in real space, comprising instructions for causing a computer to execute the.

  The distance between the pupils of the user can be estimated.

It is a hardware block diagram of the digital signage system which is one Embodiment of a gaze position detection apparatus. It is a figure which shows an example of arrangement | positioning of a camera and a display apparatus. It is a functional block diagram regarding a distance estimation process and a gaze position detection process of a pupil of a processor. It is explanatory drawing of the principle about interpupillary distance estimation. (A) is a figure which shows an example of the image of a face in case a user's face is facing with respect to a camera, (b) is a face of the case where a user inclines a face with respect to a camera It is a figure which shows an example of an image. (A) And (b) is a figure which shows an example of the relationship between a display apparatus and a camera, and the position of a pupil, and a gaze position. It is an operation | movement flowchart of the distance estimation process between pupils. It is explanatory drawing of the principle about gaze position detection. It is an operation | movement flowchart of a gaze position detection process. It is a figure which shows an example of the relationship between a display apparatus and a camera, a user's pupil gravity center, and gaze position by a modification.

Hereinafter, the gaze position detection device will be described with reference to the drawings.
The gaze position detection device captures both eyes of the user with a camera to generate an image, and detects the pupils of the left and right eyes and the gaze direction of the left and right eyes on the image. Here, when the user's face is facing the camera, that is, when facing the front of the camera, the distance from the camera to the left eye in the direction along the optical axis direction of the camera and the camera The condition that the distance to the right eye is the same is satisfied. When this condition is satisfied, the pupil position of both eyes in real space is uniquely determined based on the line-of-sight direction of both eyes and the direction from the camera toward the pupil of each eye. Therefore, this gaze position detection device is based on the line-of-sight direction for each eye calculated from the image when the user's face is facing the camera and the pupil position of each eye on the image. The pupil position and interpupillary distance of each eye in space are calculated. Then, once the interpupillary distance is calculated, the gaze position detection device calculates the gaze position based on the gaze direction of both eyes detected from the image obtained thereafter and the interpupillary distance in real space. To detect.

  In the embodiment described below, the gaze position detection device is mounted on a digital signage system, and the gaze position detection device detects the gaze position of the user of the digital signage system.

  FIG. 1 is a hardware configuration diagram of a digital signage system which is an embodiment of a gaze position detection device. The digital signage system 1 includes a display device 2, a camera 3, an input device 4, a storage medium access device 5, a memory 6, and a processor 7. Further, the digital signage system 1 may have a light source (not shown) such as an infrared light emitting diode for illuminating the eyes of the user of the digital signage system 1. Furthermore, the digital signage system may have a communication interface circuit (not shown) for connecting the digital signage system 1 to another device.

  The display device 2 includes, for example, a liquid crystal display or an organic electroluminescence display. The display device 2 displays, for example, various texts, icons, still images, or moving images according to the video signal from the processor 7.

  The camera 3 is an example of an imaging unit, and generates an image including both eyes and at least a part of the user's face. For this purpose, the camera 3 includes an image sensor having solid-state image sensors arranged in a two-dimensional manner and an imaging optical system that forms an image of a subject on the image sensor. The imaging optical system may be a single focus optical system or a variable focus optical system. The camera 3 shoots at a predetermined frame rate to generate an image during execution of the inter-pupil distance estimation process or during the gaze position detection process. The camera 3 has a resolution that allows the pupil of the user who uses the digital signage system 1 to be identified on the image. Each time the camera 3 generates an image, the camera 3 passes the image to the processor 7. In the following description, “image” represents an image generated by the camera 3 and including both eyes, at least a part of the user's face, unless otherwise specified.

  FIG. 2 is a diagram illustrating an example of the arrangement of the camera 3 and the display device 2. In this example, the camera 3 is attached to the lower side of the display screen 2 a of the display device 2. In addition, the display screen 2a of the display device 2 is an example of a surface (hereinafter referred to as a gaze target surface) that is a gaze target by the user. The optical axis OA of the camera 3 is displayed on the display screen 2a so that the entire face or part of the face of the user 200 including both eyes of the user 200 viewing the display screen 2a of the display device 2 can be photographed. It is installed so as to be parallel to the normal direction n.

  The camera 3 may be provided so as to be arranged in a direction different from the direction in which the user's eyes are arranged (for example, the horizontal direction) with respect to the gaze target surface. For example, the camera 3 may be provided above the display screen 2a. Good.

  The input device 4 includes, for example, a keyboard and a pointing device such as a mouse. An operation signal input by the user via the input device 4 is passed to the processor 7.

  The display device 2 and the input device 4 may be integrated like a touch panel display, for example. In this case, when the user touches the position of the icon displayed on the display screen of the display device 2, the input device 4 generates an operation signal corresponding to the position and outputs the operation signal to the processor 7. .

  The storage medium access device 5 is an example of a storage unit, and is a device that accesses the storage medium 8 such as a magnetic disk, a semiconductor memory card, and an optical storage medium. The storage medium access device 5 reads, for example, a computer program for pupil distance estimation processing and gaze position detection processing, which is stored in the storage medium 8 and executed on the processor 7, and passes it to the processor 7.

  The memory 6 is another example of the storage unit, and includes, for example, a readable / writable nonvolatile semiconductor memory and a readable / writable volatile semiconductor memory. The memory 6 stores a computer program for pupil distance estimation processing and gaze position detection processing, various application programs, and various data executed on the processor 7.

  Furthermore, the memory 6 may store various data used for calculating the distance between the pupils of the user and various data used for detecting the user's gaze position. For example, the memory 6 stores the positions of the display device 2 and the camera 3 in real space, the focal length of the camera 3, and the like. Furthermore, the memory 6 stores the calculated interpupillary distance.

  The processor 7 includes, for example, at least one Central Processing Unit (CPU) and its peripheral circuits. Further, the processor 7 may include a numerical arithmetic processor or a graphics processing unit (GPU). The processor 7 is connected to each part of the digital signage system 1 via a signal line, and controls the entire digital signage system 1. For example, the processor 7 causes the display device 2 to display a predetermined moving image or the like according to the operation signal received from the input device 4 and the application program being executed.

  Further, when the processor 7 detects that the user has approached the digital signage system 1, the processor 7 first obtains the distance between the pupils of the user in the real space. Then, once the interpupillary distance is obtained, the processor 7 uses the interpupillary distance to detect the user's gaze direction and gaze position by executing a gaze position detection process each time an image is obtained. .

FIG. 3 is a functional block diagram related to the interpupillary distance estimation processing and gaze position detection processing of the processor 7. The processor 7 includes a pupil detection unit 21, a line-of-sight detection unit 22, a face orientation determination unit 23, an inter-pupil distance calculation unit 24, and a gaze position detection unit 25.
Each of these units included in the processor 7 is a functional module realized by a computer program executed on the processor 7. Each of these units included in the processor 7 may be mounted as a part of a circuit in the processor 7 that realizes the function of each unit.

  Among these units included in the processor 7, the pupil detection unit 21, the line-of-sight detection unit 22, and the face orientation determination unit 23 are used in both the inter-pupil distance estimation process and the gaze position detection process. The interpupillary distance calculation unit 24 is used in the interpupillary distance estimation process. On the other hand, the gaze position detection unit 25 is used in gaze position detection processing. Hereinafter, each part which the processor 7 has is demonstrated about each of the distance estimation process between pupils, and a gaze position detection process.

(Interpupillary distance estimation processing)
In this embodiment, the processor 7 executes background difference processing between the image and the background image stored in the memory 6 every time an image is obtained from the camera 3 until the pupil distance estimation processing is started. Then, a pixel whose absolute difference value between corresponding pixels is a predetermined value or more is extracted. Then, the processor 7 starts the inter-pupil distance estimation process when the number of extracted pixels exceeds a predetermined number.
The digital signage system 1 may have a proximity sensor (not shown). Then, when the proximity sensor detects an object located in front of the camera 3, the processor 7 may start the inter-pupil distance estimation process.

  First, the principle for detecting the interpupillary distance will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a positional relationship between the display device 2 and the camera 3 and the user's pupil viewed from above along the vertical direction of the imaging surface 400 of the camera 3 in real space. Since the position of the pupil on the image represents the direction from the camera 3 toward the pupil, the direction from the camera 3 to the pupil can be known by detecting the pupil on the image. That is, based on the position of the pupil on the image, the direction from the camera 3 toward the pupil of the left eye of the user indicated by the arrow 401 and the direction from the camera 3 to the pupil of the right eye of the user indicated by the arrow 402 are directed. A direction is required.

  In addition, since it is assumed that the user looks at the gaze position on the display screen of the display device 2 with both eyes, the gaze direction of the left eye indicated by the arrow 403 and the gaze direction of the right eye indicated by the arrow 404 Intersect with the gaze position. Therefore, the intersection of the straight line along the line of sight of the left eye when viewed from the gaze position and the line along the direction toward the pupil of the left eye when viewed from the camera 3 is the intersection of the pupil of the left eye in real space. Position. Similarly, the intersection of the straight line along the line of sight of the right eye when viewed from the gaze position and the line along the direction of the right eye when viewed from the camera 3 is the pupil of the right eye in real space. It becomes the position. However, the gaze position is not known in advance. Therefore, for each gaze position candidate (four gaze position candidates 411 to 414 are shown in FIG. 4), a candidate combination of the left eye pupil position and the right eye pupil position (FIG. 4). Then, candidates 421 to 424) are obtained.

Here, when the user is facing the camera 3, the distance from the camera 3 to the left eye and the distance from the camera 3 to the right eye along the direction of the optical axis OA of the camera 3 are equal. Of the combinations of the positions of the left eye pupil and the right eye pupil, the distance from the camera 3 to the left eye is equal to the distance from the camera 3 to the right eye (candidate 422 in FIG. 4). Is uniquely determined. Therefore, when the user is facing the camera 3, the position Pr _{left in} the real space of the pupil of the left eye and the position Pr _{right in} the real space of the pupil of the right eye are obtained, and as a result, The pupil distance pp in the real space is calculated.

  Therefore, in the present embodiment, the pupil detection unit 21 detects the respective pupils of the user's left eye and right eye from the image, and the line-of-sight detection unit 22 determines the left eye based on the left eye and the right eye on the image. An eye gaze direction and a right eye gaze direction are detected. Further, the face orientation determination unit 23 determines whether or not the user's face is facing the camera 3. Then, the interpupillary distance calculation unit 24 performs the left eye gaze direction and the right eye gaze direction when the user's face is facing the camera 3, and the direction from the camera 3 toward the left eye pupil and the right eye. The interpupillary distance is calculated based on the direction toward the pupil.

  The pupil detection unit 21 detects the pupils of the user's left eye and right eye on the image every time an image is obtained until the interpupillary distance is calculated during execution of the interpupillary distance estimation process. Note that the pupil detection unit 21 only needs to execute the same processing for all images, and therefore, processing for one image will be described below.

  The pupil detection unit 21 first detects an area (hereinafter simply referred to as an eye area) in which each of the user's left and right eyes is reflected from the image. For this purpose, the pupil detection unit 21 detects the eye region using, for example, a discriminator that has been learned in advance so as to detect the eye region from the image. In this case, for example, Adaboost or Real Adaboost, a support vector machine, or a deep neural network is used as the discriminator. The pupil detection unit 21 sets a window on the image, and inputs the value of each pixel in the window or the feature amount extracted from the window to the discriminator while changing the position of the window. It is determined whether or not the eye area. Further, for example, a Haar-like feature quantity or a Histograms of Oriented Gradients feature quantity is extracted as the feature quantity.

  Alternatively, the pupil detection unit 21 may detect two regions that most closely match the template by template matching between the template representing the eye and the image, and each of the two detected regions may be used as the eye region. Alternatively, the pupil detection unit 21 may detect the eye area according to any of various other methods for detecting the eye area shown on the image.

  When the eye area is detected, the pupil detection unit 21 detects the pupil in the respective eye areas of the left and right eyes.

  For example, in order to detect a pupil, the pupil detection unit 21 performs template matching between a template corresponding to the pupil and an eye region, and detects a region having the highest degree of matching with the template in the eye region. If the highest coincidence value is higher than a predetermined coincidence threshold value, the pupil detection unit 21 determines that the detected region is a pupil region in which the pupil is reflected. Note that the size of the pupil changes according to the brightness around the user. Moreover, although the pupil is substantially circular, when the camera 3 is viewing the pupil from an oblique direction, the shape of the pupil on the image is an elliptical shape that is long in the vertical direction. Therefore, a plurality of templates may be prepared according to the size or shape of the pupil. In this case, the pupil detection unit 21 executes template matching between each template and the eye region, and obtains the highest matching degree. If the highest coincidence value is higher than the coincidence threshold, the pupil detection unit 21 determines that the region overlapping the template corresponding to the highest coincidence value is a pupil region. Note that the degree of coincidence is calculated as, for example, a normalized cross-correlation value between a template and a region overlapping the template. The coincidence threshold is set to 0.7 or 0.8, for example.

  The luminance of the region where the pupil is reflected is lower than the luminance of the surrounding region, and the pupil is substantially circular. Accordingly, the pupil detection unit 21 sets a double ring filter having two rings with different radii concentrically within the eye region. When the difference value obtained by subtracting the average luminance value of the inner pixels from the average luminance value of the pixels corresponding to the outer ring is larger than a predetermined threshold, the pupil detection unit 21 is surrounded by the inner ring. The region may be a pupil region. In addition, the pupil detection unit 21 may add to the condition for detecting the pupil region that the average luminance value of the region surrounded by the inner ring is equal to or less than a predetermined threshold value. In this case, for example, the predetermined threshold is set to a value obtained by adding 10% to 20% of the difference between the maximum luminance value and the minimum luminance value in the eye region to the minimum luminance value.

  Note that the pupil detection unit 21 may detect the pupil region using any of various other methods for detecting a region in which the pupil is shown on the image.

  The pupil detection unit 21 calculates, for each of the left and right eyes of the user, the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the pupil region, as the center of the pupil region (hereinafter simply referred to as the pupil center of gravity). As the position coordinates).

  The pupil detection unit 21 notifies the line-of-sight detection unit 22 of information representing the eye area and the position coordinates of the center of gravity of the pupil for each of the left and right eyes of the user. The pupil detection unit 21 notifies the face orientation determination unit 23 and the interpupillary distance calculation unit 24 of the position coordinates of the pupil center of gravity for the left and right eyes of the user.

  When the pupil center of gravity is detected for each of the left and right eyes of the user until the interpupillary distance is calculated during execution of the interpupillary distance estimation process, the line-of-sight detection unit 22 Detect gaze direction. Note that the line-of-sight detection unit 22 only needs to execute the same process for the left eye and the right eye, and therefore, the process for one eye will be described below.

  The line-of-sight detection unit 22 detects the line-of-sight direction based on, for example, an iris on the image. In this case, the line-of-sight detection unit 22 is, for example, Nishiuchi et al., “Study on non-contact line-of-sight detection method using elliptical approximation of iris contour”, Japan Society of Mechanical Engineers, Japan Society of Mechanical Engineers, Volume C, Volume 69, 682 No., pp. 1611-1617, 2003, may be used to detect the line-of-sight direction.

  In this case, the line-of-sight detection unit 22 first extracts an iris outline from the eye region. In general, the luminance of the pixel corresponding to the iris is lower than the luminance of the pixel corresponding to the periphery of the iris (so-called white-eye). Further, the contour of the iris (hereinafter simply referred to as the “iris contour”) has a substantially circular shape centered on the center of the pupil. Therefore, the line-of-sight detection unit 22 detects edge pixels using, for example, a Canny filter or a Sobel filter along each of a plurality of scanning lines set radially from the center of gravity of the pupil. Then, the line-of-sight detection unit 22 extracts a contour line obtained by performing ellipse approximation on the detected set of edge pixels as an iris contour. At this time, the line-of-sight detection unit 22 may extract an iris outline from edge pixels whose distance from the pupil center of gravity is within a predetermined range. The predetermined range corresponds to, for example, the maximum value of the iris size assumed on the image, with the distance from the center of the pupil to the iris contour corresponding to the minimum value of the iris size assumed on the image as a lower limit. The distance from the center of the pupil to the iris contour is set to an upper limit.

  Alternatively, the line-of-sight detection unit 22 may extract the iris contour according to any of various other methods for extracting the iris contour from the image.

Based on the ratio of the short axis R _B to the long axis R _A of the ellipse that approximates the iris contour on the image, the line-of-sight detection unit 22 is a line connecting the origin O and the center C of the eyeball in the camera coordinate system. The angle η formed by the line-of-sight vector representing the line-of-sight direction is calculated according to the following equation.

ここで、角度αは、視線ベクトルを、カメラ座標系での光軸に沿った方向と画像の水平方向に対応するカメラ座標系での方向とを含む面へ射影したときの、視線ベクトルと光軸に沿った負の方向（ユーザからカメラ３へ向かう方向）とがなす角である。なお、以下では、カメラ座標系での光軸に沿った方向をz軸方向と呼び、画像の水平方向に対応するカメラ座標系での方向をx軸方向と呼ぶ。また角度βは、視線ベクトルを、z軸方向と画像の垂直方向に対応するカメラ座標系での方向（以下、y軸方向と呼ぶ）とを含む面へ射影したときの、視線ベクトルとz軸の負の方向とがなす角である。また、γ_xは、xz平面における、光軸（すなわち、z軸）と眼球中心とがなす角度であり、γ_yは、yz平面における、光軸と眼球中心とがなす角度である。なお、γ_x及びγ_yは、画像上での瞳孔重心の位置から算出される、xz平面及びyz平面のそれぞれにおける、光軸と瞳孔重心とがなす角度により近似的に算出されてもよい。 Then, the line-of-sight detection section 22, and the angle eta, based on the angle ζ formed by the vertical axis of the minor axis R _B and the image of the ellipse approximating the iris outline, according to the following equation, calculates a unit vector e of the line-of-sight vector To do.

Here, the angle α is the line-of-sight vector and light when the line-of-sight vector is projected onto a plane including a direction along the optical axis in the camera coordinate system and a direction in the camera coordinate system corresponding to the horizontal direction of the image. It is an angle formed by a negative direction (direction from the user toward the camera 3) along the axis. In the following, the direction along the optical axis in the camera coordinate system is referred to as the z-axis direction, and the direction in the camera coordinate system corresponding to the horizontal direction of the image is referred to as the x-axis direction. The angle β is the line-of-sight vector and the z-axis when the line-of-sight vector is projected onto a plane including the z-axis direction and the direction in the camera coordinate system corresponding to the vertical direction of the image (hereinafter referred to as the y-axis direction). The angle formed by the negative direction of. Further, γ _x is an angle formed by the optical axis (that is, the z axis) and the eyeball center in the xz plane, and γ _y is an angle formed by the optical axis and the eyeball center in the yz plane. Note that γ _x and γ _y may be approximately calculated by the angle formed by the optical axis and the pupil centroid in each of the xz plane and the yz plane, which is calculated from the position of the pupil centroid on the image.

According to the modification, when the digital signage system 1 has a light source that illuminates the user's eyes, the line-of-sight detection unit 22 is a cornea reflection image (hereinafter referred to as a Purkinje image) of the light source on the image. The line-of-sight direction may be detected based on the positional relationship between the position and the pupil center of gravity.
In this case, the line-of-sight detection unit 22 detects a Purkinje image in the eye region by, for example, template matching. Then, the line-of-sight detection unit 22 specifies the line-of-sight direction corresponding to the position of the Purkinje image and the pupil center of gravity, for example, with reference to a reference table representing the relationship between the relative positional relationship between the Purkinje image and the pupil center of gravity and the line-of-sight direction. do it. The reference table may be created in advance according to the positional relationship between the camera 3 and the light source and stored in the memory 6, for example.

  The line-of-sight detection unit 22 notifies the interpupillary distance calculation unit 24 of the line-of-sight directions detected for each of the user's left eye and right eye.

  The face orientation determination unit 23 determines whether or not the user's face is facing the camera 3.

  In this embodiment, since the camera 3 is arrange | positioned under the display screen of the display apparatus 2, it is assumed that the camera 3 has faced the lower side rather than the user's face. Therefore, if the user is facing the camera 3, the vertical position of the left-eye pupil centroid on the image is substantially equal to the vertical position of the right-eye pupil centroid. On the other hand, if the user tilts his / her face with respect to the camera 3, the angle formed by the optical axis of the camera 3 and the direction from the camera 3 toward the eye closer to the camera 3 is the optical axis of the camera 3 and the camera This is larger than the angle formed by the direction from the camera 3 to the eye far from 3. As a result, the pupil centroid of the eye closer to the camera 3 is positioned above the pupil centroid of the eye farther from the camera 3 along the vertical direction on the image.

  FIG. 5A is a diagram illustrating an example of a face image when the user's face is facing the camera 3, and FIG. 5B is a view in which the user tilts the face with respect to the camera 3. It is a figure which shows an example of the image of the face in the case of being.

  As shown in FIG. 5A, when the user's face 501 is directly facing the camera 3, the vertical position of the left eye 502 and the vertical position of the right eye 503 on the image 500. Are almost identical. On the other hand, as shown in FIG. 5B, when the user's face 511 is inclined with respect to the camera 3, the eye closer to the camera 3 on the image 510 (in this example, the left eye) 512. Also, the eye far from the camera 3 (in this example, the right eye) 513 is positioned on the lower side. Thus, the face orientation determination unit 23 can determine whether or not the user's face is facing the camera 3 by comparing the positions of the right eye and the left eye in the vertical direction on the image.

  Therefore, the face orientation determination unit 23 calculates the absolute value of the difference between the vertical position of the left eye's pupil centroid on the image and the vertical position of the right eye's pupil centroid, and determines the absolute value of the difference as a confrontation. Compare with a threshold (eg, 1-2). The face orientation determination unit 23 determines that the user's face is facing the camera 3 if the absolute value of the difference is equal to or less than the facing determination threshold. On the other hand, if the absolute value of the difference is larger than the confrontation determination threshold, the face orientation determination unit 23 determines that the user's face is not facing the camera 3. Then, the face orientation determination unit 23 determines that the left eye and the right eye, in which the vertical position of the center of gravity of the pupil is located below, is farther from the camera 3. For example, if the pupil centroid of the left eye on the image is positioned below the pupil centroid of the right eye, the face orientation determination unit 23 determines that the left eye is farther from the camera 3 than the right eye. It is determined that the user's face is facing the left side of the camera 3. When viewed from the camera 3, the user is facing the right side of the camera 3. On the contrary, if the pupil centroid of the right eye on the image is positioned below the pupil centroid of the left eye, the face orientation determination unit 23 determines that the right eye is farther from the camera 3 than the left eye. That is, it is determined that the user's face is facing the right side of the camera 3. In this case, when viewed from the camera 3, the user is facing the left side of the camera 3.

  Even when the camera 3 is disposed above the display device 2, the face orientation determination unit 23 performs the vertical position of the left eye pupil centroid and the vertical position of the right eye pupil centroid, as described above. Can be determined whether or not the user's face is facing the camera 3. In this case, when it is assumed that the camera 3 is facing upward from the user's face, the face orientation determination unit 23 determines that the vertical position of the pupil center of gravity is located above the left eye and the right eye. What is necessary is just to determine that the person doing is far from the camera 3.

  The face orientation determination unit 23 may determine whether or not the user's face is facing the camera 3 according to another method for determining the face orientation from the image. For example, the face orientation determination unit 23 detects the midline and nose vertex of the user's face on the image, and if the nose vertex is located on the midline, the user's face is normal to the camera 3. You may determine that it is. In this case, for example, the face orientation determination unit 23 detects pixels having a skin color on the image, and detects a set of pixels having the detected skin color as a face region. Then, the face orientation determination unit 23 detects a mouth region in which the mouth is reflected in the face region, for example, using a mouth detection discriminator or by template matching. The face orientation determination unit 23 sets a line connecting the midpoint between the left and right eye areas and the midpoint in the horizontal direction of the mouth area as the midline of the face. Further, the face orientation determination unit 23 may detect the nose apex using a nose apex detection discriminator or by template matching. Then, the face orientation determination unit 23 calculates a distance from the vertex of the nose to the median line on the image, and if the distance is less than a predetermined threshold, the vertex of the nose is positioned on the median line. It may be determined that the user's face is directly facing the camera 3.

  The face orientation determination unit 23 notifies the interpupillary distance calculation unit 24 of the determination result as to whether or not the user's face is facing the camera 3.

  The interpupillary distance calculation unit 24 calculates the position of the pupil of the left eye and the position of the pupil of the right eye in the real space based on the image when the user's face is facing the camera 3. In the present embodiment, the interpupillary distance calculation unit 24 includes the positional relationship between the camera 3 and both eyes according to the position of the center of gravity of the pupils of both eyes on the image, the line-of-sight direction of both eyes, the camera 3 and the display device. The position of the left eye pupil and the position of the right eye pupil in the real space are calculated based on the positional relationship with the second display screen. Then, the interpupillary distance calculation unit 24 calculates the interpupillary distance from the position of the pupil of the left eye and the position of the pupil of the right eye in the real space.

FIGS. 6A and 6B are diagrams illustrating an example of the relationship between the display device 2 and the camera 3, and the position of the pupil and the gaze position. 6A and 6B, the origin O of the camera coordinate system is set to the rear principal point of the imaging optical system of the camera 3, for example. Pi _left and Pi _right correspond to the positions of the center of the pupils of the left eye and the right eye on the image, respectively, and are located at a position separated from the origin O by the focal length f of the imaging optical system of the camera 3. The position on the imaging surface 600 is represented. Pr _left and Pr _right represent the pupil centroid of the left eye and the pupil centroid of the right eye, respectively, in real space. K represents the distance from the origin O to the display screen of the display device 2 along the optical axis OA of the camera 3. d represents the distance from the origin O to the pupil centroids Pr _left and Pr _right of the user's left and right eyes in real space along the optical axis OA of the camera 3. EV _left and EV _right represent the line-of-sight direction of the user's left eye and the line-of-sight direction of the right eye, respectively. L represents a gaze position on the display screen of the display device 2.

したがって、カメラ座標系における、実空間での左眼の瞳孔重心の位置Pr_left及び右眼の瞳孔重心の位置Pr_rightは、次式で表される。

ここで、φ_leftは、カメラ３から左眼の瞳孔重心Pr_leftへ向かう方向とカメラ３の光軸OAとがなす角である。またφ_rightは、カメラ３から右眼の瞳孔重心Pr_rightへ向かう方向とカメラ３の光軸OAとがなす角である。 In the camera coordinate system, a vector PV _left representing a direction from the origin O toward the pupil centroid of the left eye and a vector PV _right representing a direction from the origin O to the pupil centroid of the right eye are respectively expressed by the following equations.

Therefore, in the camera coordinate system, the left-eye pupil centroid position Pr _left and the right-eye pupil centroid position Pr _right in real space are expressed by the following equations.

Here, φ _left is an angle formed by the direction from the camera 3 toward the pupil center of gravity Pr _{left of the} left eye and the optical axis OA of the camera 3. Φ _right is an angle formed by a direction from the camera 3 toward the right eye pupil center of gravity Pr _right and the optical axis OA of the camera 3.

ここで、σ_leftは、注視位置Lから左眼の瞳孔重心Pr_leftへ向かう方向とカメラ３の光軸OAとがなす角である。またσ_rightは、注視位置Lから右眼の瞳孔重心Pr_rightへ向かう方向とカメラ３の光軸OAとがなす角である。 On the other hand, since it is assumed that both the left eye and the right eye of the user are looking at the same gaze position, the following equation is established.

Here, σ _left is an angle formed by the direction from the gaze position L toward the pupil center of gravity Pr _{left of the} left eye and the optical axis OA of the camera 3. Σ _right is an angle formed by the direction from the gaze position L toward the right eye pupil center of gravity Pr _right and the optical axis OA of the camera 3.

Therefore, the interpupillary distance calculation unit 24 solves the equations (4) and (5) to obtain the distance d and the left eye pupil centroid position Pr _left and the right eye pupil centroid position Pr _{right in} real space. Can be calculated. Then, the interpupillary distance calculation unit 24 calculates the interpupillary distance pp in the real space according to the following equation.

  The interpupillary distance calculation unit 24 stores the calculated interpupillary distance pp in the memory 6.

  FIG. 7 is an operation flowchart of the inter-pupil distance estimation process executed by the processor 7. Every time the processor 7 receives an image from the camera 3, it is only necessary to execute an interpupillary distance estimation process according to this operation flowchart.

  The pupil detection unit 21 detects an eye area for each of the user's left and right eyes from the image (step S101). The pupil detection unit 21 detects the center of gravity of the pupil in the eye area for each of the user's left and right eyes (step S102).

  The line-of-sight detection unit 22 detects the line-of-sight direction for each of the left and right eyes of the user (step S103).

  Further, the face orientation determination unit 23 determines whether or not the user's face is facing the camera 3 based on, for example, the vertical positions of the left and right eyes on the image (step S104). If the user's face does not face the camera 3 (No at Step S104), the processor 7 ends the interpupillary distance estimation process.

  On the other hand, if the user's face is directly facing the camera 3 (step S104—Yes), the interpupillary distance calculation unit 24 sets the gaze direction for each of the left and right eyes and the direction from the camera 3 toward the center of the pupil. Based on this, the inter-pupil distance in the real space is calculated (step S105). The interpupillary distance calculation unit 24 stores the calculated interpupillary distance in the memory 6. Then, the processor 7 ends the interpupillary distance estimation process.

  Note that the processor 7 may exchange the order of the processing in step S103 and the processing in step S104. That is, when it is determined in step S104 that the user's face is facing the camera 3, the line-of-sight detection unit 22 may detect the line-of-sight direction for each of the user's left and right eyes. Further, when the face orientation determination unit 23 determines the user's face orientation without using the positions of the pupil centroids of the left and right eyes on the image, the process of step S104 may be performed first. Then, when it is determined that the user's face is facing the camera 3, the processes of steps S <b> 101 to S <b> 103 and S <b> 105 may be performed.

(Gaze position detection process)
Next, the gaze position detection process will be described. The processor 7 executes a gaze position detection process every time an image is obtained by the camera 3 until, for example, the user stops using the digital signage system 1. The processor 7 determines that the user has stopped using the digital signage system 1 when the eye area cannot be detected from the image continuously for a predetermined period (for example, 10 seconds to 1 minute), for example. Alternatively, if the digital signage system 1 has a proximity sensor (not shown), the processor 7 may determine that the user has stopped using the digital signage system 1 if the proximity sensor can no longer detect the user. Then, when the user stops using the digital signage system 1, the processor 7 deletes the interpupillary distance stored in the memory 6.

  The pupil detection unit 21 detects an eye area for each of the user's left and right eyes from the image, and detects the center of the pupil from the eye area, similarly to the inter-pupil distance estimation process. Then, the pupil detection unit 21 notifies the gaze detection unit 22, the face orientation determination unit 23, and the gaze position detection unit 25 of the position of the pupil center of gravity on the image for each of the left and right eyes of the user.

  The line-of-sight detection unit 22 also detects the line-of-sight direction for each of the user's left and right eyes, as in the inter-pupil distance estimation process. The line-of-sight detection unit 22 notifies the gaze position detection unit 25 of the line-of-sight direction for each of the left and right eyes of the user.

  Further, as in the inter-pupil distance estimation process, the face orientation determination unit 23 determines whether the user's face is facing the camera 3 and the user's face is not facing the camera 3. It is determined whether the left side or the right side of 3 is facing. The face orientation determination unit 23 notifies the gaze position detection unit 25 of the determination result.

  The gaze position detection unit 25 detects the gaze position based on the gaze direction of each of the user's left and right eyes, the position of the center of gravity of the pupil on the image, the inter-pupil distance in real space, and the determination result of the face orientation. To do.

FIG. 8 is a diagram for explaining the principle of gaze position detection. In FIG. 8, the origin O of the camera coordinate system is set to the rear principal point of the imaging optical system of the camera 3, for example. Pr _left and Pr _right represent the pupil centroid of the left eye and the pupil centroid of the right eye in real space, respectively. K represents the distance from the origin O to the display screen of the display device 2 along the optical axis OA of the camera 3. EV _left and EV _right represent the line of sight of the user's left eye and the line of sight of the right eye, respectively. L represents a gaze position on the display screen of the display device 2. Furthermore, pp represents the interpupillary distance in real space. D _left and d _right respectively represent distances from the origin O to the pupil centroids Pr _left and Pr _right of the _left and right eyes of the user in real space along the optical axis OA of the camera 3.

Similarly to the equation (4), the left eye pupil centroid position Pr _left and the right eye pupil centroid position Pr _right in the real space in the camera coordinate system are expressed by the following equations.

ここで、ユーザの顔がカメラ３に対して正対していれば、d_left=d_rightが成立する。また、図８に示されるように、ユーザが、カメラ３よりも左側を見ていれば、d_left>d_rightが成立する。逆に、ユーザが、カメラ３よりも左側を見ていれば、d_left<d_rightが成立する。そこで、注視位置検出部２５は、ユーザの顔向きから求められる、d_leftとd_rightの大小関係を用いて、（７）式及び（８）式を解くことで、d_leftとd_rightを算出できる。そして注視位置検出部２５は、算出されたd_leftまたはd_rightを（８）式に代入することで、注視位置Lを算出できる。 On the other hand, since it is assumed that both the left eye and the right eye of the user are looking at the same gaze position, the following equation is established.

Here, if the user's face is facing the camera 3, d _left = d _right is established. Also, as shown in FIG. 8, if the user is looking at the left side of the camera 3, d _left > d _right is established. Conversely, if the user is looking at the left side of the camera 3, d _left <d _right is established. Therefore, the gaze position detection unit 25 calculates d _left and d _right by solving equations (7) and (8) using the magnitude relationship between d _left and d _right obtained from the user's face orientation. it can. The gaze position detection unit 25 can calculate the gaze position L by substituting the calculated d _left or d _right into the equation (8).

  The gaze position detection unit 25 stores the gaze position L in the memory 6.

  The processor 7 tracks the user's gaze position to identify a position or range where the gaze position has stagnated for a certain time or more. Then, the processor 7 executes processing corresponding to the position or range. For example, the processor 7 may cause the display device 2 to display information related to the icon or content displayed on the display device 2 at the position or range.

  FIG. 9 is an operation flowchart of gaze position detection processing executed by the processor 7. Every time the processor 7 acquires an image from the camera 3, the processor 7 may execute a gaze position detection process according to the following operation flowchart.

  The pupil detection unit 21 detects an eye area for each of the user's left and right eyes from the image (step S201). Then, the pupil detection unit 21 detects the center of the pupil in the eye area for each of the left and right eyes of the user (step S202).

  The line-of-sight detection unit 22 detects the line-of-sight direction for each of the left and right eyes of the user (step S203).

  Further, the face orientation determination unit 23 determines the user's face orientation from the image based on, for example, the vertical positions of the left and right eyes on the image (step S204). The gaze position detection unit 25 detects the gaze position based on the gaze direction of each of the left and right eyes of the user, the direction from the camera 3 toward the center of the pupil of each of the left and right eyes, the inter-pupil distance, and the face direction ( Step S205). Then, the processor 7 ends the gaze position detection process.

  As described above, when the user's face is facing the camera, the distances from the camera to the left and right eye pupils along the optical axis of the camera are equal to each other. Using this, the gaze position detection device obtains the positions of the left and right eyes in the real space based on the gaze direction for the left and right eyes and the direction from the camera toward the pupil. The gaze position detecting device calculates the distance between the pupils of the user in the real space based on the positions of the center of the pupils of the left and right eyes in the real space. Therefore, the gaze position detection device can accurately estimate the distance between the pupils of the user in real space from the image obtained by the camera. The gaze position detection device can detect the position of the pupil of each of the left and right eyes in the real space with reference to the distance between the pupils in the real space, so that the gaze position is determined based on the gaze direction of the left and right eyes. It can be detected accurately. Therefore, this gaze position detection apparatus can accurately detect the gaze position without performing calibration processing in advance and without knowing the distance from the camera to the user in advance.

  Note that, according to the modification, the optical axis direction of the camera 3 and the normal direction of the display screen of the display device 2 may not be parallel. In this case, the distance between the camera 3 and the gaze position on the display screen along the optical axis direction of the camera 3 varies according to the gaze position on the gaze target surface. Therefore, in this modification, the interpupillary distance calculation unit 24 refers to the tilt angle of the display screen of the display device 2 with respect to the optical axis direction of the camera 3, so that the distance between the camera 3 and the gaze position is not constant. Also, the interpupillary distance is calculated. The gaze position detection unit 25 also detects the gaze position by referring to the tilt angle even if the distance between the camera 3 and the gaze position is not constant.

  FIG. 10 is a diagram illustrating an example of the relationship between the display device 2 and the camera 3, and the pupil center of gravity and gaze position according to this modification. In FIG. 10, the origin O of the camera coordinate system is set to the rear principal point of the imaging optical system of the camera 3, for example. The camera coordinate system is set so that the direction parallel to the optical axis OA of the camera 3 is the z-axis, the horizontal direction is the x-axis, and the vertical direction is the y-axis. The display screen of the display device 2 has an angle θ with respect to the z axis, and the origin O is located on a surface obtained by extending the display screen. That is, the display screen of the display device 2, which is an example of a gaze target surface, is inclined by an angle θ with respect to the optical axis OA of the camera 3. In addition, Pr represents the pupil centroid of one eye of the user in real space, and PV represents a vector from the origin O to the pupil centroid Pr. EV is a vector representing the line-of-sight direction of one of the user's eyes. L represents a gaze position on the display screen of the display device 2. D represents the distance from the origin O to the pupil center of gravity Pr of one of the users in real space along the z axis. Further, H represents a vector parallel to the display screen of the display device 2 in the yz plane.

ただし、tは、原点Oから注視位置Lまでの距離を表し、kは、瞳孔重心Prから注視位置Lまでの距離を表す。（９）式をk及びtについて解くことにより、次式が得られる。

ただし、Pr_y及びPr_zは、それぞれ、瞳孔重心Prのy軸座標及びz軸座標を表す。すなわち、Pr_z=dである。また、EV_y及びEV_zは、それぞれ、視線方向ベクトルEVのy軸成分及びz軸成分を表す。 In this case, the gaze position L is located on the display screen of the display device 2 and at a position along the vector EV representing the line-of-sight direction from the pupil center of gravity Pr of the user's eye. Therefore, the following equation is established.

However, t represents the distance from the origin O to the gaze position L, and k represents the distance from the pupil center of gravity Pr to the gaze position L. By solving the equation (9) for k and t, the following equation is obtained.

However, Pr _y and Pr _z represent the y-axis coordinate and the z-axis coordinate of the pupil center of gravity Pr, respectively. That is, Pr _z = d. EV _y and EV _z represent a y-axis component and a z-axis component of the line-of-sight direction vector EV, respectively.

  Equation (9) is established for each of the left and right eyes. Therefore, the interpupillary distance calculation unit 24 uses the t and k calculated according to the equation (10) to solve the equations (9) and (4) for the left and right eyes, respectively. The real space position of the pupil's pupil center of gravity can be calculated. Then, when the position of the real space of the center of gravity of the pupil of each of the left and right eyes is calculated, the interpupillary distance calculation unit 24 may calculate the interpupillary distance pp according to the equation (6).

  According to this modification, the gaze position detection device is configured so that the normal direction of the gaze target surface and the optical axis direction of the camera that captures the user's eyes are not parallel, that is, the distance between the camera and the gaze position is the same. Even if it is not constant, the distance between the pupils of the user can be obtained.

Similarly, the gaze position detection unit 25 uses (9) and (9) for each of the left and right eyes using t and k calculated according to the equation (10) according to the magnitude relationship between d _left and d _right based on the face orientation determination result. ), (6) and (7) are solved to calculate the gaze position L.

Note that if the origin O does not exist on the surface obtained by extending the gaze target surface, if the intercept about the z axis for the surface obtained by extending the gaze target surface is added to the right side of the upper formula of the equation (9), Good. Further, when the gaze target surface is also inclined in the horizontal direction with respect to the optical axis of the imaging unit, the equation (9) is also established for the xz plane. Therefore, in this case, the interpupillary distance calculation unit 24 uses Equations (9) and (4) relating to the xz plane for the left and right eyes, using t and k calculated according to Equation (10). By solving, the position of the real space of the center of gravity of the pupil of each of the left and right eyes can be calculated. Similarly, the gaze position detection unit 25 uses the t and k calculated according to the equation (10) according to the magnitude relationship between d _left and d _right based on the determination result of the face orientation, and the xz plane for each of the left and right eyes. The gaze position L can be calculated by solving the equations (9), (6), and (7).

According to another modification, the processor 7 performs the above-described inter-pupil distance estimation process on each of a plurality of images obtained when the user's face is directly facing the camera 3 to perform inter-pupil distance. May be calculated and stored in the memory 6. Then, the interpupillary distance calculation unit 24 may use the statistical representative value of the candidate interpupillary distance obtained from each of the plurality of images as the interpupillary distance of the user. In this case, the interpupillary distance calculation unit 24 can calculate, for example, the mode value, median value, or average value of the interpupillary distance candidates as the statistical representative value of the interpupillary distance candidates. Alternatively, the interpupillary distance calculation unit 24 may estimate the probability distribution of the interpupillary distance candidates according to, for example, a Gaussian distribution, and may use the average value in the probability distribution as the user's interpupillary distance.
According to this modification, the gaze position detection device can determine the distance between the pupils of the user with higher accuracy.

  According to still another modified example, the right and left eye gaze directions and the position of the pupil center of gravity obtained for each image obtained during the execution of the inter-pupil distance estimation process may be stored in the memory 6. Then, after estimating the interpupillary distance, the gaze position detection unit 25 may detect the gaze position for each of the images based on the gaze direction of the left and right eyes, the position of the center of gravity of the pupil, and the calculated interpupillary distance. Good. As a result, the gaze position detection device can detect the gaze position of the user before the distance between the pupils is calculated, so that the trajectory of the gaze position can be examined in more detail.

  The gaze position detection device according to the above-described embodiment or modification is not limited to a digital signage system, and may be used for a vehicle driving support system, a computer input support system, and the like.

  The computer program that realizes the function of the control unit of the gaze position detection device according to the above-described embodiment or its modification may be provided in a form recorded on a computer-readable recording medium such as a magnetic recording medium or an optical recording medium. . This recording medium does not include a carrier wave.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
Determining whether or not the user's face is facing the imaging unit from an image generated by the imaging unit and representing at least part of the user's face including the left eye and the right eye;
From the image when the user's face is directly facing the imaging unit, the respective pupils of the left eye and the right eye are detected,
Detecting the gaze direction of the left eye based on the left eye on the image;
Detecting the gaze direction of the right eye based on the right eye on the image;
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The left-eye pupil in real space based on the positional relationship between the left-eye gaze direction and the right-eye gaze direction, and the positional relationship between the imaging unit and the surface to be watched by the user And the position of the right eye pupil,
Calculating the interpupillary distance between the left eye and the right eye in real space from the position of the pupil of the left eye and the position of the right eye pupil in real space;
A computer program for detecting a gaze position for causing a computer to execute the operation.
(Appendix 2)
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The positional relationship between the direction from the imaging unit to the pupil of the left eye according to the position of the pupil of the left eye on the image when the user's face is directly facing the imaging unit, and The direction from the imaging unit to the right eye pupil according to the position of the right eye pupil on the image, the distance along the optical axis of the imaging unit from the imaging unit to the left eye, and the imaging The computer program for gaze position detection according to appendix 1, wherein the distance along the optical axis of the imaging unit from the unit to the right eye is equal.
(Appendix 3)
The normal direction of the surface to be watched and the optical axis of the imaging unit are parallel, and the positional relationship between the imaging unit and the surface to be watched is the surface to be the imaging unit and the eye to be watched. The computer program for gaze position detection according to appendix 1 or 2, including a distance between them.
(Appendix 4)
The computer program for gaze position detection according to appendix 1 or 2, wherein the positional relationship between the imaging unit and the gaze target surface includes an inclination of the gaze target surface with respect to an optical axis direction of the imaging unit.
(Appendix 5)
The calculation of the interpupillary distance between the left eye and the right eye in real space means that the interpupillary distance for each of the plurality of images when the user's face is facing the imaging unit. The statistical representative value of the interpupillary distance between the left eye and the right eye calculated for each of the plurality of images is used as the interpupillary distance, according to any one of appendices 1 to 4. Computer program for detecting the gaze position.
(Appendix 6)
From the second image generated by the imaging unit, which is different from the image in which the interpupillary distance is calculated, and which represents at least a part of the user's face including the left eye and the right eye, the left eye and the Detect each pupil of the right eye,
Detecting a second gaze direction of the left eye based on the left eye on the second image;
Detecting a second gaze direction of the right eye based on the right eye on the second image;
Determining the user's face orientation based on the second image;
The positional relationship between the imaging unit and the left and right eyes according to the position of the pupil of the left eye and the position of the pupil of the right eye on the second image, and the second line-of-sight direction of the left eye And the second eye-gaze direction of the right eye, the orientation of the user's face, the positional relationship between the imaging unit and the surface to be watched, and the inter-pupil distance on the surface to be watched Detect the gaze position of
The computer program for gaze position detection as described in any one of the supplementary notes 1-5 for making a computer perform this further.
(Appendix 7)
Determining whether or not the user's face is facing the imaging unit from an image generated by the imaging unit and representing at least part of the user's face including the left eye and the right eye;
From the image when the user's face is directly facing the imaging unit, the respective pupils of the left eye and the right eye are detected,
Detecting the gaze direction of the left eye based on the left eye on the image;
Detecting the gaze direction of the right eye based on the right eye on the image;
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The left-eye pupil in real space based on the positional relationship between the left-eye gaze direction and the right-eye gaze direction, and the positional relationship between the imaging unit and the surface to be watched by the user And the position of the right eye pupil,
Calculating the interpupillary distance between the left eye and the right eye in real space from the position of the pupil of the left eye and the position of the pupil of the right eye in real space;
Gaze position detection method including the above.
(Appendix 8)
Face orientation for determining whether or not the user's face is facing the imaging unit from an image generated by the imaging unit and showing at least part of the user's face including the left eye and the right eye A determination unit;
A pupil detection unit that detects the pupils of the left eye and the right eye from the image when the user's face is facing the imaging unit;
A line-of-sight detection unit that detects the line-of-sight direction of the left eye based on the left eye on the image and detects the line-of-sight direction of the right eye based on the right eye on the image;
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The left-eye pupil in real space based on the positional relationship between the left-eye gaze direction and the right-eye gaze direction, and the positional relationship between the imaging unit and the surface to be watched by the user And the position of the pupil of the right eye, and the position of the pupil of the left eye and the position of the right eye pupil in real space, between the left eye and the right eye in real space. An interpupillary distance calculator that calculates the interpupillary distance;
A gaze position detecting device having
(Appendix 9)
Detecting each pupil of the left eye and the right eye from an image generated by the imaging unit and representing at least a part of the user's face including the left eye and the right eye;
Detecting the gaze direction of the left eye based on the left eye on the image;
Detecting the gaze direction of the right eye based on the right eye on the image;
Determining the face orientation of the user based on the image;
The positional relationship between the imaging unit and the left and right eyes, the line-of-sight direction of the left eye, and the line of sight of the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image The gaze based on the direction, the orientation of the user's face, the positional relationship between the imaging unit and the plane to be watched by the user, and the interpupillary distance between the left eye and the right eye in real space Detect the gaze position on the target surface,
A computer program for detecting a gaze position for causing a computer to execute the operation.

1 Digital signage system (gaze position detector)
DESCRIPTION OF SYMBOLS 2 Display apparatus 2a Display screen 3 Camera 4 Input apparatus 5 Storage medium access apparatus 6 Memory 7 Processor 8 Storage medium 21 Pupil detection part 22 Eye gaze detection part 23 Face direction determination part 24 Interpupillary distance calculation part 25 Gaze position detection part

Claims (7)

Determining whether or not the user's face is facing the imaging unit from an image generated by the imaging unit and representing at least part of the user's face including the left eye and the right eye;
From the image when the user's face is directly facing the imaging unit, the respective pupils of the left eye and the right eye are detected,
Detecting the gaze direction of the left eye based on the left eye on the image;
Detecting the gaze direction of the right eye based on the right eye on the image;
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The left-eye pupil in real space based on the positional relationship between the left-eye gaze direction and the right-eye gaze direction, and the positional relationship between the imaging unit and the surface to be watched by the user And the position of the right eye pupil,
Calculating the interpupillary distance between the left eye and the right eye in real space from the position of the pupil of the left eye and the position of the right eye pupil in real space;
A computer program for detecting a gaze position for causing a computer to execute the operation.   The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The positional relationship between the direction from the imaging unit to the pupil of the left eye according to the position of the pupil of the left eye on the image when the user's face is directly facing the imaging unit, and The direction from the imaging unit to the right eye pupil according to the position of the right eye pupil on the image, the distance along the optical axis of the imaging unit from the imaging unit to the left eye, and the imaging The computer program for gaze position detection according to claim 1, further comprising: a distance along the optical axis of the imaging unit from a unit to the right eye being equal.   The normal direction of the surface to be watched and the optical axis of the imaging unit are parallel, and the positional relationship between the imaging unit and the surface to be watched is the surface to be the imaging unit and the eye to be watched. The computer program for detecting a gaze position according to claim 1 or 2, including a distance between them.   3. The computer program for gaze position detection according to claim 1, wherein the positional relationship between the imaging unit and the surface to be watched includes an inclination of the surface to be watched with respect to an optical axis direction of the imaging unit. From the second image generated by the imaging unit, which is different from the image in which the interpupillary distance is calculated, and which represents at least a part of the user's face including the left eye and the right eye, the left eye and the Detect each pupil of the right eye,
Detecting a second gaze direction of the left eye based on the left eye on the second image;
Detecting a second gaze direction of the right eye based on the right eye on the second image;
Determining the user's face orientation based on the second image;
The positional relationship between the imaging unit and the left and right eyes according to the position of the pupil of the left eye and the position of the pupil of the right eye on the second image, and the second line-of-sight direction of the left eye And the second eye-gaze direction of the right eye, the orientation of the user's face, the positional relationship between the imaging unit and the surface to be watched, and the inter-pupil distance on the surface to be watched Detect the gaze position of
The computer program for gaze position detection as described in any one of Claims 1-4 for making a computer perform this further. Determining whether or not the user's face is facing the imaging unit from an image generated by the imaging unit and representing at least part of the user's face including the left eye and the right eye;
From the image when the user's face is directly facing the imaging unit, the respective pupils of the left eye and the right eye are detected,
Detecting the gaze direction of the left eye based on the left eye on the image;
Detecting the gaze direction of the right eye based on the right eye on the image;
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The left-eye pupil in real space based on the positional relationship between the left-eye gaze direction and the right-eye gaze direction, and the positional relationship between the imaging unit and the surface to be watched by the user And the position of the right eye pupil,
Calculating the interpupillary distance between the left eye and the right eye in real space from the position of the pupil of the left eye and the position of the right eye pupil in real space;
Gaze position detection method including the above. Face orientation for determining whether or not the user's face is facing the imaging unit from an image generated by the imaging unit and showing at least part of the user's face including the left eye and the right eye A determination unit;
A pupil detection unit that detects the pupils of the left eye and the right eye from the image when the user's face is facing the imaging unit;
A line-of-sight detection unit that detects the line-of-sight direction of the left eye based on the left eye on the image and detects the line-of-sight direction of the right eye based on the right eye on the image;
The imaging unit, the left eye, and the right eye according to the position of the pupil of the left eye and the position of the pupil of the right eye on the image when the user's face is facing the imaging unit The left-eye pupil in real space based on the positional relationship between the left-eye gaze direction and the right-eye gaze direction, and the positional relationship between the imaging unit and the surface to be watched by the user Between the left eye and the right eye in real space from the position of the left eye pupil and the position of the right eye pupil in real space. An interpupillary distance calculator that calculates the distance;
A gaze position detecting device having

Images