JPWO2015012280A1 - Gaze detection device - Google Patents

Abstract

A gaze detection device capable of accurately detecting the position of a pupil of a user's eye is provided. A line-of-sight detection apparatus includes a light receiving unit, a light guiding unit, a measuring unit, and a calculating unit. The light receiving unit 22 receives light from the user's eye UE and generates a light reception signal corresponding to the intensity of the light. The light guiding unit 30 guides the light from the eye to the light receiving unit 22. The measuring unit 25 measures the positional relationship between the light guiding unit 30 and the pupil PU or iris IR of the eye based on the light reception signal. The calculating means 25 calculates the position of the user's pupil PU based on the positional relationship. [Selection] Figure 1

Description

  The present invention relates to a line-of-sight detection device.

  A technique is generally known in which a virtual image that is a virtual world image created by computer graphics or the like is superimposed on an external image that is an image of the real world and displayed on a display. In this technique, the user perceives as if the virtual image is displayed on the display together with the external image. The external image may be a real-world image that can be directly viewed by the user through transparent glass, resin, or the like, or may be an image obtained by capturing an image of the external environment with an imaging device such as a camera in advance. Good.

  In recent years, for example, a technique for enabling a user to work on virtual images and devices by detecting the position and behavior of the body such as the user's limbs and eyes has been studied. In particular, the technology that distinguishes the object ahead of the user's line of sight in the real world or video by detecting the line of sight based on the position of the pupil of the user's eye is attracting attention, and is expected to be applied in various fields. ing. For example, as a method of operating a personal computer, a technique of operating by a user's line of sight has been developed in addition to a conventional operation using a keyboard, a mouse, or the like.

  As a method for detecting the line of sight, a method for detecting the position of the pupil from the image of the user's eye imaged by the camera and detecting the line of sight based on the position of the pupil has been conventionally known (for example, the following patent document). 1). In the technique of Patent Literature 1, a user wears a spectacle-type head-mounted display, and an image of the user's eye is captured through a see-through member of the head-mounted display (a part corresponding to a spectacle lens). Then, the position of the pupil is detected from the image of the eye, and the direction of the line of sight is detected based on the position of the pupil.

  However, since the distance between the see-through member of the head-mounted display and the user's eye is not considered in the technique of Patent Document 1, the position of the pupil of the user's eye is accurately determined when the mounting state of the head-mounted display changes. There is a problem of not being detected. In addition, since the position of the pupil of the user's eye is not accurately detected, there is also a problem that the direction of the user's line of sight is not accurately detected.

JP 2010-102215 A

  The present invention has been made in view of the above problems. Accordingly, an object of the present invention is to provide a gaze detection device that can accurately detect the position of the pupil of the user's eye.

  The above object of the present invention is achieved by the following.

  (1) Light receiving means for receiving light from the user's eyes and generating a light receiving signal corresponding to the intensity of the light, light guiding means for guiding light from the eyes to the light receiving means, and based on the light receiving signal And a measuring means for measuring a positional relationship between the light guide means and the pupil or iris of the eye, and a calculating means for calculating the position of the user's pupil based on the positional relationship. Detection device.

  (2) The measurement unit measures a distance between the eyepiece surface of the light guide unit and the iris as the positional relationship, and the calculation unit calculates the position of the pupil with reference to the eyepiece surface based on the distance. And the direction of the pupil is calculated, and the line-of-sight direction is calculated based on the direction of the pupil.

  (3) The line-of-sight detection device according to (1) or (2), wherein the light guiding unit changes the course by diffracting light from the eye and guides the light to the light receiving unit.

  (4) The light receiving unit includes a compound eye imaging device, generates an image signal as the light reception signal, and the measurement unit measures the distance based on the image signal. The line-of-sight detection device according to 2) or (3).

  (5) The light receiving means includes a stereo camera as a compound eye imaging device, generates an image signal as the light reception signal, and the measuring means measures the distance based on the image signal. The visual line detection device according to (2) or (3).

  (6) The light receiving unit includes an array camera as a compound eye imaging device, generates an image signal as the light reception signal, and the measuring unit measures the distance based on the image signal. The line-of-sight detection device according to (2) or (3) above.

  (7) The line-of-sight detection device according to any one of (1) to (6), further including an illumination unit that irradiates the user's eyes with irradiation light.

  (8) It further includes an illuminating unit that irradiates the user's eyes with irradiation light, the illuminating unit irradiates pulse light as the irradiation light, and the light receiving unit includes a TOF camera and reflects from the eye. The reflected light received by the TOF camera, a delay time of the reflected light with respect to the pulsed light is calculated based on the received light signal, and the distance is measured based on the delay time (1) The line-of-sight detection device according to any one of (1) to (3).

  (9) The light guide means includes a holographic optical element, and changes the course by diffracting the irradiation light with the holographic optical element, guides it to the eye, and reflects the reflected light by the eye. The line-of-sight detection device according to (7) above, wherein

  (10) The line-of-sight detection device according to any one of (1) to (9), wherein the light guide unit includes a transparent plate, and the transparent plate includes the eyepiece surface.

  (11) The line-of-sight detection device according to any one of (1) to (10), which is configured to be worn on the head of the user.

  (12) The line-of-sight detection device according to (7), wherein the irradiation light is infrared light.

  (13) A subject imaging unit that captures an external subject, and a target ahead of the user's line of sight in the video imaged by the subject imaging unit based on the line-of-sight direction calculated by the calculation unit. The line-of-sight detection device according to any one of (2) to (12), further including: a specifying unit that specifies.

  (14) The information on the object ahead of the user's line of sight is displayed on a display unit based on a specifying unit that identifies the object ahead of the user's line of sight. Gaze detection device.

It is an external view which shows the principal part of the gaze detection apparatus in the 1st Embodiment of this invention. It is a schematic block diagram of the detection part of the gaze detection apparatus shown in FIG. It is a schematic diagram which shows the arrangement | positioning of the light and camera in the detection part shown in FIG. It is sectional drawing cut | disconnected along the IV-IV line of the gaze detection apparatus shown in FIG. It is sectional drawing for demonstrating the optical path in the light guide part of the infrared light irradiated from the light. It is sectional drawing for demonstrating the optical path in the light guide part of the reflected light from a user's eyes. It is a conceptual diagram for demonstrating the optical path of irradiation light and reflected light when a light guide part is seen from a detection part in FIG. 5 and FIG. It is a conceptual diagram for demonstrating the method of calculating a user's gaze direction. It is a conceptual diagram for demonstrating the method of calculating a user's gaze direction. It is sectional drawing for demonstrating the optical path in the light guide part of the display light from a display part. In the 2nd Embodiment of this invention, it is a figure which shows arrangement | positioning of the light of a detection part and a camera in the case of using a super-resolution type | mold array camera. It is a conceptual diagram for demonstrating the optical path of irradiation light and reflected light when a light guide part is seen from the detection part of FIG. 10A. In the 2nd Embodiment of this invention, it is a conceptual diagram which shows the other form at the time of imaging with a super-resolution type | mold array camera. In the 2nd Embodiment of this invention, it is a conceptual diagram for demonstrating the optical path of irradiation light and reflected light when a light guide part is seen from the detection part in the case of using a visual field division | segmentation type array camera. In the 2nd Embodiment of this invention, it is a conceptual diagram for demonstrating the optical path of the reflected light in the case of using an insect type | mold array camera. In the 3rd Embodiment of this invention, it is a figure which shows arrangement | positioning of the light of a detection part, and a camera. It is an external view for demonstrating the case where it has a stereo camera for imaging a to-be-photographed object in 4th Embodiment. It is an external view for demonstrating the case where it has an array camera for imaging a to-be-photographed object in 4th Embodiment.

  Hereinafter, embodiments of the eye gaze detection apparatus of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals are used for the same members. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios.

(First embodiment)
In the first embodiment, a line-of-sight detection apparatus will be described using a head-mounted display (hereinafter referred to as an HMD) that has two cameras as a compound-eye imaging apparatus and images a user's eyes in a stereo manner. FIG. 1 is an external view showing the main part of the visual line detection device according to the first embodiment of the present invention, FIG. 2 is a schematic block diagram of the detection unit of the visual line detection device shown in FIG. 1, and FIG. It is a schematic diagram which shows arrangement | positioning of the light and camera in the detection part shown in FIG. 4 is a cross-sectional view taken along line IV-IV of the visual line detection device shown in FIG. Note that the X-axis, Y-axis, and Z-axis of the orthogonal coordinate system were determined in the directions shown in FIG.

  As illustrated in FIG. 1, the line-of-sight detection device 1 according to the present embodiment includes a mounting unit 10, a detection unit 20, and a light guide unit 30. Throughout this specification, the center position of the pupil of the user's eye wearing the line-of-sight detection device 1 is referred to as “viewpoint”, and the direction of the user's line of sight is referred to as “line-of-sight direction”.

<Schematic configuration of mounting part>
The mounting unit 10 includes a pair of first support members 11R and 11L, second support members 12R and 12L, and a connecting member 13, and plays a role of fixing the detection unit 20 and the light guide unit 30 to the user's head. .

  The first support members 11R and 11L are rod-shaped members formed of, for example, a metal, a resin, or the like and having curved end portions. The first support members 11R and 11L support the light guide unit 30 at one end and the user's temporal region at the other end. It is supported by the site between the ears.

  The second support members 12R and 12L are members formed of a material such as metal or resin, for example, and are attached to the light guide 30 and supported by the user's nose.

  The connecting portion 13 is a member formed of a material such as metal or resin, for example, and connects the pair of transparent plates 31R and 31L of the light guide portion 30 to each other.

  When the line-of-sight detection device 1 is mounted on the user's head, the transparent plates 31R and 31L are positioned immediately in front of the left and right eyes, respectively, and the first support members 11R and 11L The second support members 12R and 12L are supported on both upper side portions of the nose. That is, the line-of-sight detection device 1 of the present embodiment has a shape similar to general glasses as a whole, the transparent plates 31R and 32R correspond to lenses, the support members 11R and 11L correspond to temples (temples), The two support members 12R and 12L correspond to nose pads.

<Schematic configuration of detector>
The detection unit 20 detects the viewpoint and the line-of-sight direction based on the image of the user's eye guided by the light guide unit 30. In the present embodiment, the detection unit 20 is attached to the upper part of the transparent plate 31 </ b> R of the light guide unit 30.

  As shown in FIG. 2, the detection unit 20 includes an illumination unit 21, an imaging unit 22, a display unit 23, a communication unit 24, and an arithmetic control unit 25, and these components are electrically connected by a bus or a control line 26. Connected to each other.

  The illumination unit 21 irradiates light to the user's eyes via the light guide unit 30 as illumination means. The illumination unit 21 includes lights 21A and 21B including a light emitting element (not shown) and a light emission driving unit that drives the light emitting element (see FIG. 3). In the present embodiment, the user's eyes are irradiated with light from two directions by the left and right lights 21 </ b> A and 21 </ b> B of the illumination unit 21.

  The light emitting element may be, for example, an infrared LED (Light Emitting Diode) that emits infrared light. By irradiating the user's eyes with infrared light, the user's eyes can be imaged even in places with poor visible light, such as at night or in a dark room. Note that the imaging unit 22 is configured to be sensitive to infrared light, and an image captured with infrared light is a kind of monochrome image representing the brightness of infrared light.

  In addition, by using infrared light, it is possible to avoid noise from the external image when the user's eyes are imaged, so that the image of the user's eyes can be accurately recognized. On the other hand, in imaging using visible light, the light guided from the user's eye to the imaging unit 22 includes not only the image of the user's eye but also an external image reflected on the surface of the user's eye. Therefore, there is a possibility that noise from the outside world is mixed when the user's eyes are imaged.

  The imaging unit 22 captures the right eye of the user by receiving the reflected light from the right eye of the user transmitted through the light guide unit 30 as a light receiving unit. The imaging unit 22 includes cameras 22A and 22B including an imaging element (not shown), an imaging lens that forms an image of light on the imaging element, and an imaging drive unit that drives the imaging element. The imaging unit 22 receives reflected light from the right eye of the user with the imaging element, performs photoelectric conversion, generates a video signal as a light reception signal, and transmits the video signal to the calculation control unit 25. In the present embodiment, the imaging device includes, for example, a charge coupled device (CCD) that is sensitive to infrared light, a complementary metal oxide semiconductor (CMOS), and the like.

  As shown in FIG. 3, in this embodiment, the cameras 22A and 22B are arranged separately on the left and right to function as a compound eye imaging device, and image the right eye of the user from two directions in a stereo manner. The video signal of the right eye of the user captured by the two cameras 22A and 22B is transmitted to the arithmetic control unit 25. The optical system and the image sensor of the cameras 22A and 22B may be integrally configured.

  The display unit 23 displays a predetermined image on the eyepiece surface of the transparent plate 31R. As shown in FIG. 4, in the present embodiment, the display unit 23 includes a display that is disposed on the light guide unit 30 and includes a display element (not shown) and a display driving unit that drives the display element. The display element can be, for example, a liquid crystal display element, an LED light emitting display element, an organic light emitting display element, or the like.

  The line-of-sight detection device 1 according to the present embodiment detects the user's viewpoint and line-of-sight direction, and identifies an object ahead of the user's line-of-sight direction in the predetermined video based on the detection result. The predetermined video is, for example, a video that captures the outside including the object, a menu screen for controlling the device, a video of a movie or a television program, and the like.

  The communication unit 24 includes a wired communication system or a wireless communication system transmitter and receiver, and a CPU (Central Processing Unit) (not shown) that controls the entire system including them, and between the arithmetic control unit 25 and an external device. Send and receive various data. The communication unit 24 transmits information regarding the calculated user viewpoint and the subject in the line-of-sight direction to an external device, and receives information about the subject in the line-of-sight direction. The communication unit 24 transmits the received data to the calculation control unit 25 and the display unit 23 and displays the data on the display unit.

  The arithmetic control unit 25 controls the illumination unit 21, the imaging unit 22, the display unit 23, and the communication unit 24. The arithmetic control unit 25 has a CPU and a memory (not shown). A control program is stored in the memory, and the CPU executes the control program to control the illumination unit 21, the imaging unit 22, the display unit 23, and the communication unit 24. In addition, the arithmetic control unit 25 performs distance measurement, iris shape measurement, iris tilt measurement, viewpoint detection, line-of-sight detection, gradation conversion, distortion correction, noise removal on the video signal of the user's eye imaged by the imaging unit 22. Various processes such as are executed.

  In particular, in the present embodiment, the arithmetic control unit 25 measures the positional relationship between the light guide unit 30 and the pupil or iris of the user's right eye UE based on the video signal of the user's eye, and the positional relationship. Based on the above, the viewpoint and line-of-sight direction of the user's right eye are calculated. The arithmetic control unit 25 functions as a measurement unit and a calculation unit. Information regarding the positional relationship, the viewpoint, and the line-of-sight direction is temporarily stored in the memory. Details of the measurement method of the positional relationship and the calculation method of the viewpoint and the line-of-sight direction will be described later.

  The light guide unit 30 guides the irradiation light from the illumination unit 21 to the user's right eye UE and guides the reflected light from the user's right eye UE to the imaging unit 22 as light guide means. The light guide unit 30 guides the light of the video displayed on the display unit 23 to the user's right eye UE. As shown in FIG. 4, the light guide 30 includes a transparent plate 31R and holographic optical elements (hereinafter referred to as HOE) 32A and 32B. Note that 32A may be a normal half mirror.

  If the detection unit 20 and the light guide unit 30 are arranged on the transparent plate 31L, the left eye viewpoint and the line-of-sight direction can be detected. Therefore, the line-of-sight detection device 1 of the present embodiment detects the right eye viewpoint and the line-of-sight direction. It is not limited to the case. In the following description, the user's right eye and left eye are simply referred to as the user's eye UE.

  The transparent plate 31R is a transparent plate formed of a material such as glass or resin having a refractive index larger than that of air. HOE (Holographic Optical Element) 32A and 32B are formed inside the transparent plate 31R.

  The HOEs 32A and 32B are optical elements using holograms and have the property of reflecting only light of a specific wavelength and transmitting light of other wavelengths. A hologram is produced by irradiating a photosensitive material with two highly coherent light beams such as a laser and recording the interference state.

  In the present embodiment, the HOEs 32A and 32B are designed to selectively reflect infrared (IR) light and to be a half mirror for visible light. That is, the HOEs 32A and 32B transmit visible light incident from one surface (first surface) and reflect visible light incident from the other surface (second surface). Accordingly, a part of the external image observed by the user does not become dark due to the presence of the HOE 32B. On the other hand, as for infrared light, light incident on either the first surface or the second surface is reflected.

  In the example illustrated in FIG. 4, the HOE 32 </ b> A is arranged such that the first surface faces the display unit 23 side and the second surface faces the illumination unit 21 </ b> A side. Further, the HOE 32B is arranged so that the second surface faces the user's eye UE.

  As described above, in the present embodiment, the HOEs 32A and 32B have a function of reflecting infrared light in addition to visible light for display, and thus can have both a display function and an illumination function. Moreover, in this embodiment, the user's eye UE can be imaged from substantially the front by using the HOE 32B. Therefore, it is possible to accurately detect the viewpoint and the line-of-sight direction while securing the user's field of view.

  For example, when the imaging unit 22 images the user's eye UE from two directions across the display unit 23, the central portions of the HOEs 32A and 32B are used to diffract display light from the display unit 23. . On the other hand, both ends of the HOEs 32 </ b> A and 32 </ b> B are used to diffract reflected light from the user's eye UE toward the imaging unit 22.

  Next, with reference to FIGS. 5-8B, the effect | action of the gaze detection apparatus 1 of this embodiment comprised as mentioned above is demonstrated.

  FIG. 5 is a cross-sectional view for explaining the optical path in the light guide part of the infrared light irradiated from the illumination part, and FIG. 6 is a cross-section for explaining the optical path in the light guide part of the reflected light from the user's eyes. FIG. 7 is a conceptual diagram for explaining the optical paths of the irradiation light and the reflected light when the light guide unit is viewed from the detection unit in FIGS. 5 and 6, and FIGS. 8A and 8B show the user's line-of-sight direction. It is a conceptual diagram for demonstrating the method to calculate. FIG. 9 is a cross-sectional view for explaining an optical path in a light guide portion of display light from the display portion.

  As shown in FIG. 5, the irradiation light LA emitted from the lights 21A and 21B is reflected by the HOE 32A and proceeds in the direction of the transparent plate 31R. Then, the irradiation light LA travels toward the HOE 32B while totally reflecting the interface between the transparent plate 31R and the outside air, is reflected by the HOE 32B, and reaches the user's eye UE.

  As shown in FIG. 6, the reflected light RA reflected by the user's eye UE is reflected by the HOE 32B, travels toward the HOE 32A while totally reflecting the interface between the transparent plate 31R and the external air, and the HOE 32A. And reaches the cameras 22A and 22B.

  The optical paths of the irradiated light and the reflected light when viewing the direction of the light guide from the detector in FIGS. 5 and 6 will be described below with reference to FIG. In FIG. 7, the light guide is not shown and the optical path is expressed as a straight line on the XZ plane.

  As shown in FIG. 7, the lights 21A and 21B irradiate the user's eye UE with irradiation light LA and LB from the left and right with the cameras 22A and 22B interposed therebetween. On the other hand, the reflected lights RA and RB from the user's eye UE are received by the two cameras 22A and 22B, respectively. Thus, in this embodiment, the cameras 22A and 22B can capture the user's eye UE from two different directions. Then, the video signal of the user's eye UE captured by the two cameras 22 </ b> A and 22 </ b> B is transmitted to the arithmetic control unit 25.

  The arithmetic control unit 25 measures the positional relationship between the light guide unit 30 and the pupil or iris of the user's eye UE based on the images of the user's eye UE captured by the cameras 22A and 22B, and the positional relationship Based on the above, the viewpoint and line-of-sight direction are calculated. More specifically, as shown below.

  As shown in FIGS. 8A and 8B, the origin O is a point where a light beam passing through the center of the HOE 32 </ b> B intersects the eyepiece surface ES of the light guide unit 30. The direction perpendicular to the plane of FIG. 8A from the back to the front is the X-axis direction, and the direction from the origin O toward the center of the user's eye UE is the Z-axis direction, the direction perpendicular to the X-axis direction and the Z-axis direction. Thus, an orthogonal coordinate system in which the direction from the HOE 32B toward the detection unit 20 is the Y-axis direction is set.

  In FIG. 8B, the user's eye UE includes an iris IR and a pupil PU. The iris IR and the pupil PU are substantially circular, and the circle of the iris IR is Si and the circle of the pupil PU is Sp. The center of the circle Si and the circle Sp, that is, the center (viewpoint) of the pupil PU is p0 (xp, yp, zp).

  Further, the intersection of the normal line n drawn from p0 toward the XY plane (eyepiece plane ES) of the light guide 30 and the XY plane is defined as s (xs, ys), and the azimuth angle of the normal line n (left and right angles) ) And elevation angle (up and down angles) are θx and θy, respectively. In this way, the line connecting p0 and s corresponds to the user's line of sight.

  Since the direction of the line connecting p0 and s is perpendicular to the plane including the pupil PU, it can be calculated based on the direction of the plane including the circle Sp (hereinafter referred to as the Sp plane). The arithmetic control unit 25 can calculate the distance between the eyepiece surface ES of the light guide unit 30 and the iris IR and the shape of the iris IR in the above image from the images of the user's eye UE captured by the two cameras 22A and 22B. .

  More specifically, the two cameras 22A and 22B have a predetermined baseline length, and image the user's eye UE. The arithmetic control unit 25 compares the images of the iris IR and the pupil PU from two directions imaged by the two cameras 22A and 22B, and based on the principle of stereoscopic vision (the principle of triangulation), the eyepiece of the light guide unit 30 The distance between the surface ES and the iris IR is calculated. In addition, the arithmetic control unit 25 recognizes the image and extracts the shape of the iris IR in the image. Then, the calculation control unit 25 calculates the coordinates of p0 and the inclination of the Sp plane with respect to the XY plane based on the distance and the shape.

  The shape of the iris IR in the above image changes depending on which direction the iris IR is oriented with respect to the XY plane. The arithmetic control unit 25 calculates the slope of the Sp plane with respect to the XY plane by calculating the distance between the XY plane and a plurality of points on the iris IR based on the shape of the iris IR in the video. Then, the arithmetic control unit 25 derives the normal line n based on the coordinates of p0 and the inclination of the Sp plane with respect to the XY plane, and the position s (xs, ys) where the normal line n crosses the XY plane and the direction of the normal line n The angle θx and the elevation angle θy are calculated.

  As described above, in the present embodiment, the position at which the line-of-sight detection device is mounted on the user is calculated by calculating the distance to the iris IR and the shape of the iris IR with reference to the eyepiece surface ES of the light guide unit 30. Even if changes, the three-dimensional positional relationship between the line-of-sight detection device 1 and the iris IR can be specified, and the user's viewpoint and line-of-sight direction can be accurately calculated.

  Further, in the present embodiment, the display unit 23 projects an image on the eye UE through the eyepiece surface ES of the light guide unit 30. As shown in FIG. 9, the image light IL from the display unit 23 passes through the HOE 32A, travels toward the HOE 32B while totally reflecting the interface between the transparent plate 31R and the outside air, and is reflected by the HOE 32B. Reach the user's eye UE.

  The arithmetic control unit 25 acquires in advance information on the display position of each display image displayed in the video, and collates the calculated position s (xs, ys) with the information on the display position. The display image ahead of the user's line of sight can be specified. That is, it can identify what the user is looking at. The arithmetic control unit 25 functions as a specifying unit.

  Furthermore, when an image that the user is paying attention to is specified, the image displayed in the video can be controlled according to the image that the user is viewing. For example, the icon can be selected by moving the line of sight to the icon displayed on the screen. In particular, since the line-of-sight detection apparatus 1 of the present embodiment can accurately detect the user's viewpoint and line-of-sight direction, the user can accurately select an icon even when a large number of small icons are displayed on the screen. Further, the position where the image is displayed may be controlled by the user's viewpoint and line-of-sight direction.

  The line-of-sight detection device 1 of the present embodiment described as described above has the following effects.

  Since the line-of-sight detection device 1 of the present embodiment measures the positional relationship between the light guide unit 30 and the pupil PU or iris IR of the user's eye UE, the position where the line-of-sight detection device 1 is attached to the user is determined. Even if it changes, the user's viewpoint and line-of-sight direction can be accurately detected.

  In addition, the line-of-sight detection device 1 according to the present embodiment can capture the user's eye UE by irradiating the user's eye UE with infrared light, even in places where there is little visible light such as at night or in a dark room. In addition, by using infrared light, it is possible to avoid noise from the external image when the user's eye UE is imaged, so that the image of the user's eye UE can be accurately recognized.

  In addition, since the HOEs 32A and 32B of the line-of-sight detection device 1 according to the present embodiment have a function of reflecting infrared light in addition to visible light for display, they can have both a display function and an illumination function. Moreover, in this embodiment, the user's eye UE can be imaged from substantially the front by using the HOE 32B. Therefore, it is possible to accurately detect the viewpoint and the line-of-sight direction while securing the user's field of view.

  In addition, since the imaging unit 22 of the line-of-sight detection device 1 images the user's eye UE in a stereo manner using the two cameras 22A and 22B, the optical system between the imaging unit 22 and the light guide unit 30 is minimized (optical path). The length can be kept to the shortest). As a result, the line-of-sight detection device 1 can be made thin, small, and lightweight.

(Second Embodiment)
In the first embodiment, the line-of-sight detection apparatus has been described by taking as an example an HMD that captures the user's eyes in stereo using two cameras. In the second embodiment, a line-of-sight detection apparatus will be described by taking an HMD having an array camera including a plurality of cameras as an example. In the following, detailed description of the same configuration as that of the first embodiment will be omitted to avoid duplication of description.

  In general, the array camera includes a super-resolution type, a field division type, and an insect type, and any of the array cameras can be used in this embodiment. Hereinafter, an embodiment in which the array camera is used will be described with reference to FIGS. 10A to 13.

<Super-resolution array camera>
FIG. 10A is a diagram showing the arrangement of the light and camera of the detection unit when a super-resolution array camera is used in the second embodiment, and FIG. 10B is a view of the light guide unit from the detection unit of FIG. 10A. It is a conceptual diagram for demonstrating the optical path of the irradiation light and reflected light at the time. FIG. 10C is a conceptual diagram showing another embodiment in the case of imaging with a super-resolution array camera in the second embodiment.

  As shown in FIGS. 10A and 10B, in the present embodiment, the illumination unit 21 includes lights 21A and 21B. The lights 21A and 21B irradiate the user's eye UE from two directions. The imaging unit 22 has an array camera 22C. The array camera 22C is an array of cameras arranged in a grid, and is arranged between the light 21A and the light 21B, and can capture the same region of the user's eye UE from a plurality of directions.

  As shown in FIG. 10C, when the array camera 22C is viewed in more detail, each camera of the array camera 22C includes an image sensor 221C and an imaging lens 222C. The imaging lens 222C has an angle of view α, and each camera of the array camera 22C can capture a predetermined area corresponding to the angle of view α. Therefore, each camera of the array camera 22C can overlap the imaging area with respect to other cameras while slightly different imaging areas between adjacent cameras. That is, the array camera 22C has a plurality of stereo cameras arranged.

  In the present embodiment, based on images of the user's eye UE captured from a plurality of directions by the array camera 22C, the positional relationship between the light guide unit 30 and the pupil PU or iris IR of the user's eye UE is measured, Based on the positional relationship, the viewpoint and the line-of-sight direction are calculated.

<Division-of-view array camera>
FIG. 11 is a conceptual diagram for explaining optical paths of irradiation light and reflected light when the light guide unit is viewed from the detection unit in the case of using the divided-field array camera in the second embodiment.

  In the form shown in FIG. 11, the imaging surface of the array camera 22D is divided into a plurality of regions, and the optical system is arranged so as to capture the same portion of the user's eye UE in each region. In the example shown in the figure, it is divided into two areas. In FIG. 11, the array camera 22D is divided into left and right, an optical system having two fields of view is arranged in each of the left and right regions, and the user's eyes are imaged.

  The line-of-sight detection device 1 has a positional relationship between the light guide unit 30 and the pupil PU or iris IR of the user's eye UE based on images of the user's eye UE captured from two directions of the two areas of the array camera 22D. And the viewpoint and line-of-sight direction are calculated based on the positional relationship.

<Insect type array camera>
FIG. 12 is a conceptual diagram for explaining an optical path of reflected light when an insect type array camera is used in the second embodiment.

  As illustrated in FIG. 12, the imaging unit 22 includes an array camera 22E. The array camera 22E includes a plurality of imaging lenses 222E arranged in a lattice pattern and an imaging element 221E in which pixels are arranged so that one pixel corresponds to each lens, and reflects reflected light from the user's eye UE in a plurality of directions. Can receive light. Note that the arrangement of the lights in the illumination unit 21 and the optical path of the irradiation light by the lights are the same as in the case of the field-of-view split-type array camera, and thus the description thereof is omitted.

  In the form shown in FIG. 12, the imaging system of the array camera 22E is divided into two regions, and the optical system is arranged so as to capture the same part of the user's eye UE when the field of view of the single-eye camera in each region is matched. To do.

  The line-of-sight detection device 1 has a positional relationship between the light guide unit 30 and the pupil PU or iris IR of the user's eye UE based on images of the user's eye UE captured from two directions of the two areas of the array camera 22E. And the viewpoint and line-of-sight direction are calculated based on the positional relationship.

  In the above, for convenience of explanation, the case where the array camera includes a small number of cameras has been described. However, the number of cameras included in the array camera is not limited.

  The line-of-sight detection device 1 of the present embodiment described as described above has the following effects.

  Since the line-of-sight detection apparatus 1 uses the array camera to capture the same region of the user's eye UE from a plurality of directions, the stereo method for attaching the plurality of cameras and capturing the user's eye UE is the same with a single array camera. Can be obtained. Therefore, the camera can be mounted thinly and accurately, and the position between the eyepiece surface ES of the light guide unit 30 and the pupil PU or iris IR of the user's eye UE based on the captured image of the user's eye UE The relationship can be measured accurately. As a result, the user's viewpoint and line-of-sight direction can be accurately detected.

  In addition, if the insect type array camera 22E is used, the imaging lens 222E is very thin because it is arranged on the chip without a distance, so that the thickness of the entire array camera can be reduced.

(Third embodiment)
In the first and second embodiments, the line-of-sight detection device has been described by taking the HMD that images the user's eye with the compound eye imaging device as an example. In the third embodiment, a line-of-sight detection apparatus will be described by taking an HMD having a TOF (Time Of Flight) camera as an example. In the following, detailed description of the same configuration as that of the first embodiment will be omitted to avoid duplication of description.

  FIG. 13 is a diagram showing the arrangement of the lights and cameras of the detection unit in the third embodiment.

  As shown in FIG. 13, the illumination unit 21 of the present embodiment includes an LED light 21C. The imaging unit 22 includes a TOF camera 22F. In the present embodiment, the arithmetic control unit 25 controls the illumination unit 21 to irradiate the user's eye UE with pulsed light using the LED light 21C. The TOF camera 22F receives the reflected light reflected by the iris IR and retina of the user's eye UE. The arithmetic control unit 25 two-dimensionally calculates the delay time of the reflected light with respect to the pulsed light based on the light reception signal from the TOF camera 22F. Since the delay time is proportional to the distance from the eyepiece surface ES of the light guide unit 30 to the iris IR of the user's eye UE, the relationship between the delay time and the distance is determined in advance by a lookup table or a mathematical formula. As required. Therefore, the arithmetic control unit 25 can calculate the distance based on the delay time. Note that the imaging element of the TOF camera 22F has a pixel structure that can detect the delay time of reflected pulsed light.

  The line-of-sight detection device 1 of the present embodiment described as described above has the following effects.

  The line-of-sight detection device 1 emits pulsed light from the light 21C and is received by the TOF camera 22F, calculates a delay time of reflected light with respect to the irradiated pulsed light, and the eyepiece surface ES of the light guide unit 30 based on the delay time. To the iris IR of the user's eye UE, and the tilt of the iris IR can be accurately measured. Therefore, the user's viewpoint and line-of-sight direction can be accurately detected. Note that the intensity of the pulsed light is such that it is not confused with external light.

  In this embodiment, since the light from the light 21C is received by the TOF camera 22F, a plurality of cameras are not used, and the optical system between the imaging unit 22 and the light guide unit 30 is minimized (the optical path length is reduced). The shortest). As a result, the line-of-sight detection device 1 can be made thin, small, and lightweight.

(Fourth embodiment)
In the fourth embodiment, a case will be described in which in addition to the configuration of the first embodiment, a subject imaging unit that images a subject is further included.

  FIG. 14 is an external view for explaining a case where a stereo camera for imaging a subject is provided in the fourth embodiment. FIG. 15 is a case where an array camera for imaging a subject is provided in the fourth embodiment. It is an external view for demonstrating. In the following, detailed description of the same configuration as that of the first embodiment will be omitted to avoid duplication of description.

  In the form shown in FIG. 14, the line-of-sight detection device 1 includes stereo cameras 40A and 40B as subject imaging units (subject imaging means). Stereo cameras 40A and 40B are integrated with transparent plate 31R.

  In the form shown in FIG. 15, the line-of-sight detection device 1 includes an array camera 41 as a subject imaging unit (subject imaging unit). The array camera 41 is integrated with the transparent plate 31R.

  As shown in FIGS. 14 and 15, in this embodiment, a subject is also imaged by a stereo camera, an array camera, a TOF camera, or the like, and an external space or object is recognized three-dimensionally. Therefore, the subject ahead of the user's line of sight can be accurately identified based on the video image of the subject and the detected viewpoint and line-of-sight direction.

  The line-of-sight detection device 1 of the present embodiment described as described above has the following effects in addition to the first embodiment.

  According to the present embodiment, the line-of-sight detection device 1 is mounted with the stereo cameras 40A and 40B or the array camera 41, thereby three-dimensionally representing an object in the outside world that the user wearing the line-of-sight detection device 1 is looking at. I can grasp it. Therefore, based on the user's viewpoint and line-of-sight direction, what the user is looking at in the outside world can be accurately specified.

  As described above, the visual line detection device of the present invention has been described in the embodiment. However, it goes without saying that the present invention can be appropriately added, modified, and omitted by those skilled in the art within the scope of the technical idea.

  For example, in the first to fourth embodiments, the HMD is described as an example of the visual line detection device, but the visual line detection device of the present invention is not limited to the HMD. The present invention may be configured not to be fixed to the user's head like binoculars or a vision test device, but to look into the light guide when used by the user.

  In the first and second embodiments, the positional relationship between the eyepiece surface of the light guide unit and the pupil or iris of the user's eye is irradiated with infrared light to the user's eye and the reflected light is used. Was described. However, if the reflected light from the user's eyes can be received sufficiently, such as when the line-of-sight detection device is used in a bright place, the reflected light from the user's eyes by sunlight (natural light) without using infrared light May be used.

  In addition, the iris of the user's eye can be imaged in color using the light of the display unit. For example, when the display unit has a transmissive display with a backlight, when the line-of-sight detection device is calibrated (at the start of use) or the like, the display is white so that the backlight is turned on. The eye can be illuminated with white light. In addition, even when the display unit does not have a backlight, by imaging the user's eyes irradiated with light from the image from the display unit, by performing image processing to remove the video component from the display unit, It is also possible to image the iris of the user's eye in color.

  In addition, by using an imaging device equipped with a WYRIr filter, it is possible to capture in color using visible light while imaging with high sensitivity using infrared light in one imaging unit. The WYRIr filter is a filter capable of separating and extracting RGB signals while having sensitivity in the infrared region (Ir).

  Further, as an additional function of the line-of-sight detection device, a function of simultaneously performing iris authentication, retina authentication, and the like can be provided.

  In the first to fourth embodiments, the detection unit and the light guide unit are arranged on a transparent plate corresponding to the right eye of the user wearing the visual line detection device, and the right eye viewpoint and the visual line direction are detected. I mainly explained that. However, the positions of the detection unit and the light guide unit are not limited to the transparent plate corresponding to the user's right eye. Further, the viewpoint and line-of-sight direction of the user's left eye may be detected, or the viewpoint and line-of-sight direction of both eyes may be detected.

  In the first to fourth embodiments, the case has been described in which the display light travels with total reflection on the interface between the light guide unit and external air. However, the present invention is not limited to such a case. For example, the light guide unit may be configured such that the display light is directly diffracted by the HOE and does not go through the process of totally reflecting the interface between the light guide unit and external air.

  In addition, this application is based on the Japan patent application 2013-153768 for which it applied on July 24, 2013, The content of an indication is referred as a whole by reference.

ES eyepiece,
IR iris,
IL display light,
LA, LB irradiation light,
PU pupil,
RA, RB reflected light,
The eyes of the UE user,
1 gaze detection device,
10 mounting part,
11R, 11L first support member,
12R, 12L second support member,
13 connecting part,
20 detector,
21 Lighting section,
21A-21C light,
22 imaging unit,
22A-22F camera,
23 display section,
24 communication department,
25 arithmetic control unit,
26 Bus,
30 light guide,
31R, 31L transparent plate,
32A, 32B HOE,
40A, 40B, 41 Imaging unit for subject.

Claims (14)

A light receiving means for receiving light from a user's eye and generating a light reception signal according to the intensity of the light;
A light guiding means for guiding light from the eye to the light receiving means;
Measuring means for measuring a positional relationship between the light guiding means and the pupil or iris of the eye based on the light reception signal;
A line-of-sight detection device comprising: a calculation unit that calculates a position of the user's pupil based on the positional relationship. The measuring means includes
Measure the distance between the eyepiece surface of the light guide means and the iris as the positional relationship,
The calculating means includes
Calculating the position of the pupil and the orientation of the pupil relative to the eyepiece based on the distance;
The gaze detection apparatus according to claim 1, wherein a gaze direction is calculated based on a direction of the pupil. The light guiding means includes
The line-of-sight detection device according to claim 1, wherein a path is changed by diffracting light from the eye and guided to the light receiving unit. The light receiving means includes a compound eye imaging device, generates an image signal as the light reception signal,
The line-of-sight detection apparatus according to claim 2, wherein the measurement unit measures the distance based on the image signal. The light receiving means has a stereo camera as a compound eye imaging device, generates an image signal as the light reception signal,
The line-of-sight detection apparatus according to claim 2, wherein the measurement unit measures the distance based on the image signal. The light receiving means has an array camera as a compound eye imaging device, generates an image signal as the light reception signal,
The line-of-sight detection apparatus according to claim 2, wherein the measurement unit measures the distance based on the image signal.   The line-of-sight detection apparatus according to claim 1, further comprising an illuminating unit that irradiates the user's eyes with irradiation light. And further comprising illumination means for irradiating the user's eyes with irradiation light,
The illumination means irradiates pulse light as the irradiation light,
The light receiving means has a TOF camera, receives the reflected light reflected from the eye with the TOF camera, calculates a delay time of the reflected light with respect to the pulsed light based on the received light signal, The line-of-sight detection apparatus according to claim 1, wherein the distance is measured based on the distance.   The light guide means has a holographic optical element, changes the path by diffracting the irradiation light by the holographic optical element, guides it to the eye, and receives the reflected light reflected by the eye. The line-of-sight detection apparatus according to claim 7, wherein the line-of-sight detection device is guided to the means.   The line-of-sight detection apparatus according to claim 1, wherein the light guide unit includes a transparent plate, and the transparent plate includes the eyepiece surface.   The line-of-sight detection apparatus according to claim 1, wherein the line-of-sight detection apparatus is configured to be worn on the head of the user.   The line-of-sight detection device according to claim 7, wherein the irradiation light is infrared light. An imaging means for a subject that images an external subject;
And a specifying unit configured to specify a target ahead of the user's line of sight in the video imaged by the subject imaging unit based on the line-of-sight direction calculated by the calculation unit. Item 13. The line-of-sight detection device according to any one of items 2 to 12.   The line-of-sight detection apparatus according to claim 13, wherein information on an object ahead of the user's line of sight is displayed on a display unit based on a specifying unit that identifies the object beyond the line of sight of the user.