KR20170135762A - Gaze detection system - Google Patents

Abstract

The present invention provides a gaze detecting system capable of checking whether a user is gazing at a marker or not during calibration. The system includes: a head mount display used by being mounted on a user; and a gaze detecting device detecting a gaze of the user. The head mount display includes: a display part displaying an image; a photographing part photographing the eyes of the user; and an image output part outputting an image of the eyes photographed by the photographing part to the gaze detecting device. The gaze detecting part includes: a marker image output part outputting a marker image to be displayed in the display part; and a synthetic image making part making a synthetic image by overlapping the image of the eyes photographed by the photographing part with the marker image outputted by the marker image output part.

Description

[0001] GAS DETECTION SYSTEM [0002]

The present invention relates to a line-of-sight detection system, and more particularly to a line-of-sight detection technique using a head-mounted display.

Conventionally, it has been necessary to carry out the calibration when the user performs the sight line detection to specify the place where the user is looking. Here, the calibration refers to specifying the position of the specific indicator on the user and the positional relationship between the center of the cornea of the user who watches the specific indicator and the specific indicator. By performing the calibration, the line-of-sight detection system for detecting the line-of-sight can specify the place where the user is looking.

Patent Literature 1 discloses a technique of performing visual line detection by performing calibration (see, for example, Patent Document 1).

Patent Document 1: JP-A-2012-216123

However, since the calibration is performed under the condition that it is determined that the user is watching the specific indicator, the information is obtained in a state in which the user is not watching the specific indicator. Therefore, There was a problem that it could not. In the case of the head mount display in which the inside of the user's eyes is hidden by the apparatus and the inside appearance can not be visually recognized, the above problem can not be confirmed by the operator from the surroundings whether or not the user is actually watching the specific indicator. did.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a technique capable of accurately performing calibration for realizing visual line detection of a user wearing a head mount display.

According to one aspect of the present invention, there is provided a line-of-sight detection system including a head-mounted display mounted on a user and a line-of-sight detecting apparatus for detecting a line of sight of a user, And an image output section for outputting an image including eyes of a user who has captured the image pickup section to the visual line detecting device, wherein the visual line detecting device comprises a display section for displaying a marker image to be displayed on the display section A marker image output unit for outputting the marker image output by the marker image output unit, and a marker image generated by the marker image output unit and the image of the user who is watching the marker image picked up by the image pickup unit, And a composite image output unit for outputting a composite image.

The marker image output section may change the display position of the marker image in order and output the same, and the imaging section may image the user's eye watching the marker image at least every time the display position is changed.

The marker image output unit changes the display position of the marker image to any one of a plurality of predetermined coordinate positions and outputs the marker image. The eye-gaze detection apparatus displays the image of the user's eye captured by the imaging unit, And a visual line detecting section for detecting the visual line direction of the user based on each of the images including the eyes of the user watching the marker image for each position.

The judgment unit further judges whether or not the user is watching the displayed marker image based on the image of the user's eye captured by the image pickup unit and the visual line detection system judges that the user is not watching the marker image And when the determination is made, notifying the user to watch the marker image.

When the determination unit determines that the user is watching the displayed marker image, the marker image output unit may change the display position of the marker image.

The visual line detection system may further include a determination unit that determines whether or not an image including a user's eye watching the marker image can be used as a visual line detection image by the visual line detection unit, The marker image output unit changes the display position of the marker image displayed when the image corresponding to the determination is captured to the vicinity of the center of the display unit to display the marker image, And the judging unit may judge whether or not the comparison image picked up again is usable as the image for line-of-sight detection.

Also, any combination of the above components, and the expression of the present invention may be converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium and the like is also effective as an aspect of the present invention.

According to the present invention, it is possible to provide a technique for detecting the direction of the user's gaze on which the head-mounted display is mounted.

BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is an external view showing a state in which a user mounts a head mount display according to an embodiment. FIG.
2 is a perspective view schematically showing an overview of an image display system of a head mount display according to the embodiment.
3 is a diagram schematically showing an optical configuration of an image display system of a head mount display according to the embodiment.
4 is a block diagram showing a configuration of a visual-line detecting system according to the embodiment.
Fig. 5 is a schematic diagram for explaining the calibration for detecting the gaze direction in the embodiment. Fig.
6 is a schematic diagram for explaining the position coordinates of the cornea of the user.
In Fig. 7, (a) to (c) are image views of the user's eye watching the marker image according to the embodiment.
8 is a flowchart showing the operation of the visual-line detecting system according to the embodiment.
In Fig. 9, (a) is an image showing the output position of the marker image before correction on the display screen. (b) is an image diagram showing an example of correction of the output position of the marker image.
10 is a block diagram showing a configuration of a visual line detection system.
11 is a block diagram showing the configuration of the visual-line detecting system according to the second embodiment.
12 is a diagram showing a display example of an effective visual field graph according to the second embodiment.
13 is a flowchart showing the operation of the visual-line detecting system according to the second embodiment.
14 is a flowchart showing an operation of the visual-line detecting system according to the second embodiment
15 is a diagram showing a display example of an effective visual field graph according to the third embodiment.
16 is a flowchart showing the operation of the visual-line detecting system according to the third embodiment.
17 is a diagram schematically showing an example of display of a marker image according to the fourth embodiment.
18 is a flowchart showing the operation of the visual-line detecting system according to the fourth embodiment.
Fig. 19 is a block diagram showing the configuration of a visual-line detecting system according to Embodiment 5. Fig.
20 is a head-mounted display according to Embodiment 5, wherein (a) is a plan view of the driving portion. (b) is a perspective view of the driving unit.
21 is a flowchart showing the operation of the visual-line detecting system according to the fifth embodiment.
22 is a flowchart showing the operation of the visual-line detecting system according to the fifth embodiment.

≪ Embodiment 1 >

<Configuration>

Fig. 1 schematically shows an overview of the visual-line detecting system 1 according to the embodiment. The visual-line detecting system 1 according to the embodiment includes a head-mounted display 100 and a visual-line detecting device 200. [ As shown in Fig. 1, the head mount display 100 is mounted on the head of the user 300 and used.

The visual-line detecting device 200 detects the visual direction of the right eye and the left eye of the user wearing the head-mounted display 100 to detect the focus of the user, that is, the three-dimensional image displayed on the head- And specifies a point of interest. The visual-line detecting device 200 also functions as an image generating device for generating an image displayed by the head-mounted display 100. As an example, the eye-gaze detecting apparatus 200 is a device capable of reproducing images of a stationary game machine, a portable game machine, a PC, a tablet, a smart phone, a tablet, a video player, a television, and the like. The visual-line detecting device 200 is connected to the head-mounted display 100 wirelessly or by wire. In the example shown in Fig. 1, the visual-line detecting device 200 is connected to the head-mounted display 100 wirelessly. The wireless connection of the head-mounted display 100 by the visual-line detecting device 200 can be realized by using a wireless communication technology such as existing Wi-Fi (registered trademark) or Bluetooth (registered trademark). The image transmission between the head mount display 100 and the visual-line detecting device 200 is performed in accordance with standards such as Miracast (trademark), WiGig (trademark), WHDI (trademark) .

Fig. 1 shows an example in which the head-mounted display 100 and the visual-line detecting device 200 are different devices. However, the visual-line detecting device 200 may be incorporated in the head-mounted display 100. [

The head mount display 100 includes a case 150, a mounting hole 160, and a headphone 170. The case 150 accommodates an image display system for presenting an image to the user 300 such as an image display device or a wireless transmission module such as a Wi-Fi module or a Bluetooth (registered trademark) module not shown. The mount 160 mounts the head mount display 100 on the head of the user 300. The mounting hole 160 can be realized by, for example, a belt or a stretchable band. When the user 300 mounts the head mount display 100 using the mount 160, the case 150 is disposed at a position covering the eyes of the user 300. As a result, when the user 300 mounts the head mount display 100, the field of view of the user 300 is blocked by the case 150.

The headphone 170 outputs the audio of the video reproduced by the visual-line detecting device 200. [ The headphone 170 does not need to be fixed to the head mount display 100. [ The user 300 can freely attach and detach the headphone 170 even when the head mount display 100 is mounted using the mounting hole 160. [

2 is a perspective view schematically showing an overview of the image display system 130 of the head mount display 100 according to the embodiment. More specifically, FIG. 2 is a view showing a region of the case 150 of the embodiment opposed to the cornea 302 of the user 300 when the head-mounted display 100 is mounted.

2, the convex lens 114a for the left eye is placed so that the position of the convex lens 114a is opposite to the position of the cornea 302a of the left eye of the user 300 when the user 300 is mounted on the head mount display 100 . Likewise, the convex lens 114b for the right eye is arranged so as to face the cornea 302b of the right eye of the user 300 when the user 300 is mounted on the head mount display 100. [ The convex lens 114a for the left eye and the convex lens 114b for the right eye are held by the left and right lens support portions 152a and 152b, respectively.

In the following description, the term "convex lens 114 " is simply referred to as " convex lens 114" unless otherwise specifically distinguished between the convex lens 114a for the left eye and the convex lens 114b for the right eye. Cornea 302 is simply referred to as "cornea 302" except for the case where the cornea 302a of the left eye of the user 300 and the cornea 302b of the right eye of the user 300 are specifically distinguished. The lens supporting portion 152a for the left eye and the lens supporting portion 152b for the right eye are also referred to as "lens supporting portion 152" unless otherwise specified.

The lens support portion 152 is provided with a plurality of infrared light sources 103. [ 2, an infrared light source for irradiating infrared light to the cornea 302a of the left eye of the user 300 is denoted by an infrared light source 103a, and the right eye of the user 300 An infrared light source 103b for irradiating infrared light to the cornea 302b of the eye 100b. Hereinafter, it is described as "infrared light source 103" unless the infrared light source 103a and the infrared light source 103b are specifically distinguished. In the example shown in Fig. 2, the left eye lens supporting portion 152a is provided with six infrared light sources 103a. Likewise, the right eye lens support portion 152b is provided with six infrared light sources 103b. As described above, the infrared light source 103 can be easily mounted by disposing the infrared light source 103 in the lens supporting portion 152 for holding the convex lens 114 without directly placing the infrared light source 103 on the convex lens 114 It becomes. In general, since the lens supporting portion 152 is made of resin or the like, processing for mounting the infrared light source 103 is easier than the convex lens 114 composed of glass or the like.

As described above, the lens supporting portion 152 is a member holding the convex lens 114. Therefore, the infrared light source 103 provided in the lens supporting portion 152 is arranged around the convex lens 114. [ Here, the number of the infrared light sources 103 for irradiating the infrared light to each eye is six, however, the number is not limited to this, and at least one infrared light source 103 may be provided corresponding to each eye, Or more.

3 is a diagram schematically showing the optical configuration of the image display system 130 accommodated in the case 150 according to the embodiment and is a view in which the case 150 shown in Fig. 2 is viewed from the side of the left eye side . The image display system 130 includes an infrared light source 103, an image display element 108, a hot mirror 112, a convex lens 114, a camera 116, and a first communication unit 118.

The infrared light source 103 is a light source capable of irradiating light in a near-infrared (about 700 nm to 2500 nm) wavelength band. The near-infrared light is generally in a wavelength band of invisible light which can not be observed with the naked eye of the user 300.

The image display element 108 displays an image to be presented to the user 300. The image displayed by the image display device 108 is generated by the image output unit 222 in the visual-line detecting device 200. [ The video output unit 222 will be described later. The image display device 108 can be implemented using, for example, a conventional LCD (Liquid Crystal Display) or an organic EL display (Organic Electro Luminescence Display).

The hot mirror 112 is disposed between the image display element 108 and the cornea 302 of the user 300 when the user 300 mounts the head mount display 100. [ The hot mirror 112 has the property of transmitting the visible light generated by the image display element 108, but reflecting the near infrared light.

The convex lens 114 is disposed on the side opposite to the image display device 108 with respect to the hot mirror 112. In other words, the convex lens 114 is disposed between the hot mirror 112 and the cornea 302 of the user 300 when the user 300 mounts the head mount display 100. That is, the convex lens 114 is disposed at a position opposed to the cornea 302 of the user 300 when the head-mounted display 100 is attached to the user 300.

The convex lens 114 condenses the image display light transmitted through the hot mirror 112. Thus, the convex lens 114 functions as an image enlargement unit for enlarging the image generated by the image display device 108 and presenting it to the user 300. [ 2, only one convex lens 114 is shown in FIG. 2, but the convex lens 114 may be a lens group composed of various lenses in combination, and may be a lens group having a curvature on one side and a flat lens on the other side It may be a convex lens.

A plurality of infrared light sources (103) are arranged around the convex lens (114). The infrared light source 103 emits infrared light toward the cornea 302 of the user 300.

Although not shown, the image display system 130 of the head mount display 100 according to the embodiment is provided with two image display elements 108, and displays an image for presentation on the right eye of the user 300, An image for presentation can be generated independently. Thus, the head-mounted display 100 according to the embodiment can present a parallax image for the right eye and a parallax image for the left eye on the right eye and the left eye of the user 300, respectively. Thus, the head mount display 100 according to the embodiment can present a stereoscopic image with a depth sense to the user 300. [

As described above, the hot mirror 112 transmits visible light and reflects near-infrared light. Therefore, the image light irradiated by the image display element 108 passes through the hot mirror 112 and reaches the cornea 302 of the user 300. The infrared light irradiated from the infrared light source 103 and reflected by the reflective region inside the convex lens 114 reaches the cornea 302 of the user 300.

The infrared light reaching the cornea 302 of the user 300 is reflected by the cornea 302 of the user 300 and is directed toward the convex lens 114 again. The infrared light is transmitted through the convex lens 114 and reflected by the hot mirror 112. The camera 116 has a filter for blocking visible light, and picks up near-infrared light reflected by the hot mirror 112. That is, the camera 116 is a near-infrared camera that is irradiated from the infrared light source 103 and captures the near-infrared light reflected from the eyes of the user 300 by the cornea.

Although not shown, the image display system 130 of the head mount display 100 according to the embodiment includes two cameras 116, that is, a first image pickup device 130 for picking up an image including infrared light reflected from the right eye And a second image sensing unit for sensing an image including infrared light reflected from the left eye. Thereby, it is possible to acquire an image for detecting the gaze directions of both the right eye and the left eye of the user 300. [

The first communication unit 118 outputs the image captured by the camera 116 to the visual line detection device 200 which detects the visual line direction of the user 300. [ Specifically, the first communication unit 118 transmits the image captured by the camera 116 to the visual-line detecting apparatus 200. [ The details of the visual line detecting unit 221 and the visual line detecting unit 221 that function as the visual line direction detecting unit will be described later but they are realized by a visual line detecting program executed by a CPU (Central Processing Unit) of the visual line detecting device 200. [ When the head mount display 100 has a calculation resource such as a CPU or a memory, the CPU of the head mount display 100 may execute a program for realizing the visual direction detecting section.

However, the image captured by the camera 116 includes a bright spot caused by the near-infrared light reflected from the cornea 302 of the user 300 and a bright spot caused by the near infrared light reflected from the cornea 302 of the user 300 ) Is picked up.

The configuration for presenting the image to the left eye of the user 300 in the image display system 130 according to the embodiment has been described above. same.

Fig. 4 is a block diagram of the head-mounted display 100 and the visual-line detecting device 200 relating to the visual-line detecting system 1. Fig. As shown in Fig. 4, as described above, the line of sight detection system 1 includes the head-mounted display 100 and the line of sight detection apparatus 200 that communicate with each other.

4, the head mount display 100 includes a first communication unit 118, a first display unit 121, an infrared light irradiation unit 122, an image processing unit 123, and an image pickup unit 124 Respectively.

The first communication unit 118 is a communication interface having a function of communicating with the second communication unit 220 of the visual-line detecting device 200. [ As described above, the first communication unit 118 performs communication with the second communication unit 220 by wire communication or wireless communication. Examples of usable communication standards are as described above. The first communication unit 118 transmits the image data used for the sight line detection sent from the image pickup unit 124 or the image processing unit 123 to the second communication unit 220. [ The first communication unit 118 transmits the three-dimensional image data and the marker image transmitted from the visual-line detecting apparatus 200 to the first display unit 121. [

The first display unit 121 has a function of displaying the three-dimensional image data transmitted from the first communication unit 118 on the image display device 108. [ The three-dimensional image data includes right eye parallax images and left eye parallax images, and they are parallax image pairs. The first display unit 121 displays the marker image output from the marker image output unit 223 in the designated coordinates of the image display element 108. [

The infrared light irradiating unit 122 controls the infrared light source 103 to emit infrared light to the right eye or the left eye of the user.

The image processing section 123 performs image processing on the image picked up by the image pickup section 124 and transfers it to the first communication section 118 as necessary.

The image pickup section 124 uses the camera 116 to pick up an image including near-infrared light reflected from each eye. The image pickup section 124 picks up an image including a user's eye watching the marker image displayed on the image display element 108. [ The image pickup section 124 transmits the image obtained by picking up the image to the first communication section 118 or the image processing section 123. The imaging section 124 may capture a moving image or may capture a still image at an appropriate timing (for example, a timing of irradiating near-infrared light or a timing of displaying a marker image).

4, the visual line detecting apparatus 200 includes a second communicating section 220, a visual line detecting section 221, a video output section 222, a marker image output section 223, a judging section 224 A composite image output section 225, a second display section 226, and a storage section 227,

The second communication unit 220 is a communication interface having a function of communicating with the first communication unit 118 of the head mounted display 100. As described above, the second communication unit 220 performs communication with the first communication unit 118 by wire communication or wireless communication. The second communication unit 220 transmits the three-dimensional image data transmitted from the video output unit 222, the marker image transmitted from the marker image output unit 223, and the display coordinate position thereof to the head mount display 100 do. An image including a user's eye that looks at a marker image picked up by the image pickup unit 124 transmitted from the head mount display 100 is transmitted to the determination unit 224 and the composite image output unit 225, Based on the three-dimensional image data output from the image output unit 222, an image of the eye of the user who views the displayed image to the eye-gaze detection unit 221. [

The visual line detecting unit 221 receives the image data for visual line detection of the user's right eye from the second communication unit 220 and detects the visual line direction of the user's right eye. The line of sight detecting section 221 calculates a right eye line vector indicating the direction of the eye of the user's right eye and a left eye line vector indicating the direction of the eye of the user's left eye using the method described below, And identifies the portion of the image displayed on the display element 108 that the user is watching.

The image output unit 222 generates three-dimensional image data to be displayed on the first display unit 121 of the head mount display 100 and transmits the generated three-dimensional image data to the second communication unit 220. The video output unit 222 holds the coordinate system of the output three-dimensional image and information indicating the three-dimensional position coordinates of the object displayed in the coordinate system.

The marker image output unit 223 has a function of generating a marker image to be used as an index for carrying out calibration, which is a preparation for line of sight detection, and determining the display position. The marker image output unit 223 generates a marker image and determines a display coordinate position at which the marker image should be displayed in the image display element 108. [ The marker image output unit 223 transmits the generated marker image and its display coordinate position to the second communication unit 220 and instructs the second communication unit 220 to transmit it to the head mounted display 100. [ In the present embodiment, the marker image output section 223 changes the display position of the marker image in accordance with an input instruction from the operator of the visual-line detecting device 200. [

The marker image output unit 223 receives from the determination unit 224 a message indicating that an image including the user's eyes can not be used as a line of sight detection image and a display coordinate position of the marker image at that time, And sends a new display coordinate position and marker image obtained by changing the coordinate position to the coordinate position near the center of the image display element 108 to the second communication unit 220 and instructs the second communication unit 220 to transmit it to the head mount display 100.

The determining unit 224 uses the image of the user's eye in the image based on the image including the user's eye watching the marker image transmitted from the second communication unit 220 as an image for sight line detection And determines whether or not it is possible. Specifically, the determining unit 224 specifies the black character (cornea) of the user in the image including the user's eye watching the marker image transmitted from the second communication unit 220, and determines the center position It is judged whether or not it can be specified. The judgment section 224 judges that the image including the user's eyes watching the marker image transmitted from the second communication section 220 can not exclude the image as the image for sight line detection, The contents are transmitted to the marker image output unit 223 together with the display coordinate position of the marker image.

The synthesized image output unit 225 outputs an image obtained by inverting the left and right of the display position of the marker image output by the marker image output unit 223 and the eye of the user watching the marker image transmitted from the second communication unit 220 And a synthesized image is generated. The composite image output unit 225 outputs the generated composite image to the second display unit 226. [

The second display section 226 includes a monitor for displaying an image and has a function of displaying a composite image transmitted from the composite image output section 225. [ That is, the second display unit 226 displays a composite image in which an image of the user's eye watching the marker image and the marker image displayed at the corresponding position at that time are superimposed.

The storage unit 227 is a storage medium for storing various programs and data required for the visual-line detecting apparatus 200 to operate. 4, the storage unit 227 does not show a connection line to other functional units, but each functional unit accesses the storage unit 227 and refers to a necessary program and data appropriately. Next, the detection of the gaze direction related to the embodiment will be described.

Fig. 5 is a schematic diagram for explaining the calibration for detecting the gaze direction in the embodiment. Fig. The visual line direction of the user 300 is a direction in which the image captured by the camera 116 and output to the visual line detection device 200 by the first communication unit 118 is transmitted to the visual line detection unit 221 and the visual- (221).

The marker image output unit 223 generates nine points (marker images) from points Q ₁ to Q ₉ as shown in FIG. 5 and displays them on the image display element 108 of the head mount display 100 . The line of sight detecting apparatus 200 keeps watching the user 300 in turn until it reaches the point Q ₁ to Q ₉ . At this time, the user 300 is required to watch each point with only the motion of the eyeball without moving the neck. The camera 116 captures an image including the cornea 302 of the user 300 when the user 300 is looking at nine points from points Q ₁ to Q ₉ .

Fig. 6 is a schematic diagram for explaining the position coordinates of the cornea 302 of the user 300. Fig. The line of sight detecting unit 221 in the line of sight detecting apparatus 200 analyzes the image captured by the camera 116 and detects a luminescent spot 105 derived from the infrared light. It is considered that the position of the luminescent spot 105 does not move even when the user 300 is looking at each point only by the movement of the eyeball, even if the user is watching a certain point. Therefore, the line of sight detection unit 221 sets the two-dimensional coordinate system 306 in the image captured by the camera 116 based on the detected luminescent spot 105. [

The visual line detecting section 221 also detects the center P of the cornea 302 of the user 300 by analyzing the image captured by the camera 116. [ This can be realized, for example, by using known image processing such as half conversion or edge extraction processing. The eye gaze detection section 221 can acquire the coordinates of the center P of the cornea 302 of the user 300 in the two-dimensional coordinate system 306 set.

5, the coordinates of points Q ₁ to Q _{9 in} the two-dimensional coordinate system set on the display screen displayed by the image display device 108 are Q ₁ (x ₁ , y ₁ ) ^T and Q ₂ (x ₂ , y ₂ ) ^T _{... , And} Q ₉ (x ₉ , y ₉ ) ^T , respectively. Each coordinate is, for example, a pixel number located at the center of each point. The center P of the cornea 302 of the user 300 when the user 300 is looking at the points Q ₁ to Q ₉ is defined as a point P ₁ to P ₉ . At this time, the two-dimensional coordinate system 306, that P ₁ P ₉ ~ the coordinates of each of _{P 1 (X 1, Y 1} ) T, P 2 (X 2, Y 2) of the ^T, _{... , And} P ₉ (X ₉ , Y ₉ ) ^T , respectively. Also, T represents a transpose of a vector or a matrix.

Here, a matrix M having a size of 2x2 is defined as the following equation (1).

[Number 1]

At this time, if the matrix M satisfies the following formula (2), the matrix M becomes a matrix in which the viewing direction of the user 300 is projected on the image plane displayed by the image display device 108. [

(2)

(2)

When the above formula (2) is specifically rewritten, the following formula (3) is obtained.

[Number 2]

By modifying equation (3), the following equation (4) is obtained.

[Number 3]

From here,

[Number 4]

, The following equation (5) is obtained.

y = Ax (5)

In the equation (5), the element of the vector y is known because it is the coordinates of the points Q ₁ to Q _{9 that} the visual line detecting section 221 displays on the image display element 108. The element of the matrix A can be acquired because it is the coordinates of the apex P of the cornea 302 of the user 300. [ Therefore, the line-of-sight detecting section 221 can obtain the vector y and the matrix A. [ In addition, the vector x, which is the vector that lists the elements of the transformation matrix M, is unknown. Therefore, the problem of estimating the matrix M is a problem of obtaining an unknown vector x when the vector y and the matrix A are known.

If the number of expressions is larger than the number of unknowns (i.e., the number of elements Q of the vector x) (i.e., the number of points Q presented to the user 300 at the time of the visual line detection section 221) It becomes a matter of decision. In the example shown in equation (5), since the number of equations is nine, it is a problem of crystallization.

Let the vector y and the error vector of the vector Ax be the vector e. That is, e = y-Ax. In this case, the optimal vector x _opt in the sense that the square sum of the elements of the vector e is minimized is obtained by the following equation (6).

x _opt = (A ^TA ) ^-1 A ^T y (6)

Here, "-1"

The line-of-sight detecting section 221 constructs the matrix M of the equation (1) by using the elements of the obtained vector x _opt . The visual line detecting unit 221 can detect the right eye of the user 300 in accordance with the equation (2) by using the coordinates of the vertex P of the cornea 302 of the user 300 and the matrix M. Thus, It is possible to estimate, in a two-dimensional range, which portion on the moving image displayed by the display unit 108 is being watched. Thereby, the line of sight detecting unit 221 can calculate the right eye line vector connecting the main view of the right eye on the image display element 108 and the vertex of the cornea of the right eye of the user. Similarly, the line-of-sight detecting section 221 can calculate the left-eye line vector connecting the main viewpoint of the left eye on the image display element 108 and the apex of the cornea of the left eye of the user.

7 is a diagram showing an example of a composite image output by the composite image output section 225. Fig. Figure 7 (a) is, when the head is above the right side when the seat in the mounted display 100 is viewed, that is, the marker to the position of FIG say from 5 Q ₃ picture is displayed, the user to notice the art marker image FIG. 8 is a diagram showing an example of a synthesized image obtained by synthesizing an image of a left eye and a marker image displayed at a relative position with respect to the screen at that time; Further, in a state in which the user's eyes are being viewed, the position of the marker image is symmetrical.

Figure 7 (b), the head when in the mounted display 100 the user when viewing the center upper portion, that is, also put in 5 the marker image on the point Q ₂ position is displayed, the user to notice the art marker image And a synthesized image obtained by synthesizing a marker image displayed at a relative position with respect to the image at that time. Further, in a state in which the user's eyes are being viewed, the position of the marker image is symmetrical.

Figure 7 (c) is, when the head is the marker image in the position on the left side when the user viewed in the mounted display 100, that is, the point Q ₁ is also put in 5 is displayed, the user to notice the art marker image FIG. 8 is a diagram showing an example of a synthesized image obtained by synthesizing an image of a left eye and a marker image displayed at a relative position with respect to the screen at that time;

The synthesized image is displayed on the second display section 226 so that the operator of the visual line detection system 1 can confirm whether or not the user mounting the head mount display 100 is watching the marker image at the time of calibration Can be performed. Further, although not shown in Figure 7, such a composite image is to be displayed is generated with respect to nine points Q ₁ ~ Q ₉ each shown in Fig. 7 shows an example of the left eye of the user, but it is also possible to obtain the same composite image for the right eye of the user.

<Operation>

8 is a flow chart showing the operation of the visual-line detecting system 1 at the time of calibration. The operation of the visual line detection system 1 will be described with reference to Fig.

The marker image output unit 223 of the visual line detecting apparatus 200 sets i = 1 for the marker image Q _i to be displayed (step S801).

The marker image output section 223 causes the image display element 108 of the head mount display 100 to display the marker image at the i-th display coordinate position (step S802). That is, the marker image output unit 223 generates a marker image and determines the display coordinate position. For example, if i = 1, the point Q ₁ is determined as the display coordinate position. The marker image output unit 223 transmits the marker image generated in the second communication unit 220 and its display coordinate position to the second communication unit 220. [ The second communication unit 220 transmits the transmitted marker image and its display coordinate position to the head mount display 100. [

When the first communication unit 118 of the head mount display 100 receives the marker image and its display coordinate position, it transmits it to the first display unit 121. The first display unit 121 displays the transferred marker image on the image display element 108 at the designated display coordinate position. The user watches the displayed marker image. The image pickup section 124 picks up an image including a user's eye watching the displayed marker image (step S803). The image pickup section 124 transmits the picked-up image to the first communication section 118. The first communication unit 118 transmits the image data of the image of the user's eye that observes the transmitted marker image to the visual-line detecting apparatus 200. [

The second communication unit 220 of the visual-line detecting device 200 receives the image data of the image of the user's eye watching the marker image, and transfers the image data to the composite image output unit 225. [ The synthesized image output unit 225 superimposes the marker image displayed at that time on the image of the eye of the user watching the transmitted marker image and superimposes the marker image on the inverted position of the display position, (Step S804).

The composite image output unit 225 transfers the generated composite image to the second display unit 226, and the second display unit 226 displays the transferred composite image (step S805). Thereby, the operator of the line-of-sight detection system 1 can confirm whether or not the user wearing the head-mounted display 100 is watching the marker image, and when not watching, the user is instructed to watch the marker image can do.

The marker image output unit 223 determines whether i is 9 or not (step S806). If i is not 9, the marker image output unit 223 adds 1 to i and returns to step S802. If i is 9, the determination section 224 determines whether or not each of the nine images obtained by imaging is available as line-of-sight detection data (step S807). That is, the judging unit 224 judges whether or not the corneal center of the user can be specified for each of the images of the user's eyes that watch the marker image displayed at each display coordinate position. If it can be specified, the coordinate position is stored in the storage unit 227 and used for the determinant. If it can not be specified, the determination unit 224 determines whether or not the display position of the marker image that was displayed when the user's corneal center could not be specified with respect to the marker image output unit 223, The corneal center of the user can not be specified from the image of the user's eyes.

The marker image output section 223 corrects the display coordinate position of the marker image when the image of the user who could not specify the corneal center of the user is captured to the vicinity of the center of the screen (image display element 108). Then, the display coordinate position after correction is transmitted to the second communication unit 220. The second communication unit 220 transmits the delivered display coordinate position to the head mount display 100. The first communication unit 118 transmits the received display coordinate position after correction to the first display unit 121. [ The first display unit 121 displays the marker image at the display coordinate position after the correction, and notifies the user of the marker image. The image pickup section 124 picks up an eye of the user watching the marker image displayed at the display coordinate position after the correction (step S809). The image pickup section 124 transmits the captured image to the first communication section 118 and the first communication section 118 transmits the image to the visual line detection apparatus 200. [ Then, the process returns to step S808.

On the other hand, when the determining section 224 determines that all captured images can be used as data for visual line detection, that is, when the user's corneal center can be specified from all images, And the calibration process is terminated.

This concludes the description of the operation of the visual line detection system 1 at the time of calibration.

9 is an image diagram showing an example of a change in the display coordinate position of the marker image by the marker image output section 223. 9A is a diagram showing a basic position of a display position in an image display element 108 of a marker image. 9A shows nine marker images as a whole. Actually, these marker images are sequentially displayed on the image display element 108 one by one. That is, it is possible to obtain an image of the eyes of nine users.

At this time, when each of the marker images 901a, 902a, and 903a among the marker images shown in Fig. 9A is displayed at the coordinate display position shown in Fig. 9A, the user watches the marker image It can not be used for eye detection, that is, the determination unit 224 can not specify the corneal center of the user. Then, the determination section 224 delivers the content to the marker image output section 223. [

Upon reception of this, the marker image output unit 223 corrects the display coordinate position of the marker image displayed when the user's corneal center can not be specified, to be near the center of the screen. 9 (b), the display coordinate position of the marker image 901a is set to the display coordinate position shown in the marker image 901b, and the display coordinate position of the marker image 902a is set to the marker image 902b The display coordinate position of the marker image 903a is changed to the display coordinate position shown in the marker image 903b. Then, each marker image is displayed on the image display element 108 of the head mount display 100 at the display coordinate position after the correction, and an image including the eye of the user is watched. Then, the determination section 224 determines whether or not the center of the cornea of the user can be specified in the captured image again.

In Fig. 9 (b), both the x-axis direction and the y-axis direction are set near the center of the display coordinate position of the marker image. This is because even if only one axis is corrected near the center do. When the user is not able to specify the center of the cornea from the captured image by looking at the marker image whose display position has been corrected for only one axis, the display coordinate position of the marker image is set to It may be modified as near the center.

<Summary>

As described above, the eye gaze detection system 1 according to the present invention generates a composite image by superimposing a marker image and an image of the eye of the user watching it, ) Can confirm whether or not the user is watching the marker image at the time of calibration. Corresponding to the case where the cornea of the user is covered by the lower eyelid of the user at the time of image capture and the corneal center of the user can not be specified from the captured image, the gaze detection system 1 displays the marker image The corneal center of the user can be easily specified.

&Lt; Embodiment 2 >

In the first embodiment, the operator of the visual-line detecting device 200 has a significant configuration at the time of calibration for visual-line detection. In the second embodiment, a configuration capable of acquiring the characteristics of the user 300 will also be described. The visible mode or the visible range of the user 300 to which the head mount display 100 is attached varies depending on individual differences. Accordingly, it is desired to provide a system that is rich in usability by providing images according to individual characteristics. In the second embodiment, such a line-of-sight detection system will be described.

<Configuration>

11 is a block diagram showing the configuration of the visual-point detecting system according to the second embodiment. 11, the visual line detecting system includes a head mount display 100 and a visual line detecting device 200. [ 11, the head mount display 100 includes a first communication unit 118, a first display unit 121, an infrared light irradiation unit 122, an image processing unit 123, an image pickup unit 124 . The visual line detecting apparatus 200 further includes a second communication section 220, a visual line detection section 221, a video output section 222, a reception section 228, a specification section 229, a storage section 227 . The head-mounted display 100 and the visual-line detecting device 200 shown in Fig. 11 have the same functions as the head-mounted display 100 and the visual-line detecting device 200 shown in the first embodiment, respectively. In FIG. 11, the configuration is omitted for the configuration that is not related to the second embodiment. Hereinafter, description of the functions common to the first embodiment will be omitted, and only the other functions will be described.

The video output unit 222 transmits the display image of the valid visual field specific graph to the head mount display 100 through the second communication unit 220. The first display unit 121 of the head mount display 100 displays, And displays an effective visual field specific graph transmitted to the image display element 108. [

The reception unit 228 of the visual-line detecting apparatus 200 is configured such that the user 300 mounting the head-mounted display 100 has the user 300 in the effective visual field specific graph displayed on the image display element 108 And accepts the poet information indicating the manner in which the object is viewed. The receiving unit may be provided in the visual-line detecting apparatus 200 or may receive the input of the visibility information using the connected input interface. The receiving unit may receive input of the visual information from the second communication unit 220, It may be to accept information. The input interface may be, for example, a hard key of an input panel provided in the visual line detecting device, or may be a keyboard or a touch pad connected to the visual line detecting device 200. [ Alternatively, the accepting unit 228 may accept the input of speech uttered by the user 300. In this case, by analyzing the voice in accordance with the so-called speech recognition processing, And may accept input of the poet information. The accepting unit 228 transmits the accepted view information to the specifying unit 229. [

The effective visual field specific graph is a display image for specifying the effective field of view of the user 300 to which the head mount display 100 is attached and used. Fig. 12 shows an example of the effective visual field specific graph. Fig. 12 shows a display image 1200 in a state of being displayed on the image display element 108 of the head mount display 100. Fig.

As shown in FIG. 12, the effective visual field specific graph is composed of a main viewpoint marker 1202 indicating a main viewpoint to be watched by the user and a plurality of objects arranged in a circle around the main viewpoint marker 1202 It is a picture. Here, an example in which hiragana is arranged as each of a plurality of objects is shown, but this is merely an example, and other characters or images may be used. The plurality of objects are set to be larger in size as the distance from the main viewpoint marker 1202 becomes longer as the image is of a size in accordance with the distance from (the center of) the main viewpoint marker 1202. [ That is, when the distance between the coordinates of the center of the object and the coordinates of the center of the main viewpoint marker is l and the image size of the object at that time is x 占 y, the center coordinates of the object to be displayed and the coordinates of the main viewpoint marker 1202, Is 2L, the image size of the object is 2x x 2y.

The specifying unit 229 specifies the effective field of view of the user 300 based on the visibility information transmitted from the accepting unit 228. [

The user 300 specifies to which object the user can clearly visually check the main viewpoint markers 1202 of the effective viewpoint specific graph shown in Fig. 12 displayed on the image display device 108. Fig. The information of the object that the user 300 can visually clearly recognize while observing the main viewpoint marker 1202 becomes the visibility information in the second embodiment. For example, when the object distant from the main viewpoint marker 1202 in the object clearly visible to the user is "X, Q, R, S, T, U, V, W" A circle 1201 indicated by a dotted line is an effective field of view of the user 300.

The specifying unit 229 specifies the information of the object that the user 300 can see, which is indicated by the visibility information transmitted from the accepting unit 228. The specifying unit 229 determines whether or not the user 300 is to be displayed on the basis of the coordinate system of the effective visual field graph transmitted from the video output unit 222 to the head mount display 100 and the display position on the head mount display 100. [ Specify the effective field of view (coordinate range). Concretely, the display coordinates of the object, which can be clearly recognized by the user 300 appearing with the poet information while watching the main viewpoint marker, is specified. Then, in the circle whose radius is the distance from the main viewpoint marker 1202 to the coordinates at the farthest distance from the display coordinate range of the specific object, it is specified as the effective field of view of the user.

The image output unit 222 generates a high-resolution image based on the valid view field specified by the specifying unit 229 and the main viewpoint specified by the visual-line detecting unit 221. [ The image output unit 222 generates a high-resolution image of the image portion displayed by the specifying unit 229 in a range within the effective view area specified by the sight line detecting unit 221 at a specific point in time. Also, the video output unit 222 generates a low-resolution video for the entire screen. Then, the generated low-resolution image and the high-resolution image within the valid visual field are transmitted to the head mount display 100 through the second communication unit 220. [ The video output unit 222 may generate a low-resolution video for a range outside the effective viewing range.

Thereby, the visual line detecting apparatus 200 can transmit a high-resolution image in a range according to the effective visual field of each user to the head mount display 100. [ That is, it is possible to provide a high-quality image according to the visual characteristics of each user. Further, by narrowing the range of transmission of the high-resolution image to the effective visual range of the user, the data capacity can be suppressed more than that of transmitting the high-resolution image of the entire screen. The data transmission amount can be suppressed. This can be expected, for example, when the visual-line detecting apparatus 200 receives an image from an external image transmission server and transmits the image to the head-mounted display 100. [ That is, by specifying the gaze position and effective field of view of the user in the line of sight detection apparatus 200 and sending the information to the image delivery server, the image delivery server transmits the high-resolution image within the specified range and the low- The amount of data transferred from the video delivery server to the visual-line detecting device 200 can be suppressed.

<Operation>

13 is a flowchart showing an operation when the visual field detection apparatus 200 specifies the effective field of view of the user.

After performing the calibration shown in the first embodiment, the visual line detecting apparatus 200 reads out the effective visual field specific graph from the storage section 227. Then, the read valid visual field specific graph is transmitted to the head mount display 100 via the second communication unit 220 together with the display command (step S1301). Thereby, the first display unit 121 of the head mount display 100 receives the effective visual field graph via the first communication unit 118, and displays it on the image display element 108. The user 300 specifies an object that can be clearly recognized among the objects displayed in the vicinity of the main viewpoint markers of the displayed effective viewpoint graph.

Subsequently, the acceptance unit 228 of the visual-line detecting apparatus 200 determines whether or not the visual information of the user 300, which is the information of the object that the user 300 can see with the main- (Step S1302). This may be input directly by the user 300 or may be inputted from the user 300 by receiving the information of the object that the operator of the visual line detecting apparatus 200 can recognize, Whether or not the user 300 can visually recognize the blinked object clearly can be input to the reception unit 228 by accepting an input by pushing down a simple button at the time of blinking or the like You can. When the acceptance section 228 accepts the visibility information of the user 300, the acceptance section 228 delivers the accepted visibility information to the specification section 229. [

The specification unit 229 specifies an effective visual field of the user 300 when the user 300 receives the visual information of the user 300 from the reception unit 228. A specific method of effective view of the user is as described above. The specifying unit 229 generates effective view information of the specific user (information indicating a coordinate range centered on the main viewpoint of the user 300) and stores the information in the storage unit 227 (step S1303) And terminates.

By the above processing, the visual line detecting apparatus 200 specifies the effective visual field of the user 300 on which the head mounted display 100 is mounted.

Next, a method of using a specific effective field of view will be described. 14 is a flowchart showing an operation when an image to be displayed on the head mounted display 100 is generated based on the effective field of view of the user specified by the visual line detecting device 200. [ The operation shown in Fig. 14 is the operation when the image to be displayed on the head mount display 100 is being transmitted from the visual line detecting device 200. Fig.

The video output unit 222 generates an image having a low resolution as an image to be displayed on the image display device 108 of the head mount display 100. [ Then, the video output unit 222 transmits the generated low-resolution video to the head mount display 100 through the second communication unit 220 (step S1401).

The second communication unit 220 of the visual-line detecting device 200 receives a picked-up image of the eye of the user who is looking at the image displayed on the image display device 108 from the head-mounted display 100. The second communication unit 220 transmits the received captured image to the visual line detection unit 221. [ Then, the line-of-sight detecting section 221 specifies the viewing position of the user 300 as in the first embodiment (step S1402). The visual line detecting unit 221 transmits the specific viewing position to the video output unit 222. [

The video output unit 222 receives from the storage unit 227 the viewing position of the user 300 from the visual line detection unit 221, Read information. Then, a high-resolution image up to a range of the effective visual field represented by the effective visual field information is generated centering on the transmitted viewing position (step S1403).

The video output unit 222 transmits the generated high-resolution video to the head mount display 100 through the second communication unit 220 (step S1404).

The visual line detecting apparatus 200 determines whether the video output from the video output unit 222 has ended (reached the last frame) or not from the user 300 or from the operator of the visual line detecting apparatus 200 It is determined whether or not an input is received (step S1405). When the image is not terminated and the user 300 or the operator does not receive the playback end input (NO in step S1405), the flow returns to step S1401. If the image has been terminated, or if the input from the user 300 or the operator is received (YES in step S1405), the process is terminated.

Thus, the line-of-sight detecting apparatus 200 can constantly transmit the low-resolution image continuously to the head-mounted display 100, and can continuously provide the image, and at the same time, the high- Since the image is also transmitted, it is possible to provide the user with a video of good image quality. Since the visual line detecting apparatus 200 has a configuration in which a high resolution image is provided in the effective view field of the user 300 and a low resolution image is provided in the effective view field on the head mount display 100, Resolution image to be transmitted from the apparatus 200 to the head-mounted display 100 is minimized, it is possible to suppress the amount of data transmitted from the visual-line detecting device 200 to the head-mounted display 100. [

&Lt; Embodiment 3 >

In the second embodiment, the method of specifying the effective field of view of the user 300 in accordance with the degree of visibility of a plurality of objects based on the distance from the main viewpoint markers around the main viewpoint marker has been described. In the third embodiment, a description will be given of a method of specifying an effective visual field of the user 300 in a mode different from that of the second embodiment. In the third embodiment, only the difference from the second embodiment will be described.

Fig. 15 shows a state in which the effective visual field graph according to the third embodiment is displayed on the image display element 108 of the head mount display 100. Fig.

The video output unit 222 blinks each circle of the effective visual field graph shown in Fig. 15 at predetermined intervals. That is, the display is gradually erased from the displayed state, and the display is repeated from the erased state at predetermined intervals. When the user 300 acknowledges the state, all the circles are simultaneously displayed on the system of the head mount display 100, and even if they are simultaneously erased, they are simultaneously displayed and erased according to the individual differences of humans Can not be limited. In the third embodiment, the effective field of view is specified in accordance with the method of viewing different concentric circles in each of the users.

<Configuration>

The configuration of the line of sight detection system according to the third embodiment is the same as the configuration of the line of sight detection system according to the second embodiment.

The difference is that in the third embodiment, the effective visual field graph shown in FIG. 12 is displayed while the video output unit 222 displays the effective visual field graph shown in FIG. 12. The effective visual field graph shown in Fig. 15 is an image in which a plurality of concentric circles centered on the center of the main visual point marker are displayed at regular intervals. The concentric circles are equally spaced and have the same line width. The video output unit 222 displays the concentric circles to disappear at predetermined intervals. Then, the predetermined period is displayed while changing little by little.

The accepting unit 228 accepts information that enables the user to specify the information of the period when all of the plurality of concentric circles shown in Fig. 15 appear at the same time and all of them appear to have disappeared at the same time as the visibility information.

The specifying unit 229 specifies the effective field of view of the head mounted display 100 based on the period indicated by the visibility information transmitted from the accepting unit 228. [ The specification unit 229 specifies the valid view field (effective view field distance from the main viewpoint) of the user 300 based on the valid field of view calculation function that indicates the relationship between the period and the valid field of view stored in advance in the storage unit 227, . Here, the effective visual field calculating function is a function that the effective visual field of the user 300 becomes wider (longer effective visual field distance) as the period becomes shorter, and the effective visual field of the user 300 becomes narrower (effective visual field becomes shorter) . In other words, when the user has a narrow effective view, he feels that the change occurs simultaneously even if the period of switching between display and non-display is slow. That is, it can be inferred that such a user is generally insensitive to the change of the image. In the case where the user has a wide effective view, if the period between display and non-display is slow, the change is easy to notice. That is, such a user can generally assume that the change of the image is sensitive to the change of the image.

<Operation>

16 is a flowchart showing an operation for specifying the visual field of the user 300 by the visual-line detecting apparatus 200 according to the third embodiment.

As shown in Fig. 16, the video output unit 222 displays a plurality of concentric circles to disappear at predetermined intervals (step S1601). That is, in the effective visual field graph shown in Fig. 15, the circles are displayed at the same time by repeating the display from the display at a predetermined cycle to the disappearance and disappearance. The predetermined period is given an initial value, and the image output unit 222 gradually changes the predetermined period.

The user 300 inputs the timing at which all the concentric circles are simultaneously displayed and disappear at the same time in the process of repeating the display from the disappearance to the disappearance and the disappearance from the display of the concentric group while changing the predetermined period (step S1602) . The accepting unit 228 receives this timing and delivers the predetermined period in which the video output unit 222 at that time repeats the display / non-display of the concentric group to the specifying unit 229.

The specifying unit 229 specifies the effective field of view of the user 300 using the effective field of view stored in the storage unit 227 from the transmitted predetermined period (step S1603).

With this configuration, the visual line detecting apparatus 200 can specify the effective visual field of the user 300, and can have the same effect as the effect shown in the second embodiment.

&Lt; Fourth Embodiment >

In the fourth embodiment, a display method of a marker image different from that of the first embodiment and a visual line detection method at that time will be described.

In the first embodiment, nine marker images are displayed one after another and the user's eyes are watched. However, in the fourth embodiment, in the example of performing calibration using only one marker image .

<Configuration>

The basic configuration of the visual-line detecting system according to the present embodiment is the same as that shown in the first embodiment. For this reason, it has the same configuration as the block shown in Fig. Hereinafter, the changes from the first embodiment will be described.

The video output unit 222 in the fourth embodiment transmits the entire surrounding video image to the head mounted display 100 at the time of calibration. At this time, the entire surrounding image (or a somewhat wide range, that is, an image wider than the display range of the image display element 108) includes at least one marker image. That is, the first display unit 121 of the head mount display 100 displays a marker image at predetermined coordinates in the world coordinate system. World coordinate refers to a coordinate system that represents the entire space when an image is three-dimensionally displayed. The entire surrounding image is basically a 360 degree image displayed in the world coordinate system. Since the head mount display 100 includes an acceleration sensor, the user can specify which direction the user is facing, so that the video output unit 222 receives the information of the acceleration sensor from the head mount display 100 Determines which range of video is to be transmitted, and transmits the video data.

The user 300 displays the marker image so as to be included within the display range of the head mount display 100 by moving his or her head while the head mounted display 100 is mounted so that the marker image is displayed at least 2 Look from the other direction of the dog. The camera 116 of the head mount display 100 picks up the eye of the user at that time and acquires the image as a calibration image. That is, in the first embodiment, a marker image is displayed at nine positions so as to be in a different positional relationship between the user's eye and the marker image. In contrast to this, in the fourth embodiment, But a plurality of images for calibration can be acquired by viewing this from various angles.

Figs. 17A and 17B are diagrams schematically showing the correspondence between the entire surrounding image and the display screen displayed on the head-mounted display 100. Fig. 17A and 17B show a state in which the user 300 is mounting the head mounted display 100 and the entire surrounding image 1701 transmitted from the visual line detecting apparatus 200 A display range 1702 displayed on the image display element 108 of the head mount display 100 and a marker image 1703 in the entire surrounding image 1701. FIG. The entire surrounding image 1701, the display range 1702, and the marker image 1703 shown in Figs. 17A and 17B are virtual images and actually appear as shown in Figs. 17A and 17B Please be careful not to. The position of the marker image 1703 in the world coordinates is fixed. On the other hand, when the image is displayed on the image display element 108, the display position is changed according to the direction of the face of the user 300. It is needless to say that the marker image 1703 is a target, and the shape thereof is not limited to a circle.

17A shows a state in which the marker image 1703 is not included in the display range 1702 of the display image element of the head mounted display 100. Fig. And a marker image 1703 is included in the marker image 1703. In the state shown in Fig. 17 (b), the camera 116 of the head mount display 100 picks up the eyes of the user using the near-infrared light as the light source. The user 300 moves his / her head to move the display range 1702 so that the marker image 1703 appears at a position different from the position shown in Fig. 17B within the display range 1702 And watches the marker image at that time. The camera 116 of the head mount display 100 similarly picks up the user's eyes. In the fourth embodiment, a plurality of calibration images can be obtained in this way, and the user's main viewpoint can be specified using the respective expressions shown in the first embodiment.

As a result, the marker image output unit 223 according to the fourth embodiment has a function of determining the display position of the marker image in the world coordinate system.

<Operation>

The operation of the visual-line detecting system according to the fourth embodiment will be described with reference to the flowchart of Fig.

As shown in Fig. 18, the marker image output unit 223 determines the display coordinates of the marker image in the world coordinate system (step S1801).

The video output unit 222 of the visual line detection device 200 transmits a video to be displayed on the image display device 108 via the second communication unit 220. [ The marker image output section 223 transmits the marker image together with the display coordinates to the head mount display 100 in the same manner. The first display unit 121 of the head mount display 100 detects the direction of the head mounted display 100 relative to the world coordinate system from the value of the acceleration sensor mounted on the head mount display 100, As an image, it is determined whether or not a marker image is included in the range displayed on the image display element 108 (step S1802).

When the marker image is included in the display range (YES in step S1802), the first display unit 121 displays the marker image on the corresponding position on the image display element 108 (step S1803). If the marker image is not included in the display range (NO in step S1803), the process proceeds to step S1805.

The camera 116 picks up the eyes of the user 300 watching the marker image displayed on the image display element 108 as non-visible light as a light source (step S1804). The head mount display 100 transmits the sensed image to the visual line detecting apparatus 200 and the visual line detecting apparatus 200 stores the sensed image as a calibration image in the storage section 227. [

The visual-line detecting unit 221 of the visual-line detecting device 200 determines whether or not the number of captured images necessary for calibration reaches a predetermined number (for example, 9, but not limited to this) (step S1805). If the predetermined number of sheets has been reached (YES in step S1805), the process of calibration ends. On the other hand, if the number of sheets has not reached the predetermined number (NO in step S1805), the process returns to step S1802.

Even in this manner, calibration for sight line detection can be performed similarly to the first embodiment. The calibration in the fourth embodiment may be performed, for example, during the loading of the next video in the interval between the video and the video, or in the loading screen of the game in general. In this case, it is also possible to move the marker image so that the user faces the moving marker image. In this case, the marker image may be an image of a character appearing in a video to be viewed or a game to be executed.

<Supplement 1>

The visual-line detecting system according to the present invention may be configured as follows.

(a) a display screen for displaying an image to be presented to the user, the display screen including a video display device to be used by being mounted on a head of a user; A display unit for displaying on a screen an acceptance unit for accepting visibility information indicating the manner in which the object is viewed by the user while the user is watching the predetermined display position; The visual field detection system may include a specific section that specifies the effective field of view of the user.

(b) In the line-of-sight detection system described in (a) above, the display unit displays an object having a size in accordance with a distance from the predetermined display position, with the predetermined display position as a center, The information indicating the range in which the user can clearly view the object while the user is watching the predetermined display position.

(c) In the line-of-sight detection system described in (a), the display unit may display a plurality of circles centering on the predetermined display position at a predetermined distance interval and flickering at predetermined intervals, May be information capable of specifying the predetermined period that the user can recognize if the plurality of blinking circles are simultaneously displayed or disappear while the user is watching the predetermined display position.

(d) In the line-of-sight detection system described in any one of (a) to (c), the line-of-sight detection system may include a line-of- Wherein the display unit displays an image of a high resolution within a specific effective field of view with the center of the viewing position as a center and displays an image of a low resolution outside the effective field of view.

(e) In the visual line detecting system described in (d) above, the image display apparatus is a head mount display, and the visual line detecting system generates an image to be displayed on the display screen provided in the head mount display , Transmits to the head mount display a high resolution image to be displayed within the specific effective field of view, centering the view position, and generates and transmits a low resolution image that displays at least outside the effective field of view And may further include an image generating unit.

(f) In the line-of-sight detection system described in (e) above, the image generation unit may generate and transmit a low-resolution image of the entire display image regardless of the position of the effective field of view.

(g) The line-of-sight detection system includes a video display device mounted on a head of a user, and includes a display screen for displaying an image to be presented to the user, and a marker image arranged at a specific coordinate position on the world coordinate system A display unit for displaying the marker image when the specific coordinate position is included in the display coordinate system of the display screen; and a display unit for displaying, when the marker image is displayed on the display screen, And a visual line detecting section for detecting a viewing position of the user on the display screen based on at least two other sensed images captured by the image sensing section.

(h) Further, the effective visual field specifying method according to the present invention is a visual field detecting system including a video display device having a display screen for displaying an image to be presented to the user, A display step of displaying an object on the display screen so as to be annularly expanded around a predetermined display position of the display screen as a method of specifying the effective view of the user; A reception step of receiving a viewer's view indicating a manner of viewing the object in the user; and a specifying step of specifying an effective field of view of the user based on the viewer's information.

(i) The visual-line detecting method according to the present invention is a visual-line detecting method in a visual-line detecting system including a video display device which is mounted on a head of a user and has a display screen for displaying an image presented to the user A display step of displaying, on the display screen, a marker image disposed at a specific coordinate position on the world coordinate system on the display screen when the specific coordinate position is included in the display coordinate system of the display screen; An image pickup step of picking up an image of the eye of the user who is watching the marker image in a case where the marker image is displayed on the display screen based on at least two different picked- And a visual line detecting step of detecting a viewing position of the user.

(j) In addition, the effective visual field specific program according to the present invention is a program for realizing a visual perception specific to a computer included in a visual line detection system including a video display device having a display screen for displaying an image presented to a user, A display function of displaying an object on the display screen such that the object is annularly expanded around a predetermined display position of the display screen; and a display function of displaying the object in the user And a specific function for specifying an effective visual field of the user based on the visual information.

(k) Furthermore, the visual-point detecting program according to the present invention can be applied to a computer included in a visual-line detecting system including a video display device mounted on a head of a user and displaying an image presented to a user, A display function of displaying a marker image disposed at a specific coordinate position on the coordinate system on the display screen when the specific coordinate position is included in the display coordinate system of the display screen; On the basis of at least two other sensed images captured by the image sensing function, an image of the user's eye on the display screen, A visual line detecting function for detecting the visual line detecting function.

&Lt; Embodiment 5 >

In the above-described embodiment, various techniques relating to calibration have been described. In the present embodiment, a method for reducing fatigue of a user will be described. Therefore, first, this fatigue will be described.

In the head mount display, a three-dimensional image may be displayed. However, there is a problem that the user feels fatigue when viewing a three-dimensional image. When a three-dimensional image is displayed, a display is displayed to the user rather than an actual monitor position. Due to this, the eyeball of the user tries to focus on the display position (depth) of the display. In reality, however, since the position of the monitor is within the display position of the display, the eyeball realizes that there is an actual monitor and tries to re-focus the position. In view of the three-dimensional image, since the automatic focusing operation of the eyeball occurs alternately, the user feels fatigue.

Thus, in the fifth embodiment, a visual line detection system capable of reducing fatigue of a user when stereoscopic viewing is performed is disclosed.

<Configuration>

Fig. 19 is a block diagram of the head-mounted display 100 and the visual-line detecting device 200 relating to the visual-line detecting system 1. Fig. In the present embodiment, the visual line detection system may be referred to as a stereoscopic image display system. As shown in Fig. 19, as described above, the line of sight detection system 1 includes the head-mounted display 100 and the line of sight detection apparatus 200 that communicate with each other. Here, the configuration that is different from the above embodiment will be described.

19, the head mount display 100 includes a first communication unit 118, a display unit 121, an infrared light irradiation unit 122, an image processing unit 123, an image pickup unit 124, A driving unit 125, and a driving control unit 126. [

The first communication unit 118 transfers the three-dimensional image data to the drive control unit 126 in addition to the various functions described in the above embodiment. The three-dimensional image data includes information indicating the display depth of the displayed object. Here, the display depth is the distance from the user's eyes to the display position where the object is similarly displayed by stereoscopic viewing. The three-dimensional image data includes right eye parallax images and left eye parallax images, which are parallax image pairs.

The driving unit 125 has a function of driving a motor for moving the image display device 108 so that the relative distance between the image display device 108 and the user's eye varies in accordance with a control signal transmitted from the drive control unit 126. [

The drive control unit 126 generates a control signal for moving the image display element 108 in accordance with the display depth of the displayed object using the image data transmitted from the first communication unit 118, As shown in Fig. The drive control section 126 generates a control signal according to the following driving example as a method of generating a control signal.

&Lt; Driving Example 1 &

A control signal that causes the depth of the image display element 108 to approach the display depth is generated when the difference between the display depth of the displayed display object and the depth of the image display element 108 is a predetermined threshold value or more. Here, it is assumed that the threshold value is greater than or equal to a predetermined threshold value. However, a control signal may be generated such that the depth of the image display element 108 approaches the display depth of the object without such comparison.

&Lt; Driving Example 2 &

The first display depth of the display object displayed at the first time is compared with the second display depth of the display object displayed at the second time and when the second display depth is larger than the first display depth, The display object is displayed inside the display object viewed from the user 300 rather than the display object displayed at the first time.

More details of the operation of the driving unit will be described later.

20 is a diagram showing an example of a mechanism for moving the image display element 108, that is, the monitor. Fig. 20 (a) is a plan view showing a driving portion of the image display element 108 of the head mount display 100, and shows a mechanism inside the head mount display 100. Fig. Fig. 20 (b) is a perspective view of the drive unit viewed obliquely from below, in the direction indicated by the arrow 711 in Fig. 20 (a).

20 (a) and 20 (b), the end portion (right side in the figure) of the image display element 108 is connected to the support column 701, and the support column 701 is connected to the rail 702 at an end It is fixed to be able to act. At the end of the image display element 108, a comb is provided to engage with the teeth of the belt lane 703. [ On the surface of the belt lane 703, teeth are provided as shown in Fig. 20, and these teeth also move as the motor 704 rotates. Therefore, the image display element 108 also moves in the direction indicated by the arrow 710. [ When the motor 704 makes a right turn, the image display element 108 moves in a direction away from the eye of the user 300 and makes a left turn. In this case, the image display element 108 is moved in a direction approaching the eye of the user 300 Move. Here, the motor 704 is rotated by the driving unit 125 under the control of the drive control unit 126. [ With this structure as an example, the image display element 108 of the head mount display 100 can move so that the relative distance with the eyes of the user 300 varies. It is needless to say that the method of moving the image display element 108 is merely an example and may be realized by using other methods.

<Operation>

Hereinafter, a driving method for moving the image display element 108 in the head mount display 100 will be described.

&Lt; Driving Example 1 &

21 is a flowchart showing the operation of the head mount display 100 according to the embodiment.

The video output unit 222 of the visual-line detecting device 200 transmits the video data of the stereoscopic image displayed on the image display device 108 to the second communication unit 220. [ The second communication unit 220 transmits the delivered image data to the head mount display 100. [

The first communication unit 118, upon receiving the video data, transfers the video data to the drive control unit 126. The drive control section 126 extracts the display depth information of the display object from the delivered image data (step S2101).

The drive control unit 126 determines whether or not the distance between the display depth indicated by the extracted display depth information and the depth determined from the position of the image display element 108 is equal to or larger than a predetermined threshold value (step S2102). That is, the drive control section 126 determines whether or not the distance between the display object and the image display device 108 is more than a predetermined distance. If it is determined that the distance between the display depth and the image display element 108 is equal to or larger than the predetermined threshold value (YES in step S2102), the drive control section 126 proceeds to step S2103. If it is determined that the distance is less than the predetermined threshold value (NO in step S2102), the process proceeds to step S2104.

The drive control unit 126 specifies the display depth at which the display object is viewed by the user from the extracted display depth information. Then, a control signal for moving the monitor, that is, the image display element 108, in a direction approaching a specific display depth is generated and transmitted to the driving unit 125. The driving unit 125 drives the motor 704 to move the image display element 108 based on the transmitted control signal (step S2103). The driving unit 125 transmits to the display unit 121 that the image display element 108 has been moved.

The display unit 121 displays the corresponding image on the image display element 108 when the image display element 108 is moved from the driving unit 125 (step S2104).

By repeating the processing shown in Fig. 21, the image display element 108 can be moved in accordance with the display depth of the object displaying the image display element 108 each time. That is, the difference between the display depth of the object and the position of the image display element 108 can be reduced. Therefore, the occurrence of the focus adjustment can be suppressed by the eye movement of the user 300, and fatigue to be given to the user 300 can be suppressed.

&Lt; Driving Example 2 &

22 is a flowchart showing details of the operation of the head mounted display 100 according to the embodiment. Here, the description will be made from the stage where the drive control section 126 receives the video data which is motion picture.

The drive control unit 126 extracts display depth information (hereinafter referred to as first display depth information) of the display object to be displayed at the first time of the video data from the video data (step S2201).

Next, the drive control section 126 extracts display depth information (hereinafter referred to as second display depth information) of the display object to be displayed at the second time following the first time of the video data from the video data (step S2202). Here, the second time is not necessarily immediately after the first time (after one frame), but may be after a predetermined time (for example, 1 second).

The drive control section 126 determines whether or not the second display depth indicated by the second display depth information is larger (deeper) than the first display depth indicated by the first display depth information (step S2203). This is an agreement that it is determined by the user that the object displayed at the second time is displayed inside rather than when it is displayed at the first time.

If the second display depth is larger than the first display depth (YES in step S2203), the drive control unit 126 controls the drive unit (not shown) to move the image display device 108, that is, the monitor in the direction away from the user's eyes 125, respectively. The driving unit 125 moves the image display element 108 away from the user's eyes in accordance with the control signal (step S2204).

If the second display depth is smaller than the first display depth (NO in step S2203), the drive control unit 126 controls the drive unit 126 to move the image display device 108, i.e., the monitor, (125). The driving unit 125 moves the image display device 108 in a direction approaching the user's eyes in accordance with the control signal (step S2206).

The driving unit 125 transfers the image display element 108 to the display unit 121 when the image display element 108 is moved. Then, the display unit 121 displays the image to be displayed on the image display element 108 at the second time (step S2205).

The head mount display 100 displays the image data output from the video output unit 222 of the visual-line detecting apparatus 200 until all the image data is displayed (or when the reproduction of the image is stopped by the user) Repeat the process indicated.

Thus, the focus adjustment function of the user 300 is likely to occur when the display depth of an object in the case of a moving image or the like continuously displaying an image and the distance between the image display element 108 change, It is possible to suppress this occurrence frequency.

<Summary>

As described above, the visual-line detecting system 1 according to the present invention can move the image display element 108, that is, the monitor itself according to the display depth of the object in the displayed stereoscopic image. Concretely, the position of the image display element 108 and the display depth of the stereoscopic image can be approached. The greater the distance between the position of the image display element 108 and the display depth of the stereoscopic image is, the more easily the focus adjustment of the eye becomes easier. However, by providing the configuration according to the present embodiment, It is possible to suppress the occurrence of the focus adjustment movement of the lens. Accordingly, it is possible to slightly alleviate the occurrence of the focus adjustment of the eye function based on the difference between the virtual display position of the object and the actual position of the monitor, and thus eye fatigue of the user can be suppressed.

When the visual-line detecting system 1 according to the present invention is mounted on the head-mounted display 100, it is easy to move the monitor, and the visual line can be detected. Therefore, it is possible to realize both functions of presenting a stereoscopic image to the user without giving fatigue to the user as much as possible, and of detecting a line of sight in which the user can see a point in the stereoscopic image.

In addition, in the fifth embodiment, the structure for moving the image display element 108 is not limited to the structure shown in Fig. Other structures may be adopted as long as the structure allows the image display element 108 to move in the direction indicated by the arrow 710 in Fig. 20 (a). For example, the same structure may be realized by a worm gear or the like. In the above-described embodiment, the structure shown in Fig. 20 is a structure in which the left and right (left and right in the state where the user is mounted and left and right in the longitudinal direction of the image display element 108) of the head mount display 100 However, if the image display element 108 can be moved right and left without a sense of discomfort, it may be provided on only one side.

In the fifth embodiment, one image display element 108 is used, but the present invention is not limited to this. The head mount display 100 is provided with an image display element which is opposed to the left eye of the user 300 and an image display element which is opposed to the right eye of the user 300 and two image display elements, And may be driven. Thus, fine control such as focus adjustment according to the visual acuity of the left and right eyes of the user 300 can be performed.

The image of the user 300 is captured by the hot mirror 112 as a method of picking up the eyes of the user 300. This is because the image of the user 300 is picked up by the hot mirror 112, It is possible to directly capture the eyes of the user 300 without passing through.

<Supplement 2>

The visual-point detecting system according to the fifth embodiment may be described as a stereoscopic image display system as follows.

(1) The stereoscopic image display system according to the fifth embodiment includes: a monitor for displaying a stereoscopic image to be presented to a user; a driving unit for moving the monitor so that a relative distance between the user's eyes and the user is changed; And a control unit for controlling the driving unit according to the depth of the stereoscopic image.

The control method according to Embodiment 5 of the present invention is a control method of a stereoscopic image display system for reducing the fatigue of a user in stereoscopic vision and includes a display step of displaying a stereoscopic image presented to a user on a monitor, And controlling the driving unit to move the monitor so that the relative distance with the user's eye changes according to the depth of the stereoscopic image displayed on the display unit.

The control program according to the fifth embodiment is characterized in that the computer of the stereoscopic image display system is provided with a display function for displaying a stereoscopic image presented to a user on a monitor and a display function for displaying the stereoscopic image on the monitor And controls the driving unit to move the monitor so that the relative distance of the monitor is changed.

(m) In the stereoscopic image display system according to (1), the control unit may control the driving unit in a direction in which the monitor approaches the depth at which the stereoscopic image is displayed.

(n) In the stereoscopic image display system according to the above (1) or (m), the control unit may be configured such that the depth of the stereoscopic image displayed at the second time following the first time is shorter than the depth of the stereoscopic image displayed at the first time The controller may control the driver to move the monitor closer to the user's eyes.

(o) In the stereoscopic image display system according to any one of (l) to (n), the control unit may cause the depth of the stereoscopic image displayed at the second time following the first time to be And controls the driver to move the monitor away from the eye of the user when the depth of the displayed three-dimensional image is deeper than the depth of the displayed stereoscopic image.

(p) In the stereoscopic image display system according to any one of (l) to (o), the stereoscopic image display system is mounted on a head mount display mounted on a head of a user, Visible light irradiating unit for irradiating an eye of a user with non-visible light, an imaging unit for imaging an eye of a user including non-visible light irradiated by the non-visible light irradiating unit, an image pickup unit for picking up an image picked up by the image pickup unit To an eye-gaze detecting apparatus that performs a line-of-sight detection.

<Supplement 3>

The eye-gaze detecting system according to the present invention is not limited to the above-described embodiment, and needless to say, it may be realized by another method for realizing the idea of the invention.

In the above embodiment, the position at which the marker image (bright spot) is displayed is an example, and an image of the user's eye, which is displayed at another position to detect the user's gaze, is acquired, Needless to say, it is not limited to the display position shown in the above embodiment as long as the center of the eye can be specified. In this case, the number of marker images is not limited to nine, and four expressions can be satisfied in order to specify the four elements of the matrix x. Therefore, the user's corneal center for at least four marker images It is enough if it can specify.

In the above embodiment, the image captured by the hot mirror 112 is picked up as a method of picking up the eyes of the user 300 in order to detect the line of sight of the user 300. This is because the hot mirror 112 It is possible to directly capture the eyes of the user 300 without passing through.

The marker image output section 223 changes the display position of the marker image in accordance with the input instruction from the operator of the visual line detecting apparatus 200. The marker image output section 223, The display position of the marker image may be automatically changed. For example, the marker image output unit 223 may change the display position of the marker image every predetermined time (for example, three seconds).

More preferably, the line of sight detection system 1 analyzes the captured image obtained from the head mounted display 100 to determine whether or not the user is watching the marker image. If the user determines that the user is watching the marker image , And change the display position of the marker image.

That is, in the storage section 227, an image (image obtained by imaging the center marker image of the nine marker images in a state where the user is watching the center of the image display element 108 in advance) Remember. The determination section 224 compares the stored center position of the cornea (black character) of the image with the captured image to determine whether the corneal center of the user of the image stored in the corneal center of the user of the captured image, It is determined whether or not the user is watching the marker image according to whether or not the marker image is separated by a predetermined distance (for example, 30 pixels in the pixel coordinate system of the image display element 108) in the direction in which the marker image is displayed. The determining section 224 instructs the marker image output section 223 to change the display position of the marker image when it is determined that the user is watching the marker image in the determination, (223) changes the display position of the marker image in accordance with the instruction.

When the user determines that the marker image is not being watched by the user, the marker image output section 223 displays the marker image emphatically (for example, blinking at the display position A marker image may be indicated by an icon of an arrow or the like or a sentence of "look at a marker" may be displayed). This notification may be realized by voice guidance from the headphone 170 of the head mount display 100 to perform an announcement saying "Look for a marker ". Thereby, the storage unit 227 stores the voice data, and when the marker image output unit 223 determines that the user is not watching the marker image, it transmits the voice data to the head mount display 100 , The head mount display 100 outputs the received voice data from the headphone 170. [ Further, further, the head mount display 100 shows that the judgment was successful for the user when the judging unit 224 judges that the capturing image for calibration has been successfully acquired (indicating that the calibration is successful For example, an image of "?" Or "OK" may be displayed in the marker image.

Thus, by configuring the line-of-sight detecting system 1 so as to determine whether or not the user wearing the head mount display 100 is watching the marker image, the automation of the calibration can be realized. Therefore, in the conventional calibration, the operator is required separately from the user who mounts the head mount display 100, but the calibration can be performed without the operator.

The head mount display 100 may have a configuration in which the distance between the image display element 108 and the eye of the user can be changed (moved) when the 3D image is displayed. If the virtual distance (depth) from the user's eyes to the 3D image displayed is different from the distance between the eyes of the actual user and the image display element 108, it becomes a factor of fatigue of the eyes of the user. By the configuration, the head mount display 100 can reduce the fatigue of the eyes of the user.

In the visual line detection system 1, the effective visual field of the user may be specified at the time of calibration. The user's effective visual field is a range in which the user can clearly recognize an image from the state where the user is looking at a certain point toward the end direction. In the line-of-sight detection system 1, a marker image may be displayed circularly from the center of the screen at the time of calibration to specify the effective visual field. It is also possible to specify the effective visual field of the user by specifying a period in which a plurality of concentric circles centering on one point are highlighted while the user is viewing a certain point, and at the same time, a timing which appears to disappear at the same time. If the effective visual field can be specified for each user, it is difficult for the user to recognize even if the image quality is deteriorated with respect to the out-of-view image, so that the data transmission amount of the image transmitted from the visual-line detecting device 200 to the head- can do.

In the above embodiment, the processor of the visual-line detecting apparatus 200 executes a visual-line detecting program or the like as a method of calibration in visual-line detection to specify a point that the user is watching. However, (Hardware) formed on an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration) or the like) or a dedicated circuit in the visual line detecting apparatus 200. [ Such a circuit may be realized by one or a plurality of integrated circuits, and the functions of the plurality of functional sections described in the above embodiments may be realized by one integrated circuit. LSIs are sometimes called VLSIs, Super LSIs, Ultra LSIs, etc., depending on the degree of integration. 10, the head mount display 100 includes a first communication circuit 118a, a first display circuit 121a, an infrared light irradiation circuit 122a, an image processing circuit 123a, And an image pickup circuit 124a, and the respective functions are the same as those of the respective parts having the same names as those in the above embodiment. The visual line detecting apparatus 200 further includes a second communication circuit 220a, a visual line detecting circuit 221a, a video output circuit 222a, a marker image output circuit 223a, a judging circuit 224a, The composite image output circuit 225a, the second display circuit 226a, and the storage circuit 227a, and the respective functions are the same as those of the respective parts having the same names as in the above embodiment. 10 shows an example in which the visual line detection system according to the first embodiment is implemented by a circuit. Although not shown, the visual line detection system shown in Fig. 11 or Fig. 19 may be realized by a similar circuit Needless to say.

The visual-axis detecting program may be recorded on a recording medium readable by a processor. The recording medium may be a medium such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit . The line-of-sight detecting program may be supplied to the processor through an arbitrary transmission medium (such as a communication network or a broadcast wave) capable of transmitting the line-of-sight detecting program. The present invention can also be realized in the form of a data signal included in a carrier wave, in which the line-of-sight detecting program is embodied by electronic transmission.

The line-of-sight detecting program may be executed by using, for example, a script language such as ActionScript or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), a markup language such as HTML5 Can be mounted.

A visual line detecting method according to the present invention is a method for visual line detection by a visual line detecting system including a head mount display attached to a user and a visual line detecting device for detecting the visual line of the user, Wherein the apparatus displays a marker image on the head mount display, the head mount display displays the marker image, captures an image of the user watching the marker image, And the visual line detection device generates a composite image in which the marker image and the image including the eye of the user who is looking at the captured marker image are superimposed on each other and outputs the composite image thus created Or the like.

The line of sight detecting program according to the present invention is characterized in that the computer is provided with a marker image output function for outputting a marker image to be displayed on the head mounted display and a captured image image A generating function for creating a composite image in which the marker image and the captured image are superimposed, and a program for realizing a composite image output function for outputting the composite image.

The invention is applicable to head-mounted displays.

1 line of sight detection system
100 Head Mount Display
103a infrared light source (second infrared light irradiation unit)
103b An infrared light source (first infrared light irradiation unit)
105 spots
108 image display element
112 hot mirror
114, 114a, 114b convex lens
116 Camera
118 first communication section
121 first display portion
122 infrared light irradiation unit
123 image processing section
124 image pickup section
130 image display system
150 cases
152a, 152b,
160 mounting hole
170 Headphones
200 line of sight detection device
220 second communication section
221 line-of-sight detecting section
222 Video output unit
223 marker image output unit
224 judgment section
225 composite image output section
226 Second display section
227 Memory unit

Claims (6)

1. A line-of-sight detection system including a head-mounted display mounted on a user and a line-of-sight detecting device for detecting a line of sight of the user,
Wherein the head mount display comprises:
A display section for displaying an image;
An image pickup section for picking up an image of the user's eyes;
And an image output unit for outputting, to the visual-line detecting apparatus, an image including the eye of the user, the image of which has been picked up by the image pickup unit,
The visual-
A marker image output unit for outputting a marker image to be displayed on the display unit;
A composite image creating unit that creates a composite image in which an image including a user's eye watching the marker image captured by the image capture unit is superimposed on a marker image output by the marker image output unit,
And a synthesized image output unit outputting the synthesized image. The method according to claim 1,
Wherein the marker image output unit changes the display positions of the marker images in order and outputs them,
Wherein the image capturing unit captures an image of a user who watches the marker image at least every time the display position is changed. The method of claim 2,
Wherein the marker image output unit changes the display position of the marker image to any one of a plurality of predetermined coordinate positions and outputs the marker image,
The visual-
Further comprising a line of sight detecting unit for detecting a line of sight of the user based on each of the images of the user's eye captured by the image pickup unit and the image including the user's eye watching the marker image for each of the displayed positions Eye detection system. The method of claim 3,
The line-of-
Further comprising a determination unit that determines whether or not an image including eyes of a user who is watching the marker image is usable as an image for sight line detection by the sight line detection unit,
When the determination unit determines that the image can not be used as the image for line-of-sight detection, the marker image output unit displays the display position of the marker image displayed when the image corresponding to the determination is captured, And,
Wherein the imaging unit captures an image of a user who watches a marker image in which the display position is changed,
Wherein the judging section judges whether or not the comparison image picked up again is usable as the image for line-of-sight detection. The method of claim 4,
The judging unit judges,
The image pickup unit further determines whether or not the user is watching the displayed marker image based on the image of the user's eye captured,
The line-of-
Further comprising a notifying unit for informing the user to watch the marker image when it is determined that the user is not watching the marker image. The method of claim 5,
Wherein the marker image output unit changes the display position of the marker image when the determination unit determines that the user is watching the displayed marker image.

Images