JP2010199788A - Head-mounted display - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a head-mounted display with superior ease of operation that is suitably calibrated with respect to the direction of the line of sight of a user. <P>SOLUTION: An image presenting device, which projects an image represented by content data on the side of a first position as a predetermined position on the side of a temporal part from almost the front of one eye of the user (S110, S112), is moved to the side of a second position as a predetermined position of almost the front of the one eye of the user (S106), and reference state data for controlling a predetermined operation in the head-mounted display are stored in a flash ROM (S108). <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a head mounted display that presents, for example, an image indicated by content data to a user's eyes so that the user can recognize the image, and allows the user to recognize the image.

  A technology related to a head-mounted display has been proposed in which an image indicated by content data or the like is presented to a user's eye so as to be visible, and the user recognizes the image. For example, in order for the observer to obtain a good image, it is necessary to receive light that can obtain a good image on the entire left and right pupils of the observer, so that the exit pupil or good light before and after the pupil intersects. Where the observer's pupil must be placed in the area, it is difficult to know in which direction the device should be adjusted when the pupil is misaligned, and the following configuration is proposed as a means to solve this problem. Has been. That is, an image blocking means for blocking an image display on the image display element or an image from the image display element, and a concave reflector disposed in an optical path for guiding the display image on the image display element to an observer's eyeball, And an illumination light source for determining pupil misalignment, and when adjusting the observer's pupil position, the image blocking means and the illumination light source are operated, and the pupil image of the eyeball is formed far away by the concave reflector Let According to this configuration, when the observer's pupil does not coincide with the exit pupil of the display device and there is a pupil shift, the distant image cannot be viewed in the visual axis direction by the observer, so the pupil shift does not occur. It is said that it is easily understood (see, for example, Patent Document 1).

  As another technique, a technique has been proposed that uses a movement pattern of an eyeball so that a wearer can switch between an electronic image and an external image without using a hand. That is, on the condition that the convergence angle formed by the left and right eyeballs is calculated, the state of the relationship of the convergence angle with respect to the predetermined reference convergence angle changes, and the duration of the same state with respect to the predetermined reference time is long A configuration is proposed in which the convergence pattern extraction processing device outputs a change signal instructing switching between an electronic image and an external image to the image switching control device (see, for example, Patent Document 2).

JP-A-7-115607 JP-A-8-160345

  By the way, the structure which controls the operation | movement of a head mounted display according to the gaze direction of the user with whom the head mounted display is mounted | worn can improve the operativity of a head mounted display. However, such control is realized by appropriately detecting the user's line-of-sight direction. Therefore, in a head mounted display capable of realizing such control, the user's line-of-sight direction corresponding to the operation must be appropriately adjusted (calibrated). Since there are individual differences in how to move the eyes, it is necessary to perform appropriate calibration for each user.

  An object of the present invention is to provide a head mounted display excellent in operability capable of appropriately performing calibration relating to a user's line-of-sight direction.

  The head mounted display of the present invention made in view of the above-described problem enables an image presentation device that emits image light indicating an image to be arranged at the first position and the second position, and the image presentation device is arranged at the first position. Thus, when the line of sight of the user is the first direction, the first operation of the head mounted display is controlled, and the reference state information is stored when the arrangement of the image presentation device is at the second position. Is. Here, the reference state information is information corresponding to the second state information indicating the state of one eye in which the user's line-of-sight direction is in the first direction in a state where the image presentation device is disposed at the second position. . The head mounted display of the present invention acquires first state information indicating the state of one eye of the user in a state where the image presentation device is disposed at the first position, and the acquired one state information and the reference state By comparing with the information, it is detected that the line-of-sight direction of the user is the first direction in a state where the image presentation device is arranged at the first position.

  A first problem-solving means reflecting the present invention is a head-mounted display that is mounted on a user's head and allows the user to visually recognize an image, and the image light that indicates the image to the user's eyes. An image presentation device that emits light, a first position that is a predetermined position on the temporal side from approximately the front of one eye of the user, and a second position that is a predetermined position approximately in front of the one eye And an arrangement means for arranging the image presentation device, and in a state where the image presentation device is arranged at the first position, first state information indicating the state of the one eye is acquired, and the image presentation device State information acquisition means for acquiring second state information indicating the state of the one eye in which the user's line-of-sight direction is in the first direction in a state in which the user is positioned at the second position; and the second state information Storage means for storing reference state information corresponding to the state, and the state The first state information acquired by the information acquisition unit is compared with the reference state information stored in the storage unit, and the one of the users in the state where the image presentation device is arranged at the first position Detection means for detecting that the line-of-sight direction of the eye is the first direction, and operation control means for controlling the first operation of the head mounted display when the detection means detects the first direction. A command acquisition unit that acquires an execution command that instructs the storage unit to store the reference state information; and when the command acquisition unit acquires the execution command, the arrangement of the image presentation device in the arrangement unit is: The state information acquisition means acquires the moving means for moving from the first position to the second position, and the image presentation apparatus is located at the second position by the moving means. The reference state information corresponding to the second state information, a head-mounted display, characterized by comprising: a storage control means for storing in the storage means.

  According to this, with respect to the first movement controlled based on the user's line-of-sight direction, the image presentation device displays the second state information that is the basis of the reference state information serving as the reference approximately in front of one eye of the user. It can be acquired in the arranged state (the state where the image presentation device is arranged at the second position).

  The second problem-solving means is the head-mounted display of the first problem-solving means, and the moving means, when the storage control means stores the reference state information in the storage means, The arrangement of the image presentation device is moved from the second position to the first position.

  According to this, when the reference state information is stored, the arrangement of the image presentation device can be returned to the first position.

  The third problem solving means is a head-mounted display of the first or second problem solving means, wherein the reference state information is the second state information, and the image presentation device is disposed at the first position. The state information is corrected so as to indicate the state of the one eye in which the user's line-of-sight direction is in the first direction.

  According to this, it is possible to appropriately detect that the line-of-sight direction of one eye of the user is the first direction in a state where the image presentation device is arranged at the first position.

  The fourth problem solving means is a head-mounted display according to any one of the first to third problem solving means, wherein the image presentation device includes image presentation means for emitting the image light, and the image presentation means. Reflecting means for reflecting image light emitted from the user in the direction of the eyes of the user, wherein the arranging means arranges the reflecting means at the first position and the second position, and the moving means includes The reflecting means is moved from the first position to the second position.

  According to this, when acquiring 2nd status information, the structure moved to a 2nd position can be decreased.

  ADVANTAGE OF THE INVENTION According to this invention, the head mounted display excellent in the operativity which can perform the calibration regarding a user's gaze direction appropriately can be obtained.

The figure which shows the external appearance of the head mounted display in embodiment of this invention The figure which shows the state which mounted | wore the head mounted display of embodiment which concerns on this invention. The figure which shows the functional block of the image presentation apparatus of embodiment which concerns on this invention. The figure which shows the structure of the image presentation mounting part of embodiment which concerns on this invention. The figure which shows the moving mechanism of the image presentation apparatus and half mirror of embodiment which concerns on this invention The figure which shows the state by which the half mirror attached to the image presentation apparatus of embodiment which concerns on this invention was arrange | positioned in the 2nd position. (A) And (b) is a figure explaining the movement of the image presentation apparatus main body (CCD camera) and half mirror of embodiment which concerns on this invention. The figure which shows the processing flow of the main process of embodiment which concerns on this invention. The figure which shows the processing flow of the calibration of embodiment which concerns on this invention The figure which shows the image shown to a user in the calibration of embodiment which concerns on this invention The figure explaining the correction process of embodiment which concerns on this invention The figure which shows the processing flow of the content presentation of embodiment which concerns on this invention.

  Embodiments for implementing the above problem solving means reflecting the present invention will be described below in detail with reference to the drawings. In addition, the said subject solution means is not limited to the structure described below, Various structures can be employ | adopted in the same technical idea. For example, in the following description, an integrated head-mounted display (Head Mounted Display / hereinafter referred to as “HMD”) will be described as an example. However, the head-mounted display main body and an image indicated by content data on the head-mounted display main body are described. It is also possible to connect the control box that provides

(Overview of head mounted display)
1 shows the configuration of the HMD. More specifically, FIG. 1A shows a top view of the HMD, FIG. 1B shows a front view of the HMD, and FIG. 1C shows a left side view of the HMD. FIG. 2 shows a state in which the user wears the HMD. 1 and 2, a moving mechanism for moving the image presentation device (half mirror) 3347 is omitted (see FIGS. 5 to 7).

  The HMD 100 includes temples 104A and 104B each having a modern 102A and 102B that hits the user's ear at one end, armatures 106A and 106B coupled to the other ends of the temples 104A and 104B via hinges 112A and 112B, and armor 106A. , 106B and a nose pad 110 attached to the center of the front frame 108 and abutting the user's nose form a skeleton. Temples 104A and 104B can be folded with hinges 112A and 112B formed on armor 106A and 106B. The structure of the skeleton part of the HMD 100 is the same as, for example, normal glasses, and the HMD 100 is supported on the user's face by the modern 102A and 102B and the nose pad 110 when worn on the user. (See FIG. 2). In FIG. 1B, drawing of the moderns 102A and 102B and the temples 104A and 104B is omitted.

  An image presentation device 114 is movably attached to the HMD 100 from the position shown in FIG. 1 to the front side of the user's left eye 118 (details will be described later). When the image presentation device 114 is attached to the HMD 100, the image presentation device 114 is disposed at a position that is substantially the same height as the left eye 118 of the user wearing the HMD 100. Further, the image presentation device 114 includes various configurations therein as described later. The content data is processed (rendered) by a configuration included in the image presentation device 114, and a predetermined image is generated. Then, image light indicating the generated image is optically emitted from the image presentation unit 310 (see FIG. 3) built in the image presentation device 114 toward the half mirror 116. The half mirror 116 is attached to a predetermined position of the image presentation device 114 that incorporates the image presentation unit 310 and the like, and constitutes the image presentation device 114.

  The image light emitted from the image presentation unit 310 is reflected by the half mirror 116 and is incident on the user's left eye 118, in other words, presented (projected) so as to be visible. Thereby, the user recognizes a predetermined image. Here, in FIG. 1A, reference numeral 120a indicates image light relating to an image emitted from the image presentation unit 310, and reference numeral 120b indicates image light reflected by the half mirror 116 and incident on the left eye 118 of the user. Indicates. The image presentation unit 310 scans the image light 120a and 120b in accordance with the acquired image signal in a two-dimensional direction, guides the scanned image light 120a and 120b to the user's left eye 118, and displays the image on the retina. In addition to a retinal scanning type display that forms a liquid crystal display, a liquid crystal display, an organic EL (Organic Electroluminescence) display, and other devices may be used.

  Here, a CCD camera 300 capable of capturing an image of the user's left eye 118 is attached to the upper surface of the image presentation device 114.

(Configuration of image presentation device)
FIG. 3 shows functional blocks of the image presentation device 114. The image presentation device 114 includes a CPU 202 that controls the device itself and a program ROM 204 that stores various programs. Further, the image presentation device 114 includes a nonvolatile flash ROM 206 that stores content data, reference state data, and other various data, and a RAM 208 as a work area. Here, the reference state data is data (information) indicating the reference state corresponding to the state of the left eye 118 (see FIG. 1) in which the user's line-of-sight direction is the left direction and the right direction (details will be described later). It is.

  In addition, the image presentation device 114 includes a CCD camera connection I / F 210 to which the CCD camera 300 is connected. The CCD camera connection I / F 210 includes a CCD camera connection controller 2102 and a CCD camera VRAM (Video RAM) 2104. Here, the CCD camera connection controller 2102 controls the input (reception) of the captured image data from the CCD camera 300 based on a command from the CPU 202. A CCD camera VRAM (Video RAM) 2104 stores an image captured by the CCD camera 300.

  In addition, the image presentation device 114 includes an image presentation unit connection I / F 212 to which the image presentation unit 310 is connected. The image presentation unit connection I / F 212 includes an image presentation unit connection controller 2122 and an image presentation unit VRAM 2124. Here, the image presentation unit connection controller 2122 controls input / output (transmission / reception) of signals including various data with the image presentation unit 310 based on a command from the CPU 202. The image presentation unit VRAM 2124 stores an image output to the image presentation unit 310. The image presenting unit 310 emits image light 120 a that is stored in the image presenting unit VRAM 2124 and that indicates a predetermined image generated by rendering input by the image presenting unit connection controller 2122. The emitted image light 120 a is reflected by the half mirror 116.

  In addition, the image presentation device 114 includes a peripheral I / F 214 and a communication unit 216. Here, the peripheral I / F 214 is connected to, for example, a lamp 320 that illuminates an imaging position by the CCD camera 300 and a power switch 330 of the HMD 100. The communication unit 216 receives an input of a predetermined instruction from, for example, a keyboard (not drawn in FIG. 3).

  Furthermore, the image presentation device 114 includes an image presentation device moving unit controller 218 that controls the movement of the image presentation device 114 based on a command from the CPU 202. The image presentation device moving unit controller 218 controls a motor 154 (see FIG. 5B) that is a drive source for moving the image presentation device 114. Note that, for example, when adopting a configuration in which the movement of the image presentation device 114 is performed by a user's manual operation (manual), the function of controlling the motor 154 may not be included. The image presentation device moving unit controller 218 is a position for detecting that the image presentation device 114 (half mirror 116) has been moved to the predetermined first position and has been moved to the predetermined second position. A sensor 340 is connected. The image presentation device moving unit controller 218 controls the input (reception) of the signal from the position sensor 340 based on a command from the CPU 202. The position information of the image presentation device 114 input from the position sensor 340 is transmitted to the CPU 202.

  The first position is a predetermined position on the left head side from approximately the front of the left eye 118 of the user. In other words, the first position emits the image light 120a of the image indicated by the content data, and when the image light 120b is incident on the left eye 118 of the user via reflection by the half mirror 116, the image presentation device This is the position where the half mirror 116 attached to 114 is disposed. In other words, the HMD 100 exhibits the essential functions of the HMD 100 such that the image (image light 120a, 120b) indicated by the content data is presented to the user's left eye 118 so that the user can recognize the image. In this case, the half mirror 116 attached to the image presentation device 114 is disposed at the first position.

  On the other hand, the second position is a predetermined position substantially in front of the user's left eye 118. In other words, the second position stores, for example, reference state data (details will be described later) serving as a reference for controlling the operation (for example, reproduction start and reproduction stop) of the HMD 100 (calibration is executed). The half mirror 116 is disposed.

  By presenting the image at the first position so as to be visible and executing the calibration at the second position, there are the following advantages. At the first position, the image light is emitted from a direction deviated by a predetermined position on the left head side from substantially the front of the left eye 118. Thereby, when the user is normally looking forward, the image light does not enter the left eye 118. That is, since the user cannot visually recognize the image light, the user can visually recognize only the front outside world without being disturbed by the image light. When the user looks at a predetermined position on the left head side (leftward by θ in FIG. 5 described later), the image light is incident on the left eye 118 and the image light can be viewed. As a result, the image light can be visually recognized by looking at a predetermined position on the left side of the head only when it is necessary to see the image light, and when it is not necessary, the front can be seen and the outside world can be seen without being disturbed by the image light. Predetermined work can be performed. In particular, when an image is formed on the retina by scanning with laser light, the laser light has a small beam diameter. Therefore, the user simply moves his / her line of sight from a predetermined position on the left side of the head to move the laser light. The left eye 118 cannot be entered. However, even if the calibration is performed at the first position, it is necessary to always turn the line of sight toward a predetermined position on the left side of the head in order to visually recognize the calibration guidance. Since the user's line of sight is fixed at a predetermined position on the left side of the head, the user moves the line of sight up and down, left and right, even if guidance for moving the line of sight up and down and left and right is displayed. It becomes difficult to obtain data. That is, calibration cannot be performed satisfactorily. On the other hand, since the second position is a predetermined position substantially in front of the left eye 118 of the user, the calibration guidance is displayed, and the user acquires the reference state data by moving the line of sight up, down, left, and right. can do. However, if the work is performed in the second state, the work environment and the display of the work instruction manual and manual for the predetermined work always appear to overlap each other, and the image light interferes with the work. Therefore, in the present embodiment, it is possible to satisfactorily perform the calibration by executing the calibration at the second position without causing an obstacle to the work by presenting the image at the first position so as to be visible. Yes.

  The CPU 202 acquires a predetermined image by executing a program for reproducing (rendering) the content data stored in the program ROM 204 on the RAM 208. Then, a program for controlling the image presentation unit connection I / F 212 stored in the program ROM 204 is executed on the RAM 208, and an image signal including the image is output to the image presentation device 100 as an image presentation unit connection controller. 2122 is commanded.

  Further, the CPU 202 is indicated by the image of the left eye 118 of the user and the reference state data which are input from the CCD camera 300 via the CCD camera connection I / F 210 and stored in the CCD camera VRAM 2104 by the CCD camera connection controller 2102. Using the image, an analysis program (for example, a program for pattern matching) stored in the program ROM 204 is executed on the RAM 208 to analyze the state of the left eye 118 of the user. In addition, the CPU 202 executes a program for controlling the HMD 100 stored in the program ROM 204 on the RAM 208, and controls the operation of the HMD 100 based on the analysis result.

  Further, the CPU 202 acquires an instruction received by the communication unit 216 or an instruction via an operation unit included in the HMD 100 not illustrated in FIG. 3, and a calibration program stored in the program ROM 204 based on the instruction. By executing this, the process described later (calibration shown in FIG. 9) is executed.

  Therefore, the CPU 202 executes various programs stored in the program ROM 204 on the RAM 208 by using various data such as content data, an image of the left / left eye 118 imaged by the CCD camera 300 and an image indicated by the reference state data. Thus, various function means (for example, detection means, operation control means, command acquisition means, storage control means) are configured.

(Configuration of image presentation unit)
FIG. 4 shows the configuration of the image presentation device. The image presentation unit 310 includes a scanned image light generation unit 3121, a collimating optical system 3122, a horizontal scanning unit 3123, a vertical scanning unit 3124, a relay optical system 3125, and a relay optical system 3126.

  The scanned image light generation unit 3121 is a device that reads the image signal output from the image presentation unit connection I / F 212 for each dot clock and modulates the intensity according to the read image signal to generate the scanned image light. The scanned image light generation unit 3121 includes a signal processing circuit 3211, a light source unit 3212, and a light combining unit 3213.

  The signal processing circuit 3211 is connected to the image presentation unit connection I / F 212. The signal processing circuit 3211 is based on the “image signal” input from the image presentation unit connection I / F 212 and serves as an element for generating scanned image light B (blue), G (green), and R (red). The image signals 3214a to 3214c are generated and output to the light source unit 3212. The signal processing circuit 3211 is connected to a horizontal scanning control circuit 3123b of the horizontal scanning unit 3123 described later. The signal processing circuit 3211 generates a horizontal drive signal 3215 based on the “image signal” input from the image presentation unit connection I / F 212 and outputs the horizontal drive signal 3215 to the horizontal scanning control circuit 3123b. Further, the signal processing circuit 3211 is connected to a vertical scanning control circuit 3124b described later. The signal processing circuit 3211 generates a vertical drive signal 3216 based on the “image signal” input from the image presentation unit connection I / F 212 and outputs the vertical drive signal 3216 to the vertical scanning control circuit 3124b.

  The light source unit 3212 includes a B laser driver 3212a, a G laser driver 3212b, an R laser driver 3212c, a B laser 3212d, a G laser 3212e, and an R laser 3212f. The B laser driver 3212a drives the B laser 3212d based on the B (blue) image signal 3214a output from the signal processing circuit 3211 for each dot clock. The B laser 3212d emits intensity-modulated blue laser light based on the B (blue) image signal 3214a. Similarly, the G laser 3212e and the R laser 3212f emit green laser light and red laser light, which are respectively intensity-modulated.

  Each of the lasers 3212d to 3212f includes a semiconductor laser and a solid-state laser with a harmonic generation function. When a semiconductor laser is used, the drive current is directly modulated to modulate the intensity of the laser beam. When using a solid state laser with a harmonic generation function, each of the lasers 3212d to 3212f is provided with an external modulator to modulate the intensity of the laser light. Note that the efficiency of harmonic generation is not high, and the loss of the external modulator is also added, so that the power consumption of the solid-state laser with the harmonic generation function is increased. Therefore, it is preferable to use a semiconductor laser for each of the lasers 3212d to 3212f. .

  The light combining unit 3213 includes collimating optical systems 3213a to 3213c, dichroic mirrors 3213d to 3213f, and a coupling optical system 3213g. The collimating optical systems 3213a to 3213c are arranged in front of the lasers 3212d to 3212f, respectively, and collimate the laser beams emitted from the lasers 3212d to 3212f. The dichroic mirrors 3213d to 3213f are arranged in front of the collimating optical systems 3213a to 3213c, respectively, and each laser beam paralleled by the collimating optical systems 3213a to 3213c is used only for laser beams having a wavelength in a predetermined range. Optionally, reflect or transmit.

  The coupling optical system 3213g is disposed in front of the dichroic mirror 3213d. Blue laser light transmitted through the dichroic mirror 3213d and green laser light and red laser light reflected by the dichroic mirrors 3213e and 3213f are incident on the coupling optical system 3213g. The coupling optical system 3213 g condenses (mixes) the laser beams of the three primary colors and enters the optical fiber 3127. Note that white can be expressed by equalizing the intensity of each of the blue laser light, the green laser light, and the red laser light.

  In order to irradiate the laser beam incident on the optical fiber 3127 as an image, the horizontal scanning unit 3123 and the vertical scanning unit 3124 scan the laser beam in the horizontal direction and the vertical direction to generate scanning image light.

  The horizontal scanning unit 3123 includes a resonant deflection element 3123a, a horizontal scanning control circuit 3123b, and a horizontal scanning angle detection circuit 3123c. The laser light incident on the optical fiber 3127 is collimated by the collimating optical system 3122 and is incident on the resonant deflection element 3123a. The resonant deflection element 3123a has a reflection surface 3123d that is swung by the horizontal scanning control circuit 3123b, and scans the incident laser light in the horizontal direction by being reflected by the swinging reflection surface 3123d. The horizontal scanning control circuit 3123b generates a driving signal for swinging the reflecting surface 3123d of the resonant deflection element 3123a based on the horizontal driving signal 3215 output from the signal processing circuit 3211. The horizontal scanning angle detection circuit 3123c detects a swing state such as a swing range and a swing frequency of the reflection surface 3123d of the resonant deflection element 3123a based on the displacement signal output from the resonant deflection element 3123a. A signal indicating the swing state is output to the image presentation unit connection I / F 212.

  The vertical scanning unit 3124 includes a deflection element 3124a, a vertical scanning control circuit 3124b, and a vertical scanning angle detection circuit 3124c. The deflection element 3124a has a reflection surface 3124d that is oscillated by the vertical scanning control circuit 3124b. The incident laser beam is reflected by the oscillating reflection surface 3124d and scanned in the vertical direction to scan two-dimensionally. The emitted image light is emitted to the relay optical system 3126. Based on the vertical drive signal 3216 output from the signal processing circuit 3211, the vertical scanning control circuit 3124b generates a drive signal for swinging the reflection surface 3124d of the deflection element 3124a. Based on the displacement signal output from the deflection element 3124a, the vertical scanning angle detection circuit 3124c detects the oscillation state such as the oscillation range and oscillation frequency of the reflection surface 3124d of the deflection element 3124a, and detects the oscillation state. Is output to the image presentation unit connection I / F 212.

  The relay optical system 3125 is disposed between the resonant deflection element 3123a and the deflection element 3124a. The relay optical system 3125 causes the laser beam scanned in the horizontal direction by the reflection surface 3123d of the resonance type deflection element 3123a to enter the reflection surface 3124d of the deflection element 3124a.

  Based on the “image signal” input from the image presentation unit connection I / F 212, the signal processing circuit 3211 outputs a horizontal drive signal 3215 and a vertical drive signal 3216 to the horizontal scanning control circuit 3123b and the vertical scanning control circuit 3124b, respectively. Then, image light is generated by changing the scanning angle of the reflecting surfaces 3123d and 3124d.

  The scanning angles of the reflection surfaces 3123d and 3124d thus changed are detected as detection signals by the horizontal scanning angle detection circuit 3123c and the vertical scanning angle detection circuit 3124c, and the detection signals are input to the image presentation unit connection I / F 212 to be This is fed back to the drive signal 3215 and the vertical drive signal 3216.

  The relay optical system 3126 includes lens systems 3126a and 3126b having a positive refractive power. The image light emitted from the deflecting element 3124a is converted into convergent image light by the lens system 26a so that the center lines of the scanned image light are made substantially parallel to each other. The converged image light is converted into substantially parallel scanned image light by the lens system 26b, and is condensed so that the center line of the scanned image light converges on the pupil Ea of the user.

  In the present embodiment, laser light incident from the optical fiber 3127 is scanned in the horizontal direction by the horizontal scanning unit 3123 and then scanned in the vertical direction by the vertical scanning unit 3124. The arrangement of the vertical scanning unit 3124 may be changed, and after the vertical scanning unit 3124 is scanned in the vertical direction, the horizontal scanning unit 3123 may scan in the horizontal direction.

(Movement mechanism of image presentation device)
FIG. 5 is a diagram illustrating a moving mechanism of the image presentation device. In FIG. 5, the half mirror 116 attached to the image presentation device 114 is disposed at the first position described above. In FIG. 5, the CCD camera 300 (see, for example, FIG. 1) is omitted.

  The moving mechanism of the image presentation device moves the base 150 arranged to face the image presentation device 114, the movement guide 152 formed on the surface of the base 150 facing the image presentation device 114, and the image presentation device 114. And a motor 154 which is a drive source for the above. A gear 152 a is formed on the movement guide 152, and a small gear 154 a that meshes with the gear formed on the movement guide 152 is attached to the tip of the rotating shaft of the motor 154. A cam roller 156 that engages with the movement guide 152 is attached to the image presentation device 114.

  For example, when the motor 154 rotates in one direction, the rotation of the small gear 154a in one direction is transmitted to the gear 152a, and the image presentation device 114 moves from the first position side to the second position side. At this time, the image presentation device 114 moves along the movement guide 152 by the action of the cam roller 156 engaged with the movement guide 152. In addition, when the motor 154 rotates in the other direction, the rotation of the small gear 154a in the other direction is transmitted to the gear 152a, and the image presentation device 114 starts from the second position side illustrated in FIG. Move to the position side as above. In the base 150, the above-described position sensor 340 is installed on each of the first position side and the second position side. When the image presentation device 114 moves to the first position side, the position sensor 340 installed on the first position side detects the image presentation device 114. The position sensor 340 is configured by a proximity switch, for example. Further, the position sensor 340 can also be configured by a switch and a protrusion, a Hall element and a magnet, and a reflective optical sensor.

  When the image presentation device 114 moves and the half mirror 116 is disposed at the first position, the image light 120b emitted from the image presentation unit 310 of the image presentation device 114 (see the image light 120a) and reflected by the half mirror 116 is obtained. Then, it enters the left eye 118 at a predetermined angle θ with respect to the front of the left eye 118. The user can visually recognize the image indicated by the image light 120b by directing his / her line of sight toward the image light 120b, and can visually recognize the outside world by directing his / her line of sight toward the front of the left eye 118.

  The base 150 is attached to the temple 104A by a predetermined method, for example. The motor 154 is attached to the base 150.

  FIG. 6 is a diagram illustrating a state in which the image presentation device has moved and the half mirror is disposed at the second position. In FIG. 6, the CCD camera 300 (see, for example, FIG. 1) is omitted. When the image presentation device 114 moves to the second position side, the position sensor 340 (not shown in FIG. 6, see FIG. 5B) installed on the second position side detects the image presentation device 114. When the image presentation device 114 moves and the half mirror 116 is arranged at the second position, the image light 120b emitted from the image presentation unit 310 of the image presentation device 114 (see the image light 120a) and reflected by the half mirror 116 is obtained. The light enters the left eye 118 from the front side of the face. The user visually recognizes the image light 120b emitted from the image presentation unit 310 of the image presentation device 114 disposed at the second position in a state where the line-of-sight direction is in front of the left eye 118. Therefore, the user can move the line-of-sight direction up, down, left, and right, for example, according to the calibration guidance indicated by the image light 120b. Details of calibration will be described later.

  Fig.7 (a) is a figure which shows the movement locus | trajectory of an image presentation apparatus and a half mirror. As is clear from this figure, the main body of the image presentation device 114 moves substantially horizontally on the arc M centered on the left eye 118, and the half mirror 116 moves substantially horizontally on the arc N centered on the left eye 118. . The CCD camera 300 disposed on the upper surface of the image presentation device 114 also moves substantially horizontally on the arc M together with the image presentation device 114. The direction (angle) at which the CCD camera 300 images the left eye 118 is larger than the incident angle of the image light 120b reflected by the half mirror 116 (the angle with respect to the front of the left eye 118; see angle θ in FIG. 5A). This is the direction.

  FIG. 7B is a diagram showing the amount of change when the half mirror attached to the image presentation device moves from the first position to the second position (illustration of the CCD camera 300 is omitted). In FIG. 7B, reference numerals 114 ′, 116 ′, 120a ′, and 120b ′ indicate the respective parts (drawn by two-dot chain lines) on the first position side, and reference numerals 114, 116, 120a, and 120b indicate Each part on the second position side is shown. As is clear from FIG. 7B, the half mirror 116 moves by an angle θ about the left eye 118 as the image presentation device 114 moves, and is disposed at the second position. Note that the amount of movement when the image presentation device 114 moves the movement guide 152 (see FIGS. 5 and 6) is an angle θ with the left eye 118 as the center.

  Hereinafter, various processes executed when the CPU 202 processes a predetermined program stored in the program ROM 204 on the RAM 208 will be described.

(Main process)
FIG. 8 is a diagram illustrating a processing flow of the main processing. This main processing is performed by the CPU 202 reading out a main processing program stored in the program ROM 204 and executing this program on the RAM 208 based on the acquired various data. The main process is started and executed on condition that the user turns on the power switch 330 of the HMD 100.

  The CPU 202 that has started this process first initializes the RAM 208 (S100). Then, the CPU 202 executes predetermined control so that the image light 120a indicating the menu screen is emitted from the image presentation unit 310 (S102). In step S <b> 102, the CPU 202 gives a command for inputting a menu image to the image presentation unit 310 to the image presentation unit connection controller 2122. The image presentation unit 310 receives an input of the menu screen from the image presentation unit connection controller 2122, and emits image light 120a indicating this. The menu screen includes an item for designating the start of content presentation (reproduction of content data) and an item for designating the start of calibration.

  After executing S102, the CPU 202 determines whether or not the user has selected to execute calibration according to the presentation of the menu (S104). Here, for example, the user selects an item presented via a keyboard connected to the communication unit 216 or an operation unit not shown in FIG. As a result of the determination, if the start of content presentation is specified (S104: No), the CPU 202 shifts the process to S112. On the other hand, when the start of calibration is designated (S104: Yes), the CPU 202 moves the image presentation device 114 to the second position side (left eye so that the half mirror 116 is disposed at the second position). The movement of 118 to the front front side is controlled (S106). Specifically, a predetermined command is given to the image presentation device moving unit controller 218 so that the motor 154 rotates and the movement of the image presentation device 114 to the second position side is started.

  Then, the CPU 202 acquires a signal output from the position sensor 340 installed on the second position side via the image presentation device moving unit controller 218, and starts and executes calibration (S108). Details of the calibration will be described later.

  After executing the calibration of S108, the CPU 202 controls the movement of the image presentation device 114 toward the first position (movement toward the left head side) so that the half mirror 116 is disposed at the first position (S110). ). Specifically, a predetermined command is given to the image presentation device moving unit controller 218 so that the motor 154 rotates and the movement of the image presentation device 114 to the first position side is started. Thereafter, the CPU 202 shifts the process to S112. The specific process of S110 is the same as that of S106 described above, and a description thereof will be omitted.

  The CPU 202 that has determined that the movement of the image presentation device 114 to the first position side has been completed based on the signal from the position sensor 340 installed on the first position side, continues the content data stored in the flash ROM 206 Playback (content presentation processing) is started and executed (S112). Details of the content presentation processing will be described later. Then, the CPU 202 determines whether or not a power OFF command (end command) has been input (S114). Here, the power-off command is input by the user operating the power switch 330. As a result of the determination, if the end command is not input (S114: No), the CPU 202 returns the process to S102, whereas if the end command is input (S114: Yes), the CPU 202 turns off the power, The main process ends.

(Calibration)
FIG. 9 is a diagram showing a calibration processing flow. This calibration is performed by the CPU 202 reading out a calibration program stored in the program ROM 204 and executing the program on the RAM 208 based on the acquired various data. Calibration is started and executed in S112 of the main process.

  The CPU 202 that has started this process generates an image centered on the gazing point P for calibration as shown in FIG. 10A, and performs predetermined control so that the image light 120a indicating the image is emitted. Execute (S200). In S <b> 200, the CPU 202 gives an instruction to input an image shown in FIG. 10A to the image presentation unit 310 to the image presentation unit connection controller 2122. The image presentation unit 310 receives the input of the image shown in FIG. 10A from the image presentation unit connection controller 2122 and emits image light 120a indicating the input.

  After executing S200, the CPU 202 generates an image (see FIG. 10B) in which the position of the gazing point P is moved to the left by a predetermined distance, and executes control similar to S200 so that the generated image is presented. (S202). The user visually recognizes and recognizes the gazing point P based on the image light 120 b emitted from the image presentation unit 310. Note that the white circles shown in FIG. 10B are drawn to explain the left-eye line input position L described later, and are not included in the actually generated image.

  The CPU 202 generates an image in which the gazing point P has moved to the left, and determines whether or not the position of the gazing point P is the left gaze input position L, which is the final movement position (S204). Here, when the CPU 202 obtains an instruction to determine the line-of-sight input position, the CPU 202 determines the left-side line-of-sight input position L, which is the final movement position. An instruction to determine the line-of-sight input position is input when the user operates a keyboard connected to the communication unit 216 or an operation unit included in the HMD 100. In the HMD 100 that controls a predetermined operation based on the user's line-of-sight direction, the configuration in which the line-of-sight input position is determined based on the input from the user can realize an appropriate calibration according to the user's preference. it can. As a result of the determination, if the position of the gazing point P is not the left gaze input position L (S204: No), the CPU 202 returns the process to S202, generates an image in which the gazing point P is sequentially moved to the left, and is generated. Control is performed so that images are presented sequentially. The user continues to visually recognize the gazing point P that moves to the left based on the sequentially presented images.

  On the other hand, as a result of the determination, when the position of the gazing point P is the left gaze input position L (S204: Yes), the CPU 202 visually recognizes the gazing point P at the left gaze input position L, That is, an image of the left eye 118 whose line-of-sight direction is at the left-eye line input position L is acquired, and predetermined processing is executed. Then, the CPU 202 stores the left reference state data obtained by the predetermined processing in the flash ROM 206 (S206).

  The process of S206 will be specifically described. First, the CPU 202 gives a predetermined command to the CCD camera connection controller 2102. The CCD camera connection controller 2102 that has acquired this command stores, in the CCD camera VRAM 2104, the image of the left eye 118 that is captured by the CCD camera 300 and whose viewing direction is at the left viewing line input position L. The CPU 202 corrects the image stored in the CCD camera VRAM 2104 based on the angle θ (see FIG. 7B). With this correction, an image showing the left eye 118 (the left eye 118 whose line-of-sight direction is at the left-eye line input position L) captured by the CCD camera 300 in a state where the half mirror 116 is disposed at the second position is the half mirror 116. Is corrected to an image showing the left eye 118 imaged in the state of being arranged at the first position. This point will be described with reference to FIG. FIG. 11A shows the left eye 118 imaged by the CCD camera 300 at the second position. Here, the point O at which the dotted lines intersect indicates the center point of the lens of the CCD camera 300. A straight line Q indicates a captured image plane. Here, when the point O and the straight line Q are rotated by an angle θ (see FIG. 7B) in the state of FIG. 11A, each point constituting the image of the black eye 118a with respect to the outer shape of the left eye 118. Therefore, this deviation is obtained in advance at each point of the black eye 118a and stored in the flash ROM 206, for example. The CPU 202 corrects the image shown in FIG. 11A to the state shown in FIG. 11B based on the deviation of each point of the black eye 118a stored in the flash ROM 206 in advance.

  Then, the CPU 202 stores the corrected image in the flash ROM 206 as left reference state data. In the present embodiment, the left reference state data is associated with an operation realized in the HMD 100, specifically, with the start of reproduction of content data stored in the flash ROM 206 (refer to “content presentation processing” described later for details).

  After executing S206, as in S200, the CPU 202 generates an image centered on the gazing point P for calibration as shown in FIG. 10 (c), and outputs image light 120a indicating the image. Execute the control. Note that the processing executed in S206 is the same as that in S200, and a description thereof will be omitted.

  Then, the CPU 202 generates an image (see FIG. 10D) in which the position of the gazing point P is moved to the right opposite to S202 by a predetermined distance, and performs the same control as S206 (S200) so that this is presented. Execute (S210). The user visually recognizes and recognizes the gazing point P based on the image light 120 b emitted from the image presentation unit 310 and reflected by the half mirror 116. Note that the white circles shown in FIG. 10D are drawn to explain the right-hand line-of-sight input position R described later, and are not included in the actually generated image.

  The CPU 202 generates an image in which the gazing point P has moved to the right, and determines whether or not the position of the gazing point P is the right gaze input position R that is the final position of movement (S212). The CPU 202 determines the right gaze input position R by the same method as that for the left gaze input position L. As a result of the determination, when the position of the gazing point P is not the right gaze input position R (S212: No), the CPU 202 returns the process to S210, generates an image in which the gazing point P is sequentially moved to the right, and is generated. Control is performed so that images are presented sequentially. The user continues to visually recognize the gazing point P that moves to the right based on the sequentially presented images.

  On the other hand, as a result of the determination, when the position of the gazing point P is the right gaze input position R (S212: Yes), the CPU 202 visually recognizes the gazing point P at the right gaze input position R, That is, the image of the left eye 118 whose line-of-sight direction is at the right-eye line input position R is acquired, and the correction process described in relation to S206 is executed. Then, the CPU 202 stores the right reference state data obtained by the correction process in the flash ROM 206 (S214). After executing S214, the CPU 202 returns the process to S110 in FIG.

  In the present embodiment, the right reference state data is associated with the operation realized in the HMD 100, specifically, the reproduction stop of the content data stored in the flash ROM 206 (refer to “content presentation process” described later for details). Note that the processing executed in S214 is the same as that in S206 described above, and a description thereof is omitted.

(Content presentation process)
FIG. 12 is a diagram illustrating a processing flow of content presentation processing. The processing for content presentation is performed by the CPU 202 reading out the content presentation program stored in the program ROM 204 and executing the program on the RAM 208 based on the acquired various data. The content presentation process is started and executed in S112 of the main process.

  The CPU 202 that has started this process determines whether or not the line of sight of the left eye 118 is the left side (S300). Here, the CPU 202 performs pattern matching using the captured image indicating the state of the left eye 118 captured by the CCD camera 300 and stored in the CCD camera VRAM 2104 and the image indicated by the left reference state data in S300. Make a decision. Note that the left reference state data is stored in the flash ROM 206 in S206 of calibration in FIG.

  As a result of the determination in S300, if the line-of-sight direction is the left side, in other words, if it is determined that there is an identity between both images as a result of pattern matching (S300: Yes), the CPU 202 stores in the flash ROM 206. The reproduction of the content data thus started is started (S302). Specifically, the CPU 202 renders the content data and stores the image generated by the rendering in the image presentation unit VRAM 2124. Then, the CPU 202 gives a predetermined command to the image presentation unit connection controller 2122 so that the image light 120 a indicating the image stored in the storage in the image presentation unit VRAM 2124 is emitted from the image presentation unit 310. On the other hand, as a result of the determination in S300, if the line-of-sight direction is not the left side, in other words, if it is determined that there is no identity between the two images as a result of pattern matching (S300: No), the CPU 202 advances the process to S304. Transition.

  In step S304, the CPU 202 determines whether the line-of-sight direction of the left eye 118 is the right side. Here, the CPU 202 performs the determination in S304 by the same method as in S300. That is, pattern matching is performed using the captured image indicating the state of the left eye 118 stored in the CCD camera VRAM 2104 and the image indicated by the right reference state data. The right reference state data is stored in the flash ROM 206 in S214 of calibration in FIG.

  As a result of the determination in S304, if the line-of-sight direction is the right side (S304: Yes), the CPU 202 stops the reproduction started in S302 (S306), and returns the process to S114 in FIG. On the other hand, when the line-of-sight direction is not the right side (S304: No), the CPU 202 returns the process to S114 of FIG. 8 without executing S306.

(Advantageous effects based on this embodiment)
The HMD 100 according to the present embodiment first generates an image centered on the gazing point P, presents it to the user, and then moves the gazing point P in a predetermined direction to the final line-of-sight input positions L and R. A configuration was adopted in which calibration was performed by moving sequentially. According to this, the user can continue to visually recognize the gazing point P that sequentially moves to the final line-of-sight input positions L and R. As a result, the left eye 118 of the user whose line-of-sight direction is at a predetermined line-of-sight input position can be captured by the CCD camera 300, and the reference state data based on the captured image can be stored in the flash ROM 206.

  When an image in which the gazing point P is displayed at the line-of-sight input positions L and R from the beginning is presented to the user, for example, the user does not notice the gazing point P, and as a result, the user sets the gazing point P. It is possible to assume a situation where the user does not visually recognize. In this case, the left eye 118 of the user whose line-of-sight direction is at the line-of-sight input position cannot be imaged by the CCD camera 300, and as a result, the reference state data cannot be stored in the flash ROM 206.

(Modification)
(1) In the above description, the monocular HMD 100 for the left eye 118 has been described as an example. However, the configuration of the present embodiment can also be adopted in an HMD that presents images to both eyes so as to be visible. In this case, a configuration is adopted in which the image presentation device for the right eye and the image presentation device for the left eye are movable in front of each eye and each temporal region. Then, calibration is performed for each eye. In addition, when each image obtained by imaging the left and right eyes (each captured by two CCD cameras) matches each side reference state data, the operation of the HMD (for example, reproduction start of content data) is controlled. .

  Note that even a binocular HMD can be configured such that calibration is executed only for the state of the left eye 118 and the operation of the HMD is controlled based only on the left eye 118, for example. Moreover, it can also be set as the structure based on the combination of both eyes.

  (2) In the above description, the left and right gaze directions are described as an example. However, in addition to the configuration based on the left and right, for example, a configuration based on the top or bottom may be used. In addition, the operation of the HMD 100 associated with the line-of-sight direction can be an operation other than the start and stop of reproduction.

  (3) In the above description, the HMD 100 in which the half mirror 116 moves by attaching the half mirror 116 to the image presentation device 114 and moving the image presentation device 114 has been described as an example. However, other configurations can be employed. For example, the half mirror 116 can be moved alone. In this case, the half mirror 116 is independently moved and arranged at each position shown in FIG. 5A and FIG. 6A, for example. In an HMD that employs a diffraction grating or hologram instead of the half mirror 116, the diffraction grating or hologram is moved between the first position and the second position and arranged at each position. And the operation | movement similar to each operation | movement mentioned above is performed. In the case of an HMD that does not employ a reflecting means such as the half mirror 116, the image presentation device (image presentation unit) is disposed at the position where the half mirror 116 illustrated in FIGS. 5A and 6 is disposed. The same operation as each operation is executed.

  (4) In the above description, the configuration in which the determination in S300 and S304 of the content presentation process illustrated in FIG. However, other configurations can be employed. For example, the ratio of white eyes and black eyes represented in the image captured by the CCD camera 300 is compared with the ratio of white eyes and black eyes represented in the image indicated by the reference state data. It can also be set as the structure which performs judgment.

  (5) In the above description, the configuration in which the image of the left eye 118 imaged by the CCD camera 300 is corrected and the reference state data indicating the corrected state is stored is described as an example (the calibration S206 and the calibration shown in FIG. 9). (See S214). However, other configurations can be employed. For example, the captured image of the left eye 118 may be stored in the flash ROM 206, and the same correction as the correction executed in S206 or the like may be executed each time the content presentation process S300 and S304 shown in FIG. it can.

  (6) In the above description, the configuration in which the reference state data is stored in the flash ROM 206 in the calibration S206 and S214 illustrated in FIG. However, other configurations can be employed. For example, the main state shown in FIG. 7 may be configured to store the reference state data in the flash ROM 206. In this case, when the determinations of S204 and S212 in FIG. 9 are affirmed (S204, S212: Yes), the image captured by the CCD camera 300 is stored in the CCD camera VRAM 2104 as it is, for example.

100 HMD
114 Image presentation device 202 CPU
204 Program ROM
206 flash ROM
208 RAM
300 CCD camera 340 Position sensor

Claims (4)

A head-mounted display that is mounted on a user's head and allows the user to visually recognize an image,
An image presentation device that emits image light indicating the image to the eyes of the user;
The image presentation device is arranged at a first position, which is a predetermined position from the front side of the user's one eye to the temporal region, and a second position, which is a predetermined position substantially in front of the one eye. Arrangement means to arrange;
The first state information indicating the state of the one eye is acquired in a state where the image presentation device is disposed at the first position, and the use is performed in a state where the image presentation device is disposed at the second position. State information acquisition means for acquiring second state information indicating the state of the one eye whose gaze direction of the person is in the first direction;
Storage means for storing reference state information corresponding to the second state information;
The first state information acquired by the state information acquisition unit is compared with the reference state information stored in the storage unit, and the user of the user in a state where the image presentation device is arranged at the first position is compared. Detecting means for detecting that the line-of-sight direction of one eye is the first direction;
An operation control means for controlling a first operation of the head mounted display when the detection means detects that the first direction is detected;
Command acquisition means for acquiring an execution command for instructing storage of the reference state information in the storage means;
Moving means for moving the arrangement of the image presentation device in the arrangement means from the first position to the second position when the instruction acquisition means acquires the execution instruction;
Storage control means for storing, in the storage means, reference state information corresponding to the second state information acquired by the state information acquisition means in a state where the image presentation device is arranged at the second position by the moving means; A head-mounted display comprising:   The moving means moves the arrangement of the image presentation device in the arrangement means from the second position to the first position when the storage control means stores the reference state information in the storage means. The head mounted display according to claim 1.   The reference state information indicates the state of the one eye in which the user's line-of-sight direction is in the first direction in a state where the image presentation device is arranged at the first position. The head mounted display according to claim 1, wherein the state information is corrected as described above. The image presentation device includes:
Image presentation means for emitting the image light;
Reflecting means for reflecting image light emitted from the image presentation means in the direction of the user's eyes,
The arrangement means arranges the reflection means at the first position and the second position,
The head mounted display according to any one of claims 1 to 3, wherein the moving means moves the reflecting means from the first position to the second position.