WO2013179425A1 - Display device, head-mounted display, calibration method, calibration program, and recording medium - Google Patents

Abstract

This display device is an optically transmissive type and displays added information added to a real environment seen by a user. A position detection means in said display device detects the position of a target object that exists in the real environment, and a gaze-direction detection means detects the direction of the user's gaze when the user is looking at the target object. A calibration means computes calibration data for converting from a first coordinate system defined by the position detection means to a second coordinate system defined by the display performed by the display device. Specifically, the calibration means computes said calibration data on the basis of the position detected by the position detection means and the gaze direction detected by the gaze-direction detection means.

Description

Display device, head mounted display, calibration method, calibration program, and recording medium

The present invention relates to a technical field for additionally presenting information to a real environment.

Conventionally, a technology related to augmented reality (AR) in which additional information such as computer graphics (CG) and characters is additionally presented to the real environment using an optically transmissive display device such as a head-mounted display. Has been proposed. In addition, regarding AR using such an optically transmissive display device (hereinafter referred to as “optically transmissive AR” as appropriate), a deviation between the position of the real image and the display position of information viewed from the user's viewpoint is detected. There has been proposed a technique for performing calibration for correction.

For example, Non-Patent Document 1 describes a technique in which a user matches a marker position in a real environment and a display position on a display, and performs calibration based on information at that time. Further, Patent Document 1 does not perform calibration for each user, but notifies the deviation between the position of the eyeball at the time of previous calibration and the current position of the eyeball, so that the composite position can be obtained without recalibration. It is described that the deviation is eliminated.

Further, in Patent Document 2, regarding a device having an optically transmissive head-mounted display, an external shooting camera, and a line-of-sight detection means, a specific range in the external shooting camera is selected by the movement of the user's line of sight. A technique for fetching and processing a selection range as image information from a camera is described. In this technique, for example, while reading English, image processing is performed on a designated area by line of sight, and the data is displayed after being read and translated. Japanese Patent Application Laid-Open No. 2004-228620 describes that a pupil position is accurately detected and a display position is corrected in accordance with the position of the pupil for a medical display device.

JP 2006-133688 A JP 2000-152125 A JP 2008-18015 A

In the technique described in Non-Patent Document 1, the user needs to perform the work of adjusting the marker position in the real environment and the display position of the display a plurality of times. Therefore, it is necessary for the user to move the marker and the head many times, which tends to be troublesome and time consuming. In addition, when the display device is for both eyes, it is necessary to perform calibration separately for the left and right eyes.

On the other hand, the technique described in Patent Document 1 requires a configuration that allows the eye position to be freely adjusted, and is not applicable when it is desired to change the setting or position of a camera or display device. Further, in the technique described in Patent Document 2, since the calibration is not performed, there is a possibility that the display position is shifted from a desired position. Further, the technique described in Patent Document 3 cannot be applied when it is desired to change the setting or position of a camera or a display device.

Examples of the problem to be solved by the present invention include the above. It is an object of the present invention to provide a display device, a head mounted display, a calibration method and a calibration program, and a recording medium that can be easily calibrated.

In the first aspect of the present invention, an optically transmissive display device that displays additional information with respect to a real environment visually recognized by a user includes a position detection unit that detects a position of an object existing in the real environment, Based on the line-of-sight detection unit that detects the line-of-sight direction when the user is directing the line of sight to the object, the position detected by the position detection unit, and the line-of-sight direction detected by the line-of-sight detection unit, Calibration means for calculating calibration data for converting from a first coordinate system defined by the position detection means to a second coordinate system defined by display by the display device.

In the invention according to claim 10, an optically transmissive head mounted display that displays additional information with respect to a real environment visually recognized by a user includes a position detection unit that detects a position of an object existing in the real environment, Based on gaze direction detection means for detecting a gaze direction when the user is gazeing at the object, the position detected by the position detection means, and the gaze direction detected by the gaze direction detection means Calibration means for calculating calibration data for conversion from the first coordinate system defined by the position detection means to the second coordinate system defined by display by the display device.

In the invention according to claim 11, the calibration method executed by the optical transmission type display device that displays the additional information with respect to the real environment visually recognized by the user detects the position of the object existing in the real environment. A position detection step, a line-of-sight detection step for detecting a line-of-sight direction when the user is directing a line of sight to the object, and the position detected by the position detection step and the line-of-sight detected by the line-of-sight detection step And a calibration step of calculating calibration data for conversion from the first coordinate system defined in the position detection step to the second coordinate system defined by the display by the display device based on the direction. .

In a twelfth aspect of the present invention, a calibration program executed by an optically transmissive display device that includes a computer and displays additional information with respect to a real environment visually recognized by a user, sets the computer to the real environment. Position detecting means for detecting the position of an existing object, eye gaze direction detecting means for detecting a gaze direction when the user is looking at the object, and the position and gaze direction detected by the position detecting means Based on the line-of-sight direction detected by the detection means, calibration data for converting from the first coordinate system defined by the position detection means to the second coordinate system defined by the display by the display device is calculated. Function as calibration means.

In the invention described in claim 13, the recording medium records the calibration program described in claim 12.

It is an external view which shows schematic structure of HMD. It is a figure which shows the internal structure of HMD schematically. The figure for demonstrating the reason for calibrating is shown. An example of the marker used in 1st Example is shown. It is a block diagram which shows the structure of the control part which concerns on 1st Example. It is a flowchart which shows the whole process of HMD. It is a flowchart which shows the calibration process which concerns on 1st Example. It is a block diagram which shows the structure of the control part which concerns on 2nd Example. It is a flowchart which shows the calibration process which concerns on 2nd Example.

In one aspect of the present invention, an optically transmissive display device that displays additional information with respect to a real environment visually recognized by a user includes a position detection unit that detects a position of an object existing in the real environment, and the user Based on the gaze direction detecting means for detecting the gaze direction when the gaze is directed toward the object, the position detected by the position detecting means and the gaze direction detected by the gaze direction detecting means, Calibration means for calculating calibration data for conversion from the first coordinate system defined by the position detection means to the second coordinate system defined by the display by the display device.

The above display device is configured to be able to realize an optically transmissive AR, and displays additional information for a real environment visually recognized by the user. The position detection unit detects the position of the target existing in the real environment, and the line-of-sight direction detection unit detects the line-of-sight direction when the user directs the line of sight toward the target. Examples of the “object” include various things such as artificial feature points (such as markers) and natural feature points that exist in the real environment. The calibration means is a calibration for correcting the positional deviation between the real image and the display viewed from the user's viewpoint, which is caused by the position and orientation relationship between the position detection means, the display device, and the user's eye in the optical transmission type AR. I do. Specifically, the calibration means is based on the position of the object detected by the position detection means and the line-of-sight direction detected by the line-of-sight direction detection means based on the first coordinate system defined by the position detection means. Calibration data for conversion to the second coordinate system defined by the display is calculated.

According to the above display device, since the calibration is performed based on the line-of-sight direction, the user only has to perform an operation of directing the line of sight toward the object at the time of calibration. Such work can be said to be less burdened on the user at the time of calibration than the work of moving the display device or marker and aligning the position of the cross display and the marker as described in Non-Patent Document 1. It can be said that it does not take time and effort. Further, when the display device is for both eyes, the right and left eyes can be calibrated at the same time by providing the line-of-sight direction detection means on the left and right. As mentioned above, according to said display apparatus, the burden of the user at the time of calibration can be reduced effectively.

In one aspect of the above-described display device, the line-of-sight direction detection unit detects the line-of-sight direction when operating an input unit for inputting that the user gazes. In this aspect, the user informs that the user is gazing by operating an input means such as a button. Thereby, it is possible to appropriately detect the line-of-sight direction when the user is gazing at the object.

In another aspect of the display device, the line-of-sight direction detection unit detects the line-of-sight direction when the user performs a gaze operation for a predetermined time. This makes it possible to suppress disturbances in the line of sight and movement of the head, as compared with the case of notifying gaze by operating buttons, etc., reducing the error factor during calibration and improving calibration accuracy. it can.

In another aspect of the display device, the line-of-sight direction detection unit detects the line-of-sight direction when the user blinks. In this way, it is possible to suppress disturbances in the line of sight and movement of the head as compared with the case of notifying the gaze by operating buttons, etc., reducing the error factor during calibration and improving the calibration accuracy. Can do.

In a preferred example, the position detection means detects the position based on a photographed image obtained by photographing the real environment with an image acquisition device, and the first coordinate system is a coordinate system based on the image acquisition device. It is.

In a preferred example, the object is a marker used for calculating the calibration data by the calibration means. Thereby, the position of the object can be easily detected during calibration.

In a preferred example, the line-of-sight direction detecting means obtains coordinates in the second coordinate system corresponding to the intersection of the line-of-sight direction and the display surface of the display device, and the calibration means is determined by the line-of-sight direction detecting means. The calibration data is calculated based on the obtained coordinates and the position detected by the position detecting means.

In another aspect of the above display device, the display device further includes a specifying unit that specifies the object to be watched by the user, and the gaze direction detecting unit is configured to allow the user to specify the object specified by the specifying unit. The direction of the line of sight when gazing is detected. As a result, the user can perform calibration only by performing an operation such as gazing at the designated object.

Preferably, in the above display device, the designation unit can designate the object by displaying an image corresponding to the object to be watched. As a result, the user can appropriately grasp the object to be watched.

In another aspect of the present invention, an optically transmissive head mounted display that displays additional information with respect to a real environment visually recognized by a user includes a position detection unit that detects a position of an object existing in the real environment, Based on the line-of-sight detection unit that detects the line-of-sight direction when the user is directing the line of sight to the object, the position detected by the position detection unit, and the line-of-sight direction detected by the line-of-sight detection unit, Calibration means for calculating calibration data for converting from a first coordinate system defined by the position detection means to a second coordinate system defined by display by the display device.

In still another aspect of the present invention, a calibration method executed by an optically transmissive display device that displays additional information with respect to a real environment visually recognized by a user detects a position of an object existing in the real environment. A position detection step, a line-of-sight detection step for detecting a line-of-sight direction when the user is directing a line of sight to the object, and the position detected by the position detection step and the line-of-sight detected by the line-of-sight detection step And a calibration step of calculating calibration data for conversion from the first coordinate system defined in the position detection step to the second coordinate system defined by the display by the display device based on the direction. .

In still another aspect of the present invention, a calibration program executed by an optically transmissive display device that includes a computer and displays additional information with respect to a real environment visually recognized by a user includes the computer in the real environment. Position detecting means for detecting the position of an existing object, eye gaze direction detecting means for detecting a gaze direction when the user is looking at the object, and the position and gaze direction detected by the position detecting means Based on the line-of-sight direction detected by the detection means, calibration data for converting from the first coordinate system defined by the position detection means to the second coordinate system defined by the display by the display device is calculated. Function as calibration means.

The above calibration program can be suitably handled in a state of being recorded on a recording medium.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[First embodiment]
First, the first embodiment will be described.

(Device configuration)
FIG. 1 is an external view showing a schematic configuration of a head mounted display (hereinafter, appropriately referred to as “HMD”) according to the first embodiment. As shown in FIG. 1, the HMD 100 mainly includes a transmissive display unit 1, an imaging unit 2, and a mounting unit 3. The HMD 100 is configured as a glasses, and the user wears the HMD 100 on the head for use. The HMD 100 realizes AR (augmented reality) by displaying CG as an example of “additional information” in the present invention on the transmissive display unit 1 so as to correspond to the position of the marker provided in the real environment. . The HMD 100 is an example of the “display device” in the present invention.

The imaging unit 2 includes a camera, and photographs the real environment in front of the user with the HMD 100 mounted. The imaging unit 2 is provided between two transmissive display units 1 arranged side by side. In the present embodiment, the marker position and the like are detected based on the captured image of the imaging unit 2.

The mounting unit 3 is a member (glass frame-shaped member) configured to be mounted on the user's head, and configured to sandwich the user's head from both the left and right sides.

The transmissive display unit 1 is configured to be optically transmissive, and one transmissive display unit 1 is provided corresponding to each of the left and right eyes of the user. The user sees the real environment through the transmissive display unit 1 and sees the CG displayed on the transmissive display unit 1, so that a CG that does not exist in the real environment exists as if in the real environment. Can feel like. That is, AR (augmented reality) can be realized.

In order to detect a three-dimensional position of a marker or the like using a captured image of the imaging unit 2 (details will be described later), the imaging unit 2 is preferably configured with a stereo camera. However, it is not limited to using a stereo camera, and in another example, a monocular camera can be used for the imaging unit 2. In this case, a three-dimensional position may be detected by using a marker or picture marker of a known size or feature, a three-dimensional object, or using a difference in viewpoint due to camera movement. In yet another example, a three-dimensional position can be detected by using a TOF (Time-of-Flight) camera and a visible light camera together in the imaging unit 2. In yet another example, a three-dimensional position can be detected using pattern projection with a laser or projector, and triangulation using a camera.

FIG. 2 is a diagram schematically showing the internal structure of the HMD 100. As shown in FIG. As shown in FIG. 2, the HMD 100 includes a control unit 5, a near-infrared light source 6, and a line-of-sight direction detection unit 7 in addition to the transmission display unit 1 and the imaging unit 2 described above. The transmissive display unit 1 includes a display unit 1a, a lens 1b, and a half mirror 1c (see the area surrounded by the broken line).

The display unit 1a is composed of an LCD (Liquid Crystal Display), a DLP (Digital Light Processing), an organic EL, and the like, and emits light corresponding to an image to be displayed. A configuration in which light from a laser light source is scanned by a mirror may be applied to the display unit 1a. The light emitted from the display unit 1a is magnified by the lens 1b and then reflected by the half mirror 1c and enters the user's eyes. Thereby, the user visually recognizes a virtual image formed on the surface indicated by reference numeral 4 in FIG. 2 (hereinafter, referred to as “display surface 4” as appropriate) through the half mirror 1c.

Near-infrared light source 6 irradiates near-infrared rays toward the eyeball. The line-of-sight direction detection unit 7 detects the user's line-of-sight direction by detecting the near-infrared reflected light (Purkinje image) and the position of the pupil on the cornea surface. For example, for the detection of the gaze direction, a known corneal reflection method (in one example, “Yusuke Sakashita, Hironobu Fujiyoshi, Yutaka Hirata,“ Measurement of three-dimensional eye movement by image processing ”, Experimental Mechanics, Vol. 6, No. .3, pp.236-243, September 2006 ”). In this method, the user is able to detect the location on the display of the HMD 100 more accurately by performing the work of gazing at the display of the HMD 100 a plurality of times and performing calibration for the detection of the line-of-sight direction. The line-of-sight direction detection unit 7 supplies information regarding the line-of-sight direction detected in this way to the control unit 5. The line-of-sight direction detection unit 7 is an example of the “line-of-sight direction detection unit” in the present invention.

The control unit 5 includes a CPU, RAM, ROM, and the like (not shown), and performs overall control of the HMD 100. Specifically, the control unit 5 calibrates the display position of the CG based on the photographed image captured by the imaging unit 2 and the line-of-sight direction detected by the line-of-sight direction detection unit 7, and renders the CG to be displayed. Etc. Details of the control performed by the control unit 5 will be described later.

It should be noted that the detection of the line-of-sight direction is not limited to the method described above. In another example, the line-of-sight direction can be detected by photographing the eyeball reflected by the infrared half mirror. In still another example, the line of sight can be detected by detecting a pupil, an eyeball, or a face with a monocular camera. In yet another example, the line-of-sight direction can be detected using a stereo camera. Further, there is no limitation to using a non-contact type gaze direction detection method, and a contact type gaze direction detection method may be used.

(Calibration method)
Next, the calibration method according to the first embodiment will be specifically described.

FIG. 3 is a diagram for explaining the reason for performing calibration. As shown in FIG. 3A, in the HMD 100, the position of the user's eyes and the position of the imaging unit 2 are different from each other, so that an image (captured image) acquired by the imaging unit 2 and an image reflected in the user's eyes are displayed. Are different from each other. For example, it is assumed that the user's eyes, the imaging unit 2, and the marker 200 provided in the real environment are in a positional relationship as shown in FIG. In this case, in the image P1 acquired by the imaging unit 2 (see FIG. 3B), the marker 200 is located on the left side of the image, but the image P3 reflected in the user's eyes (see FIG. 3C). ), The marker 200 is positioned on the right side of the image. The marker 200 is provided on the object 400 in the real environment. The marker 200 is one of objects to which additional information such as CG should be presented.

Here, when the position of the marker 200 is detected based on the image P1 acquired by the imaging unit 2, and the CG 300 is directly combined with the detected position in the image P1, the image P2 in which the positions of the CG 300 and the marker 200 coincide with each other. Will be generated. However, an optically transmissive display device such as the HMD 100 needs to perform calibration according to the difference between the position of the user's eyes and the position of the imaging unit 2. This is because if the calibration is not performed, the position and orientation (direction) of the marker 200 and the CG 300 may be shifted from the viewpoint of the user as shown in the image P4 in FIG. It is.

Therefore, in the first embodiment, the control unit 5 performs correction so that the positions and orientations (directions) of the CG 300 and the marker 200 coincide with each other as in an image P5 in FIG. Specifically, the control unit 5 performs the correction by converting an image based on the imaging unit 2 into an image of the HMD 100 based on the eye. That is, the control unit 5 performs conversion from a coordinate system in the imaging unit 2 (hereinafter referred to as “imaging coordinate system”) to a coordinate system in display by the HMD 100 (hereinafter referred to as “display coordinate system”). . The imaging coordinate system is an example of the “first coordinate system” in the present invention, and the display coordinate system is an example of the “second coordinate system” in the present invention.

In the first embodiment, the controller 5 performs a process of calculating calibration data, which is a matrix for converting from the imaging coordinate system to the display coordinate system (hereinafter referred to as “calibration process” as appropriate). The calibration data is determined by the relationship between the position and orientation of the display surface 4, the imaging unit 2, and the eye. When the display surface 4, the imaging unit 2, and the eye all move in the same direction and the same angle, There is no problem using the same calibration data. Therefore, the HMD 100 first calculates calibration data (e.g., calculates calibration data at the start of use of the HMD 100 or when requested by the user), and then uses the calculated calibration data to correct the above-described deviation. Make corrections.

Specifically, in the first embodiment, the control unit 5 calculates calibration data based on the gaze direction detected by the gaze direction detection unit 7 and the captured image captured by the imaging unit 2. In this case, in the first embodiment, using a real environment in which a plurality of calibration markers (hereinafter simply referred to as “markers”) are arranged, the control unit 5 uses a captured image of such a real environment. From the imaging coordinate system to the display coordinate system based on the detected marker position in the imaging coordinate system and the line-of-sight direction detected by the line-of-sight direction detection unit 7 when the user is gazing at the marker in the real environment. Calculate calibration data for conversion to. More specifically, the control unit 5 determines the position of the marker in the imaging coordinate system and the coordinates of the display coordinate system corresponding to the intersection of the user's line-of-sight direction and the display surface 4 (hereinafter referred to as “line-of-sight direction coordinates”). Based on this, calibration data is calculated.

In the first embodiment, the control unit 5 obtains the marker position and the line-of-sight direction coordinate in the imaging coordinate system for the plurality of calibration markers, and the obtained position of the plurality of marker and the plurality of line-of-sight direction coordinates. Based on the above, calibration data is calculated. Specifically, the control unit 5 designates one marker from among a plurality of markers, and performs a process of designating another marker after the user gazes at the marker, so that each time, A plurality of marker positions and a plurality of gaze direction coordinates are obtained by obtaining a marker position and a gaze direction coordinate in the imaging coordinate system. In this case, the control unit 5 displays an image for designating a marker to be watched (hereinafter referred to as “gaze target image”), and when the user gazes at the marker corresponding to the gaze target image, By pressing a button as a user interface for calibration (UI: User 制 御 Interface), the control unit 5 is notified.

As described above, the control unit 5 is an example of the “position detecting means”, “calibration means”, and “designating means” in the present invention.

FIG. 4 shows an example of a marker used in the first embodiment. FIG. 4 illustrates a case where nine markers 200a to 200i with numbers 0 to 8 are used. 4A shows a view of the markers 200a to 200i observed from the front, and FIG. 4B shows a view of the markers 200a to 200i observed from above. The markers 200a to 200i have substantially the same size.

As shown in FIG. 4 (a), the markers 200a to 200i are arranged such that their positions in the upper, lower, left and right directions are different from each other. Further, as shown in FIG. 4B, the markers 200a to 200e and the markers 200f to 200i are arranged so that the front and rear positions are different. Specifically, the markers 200a to 200e are disposed on the back side, and the markers 200f to 200i are disposed on the near side. For example, when the control unit 5 designates one marker to be watched by the user among the markers 200a to 200i, the control unit 5 displays a gaze target image including a number (any one of 0 to 8) attached to the marker. Let

In addition, although the example using a rectangular marker was shown in FIG. 4, it is not limited to this. Various markers can be used as long as the three-dimensional position can be detected and the user can gaze. For example, a circular marker, a polygonal marker, a marker composed of a reflective object or a light emitting object, a marker of an image registered in advance, or the like can be used.

Further, the arrangement of the markers is not limited to that shown in FIG. Various arrangements can be applied as long as the center positions of the markers are not on the same plane.

Furthermore, it is not limited to using a plurality of markers, and only one marker may be used. One marker is used for one gaze, and it is only necessary that the position of each marker is different when gazes a plurality of times. Therefore, this can be realized with one marker without using a plurality of markers. Because it can. For example, the same thing as the case where a plurality of markers are used by moving one marker or moving the user's head with one marker fixed each time when the user gazes a plurality of times. realizable.

(Configuration of control unit)
Next, a specific configuration of the control unit 5 according to the first embodiment will be described with reference to FIG.

FIG. 5 is a block diagram illustrating a configuration of the control unit 5 according to the first embodiment. As shown in FIG. 5, the control unit 5 mainly includes a calibration unit 51, a conversion matrix calculation unit 52, a rendering unit 53, and a selector (SEL) 54.

The button 8 is a button that is pressed when the user gazes at the marker as described above. When the button 8 is pressed by the user, it outputs a gaze completion signal indicating that the user gazes at the marker to the gaze direction detection unit 7 and the calibration unit 51. The button 8 is an example of the “input unit” in the present invention.

When the gaze completion signal is input from the button 8, the gaze direction detection unit 7 detects the gaze direction of the user at this time. Specifically, the line-of-sight direction detection unit 7 obtains line-of-sight direction coordinates (Xd, Yd), which are coordinates of the display coordinate system, corresponding to the intersection of the line-of-sight direction and the display surface 4 when the user is gazing. The line-of-sight direction coordinates (Xd, Yd) are output to the calibration unit 51.

The calibration unit 51 includes a calibration control unit 51a, a gaze target selection unit 51b, a gaze direction coordinate accumulation unit 51c, a calibration marker position detection unit 51d, a marker position accumulation unit 51e, and a calibration data calculation unit 51f. . The calibration unit 51 calculates calibration data M by performing calibration processing when a calibration start trigger is input by pressing a predetermined button (not shown) or the like.

The calibration control unit 51a controls the calibration process. Specifically, the calibration control unit 51a controls the gaze target selection unit 51b, the calibration data calculation unit 51f, and the selector 54. The calibration control unit 51a starts the calibration process when the calibration start trigger is input. Specifically, when a calibration start trigger is input, the calibration control unit 51a responds to a gaze completion signal from the button 8 and updates a display update signal for updating a gaze target image that specifies a marker to be gazeed by the user. Is output to the gaze target selection unit 51b. In addition, when the gaze completion signal from the button 8 is input a predetermined number of times, the calibration control unit 51 a outputs a calculation trigger to the calibration data calculation unit 51 f and outputs a mode switching signal to the selector 54. As will be described later, the calibration data calculation unit 51f calculates calibration data M when a calculation trigger is input. In addition, when a mode switching signal is input, the selector 54 outputs data to the display unit 1a as data corresponding to the gaze target image (gaze target image data) and image data such as CG to be displayed as additional information. The mode is switched with (display data).

The gaze target selection unit 51b generates a gaze target image for designating a marker to be watched by the user when the display update signal is input from the calibration control unit 51a. Specifically, the gaze target selection unit 51b selects one marker to be watched by the user from among a plurality of markers (specifically, selects one marker that has not yet been watched by the user in the current calibration process). Then, gaze target image data corresponding to the marker is generated. For example, the gaze target selection unit 51b generates a gaze target image that includes numbers and characters attached to the selected marker. Then, the gaze target selection unit 51 b outputs the generated gaze target image data to the selector 54. Note that the gaze target selection unit 51b corresponds to an example of a “designating unit” in the present invention.

The gaze direction coordinate accumulation unit 51c receives the gaze direction coordinates (Xd, Yd) from the gaze direction detection unit 7, and accumulates the gaze direction coordinates (Xd, Yd). Note that the line-of-sight direction coordinates (Xd, Yd) correspond to the position coordinates of the marker with reference to the display coordinate system.

The calibration marker position detection unit 51d receives the captured image captured by the imaging unit 2, and detects the three-dimensional position of the calibration marker from the captured image. Specifically, the calibration marker position detection unit 51d identifies coordinates (Xc, Yc, Zc) indicating the position of the calibration marker based on the image data corresponding to the captured image, and identifies the identified coordinates (Xc , Yc, Zc) is output to the marker position accumulating unit 51e. The marker position accumulation unit 51e accumulates the coordinates (Xc, Yc, Zc) output from the calibration marker position detection unit 51d. The coordinates (Xc, Yc, Zc) correspond to the position coordinates of the marker based on the imaging coordinate system. Thus, the calibration marker position detection unit 51d corresponds to an example of the “position detection unit” in the present invention.

When a calculation trigger is input from the calibration control unit 51a, the calibration data calculation unit 51f accumulates the plurality of gaze direction coordinates (Xd, Yd) accumulated in the gaze direction coordinate accumulation unit 51c and the marker position accumulation unit 51e. The position coordinates (Xc, Yc, Zc) of the plurality of markers are read out. Then, the calibration data calculation unit 51f converts the imaging coordinate system into the display coordinate system based on the read plurality of gaze direction coordinates (Xd, Yd) and the plurality of position coordinates (Xc, Yc, Zc). Calibration data M is calculated. When the calibration data calculation unit 51f finishes calculating the calibration data M, the calibration data calculation unit 51f outputs a calculation end signal indicating that to the calibration control unit 51a. Thus, the calibration data calculation unit 51f corresponds to an example of “calibration means” in the present invention.

Next, the conversion matrix calculation unit 52 includes a marker detection unit 52a and an Rmc calculation unit 52b. The conversion matrix calculation unit 52 calculates a conversion matrix Rmc for converting from the coordinate system in the marker (hereinafter referred to as “marker coordinate system”) to the imaging coordinate system.

The marker detection unit 52a detects the position and size of the marker in the captured image captured by the imaging unit 2.

The Rmc calculation unit 52b calculates a conversion matrix Rmc for converting from the marker coordinate system to the imaging coordinate system based on the position and size of the marker detected by the marker detection unit 52a. The Rmc calculation unit 52 b outputs the calculated conversion matrix Rmc to the rendering unit 53. By updating the transformation matrix Rmc, CG is displayed so as to follow the marker.

Next, the rendering unit 53 includes a CG data storage unit 53a, a marker to imaging coordinate conversion unit 53b, and an imaging to display conversion unit 53c. The rendering unit 53 performs rendering for the CG to be displayed.

The CG data storage unit 53a is a storage unit that stores CG data (CG data) to be displayed. The CG data storage unit 53a stores CG data defined by the marker coordinate system. The CG data stored in the CG data storage unit 53a is three-dimensional (3D) data. Hereinafter, the CG data stored in the CG data storage unit 53a is appropriately referred to as “marker coordinate system data”.

The marker-to-imaging coordinate conversion unit 53b receives the conversion matrix Rmc from the conversion matrix calculation unit 52, and converts the CG data stored in the CG data storage unit 53a from the marker coordinate system to the imaging coordinate system based on the conversion matrix Rmc. Convert to Hereinafter, the CG data based on the coordinate system of the imaging unit 2 after being converted by the marker-to-imaging coordinate conversion unit 53b is appropriately referred to as “imaging coordinate system data”.

The imaging to display conversion unit 53c receives the calibration data M from the calibration unit 51, and converts the imaging coordinate system data (3D) input from the marker to imaging coordinate conversion unit 53b into display data based on the calibration data M. (Coordinate transformation and projection transformation). The display data is two-dimensional (2D) data based on the display coordinate system. The imaging to display conversion unit 53 c outputs display data to the selector 54.

The selector 54 selectively outputs the gaze target image data input from the calibration unit 51 and the display data input from the rendering unit 53 to the display unit 1a in response to the mode switching signal from the calibration unit 51. The selector 54 outputs the gaze target image data to the display unit 1a when the calibration process is performed, and outputs the display data to the display unit 1a when the CG is to be displayed on the HMD 100. The display unit 1a displays a gaze target image based on the gaze target image data, and displays CG based on the display data.

(Processing flow)
Next, a processing flow according to the first embodiment will be described with reference to FIGS.

FIG. 6 is a flowchart showing the overall processing of the HMD 100.

First, in step S10, calibration processing is performed. Details of the calibration process will be described later. Next, in step S <b> 20, a captured image of the real environment is acquired by the imaging unit 2. That is, the HMD 100 acquires a captured image of the real environment by capturing the real environment with the imaging unit 2.

Next, in step S30, the conversion matrix calculation unit 52 detects a marker as a target for which additional information such as CG is to be presented, and calculates the conversion matrix Rmc. That is, the marker detection unit 52a of the transformation matrix calculation unit 52 detects the position, orientation (direction), and size of the marker provided in the real environment based on the captured image of the real environment acquired by the imaging unit 2. Based on the detected position, orientation (direction), and size of the marker, the Rmc calculation unit 52b of the conversion matrix calculation unit 52 calculates the conversion matrix Rmc.

Next, in step S40, a drawing process for generating display data of CG to be displayed is performed. In the drawing process, first, the marker coordinate system data stored in the CG data storage unit 53a is converted into imaging coordinate system data by the marker to imaging coordinate conversion unit 53b based on the conversion matrix Rmc. Next, the imaging coordinate system data is converted into display data based on the calibration data M by the imaging to display conversion unit 53c. The display data generated in this way is input to the display unit 1 a via the selector 54.

Next, in step S50, the HMD 100 displays a CG based on the display data. In step S60, it is determined whether or not to end the display of the CG by the HMD 100. If it is determined to end (step S60: Yes), the display of the CG is ended. If it is determined not to end (step S60: No), the process according to step S10 is performed again.

FIG. 7 is a flowchart showing the calibration process in step S10 described above.

First, in step S101, the user calibrates the detection of the line-of-sight direction by performing the work of watching the display of the HMD 100 a plurality of times.

Next, in step S102, the calibration unit 51 designates a marker to be watched by the user. Specifically, a gaze target image corresponding to the marker selected by the gaze target selection unit 51b of the calibration unit 51 is displayed. In step S103, it is determined whether or not the button 8 has been pressed. That is, it is determined whether or not the user has gazed at the designated marker.

When the button 8 is pressed (step S103: Yes), the process proceeds to step S104. On the other hand, when the button 8 is not pressed (step S103: No), the determination according to step S103 is performed again. That is, the determination in step S103 is repeated until the button 8 is pressed.

In step S104, the gaze direction detection unit 7 detects the gaze direction of the user. Specifically, the line-of-sight direction detection unit 7 obtains line-of-sight direction coordinates (Xd, Yd), which are coordinates of the display coordinate system, corresponding to the intersection of the user's line-of-sight direction and the display surface 4.

Next, in step S105, the captured image of the real environment is acquired by the imaging unit 2. In step S106, the calibration marker position detection unit 51d of the calibration unit 51 detects the three-dimensional position of the calibration marker from the captured image. Specifically, the calibration marker position detection unit 51d obtains coordinates (Xc, Yc, Zc), which are the position coordinates of the marker with reference to the imaging coordinate system, based on the image data corresponding to the captured image.

Next, in step S107, the gaze direction coordinates (Xd, Yd) obtained in step S104 and the marker position coordinates (Xc, Yc, Zc) obtained in step S106 are accumulated. Specifically, the gaze direction coordinates (Xd, Yd) are accumulated in the gaze direction coordinate accumulation unit 51c, and the marker position coordinates (Xc, Yc, Zc) are accumulated in the marker position accumulation unit 51e.

Next, in step S108, it is determined whether or not the processing in steps S102 to S107 has been performed a predetermined number of times. The predetermined number of times used for the determination is determined according to, for example, the accuracy of the calibration process.

When the processing of steps S102 to S107 is performed a predetermined number of times (step S108: Yes), the calibration data M is calculated by the calibration data calculation unit 51f of the calibration unit 51 (step S109). Specifically, the calibration data calculation unit 51f includes a plurality of gaze direction coordinates (Xd, Yd) accumulated in the gaze direction coordinate accumulation unit 51c, and a plurality of marker position coordinates accumulated in the marker position accumulation unit 51e ( Calibration data M is calculated based on Xc, Yc, Zc). Then, the process ends. On the other hand, if the processes in steps S102 to S107 have not been performed a predetermined number of times (step S108: No), the process related to step S102 is performed again.

Note that since the human gaze direction tends to be unstable even when gazing, the calibration data M using only the gaze direction at the timing when the button 8 is pressed may increase the error of the calibration data M. . Therefore, it is preferable to obtain the gaze direction data for about 1 second before and after the timing when the button 8 is pressed, and perform the averaging process and the histogram process on the data to determine the gaze direction. By doing so, it is possible to reduce the error of the calibration data M.

(Action / Effect)
Here, the operation and effect of the first embodiment described above will be described in comparison with the technique described in Non-Patent Document 1 described above. In the technology described in Non-Patent Document 1, it is necessary for the user to repeat the work of aligning the displayed cross and the actual marker position and informing the calculator using the buttons during calibration. Therefore, the user needs to move the display device and the marker and repeat the position adjustment between the display and the marker, which tends to be troublesome and time consuming. In addition, when the display device is for both eyes, it is necessary to perform calibration separately for the left and right eyes.

On the other hand, in the first embodiment, the gaze direction detection function is used, so that the calibration can be performed only by gazing at the cross display or the actual marker. The work of gazing as in the first embodiment is more burdensome on the user during calibration than the work of moving the display device or marker and aligning the position of the cross display and the marker as described in Non-Patent Document 1. Can be said to be small, meaning that it takes less time and effort. When the HMD 100 is for both eyes, the calibration processing can be performed simultaneously on the left and right sides by providing the line-of-sight direction detection units 7 according to the first embodiment on the left and right sides. As described above, according to the first embodiment, it is possible to reduce the burden on the user at the time of calibration as compared with the technique described in Non-Patent Document 1.

In addition, unlike the techniques described in Patent Documents 1 and 3 described above, the first embodiment is calibrated to appropriately respond to changes in the settings and positions of the imaging unit 2 and the HMD 100, as well as changes in the position of the eyes. Can do.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the first embodiment, calibration is performed using a marker, but in the second embodiment, calibration is performed using natural feature points existing in the real environment instead of using artificial feature points such as markers. That is, in the second embodiment, the calibration is performed by causing the user to sequentially watch the surrounding natural feature points. Specifically, in the second embodiment, the surrounding real environment is imaged by the imaging unit 2, and the imaging direction of the imaging unit 2 (hereinafter referred to as “optimal imaging”) including the natural feature point optimal for the calibration process. Calibration is performed using “direction” or “optimal direction”.

Here, the optimal shooting direction will be specifically described. In order to accurately obtain the calibration data M, it is desirable that the list of natural feature points to be watched by the user is scattered in the horizontal direction, the vertical direction, and the depth direction with respect to the imaging unit 2. Therefore, it can be said that the shooting direction of the imaging unit 2 is preferably a direction in which many natural feature points are scattered and exist. In addition, it is desirable that the natural feature point list is easy to detect a three-dimensional position. In order to detect the three-dimensional position of a natural feature point, it is necessary to accurately match the natural feature point in an image taken from a plurality of viewpoints. For example, if there are similar patterns such as wall tiles, matching errors tend to increase between images. In addition, for example, if the object is a moving object such as a leaf, the three-dimensional position is different between images having different photographing timings, so that it can be said that the three-dimensional position cannot be obtained accurately. From the above, it is desirable to use a shooting direction in which a plurality of natural feature points that are not similar to each other and whose positions do not move are scattered as the optimum shooting direction.

FIG. 8 is a block diagram showing the configuration of the control unit 5x according to the second embodiment. In addition, the component which attached | subjected the code | symbol same as the control part 5 shown in FIG. 5 shall have the same meaning, and the description is abbreviate | omitted. Further, components and processes not particularly described here are the same as those in the first embodiment.

As shown in FIG. 8, the control unit 5x has a calibration unit 51x instead of the calibration unit 51. The calibration unit 51x includes a calibration control unit 51ax, a gaze target selection unit 51bx, a gaze direction coordinate accumulation unit 51c, a feature point position detection unit 51dx, a feature point position accumulation unit 51ex, a calibration data calculation unit 51f, and an optimum A direction determination unit 51g.

The optimal direction determination unit 51g receives the captured image captured by the imaging unit 2, and determines whether the captured image includes the optimal capturing direction for the calibration process. Specifically, the optimal direction determination unit 51g performs image analysis on a captured image of the surroundings to detect a shooting direction in which a plurality of natural feature points that are not similar to each other and do not move are scattered. To do. When the optimal shooting direction is detected from the captured image, the optimal direction determination unit 51g outputs an optimal direction detection signal indicating that the optimal shooting direction is included in the captured image to the calibration control unit 51ax. Thus, the optimum direction determination unit 51g corresponds to an example of the “determination unit” in the present invention.

When the optimal direction detection signal is input from the optimal direction determination unit 51g, the calibration control unit 51ax updates the gaze target image that designates the natural feature point to be watched by the user in accordance with the gaze completion signal from the button 8. The display update signal is output to the gaze target selection unit 51bx.

The gaze target selection unit 51bx, when the display update signal is input from the calibration control unit 51ax, causes the user to gaze among the natural feature points included in the photographed image (image corresponding to the optimum photographing direction). A point is selected, and gaze target image data corresponding to the natural feature point is generated. For example, the gaze target selection unit 51bx highlights the natural feature point to be watched by the user in the reduced image (in one example, an image in which the natural feature point to be watched is displayed in a specific color, , An image in which the natural feature points are circled. Then, the gaze target selection unit 51bx outputs the generated gaze target image data to the selector 54. The gaze target selection unit 51bx corresponds to an example of the “designating unit” in the present invention.

The feature point position detection unit 51dx receives the captured image captured by the imaging unit 2, and detects the three-dimensional position of the natural feature point that the user gazes from the captured image. Specifically, the feature point position detection unit 51dx, based on image data corresponding to the captured image, coordinates (Xc ′, Yc ′, Zc ′) indicating the position of the natural feature point selected by the gaze target selection unit 51bx. ) Is specified, and the specified position coordinates (Xc ′, Yc ′, Zc ′) are output to the feature point position storage unit 51ex. The feature point position accumulation unit 51ex accumulates the position coordinates (Xc ′, Yc ′, Zc ′) output from the feature point position detection unit 51dx. Note that the position coordinates (Xc ′, Yc ′, Zc ′) correspond to the position coordinates of the natural feature point based on the imaging coordinate system. Thus, the feature point position detection unit 51dx corresponds to an example of the “position detection unit” in the present invention.

When a calculation trigger is input from the calibration control unit 51ax, the calibration data calculation unit 51fx accumulates the plurality of gaze direction coordinates (Xd, Yd) accumulated in the gaze direction coordinate accumulation unit 51c and the feature point position accumulation unit 51ex. The position coordinates (Xc ′, Yc ′, Zc ′) of the plurality of natural feature points are read out. Then, the calibration data calculation unit 51fx converts the imaging coordinate system to the display coordinate system based on the read plurality of gaze direction coordinates (Xd, Yd) and the plurality of position coordinates (Xc ′, Yc ′, Zc ′). Calibration data M, which is a matrix for the calculation, is calculated. Thus, the calibration data calculation unit 51fx corresponds to an example of “calibration means” in the present invention.

Next, a processing flow according to the second embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing the calibration process according to the second embodiment. This calibration process is executed in step S10 described above (see FIG. 6) instead of the calibration process shown in FIG.

First, in step S111, the user calibrates the detection of the line-of-sight direction by performing the work of gazing at the display of the HMD 100 a plurality of times.

Next, in step S112, a captured image of the real environment is acquired by the imaging unit 2. Specifically, the HMD 100 acquires a captured image of the real environment in a relatively wide range obtained by capturing the real environment around the user by the imaging unit 2.

Next, in step S113, the optimum shooting direction included in the shot image is detected by the optimum direction determination unit 51g of the calibration unit 51x. Specifically, the optimum direction determination unit 51g performs image analysis on a captured image to detect a shooting direction in which a plurality of natural feature points that are not similar to each other and whose positions do not move are scattered. When the optimal shooting direction is included in the captured image (step S114: Yes), the process proceeds to step S116. On the other hand, when the optimal shooting direction is not included in the captured image (step S114: No), the process proceeds to step S115. In this case, the user is instructed to move the place (step S115), and the process related to step S111 is performed again. That is, the user is moved to another place and the surroundings are photographed again.

In step S116, the direction of the user's head is guided to the detected optimum photographing direction. For example, the direction of the user's head is guided by displaying an image of an arrow indicating the optimum shooting direction.

Next, in step S117, a natural feature point to be watched by the user is designated by the calibration unit 51x. Specifically, a gaze target image corresponding to the natural feature point selected by the gaze target selection unit 51bx of the calibration unit 51x is displayed. In step S118, it is determined whether or not the button 8 has been pressed. That is, it is determined whether or not the user has focused on the designated natural feature point.

When the button 8 is pressed (step S118: Yes), the process proceeds to step S119. On the other hand, when the button 8 is not pressed (step S118: No), the determination according to step S118 is performed again. That is, the determination in step S118 is repeated until the button 8 is pressed.

In step S119, the visual line direction detection unit 7 detects the visual line direction of the user. Specifically, the line-of-sight direction detection unit 7 obtains line-of-sight direction coordinates (Xd, Yd), which are coordinates of the display coordinate system, corresponding to the intersection of the user's line-of-sight direction and the display surface 4.

Next, in step S120, a captured image of the real environment (an image corresponding to the optimal shooting direction) is acquired by the imaging unit 2. In step S121, the feature point position detection unit 51dx of the calibration unit 51x detects the three-dimensional position of the natural feature point watched by the user from the captured image. Specifically, the feature point position detection unit 51dx uses the position coordinates (Xc ′, Yc ′, Zc ′) of the natural feature point selected by the gaze target selection unit 51bx based on the image data corresponding to the captured image. Ask.

Next, in step S122, the gaze direction coordinates (Xd, Yd) obtained in step S119 and the natural feature point position coordinates (Xc ′, Yc ′, Zc ′) obtained in step S121 are accumulated. . Specifically, the gaze direction coordinates (Xd, Yd) are accumulated in the gaze direction coordinate accumulation unit 51c, and the position coordinates (Xc ′, Yc ′, Zc ′) of the natural feature points are accumulated in the feature point position accumulation unit 51ex. The

Next, in step S123, it is determined whether or not the processes in steps S117 to S122 have been performed a predetermined number of times. The predetermined number of times used for the determination is determined according to, for example, the accuracy of the calibration process.

When the processes of steps S117 to S122 are performed a predetermined number of times (step S123: Yes), the calibration data M is calculated by the calibration data calculation unit 51fx of the calibration unit 51x (step S124). Specifically, the calibration data calculation unit 51fx includes a plurality of gaze direction coordinates (Xd, Yd) accumulated in the gaze direction coordinate accumulation unit 51c and a plurality of natural feature points accumulated in the feature point position accumulation unit 51ex. Calibration data M is calculated based on the position coordinates (Xc ′, Yc ′, Zc ′). Then, the process ends. On the other hand, when the processes of steps S117 to S122 have not been performed a predetermined number of times (step S123: No), the process according to step S117 is performed again.

According to the second embodiment described above, as with the first embodiment, it is possible to appropriately reduce the burden on the user during calibration. Further, according to the second embodiment, unlike the first embodiment, by using the natural feature points, it is possible to appropriately calibrate even in an environment where there are no markers.

[Modification]
Below, the modification suitable for said Example is demonstrated. It should be noted that the following modifications can be applied to the above-described embodiments in any combination.

(Modification 1)
In the above-described embodiment, the gaze is notified to the HMD 100 by pressing the button 8 when the user gazes. However, when the operation of pressing the button 8 is performed, the concentration power is reduced by the operation, and the direction of the line of sight is disturbed, or the operation of pressing the button 8 also affects the position of the head. Therefore, in another example, instead of notifying the completion of the gaze with the button 8, the gaze can be completed when the user performs a gaze operation for a predetermined time. That is, the line-of-sight direction can be detected at the timing when the user performs a gaze operation for a predetermined time. As a result, it is possible to suppress disturbances in the line-of-sight direction and movement of the head, reduce error factors during calibration, and improve calibration accuracy.

In yet another example, when the user blinks during the gaze operation, the gaze can be completed. That is, the line-of-sight direction can be detected at the timing when the user blinks. Also by this, it is possible to suppress the disturbance of the line-of-sight direction and the movement of the head, reduce the error factor at the time of calibration, and improve the calibration accuracy.

It should be noted that when any one of the conditions that the user has performed the gaze operation for a predetermined time and the user has blinked during the gaze operation is satisfied, the gaze operation may be treated as completion.

(Modification 2)
In the above-described embodiment, an example in which calibration for detection of the line-of-sight direction is performed manually is shown, but the calibration may be automatically performed. In that case, it is not necessary to perform the processing of steps S101 and S111 in the calibration processing shown in FIGS. Further, when a detection method that does not require calibration for the detection of the line-of-sight direction is used, it is not necessary to perform the processes of steps S101 and S111.

(Modification 3)
In the above-described embodiment, the three-dimensional position of the marker or the natural feature point is detected based on the captured image of the imaging unit 2 (camera). However, instead of the imaging unit 2, a magnetic sensor, an ultrasonic sensor, The three-dimensional position of the target (object) may be detected using a gyro sensor, an acceleration sensor, an angular velocity sensor, a GPS receiver, or a wireless communication device. In this case, the position of the target such as a marker may be fixed, or a sensor may be attached to the target to detect the position and direction of the sensor itself attached to the HMD 100 with respect to the reference position of the environment. When the imaging unit 2 (camera) is used as in the above-described embodiment, a method for detecting the position of the target and a method for estimating the position of the fixed target by detecting the position and direction of the camera. Both of these are applicable.

In addition to the marker detection as described above, the marker detection unit 52a may use an image marker or a natural feature point.

Furthermore, a plurality of detection methods may be used in combination for three-dimensional position detection. However, it is necessary to use the same sensor output in the calibration marker position detection unit 51d and the marker detection unit 52a.

(Modification 4)
The present invention is not limited to application to the HMD 100. The present invention can be applied to various see-through displays capable of realizing an optically transmissive AR. For example, it can be applied to a head-up display (HUD), a see-through display, and the like.

The present invention can be used for an optical transmission type display device such as a head mounted display.

DESCRIPTION OF SYMBOLS 1 Transmission type display part 1a Display part 2 Imaging part 3 Mounting part 4 Display surface 5 Control part 6 Near-infrared light source 7 Gaze direction detection part 51 Calibration part 52 Conversion matrix calculation part 53 Rendering part 100 Head mounted display (HMD)
200 markers

Claims (13)

1. An optically transmissive display device that displays additional information for a real environment visually recognized by a user,
   Position detecting means for detecting the position of an object existing in the real environment;
   Gaze direction detecting means for detecting a gaze direction when the user is gazeing at the object;
   Based on the position detected by the position detection unit and the line-of-sight direction detected by the line-of-sight direction detection unit, the display unit defines the first coordinate system defined by the position detection unit. A display device comprising: calibration means for calculating calibration data for conversion to the second coordinate system.
2. 2. The display device according to claim 1, wherein the line-of-sight direction detecting unit detects the line-of-sight direction when operating an input unit for inputting that the user gazes.
3. 2. The display device according to claim 1, wherein the gaze direction detection unit detects the gaze direction when the user performs a gaze operation for a predetermined time.
4. The display device according to claim 1, wherein the line-of-sight direction detection means detects the line-of-sight direction when the user blinks.
5. The position detection means detects the position based on a photographed image obtained by photographing the real environment with an image acquisition device,
   The display device according to claim 1, wherein the first coordinate system is a coordinate system based on the image acquisition device.
6. The display device according to any one of claims 1 to 5, wherein the object is used for calculation of the calibration data by the calibration means and is a marker.
7. The line-of-sight direction detection means obtains coordinates in the second coordinate system corresponding to the intersection of the line-of-sight direction and the display surface by the display device,
   7. The calibration data according to claim 1, wherein the calibration unit calculates the calibration data based on the coordinates obtained by the line-of-sight direction detection unit and the position detected by the position detection unit. The display device according to one item.
8. Further comprising designation means for designating the object to be watched by the user;
   The said gaze direction detection means detects the gaze direction when the said user is gazing at the said object designated by the said designation means, The one of Claim 1 thru | or 7 characterized by the above-mentioned. Display device.
9. The display device according to claim 8, wherein the designation unit designates the target object by displaying an image corresponding to the target object to be watched.
10. An optically transmissive head mounted display that displays additional information for a real environment visually recognized by a user,
    Position detecting means for detecting the position of an object existing in the real environment;
    Gaze direction detecting means for detecting a gaze direction when the user is gazeing at the object;
    Based on the position detected by the position detection unit and the line-of-sight direction detected by the line-of-sight direction detection unit, the display unit defines the first coordinate system defined by the position detection unit. And a calibration means for calculating calibration data for conversion to the second coordinate system.
11. A calibration method executed by an optically transmissive display device that displays additional information with respect to a real environment visually recognized by a user,
    A position detecting step for detecting a position of an object existing in the real environment;
    A line-of-sight direction detection step of detecting a line-of-sight direction when the user is directing a line of sight to the object;
    Based on the position detected by the position detection step and the line-of-sight direction detected by the line-of-sight direction detection step, the display device defines the first coordinate system defined by the position detection step. And a calibration step for calculating calibration data for conversion to the second coordinate system.
12. A calibration program that is executed by an optically transmissive display device that has a computer and displays additional information for a real environment visually recognized by the user,
    The computer,
    Position detecting means for detecting the position of an object existing in the real environment;
    Gaze direction detection means for detecting a gaze direction when the user is gazeing at the object;
    Based on the position detected by the position detection unit and the line-of-sight direction detected by the line-of-sight direction detection unit, the display unit defines the first coordinate system defined by the position detection unit. A calibration program which functions as calibration means for calculating calibration data for conversion to the second coordinate system.
13. A recording medium in which the calibration program according to claim 12 is recorded.

Images