JP2016057634A - Head-mounted display, calibration method, calibration program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted display that can be easily calibrated, and to provide a calibration method, a calibration program, and a recording medium.SOLUTION: A head-mounted display includes image capturing means and an image display device, and allows a user to view information based on an image captured by the image capturing means on the image display device. Calibration means adjusts positional relationship, in a field of view of the user, between an object under observation being observed via the image display device and presented information based on a mirror image, captured by the image capturing means, of the user wearing the head-mounted display. The head-mounted display can thus be calibrated easily.SELECTED DRAWING: Figure 2

Description

  The present invention relates to the technical field of head mounted displays.

  Conventionally, a technique related to AR (Augmented Reality) in which a virtual image is superimposed and displayed on a real space using a head mounted display or the like has been proposed. In addition, with regard to such AR, a technique for performing position / orientation alignment (calibration) between the real space and the virtual space has been proposed. For example, Patent Document 1 discloses a technique for performing calibration using a user's posture and values of sensors that measure the position and posture. Specifically, in the technique, calibration is performed by causing the user to take a posture in which the display and the marker coincide with each other. In addition to this, a technique for performing calibration using such a marker is disclosed in, for example, Patent Document 2 and Non-Patent Documents 1 to 3. Furthermore, for example, Patent Document 3 discloses a technique for performing calibration by converting the display on the display by the user operating the button so that the display on the display matches the landscape.

JP 2002-229730 A JP 2009-284175 A JP2011-27543A

Hirokazu Kato, Mark Billinghurst, Koichi Asano, Keihachiro Tachibana, "Augmented Reality System Based on Marker Tracking and Its Calibration", Transactions of the Virtual Reality Society of Japan, Vol. 4 (1999), No. 4, pp.607 -616 Mihran Tuceryan, Yakup Genc, Nassir Navab, Single point active alignment method (SPAAM) for optical see-through HMD calibration for augmented reality, Teleoperators and Virtual Environments, Vol. 11, Issue 3, pp. 259-276, June 2002, published by the MIT Press Kiyohide Sato, Hiroyuki Yamamoto, Hideyuki Tamura, alignment method of real space and virtual space by combination of camera and 3D sensor, TVRSJ VoL4 No.i, pp.295-302

  However, with the techniques described in Patent Documents 1 to 3 and Non-Patent Documents 1 to 3, when calibration is performed, a predetermined work for adjusting the positional relationship between the real space and the virtual space is imposed on the user. Therefore, it was troublesome for the user.

  Examples of the problem to be solved by the present invention are as described above. It is an object of the present invention to provide a head mounted display, a calibration method, a calibration program, and a recording medium that can be easily calibrated.

  The invention according to claim 1 is a head-mounted display having an imaging unit and an optically transmissive image display device, and taking a mirror image of a user captured by the imaging unit in a coordinate system defined by the imaging unit. The image display device is converted into a first coordinate system defined by a center position in a real image of the user's eyeball based on an image, and the first coordinate system is converted into a coordinate system defined by the image display device. Calibration means for calibrating the positional relationship in the user's field of view between the object observed via the information and the information based on the photographed image of the photographing means.

  According to a ninth aspect of the present invention, there is provided a calibration method executed by a head-mounted display having an imaging unit and an optically transmissive image display device, wherein information based on a captured image of the imaging unit is visually recognized on the image display device. The coordinate system defined by the imaging unit is converted into a first coordinate system defined by the center position in the real image of the user's eyeball based on the captured image of the mirror image of the user captured by the imaging unit; The position in the field of view of the user of the object observed through the image display device and the information based on the photographed image of the photographing means by converting the first coordinate system to a coordinate system defined by the image display device A calibration process for calibrating the relationship is provided.

  According to a tenth aspect of the present invention, there is provided a head-mounted display having a photographing unit and an optically transmissive image display device, allowing information based on a photographed image of the photographing unit to be visually recognized on the image display device, and including a computer. A calibration program to be executed, wherein a first coordinate defined by a center position in a real image of the user's eyeball based on a photographed image of a mirror image of the user photographed by the photographing means in a coordinate system defined by the photographing means The information of the object observed through the image display device and the information based on the photographed image of the photographing means by transforming the first coordinate system into a coordinate system defined by the image display device. The computer is caused to function as calibration means for calibrating the positional relationship in the user's field of view.

It is a block diagram which shows schematic structure of HMD which concerns on a present Example. It is the figure which showed roughly the whole structure of HMD which concerns on a present Example. It is a side view which shows schematic structure of the image display device of HMD which concerns on a present Example. The figure for demonstrating the problem that presentation information will be displayed on the position which shifted | deviated is shown. The figure for demonstrating the outline | summary of the calibration method which concerns on a present Example is shown. The figure for demonstrating the coordinate system used by a present Example is shown. It is a flowchart which shows the coordinate transformation method which concerns on a present Example. The relationship between the coordinates in the eye coordinate system and the projected coordinates is shown. It is a flowchart which shows the calculation method of the coordinate transformation parameter which concerns on a present Example.

  In one aspect of the present invention, a head-mounted display that includes a photographing unit and an optically transmissive image display device, and that causes information based on a photographed image of the photographing unit to be visually recognized on the image display device is photographed by the photographing unit. Calibration for adjusting the positional relationship in the user's field of view between the observation object observed through the image display device and the information based on the captured image of the mirror image of the user wearing the head mounted display Calibration means for performing

  The above-described head mounted display includes an imaging unit and an optically transmissive image display device, and allows a user to visually recognize information (presentation information) based on a captured image of the imaging unit on the image display device. The calibration unit is configured to determine the positional relationship in the user's field of view between the observation target observed through the image display device and the presentation information based on the captured image of the mirror image of the user wearing the head mounted display, which is captured by the imaging unit. adjust. As a result, the calibration is automatically performed when the user faces the mirror or the like, so that it is possible to reduce the user's trouble at the time of calibration as compared with the technique described in the prior art document described above.

  In one aspect of the head mounted display, the calibration unit can perform the calibration based on a positional relationship between the imaging unit and the user's eyeball included in the mirror image. According to this aspect, calibration can be performed by simple processing.

  In another aspect of the head-mounted display, the calibration unit performs the calibration based on a positional relationship between the imaging unit, the image display device, and the user's eyeball included in the captured image of the mirror image. Can do. According to this aspect, even when the positional relationship between the photographing unit and the image display device is not fixed, calibration can be performed with high accuracy.

  Preferably, in the above-described head mounted display, a marker is added to the photographing unit, and the calibration unit is included in the captured image of the mirror image based on the marker added to the photographing unit, The calibration can be performed.

  Preferably, a marker is added to the mirror forming the mirror image, and the calibration means performs the calibration based on the marker added to the mirror included in the captured image of the mirror image. It can be carried out. In the present specification, the “mirror” includes not only a mirror in a pure sense but also a mirror having a reflectance higher than a predetermined value (for example, glass).

  Preferably, a marker is added to the image display device, and the calibration unit performs the calibration based on the marker added to the image display device, which is included in the captured image of the mirror image. It can be carried out.

  In another aspect of the above head-mounted display, the calibration unit coordinates coordinates from a coordinate system defined by the imaging unit to a coordinate system defined by the image display device based on the captured image of the mirror image. The calibration is performed by obtaining a coordinate conversion parameter used for conversion. In this aspect, the calibration unit determines conversion from the coordinate system defined by the imaging unit to the coordinate system defined by the image display device as calibration of the head mounted display.

  In another aspect of the above head-mounted display, display control for performing control to display the information on the image display device by performing the coordinate conversion using the coordinate conversion parameter obtained by the calibration unit. Means are further provided. Thereby, the information display position on the image display device superimposed on the user's observation image that has passed through the image display device can be appropriately matched with the positional relationship between the captured image and the information based thereon.

  Preferably, in the above head mounted display, the calibration means is based on a first estimation means for estimating an optical center position and an optical axis direction in the mirror image of the photographing means from the photographed image of the mirror image, and from the photographed image of the mirror image. A second estimation unit that estimates a center position in a mirror image of the user's eyeball; and a mirror plane that generates the mirror image is estimated based on the optical center position and the optical axis direction estimated by the first estimation unit. A center position in the real image of the user's eyeball is estimated based on the third estimation means, the mirror plane estimated by the third estimation means, and the center position estimated by the second estimation means. 4 estimation means, and based on the center position estimated by the fourth estimation means, the coordinates of the coordinate system defined by the imaging means are determined by the user. Based on the first parameter calculating means for obtaining a first parameter for converting into coordinates of a coordinate system defined by a sphere as one of the coordinate conversion parameters, and the center position estimated by the fourth estimating means. A second parameter calculating means for obtaining, as one of the coordinate conversion parameters, a second parameter for projecting a coordinate of a coordinate system defined by the user's eyeball onto the image display device; Based on the center position estimated by the means, the coordinates of the coordinate system defined by the user's eyeball, the coordinates defined by the head mounted display for displaying the information on the image display device And third parameter calculation means for obtaining a third parameter for conversion to system coordinates as one of the coordinate conversion parameters. That. Thereby, a coordinate conversion parameter can be calculated | required appropriately. Note that “real image” means an image that is a generation source of a mirror image.

  Preferably, in the above head mounted display, the calibration means is estimated by a fifth estimation means for estimating a position and a direction in the mirror image of the image display device from the captured image of the mirror image, and the third estimation means. And 6th estimating means for estimating the position and direction in the real image of the image display device based on the mirror plane and the position and direction estimated by the fifth estimating means, The two-parameter calculating unit obtains the second parameter based on the center position estimated by the fourth estimating unit and the position and the direction estimated by the sixth estimating unit, and calculates the third parameter. The means includes the center position estimated by the fourth estimation means and the position estimated by the sixth estimation means. Based on the fine said direction, determining the third parameter. In this case, the position and direction of the image display device are estimated, and coordinate conversion parameters are obtained based on the position and the direction. Thereby, the coordinate conversion parameter can be obtained with high accuracy.

  In the above head mounted display, preferably, the center position of the user's eyeball is a pupil center position or an optical center position. When the pupil center position is employed, the position of the eyeball can be easily obtained from the captured image. When the optical center position is adopted, the coordinate conversion parameter can be obtained with high accuracy.

In a preferred example, a stereo camera can be used as the photographing means.
In a preferred example, an image taken in a state where the user and the mirror are generally facing each other can be used as the captured image of the mirror image.

  In another aspect of the present invention, there is provided a calibration method executed by a head-mounted display having an imaging unit and an optically transmissive image display device, and causing information based on a captured image of the imaging unit to be visually recognized on the image display device. The positional relationship in the user's field of view between the observation object and the information observed through the image display device based on the captured image of the mirror image of the user wearing the head-mounted display, which is imaged by the imaging means A calibration process for performing calibration to adjust the.

  According to still another aspect of the present invention, there is provided a head-mounted display having a photographing unit and an optically transmissive image display device, allowing information based on a photographed image of the photographing unit to be visually recognized on the image display device, and including a computer. The calibration program to be executed is based on the captured image of the mirror image of the user wearing the head-mounted display, which is captured by the imaging unit, and the observation target and the information that are observed through the image display device. The computer is caused to function as calibration means for performing calibration for adjusting the positional relationship in the user's field of view.

  The above calibration program can be suitably handled in a state where it is recorded on a recording medium.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[Device configuration]
First, the configuration of the head mounted display (hereinafter referred to as “HMD” where appropriate) according to the present embodiment will be described.

  FIG. 1 is a block diagram illustrating a schematic configuration of the HMD 1 according to the present embodiment. As shown in FIG. 1, the HMD 1 mainly includes a control unit 11, a camera 12, and an optically transmissive image display device 13. The HMD 1 is configured in, for example, a glasses type, and is configured to be wearable on the user's head. In addition, the HMD 1 uses the image display device 13 to enable a user to view an object existing in the outside world (for example, a person, an object, an article, a landscape, etc. in a real space), and an image of the object (real AR (augmented reality) is realized by displaying a virtual image superimposed on the image.

  The camera (photographing means) 12 photographs the front of the HMD 1 and generates a photographed image. The camera 12 is a pinhole camera whose internal parameters are known, or a camera whose lens distortion is corrected and whose internal parameters are known.

  The image display device 13 causes the user to visually recognize information corresponding to the captured image (hereinafter referred to as “presentation information”) overlapping the image of the object.

  The control unit 11 includes a CPU, a RAM, a ROM, and the like (not shown) and performs overall control of the HMD 1. Specifically, the control unit 11 acquires a captured image from the camera 12 and performs control for causing the user to visually recognize the presentation information on the image display device 13. In this case, the control unit 11 analyzes the captured image to determine the presentation information to be presented, determines the presentation position where the presentation information is presented, and performs control to display the presentation information at the presentation position. .

  The control unit 11 corresponds to an example of “calibration means” and “display control means” in the present invention. Specifically, the control unit 11 includes “first estimation unit”, “second estimation unit”, “third estimation unit”, “fourth estimation unit”, “fifth estimation unit”, “sixth estimation unit”. "," First parameter calculation means "," second parameter calculation means ", and" third parameter calculation means ". Details of these will be described later.

  FIG. 2 is a diagram schematically illustrating the overall configuration of the HMD 1 according to the present embodiment. As shown in FIG. 2, the HMD 1 is configured as a glasses, and the camera 12 and the image display device 13 are fixed to a frame 14. In addition, illustration of the control part 11 is abbreviate | omitted in FIG. In FIG. 2, only the image display device 13 for the left eye is shown, and the illustration of the image display device 13 for the right eye is omitted. Note that the image display device 13 is not limited to be applied to both eyes, and the image display device 13 may be applied to only one eye. That is, an image may be displayed only on one eye.

  As shown in FIG. 2, a marker 12a and a marker 13a are added to the camera 12 and the image display device 13, respectively. These markers 12a and 13a are used in a calibration method described later. It is assumed that the control unit 11 knows the size and shape of the markers 12a and 13a. In addition, it is assumed that the control unit 11 also grasps the positional relationship between the optical center of the camera 12 and the marker 12a and the positional relationship between the image display device 13 and the marker 13a.

  In addition, as shown in FIG. 2, it is not limited to providing the four markers 12a and 13a, and it is not limited to providing the markers 12a and 13a in the positions as shown in FIG. Further, the HMD 1 is not limited to being configured as a glasses, but may be configured as a helmet.

  FIG. 3 is a side view illustrating a schematic configuration of the image display device according to the present embodiment. The image displayed on the LCD 13b forms a virtual image 40 (hereinafter sometimes referred to as “virtual screen” or simply “screen”) at a position h from the user's eyes using the lens 13c and the half mirror 13d. . Although there is a possibility that the virtual image 40 may be distorted by the lens 13c, the control unit 11 corrects the image. It is assumed that the control unit 11 knows the positional relationship between the image display device 13 and the virtual screen 40. The image display device of the present embodiment is not particularly limited. You may have the light-guide plate which consists of optical glass, a plastic material, etc. Further, for example, OELD or the like may be used instead of the LCD. Further, a hologram optical element or the like may be used instead of the half mirror. Two or more lenses may be used, and the lens may not be used.

[Outline of calibration method]
Next, an outline of the calibration method according to the present embodiment will be described.
As described above, when realizing the AR (augmented reality) for presenting the presentation information in the real space, the control unit 11 analyzes the captured image of the camera 12 and determines the presentation information and the presentation position based on the analysis. The image is displayed on the image display device 13 and the user visually recognizes the virtual screen 40. By the way, the viewpoint (where the user is looking from), the viewing angle (viewing angle), and the visual axis tend to be different between the landscape actually viewed by the user and the captured image of the camera 12. Therefore, when the presentation information is displayed at the presentation position determined by analyzing the captured image of the camera 12, the presentation information may be displayed at a position shifted in the real space that the user is looking at.

  FIG. 4 is a diagram for explaining the problem that the presentation information is displayed at a shifted position. FIG. 4A shows an example of an image actually viewed by a user wearing the HMD 1. Specifically, FIG. 4A shows an example in which presentation information (characters such as facility A and facility B, a circle surrounding a building, and a leader line) is displayed at an appropriate position in the user's field of view. Is shown. FIG. 4B shows an example of an image captured by the camera 12. Specifically, FIG. 4B shows an example in which the captured image of the camera 12 is shifted from the scenery that the user actually sees. FIG. 4C shows another example of an image visually recognized by the user wearing the HMD 1. Specifically, FIG. 4C illustrates an example in which the presentation information is displayed at the presentation position determined by analyzing the captured image as illustrated in FIG. Thus, it can be seen that the presentation information is not displayed at an appropriate position in the user's visual field. That is, it can be seen that the presentation information is displayed at a position shifted from the position where the presentation information is to be displayed.

  In the present embodiment, the HMD 1 is calibrated in order to solve such a problem that the presentation information is displayed at a shifted position. Here, “calibration” corresponds to determining conversion from a coordinate system (camera coordinate system) defined by the camera 12 to a coordinate system (screen coordinate system) defined by the virtual screen 40. . Specifically, in this embodiment, a mirror image of the user wearing the HMD 1 is captured by the camera 12, and based on the captured image, the user's observation target and presentation information observed via the image display device 13 are displayed. Adjust the positional relationship in the field of view.

  FIG. 5 is a diagram for explaining the outline of the calibration method according to the present embodiment. In this embodiment, when performing calibration, a mirror 30 as shown in FIG. 5A is used together with the HMD 1. Then, as shown in FIG. 5B, a mirror image of the user wearing the HMD 1 formed on the mirror 30 is taken by the camera 12. When performing such shooting, the mirror 30 is arranged so that the reflecting surface faces the user, faces the user's face substantially, and the user's head is reflected. The mirror 30 need not be a special one, and may be a familiar one such as a hand mirror.

  The control unit 11 estimates the positional relationship between the camera 12, the image display device 13, and the eyes 20 based on a captured image of a mirror image of the user wearing the HMD 1. Specifically, the control unit 11 estimates the optical center position and the optical axis direction of the camera 12 by detecting the marker 12a added to the camera 12 from the captured image. In this case, the control unit 11 estimates the optical center position and the optical axis direction of the camera 12 based on the size and shape of the marker 12a and the positional relationship between the optical center of the camera 12 and the marker 12a. In addition, the control unit 11 estimates the position and direction of the image display device 13 by detecting the marker 13 a added to the image display device 13 from the captured image. In this case, the control unit 11 estimates the position and direction of the image display device 13 based on the size and shape of the marker 13a and the positional relationship between the image display device 13 and the marker 13a.

  Further, the control unit 11 estimates the three-dimensional position of the eye 20 by detecting the black eye portion of the user's eye 20 from the captured image (assuming the size of the black eye is known). In this case, the control unit 11 estimates the pupil center position as the position of the eye 20. Since the center position of the pupil changes depending on the eye movement, when taking a picture, the user should pay attention so that the visual axis (viewing direction) faces the mirror 30 substantially. Do not let the.

  The controller 11 calibrates the HMD 1 based on the optical center position and optical axis direction of the camera 12, the position and direction of the image display device 13, and the pupil center position estimated from the captured image in this way. Specifically, as the calibration, the control unit 11 performs a process of obtaining parameters (coordinate conversion parameters) necessary in the process of coordinate conversion from the camera coordinate system to the screen coordinate system. Details of the calculation method of the coordinate conversion parameter will be described later.

[Definition of coordinate system]
Next, a coordinate system used in the present embodiment will be described with reference to FIG. As shown in FIG. 6, in this embodiment, a camera coordinate system, an eye coordinate system, an image display device coordinate system, and a screen coordinate system are used. Such a coordinate system is used for the coordinate conversion from the camera coordinate system to the screen coordinate system when the control unit 11 displays the presentation information on the image display device 13 based on the captured image.

The camera coordinate system is a three-dimensional coordinate system for the optical center of the camera 12 as the origin O _c. The camera coordinate system, the optical axis of the camera 12 and Z _c axis, respectively a direction parallel to the x-axis and y axis of the photographic image and X _c-axis and Y _c axis. The eye coordinate system corresponds to the visual perception of the user of the HMD 1 and is a three-dimensional coordinate system with the user's pupil center as the origin O _e . Eye coordinate system, the plane mirror 30 is present (hereinafter, referred to as "mirror plane".) The normal direction of the Z _e axis, the X _c-axis and a respective direction parallel to the Y _c-axis of the camera coordinate system X and _e axis and _{Y e} axis.

The screen coordinate system is image coordinates of the virtual screen 40. The image display device coordinate system is a coordinate system defined by the control unit 11 in order to display the presentation information on the virtual screen 40, and is a three-dimensional coordinate system having the virtual screen 40 as a projection plane. In the image display device coordinate system, the Z _h axis intersects with the center of the virtual screen 40, and the directions parallel to the x axis and the y axis of the screen coordinate system are the X _h axis and the Y _h axis, respectively.

[Coordinate conversion method]
Next, a coordinate conversion method according to the present embodiment will be described. Here, an outline of coordinate transformation will be briefly described. First, the control unit 11 analyzes a captured image of the camera 12 and obtains coordinates for presenting presentation information (hereinafter referred to as “presentation coordinates”). Next, the control unit 11 converts the presentation coordinates in the camera coordinate system into the eye coordinate system, and obtains projection coordinates obtained by projecting the converted coordinates onto the virtual screen 40. Next, the control unit 11 converts the projected coordinates into the image display device coordinate system, and perspective-projects the converted coordinates onto the screen coordinate system.

  FIG. 7 is a flowchart illustrating the coordinate conversion method according to the present embodiment. This flow is repeatedly executed by the control unit 11 of the HMD 1.

First, in step S101, the control unit 11 analyzes a captured image of the camera 12, and based on the analysis result, the presentation information to be presented and the presentation coordinates L _c (presented in the camera coordinate system) for presenting the presentation information. Shall be determined). Specifically, the control unit 11 detects a target object on which the presentation information is to be presented, using a method such as AR marker recognition or specific object recognition, and calculates the position and orientation of the target object. By doing this, the presentation coordinate L _c for the presentation information is determined. Then, the process proceeds to step S102.

In step S102, the control unit 11 uses the following equation (1), the presentation coordinates L _c of the camera coordinate system are converted into coordinates L _e of the eye coordinate system.

式（１）中の「Ｒ_１」及び「ｔ_１」は、それぞれ、カメラ座標系から目座標系へ変換するための回転行列及び並進ベクトルである。また、「Ｒ_１」及び「ｔ_１」は、上記した座標変換パラメータの１つである。以上のステップＳ１０２の後、処理はステップＳ１０３に進む。

“R ₁ ” and “t ₁ ” in Equation (1) are a rotation matrix and a translation vector for converting from the camera coordinate system to the eye coordinate system, respectively. “R ₁ ” and “t ₁ ” are _{one of} the coordinate conversion parameters described above. After step S102 described above, the process proceeds to step S103.

In step S < _b > 103, the control unit 11 calculates a projected coordinate M _e (which is defined in the eye coordinate system) obtained by projecting the coordinate Le in the eye coordinate system onto the virtual screen 40. Figure 8 shows the relationship between the coordinates L _e and the projection coordinates M _e. As can be seen from FIG. 8, that determine the projected coordinates M _e has the same meaning as determining the intersection of the plane and the straight line OeLe virtual screen 40. When the unit normal vector in the plane of the virtual screen 40 defined by the eye coordinate system is “n _e ” and the distance between the plane of the virtual screen 40 and the origin O _e of the eye coordinate system is “h”, the control unit 11, using the following equation (2) and (3), to convert the coordinates _{L e} to the projection coordinates _{M e.} In addition, a well-known method can be used for the method of calculating | requiring the intersection of a plane and a straight line.

具体的には、制御部１１は、式（３）に「Ｌ_ｅ」、「ｎ_ｅ」及び「ｈ」を代入することで「α」を求め、この「α」を式（２）に代入することで、座標Ｌ_ｅから投影座標Ｍ_ｅを求める。なお、「ｎ_ｅ」及び「ｈ」は、上記した座標変換パラメータの１つである。以上のステップＳ１０３の後、処理はステップＳ１０４に進む。

Specifically, the control unit 11 obtains “α” by substituting “L _e ”, “n _e ”, and “h” into Equation (3), and substitutes this “α” into Equation (2). by obtains the projected coordinates _{M e} from the coordinates _{L e.} “N _e ” and “h” are one of the coordinate conversion parameters described above. After step S103, the process proceeds to step S104.

In step S104, the control unit 11 uses the following equation (4), the projected coordinates M _e of the eye coordinate system into a coordinate M _h in the image display device coordinate system.

式（４）中の「Ｒ_２」及び「ｔ_２」は、それぞれ、目座標系から画像表示デバイス座標系へ変換するための回転行列及び並進ベクトルである。また、「Ｒ_２」及び「ｔ_２」は、上記した座標変換パラメータの１つである。以上のステップＳ１０４の後、処理はステップＳ１０５に進む。

“R ₂ ” and “t ₂ ” in Expression (4) are a rotation matrix and a translation vector for converting from the eye coordinate system to the image display device coordinate system, respectively. “R ₂ ” and “t ₂ ” are one of the coordinate conversion parameters described above. After step S104 described above, the process proceeds to step S105.

In step S <b> 105, the control unit 11 perspectively projects the coordinate M _h in the image display device coordinate system to the coordinate m _h in the screen coordinate system using the following equation (5).

式（５）より得られる座標ｍ_ｈが、仮想スクリーン４０の表示座標である。式（５）において、「ｗｉｄｔｈ」及び「ｈｅｉｇｈｔ」は仮想スクリーン４０の画像サイズであり、「ｋ_ｘ」及び「ｋ_ｙ」はＬＣＤ１３ｂのドットピッチとＬＣＤ１３ｂから仮想スクリーン４０までの光路で決まる係数である。以上のステップＳ１０５の後、処理は終了する。

The coordinates m _h obtained from the equation (5) are the display coordinates of the virtual screen 40. In Expression (5), “width” and “height” are the image sizes of the virtual screen 40, and “k _x ” and “k _y ” are coefficients determined by the dot pitch of the LCD 13 b and the optical path from the LCD 13 b to the virtual screen 40. is there. After the above step S105, the process ends.

In the above, on the assumption that the plane of the virtual screen 40 is inclined with respect to the eye's line of sight (the clogging th coordinate system Z _h axis Z _e axis and the image display device coordinate system is not parallel), An example of coordinate transformation is given. In another example, assuming that the eye gaze to the plane of the virtual screen 40 are orthogonal (and the Z _h axis Z _e axis of clogging th coordinate system and the image display device coordinate system is parallel), Coordinate transformation can be performed. In this case, perspective projection can be performed directly from the eye coordinate system to the screen coordinate system. That is, it is not necessary to perform the process of step S104.

[Calculation method of coordinate transformation parameters]
Next, a method for calculating coordinate conversion parameters in the present embodiment will be described. In this embodiment, the coordinate conversion parameter is calculated as the calibration of the HMD 1. As described above, the coordinate transformation parameters are the rotation matrices R ₁ and R ₂ , the translation vectors t ₁ and t ₂ , the unit normal vector n _e of the plane of the virtual screen 40, and the plane and eye coordinate system of the virtual screen 40. which is a distance h between the origin _{O e.} “R ₁ ” and “t ₁ ” are parameters for converting from the camera coordinate system to the eye coordinate system, and are hereinafter collectively referred to as “first parameters”. “N _e ” and “h” are parameters for projecting coordinates in the eye coordinate system onto the plane of the virtual screen 40, and hereinafter, these are collectively referred to as “second parameters”. “R ₂ ” and “t ₂ ” are parameters for converting from the eye coordinate system to the image display device coordinate system. Hereinafter, these are collectively referred to as “third parameter”.

Here, an outline of a method for calculating the coordinate conversion parameter will be briefly described. First, the control unit 11 acquires a captured image of a mirror image of a user wearing the HMD 1 captured by the camera 12. Next, from the captured image, the control unit 11 includes a marker 12a added to the camera 12 (hereinafter referred to as “camera marker” as appropriate), a user's eyes 20, and a marker 13a added to the image display device 13. (Hereinafter referred to as “screen marker” as appropriate) is detected, and the position and direction of the mirror image in the camera coordinate system are estimated for each of the camera 12, the eye 20, and the image display device 13. Next, the control unit 11 estimates the mirror plane based on the position and direction of the mirror image of the camera 12, and for each of the eyes 20 and the image display device 13, the real image in the camera coordinate system (becomes the generation source of the mirror image). The position and direction of the image are estimated. The control unit 11 knows the positional relationship between the image display device 13 and the virtual screen 40 and estimates the position and direction of the virtual screen 40 in the camera coordinate system. Since the position / direction relationship between the camera 12, the eye 20, and the virtual screen 40 can be understood from the above estimation, the control unit 11 determines the coordinate conversion parameter (specifically, the first conversion parameter) based on the position / direction relationship. Parameters R ₁ , t ₁ , second parameters n _e , h, and third parameters R ₂ , t ₂ ).

  FIG. 9 is a flowchart illustrating a method for calculating coordinate conversion parameters according to the present embodiment. This flow is executed by the control unit 11 of the HMD 1. For example, this flow is executed when the HMD 1 is attached to the user.

  First, in step S <b> 201, the control unit 11 acquires a captured image of a mirror image of a user wearing the HMD 1 captured by the camera 12. For example, the control unit 11 reflects the state in which the HMD 1 is mounted on the mirror 30 and notifies the user that this state should be captured by the camera 12, thereby acquiring a captured image. Thereafter, the process proceeds to steps S202, S203, and S204.

  In step S202, the control unit 11 detects the position of the camera marker on the image from the captured image. In step S203, the control unit 11 detects the position of the user's eye 20 on the image from the captured image. In step S204, the control unit 11 detects the position of the screen marker on the image from the captured image. After the above steps S202, S203, and S204, the process proceeds to steps S205, S206, and S207.

  In step S205, the control unit 11 performs image analysis at the position detected in step S202, and uses the position and orientation of the camera marker, and the optical center position and optical axis direction of the mirror image of the camera 12 in the camera coordinate system. Is estimated. Thereafter, the process proceeds to step S208. In step S206, the control unit 11 performs image analysis at the position detected in step S203, and estimates the pupil center position of the mirror image of the eye 20 in the camera coordinate system using the position and size of the eye 20. . In step S207, the control unit 11 performs image analysis at the position detected in step S204, and estimates the position and direction of the mirror image of the image display device 13 in the camera coordinate system using the position and orientation of the screen marker. To do.

In step S208, the control unit 11 estimates the mirror plane based on the optical center position and optical axis direction of the mirror image of the camera 12 estimated in step S205. Specifically, the control unit 11 estimates the unit normal vector of the mirror plane and the distance from the origin O _{c in} the camera coordinate system. Mirror plane intersects at the midpoint of the optical center position and the origin O _c of the mirror image of the camera 12. Also, the normal of the mirror plane, the optical axis of the mirror image of the camera 12 is a direction rotated 1/2 to Z _c-axis direction. After the above step S208, the process proceeds to steps S209 and S210.

  In step S209, the control unit 11 estimates the pupil center position of the real image of the eye 20 in the camera coordinate system. Specifically, the control unit 11 estimates the pupil center position of the real image of the eye 20 by folding back the pupil center position of the mirror image of the eye 20 estimated in step S206 at the mirror plane estimated in step S208. . In step S210, the control unit 11 estimates the center position and direction of the virtual screen 40 in the camera coordinate system. Specifically, the control unit 11 folds the position and direction of the mirror image of the image display device 13 estimated in step S207 on the mirror plane estimated in step S208, so that the position of the real image of the image display device 13 and The direction is estimated, and the center position and direction of the virtual screen 40 in the camera coordinate system are estimated using the positional relationship between the image display device 13 and the virtual screen 40 that is grasped by the control unit 11. As described above, the positional relationship and the directional relationship among the camera 12, the eye 20, and the virtual screen 40 are determined under the camera coordinate system. After steps S209 and S210, the process proceeds to steps S211, S212, and S213.

In step S211, the control unit 11 uses the first parameters R ₁ and t ₁ (rotation matrix) for converting from the camera coordinate system to the eye coordinate system based on the pupil center position of the real image of the eye 20 estimated in step S209. R ₁ , translation vector t ₁ ) is determined. In step S212, the control unit 11 determines the coordinates in the eye coordinate system based on the pupil center position of the real image of the eye 20 estimated in step S209 and the center position and direction of the virtual screen 40 estimated in step S210. To the virtual screen 40, the second parameters n _e , h (the unit normal vector n _e of the plane of the virtual screen 40 in the eye coordinate system, the plane of the virtual screen 40 and the origin O _e of the eye coordinate system) Find the distance h). In step S213, the control unit 11 displays an image from the eye coordinate system based on the pupil center position of the real image of the eye 20 estimated in step S209 and the center position and direction of the virtual screen 40 estimated in step S210. Third parameters R ₂ and t ₂ (rotation matrix R ₂ , translation vector t ₂ ) for conversion to the device coordinate system are obtained. After the above steps S211, S212, and S213, the process ends.

  Then, the control part 11 performs control which displays presentation information on the image display device 13 by performing coordinate conversion using the coordinate conversion parameter calculated | required as mentioned above.

[Effects of this embodiment]
According to the present embodiment described above, the HMD 1 can be easily calibrated. Specifically, in this embodiment, calibration is automatically performed when the user faces the mirror 30, so that the user's trouble at the time of calibration is reduced as compared with the technique described in the above-described prior art document. can do. In addition, since such calibration can be realized by using a mirror 30 (hand mirror or the like) that is close to the user, according to the present embodiment, a special one for calibration (for example, a marker for calibration) is carried. Or embedded in the usage environment.

[Modification]
Below, the modification of an above-described Example is presented. The following modifications can be implemented in any combination.

(Modification 1)
A stereo camera may be used as the camera 12 described above. When a stereo camera is used as the camera 12, the estimation accuracy for the position and direction can be improved. If a stereo camera is used, the three-dimensional position can be estimated from the captured image even if the sizes of the markers 12a and 13a are unknown. Therefore, it is not necessary to let the control unit 11 know the size of the markers 12a and 13a in advance.

(Modification 2)
In the above description, the markers 12a and 13a are used to estimate the positions and directions of the camera 12 and the image display device 13 from the captured images. However, the markers 12a and 13a may not be used. In another example, the positions and directions of the camera 12 and the image display device 13 can be estimated using the shape characteristics of the camera 12 and the shape characteristics of the image display device 13 without using the markers 12a and 13a. .

(Modification 3)
In the above embodiment, the pupil center position is used as the position of the user's eye 20, but the optical center position of the eye 20 may be used instead of the pupil center position. If the relationship between the pupil center position and the optical center position is measured in advance, the optical center position of the eye 20 can be estimated from the captured image. When such an optical center position is used, the coordinate conversion parameter can be obtained with higher accuracy than when the pupil center position is used.

(Modification 4)
In the above, the embodiment has been described in which the position and direction of the image display device 13 are estimated and the coordinate conversion parameter is obtained based on the position and the direction. In another example, the coordinate conversion parameter can be obtained without estimating the position and direction of the image display device 13. When the positional relationship between the camera 12 and the image display device 13 is fixed (for example, when the frame 14 is appropriately fixed), the position and direction of the image display device 13 are uniquely determined. The position and direction of the virtual screen 40 need not be estimated. In this case, the processing of steps S204, S207, and S210 shown in FIG. 9 need not be performed. That is, it is not necessary to detect the screen marker, estimate the position and direction of the mirror image of the image display device 13, and estimate the center position and direction of the virtual screen 40.

(Modification 5)
The use of the mirror 30 for obtaining a mirror image is not limited. Instead of the mirror 30, a mirror image of the captured image may be obtained by using a mirror having a reflectance higher than a predetermined value (for example, glass).

(Modification 6)
In the above, in order to estimate the mirror plane, the optical center position and the optical axis direction of the mirror image of the camera 12 are estimated. However, the mirror plane is estimated by adding a marker to the mirror and photographing it. May be. In this case, steps S202, S205, and S208 shown in FIG. 9 do not have to be performed.

(Modification 7)
In the above description, the LCD 13b, the lens 13c, and the half mirror 13d are used as the optically transmissive image display device. In another example, a transparent display using a transparent organic EL or a transparent LCD, or an image projected by an LCD projector on a translucent plate such as frosted glass may be used.

1 Head mounted display (HMD)
DESCRIPTION OF SYMBOLS 11 Control part 12 Camera 12a, 13a Marker 13 Image display device 13b LCD
13c lens 13d half mirror 14 frame

Claims (10)

A head-mounted display having an imaging means and an optically transmissive image display device,
The coordinate system defined by the imaging unit is converted into a first coordinate system defined by a center position in a real image of the user's eyeball based on a captured image of the user's mirror image captured by the imaging unit, and the first coordinate By converting the system to a coordinate system defined by the image display device, the positional relationship in the user's field of view between the object observed through the image display device and the information based on the photographed image of the photographing means is calibrated. A head mounted display provided with a calibration means. The calibration means includes
Based on the center position, the coordinates of the coordinate system defined by the user's eyeball are converted into the coordinates of the coordinate system defined by the head mounted display in order to display the information on the image display device. Parameter calculating means for obtaining parameters for
The head mounted display according to claim 1, further comprising: The calibration means includes
First estimation means for estimating a center position in a real image of the user's eyeball from the captured image of the mirror image;
Second estimation means for estimating an optical center position and an optical axis direction in a mirror image of the imaging means from the captured image of the mirror image;
Third estimation means for estimating a center position in a mirror image of the user's eyeball from the captured image of the mirror image;
Fourth estimation means for estimating a mirror plane that has generated the mirror image based on the optical center position and the optical axis direction estimated by the second estimation means;
The first estimating means estimates a center position in a real image of the user's eyeball based on the mirror plane estimated by the fourth estimating means and the center position estimated by the third estimating means. The head-mounted display according to claim 1, wherein: A marker is added to the photographing means,
The head mounted display according to claim 1, wherein the calibration unit performs the calibration based on the marker added to the imaging unit included in the captured image of the mirror image. A marker is added to the image display device,
The head mounted display according to claim 1, wherein the calibration unit performs the calibration based on the marker added to the image display device included in the captured image of the mirror image.   The display control means for controlling the display of the information on the image display device by performing the coordinate conversion using the coordinate conversion parameters obtained by the calibration means. 2. The head mounted display according to 2. The calibration means includes
Fifth estimation means for estimating the position and direction in the mirror image of the image display device from the captured image of the mirror image;
Sixth estimation means for estimating a position and a direction in a real image of the image display device based on the mirror plane estimated by the fourth estimation means and the position and the direction estimated by the fifth estimation means; Further comprising
The calibration means is an image display device capable of optically transmitting the first coordinate system based on the center position estimated by the first estimation means and the position and direction estimated by the sixth estimation means. Find the parameters to convert to the coordinate system specified in
The calibration means includes coordinates of a coordinate system defined by the user's eyeball based on the center position estimated by the first estimation means and the position and direction estimated by the sixth estimation means. The head mounted display according to claim 3, wherein a parameter for converting the information into coordinates of a coordinate system defined by the head mounted display in order to display the information on the image display device is obtained. .   The head mounted display according to any one of claims 1 to 6, wherein a pupil center position or an optical center position is used as a center position of the user's eyeball. A calibration method executed by a head-mounted display that has an imaging unit and an optically transmissive image display device, and visually recognizes information based on a captured image of the imaging unit on the image display device,
The coordinate system defined by the imaging unit is converted into a first coordinate system defined by a center position in a real image of the user's eyeball based on a captured image of the user's mirror image captured by the imaging unit, and the first coordinate By converting the system to a coordinate system defined by the image display device, the positional relationship in the user's field of view between the object observed through the image display device and the information based on the photographed image of the photographing means is calibrated. A calibration method comprising a calibration step. A calibration program executed by a head-mounted display including a photographing unit and an optically transmissive image display device, causing information based on a photographed image of the photographing unit to be visually recognized on the image display device, and including a computer;
The coordinate system defined by the imaging unit is converted into a first coordinate system defined by a center position in a real image of the user's eyeball based on a captured image of the user's mirror image captured by the imaging unit, and the first coordinate By converting the system to a coordinate system defined by the image display device, the positional relationship in the user's field of view between the object observed through the image display device and the information based on the photographed image of the photographing means is calibrated. A calibration program for causing the computer to function as calibration means.

Images