JP2017044756A - Image display device, image display method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display device that contributes to the ability of a HMD wearer to recognize that an AR image is superposed correctly on an object.SOLUTION: The image display device comprises: an optical beacon reception unit for receiving a signal transmitted from a first optical beacon arranged in accordance with the predetermined position of an object, the signal including first optical beacon identification information; a corneal ball position estimation unit for detecting an area pertaining to the pupil of the eye from image data in which the pupil of a wearer is imaged, and estimating the center position of a corneal ball on the basis of the size and shape of the detected pupil; an optical beacon position estimation unit for estimating the position of the first optical beacon on the display surface of a virtual image visually recognized by the wearer on the basis of the position in the image data of the image of the first beacon reflected in the pupil and the estimated center position of the corneal ball; and an augmented reality display unit for displaying an image pertaining to augmented reality associated with the first optical beacon identification information at the estimated position of the first optical beacon by being superposed on an actual background.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image display device, an image display method, and a program. In particular, the present invention relates to an image display apparatus, an image display method, and a program that provide display related to augmented reality (AR).

  In recent years, research and development of products related to HMD (Head Mounted Display) to be mounted on the head have been actively conducted. The situation in which the HMD is used includes equipment maintenance and luggage collection in a warehouse. For example, an operator in charge of the above work can be expected to improve work efficiency by wearing an HMD. This is because the HMD can project an instruction related to a virtual image work location and work content so as to be superimposed on an object (work location) visually observed by the worker. As a result, the operator does not need to compare the work manual with the work object, and can reliably grasp the work location. In this way, a product in which a marker or the like is pasted on a work object, an image of the marker is captured by a camera mounted on the HMD, and the operator's field of view is recognized, and instructions on the work location and work content are projected, System development is underway.

  Patent Document 1 discloses a system that displays an apparatus maintenance command and an operation command using augmented reality.

  In Patent Document 2, a system that displays an AR image so as to be aligned with an object in a scene viewed by a wearer of the HMD, and that requests the HMD wearer to make an adjustment related to the display position of the image. It is disclosed.

  Patent Document 3 discloses a technique for calibrating the positional relationship between an object and a virtual reality provided by the HMD when a wearer of the HMD faces a mirror.

Special table 2009-538487 International Publication No. 2013/145147 International Publication No. 2013/128612

  Each disclosure of the above prior art document is incorporated herein by reference. The following analysis was made by the present inventors.

  In the existing system, the HMD analyzes the image captured by the mounted camera, detects the marker (AR marker) in the scene (real image of the field of view, real scenery), and determines the position of the marker in the captured image. It has recognized. Therefore, if the positional relationship between the HMD and the eyeball is fixed (fixed), it is easy to adjust the position between the scene (real image of the field of view) and the AR (Augmented Reality) image, or once adjusted, again. No adjustment is required. However, in actual use, it is difficult to assume that the positional relationship between the HMD and the eyeball does not change. When these positional relationships vary, it is necessary to correct variation (displacement) in the positional relationship between the scene and the AR image.

  This is also the case with the technology disclosed in Patent Document 1, in which the relative position between the camera mounted on the HMD and the eyeball of the wearer is shifted, resulting in the direction of the marker in the image captured by the camera, The direction of the marker that is actually visually recognized by the wearer deviates. That is, the relative position between the camera and the wearer's eyes is displaced, and the direction of the marker in the image captured by the camera may not match the direction of the marker viewed by the wearer.

  Moreover, in the conventional HMD as disclosed in Patent Document 1, the position of the marker visually recognized by the wearer and the position of the virtual image display (display surface that the HMD wearer recognizes that the AR image is displayed). There is no means for detecting. For example, in Patent Document 1, the position of the marker and the position of the virtual image display are determined by initial setting. Since the above means does not exist, in order to correctly perform the function of the HMD, the positional relationship between the HMD, the wearer, and the scene is determined so that the wearer can recognize that the position of the marker and the position of the AR image overlap. After fixing, it was necessary to measure and acquire the correspondence between the position on the image captured by the camera and the position on the display of the virtual image visually recognized by the wearer. However, there is a problem that such measurement is complicated.

  In addition, if the positional relationship between the three elements (HMD, wearer, scene) changes while using the HMD, the position on the image captured by the camera and the display that the wearer visually recognizes Since the correspondence relationship between the positions above changes, each time the wearer uses the HMD, it is necessary to wear and fix the HMD and remeasure the positional relationship between the two images. Also, in Patent Document 2, the user performs the alignment for adjusting the positional relationship between the object and the AR image. However, if the user requests a complicated alignment operation every time the user feels uncomfortable with the AR image, the HMD The work efficiency of using is reduced. In this respect, the technique disclosed in Patent Document 3 is the same, and if the wearer faces the mirror every time alignment is required, the work efficiency decreases.

  As described above, in the existing system including the system disclosed in Patent Document 1, the position on the image captured by the camera in advance in order to correctly superimpose the virtual image work instruction in accordance with the actual work location visually recognized by the worker. And a complicated operation of measuring the correspondence between the position of the virtual image visually recognized by the wearer on the display. In addition, the above-described correspondence may vary during use of the HMD, and there is a problem that the measurement operation (alignment) is re-executed or requested from the user each time. Alternatively, if the target object and the AR image remain shifted, problems such as instructing an operator to an incorrect work location may occur.

  An object of the present invention is to provide an image display device, an image display method, and a program that contribute to the fact that an HMD wearer can visually recognize that an AR image is correctly superimposed on an object.

  According to the 1st viewpoint of this invention, it is a signal transmitted from the 1st optical beacon arrange | positioned according to the position of the predetermined object, Comprising: The signal containing the identification information of the said 1st optical beacon is received. Detecting an area related to the wearer's pupil from image data obtained by capturing an optical beacon receiving unit and at least the wearer's pupil wearing the device, and based on the size and shape of the detected pupil, A corneal sphere position estimating unit that estimates a center position of the wearer's corneal sphere, a position in the image data of an image of a first optical beacon reflected on the wearer's pupil, and the estimated center of the corneal sphere And an optical beacon position estimating unit for estimating a position of the first optical beacon on a display surface of a virtual image visually recognized by the wearer, and the estimated position of the first optical beacon. Regarding optical beacon identification information An image relating to the Tagged augmented reality displays superimposed on the real scenery, and a augmented reality display unit, an image display apparatus is provided.

  According to the 2nd viewpoint of this invention, it is a signal transmitted from the 1st optical beacon arrange | positioned according to the position of the predetermined object, Comprising: The signal containing the identification information of the said 1st optical beacon is received. Detecting a region related to the wearer's pupil from image data obtained by imaging at least a wearer's pupil wearing the device, and based on the detected size and shape of the pupil Estimating the central position of the corneal sphere, based on the step, the position in the image data of the image of the first optical beacon reflected on the wearer's pupil, and the estimated central position of the corneal sphere, Estimating the position of the first optical beacon on the display surface of the virtual image viewed by the wearer, and augmented reality associated with the identification information of the first optical beacon at the estimated position of the first optical beacon Drawings related to A display superimposed on the actual landscape, including a step, an image display method is provided.

According to the 3rd viewpoint of this invention, it is a signal transmitted from the 1st optical beacon arrange | positioned according to the position of the predetermined object, Comprising: The signal containing the identification information of the said 1st optical beacon is received. And a region related to the wearer's pupil is detected from image data obtained by imaging the at least the wearer's pupil wearing the device, and the wearer is based on the detected size and shape of the pupil. Estimating the center position of the corneal sphere, based on the processing, the position in the image data of the image of the first optical beacon reflected on the wearer's pupil, and the estimated center position of the corneal sphere, A process of estimating the position of the first optical beacon on the display surface of the virtual image visually recognized by the wearer, and the augmented reality associated with the identification information of the first optical beacon at the estimated position of the first optical beacon Images of the real Display superimposed on the color, a program to be executed by the onboard computer processing and, to the image display apparatus is provided.
This program can be recorded on a computer-readable storage medium. The storage medium may be non-transient such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, or the like. The present invention can also be embodied as a computer program product.

  According to each viewpoint of the present invention, an image display device, an image display method, and a program are provided that contribute to the fact that the wearer of the HMD can visually recognize that the AR image is correctly superimposed on the object.

It is a figure for demonstrating the outline | summary of one Embodiment. It is a figure which shows an example of the whole structure of the system which concerns on 1st Embodiment. It is a figure which shows an example of the internal structure of HMD which concerns on 1st Embodiment. It is a figure which shows an example of the top view (a) and front view (b) of HMD which concern on 1st Embodiment. It is a figure which shows an example of the optical beacon reflected in a pupil. It is a figure which shows an example of the whole operation | movement which concerns on the positional relationship calculation of the target object by HMD which concerns on 1st Embodiment. It is a flowchart which shows an example of operation | movement of an optical beacon receiving part. It is the figure (a) imitating the cross section of an actual eyeball, and the figure which shows an example of the geometric model (b) of an eyeball. It is a figure which shows the relationship between the position and direction of an eyeball, and the projection image of a pupil by a weak perspective projection model. It is a flowchart which shows an example of operation | movement of a cornea ball position estimation part. It is a figure for demonstrating the relationship between the position of the image of the optical beacon in the image data which the camera part imaged, and the direction of the optical beacon which a wearer visually recognizes. It is a figure which shows an example of the positional relationship of the optical beacon which exists on the straight line which connects the image of the optical beacon on the surface of a corneal sphere, and an optical beacon. It is a figure which shows an example of the target object light beacon which exists in the display of the virtual image visually recognized by the wearer of HMD. It is a figure for demonstrating the table information stored in the memory | storage part. It is a flowchart which shows an example of operation | movement of an optical beacon position estimation part. It is a flowchart which shows an example of operation | movement of an optical beacon position estimation part. It is a flowchart which shows an example of operation | movement of an augmented reality display part.

  First, an outline of one embodiment will be described. Note that the reference numerals of the drawings attached to the outline are attached to the respective elements for convenience as an example for facilitating understanding, and the description of the outline is not intended to be any limitation.

  The image display apparatus 100 according to an embodiment includes an optical beacon receiving unit 101, a corneal sphere position estimating unit 102, an optical beacon position estimating unit 103, and an augmented reality display unit 104. The optical beacon receiving unit 101 is a signal transmitted from a first optical beacon (for example, the target optical beacon 11 shown in FIG. 2) arranged according to a predetermined position of the target, and the first optical beacon A signal including the identification information is received. The corneal sphere position estimation unit 102 detects a region related to the wearer's pupil from image data obtained by imaging the wearer's pupil wearing at least the own device (image display device 100; HMD), and detects the detected pupil. Based on the size and shape, the center position of the wearer's corneal sphere is estimated. The optical beacon position estimating unit 103 displays a virtual image display surface visually recognized by the wearer based on the position in the image data of the image of the first optical beacon reflected on the wearer's pupil and the estimated center position of the corneal sphere. The position of the first optical beacon at is estimated. The augmented reality display unit 104 superimposes and displays an image related to augmented reality associated with the identification information of the first optical beacon on the estimated position of the first optical beacon.

  The image display device 100 calculates the size and shape (outer shape of the pupil) of the wearer's pupil from image data relating to the wearer's pupil of the image display device 100 captured by an imaging unit such as a camera. Further, the image display device 100 calculates a three-dimensional geometric model of the eyeball from the calculated size and shape of the pupil and the position in the image data of the image of the first optical beacon reflected from the surface of the corneal sphere. By using it, the position on the display of the virtual image of the first optical beacon seen from the wearer is calculated (estimated). By displaying the augmented reality image (AR image) associated with the first optical beacon at the estimated position of the first optical beacon, the wearer of the image display device 100 overlaps the object and the augmented reality image. Can be visually recognized. That is, according to the image display device 100 according to an embodiment, the AR image can be drawn so as to overlap the object as viewed from the wearer. In addition, work that is performed every time the wearer wears and fixes the HMD (work that measures the correspondence between the position on the image captured by the camera and the position on the display of the virtual image that the wearer sees). It becomes unnecessary, and work efficiency can be improved.

  Hereinafter, specific embodiments will be described in more detail with reference to the drawings. In addition, in each embodiment, the same code | symbol is attached | subjected to the same component and the description is abbreviate | omitted.

[First Embodiment]
The first embodiment will be described in more detail with reference to the drawings.

  FIG. 2 is a diagram illustrating an example of the overall configuration of the system according to the first embodiment. Referring to FIG. 2, the user wears an HMD 10 that is an image display device. In addition, an object on which the AR image is superimposed exists in the scene (real image of the field of view, real scenery) of the wearer of the HMD 10. Moreover, the object light beacons 11-1 to 11-4 are arranged so as to capture the object (according to the position of the object). The object light beacons 11-1 to 11-4 function as markers for designating positions for displaying the AR images (function as AR markers). In the following description, when there is no particular reason for distinguishing the object light beacons 11-1 to 11-4, they are simply referred to as “object light beacons 11”.

  In the example of FIG. 2, the object light beacons 11 are composed of four optical beacons. An optical beacon is a device that emits visible light or infrared light that is fixed at a predetermined location. The optical beacon forming the object optical beacon 11 transmits a detection signal (beacon signal) including its own identification information (hereinafter referred to as ID (Identification) information) to a predetermined light pulse pattern or the like (light (Send by pulse position modulation etc.) Such communication using optical beacons is a form of optical space communication. For example, the following document “AV & IT Standardization Committee,“ CP-1223 ”established by JEITA (Japan Electronics and Information Technology Industries Association). Reference may be made to the Visible Light Beacon System “May 2013”.

  A typical light source of an optical beacon is an LED (Light Emitting Diode), but the light source of the optical beacon is not limited to an LED. In other words, the light source of the optical beacon may be any light source as long as it generates a bright spot within a predetermined size and can perform the above-described light modulation. Moreover, although 1st Embodiment demonstrates the number of the optical beacons which comprise the target object optical beacon 11 as four, it is not the meaning which limits the number of optical beacons. That is, the number of optical beacons may be other numbers such as three or more and five or more. However, with three, the AR image deformation method is limited, and with five or more, the amount of processing to be described later increases. Therefore, the number of optical beacons constituting the object optical beacon 11 is preferably four from these viewpoints. .

  From the above viewpoint, in the first embodiment, a set of object light beacons 11 is configured by four optical beacons, and the distances between the object optical beacons 11 are grasped in advance. .

  FIG. 3 is a diagram illustrating an example of an internal configuration of the HMD 10 according to the first embodiment. Referring to FIG. 3, the HMD 10 includes a camera unit 20, a display unit 30, a storage unit 40, a control unit 50, a camera control unit 60, and a display control unit 70. Each part shown in FIG. 3 is mutually connected by an internal bus (data line, control signal line, etc.).

  The camera unit 20 is a camera module including an image sensor 21 and a camera lens 22.

  The display unit 30 is a display module including a display unit 31 including a light source, a lens unit 32, a light guide unit 33, and a glass unit 34. Furthermore, the glass part 34 includes a transflective mirror 35 disposed so as to be positioned in front of the pupil of the wearer of the HMD 10 and a plurality of glass part optical beacons 36-1 to 36-4. . In the following description, when there is no particular reason for distinguishing the glass part optical beacons 36-1 to 36-4, they are simply expressed as “glass part optical beacons 36”. Moreover, in the following description, when it is not necessary to distinguish the object light beacon 11 and the glass part light beacon 36, they are simply referred to as “light beacon”.

  The light emitted by the glass part light beacon 36 does not need to be visible light, and is light outside the detection wavelength range of the eye of the wearer of the HMD 10 and within the detection wavelength range of the imaging device 21 of the camera part 20 (for example, the eye It is desirable that it is near infrared light having a wavelength that does not easily adversely affect the eye and is within the safety light intensity for the eyes. As described above, the object light beacon 11 can be a device that emits infrared light. That is, the light emitted from the object light beacon 11 does not need to be visible light, and is light outside the detection wavelength range of the eye of the wearer of the HMD 10 and within the detection wavelength range of the imaging device 21 of the camera unit 20 (for example, , Near-infrared light having a wavelength that has little adverse effect on the eyes, and within a safe light intensity for the eyes). However, when the object light beacon 11 is invisible light, the object light beacon 11 in the following description cannot be visually recognized. However, if the object light beacon 11 is within the detection wavelength range of the camera unit 20, the glass unit 34 of the HMD 10. For each wearer of the HMD 10, the positional relationship between the upper glass part light beacon 36 and the object light beacon 11, which is a real image in which an actual bright spot is arranged, is detected, and the actual image and the AR image are aligned. It is possible to project an AR image position appropriate for the specified position on the field of view. Further, the intensity of light emitted from the object light beacon 11 and the glass part light beacon 36 is sufficiently low so as not to affect the retina.

  The storage unit 40 stores programs and data necessary for the operation of the control unit 50. For example, the storage unit 40 includes information on the object light beacon 11 including four optical beacons (for example, ID information, a distance between the light beacons) and an AR image associated with the object light beacon 11. A plurality of pieces of information as one set are stored. Moreover, the memory | storage part 40 memorize | stores ID information regarding the glass part optical beacon 36 which consists of four optical beacons. Alternatively, the storage unit 40 stores information such as the focal length of the camera lens 22 and the pixel pitch of the image sensor 21 that are parameters of the camera unit 20. Further, the storage unit 40 stores information such as the pupil radius, the radius of the corneal sphere, and the length from the center of the pupil to the center of the corneal sphere, which are parameters of the geometric model of the eyeball. Furthermore, the memory | storage part 40 memorize | stores the corresponding table information of the position of the optical beacon in the surface enclosed by the glass part optical beacon 36, and the position of the optical beacon in the display of a virtual image.

  The control unit 50 is configured by an arithmetic device such as a CPU (Central Processing Unit), and implements various functions by reading out various programs from the storage unit 40 and executing the programs. The control unit 50 includes an optical beacon receiving unit 51, a corneal sphere position estimating unit 52, an optical beacon position estimating unit 53, and an augmented reality display unit 54. Details of these parts will be described later. The control unit 50 can also be realized by a computer program that causes a computer mounted on the HMD 10 to execute each process described in detail later using the hardware thereof.

  The control unit 50 instructs the camera control unit 60 and the display control unit 70 to operate. In addition, the control unit 50 performs access to the storage unit 40 (reading data and writing data).

  The camera control unit 60 takes in the image data captured by the camera unit 20 and writes it in the storage unit 40.

  The display control unit 70 reads the image data from the storage unit 40 and outputs it to the display unit 31.

  FIG. 4 is a diagram illustrating an example of a plan view (a) and a front view (b) of the HMD 10 according to the first embodiment. Referring to FIG. 4A, the light emitted from the light source of the display unit 31 passes through the lens unit 32 and the light guide unit 33 and reaches the semi-transmissive mirror 35. The light that reaches the semi-transmissive mirror 35 is reflected by the semi-transmissive mirror 35. The light guide 33 may be a hollow pipe or a bundle of optical fiber cables. The transmissivity of the semi-transmissive mirror 35 is a predetermined value within a range where the virtual image projected by the HMD 10 can be recognized without impeding the visual field of the wearer, and the transmissivity is not limited to 50%.

  As shown in FIG. 4A, the semi-transmissive mirror 35 is arranged in front of the wearer of the HMD 10. Therefore, the wearer of the HMD 10 can visually recognize a virtual image that overlaps the scene (real image of the field of view, real image of the scenery) that is transmitted through the semi-transmissive mirror 35. That is, the wearer of the HMD 10 can recognize a virtual image display (virtual image display surface; a display surface that the wearer of the HMD 10 recognizes that the AR image is displayed) as if it were an actual object. That is, the range of the virtual image visually recognized by the wearer is a virtual image display (see FIG. 4A).

  The camera unit 20 including the imaging element 21 and the camera lens 22 is fixed so that the wearer's eyes (eyeball including the cornea portion) fall within the angle of view of the camera unit 20. The image captured by the camera unit 20 is such that the cornea of the eyeball corresponds to a convex mirror, and the depth of field is such that two images of the object light beacon 11 reflected by the wearer's pupil and the cornea surface are in focus at the same time. Is set. In other words, the cornea of the eyeball functions as a convex mirror having a very short focal length (the same function as the concave lens), and therefore the camera lens 22 set so as to focus on the image of the object light beacon 11 reflected by the cornea surface. The rear depth of field required for the camera is extremely shorter than the distance from the focus position on the optical path from the camera lens 22 to the object light beacon 11 to the object light beacon 11. The pupil of the wearer of the HMD 10 and the optical beacon in the scene reflected by the corneal surface are simultaneously imaged with sufficient resolution (see FIG. 5).

  The glass part optical beacon 36 is a set of optical beacons arranged on the glass part 34 so as to surround the semi-transmissive mirror 35 (see FIG. 4B). That is, the glass part light beacon 36 and the object light beacon 11 arranged in the scene have physically different configurations.

[Description of operation]
Next, the operation of the HMD 10 according to the first embodiment will be described. When the wearer of the HMD 10 visually recognizes the object in the scene, the object light beacon 11 is also visually recognized (the object light beacon 11 also enters the wearer's field of view).

  The image captured by the camera unit 20 of the HMD 10 with the wearer visually recognizing the object light beacon 11 includes images of the object light beacon 11 and the glass part light beacon 36 reflected by the wearer's pupil. . For example, as shown in FIG. 5, the image captured by the camera unit 20 includes images of the object light beacon 11 and the glass part light beacon 36 reflected by the wearer's pupil.

  Since the reflection of light at the pupil is a reflection at the corneal surface and the corneal surface is a curved surface, the positional relationship between the images of multiple optical beacons reflected at the pupil is the positional relationship when the optical beacons are projected onto a plane. Unlike that, it becomes distorted. Hereinafter, the calculation operation of the positional relationship between the display of the virtual image viewed by the wearer and the object in the scene from the position of the image of the optical beacon captured by the camera unit 20 will be described.

  First, the entire calculation operation will be described with reference to FIG.

  When the HMD 10 starts operation, the camera unit 20 of the HMD 10 repeats imaging at a constant period and outputs image data obtained by imaging to the camera control unit 60 before executing various processes shown in FIG. The camera control unit 60 writes the acquired image data to the storage unit 40, and each time the writing of one frame is completed, the camera control unit 60 instructs the control unit 50 (more specifically, the optical beacon reception unit 51 and the corneal sphere position estimation unit 52). The image data writing completion is notified.

  Each of the optical beacon receiving unit 51 and the corneal sphere position estimating unit 52 that has received the notification executes an optical beacon receiving process and a corneal sphere position estimating process (steps S01 and S02). When the above two processes are completed, the optical beacon position estimation unit 53 executes an optical beacon position estimation process (step S03). Thereafter, the augmented reality display unit 54 executes an AR image drawing process (step S04).

[Operation of optical beacon receiver]
The optical beacon receiving unit 51 is a means for receiving a signal that is transmitted from an optical beacon arranged according to a predetermined position of an object and that includes identification information of the optical beacon. Specifically, the optical beacon receiving unit 51 analyzes the image data captured by the camera unit 20, reads the ID transmitted by the optical beacon, and the presence of the optical beacon reflected by the wearer's pupil in the image data It is a means to detect.

  FIG. 7 is a flowchart illustrating an example of the operation of the optical beacon receiving unit 51.

  In step S <b> 101, the optical beacon receiving unit 51 detects a detection signal transmission region from the image data written in the storage unit 40 when receiving a notification of completion of writing of one frame from the camera control unit 60. The detection signal transmission area is an area where the luminance changes in the same pattern as a predetermined detection signal pattern. The optical beacon receiving unit 51 detects the detection signal transmission region by comparing image data of a predetermined number of frames captured by the camera unit 20. When the optical beacon receiving unit 51 detects the detection signal transmission region (a region where the luminance changes in the same pattern as the detection signal), the optical beacon receiving unit 51 selects the optical beacon image candidate (hereinafter referred to as an optical beacon candidate). Set to (notation).

  The optical beacon receiving unit 51 determines whether or not signal restoration processing and error detection processing described later have been applied to all acquired optical beacon candidates (step S102).

  If the signal restoration process or the like has been completed for all optical beacon candidates (step S102, Yes branch), the optical beacon reception unit 51 executes the processes according to steps S107 and S108. If there is an optical beacon candidate that is not subjected to signal restoration processing or the like (step S102, No branch), the optical beacon receiving unit 51 selects one candidate from the plurality of optical beacon candidates, and the selected optical beacon candidate is selected. Demodulation processing is performed on the luminance change pattern to restore the signal, and error detection processing is performed on the restored signal (step S103). As a result of signal restoration, ID information of each optical beacon is obtained.

  As a modulation method used for an optical beacon, there is baseband-pulse position modulation or the like. Further, in the demodulation of the modulation method, for example, by measuring the pulse width of the light and the interval between the pulses, the reference time interval and the position (time) of the light pulse in the time interval are specified and dealt with. A data string is obtained by applying a binary value (0/1 information). Then, the ID information and the error detection code are restored based on a predetermined frame structure.

  The optical beacon receiving unit 51 calculates an error detection code from the previously acquired data sequence and compares it with the restored error detection code to determine whether there is an error in the restored signal. In addition, quaternary pulse position modulation (4PPM; 4 Pulse Position Modulation) is mentioned as a modulation system normally used in visible light communication. In order to demodulate a signal modulated by this method, the pulse width and interval of light are usually detected.

  Further, modulation methods using 4PPM include a method of superimposing subcarriers and a method of not superimposing subcarriers. In the first embodiment, an image sensor used as a receiving element usually does not have an analog circuit that divides a signal at a subcarrier frequency, and separates incident light for each sensor pixel by a lens. Therefore, considering that the influence of the interference light is small, a method that does not superimpose subcarriers is used. However, if various conditions (for example, power consumption, cost, etc.) are prepared, an image sensor corresponding to a frame rate sufficiently higher than the frequency of the subcarrier is used, thereby using a modulation method by 4PPM that superimposes the subcarrier. You can also Note that the 4PPM modulation method that does not superimpose subcarriers is also used in FIR (Fast Infrared) of version 1.1 of IrDA (Infrared Data Association).

  As described above, the optical beacon receiving unit 51 applies the ID information of the optical beacon by applying a predetermined frame structure to the data sequence (0/1 data sequence) acquired by demodulating the optical signal. Restore. For the frame structure used for restoring the ID information, reference can be made to FIG. 5 of the document “Hideki Ueno, Yoshiyuki Sato, Atsushi Kataoka,“ Visible Light ID System ”, Toshiba Review Vol.62 No.5, 2007”.

  The error detection code used in the first embodiment is a CRC (Cyclic Redundancy Check) normally used in visible light communication. Although the error detection code need not be used, the error detection code is used in the first embodiment in consideration of the necessity of using the error detection code in the optical space communication standard.

  The optical beacon receiving unit 51 determines whether or not an error is detected from the optical beacon candidates (step S104). If an error is detected (step S104, Yes branch), the process returns to step S102 and continues the processing.

  When an error is not detected for the optical beacon candidate (step S104, No branch), the optical beacon receiving unit 51 recognizes the optical beacon candidate (detection signal transmission region) as an optical beacon image, and Calculate the center of gravity position. The optical beacon receiving unit 51 determines the calculated position of the center of gravity as the coordinate position of the optical beacon (optical beacon image) in the image data captured by the camera unit 20 (step S105).

  The optical beacon receiving unit 51 associates the received (decoded) ID information with the coordinate position and stores it in the storage unit 40 (step S106), and returns to the process of step S102.

  When the process for acquiring the ID information is completed for all optical beacon candidates (step S102, Yes branch), the optical beacon receiving unit 51 determines whether there is an area determined as an optical beacon in the optical beacon candidates. Is determined (step S107).

  If there is an area determined to be an optical beacon image (step S107, Yes branch), the optical beacon reception unit 51 notifies the optical beacon position estimation unit 53 of completion of optical beacon reception (step S108). If there is no region determined to be an optical beacon image (step S107, No branch), the optical beacon receiving unit 51 ends the process.

  As described above, the signals transmitted by the object light beacon 11 and the glass portion light beacon 36 are modulated by a predetermined method, and the light beacon receiving unit 51 analyzes a plurality of image data corresponding to one frame. As a result, the signal transmitted by the optical beacon is demodulated and the identification information of the optical beacon is acquired.

[Operation of corneal sphere position estimation unit]
The corneal sphere position estimation unit 52 detects a region related to the wearer's pupil from image data obtained by imaging at least the wearer's pupil wearing the HMD 10, and wears the cornea according to the size and shape of the detected pupil. This is a means for estimating the center position of a person's corneal sphere. More specifically, the corneal sphere position estimation unit 52 detects the pupil from the image data captured by the camera unit 20 and calculates the center position of the corneal sphere based on the size and shape. In the following description, the position of the corneal sphere or the like starts from the principal point of the camera lens 22 of the camera unit 20 (origin).

  The relationship between the outer shape (shape and size) of the pupil in the image captured by the camera and the position and orientation of the eyeball is a known fact. For example, the document “C. Nitschke, A. Nakazawa and H. Takemura,“ “Display-Camera Calibration from Eye Reflection”, Proc. International Conference Computer Vision (ICCV), pp. 1226-1233, 2009 ”. Alternatively, the above facts are also described in the literature “Atsushi Nakazawa, Christian Nichuke,“ A method of estimating a gaze point robust to parallax errors using the corneal imaging method ”, 19th Image Sensing Symposium, June 2013”. ing.

  FIG. 8 is a schematic diagram (FIG. 8A) simulating a cross section of an actual eyeball and an example of a geometric model of the eyeball (FIG. 8B). Referring to FIG. 8A, since the cornea portion of the actual eyeball protrudes, two spheres of the eyeball body and the corneal sphere are used in the approximation by the geometric model (see FIG. 8B).

  Note that fixed values are used for the parameters of the geometric model of the eyeball. More specifically, the geometric model of the eyeball is approximated by a plurality of parameters, and fixed values are used for each parameter (rL = 5.6 mm, rC = 7.7 mm, dLC = 5.6 mm; FIG. b)). It should be noted that although there are individual differences in the actual size of the eyeball, it is known that the accuracy of the surface reflection analysis result of the eyeball has sufficient accuracy even if a fixed value is used as a parameter of the eyeball geometric model. Yes.

  FIG. 9 is a diagram illustrating the relationship between the position and orientation of the eyeball and the projection image of the pupil using a weak perspective projection model. Note that the perspective projection model is inherently non-linear, and the weak perspective projection model is a projection model that approximates the perspective projection model linearly. The weak perspective projection model is a technique used in the field of computer vision.

  Referring to FIG. 9, the image captured by the camera unit 20 matches the image projected on the image plane 201 shown in FIG. The pupil projection image 202 projected onto the image plane 201 has an elliptical shape indicated by a major axis rmax, a minor axis rmin, a center iL, and a rotation angle Φ. Even when the weak perspective projection model is used, it has been confirmed that the accuracy of the result of the surface reflection analysis of the eyeball is sufficient.

  The corneal sphere position estimation unit 52 calculates the shape and size of the pupil shown in the image captured by the camera unit 20. The corneal sphere position estimation unit 52 needs to detect the outer shape of the pupil prior to the calculation. At that time, it is assumed that a part of the pupil is hidden by eyelashes or eyelashes. Even in such a case, there is a method for accurately detecting the outer shape of the pupil. For example, in the following literature “Tokuo Tsuji, Shingo Shibata, Tsutomu Hasegawa, Ryo Kurazume,“ Iris Detection Method Using LMedS for Gaze Measurement ”, Image Recognition and Understanding Symposium (MIRU 2004), July 2004” The described method can be used.

When detecting the outer shape of the pupil, the corneal sphere position estimation unit 52 uses the following developed ellipse equation (1).
Ax ² + Bxy + Cy ² + Dx + Ey + F = 0 (1)
However, 4AC-B ² > 0

  The corneal sphere position estimation unit 52 detects the outer shape of the pupil using a minimum median method (LMedS; Least Median Square). Note that the minimum median method repeatedly extracts several samples and applies them to the least squares method (LSM) to estimate when the median of the square error in all measured values is the smallest. This is a method that is regarded as a correct estimation. The details are described in the document “P.J. Rousseeuw and A.M. Leroy,“ Robust Regression and Outlier Detection ”, Wiley, New York, 1987”.

  The corneal sphere position estimating unit 52 uses, for example, a random 5 points from the candidate points of the outline of the pupil including noise as a point to be substituted for obtaining the coefficient of the ellipse represented by the above formula (1) according to the minimum median method. Select a point. Then, the corneal sphere position estimation unit 52 uses a median value of the squares of the distances between the ellipse represented by the obtained coefficient and all candidate points as a value for evaluating the coefficient.

Next, the corneal sphere position estimation unit 52 obtains the ellipse parameters from the following formulas (2) to (5) from the coefficients of the developed ellipse formula.

X-coordinate x0 of the center iL = (BE-2CD) / (4AC-B ² ) (2)

Y coordinate y0 of center iL = (BD-2AE) / (4AC-B ² ) (3)

Rotation angle Φ = (1/2) tan ⁻¹ (B / (AC)) (4)

(Long axis rmax, short axis rmin) = √ ((2 (A · x0 ² + B · x0 · y0 + C · y0 ² −F)) / (A + C ± (B / sin2Φ))) (5)
However, a larger value is rmax and a smaller value is rmin.

Next, the corneal sphere position estimation unit 52 calculates the posture (position, direction) of the eyeball using the ellipse parameters. The corneal sphere position estimation unit 52 uses the major axis rmax of the projection image 202 of the pupil, the parameter rL that is a parameter of the three-dimensional geometric model of the eyeball, and the focal length f of the camera unit 20 to use the corneal sphere. A distance d between the center position 203 and the image plane 201 is calculated (see equation (6)).

d = f · rL / rmax (6)

Further, the corneal sphere position estimation unit 52 calculates the tilt τ in the depth direction of the pupil using the major axis rmax and the minor axis rmin of the projected image of the pupil (see Expression (7)).

τ = ± arccos (rmin / rmax) (7)

Furthermore, the corneal sphere position estimation unit 52 calculates the line-of-sight vector g from the rotation angle Φ of the projected image of the pupil and the tilt τ in the depth direction of the pupil (see Expression (8)).

g = [sinτsinΦ−sinτcosΦ−cosτ] ^T (8)

Note that “T” in Equation (8) indicates that the matrix of Equation (8) is a transposed matrix.

  The center point L of the pupil can be calculated from the center iL of the projected image of the pupil, the focal length f of the camera unit 20, and the distance d between the center point C of the corneal sphere 203 and the image plane 201. . The center point C of the corneal sphere 203 is a parameter used in the pupil center point L, the line-of-sight vector g, and the weak perspective projection model, and is a parameter dLC which is a three-dimensional parameter of the eyeball. And can be calculated from

  Next, details of the operation of the corneal sphere position estimation unit 52 will be described with reference to the flowchart shown in FIG.

  In step S <b> 201, when the corneal sphere position estimation unit 52 receives a notification of completion of writing of one frame from the camera control unit 60, it binarizes (white / black) the image data written in the storage unit 40. By such processing, it is possible to distinguish between a dark pixel with a pupil or shadow and a bright pixel with a sclera or skin.

  Note that as a binarization method, a method of calculating a threshold value such that black has a predetermined ratio from the luminance distribution of all pixels of the image and comparing each pixel with the threshold value can be employed. Alternatively, a method of comparing the brightness of pixels adjacent to an arbitrary pixel in order, and determining that the difference is larger when it is brighter than a predetermined value, and is black when it is larger when it is darker May be adopted.

  In step S202, the corneal sphere position estimation unit 52 performs an opening process on the binarized image by a contraction process and an expansion process. Note that the contraction process and the expansion process are methods for image processing, and are processes for a binarized monochrome image. Specifically, the dilation process is a process of performing an operation of replacing a white pixel with a white pixel even if there is one pixel around a certain pixel. The erosion process is a process of performing an operation of replacing a black pixel if there is a black pixel in the periphery.

  The corneal sphere position estimation unit 52 executes the opening process by repeating the expansion / contraction process the same number of times.

  In step S <b> 203, the corneal sphere position estimation unit 52 determines whether there is a dark pixel (pixel value is black) region in the image data for which the opening process has been completed. When there is no dark pixel region (step S203, No branch), the corneal sphere position estimating unit 52 determines that the pupil is not photographed because the wearer closed his eyes, and the process ends.

  When there is a dark pixel area (when it is determined that the pupil is reflected in the image data; Step S203, Yes branch), the corneal sphere position estimation unit 52 determines each boundary pixel in the binarized image. The coordinates are acquired (step S204). Specifically, the corneal sphere position estimation unit 52 performs edge detection processing on the binarized image and extracts pixels that serve as boundaries between the dark pixel area and the bright pixel area. Get the coordinates.

  The acquired boundary pixels are candidates for pixels on the outer periphery of the pupil (the boundary between the cornea and sclera). When the corneal sphere position estimation unit 52 executes the edge detection process, pixels on the outer periphery of the pupil and pixels on the outer periphery of the shadow area are extracted in a mixed manner. The coordinates here mean the coordinates of the extracted individual pixels.

  In step S <b> 205, the corneal sphere position estimation unit 52 calculates each coefficient that forms an ellipse equation corresponding to the outer shape of the pupil from the extracted boundary pixels using the minimum median method (LMedS).

  In step S206, the corneal sphere position estimation unit 52 determines whether or not the coefficient of the ellipse equation has been calculated. If the coefficient cannot be calculated (if no coefficient is found), the process ends (step S206, No branch). When the coefficient can be calculated (step S206, Yes branch), the corneal sphere position estimation unit 52 calculates the center point C of the corneal sphere (steps S207 to S212).

  First, the corneal sphere position estimation unit 52 calculates the position of the center iL of the projected image of the pupil, the long axis rmax, the short axis rmin, the rotation angle Φ, the coefficient of the elliptic formula, the formula for calculating the elliptic parameter, Calculation is performed based on the pixel pitch of the image sensor 21 of the unit 20 (step S207).

  Next, the corneal sphere position estimation unit 52 determines the distance d between the center L of the pupil and the image plane, the long axis rmax, the parameter rL that is a parameter of the three-dimensional geometric model of the eyeball, and the focal length f of the camera unit. , To calculate (step S208).

  Next, the corneal sphere position estimation unit 52 calculates the position of the center L of the pupil using the position of the center iL, the focal length f of the camera unit, and the distance d between the center L of the pupil and the image plane. (Step S209).

  Next, the corneal sphere position estimating unit 52 calculates the inclination τ in the depth direction of the pupil using the major axis rmax and the minor axis rmin (step S210).

  Next, the corneal sphere position estimation unit 52 calculates the line-of-sight vector g using the rotation angle Φ of the pupil projection image and the tilt τ in the depth direction of the pupil (step S211).

  Next, the corneal sphere position estimation unit 52 calculates the center point C of the corneal sphere using the position of the pupil center L, the line-of-sight vector g, and the eyeball geometric model parameter dLC (step S212; (See FIG. 9).

  When the calculation of the center point C of the corneal sphere is completed, the corneal sphere position estimation unit 52 writes the calculation result in the storage unit 40 and notifies the optical beacon position estimation unit 53 of the completion of the eyeball position estimation (step S213). ).

  As described above, the corneal sphere position estimation unit 52 projects the wearer's pupil onto the image plane 201 at the focal position (position away from the principal point of the camera lens 22 by the focal distance f) when the image data is generated. Is specified by the coefficient of the equation representing the ellipse. Thereafter, the corneal sphere position estimating unit 52 determines the distance d between the center L of the pupil and the image plane, the long axis rmax, the parameter rL that is a parameter of the three-dimensional geometric model of the eyeball, the focal length f of the camera unit, Calculate using. Next, the corneal sphere position estimation unit 52 calculates the position of the center L of the pupil using the position of the center iL, the focal length f of the camera unit, and the distance d between the center L of the pupil and the image plane. To do. Thereafter, the corneal sphere position estimating unit 52 calculates the inclination τ of the wearer's pupil using the lengths of the major axis rmax and the minor axis rmin of the ellipse indicating the wearer's pupil. In addition, the corneal sphere position estimation unit 52 determines the direction in which the wearer's pupil faces based on the calculated inclination τ of the wearer's pupil and the rotation angle Φ of the wearer's pupil projected on the image plane 201. The gaze vector g shown is calculated. Further, the corneal sphere position estimation unit 52 is based on the calculated line-of-sight vector g, the predetermined parameter dLC of the eyeball geometric model, and the distance d between the center position of the wearer's corneal sphere and the image plane. The center point C (center position) of the wearer's corneal sphere is estimated.

[Operation of optical beacon position estimation unit]
The optical beacon position estimation unit 53 displays a virtual image display surface that the wearer visually recognizes based on the position in the image data of the image of the optical beacon reflected on the pupil of the wearer of the HMD 10 and the estimated center position of the corneal sphere. This is means for estimating the position of the optical beacon in the (virtual image display). More specifically, the optical beacon position estimating unit 53 determines the position of the image of the optical beacon detected by the optical beacon receiving unit 51 (the optical beacon reflected on the wearer's pupil in the image data captured by the camera unit 20). And a corneal sphere center point C estimated by the corneal sphere position estimating unit 52 (a corneal sphere center point in the image data captured by the camera unit 20). And calculate the positional relationship between the optical beacon.

  FIG. 11 is a diagram for explaining the relationship between the position of the optical beacon image in the image data captured by the camera unit 20 and the direction of the optical beacon visually recognized by the wearer. First, the optical beacon position estimating unit 53 determines the position of the optical beacon image on the surface of the corneal sphere, the focal length f that is a parameter related to the camera unit 20, and the optical beacon image in the image data captured by the camera unit 20. , The center point C of the corneal sphere, and the radius rC of the corneal sphere (see FIG. 8B).

  As shown in FIG. 11, since the reflection angle in the reflection on the corneal sphere surface is the same as the incident angle, the optical beacon position estimation unit 53 performs the image of the optical beacon on the principal point O of the camera lens 22 of the camera unit 20 and the corneal sphere surface. The direction of the straight line g connecting the optical beacon image on the corneal sphere surface and the actual optical beacon is calculated from the straight line 211 connecting the corneal sphere center point C and the optical beacon image on the corneal sphere surface. To do. Note that, even if the optical beacon is either the object optical beacon 11 or the glass part optical beacon 36, since four optical beacons are one set, a straight line g connecting the optical beacon image on the corneal sphere surface with the optical beacon is Four are calculated as one set.

  In order to obtain the direction of the optical beacon visually recognized by the wearer of the HMD 10, a straight line 213 connecting the center point C of the corneal sphere and the optical beacon is required. In order to specify the position of the optical beacon on the straight line connecting the optical beacon image on the corneal sphere surface and the optical beacon, the optical beacon position estimating unit 53 uses the distance between the optical beacons that are known in advance.

  FIG. 12 is a diagram illustrating an example of the positional relationship of the optical beacons on the straight line g connecting the optical beacon image on the corneal sphere surface and the optical beacon. In the first embodiment, four optical beacons are one set, and the distance between the optical beacons is known information in advance. Therefore, on the straight line connecting the optical beacon image on the corneal sphere surface with the optical beacon. The point where this distance relationship (positional relationship between optical beacons) is established is the actual position of the optical beacon.

  For example, as shown in FIG. 12A, a case is considered in which the distance between each pair of optical beacons P1 to P4 is determined as d12, d23, d34, and d41. Moreover, the straight line g corresponding to each of the optical beacons P1 to P4 is denoted as straight lines g1 to g4. The positions of the optical beacons P1 to P4 that satisfy the distances (d12, d23, d34, d41) between the optical beacons shown in FIG. 12A can be obtained as follows.

  First, the optical beacon position estimating unit 53 calculates a point p1max that is a position that the optical beacon P1 can take on the straight line g1 and is farthest from the eyeball as follows. The optical beacon position estimating unit 53 assumes a point p1 at an arbitrary position on the straight line g1, and calculates the shortest distance dp1-g2min between p1 and the straight line g2.

  Next, the optical beacon position estimating unit 53 compares the calculated shortest distance dp1-g2min with the distance d12. If the shortest distance dp1-g2min is greater than a certain value as a result of the comparison, the optical beacon position estimating unit 53 assumes that the position of the point p1 is far from the eyeball (a position far from the predetermined predetermined distance). Suppose) and again calculate the shortest distance dp1-g2min.

  On the other hand, if the shortest distance dp1-g2min is smaller than a certain value, the optical beacon position estimating unit 53 assumes that the position of the point p1 is close to the eyeball (assumed to be close to a predetermined distance). Then, the shortest distance dp1-g2min is calculated again.

  If the difference between the shortest distance dp1−g2min and the distance dp12 is within a predetermined range, the optical beacon position estimating unit 53 determines the position of p1 as p1max.

  Next, the optical beacon position estimating unit 53 assumes that the point p1 on the straight line g1 exists on p1max. Then, the optical beacon position estimation unit 53 calculates a point p2 on the straight line g2 that exists at the position of the distance d12 from the point p1.

  At this time, there may be two points p2 on the straight line g2 at a distance d12 away from the point p1 (one point in the direction approaching the pupil and one point in the direction far from the pupil; FIG. 12). (See (b)). Next, the optical beacon position estimating unit 53 calculates a point p3 on the straight line g3 that exists at a distance d23 from the point p2. Also in this case, there may be two points p3 for each of the two points p2 (the point p3 may be four places). Next, the optical beacon position estimating unit 53 calculates a point p4 on the straight line g4 that exists at a distance d34 from the point p3. Also in this case, there is a possibility that there are two points p4 for each of the four points p3 (the point p4 may be eight places). Next, the optical beacon position estimating unit 53 calculates a point p1 'on the straight line g1 that exists at a distance d41 from the point p4. In this case, there may be two points p1 'for each of the eight points p4 (there may be 16 points p1').

  Next, the optical beacon position estimating unit 53 calculates a distance dp1-p1 'between the initially assumed point p1 and a maximum of 16 points p1' calculated based on the point p1. If all the calculated dp1-p1 ′ are equal to or greater than a certain value, the optical beacon position estimating unit 53 assumes again the position of the point p1 to a position close to the eyeball (position close to a predetermined value), and the distance The calculation of dp1−p1 ′ and the comparison with a constant value (threshold value) are repeated.

  As a result of the above iterative process, if the optical beacon position estimating unit 53 detects a distance dp1-p1 ′ that is equal to or smaller than a certain value, the optical beacon P1 is set to a set of points p1, p2, p3, and p4. It is determined at a position of ~ P4.

  When the wearer of the HMD 10 visually recognizes the object through the virtual image display, the presence of the virtual image display can be confirmed within the range surrounded by the glass part light beacon 36. Furthermore, the wearer visually recognizes the object light beacon 11 in the range of the virtual image display.

  FIG. 13 is a diagram illustrating an example of the object light beacon 11 existing in the display range of the virtual image visually recognized by the wearer of the HMD 10.

  The optical beacon position estimating unit 53 acquires the coordinates on the display 221 of the virtual image that overlaps with the object optical beacon 11 when viewed by the wearer (viewed from the wearer so as to overlap). At that time, the optical beacon position estimating unit 53 passes a straight line (straight line 213 shown in FIG. 11) connecting the center point C of the corneal sphere and the optical beacon over the virtual plane 222 surrounded by the glass part optical beacon 36. Calculate points.

  Thereafter, the optical beacon position estimating unit 53 converts the calculation result into coordinates from the correspondence between the position on the virtual surface 222 surrounded by the glass part optical beacon 36 described later and the position on the display 221 of the virtual image.

  The virtual surface 222 surrounded by the glass part optical beacons 36 is calculated from a plane formula by the least square method or the like from the positions of the four glass part optical beacons 36, and the four glass part optical beacons 36 are projected on the plane. It can be calculated as a surface surrounded by points (projection points).

  Here, the information regarding the correspondence between the position on the virtual surface 222 surrounded by the glass part optical beacon 36 and the position on the display 221 of the virtual image may be calculated each time or written in advance as table information in the storage unit 40. Alternatively, the position may be selected from the position of the center point C of the corneal sphere.

  As shown in FIG. 4, the wearer of the HMD 10 visually recognizes a virtual image display reflected by the semi-transmissive mirror 35. Therefore, the eyeballs and glasses of the wearer change depending on various characteristics such as the size and refractive index of the display unit 31, the lens unit 32, the light guide unit 33, and the semi-transmissive mirror 35, and individual differences in the distance between both eyes. The correspondence between the position of the virtual surface 222 surrounded by the glass part optical beacon 36 visually recognized by the wearer and the position of the virtual image display 221 is determined by the positional relationship with the part 34.

  Among the various elements that determine the correspondence between the position of the virtual surface 222 surrounded by the glass part light beacon 36 and the position of the virtual image display 221, elements other than the positional relationship between the wearer's eyeball and the glass part 34 are changed. No (fixed). Therefore, for example, by writing table information in which the position on the virtual surface 222 surrounded by the glass part light beacon 36 and the position on the display 221 of the virtual image are associated with each other in the storage unit 40, The table information can be selected based on the center point C of the corneal sphere calculated from the captured image data and the position of the glass part light beacon 36.

  The table information to be written in the storage unit 40 changes the relative position of the wearer's eyeball with respect to the glass unit 34 by ray tracing simulation, and sets each change point to the center of the corneal sphere with respect to the glass unit optical beacon 36. It can be created by associating with the position of the point C.

  FIG. 14 is a diagram illustrating an example of table information that associates the center point C of the corneal sphere of the pupil with the position of the virtual image display on the virtual plane. Referring to FIG. 14A, the table information defines the display position (coordinates) of the virtual image on the virtual surface in accordance with the relative position of the center point C of the pupil with respect to the virtual surface 222. For example, when the position of the center point C of the pupil is in the first row of FIG. 14A, the position of the virtual image display 221 on the virtual plane 222 formed by the glass part light beacon 36 is as shown in FIG. It is prescribed as follows. Similarly, when the position of the center point C is the second line in FIG. 14A, the position of the virtual image display 221 is defined as shown in FIG. Alternatively, when the relative position of the corneal sphere center point C with respect to the virtual surface 222 by the glass part light beacon 36 is an intermediate position of the position in the table (when not registered in the table), the closest position in the table Or 3 points that are close in distance with respect to the relative position of the center point C (3 points are necessary because of 3D coordinates). May be.

  Next, the operation of the optical beacon position estimating unit 53 will be described with reference to FIGS. 15 and 16.

  In FIG.15 S301, the optical beacon position estimation part 53 determines whether 1 set of the object optical beacon 11 was discovered. Specifically, when the optical beacon position estimation unit 53 receives the notification related to the completion of optical beacon reception from the optical beacon reception unit 51 and the notification related to completion of eyeball position estimation from the corneal ball position estimation unit 52, The ID information of the optical beacon written in the storage unit 40 (the ID information of the optical beacon received by the optical beacon receiving unit 51) is compared with the ID information registered in advance.

  As a result of the comparison, when a set of object light beacons 11 is found (step S301, Yes branch), the optical beacon position estimating unit 53 executes the processing after step S302. If the object light beacon 11 is not found as a result of the comparison (step S301, No branch), the light beacon position estimating unit 53 ends the process.

  In step S302, the optical beacon position estimation unit 53 determines whether or not one set of the glass part optical beacons 36 has been found, as in the previous step. If the set of the glass part optical beacon 36 is found (step S302, Yes branch), the optical beacon position estimation part 53 performs the process after step S303. If the pair of glass part optical beacons 36 is not found (step S302, No branch), the optical beacon position estimating part 53 ends the process.

  In step S303, the optical beacon position estimating unit 53 calculates a straight line connecting the optical beacon image and the optical beacon on the corneal sphere surface. In that case, the optical beacon position estimation part 53 calculates the position of the image of the optical beacon on the corneal sphere surface for each of the object optical beacon 11 and the glass part optical beacon 36.

  The optical beacon position estimation unit 53 includes known information (focal length f of the camera unit 20, pixel pitch of the image sensor 21, and radius rC of the corneal sphere) written in the storage unit 40, and an optical beacon reception unit 51. Written in the storage unit 40 (the position of the optical beacon image in the image data captured by the camera) and the information written in the storage unit 40 by the corneal sphere position estimation unit 52 (the position of the center point C of the corneal sphere). ), The position of the image of the optical beacon on the surface of the corneal sphere is calculated. The optical beacon position estimating unit 53 calculates the direction of the straight line connecting the optical beacon image on the corneal sphere surface and the optical beacon using the straight line connecting the principal point of the camera lens and the optical beacon image on the corneal sphere surface.

  Next, the optical beacon position estimation unit 53 calculates the position of the optical beacon (step S304; see FIG. 15).

  Referring to FIG. 16, the optical beacon position estimating unit 53 assumes a point p1 at an arbitrary position on a straight line g1 connecting the optical beacon P1 and the image of the optical beacon on the corneal sphere surface among the set of optical beacons. The shortest distance dp1-g2min is calculated (steps S401 and S402).

  When the difference between the shortest distance dp1−g2min and the distance d12 is equal to or greater than a certain value (predetermined value) (step S403, Yes branch), the optical beacon position estimation unit 53 re-assums the point p1 at the position shifted ( Then, the shortest distance dp1-g2min is calculated again (step S402). Specifically, if the shortest distance dp1-g2min is larger than a certain value, the optical beacon position estimating unit 53 assumes the position of the point p1 as a position far from the eyeball, and calculates the shortest distance dp1-g2min again. On the other hand, if the shortest distance dp1-g2min is smaller than a certain value, the position of the point p1 is assumed to be close to the eyeball, and the shortest distance dp1-g2min is calculated again.

  If the difference between the distance dp1−g2min and the distance d12 is equal to or smaller than a certain value (step S403, No branch), the optical beacon position estimating unit 53 sets the position of the point p1 to p1max, and the point p1 is on the straight line g1. It is assumed that it exists at p1max (step S405).

  In step S406, the optical beacon position estimating unit 53 calculates the points p1 to p2, the points p2 to p3, the points p3 to p4, and the points p4 to p1 'by the method described above. Thereafter, the optical beacon position estimating unit 53 calculates the distance dp1-p1 'between the point p1 and a maximum of 16 points p1' calculated based on the point p1.

  The optical beacon position estimating unit 53 determines whether or not there is a distance dp1-p1 'that is equal to or smaller than a certain value among all the distances dp1-p1' (step S407). If all the distances dp1-p1 'are equal to or greater than a certain value (step S407, No branch), the optical beacon position estimation unit 53 assumes again the position of the point p1 to a position close to the eyeball (step S408).

  Thereafter, the optical beacon position estimating unit 53 determines whether or not the distance between the position of p1 and the eyeball is equal to or less than a certain value (step S409), and determines whether or not there is room for moving the position of p1. judge. If the point p1 can be moved (step S409, No branch), the optical beacon position estimating unit 53 returns to steps S406 and S407, and calculates the distance dp1-p1 ′ and compares the calculated value with a constant value. repeat.

  Finally, when the distance dp1-p1 'below the certain value cannot be found (step S409, Yes branch), the optical beacon position estimating unit 53 ends the process.

  When a distance dp1-p1 ′ that is equal to or smaller than a certain value is found (step S407, Yes branch), the optical beacon position estimation unit 53 determines the position of the optical beacon as a set of the points p1, p2, p3, and p4. (Step S410).

  Thus, the process of step S304 in FIG. 15 (process for calculating the position of the optical beacon) ends, and the process returns to the upper process (process in FIG. 15).

  The optical beacon position estimation unit 53 repeats the process of step S304 for each set of optical beacons, and calculates the positions of all optical beacons in the set. When the calculation cannot be performed (step S305, No branch), the optical beacon position estimating unit 53 ends the process. When the calculation is completed (step S305, Yes branch), the optical beacon position estimation unit 53 calculates a straight line connecting the center point C of the corneal sphere and the object optical beacon 11 (step S306).

  Next, the optical beacon position estimating unit 53 calculates the virtual plane 222 surrounded by the glass unit optical beacon 36 (step S307).

  The optical beacon position estimating unit 53 calculates the point through which the straight line connecting the center point C of the corneal sphere and the object optical beacon 11 (the straight line calculated in step S306) passes through the virtual plane 222 calculated in step S307. (Step S308).

  The optical beacon position estimation unit 53 determines whether or not the optical beacon has passed through the range (virtual plane 222) surrounded by the glass part optical beacon 36 (step S309), and if it does not pass (step S309, No branch) ) Ends the process.

  When passing (step S309, Yes determination), the optical beacon position estimating unit 53 determines the center point (center position) C of the wearer's corneal sphere from the center point C of the corneal sphere and the position of the glass part optical beacon 36. And the relative position of the glass part is calculated. Thereafter, the optical beacon position estimating unit 53 selects a corneal sphere from among a plurality of pieces of table information (see FIG. 14A) regarding the position on the display 221 of the virtual image on the virtual surface 222 surrounded by the glass unit optical beacon 36. One table associated with the center point C and the position of the glass optical beacon 36 is selected (step S310).

  Next, the optical beacon position estimation unit 53 coordinates the virtual image on the display 221 that overlaps the object optical beacon 11 when viewed from the wearer with respect to a point passing through the virtual surface 222 surrounded by the glass optical beacon 36. (Coordinates are calculated; step S311).

  When the coordinates cannot be calculated (step S312, No branch), the optical beacon position estimating unit 53 ends the process. If the coordinates can be calculated (step S312, Yes branch), the optical beacon position estimating unit 53 receives the calculated coordinates, and the optical beacon ID information (the optical beacon receiving unit 51 has written in the storage unit 40). Write to the storage unit 40 in association with the optical beacon ID information).

  Thereafter, the optical beacon position estimating unit 53 notifies the augmented reality display unit 54 of completion of optical beacon position estimation (step S313).

  As described above, the optical beacon position estimating unit 53 is formed by the plurality of glass part optical beacons 36 based on the position in the image data of the image of the glass part optical beacon 36 reflected on the wearer's pupil included in the image data. The position of the virtual image display 221 on the virtual plane 222 is specified based on the relative positional relationship between the virtual plane 222 and the center point C (center position) of the wearer's corneal sphere. To do. The light beacon position estimating unit 53 calculates the position of the object light beacon 11 based on the position in the image data of the image of the object light beacon 11 reflected on the wearer's pupil included in the image data. By calculating a straight line connecting the position of the beacon and the center position of the wearer's corneal sphere, and converting the coordinates of the passing point through which the straight line passes through the virtual plane into coordinates on the virtual image display 221, a virtual image display The position of the object light beacon 11 at 221 is estimated.

[Operation of augmented reality display]
The augmented reality display unit 54 superimposes and displays an image related to augmented reality associated with the identification information of the object light beacon 11 on the estimated position of the object light beacon 11 on the actual scene (scene). Means. More specifically, the augmented reality display unit 54 reads the AR image based on the ID information of the optical beacon, and the wearer visually recognizes the virtual image display 221 and the positional relationship between the optical beacon visually recognized by the wearer of the HMD 10. The AR image is drawn on the display so as to match the object of the current scene.

  FIG. 17 is a flowchart illustrating an example of the operation of the augmented reality display unit 54.

  In step S501, when the augmented reality display unit 54 receives the notification of the completion of the optical beacon position estimation by the optical beacon position estimation unit 53, the received optical beacon ID information (the ID information written by the optical beacon reception unit 51), The coordinates on the display 221 of the virtual image (the coordinates written by the optical beacon position estimating unit 53 and overlapping the object optical beacon 11 visually recognized by the wearer) are read from the storage unit 40.

  Thereafter, the augmented reality display unit 54 reads out an AR image (an AR image associated with the ID information) stored in advance in the storage unit 40 using the ID information as a search key (step S502).

  It is assumed that the AR image is similar to the quadrangle of four sides d12, d23, d34, and d41, and is displayed so as to overlap the quadrangle having the pair of object light beacons 11 as vertices. It is the contents. Further, the quadrangle surrounded by the coordinates of the virtual image display 221 that overlaps the object light beacon 11 visually recognized by the wearer changes in shape depending on the angle at which the wearer views the object. Therefore, the augmented reality display unit 54 calculates a deformed image of the AR image so as to match the changing shape (step S503). The above calculation is a projective transformation performed based on the four vertices of the AR image and the four coordinates on the display 221 of the virtual image overlapping the object light beacon 11 visually recognized by the wearer.

  Thereafter, the augmented reality display unit 54 writes the modified image of the AR image in the storage unit 40 and instructs the display control unit 70 to send the image data to the display unit 31 (step S504).

  Upon receiving the instruction, the display control unit 70 reads the deformed AR image from the storage unit 40 and outputs it to the display unit 31.

  As described above, in the system according to the first embodiment, one set of optical beacons whose positional relationship is known in advance is arranged in the glass unit 34 of the HMD 10 and one or more sets in the scene. In addition, in a state where the wearer of the HMD 10 visually recognizes the scene, the camera unit 20 mounted on the HMD 10 images the eyes of the wearer. The HMD 10 uses a three-dimensional geometric model of the eyeball from the position in the image of the outer shape of the wearer's pupil reflected in the captured image and the image of the optical beacon reflected on the cornea surface. Calculate the position of the optical beacon that is looking at. Thereafter, the HMD 10 estimates the relationship between the position of the virtual image displayed by the wearer and the position of the scene. That is, the HMD 10 detects the positional relationship between the glass part light beacon 36 on the glass part 34 of the HMD 10 and the object light beacon 11 which is a real image in which an actual bright spot is arranged, and the actual image and the AR image are detected. Perform alignment. As a result, it is possible to project an appropriate AR image position for each wearer of the HMD 10 at a designated position on the field of view.

  The HMD 10 according to the first embodiment is an aerial image of an AR image by using an optical see-through type (HMD; Head Mounted Display). When the AR image is displayed to the wearer of the HMD, the light emitted from the light source having the function of an optical beacon arranged in the scene and the glass part of the HMD and reflected on the cornea surface of the wearer is converted into the HMD. Take a picture with the camera mounted on the camera. Further, the HMD 10 calculates the position of the optical beacon using the geometric model of the eyeball, so that the scene (real image such as landscape) that the wearer is viewing and the virtual image display (the function of the display that projects the virtual image) A positional relationship with a virtual image plane (by a video projection device such as an HMD) is estimated, and an AR image that is a virtual image is projected at an appropriate position in the scene.

  The optical beacon described in the first embodiment is a device for position guidance using light having an identification function such as blinking. In the first embodiment, the light of the object light beacon 11 is transmitted from a pseudo point light source such as an LED (Light Emitting Diode) or the like through the semi-transparent mirror 35 or the glass part 34, and the light intensity is attenuated so that the wearer's cornea On the other hand, as for the light of the glass part light beacon 36, the light of the light source directly enters the wearer's cornea and the real image enters the wearer's field of view. However, instead of the glass part optical beacon 36, a configuration in which the bright part of the virtual image having the role of the optical beacon is superimposed on the wearer's visual field to form the glass part optical beacon may be used. That is, a pseudo-point light source such as an LED is installed in one of the optical paths from the display unit 31 to the semi-transmissive mirror 35, and an aerial image (virtual image) of a dotted luminescent spot is formed in the wearer's field of view. You may make it tie. Alternatively, a bright spot that blinks in the same manner as an optical beacon (or may be modulated in the same manner) is displayed on an arbitrary pixel or a set of pixels in the display area of the display unit 31 and appears as an aerial image in the wearer's field of view. A pixel on a virtual image display may be used as a glass part optical beacon.

  Although the industrial applicability is clear from the above explanation, the present disclosure is suitable for a case where an aerial image of an AR image is projected using a see-through head mounted display and the AR image is provided to the wearer. It is.

  A part or all of the above embodiments can be described as in the following supplementary notes, but is not limited thereto.

[Appendix 1]
This is the same as the image display device according to the first viewpoint described above.
[Appendix 2]
A plurality of second light beacons disposed around a semi-transparent mirror disposed in front of the wearer's pupil;
The optical beacon position estimating unit
Based on the position in the image data of the image of the second light beacon reflected on the wearer's pupil included in the image data, a plane formed by the plurality of second light beacons is calculated as a virtual surface,
The image display device according to appendix 1, wherein a position of a display surface of the virtual image on the virtual surface is specified based on a relative positional relationship between the virtual surface and a center position of the wearer's corneal sphere.
[Appendix 3]
The optical beacon position estimating unit
Based on the position in the image data of the image of the first light beacon reflected on the wearer's pupil included in the image data, the position of the first light beacon is calculated,
Calculating a first straight line connecting the calculated position of the first optical beacon and the center position of the wearer's corneal sphere;
Estimating the position of the first optical beacon on the display surface of the virtual image by converting the coordinates of a passing point through which the first straight line passes through the virtual surface into coordinates on the display surface of the virtual image; The image display device according to attachment 2.
[Appendix 4]
The optical beacon position estimating unit
For each of the plurality of first light beacons, a second straight line connecting the position in the image data of the image of the first optical beacon reflected on the wearer's pupil included in the image data and the first optical beacon As well as calculating
Assuming the first optical beacon corresponding to one of the calculated second straight lines, the assumed first optical beacon, and the first optical beacon corresponding to the assumed first optical beacon. The calculated second straight line is assumed to have the shortest distance from the assumed first optical beacon on the other second straight line adjacent to the second straight line. Repeat about the
The first optical beacon initially assumed on the second straight line, and the first optical beacon on the second straight line assumed to be the first optical beacon calculated by the iterative process; The image display device according to appendix 3, wherein the assumed position of the first optical beacon when the distance between and is less than or equal to the predetermined value is the position of the first optical beacon. .
[Appendix 5]
The corneal sphere position estimating unit specifies the size and shape of the wearer's pupil projected on the image plane at the focal position when the image data is generated, using a coefficient of an expression representing an ellipse, The image display apparatus as described in any one of thru | or 4.
[Appendix 6]
The corneal sphere position estimating unit
Using the length of the major axis and minor axis of the ellipse indicating the wearer's pupil, the tilt of the wearer's pupil is calculated,
Based on the calculated inclination of the wearer's pupil and the rotation angle of the wearer's pupil projected on the image plane, a line-of-sight vector indicating a direction in which the wearer's pupil faces is calculated,
Based on the calculated line-of-sight vector, predetermined parameters of the eyeball geometric model, and the distance between the center position of the wearer's corneal sphere and the image plane, the center position of the wearer's corneal sphere The image display device according to appendix 5, wherein
[Appendix 7]
The signal transmitted by the first optical beacon is modulated by a predetermined method,
The optical beacon receiving unit demodulates a signal transmitted by the first optical beacon by analyzing a plurality of image data corresponding to one frame, and acquires identification information of the first optical beacon. The image display apparatus as described in any one of thru | or 6.
[Appendix 8]
The augmented reality display unit
Appendices 1 thru 8. The image display device according to any one of 7.
[Appendix 9]
The image display device according to appendix 7 or 8, wherein the optical beacon receiving unit restores a signal transmitted by the first optical beacon and performs error detection processing on the restored signal.
[Appendix 10]
The image display device according to any one of appendices 1 to 9, further comprising a storage unit that stores at least table information that associates a center point of a corneal sphere calculated in advance with a position of the display surface of the virtual image on the virtual surface. .
[Appendix 11]
This is the same as the image display method according to the second viewpoint described above.
[Appendix 12]
It is as the program which concerns on the above-mentioned 3rd viewpoint.
Note that the forms of Supplementary Notes 11 and 12 can be expanded from the form of Supplementary Note 2 to the form of Supplementary Note 10, similarly to the form of Supplementary Note 1.

  Each disclosure of the cited patent documents and the like cited above is incorporated herein by reference. Within the scope of the entire disclosure (including claims) of the present invention, the embodiments and examples can be changed and adjusted based on the basic technical concept. In addition, various combinations or selections of various disclosed elements (including each element in each claim, each element in each embodiment or example, each element in each drawing, etc.) within the scope of the entire disclosure of the present invention. Is possible. That is, the present invention of course includes various variations and modifications that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea. In particular, with respect to the numerical ranges described in this document, any numerical value or small range included in the range should be construed as being specifically described even if there is no specific description.

10 HMD (Head Mounted Display)
11, 11-1 to 11-4 Object light beacon 20 Camera unit 21 Imaging element
22 Camera lens 30 Display unit 31 Display unit 32 Lens unit 33 Light guide unit 34 Glass unit 35 Semi-transmissive mirror 36, 36-1 to 36-4 Glass unit optical beacon 40 Storage unit 50 Control unit 51, 101 Optical beacon receiving unit 52 , 102 corneal sphere position estimation unit 53, 103 optical beacon position estimation unit 54, 104 augmented reality display unit 60 camera control unit 70 display control unit 100 image display device 201 image plane 202 projected image 203 corneal sphere 211 to 213 straight line 221 Virtual image display 222 Virtual surface

Claims (10)

An optical beacon receiving unit that receives a signal including identification information of the first optical beacon, which is a signal transmitted from a first optical beacon arranged according to a predetermined position of the object;
An area related to the wearer's pupil is detected from image data obtained by capturing at least the wearer's pupil wearing the apparatus, and the wearer's corneal sphere is detected based on the detected size and shape of the pupil. A corneal sphere position estimator for estimating the center position;
Based on the position in the image data of the image of the first optical beacon reflected on the wearer's pupil and the estimated center position of the corneal sphere, the first image on the display surface of the virtual image viewed by the wearer. An optical beacon position estimation unit for estimating the position of the optical beacon;
An augmented reality display unit that displays an image related to augmented reality associated with the identification information of the first optical beacon at the estimated position of the first optical beacon, superimposed on a real scene;
An image display device comprising: A plurality of second light beacons disposed around a semi-transparent mirror disposed in front of the wearer's pupil;
The optical beacon position estimating unit
Based on the position in the image data of the image of the second light beacon reflected on the wearer's pupil included in the image data, a plane formed by the plurality of second light beacons is calculated as a virtual surface,
The image display device according to claim 1, wherein a position of the display surface of the virtual image on the virtual surface is specified based on a relative positional relationship between the virtual surface and a center position of the wearer's corneal sphere. The optical beacon position estimating unit
Based on the position in the image data of the image of the first light beacon reflected on the wearer's pupil included in the image data, the position of the first light beacon is calculated,
Calculating a first straight line connecting the calculated position of the first optical beacon and the center position of the wearer's corneal sphere;
Estimating the position of the first optical beacon on the display surface of the virtual image by converting the coordinates of a passing point through which the first straight line passes through the virtual surface into coordinates on the display surface of the virtual image; The image display device according to claim 2. The optical beacon position estimating unit
For each of the plurality of first light beacons, a second straight line connecting the position in the image data of the image of the first optical beacon reflected on the wearer's pupil included in the image data and the first optical beacon As well as calculating
Assuming the first optical beacon corresponding to one of the calculated second straight lines, the assumed first optical beacon, and the first optical beacon corresponding to the assumed first optical beacon. The calculated second straight line is assumed to have the shortest distance from the assumed first optical beacon on the other second straight line adjacent to the second straight line. Repeat about the
The first optical beacon initially assumed on the second straight line, and the first optical beacon on the second straight line assumed to be the first optical beacon calculated by the iterative process; The image display according to claim 3, wherein when the distance between the first optical beacon and the first optical beacon is equal to or less than the predetermined value, the assumed position of the first optical beacon is the first optical beacon position. apparatus.   The corneal sphere position estimation unit specifies the size and shape of the wearer's pupil projected on an image plane at a focal position when the image data is generated, using a coefficient of an expression representing an ellipse. The image display device according to any one of 1 to 4. The corneal sphere position estimating unit
Using the length of the major axis and minor axis of the ellipse indicating the wearer's pupil, the tilt of the wearer's pupil is calculated,
Based on the calculated inclination of the wearer's pupil and the rotation angle of the wearer's pupil projected on the image plane, a line-of-sight vector indicating a direction in which the wearer's pupil faces is calculated,
Based on the calculated line-of-sight vector, predetermined parameters of the eyeball geometric model, and the distance between the center position of the wearer's corneal sphere and the image plane, the center position of the wearer's corneal sphere The image display device according to claim 5, wherein The signal transmitted by the first optical beacon is modulated by a predetermined method,
The optical beacon receiving unit demodulates a signal transmitted by the first optical beacon by analyzing a plurality of image data corresponding to one frame, and acquires identification information of the first optical beacon. The image display device according to any one of 1 to 6. The augmented reality display unit
The projected position of the augmented reality image associated with the identification information of the first optical beacon is subjected to projective transformation at the estimated position of the first optical beacon, and then superimposed and displayed on a real scene. The image display apparatus as described in any one of thru | or 7. Receiving a signal that is transmitted from a first optical beacon arranged according to a predetermined position of an object and includes identification information of the first optical beacon;
An area related to the wearer's pupil is detected from image data obtained by capturing at least the wearer's pupil wearing the apparatus, and the wearer's corneal sphere is detected based on the detected size and shape of the pupil. Estimating the center position; and
Based on the position in the image data of the image of the first optical beacon reflected on the wearer's pupil and the estimated center position of the corneal sphere, the first image on the display surface of the virtual image viewed by the wearer. Estimating the position of the optical beacon; and
Displaying an image related to augmented reality associated with the identification information of the first optical beacon at the position of the estimated first optical beacon, superimposed on a real scene; and
An image display method including: A signal transmitted from a first optical beacon arranged in accordance with a predetermined position of an object, and receiving a signal including identification information of the first optical beacon;
An area related to the wearer's pupil is detected from image data obtained by capturing at least the wearer's pupil wearing the apparatus, and the wearer's corneal sphere is detected based on the detected size and shape of the pupil. Processing to estimate the center position;
Based on the position in the image data of the image of the first optical beacon reflected on the wearer's pupil and the estimated center position of the corneal sphere, the first image on the display surface of the virtual image viewed by the wearer. Processing to estimate the position of the optical beacon;
A process of displaying an image related to augmented reality associated with the identification information of the first optical beacon at the estimated position of the first optical beacon while being superimposed on a real scene; and
For causing a computer mounted on the image display apparatus to execute the program.

Images