JP6349660B2 - Image display device, image display method, and image display program - Google Patents

Description

  The present invention relates to an image display device, an image display method, and an image display program.

In recent years, augmented reality (AR: Augmen) displays additional information superimposed on the real environment.
A technique called “ted Reality” has attracted attention. If AR technology is used, various contents (also referred to as “AR images”) such as guidance and advertisements can be superimposed and displayed in an actual environment. As a typical apparatus using the AR technology, there is a see-through type head mounted display (HMD).

  In general, a head-mounted display is provided with a camera for imaging the outside world. For example, an image captured by a camera is used for recognizing the outside world observed by an observer (user) wearing a head-mounted display on the head and displaying an AR image at an appropriate position. The

  At this time, if there is a difference between the external world recognized from the image captured by the camera and the external world actually observed by the observer, the AR image cannot be displayed at the intended position. Therefore, in order to prevent such a shift, there is a desire to completely match the observer's line-of-sight direction with the camera's imaging direction.

  However, in the head-mounted display, if the observer's line-of-sight direction and the imaging direction of the camera are made to coincide completely, a complicated optical system is obtained. Therefore, the camera is often installed at a position that is not on the observer's line of sight (for example, a frame portion of a head-mounted display).

  When the camera is installed at such a position, naturally, there is a difference between the outside world recognized from the image captured by the camera and the outside world actually observed by the observer. Therefore, in order to reduce this shift, a technique for correcting an image captured by a camera has been developed (for example, Patent Document 1). If the technique described in Patent Literature 1 is used, an AR image can be displayed in an appropriate position, in an appropriate size, in an appropriate orientation and superimposed on an actual environment.

JP 2009-284175 A

  However, there is a case where a deviation occurs in the depth direction (observer's line-of-sight direction) between the outside world recognized from the captured image and the outside world actually observed by the observer. Nothing is considered about. When such a shift in the depth direction occurs, the AR image cannot be displayed at an appropriate position even using the technique of Patent Document 1. That is, the display position of the AR image (the position that appears to exist) is shifted toward the front or back from the object (person or the like) that the observer is gazing at. In such a state, it is necessary for the observer to observe the object and the AR image while alternately focusing on the object and the AR image, and the eyes become tired.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image display apparatus, an image display method, and an image display program that do not require the observer to perform special focusing for observing an AR image. And

(1) An image display device that projects image light according to content information onto an observer's eye while transmitting light from the outside, and an imaging unit that images the outside and generates a captured image; Using the captured image generated by the unit, a selection unit that selects a feature point that exists in the outside world, a distance measurement unit that measures a distance from the installation position of the imaging unit to the feature point,
Using the distance measured by the distance measuring unit, a calculation unit that calculates the position of the feature point when the installation position of the imaging unit is used as a reference as a first reference position, and the observer uses the feature A reception unit that receives designation of the observer that the feature point and the content information are a second reference position that can be viewed in focus in the depth direction when the direction of the point is viewed; In order to correct the first reference position to the second reference position based on a deviation between the first reference position calculated by the calculation unit and the second reference position received by the receiving unit. An image display device comprising: a reference creation unit that creates the reference data.
(2) An image display device that projects image light according to content information to an observer's eye while transmitting light from the outside, and an imaging unit that images the outside and generates a captured image; Using the captured image generated by the unit, the selection unit that selects a feature point that actually exists in the outside world, and the position of the feature point based on the installation position of the imaging unit is calculated as a first reference position The second reference position where the feature point and the content information can both be viewed in the depth direction when the observer sees the direction of the feature point. The first reference based on a deviation between the accepting unit that receives the designation of the observer, the first reference position calculated by the calculating unit, and the second reference position received by the receiving unit Position the second reference position And a reference generating unit configured to generate a reference data for correcting the said imaging unit is a stereo camera or array camera,
The image display device, wherein the calculation unit calculates the first reference position using a plurality of captured images generated by the stereo camera or the array camera.
(3) An image display device that projects image light according to content information to an observer's eye while transmitting light from the outside, and an imaging unit that captures the outside and generates a captured image;
Using the captured image generated by the imaging unit, a selection unit that selects a feature point that actually exists in the outside world, and a position of the feature point when the installation position of the imaging unit is used as a reference, a first reference position And a second reference position that allows the feature point and the content information to be viewed in focus in the depth direction when the observer sees the direction of the feature point. The first reference position calculated by the calculation unit and the second reference position received by the reception unit based on a shift between the reception unit that receives the observer's designation as being; A reference creation unit that creates reference data for correcting one reference position to the second reference position, and the reception unit focuses the displayed content information together with the feature points. The second reference position is designated by the observer's operation of moving in the depth direction to a possible position and then moving the content information so as to overlap the feature point in a plane perpendicular to the depth direction. Accept image display device.

( 4 ) The display unit further includes a display unit that displays the content information at the first reference position calculated by the calculation unit, and the reception unit displays the content information displayed on the display unit as the content information. The image display device according to any one of (1) to (3) , wherein designation of the second reference position is received by an operation of the observer that is moved to a feature point position.

( 5 ) The image processing apparatus further includes a correction unit that corrects a display position of new content information using the reference data created by the reference creation unit, the correction unit corresponding to an object being watched by the observer. The image display device according to any one of (1) to ( 4 ), wherein the display position of the new content information is changed.

( 6 ) An image display method for projecting image light according to content information onto the eyes of an observer while transmitting light from the outside, and (a) capturing the outside and generating a captured image; (B) using the captured image generated in the step (a), selecting a feature point that exists in the outside world; and (c) selecting from the installation position of the imaging device in the step (b) A step of measuring a distance to the feature point; and ( d ) using the distance measured in the step (c) as a reference when an installation position of the imaging device that performs imaging in the step (a) is used as a reference. the position of the feature point, a step of calculating a first reference position, the content information, and the feature point when the (e) the viewer sees the direction of the feature point, in the depth direction, co A step of accepting the viewer's specified as the second reference position which can be viewed Focus, said first reference position calculated in (f) said step (d), the step (e And a step of creating reference data for correcting the first reference position to the second reference position based on a deviation from the second reference position received in step (2).
(7) An image display method for projecting image light according to content information to an observer's eyes while transmitting light from the outside world, and (a) imaging the outside world to generate a captured image; (B) using the captured image generated in step (a) to select a feature point that exists in the outside world; and (d) using the installation position of the imaging device that performs imaging in step (a) as a reference. A step of calculating the position of the feature point as a first reference position, and (e) when the observer sees the direction of the feature point, the feature point and the content information in the depth direction A step of accepting the observer's designation of being a second reference position where both can be viewed in focus; (f) the first reference position calculated in step (d); and the step e) generating reference data for correcting the first reference position to the second reference position based on a deviation from the second reference position received in e), wherein the imaging device An image display method, wherein the step (d) calculates the first reference position using a plurality of captured images generated by the stereo camera or the array camera.
(8) An image display method for projecting image light according to content information to an observer's eyes while transmitting light from the outside, and (a) capturing the outside and generating a captured image; (B) using the captured image generated in step (a) to select a feature point that exists in the outside world; and (d) using the installation position of the imaging device that performs imaging in step (a) as a reference. A step of calculating the position of the feature point as a first reference position, and (e) when the observer sees the direction of the feature point, the feature point and the content information in the depth direction A step of accepting the observer's designation of being a second reference position where both can be viewed in focus; (f) the first reference position calculated in step (d); and the step e) generating reference data for correcting the first reference position to the second reference position based on the deviation from the second reference position received in e), and performing the step (e ) Moves the displayed content information in the depth direction to a position where it can be viewed together with the feature points, and then moves the content information in the plane perpendicular to the depth direction. An image display method for accepting designation of the second reference position by an operation of the observer, which is moved so as to overlap with each other.

( 9 ) ( g ) A step of displaying the content information so as to exist at the first reference position calculated in the step ( d ) is further performed, and the step ( e ) is displayed in the step ( g ). The image display method according to any one of (6) to (8) , wherein the designation of the second reference position is received by the operation of the observer, the content information being moved to the position of the feature point .

( 10 ) (h) A step of correcting the display position of the new content information is further performed using the reference data created in the step ( f ), and the step (h) The image forming method according to any one of ( 6 ) to ( 9 ), wherein a display position of the new content information is changed according to a target object.

( 11 ) A step of (a) imaging the outside world and generating a captured image by a computer functioning as an image display device that projects image light according to content information to the viewer's eyes while transmitting light from the outside world (B) using the captured image generated in step (a) to select a feature point that exists in the outside world; and (c) selecting from the installation position of the imaging device in step (b) measuring the distance to the feature points were referenced using said measured distance, the installation position of the imaging device for imaging in said step (a) in; (d) step (c) said feature point position in the case, a step of calculating a first reference position, the content information, and the feature point when the (e) the viewer sees the direction of the feature point A step of accepting both specified second reference that can be seen by focusing position, (f) and said first reference position calculated in step (d), accepted in the step (e) the And a step of creating reference data for correcting the first reference position to the second reference position based on a deviation from the second reference position.
(12) A step of (a) imaging the outside world and generating a captured image by a computer functioning as an image display device that projects image light according to the content information to the viewer's eyes while transmitting light from the outside world And (b) using the captured image generated in step (a) to select feature points that exist in the outside world; and (d) the installation position of the imaging device that performs imaging in step (a). A step of calculating the position of the feature point when used as a reference as a first reference position; and (e) when the observer sees the direction of the feature point, the feature point and the content information are Receiving the observer's designation of being a second reference position that can be viewed in focus in both directions, (f) the first calculated in step (d) Creating reference data for correcting the first reference position to the second reference position based on the deviation between the quasi-position and the second reference position received in step (e); The imaging device is a stereo camera or an array camera, and the step (d) uses the plurality of captured images generated by the stereo camera or the array camera to perform the first reference position. An image display program for calculating
(13) A step of generating a captured image by imaging the outside world on a computer that functions as an image display device that projects image light according to content information to the viewer's eyes while transmitting light from the outside world. And (b) using the captured image generated in step (a) to select feature points that exist in the outside world; and (d) the installation position of the imaging device that performs imaging in step (a). A step of calculating the position of the feature point when used as a reference as a first reference position; and (e) when the observer sees the direction of the feature point, the feature point and the content information are Receiving the observer's designation of being a second reference position that can be viewed in focus in both directions, (f) the first calculated in step (d) Creating reference data for correcting the first reference position to the second reference position based on the deviation between the quasi-position and the second reference position received in step (e); In the step (e), the displayed content information is moved in the depth direction to a position where it can be viewed in focus with the feature points, and then in a plane perpendicular to the depth direction. In the image display program, the content information is moved so as to be superimposed on the feature point, and the designation of the second reference position is received by the operation of the observer.

(14) ( g ) causing the computer to further execute a step of displaying the content information so that the content information exists at the first reference position calculated in the step ( d ), and the step ( e ) includes the step ( e ) The image display program according to any one of (11) to (13), wherein the designation of the second reference position is received by an operation of moving the content information displayed in g ) to the position of the feature point.

( 15 ) (h) Using the reference data created in step ( f ), the computer further executes a step of correcting a display position of new content information, and the step (h) The image display program according to any one of ( 11 ) to ( 14 ), wherein the display position of the new content information is changed according to the object being watched by the user.

  According to the present invention, a reference for correcting a deviation between the outside world recognized from the captured image and the outside world actually observed by the observer is created. Thereby, it is possible to display such that the AR image is present in the vicinity of the target object being watched by the observer without complicating the optical system. As a result, the observer can focus and observe both the object and the AR image at the same time, and the eyes are not tired.

It is a figure which shows the example of an external appearance of the head mounted display (HMD) apparatus which concerns on 1st embodiment. It is a schematic block diagram of AR provision apparatus provided in a HMD apparatus. It is a block diagram which shows the hardware structural example of an HMD apparatus. It is a block diagram which shows the function structural example of an HMD apparatus. It is a flowchart which shows the procedure of a calibration process. It is a figure which shows the example of the positional relationship of the HMD apparatus with which the observer's head was mounted | worn, and the object which exists in an imaging direction. It is a figure which shows an example of a captured image. It is a figure for demonstrating the method to adjust the position of AR marker. It is a figure for demonstrating the adjustment method of the depth direction (Z direction) of AR marker. It is a figure for demonstrating the adjustment method of the left-right direction (X direction) and the up-down direction (Y direction) of AR marker. It is a schematic block diagram of AR provision apparatus which concerns on 2nd embodiment. It is a flowchart which shows the procedure of AR image display processing. It is a figure which shows the example of a display when the display position of AR image is changed according to the target object which the observer is gazing at. It is a figure which shows the modification about a display apparatus at the time of adjustment of AR marker. It is a figure which shows the example of the captured image when an observer's head shakes, and an observer's visual field.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios.

<First embodiment>
FIG. 1 is a diagram illustrating an appearance example of a head mounted display (HMD) device according to the first embodiment. FIG. 2 is a schematic configuration diagram of an AR providing apparatus provided in the HMD apparatus. FIG. 3 is a block diagram illustrating a hardware configuration example of the HMD device.

  Hereinafter, a schematic configuration of the HMD device 10, particularly a hardware configuration, will be described with reference to FIGS. 1 to 3.

<HMD device 10 (hardware configuration)>
The HMD device 10 according to the first embodiment is an image display device used by being mounted on the head of a user (hereinafter also referred to as “observer”). For example, the HMD device 10 is a see-through type image display device that projects image light according to content information desired to be presented to an observer's eyes as an AR (Augmented Reality) image while transmitting light from the outside world. The AR image is displayed as if it exists in the actual environment (real space) that the observer is observing.

  As shown in FIG. 1, the HMD device 10 includes a frame 11, a transparent member 12, an imaging device 13, an AR providing device 20, and a main body 30.

  The frame 11 has a shape that allows the HMD device 10 to be mounted on the observer's head, and has, for example, an eyeglass shape.

  The transparent member 12 is a transparent material (glass, plastic, film, etc.) that can transmit light from the outside world and reach the eyes of the observer so that the observer wearing the HMD device 10 can observe the outside world. It is formed by. The transparent member 12 is fitted and fixed to the rim of the frame 11 (the portion that supports the transparent member 12). The transparent member 12 may not be provided on the rim of the frame 11 and light from the outside world may directly reach the observer's eyes.

  The imaging device 13 images the outside of the imaging lens in the optical axis direction (hereinafter also referred to as “imaging direction”). For example, the imaging device 13 is a digital camera or a video camera provided with an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). For example, a captured image (moving image) such as JPEG or MPEG2 / 4 is generated by the imaging of the imaging device 13. In addition, the imaging device 13 of 1st embodiment is installed in the flame | frame 11 part which is not on an observer's eyes | visual_axis as shown in FIG.

  The imaging device 13 also functions as a distance measuring device that can measure the distance to a feature point (for example, a human face or an object having a characteristic shape) existing in the imaging direction. Note that the imaging device 13 may have any form, name, and structure as long as it can measure the distance to the feature point. A stereo camera, an array camera, a near-infrared light emitting element (near red) An external laser, etc.), a near-infrared light receiving element, an ultrasonic sensor or the like. In addition, the imaging device 13 may include sensors necessary for detecting the attitude of the HMD device 10, such as a triaxial acceleration sensor and a gyro sensor.

  The AR providing device 20 displays an image (AR image) that is additionally superimposed on light from the outside. For example, the AR providing apparatus 20 displays a balloon including information such as a name and a work place as if there is a person near the person existing in the viewing direction of the observer. Such a display is realized by adjusting the position, size, orientation, shape, color (saturation, brightness, contrast, etc.), shadow, etc. of the AR image. The AR providing apparatus 20 may be a binocular type instead of the monocular type as shown in FIG. A specific configuration of the AR providing apparatus 20 will be described later.

  The main body 30 has a configuration necessary for controlling the imaging device 13 and the AR providing device 20 to function as a head mounted display as a whole. A specific configuration of the main body 30 will be described later.

(Specific Configuration of AR Providing Device 20)
Next, a specific configuration of the AR providing device 20 will be described.

  As illustrated in FIG. 2, the AR providing device 20 includes a display device 21, a lens 22, and a light guide unit 23.

  The display device 21 displays an AR image. As shown in FIG. 2, the image light output from the display device 21 enters the light guide unit 23 through the lens 22. For example, the display device 21 employs an LED (Light Emitting Diode) display, a liquid crystal display, an organic EL (Electro Luminescence) display, and the like.

  Further, the display device 21 is installed to be movable along the optical axis direction (+ L direction or −L direction) shown in FIG. When the display device 21 moves in a direction away from the light guide 23 (+ L direction), the optical path length from the display device 21 to the eyes of the observer becomes longer. At this time, the observer seems to have moved the AR image in the far direction (+ Z direction). On the contrary, when the display device 21 moves in the direction approaching the light guide 23 (−L direction), the optical path length from the display device 21 to the eyes of the observer is shortened. At this time, the observer seems to have moved in the direction in which the AR image approaches (−Z direction).

  In the following, the midpoint between the observer's eyes is the coordinate origin (0, 0, 0), the horizontal direction as viewed from the observer is the X axis, and the vertical direction as viewed from the observer is the Y axis. The front-rear direction (front direction, depth direction) viewed from the viewpoint is taken as the Z axis. Such a coordinate system is referred to as an observer-based coordinate system XYZ.

  The lens 22 is composed of a plurality of lens groups. Each lens constituting this lens group is independently movable in the optical axis direction (+ L direction or -L direction). The focal length can be freely changed by adjusting the positional relationship of each lens. Accordingly, for example, the ratio (ΔZ / ΔL) of the movement amount ΔZ of the AR image (the position where the AR image appears to be present when viewed by the observer) to the movement amount ΔL of the display device 21 can be adjusted.

  The light guide 23 guides the image light from the display device 21 to the eyes of the observer. As shown in FIG. 2, the light guide 23 includes a transparent plate 24 and a first HOE (Holographic Optical Element) 25.

  The transparent plate 24 is a transparent plate formed of a material such as glass or resin having a refractive index larger than that of air.

  The first HOE 25 is an optical element using a hologram and has a property of reflecting only light of a specific wavelength and transmitting light of other wavelengths. In the first embodiment, the first HOE 25 reflects and reflects the image light from the display device 21 to the observer's eyes while diffracting and reflecting the image light, and transmits the light from the outside to the observer's eyes. Type holographic optical element. That is, the first HOE 25 functions as a combiner that enables simultaneous observation of the outside world and the AR image. In place of the first HOE 25, a half mirror, a polarization beam splitter, or the like may be used.

(Specific configuration of main body 30)
Next, a specific configuration of the main body 30 will be described.

  As shown in FIG. 3, the main body unit 30 includes a CPU (Central Processing Unit) 31, a memory 32, a storage 33, an input device 34, a drive device 35, and a display controller 36.

  The CPU 31 is a control circuit composed of a processor or the like that controls the above-described units and executes various arithmetic processes according to a program. Each function of the HMD device 10 is exhibited by the CPU 31 executing a corresponding program. The

  The memory 32 is a high-speed accessible main storage device that temporarily stores programs and data as a work area. As the memory 32, for example, a DRAM (Dynamic Random Access Memory), an SDRAM (Synchronous Dynamic Access Memory), an SRAM (Static Random Access Memory), or the like is adopted.

  The storage 33 is a large-capacity auxiliary storage device that stores various programs including an operating system and various data. For example, a flash memory or the like is employed as the storage 13.

  The input device 34 includes physical direction keys and buttons for inputting instructions for selecting and moving the AR image. Moreover, you may utilize an external controller (for example, a smart phone, a remote controller, etc.) as the input device 34. FIG.

  The drive device 35 is a device that moves the display device 21 and the lens 22 in the optical axis direction. For the drive device 35, for example, a stepping motor, a voice coil motor, or the like is employed.

  The display controller 36 causes the display device 21 to display an AR image or the like. For example, the display controller 36 reads out an AR image from the memory 32 every predetermined display period, converts it into a signal for input to the display device 21, and generates a pulse signal such as a horizontal synchronizing signal and a vertical synchronizing signal. .

  The HMD device 10 having the hardware configuration as described above has the following functional configuration.

  FIG. 4 is a block diagram illustrating a functional configuration example of the HMD device.

<HMD device 10 (functional configuration)>
As illustrated in FIG. 4, the HMD device 10 includes an imaging unit 41, a feature point selection unit 42, a distance measurement unit 43, a first reference calculation unit 44, a display control unit 45, and a second reference as functional configurations. A reception unit 46, a reference creation unit 47, and an AR application unit 48 are included.

  Here, the function units 41 to 47 in the broken line frame correct the deviation between the external world recognized from the captured image obtained by the imaging of the imaging device 13 and the external world actually observed by the observer. The present invention relates to a calibration function for creating a standard for the above. On the other hand, the AR application unit 48 relates to a function for displaying an AR image so that an AR image exists in the vicinity of an object in the direction of the observer's line of sight, using a reference created by a calibration function.

  Hereinafter, the functional units 41 to 48 will be described in detail. The function units 41 to 48 are realized by the CPU 31 reading out a program installed in the storage 33 to the memory 32 and executing the program. Further, the present invention is not limited to this, and hardware such as ASIC may be used.

  The imaging unit 41 controls the imaging device 13 to image the outside world. Thereby, a captured image is generated.

  The feature point selection unit 42 uses the captured image generated by the imaging unit 41 to select a feature point that exists in the outside world.

  The distance measuring unit 43 controls the imaging device 13 to measure the distance in the imaging direction from the installation position of the imaging device 13 to the feature point selected by the feature point selection unit 42.

  The first reference calculation unit 44 calculates the position of the feature point based on the installation position of the imaging device 13 as the first reference position.

  The display control unit 45 causes the display device 21 to display an AR image. For example, the display control unit 45 displays the AR marker at the first reference position calculated by the first reference calculation unit 44.

  The second reference acceptance unit 46 accepts designation of a second reference position at which both the feature point and the AR image can be viewed when the observer sees the direction of the feature point that exists in the outside world. . For example, the second reference receiving unit 46 first performs an operation of moving the AR marker displayed by the display control unit 45 to a position in the depth direction where the AR marker can be focused and viewed together with the feature points that exist in the outside world. Accept. Next, designation of the second reference position is accepted by an operation of moving the AR marker in a plane perpendicular to the depth direction (up / down / left / right direction).

  The reference creation unit 47 sets the first reference position based on the difference between the first reference position calculated by the first reference calculation unit 44 and the second reference position received by the second reference reception unit 46. Create reference data for correction to two reference positions.

  The AR application unit 48 provides various functions involving display of an AR image. For example, the AR application unit 48 displays an AR image (such as a balloon including information such as a name and work place) in the vicinity of an object (such as a person) in the observer's line of sight. At this time, the AR application unit 48 displays the AR image after correcting the display position by applying the reference data created by the reference creation unit 47.

  Detailed operations of the functional units 41 to 48 will be described in detail below.

<Operation of HMD Device 10>
FIG. 5 is a flowchart showing a procedure of calibration processing executed in the HMD apparatus. FIG. 6 is a diagram illustrating an example of a positional relationship between the HMD device mounted on the observer's head and an object existing in the imaging direction. FIG. 7 is a diagram illustrating an example of a captured image. FIG. 8 is a diagram for explaining a method of adjusting the position of the AR marker. FIG. 8A is a diagram showing the position of the AR marker before adjustment, and FIG. 8B is a diagram showing the position of the AR marker after adjustment. FIG. 9 is a diagram for explaining a method of adjusting the depth direction (Z direction) of the AR marker. 9A is a diagram illustrating an example of a captured image, FIG. 9B is a diagram illustrating an example of the field of view of an observer before AR marker adjustment, and FIG. 9C is a diagram illustrating an AR marker. It is a figure which shows the example of the observer's visual field after adjustment. FIG. 10 is a diagram for explaining an adjustment method of the AR marker in the left-right direction (X direction) and the up-down direction (Y direction). 10A is a diagram showing an example of the observer's field of view before the AR marker adjustment, and FIG. 10B is a diagram showing an example of the observer's field of view after the AR marker adjustment. (C) is a figure which shows the example of the visual field of the observer after AR marker adjustment about all the feature points.

  Hereinafter, the procedure of the calibration process will be described with reference to FIGS.

(Calibration process)
For example, when the power of the main body is turned on, the HMD device 10 starts the calibration process shown in FIG. However, the timing for starting the calibration process is not limited to this, and the calibration process may be started when an operation for starting the calibration process is performed on the input device 34. Further, the calibration process may be started when it is detected that the HMD device 10 is mounted on the observer's head.

[Step S101]
When the calibration process is started, the HMD device 10 functions as the imaging unit 41 and images the outside world. Specifically, the HMD device 10 controls the imaging device 13 to start imaging, thereby generating a captured image in the optical axis direction of the imaging lens.

  For example, as in the example shown in FIG. 6, it is assumed that an observer wearing the HMD device 10 on the head is observing a cube-shaped object that exists in the outside world. At this time, when the imaging device 13 captures an image of a landscape in the optical axis direction (Z ′ direction) of the imaging lens, a captured image as in the example illustrated in FIG. 7 is generated.

  In the following, as shown in FIGS. 6 and 7, the center of the imaging lens of the imaging device 13 is the coordinate origin (0, 0, 0), the horizontal axis of the imaging device 13 is the X ′ axis, and the vertical axis Is the Y ′ axis, and the optical axis (imaging direction) is the Z ′ axis. Such a coordinate system is referred to as an image pickup apparatus reference coordinate system X′Y′Z ′.

  Further, the HMD device 10 stores the generated captured image in the memory 32 or the storage 33.

[Step S102]
Next, the HMD device 10 functions as the feature point selection unit 42, and uses the captured image generated in step S101 to select feature points that exist in the outside world. For example, the HMD device 10 extracts a characteristic portion in the captured image as a feature point using a general method such as edge detection or face recognition. Thereafter, the HMD device 10 selects one feature point from the extracted feature points. For example, in the case of a captured image as in the example illustrated in FIG. 7, the feature point selection unit 42 selects the vertices of a cube-shaped object as feature points (A, B, C, D).

[Step S103]
Subsequently, the HMD device 10 functions as the first reference calculation unit 44, and calculates the position when the feature point selected in step S102 is based on the installation position of the imaging device 13 as the first reference position. . That is, the HMD device 10 calculates the position (three-dimensional coordinate) of the feature point in the imaging device reference coordinate system X′Y′Z ′ as the first reference position. Note that the X′Y ′ coordinate of the feature point is calculated using a captured image, and the Z ′ coordinate of the feature point is calculated using a value (distance) measured by the distance measuring unit 43.

  More specifically, the HMD device 10 sets the x coordinate and y coordinate of the feature point in the captured image generated in step S101 as the X ′ coordinate and the Y ′ coordinate of the first reference position.

  Furthermore, the HMD device 10 functions as the distance measuring unit 43 and measures the distance from the installation position of the imaging device 13 to the feature point. For example, the HMD device 10 controls the imaging device 13 to return to the feature point out of the laser light emitted from the near-infrared laser (not shown) so as to scan the field of view of the observer. Receives laser light. At this time, the distance in the depth direction to the feature point is calculated by measuring the time required for the laser beam to reciprocate to the feature point. When the imaging device 13 is a stereo camera or an array camera, the HMD device 10 calculates a distance in the depth direction to the feature point using a plurality of captured images captured by the stereo camera or the array camera. To do. Then, the HMD device 10 sets the distance in the depth direction calculated here as the Z ′ coordinate of the first reference position.

[Step S104]
Next, the HMD device 10 functions as the display control unit 45 and displays it as if an AR marker exists at the first reference position (X ′, Y ′, Z ′) calculated in step S103. For example, the HMD device 10 displays a black circle AR image as shown in FIG. 8A on the display device 21 as the AR marker M. At this time, the HMD device 10 assumes that the observer-based coordinate system XYZ and the imaging device-based coordinate system X′Y′Z ′ are the same, and calculates the first reference position ( X ′, Y ′, Z ′) is displayed as if the AR marker M exists. In the following, the coordinates (two-dimensional coordinates) on the display device 21 when the AR marker M is displayed as if present at the first reference position are set as the first reference coordinates K (x, y).

  Note that the HMD device 10 displays the display position, size, orientation, shape, and color of the AR marker in order to display the AR marker as if it exists at the first reference position (X ′, Y ′, Z ′). (Saturation, brightness, contrast, etc.), shadows, etc. are adjusted. More specifically, regarding the display position, the distance from an object (for example, a cube-shaped object, etc.) existing around the AR marker may be adjusted. The size, orientation, and shape may be adjusted according to a general perspective method. As for colors, it is only necessary to adjust the saturation, lightness, and contrast so that distant objects appear more blurred. For the shadow, the direction and length of the shadow may be adjusted.

  However, since the first reference position calculated in step S103 is based on the position of the imaging device 13 and is different from the viewpoint of the observer, it is displayed as if the AR marker exists at the first reference position. Even so, it may deviate from the expected position. For example, even if it is displayed as if the AR marker M is present at the first reference position shown in FIG. 9 (A), as shown in FIG. 8 (A) and FIG. , The AR marker M may appear to deviate from the feature point A.

[Step S105]
Accordingly, the HMD device 10 functions as the second reference receiving unit 46, and when the observer sees the direction of the feature point, the second reference position where both the feature point and the AR marker can be seen in focus. The specification of is accepted. For example, the HMD device 10 uses the AR marker M displayed on the display device 21 as the feature point A while the observer looks at the direction of one feature point A as in the example shown in FIG. The designation of the second reference position is accepted by a manual operation for moving to the overlapping position. Note that the coordinates (two-dimensional coordinates) on the display device 21 when the AR marker M is displayed as if it exists at the second reference position are the second reference coordinates K ′ (x ′, y ′). Put.

  In order to move the AR marker to the second reference position, the driving device 35 of the HMD device 10 first moves the display device 21 and the lens 22 in the optical axis direction according to the operation of the input device 34 by the observer (FIG. 2). In the L direction). Accordingly, the AR marker appears to move in the depth direction (Z direction) as viewed from the observer. Finally, as shown in FIG. 9C, the driving device 35 allows the observer to focus on the AR marker M and the feature point A in accordance with the operation of the input device 34 by the observer. The AR marker M is moved in the depth direction to the position determined to be. As a result, the observer can see both the feature point A and the AR marker M in focus.

  However, even if it is possible to see both the feature point and the AR marker in focus, as shown in FIGS. 9C and 10A, the left and right direction (X direction) as viewed from the observer. , It may appear shifted in the vertical direction (Y direction).

  Therefore, the driving device 35 of the HMD device 10 next moves the display position of the AR marker in the horizontal direction (X direction) and the vertical direction (Y direction) according to the operation of the input device 34 by the observer. Finally, as shown in FIG. 10B, the drive device 35 moves the AR marker M left and right until the observer overlaps the AR marker M and the feature point A according to the operation of the input device 34 by the observer. Move it up and down. Then, the HMD device 10 sets the position (X, Y, Z) when the AR marker M overlaps with the feature point A as the second reference position.

[Step S106]
Next, the HMD device 10 determines whether or not the four feature points have been processed. Specifically, the HMD device 10 determines that the first reference coordinate K and the second reference coordinate K ′ have been obtained for the four feature points, and has determined that the processing has been completed. Determines that it has not been processed.

  If it is determined that the four feature points have not been processed (step S106: NO), the HMD device 10 repeats the processing from step S101 to step S105 until the four feature points have been processed. As a result, the first reference position is also calculated for the remaining feature points (B, C, D, etc.), and the second reference position is also designated (FIG. 10C). Then, the HMD device 10 obtains the first reference coordinate K and the second reference coordinate K ′ for the four feature points (A, B, C, D).

  If the HMD device 10 determines that the four feature points have been processed (step S106: YES), the process proceeds to step S107.

[Step S107]
The HMD device 10 functions as the reference creation unit 47, and sets the second reference position to the second reference position based on the difference between the first reference position calculated in step S103 and the second reference position received in step S105. Create reference data for correction to the reference position.

  Specifically, the HMD device 10 changes the first reference coordinates K (x, y) for all feature points (A, B, C, D) to the second reference coordinates K ′ (x ′, y ′). A convertible homography matrix H is obtained. That is, the reference creation unit 47 obtains a homography matrix H that satisfies HK = sK ′. Here, the homography matrix H is a 3 × 3 matrix, and s is a scale factor. Specifically, HK = sK ′ can be described as in Equation 1 below.

  A method for obtaining the homography matrix H satisfying Equation 1 will be briefly described below. Specifically, the homography matrix H is obtained by the following procedures (i) to (iv).

  (I) The HMD device 10 substitutes “1” for the scale factor s of Equation 1 in order to fix the scaling of the homography. As a result, the following formula 2 is obtained.

  (Ii) The HMD device 10 decomposes the determinant of Equation 2 and eliminates s to obtain Equation 3 below.

  (Iii) The HMD device 10 obtains the following formula 4 when formula 3 is arranged and expressed in the form of a matrix.

(Iv) The HMD device 10 obtains parameters h ₁₁ to h ₃₂ by obtaining an inverse matrix for the 6 × 2 matrix of Equation 4 and multiplying the obtained inverse matrix on both sides from the left.

[Step S108]
Subsequently, the HMD device 10 functions as the reference creation unit 47, and stores the homography matrix H having the parameters h11 to h32 obtained in step S107 as elements in the memory 32 and the storage 33 as reference data.

  Thereafter, the HMD device 10 ends the calibration process.

  By executing the above calibration processing in the HMD device 10, it is possible to create a reference for correcting a deviation between the outside world recognized from the captured image and the outside world actually observed by the observer. it can. If the AR application unit 48 corrects the display position of the AR image using the created reference, the AR image is present in the vicinity of the object being observed by the observer without complicating the optical system. Display as if it is. As a result, the observer can focus and observe both the object and the AR image at the same time, and the eyes are not tired.

  Note that each processing unit of the above-described flowchart is divided according to main processing contents in order to facilitate understanding of the HMD device 10. The invention of the present application is not limited by the method of classification of the processing steps and the names thereof. The processing performed in the HMD device 10 can be divided into more processing steps. One processing step may execute more processes.

<Second embodiment>
In the first embodiment described above, when there are a plurality of objects (for example, a person) in the imaging direction (Z direction), the object being watched by the observer cannot be accurately identified, and the AR image It is impossible to determine in which object the object should be displayed.

  Therefore, in order to solve this problem, the HMD device 10 according to the second embodiment detects an observer's line-of-sight direction and enables an AR image to be displayed in the vicinity of an object existing ahead of the line-of-sight direction. .

  In the following description, differences from the first embodiment will be mainly described, and the same reference numerals will be given to the same configurations, and description thereof will be omitted.

  FIG. 11 is a schematic configuration diagram of an AR providing apparatus according to the second embodiment.

  The HMD device 10 according to the second embodiment includes an AR providing device 20 that is partially different from the first embodiment. As shown in FIG. 11, the AR providing device 20 includes a line-of-sight detection device 26 and a second HOE 27 in addition to the configuration of the display device 21, the lens 22, the light guide 23, the transparent plate 24, and the first HOE 25.

  The line-of-sight detection device 26 includes an illumination device 26A and an imaging device 26B for line-of-sight detection, and the second HOE 27 has the same function as the first HOE 25.

  As shown in FIG. 11, the irradiation light (broken line) irradiated from the illumination device 26 </ b> A is reflected by the second HOE 27 and travels in the direction of the transparent plate 24. Then, the irradiation light travels toward the first HOE 25 while being totally reflected in the light guide portion 23, is reflected by the first HOE 25, and reaches the observer's eyes. The reflected light (solid line) reflected by the observer's eyes is reflected by the first HOE 25 and travels toward the second HOE 27 while totally reflecting the light guide 23 in the direction opposite to the irradiation light. Then, it is reflected by the second HOE 27 and reaches the imaging device 26B.

  The HMD device 10 functions as the imaging unit 41 and obtains a captured image related to the eyes of the observer by controlling the line-of-sight detection device 26. The HMD device 10 functions as the AR application unit 48, analyzes a captured image related to the eyes of the observer, and detects the direction of the observer's line of sight.

  Next, an AR image display process executed in the HMD device 10 according to the second embodiment having the above-described line-of-sight detection function will be described.

  FIG. 12 is a flowchart illustrating the procedure of the AR image display process. FIG. 13 is a diagram illustrating a display example when the display position of the AR image is changed according to the object that the observer is gazing at. FIG. 13A is a display example when the observer is looking at “Mr. A”, and FIG. 13B is a display example when the observer is looking at “Mr. B”.

  Hereinafter, the procedure of the AR image display process will be described with reference to FIGS. 12 and 13.

(AR image display processing)
For example, the HMD device 10 starts the AR image display process shown in FIG. 12 when the calibration process of the first embodiment is completed. However, the timing of starting the AR image display process is not limited to this, and the AR image display process may be started when an operation for starting the AR image display process is performed on the input device 34. .

[Step S201]
When the AR image display process is started, the HMD device 10 functions as the imaging unit 41 and images the observer's eyes. Specifically, the HMD device 10 controls the illumination device 26A and the imaging device 26B to start imaging, thereby generating a captured image related to the observer's eyes. The HMD device 10 stores the generated captured image in the memory 32 or the storage 33.

[Step S202]
Next, the HMD device 10 functions as the AR application unit 48, analyzes the captured image generated in step S201, and recognizes an object that can be an object for displaying the AR image in the vicinity. For example, the HMD device 10 extracts a characteristic region (for example, a face or the like) in the captured image using a general method such as edge detection or face recognition.

[Step S203]
In step S202, if there is no target object that displays an AR image in the vicinity in step S202 (step S203: NO), the HMD device 10 repeats the processes of steps S201 and S202 until the target object is found.

  Then, when at least one or more objects are found (step S203: YES), the HMD device 10 advances the process to step S204.

[Step S204]
The HMD device 10 functions as the AR application unit 48 and detects the observer's line of sight. For example, the HMD device 10 obtains a captured image relating to the eyes of the observer using the AR providing device 20 illustrated in FIG. Then, the HMD device 10 measures the positional relationship between the light guide 23 and the pupil or iris of the observer's eye based on the obtained captured image, and based on the positional relationship, the observer's viewpoint and Detect gaze direction.

[Step S205]
Next, the HMD device 10 functions as the AR application unit 48 and determines whether or not there is an object for displaying an AR image in the vicinity of the viewer in the direction of the line of sight detected in step S204. For example, the HMD device 10 may determine by the following processes (α) to (γ).

  (Α) A straight line passing through the observer's viewpoint (coordinates) and extending in the observer's line-of-sight direction (direction vector) detected in step S204 is obtained.

  (Β) A region (range) in the real space (three-dimensional space) occupied by the object recognized in step S202 is obtained. For example, similar to the processing in step S103 described above, the image can be obtained by using the captured image obtained by the imaging device 13 and the distance in the depth direction calculated by the imaging device 13.

  In order to simplify the process, the coordinates of a point representing the object may be obtained instead of the area occupied by the object.

  (Γ) If the straight line obtained in (α) intersects with the region obtained in (β), it is determined that there is an object for displaying an AR image in the vicinity of the observer's line-of-sight direction. On the other hand, if the straight line obtained in the above (α) does not intersect the area obtained in the above (β), it is determined that there is no object for displaying the AR image in the vicinity of the observer's line of sight.

  When only the point representing the object is obtained in (β), the distance between the straight line obtained in (α) and the representative point obtained in (β) may be obtained. If the distance obtained here is within a predetermined threshold, there is an object for displaying the AR image in the vicinity of the viewer's line of sight.

  If there is no object that displays the AR image in the vicinity of the observer's line of sight (step S205: NO), the HMD device 10 does not need to display the AR image, so the process is performed in step S201. Return to. On the other hand, if there is an object for displaying the AR image in the vicinity of the observer's line of sight (step S205: YES), the process proceeds to step S206.

[Step S206]
The HMD device 10 functions as the AR application unit 48 and corrects the display position of the AR image. Specifically, the HMD device 10 corrects the display position of the AR image by applying the reference data created by the calibration process of the above embodiment. For example, if the position (two-dimensional coordinates) on the display device 21 when the AR image is displayed near the object being observed by the observer is converted by the homography matrix H calculated in step S107 described above. Good.

[Step S207]
The HMD device 10 displays the AR image at the display position (position represented by two-dimensional coordinates on the display device 21) after being corrected by the process of step S206. By the processing in step S207, it is possible to display as if the AR image is present in the vicinity of the object being observed by the observer. For example, as shown in FIG. 13A, a name (a speech bubble “Mr. A”) can be displayed next to the person in front. At this time, the observer can observe both Mr. A who exists in the line-of-sight direction and the name (AR image) in focus at the same time. Further, the HMD device 10 does not display an AR image next to Mr. B (a person existing behind Mr. A) who does not exist in the viewing direction of the observer. This prevents display of an AR image that cannot be observed in focus at the same time as Mr. A who exists in the line-of-sight direction.

[Step S208]
The HMD device 10 functions as the AR application unit 48 and determines whether or not an instruction to end the AR image display process has been given by the observer. Specifically, the HMD device 10 may determine whether or not the input device 34 has been operated to end the AR image display process.

  If there is no instruction to end the AR image display process (step S208: NO), the HMD device 10 returns the process to step S201 and continues the AR image display process. At this time, if the observer's line-of-sight direction changes in the subsequent step S204, the HMD device 10 displays the name ("Mr. B" next to the person in the back where the line-of-sight is newly directed, as shown in FIG. Can be displayed (steps S206 and S207). Also in this case, the observer can observe by focusing on both Mr. B existing in the new line-of-sight direction and the name (AR image) at the same time. Further, the HMD device 10 does not display an AR image beside Mr. A who is not in the observer's new line-of-sight direction.

  On the other hand, when an instruction to end the AR image display process is given (step S208: YES), the HMD device 10 ends the AR image display process.

  By executing the above AR image display processing in the HMD device 10, it is possible to change the display target of the AR image while detecting the line of sight of the observer. Therefore, it is possible to display as if the AR image is present in the vicinity of the target object being watched by the observer. As a result, the observer can always focus and observe both the object in the line of sight and the AR image at the same time, and the eyes are not tired.

<Modification>
In addition, each said embodiment intends to illustrate the summary of this invention, and does not limit this invention. Many alternatives, modifications, and variations will be apparent to those skilled in the art.

  For example, in the first embodiment, as shown in FIG. 7, only the AR marker is displayed when the AR marker is adjusted. Therefore, it is difficult for the observer to understand what kind of work should be performed.

  Therefore, as a modification of the display on the display device 21 during the adjustment of the AR marker, the HMD device 10 displays, on the display device 21, the point A ′ that the observer should look at during the adjustment of the AR marker together with the AR marker. May be. FIG. 14 is a diagram showing a modification of the display on the display device 21 when adjusting the AR marker. By adding and displaying the image within the thick frame in FIG. 14, the observer can easily grasp where to view and adjust the AR marker.

  In addition to the AR marker, the HMD device 10 may display, for example, a message or a graphic for urging the observer to make adjustments on the display device 21.

  Further, in each of the above embodiments, the frame 11 of the HMD device 10 has a glasses shape. However, the present invention is not limited to this. The frame 11 only needs to be able to mount the HMD device 10 on the observer's head and to place the AR providing device 20 in front of the observer, and the name, shape, dimensions, material, and the like are not particularly limited.

  In the first embodiment, the coordinate origin is the midpoint between the eyes of the observer in the observer-based coordinate system XYZ. However, the present invention is not limited to this. For example, the observer reference coordinate origin may be the center position of the dominant eyeball, or may be the position designated by the observer as the center position of both eyes by manual operation.

  In the first embodiment, the second reference position is designated by moving the AR marker. However, the present invention is not limited to this, and the second reference position may be designated by a method other than the method of moving the AR marker. For example, instead of the AR marker, a second reference position may be designated by moving, rotating, or changing the scale of an AR image of a figure such as a line or a plane.

  In the first embodiment, it is preferable that the feature points (A, B, C, D) selected in step S102 be as far as possible from the position of the observer. For example, each feature point (A, B, C, D) is preferably four points in a plane subject.

  FIG. 15 is a diagram illustrating an example of a captured image and an observer's field of view when the observer's head is shaken. FIG. 15A is a diagram illustrating an example of a captured image when the observer's head is shaken, and FIG. 15B is an example of the observer's field of view when the observer's head is shaken. FIG.

  As shown in FIG. 15A, in the first embodiment described above, the observer's head is shaken during the adjustment of the AR marker, and the feature point moves in the captured image captured by the imaging device 13. There is. However, even when the observer's head is shaken in this way, as shown in FIG. 15B, the display position of the AR marker is linked to the position of the feature point in the captured image. As in the case where the head is fixed without shaking, the AR marker can be adjusted.

  The above-described configuration of the HMD device 10 has described the main configuration in describing the features of the above-described embodiments and modifications, and is not limited to the above-described configuration. Further, the configuration of the general HMD device 10 is not excluded.

  In addition, each functional configuration of the HMD device 10 described above is classified according to main processing contents in order to facilitate understanding of each functional configuration. The present invention is not limited by the way of classification and names of the constituent elements. Each functional configuration can be classified into more components according to the processing content. Moreover, it can also classify | categorize so that one component may perform more processes.

  The processing of each functional configuration of the HMD device 10 described above can also be realized by a dedicated hardware circuit. In this case, it may be executed by one hardware or a plurality of hardware.

  The program for operating the HMD device 10 may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, or a CD-ROM, or may be provided online via a network such as the Internet. . In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in the memory 32, the storage 33, or the like. Further, this program may be provided as, for example, a single application software, or may be incorporated in the software of each device as one function of the HMD device 10.

10 HMD device,
11 frames,
12 Transparent member,
13 imaging device,
20 AR providing device,
21 display device,
22 lenses,
23 light guide,
24 Transparent plate,
25 First HOE,
26 gaze detection device,
26A lighting device,
26B An imaging device for detecting a line of sight,
27 Second HOE,
30 body part,
31 CPU,
32 memory,
33 storage,
34 input devices,
35 drive unit,
36 Display controller,
41 imaging unit,
42 feature point selection unit,
43 Ranging section,
44 1st standard calculation part,
45 Display control unit,
46 Second Standard Reception Department,
47 Standard creation department,
48 AR application part,
MAR marker.

Claims (15)

An image display device that projects image light according to content information to an observer's eye while transmitting light from the outside world,
An imaging unit that images the outside world and generates a captured image;
Using the captured image generated by the imaging unit, a selection unit that selects feature points that exist in the outside world; and
A distance measuring unit for measuring a distance from an installation position of the imaging unit to the feature point;
Using the distance measured by the distance measuring unit, a calculation unit that calculates the position of the feature point when the installation position of the imaging unit is used as a reference, and a first reference position;
When the observer sees the direction of the feature point, the observer sees that the feature point and the content information are the second reference position where the feature point and the content information can both be viewed in focus in the depth direction. A reception unit that accepts the designation;
For correcting the first reference position to the second reference position based on a deviation between the first reference position calculated by the calculation unit and the second reference position received by the receiving unit. A reference creation unit for creating reference data;
An image display device comprising: An image display device that projects image light according to content information to an observer's eye while transmitting light from the outside world,
An imaging unit that images the outside world and generates a captured image;
Using the captured image generated by the imaging unit, a selection unit that selects feature points that exist in the outside world; and
A calculation unit that calculates the position of the feature point when the installation position of the imaging unit is a reference, as a first reference position;
When the observer sees the direction of the feature point, the observer sees that the feature point and the content information are the second reference position where the feature point and the content information can both be viewed in focus in the depth direction. A reception unit that accepts the designation;
For correcting the first reference position to the second reference position based on a deviation between the first reference position calculated by the calculation unit and the second reference position received by the receiving unit. A standard creation unit for creating standard data ,
The imaging unit is a stereo camera or an array camera,
The image display device , wherein the calculation unit calculates the first reference position using a plurality of captured images generated by the stereo camera or the array camera . An image display device that projects image light according to content information to an observer's eye while transmitting light from the outside world,
An imaging unit that images the outside world and generates a captured image;
Using the captured image generated by the imaging unit, a selection unit that selects feature points that exist in the outside world; and
A calculation unit that calculates the position of the feature point when the installation position of the imaging unit is a reference, as a first reference position;
When the observer sees the direction of the feature point, the observer sees that the feature point and the content information are the second reference position where the feature point and the content information can both be viewed in focus in the depth direction. A reception unit that accepts the designation;
For correcting the first reference position to the second reference position based on a deviation between the first reference position calculated by the calculation unit and the second reference position received by the receiving unit. A standard creation unit for creating standard data ,
The reception unit moves the displayed content information in the depth direction to a position where the content point can be focused and viewed together with the feature point, and then moves the content information in a plane perpendicular to the depth direction. An image display device that accepts designation of the second reference position by an operation of the observer that is moved so as to overlap with a feature point . A display unit for displaying the content information so as to exist at the first reference position calculated by the calculation unit;
The said reception part receives designation | designated of said 2nd reference position by operation of the said observer who moves the said content information displayed on the said display part to the position of the said feature point , Any one of Claims 1-3 the image display apparatus according to an item or. A correction unit that corrects the display position of new content information using the reference data created by the reference creation unit;
Wherein the correction unit in accordance with the object to the observer is gazing, to change the display position of the new content information, the image display apparatus according to any one of claims 1-4. An image display method for projecting image light according to content information to an observer's eye while transmitting light from the outside world,
(A) imaging the outside world to generate a captured image;
(B) using the captured image generated in step (a), selecting a feature point that exists in the outside world;
(C) measuring the distance from the installation position of the imaging device to the feature point selected in step (b);
( D ) Using the distance measured in step (c) as the first reference position, the position of the feature point when the installation position of the imaging device that performs imaging in step (a) is used as a reference A calculating step;
( E ) When the observer sees the direction of the feature point, the feature point and the content information are in a depth direction, and the second reference position can be viewed in focus in the depth direction. A step of accepting an observer's designation;
( F ) Based on the difference between the first reference position calculated in step ( d ) and the second reference position received in step ( e ), the first reference position is changed to the second reference position. Creating reference data for correcting the reference position;
Image display method. An image display method for projecting image light according to content information to an observer's eye while transmitting light from the outside world,
(A) imaging the outside world to generate a captured image;
(B) using the captured image generated in step (a), selecting a feature point that exists in the outside world;
( D ) calculating the position of the feature point as a first reference position when the installation position of the imaging device that performs imaging in step (a) is used as a reference;
( E ) When the observer sees the direction of the feature point, the feature point and the content information are in a depth direction, and the second reference position can be viewed in focus in the depth direction. A step of accepting an observer's designation;
( F ) Based on the difference between the first reference position calculated in step ( d ) and the second reference position received in step ( e ), the first reference position is changed to the second reference position. Creating reference data for correcting to the reference position ,
The imaging device is a stereo camera or an array camera,
The step (d) is an image display method in which the first reference position is calculated using a plurality of captured images generated by the stereo camera or the array camera . An image display method for projecting image light according to content information to an observer's eye while transmitting light from the outside world,
(A) imaging the outside world to generate a captured image;
(B) using the captured image generated in step (a), selecting a feature point that exists in the outside world;
( D ) calculating the position of the feature point as a first reference position when the installation position of the imaging device that performs imaging in step (a) is used as a reference;
( E ) When the observer sees the direction of the feature point, the feature point and the content information are in a depth direction, and the second reference position can be viewed in focus in the depth direction. A step of accepting an observer's designation;
( F ) Based on the difference between the first reference position calculated in step ( d ) and the second reference position received in step ( e ), the first reference position is changed to the second reference position. Creating reference data for correcting to the reference position ,
The step (e) moves the displayed content information in the depth direction to a position where it can be seen together with the feature point, and then moves the content information in a plane perpendicular to the depth direction. An image display method for accepting designation of the second reference position by an operation of the observer, wherein the second reference position is moved so as to overlap the feature point . ( G ) further performing a step of displaying the content information so as to exist at the first reference position calculated in the step ( d );
Wherein step (e) is the content information displayed in the step (g), is moved to the position of the feature point, by the observer's operation, it receives designation of the second reference position, according to claim 6 The image display method as described in any one of -8 . (H) further performing the step of correcting the display position of the new content information using the reference data created in the step ( f ),
Wherein step (h), depending on the object to the observer is gazing, to change the display position of the new content information, the image display method according to any one of claims 6-9. To a computer that functions as an image display device that projects image light according to content information to the viewer's eyes while transmitting light from the outside world,
(A) imaging the outside world to generate a captured image;
(B) using the captured image generated in step (a), selecting a feature point that exists in the outside world;
(C) measuring the distance from the installation position of the imaging device to the feature point selected in step (b);
( D ) Using the distance measured in step (c) as the first reference position, the position of the feature point when the installation position of the imaging device that performs imaging in step (a) is used as a reference A calculating step;
( E ) When the observer sees the direction of the feature point, the feature point and the content information are in a depth direction, and the second reference position can be viewed in focus in the depth direction. A step of accepting an observer's designation;
( F ) Based on the difference between the first reference position calculated in step ( d ) and the second reference position received in step ( e ), the first reference position is changed to the second reference position. Creating reference data for correcting the reference position;
An image display program that executes To a computer that functions as an image display device that projects image light according to content information to the viewer's eyes while transmitting light from the outside world,
(A) imaging the outside world to generate a captured image;
(B) using the captured image generated in step (a), selecting a feature point that exists in the outside world;
( D ) calculating the position of the feature point as a first reference position when the installation position of the imaging device that performs imaging in step (a) is used as a reference;
( E ) When the observer sees the direction of the feature point, the feature point and the content information are in a depth direction, and the second reference position can be viewed in focus in the depth direction. A step of accepting an observer's designation;
( F ) Based on the difference between the first reference position calculated in step ( d ) and the second reference position received in step ( e ), the first reference position is changed to the second reference position. Creating reference data for correcting to a reference position ; and
The imaging device is a stereo camera or an array camera,
The step (d) is an image display program for calculating the first reference position using a plurality of captured images generated by the stereo camera or the array camera . To a computer that functions as an image display device that projects image light according to content information to the viewer's eyes while transmitting light from the outside world,
(A) imaging the outside world to generate a captured image;
(B) using the captured image generated in step (a), selecting a feature point that exists in the outside world;
( D ) calculating the position of the feature point as a first reference position when the installation position of the imaging device that performs imaging in step (a) is used as a reference;
( E ) When the observer sees the direction of the feature point, the feature point and the content information are in a depth direction, and the second reference position can be viewed in focus in the depth direction. A step of accepting an observer's designation;
( F ) Based on the difference between the first reference position calculated in step ( d ) and the second reference position received in step ( e ), the first reference position is changed to the second reference position. Creating reference data for correcting to a reference position ; and
The step (e) moves the displayed content information in the depth direction to a position where it can be seen together with the feature point, and then moves the content information in a plane perpendicular to the depth direction. An image display program for accepting designation of the second reference position by an operation of the observer, which is moved so as to overlap the feature point . ( G ) causing the computer to further execute a step of displaying the content information so as to exist at the first reference position calculated in the step ( d );
Wherein step (e) is the content information displayed in the step (g), is moved to the position of the feature point, by the observer's operation, receives designation of the second reference position, according to claim 11 The image display program according to any one of to 13. (H) causing the computer to further execute a step of correcting the display position of the new content information using the reference data created in the step ( f ),
The image display program according to any one of claims 11 to 14 , wherein the step (h) changes a display position of the new content information in accordance with an object being watched by the observer.