JP2015061200A - Image display device, image display method, and image display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display device, an image display method, and an image display program capable of display natural for an observer even when a camera swings.SOLUTION: An image display device 10 to be used while being worn by an observer comprises: an inertia sensor 14 which is installed at a first position A of a body portion (frame 11) and detects swinging of the observer; an imaging apparatus 13 which is installed at a second position B different from the first position A of the body portion and generates an imaged image by imaging an outside world; a swing amount calculation unit which calculates the swing amount of the imaging apparatus 13 on the basis of the swing amount detected by the inertia sensor 14; and a tracking unit which tracks a predetermined target in the imaged image on the basis of the swing amount of the imaging apparatus 13 calculated by the swing amount calculation unit.

Description

  The present invention relates to an image display device, an image display method, and an image display program.

  There is a display system that superimposes and displays a virtual object drawn with additional information or computer graphics (CG) on a captured image of a real space. According to such a display system, it is possible to make it appear to the observer as if additional information or a CG-drawn virtual object exists in the real space. In general, such a technique is referred to as mixed reality, augmented reality, or the like, but hereinafter referred to as AR technique. As a typical apparatus using the AR technology, there is a head mounted display (HMD: Head Mounted Display) or the like.

  Some head mounted displays perform distortion correction for correcting distortion caused by characteristics of a camera lens (for example, Patent Document 1). In addition, there are some that perform object recognition, tracking, and the like on a target (for example, a person) existing in front of the observer using a captured image obtained by a camera that captures the visual field range of the observer.

JP 2010-91870 A

  However, since the head mounted display is used by being attached to an observer (for example, the head), the camera is also shaken when the observer shakes. At this time, if distortion correction or the like is performed without taking camera shake into consideration, the object recognition or tracking target cannot be accurately recognized. For example, when an observer uses a head-mounted display while walking, it is difficult to continue to accurately recognize an object recognition or tracking target.

  If a target for object recognition or tracking is mistakenly recognized, a pop-up image is displayed near the target, resulting in an unnatural image. In addition, when processing that requires accurate recognition of the position and shape of the target, such as processing for removing a specific target region from the captured image (cancellation image processing), the image becomes more unnatural. End up.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image display device, an image display method, and an image display program that enable natural display for an observer even when the camera shakes. To do.

  (1) An image display device that is used by being attached to an observer, and is installed at a first position of the main body part, and detects a shake of the observer, and the first position of the main body part. An image pickup unit that is installed at a second position different from the image pickup unit that picks up the outside world and generates a picked-up image, and a shake amount calculation that calculates a shake amount of the image pickup unit based on a shake amount detected by the shake detection unit An image display device comprising: a tracking unit that tracks a predetermined target in the captured image based on the swing amount of the imaging unit calculated by the swing amount calculation unit.

  (2) The image according to (1), wherein the imaging unit is a camera that uses a camera lens during imaging, and further includes a distortion correction unit that corrects distortion generated in the captured image due to characteristics of the camera lens. Display device.

  (3) The image display according to (2), wherein the distortion correction unit limits a range of the captured image in which distortion is corrected based on the shake amount of the imaging unit calculated by the shake amount calculation unit. apparatus.

  (4) The image display device according to any one of (1) to (3), wherein the tracking unit compares the plurality of captured images captured at different timings to track the predetermined target. .

  (5) The tracking unit limits the range of the captured image to be compared to track the predetermined target based on the shaking amount of the imaging unit calculated by the shaking amount calculation unit. ) Image display device.

  (6) The image display device according to any one of (2) to (5), wherein the distortion correction by the distortion correction unit and the tracking by the tracking unit are performed in parallel.

  (7) The shake detection unit that is installed at the first position of the main body portion of the image display device that is used by being attached to the observer is different from the first position of the main body portion. An image display method in an image display device, comprising: an imaging unit that is installed at a second position and that captures an image of the outside world to generate a captured image, and (a) based on a shaking amount detected by the shaking detection unit Performing a step of calculating a shaking amount of the imaging unit, and a step of (b) tracking a predetermined target in the captured image based on the shaking amount of the imaging unit calculated in the step (a). Image display method.

  (8) The imaging unit according to (7), wherein the imaging unit is a camera that uses a camera lens at the time of imaging, and (c) further corrects distortion generated in the captured image due to characteristics of the camera lens. Image display method.

  (9) The image according to (8), wherein the step (c) limits a range of the captured image for correcting distortion based on the amount of shaking of the imaging unit calculated in the step (a). Display method.

  (10) The image according to any one of (7) to (9), wherein the step (b) tracks the predetermined target by comparing a plurality of the captured images captured at different timings. Display method.

  (11) The step (b) limits the range of the captured image to be compared for tracking the predetermined target based on the amount of shaking of the imaging unit calculated in the step (a). The image display method according to (10).

  (12) The image display method according to any one of (8) to (11), wherein the distortion correction in step (c) and the tracking in step (b) are performed in parallel.

  (13) The shake detection unit that is installed at the first position of the main body portion of the image display device that is used by being attached to the observer is different from the first position of the main body portion. A computer that functions as an image display device that is installed at a second position and that captures the outside world to generate a captured image. (A) Based on the amount of shaking detected by the shaking detection unit, A step of calculating an amount of shaking of the imaging unit; and (b) tracking a predetermined target in the captured image based on the amount of shaking of the imaging unit calculated in step (a). Image display program.

  (14) The imaging unit is a camera that uses a camera lens during imaging, and (c) further causes the computer to execute a step of correcting distortion generated in the captured image due to characteristics of the camera lens. ) Image display program.

  (15) The image according to (14), wherein the step (c) limits a range of the captured image whose distortion is corrected based on the shaking amount of the imaging unit calculated in the step (a). Display program.

  (16) The image according to any one of (13) to (15), wherein the step (b) compares the plurality of captured images captured at different timings to track the predetermined target. Display program.

  (17) The step (b) limits the range of the captured image to be compared to track the predetermined target based on the shaking amount of the imaging unit calculated in the step (a). The image display program according to (16).

  (18) The image display program according to any one of (14) to (17), wherein the distortion correction in the step (c) and the tracking in the step (b) are performed in parallel.

  According to the present invention, when the shaking of the observer is detected, the shaking amount of the imaging unit is calculated, and a specific target (for example, a person) in the captured image is tracked based on the shaking amount. This makes it possible to accurately recognize the position and shape of a specific target in the captured image. As a result, even when a pop-up image is displayed or cancel image processing is performed, a natural display for the observer is possible.

It is a figure which shows the example of an external appearance of the head mounted display (HMD) apparatus which concerns on this embodiment. It is a block diagram which shows the hardware structural example of an HMD apparatus. It is a block diagram which shows the function structural example of an HMD apparatus. It is a flowchart which shows the procedure of AR image display processing. It is a figure which shows the example of the coordinate system set by the HMD apparatus with which the observer's head was mounted | worn, and initial setting. It is a figure which shows the example of a captured image. It is a figure for demonstrating the method to calculate the shaking amount of an imaging device. It is a figure for demonstrating distortion correction. It is a figure for demonstrating a tracking process. It is a figure for demonstrating the method of performing distortion correction and a tracking process with respect to the one part pixel of a captured image. It is a figure which shows the example of a display of AR image. It is a figure which shows the example of a cancellation image.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios.

  FIG. 1 is a diagram illustrating an appearance example of a head mounted display (HMD) device according to the present embodiment. FIG. 2 is a block diagram illustrating a hardware configuration example of the HMD device.

  Hereinafter, with reference to FIG. 1 and FIG. 2, a schematic configuration of the HMD device 10, particularly a hardware configuration will be described.

<HMD device 10 (hardware configuration)>
The HMD device 10 according to the present embodiment is an image display device that is used by being mounted on the head of a user (hereinafter also referred to as “observer”). For example, the HMD device 10 is a see-through type image display device that projects image light according to content information desired to be presented to an observer's eyes as an AR (Augmented Reality) image while transmitting light from the outside world. The AR image is displayed as if it exists in the actual environment (real space) that the observer is observing.

  As shown in FIG. 1, the HMD device 10 includes a frame 11, a transparent member 12, an imaging device 13, an inertial sensor 14, an AR providing device 20, and a main body 30.

  The frame 11 has a shape that allows the HMD device 10 to be mounted on the observer's head, and has, for example, an eyeglass shape.

  The transparent member 12 is a transparent material (glass, plastic, film, etc.) that can transmit light from the outside world and reach the eyes of the observer so that the observer wearing the HMD device 10 can observe the outside world. It is formed by. The transparent member 12 is fitted and fixed to the rim of the frame 11 (the portion that supports the transparent member 12). The transparent member 12 may not be provided on the rim of the frame 11 and light from the outside world may directly reach the observer's eyes.

  The imaging device 13 images the outside of the camera lens in the optical axis direction (hereinafter also referred to as “imaging direction”). For example, the imaging device 13 is a digital camera or a video camera provided with an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). For example, a captured image (moving image) such as JPEG or MPEG2 / 4 is generated by the imaging of the imaging device 13.

  The inertial sensor 14 is a sensor that detects shaking of the HMD device 10 and provides data necessary for calculating the shaking amount of the imaging device 13 at that time. For example, the inertial sensor 14 includes an acceleration sensor, a gyro sensor, or the like. The inertial sensor 14 may have any form, name, and structure as long as it can provide data necessary for calculating the amount of shaking of the imaging device 13.

  The imaging device 13 and the inertial sensor 14 of the present embodiment are installed in a main body portion (for example, a frame 11 portion as shown in FIG. 1). For example, the inertial sensor 14 is installed at a first position A shown in FIG. 1, and the imaging device 13 is installed at a second position B different from the first position A.

  The AR providing device 20 displays an image (AR image) that is additionally superimposed on light from the outside. For example, the AR providing apparatus 20 displays a pop-up image including information such as a name and a place of work near a person existing in the observer's line of sight. Such a display is realized by adjusting the position, size, orientation, shape, color (saturation, brightness, contrast, etc.), shadow, etc. of the AR image. The AR providing apparatus 20 may be a binocular type instead of the monocular type as shown in FIG.

  The main body 30 has a configuration necessary for controlling the imaging device 13, the inertial sensor 14, and the AR providing device 20 to function as a head-mounted display as a whole. A specific configuration of the main body 30 will be described later.

(Specific configuration of main body 30)
Next, a specific configuration of the main body 30 will be described.

  As shown in FIG. 2, the main body 30 includes a CPU (Central Processing Unit) 31, a memory 32, a storage 33, an input device 34, a drive device 35, and a display controller 36.

  The CPU 31 is a control circuit composed of a processor or the like that controls the above-described units and executes various arithmetic processes according to a program. Each function of the HMD device 10 is exhibited by the CPU 31 executing a corresponding program. The

  The memory 32 is a high-speed accessible main storage device that temporarily stores programs and data as a work area. As the memory 32, for example, a DRAM (Dynamic Random Access Memory), an SDRAM (Synchronous Dynamic Access Memory), an SRAM (Static Random Access Memory), or the like is adopted.

  The storage 33 is a large-capacity auxiliary storage device that stores various programs including an operating system and various data. For example, a flash memory or the like is employed as the storage 13.

  The input device 34 includes physical keys and buttons for inputting instructions such as turning on the power of the HMD device 10. Moreover, you may utilize an external controller (for example, a smart phone, a remote controller, etc.) as the input device 34. FIG.

  The drive device 35 is a device that moves the display device 21 and the lens 22 in the optical axis direction. For the drive device 35, for example, a stepping motor, a voice coil motor, or the like is employed.

  The display controller 36 causes the display device 21 to display an AR image or the like. For example, the display controller 36 reads out an AR image from the memory 32 every predetermined display period, converts it into a signal for input to the display device 21, and generates a pulse signal such as a horizontal synchronizing signal and a vertical synchronizing signal. .

  The HMD device 10 having the hardware configuration as described above has the following functional configuration.

  FIG. 3 is a block diagram illustrating a functional configuration example of the HMD device.

<HMD device 10 (functional configuration)>
As shown in FIG. 3, the HMD device 10 includes, as a functional configuration, a control unit 41, a shake detection unit 42, an imaging control unit 43, a shake amount calculation unit 44, a tracking unit 45, a distortion correction unit 46, A display control unit 47 and an application unit 48 are provided.

  Hereinafter, the functional units 41 to 48 will be described in detail. The function units 41 to 48 are realized by the CPU 31 reading out a program installed in the storage 33 to the memory 32 and executing the program. Further, the present invention is not limited to this, and hardware such as ASIC may be used.

  The control unit 41 controls the entire HMD device 10.

  The shake detection unit 42 detects the shake of the observer wearing the HMD device 10. For example, the shake detection unit 42 acquires the sensor value of the inertial sensor 14 installed at the first position A, and detects the observer's shake.

  The imaging control unit 43 controls the imaging device 13 installed at the second position B to image the outside world. Thereby, a captured image is generated.

  The shaking amount calculation unit 44 calculates the shaking amount of the imaging device 13 based on the shaking amount detected by the shaking detection unit 42.

  The tracking unit 45 tracks a predetermined target (for example, a person) in the captured image based on the shaking amount of the imaging device 13 calculated by the shaking amount calculation unit 44.

  The distortion correction unit 46 corrects distortion generated in the captured image due to the characteristics of the camera lens of the imaging device 13.

  The display control unit 47 moves the positions of the display device 21 and the lens 22 and changes the display content of the display device 21 to determine whether an AR image exists at an arbitrary position within the visual field range of the observer. Display like this.

  The application unit 48 provides various functions involving display of AR images. For example, the application unit 48 displays an AR image (such as a pop-up image including information such as a name and work place) in the vicinity of a predetermined target (for example, a person) tracked by the tracking unit 45. To do.

  Detailed operations of the functional units 41 to 48 will be described in detail below.

<Operation of HMD Device 10>
FIG. 4 is a flowchart showing a procedure of AR image display processing executed in the HMD device. FIG. 5 is a diagram illustrating an example of an HMD device mounted on the observer's head and a coordinate system set by initial setting. FIG. 6 is a diagram illustrating an example of a captured image. FIG. 7 is a diagram for explaining a method of calculating the amount of shaking of the imaging apparatus. FIG. 7A is a diagram showing a state of the HMD device before the observer shakes, and FIG. 7B is a diagram showing a state of the HMD device after the observer is shaken. FIG. 8 is a diagram for explaining the distortion correction. FIG. 9 is a diagram for explaining the tracking process. FIG. 10 is a diagram for describing a method of performing distortion correction and tracking processing on a part of pixels of a captured image. FIG. 10A is a diagram illustrating an example of a region that is a target of distortion correction and tracking processing, and FIG. 10B is a diagram illustrating a correction example of a region that is a target of distortion correction and tracking processing. . FIG. 11 is a diagram illustrating a display example of an AR image.

  Hereinafter, the procedure of the AR image display process will be described with reference to FIGS.

(AR image display processing)
For example, when the power of the main body is turned on, the HMD device 10 starts the AR image display process shown in FIG. However, the timing of starting the AR image display process is not limited to this, and the AR image display process may be started when an operation for starting the AR image display process is performed on the input device 34. . Alternatively, the AR image display process may be started when it is detected that the HMD device 10 is mounted on the observer's head.

[Step S101]
When the AR image display process is started, the HMD device 10 functions as the control unit 41 and performs initial setting of the HMD device 10. For example, in the HMD device 10, as in the example illustrated in FIG. 5, it is assumed that an observer wearing the HMD device 10 on the head is observing a cubic object that exists in the outside world. At this time, for example, the HMD device 10 uses the center of the observer's visual field as the coordinate origin (0, 0, 0), the horizontal axis when the observer stands upright as the X axis, and the vertical axis as the Y axis. The optical axis (imaging direction) of the device 13 is taken as the Z axis. The reason why the center of the observer's field of view is the coordinate origin is that the HMD device 10 is likely to be the center point when it rotates around the X, Y, and Z axes. Such a coordinate system is referred to as an observer-based coordinate system XYZ.

[Step S102]
Next, the HMD device 10 functions as the imaging control unit 43 and images the outside world. Specifically, the HMD device 10 controls the imaging device 13 to start imaging, thereby generating a captured image in the optical axis direction of the camera lens. For example, in the situation shown in FIG. 5, when a scene in the optical axis direction (Z direction) of the camera lens is imaged by the imaging device 13, a captured image like the example shown in FIG. The HMD device 10 stores the generated captured image in the memory 32 or the storage 33.

[Step S103]
Subsequently, the HMD device 10 functions as the shake detection unit 42 and acquires a sensor value from the inertial sensor 14. For example, the HMD device 10 acquires accelerations in the three directions of the XYZ axes at the position of the inertial sensor 14.

[Step S104]
Next, the HMD device 10 functions as the shake detection unit 42 and detects the shake of the observer wearing the HMD device 10. Specifically, the HMD device 10 determines whether or not the observer is shaking. For example, the HMD device 10 determines that the observer is shaking if the sensor value acquired in step S103 exceeds a predetermined threshold, and the observer is not shaking if the sensor value is less than the predetermined threshold. Is determined.

  When the HMD device 10 determines that the observer is shaking as described above (step S104: YES), the process proceeds to step S105. If the HMD device 10 determines that the observer is not shaken (step S104: NO), the process returns to step S102, and the processes of steps S102 and S103 are performed until the observer shake is detected. repeat.

[Step S105]
The HMD device 10 functions as the shaking amount calculation unit 44 and calculates the shaking amount of the imaging device 13 based on the shaking amount of the observer detected in step S104.

For example, assuming that the coordinates of the first position A of the inertial sensor 14 before shaking are (X _A , Y _A , Z _A ) (FIG. 7 (A)), the inertial sensor 14 is A when the observer's head shakes. It moves to '(X _A ', Y _A ', Z _A ') (FIG. 7B).

At this time, the HMD device 10 obtains R _Z on the assumption that the relational expression of A ′ = R _Z · A is satisfied. Here, R _Z is a rotation matrix that rotates α (deg) around the Z axis, and is represented by the following determinant (Formula 1).

Here, the rotation angle α (deg) around the Z axis can be obtained by using the sensor value (acceleration in three directions of the XYZ axes) acquired in step S103. Specifically, the HMD device 10 determines the amount of movement of the inertial sensor 14 on the XY plane (direction vector from A to A ′ in the XY plane) by integrating the accelerations in the three directions of the XYZ axes. . Then, the HMD device 10 adds the obtained movement amount (direction vector) to the coordinates of the first position A of the inertial sensor 14 before shaking to obtain the coordinate A ′ of the position of the inertial sensor 14 after shaking. Next, based on the positional relationship between A and A ′ (for example, by a relational expression such as A · A ′ = | A || A ′ | · cosα), the HMD device 10 rotates the rotation angle α (deg) around the Z axis. ) Thus, the rotation angle α (deg) around the Z axis is obtained, and the above R _Z is obtained.

Then, when the rotational inertia sensor 14 as described above, with this, the image pickup apparatus 13 is also rotated R _Z. Therefore, the coordinates of the second position B of the imaging device 13 before shaking are set to (X _B , Y _B , Z _B ) (FIG. 7A), and B ′ (X _B ′, Y _B ′, Z after shaking). _When moving to _B ′) (FIG. 7B), the relational expression of B ′ = R _Z · B is established. Therefore, the HMD device 10 obtains the position B ′ of the imaging device 13 after shaking by multiplying the coordinates (X _B , Y _B , Z _B ) of the second position B of the imaging device 13 before shaking by R _Z. be able to.

Further, the HMD device 10 rotates not only around the Z axis as described above but also around the X axis and around the Y axis. Therefore, the rotation matrix R _X that rotates β (deg) around the X axis and the rotation matrix R _Y that rotates γ (deg) around the Y axis are also obtained by the same method as R _Z described above. However, R _X is represented by the following determinant (Formula 2), and R _Y is represented by the following determinant (Formula 3).

Then, the HMD device 10 obtains the position B ′ in the three-dimensional space of the image pickup device 13 after shaking by multiplying B by the rotation matrices R _X , R _Y , and R _Z in the XYZ axis directions. That is, the HMD device 10 may perform the operation of B ′ = R _X · R _Y · R _Z · B. Thereafter, the HMD device 10 obtains a distance (direction vector) between BB ′ as the amount of shaking of the imaging device 13. Then, the HMD device 10 stores the obtained shaking amount of the imaging device 13 in the memory 32 or the storage 33.

[Step S106]
Next, the HMD device 10 functions as the distortion correction unit 46 and corrects distortion generated in the captured image due to the characteristics of the camera lens of the imaging device 13. For example, the HMD device 10 performs correction for reading out one pixel at a time from a distorted captured image as shown in FIG. 8A and moving it to a predetermined position so as to obtain an image without distortion as shown in FIG. 8B. Do.

  However, in this embodiment, for the purpose of speeding up the distortion correction, reducing the load on the CPU 31, reducing the usage rate of the memory 32, and the like, the range of the captured image for correcting the distortion based on the amount of shaking of the imaging device 13 is set. limit. That is, distortion correction is performed only for some pixels without correcting distortion for all pixels of the captured image. For example, as shown in FIG. 10 (A), the HMD device 10 extracts a region (within a broken line) M including the entire predetermined target (for example, a cubic object) from the captured image, Perform distortion correction. If the observer's shaking is detected in step S104, as shown in FIG. 10B, the cube-shaped object is moving in the captured image, and thus the region M is also moved. Specifically, the HMD device 10 moves the region M to the region M ′ (in the solid line) by the amount of shaking of the imaging device 13 obtained in step S105. Then, the HMD device 10 performs distortion correction only on pixels that may fall within the region M ′ due to distortion correction.

[Step S107]
After that, the HMD device 10 functions as the tracking unit 45, and based on the shake amount of the imaging device 13 calculated by the shake amount calculation unit 44, a predetermined target in the captured image (for example, a cubic object shown in FIG. 6). And people). Specifically, the HMD device 10 tracks a predetermined target by comparing a plurality of captured images captured at different timings. For example, as in the example illustrated in FIG. 9, the HMD device 10 reads and compares (matches) two captured images that are continuously captured, and the predetermined image that was present in the previously captured image. It is specified where the target of has moved in the captured image captured later.

  However, for the purpose of speeding up the tracking process, reducing the load on the CPU 31, and reducing the usage rate of the memory 32, the range of captured images to be compared to track a predetermined target based on the amount of shaking of the imaging device 13 Limit. That is, the search target is not searched from the entire captured image, but is searched from a partial region of the captured image. Specifically, as shown in FIG. 10B, the HMD device 10 searches for the tracking target from the region M ′.

[Step S108]
Next, the HMD device 10 functions as the application unit 48 and the display control unit 47, and displays the AR image on the display device 21 as if the AR image exists in the vicinity of the predetermined target tracked in step S107. For example, as shown in FIG. 11, a name (a pop-up image “Mr. A”) can be displayed next to a person to be tracked.

[Step S109]
Thereafter, the HMD device 10 functions as the control unit 41 and determines whether or not an instruction to end the AR image display process has been given by the observer. Specifically, the HMD device 10 may determine whether or not the input device 34 has been operated to end the AR image display process.

  If there is no instruction to end the AR image display process (step S109: NO), the HMD device 10 returns the process to step S102 and continues the AR image display process. On the other hand, when an instruction to end the AR image display process is given (step S109: YES), the HMD device 10 ends the AR image display process.

  The AR image display process described above is executed in the HMD device 10 to calculate the amount of shaking of the imaging device 13 when an observer's shaking is detected, and based on this amount of shaking, a specific target (for example, in the captured image) , People, etc.). At this time, the amount of shaking of the inertial sensor 14 is not used as it is, but the amount of shaking of the imaging device 13 relative to the amount of shaking of the inertial sensor 14 is calculated. Thereby, the amount of shaking of the imaging device 13 can be accurately calculated regardless of the attachment position (relative position) of the inertial sensor 14 and the imaging device 13. Since the specific target in the captured image is tracked based on the shake amount of the imaging device 13 accurately calculated in this way, the position and shape of the specific target can be recognized with high accuracy. As a result, even when a pop-up image as shown in FIG. 11 is displayed, a natural display for the observer is possible. In addition, since the amount of shaking of the imaging device 13 is taken into consideration, the range of distortion correction and tracking processing can be limited to a part of the region in the captured image. Therefore, the processing can be speeded up, the load on the CPU can be reduced, and the memory efficiency can be improved.

  Each processing unit in the above-described flowchart is divided according to main processing contents in order to facilitate understanding of the HMD device 10. The invention of the present application is not limited by the method of classification of the processing steps and the names thereof. The processing performed in the HMD device 10 can be divided into more processing steps. One processing step may execute more processes.

<Modification>
In addition, said embodiment intends to illustrate the summary of this invention, and does not limit this invention. Many alternatives, modifications, and variations will be apparent to those skilled in the art.

  For example, in the above-described embodiment, the HMD device 10 displays a name (a pop-up image “Mr. A”) beside the person to be tracked in step S108 (FIG. 11). However, the present invention is not limited to this. For example, the HMD device 10 may display an AR image (referred to as a “cancel image”) superimposed on the tracking target so that the tracking target is not visible to the observer.

  FIG. 12 is a diagram illustrating an example of a cancel image. FIG. 12A is a diagram illustrating the observer's field of view in a state where the cancel image is not displayed, and FIG. 12B is a diagram illustrating the observer's field of view in a state where the cancel image is displayed in an overlapping manner. It is.

  When the tracking target has a complicated shape like the tree shown in FIG. 12A, it is difficult to accurately overlay the cancel images. However, if a specific target in the captured image is tracked based on the accurately calculated shake amount of the imaging device 13 as in the above-described embodiment, even if the shape is a complicated shape such as a tree, cancellation is possible. Images can be accurately superimposed (FIG. 12B).

  In the embodiment and the modification, the HMD device 10 is a see-through image display device. However, the present invention is not limited to this, and the HMD device 10 may be a non-transmissive image display device.

  In the above-described embodiment and modification, distortion correction (step S106) and tracking processing (step S107) are sequentially performed. However, the present invention is not limited to this, and distortion correction and tracking processing may be performed in parallel. In this case, every time one pixel is read from the captured image, the HMD device 10 performs both distortion correction and a process of comparing (matching) with another captured image in order to track a predetermined target.

In the embodiment, the relative position of the imaging device 13 with respect to the inertial sensor 14 is obtained using the rotation matrices R _X , R _Y , and R _Z. However, the present invention is not limited to this, and may be obtained by another calculation method as long as the accurate position of the imaging device 13 is obtained.

  The above-described configuration of the HMD device 10 has described the main configuration in describing the features of the above-described embodiments and modifications, and is not limited to the above-described configuration. Further, the configuration of the general HMD device 10 is not excluded.

  In addition, each functional configuration of the HMD device 10 described above is classified according to main processing contents in order to facilitate understanding of each functional configuration. The present invention is not limited by the way of classification and names of the constituent elements. Each functional configuration can be classified into more components according to the processing content. Moreover, it can also classify | categorize so that one component may perform more processes.

  The processing of each functional configuration of the HMD device 10 described above can also be realized by a dedicated hardware circuit. In this case, it may be executed by one hardware or a plurality of hardware.

  The program for operating the HMD device 10 may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, or a CD-ROM, or may be provided online via a network such as the Internet. . In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in the memory 32, the storage 33, or the like. Further, this program may be provided as, for example, a single application software, or may be incorporated in the software of each device as one function of the HMD device 10.

10 HMD device,
11 frames,
12 Transparent member,
13 imaging device,
14 Inertial sensor,
20 AR providing device,
21 display device,
22 lenses,
30 body part,
31 CPU,
32 memory,
33 storage,
34 input devices,
35 drive unit,
36 Display controller,
41 control unit,
42 Shake detection unit,
43 imaging control unit,
44 Shaking amount calculation unit,
45 Tracking unit,
46 Distortion corrector,
47 display control unit,
48 Application part.

Claims (18)

An image display device used by being attached to an observer,
A shake detection unit that is installed at a first position of the main body part and detects the shake of the observer;
An imaging unit that is installed at a second position different from the first position of the main body part, images the outside world, and generates a captured image;
Based on the shake amount detected by the shake detection unit, a shake amount calculation unit that calculates the shake amount of the imaging unit;
A tracking unit that tracks a predetermined target in the captured image based on the amount of shaking of the imaging unit calculated by the shaking amount calculation unit;
An image display device comprising: The imaging unit is a camera that uses a camera lens during imaging,
The image display apparatus according to claim 1, further comprising a distortion correction unit that corrects distortion generated in the captured image due to characteristics of the camera lens.   The image display device according to claim 2, wherein the distortion correction unit limits a range of the captured image in which distortion is corrected based on the shake amount of the imaging unit calculated by the shake amount calculation unit.   The image display device according to claim 1, wherein the tracking unit tracks the predetermined target by comparing a plurality of the captured images captured at different timings.   5. The tracking unit according to claim 4, wherein the tracking unit limits a range of the captured image to be compared in order to track the predetermined target based on a shaking amount of the imaging unit calculated by the shaking amount calculation unit. Image display device.   The image display device according to claim 2, wherein the distortion correction by the distortion correction unit and the tracking by the tracking unit are performed in parallel. A shake detection unit that is installed at the first position of the main body portion of the image display device used by being attached to the observer, and detects the shake of the observer;
An image display method in an image display device, the image display device comprising: an imaging unit that is installed at a second position different from the first position of the main body part and images the outside world to generate a captured image;
(A) calculating a shake amount of the imaging unit based on a shake amount detected by the shake detection unit;
(B) tracking a predetermined target in the captured image based on the shaking amount of the imaging unit calculated in the step (a);
Image display method. The imaging unit is a camera that uses a camera lens during imaging,
The image display method according to claim 7, further comprising: (c) correcting distortion generated in the captured image due to characteristics of the camera lens.   The image display method according to claim 8, wherein the step (c) limits a range of the captured image in which distortion is corrected based on the shake amount of the imaging unit calculated in the step (a).   The image display method according to claim 7, wherein the step (b) compares the plurality of captured images captured at different timings to track the predetermined target.   The step (b) limits the range of the captured image to be compared to track the predetermined target based on the amount of shaking of the imaging unit calculated in the step (a). The image display method described.   The image display method according to any one of claims 8 to 11, wherein the distortion correction in the step (c) and the tracking in the step (b) are performed in parallel. A shake detection unit that is installed at the first position of the main body portion of the image display device used by being attached to the observer, and detects the shake of the observer;
In a computer that functions as an image display device, which is installed at a second position different from the first position of the main body portion, and includes an imaging unit that images the outside world and generates a captured image.
(A) calculating a shake amount of the imaging unit based on a shake amount detected by the shake detection unit;
(B) tracking a predetermined target in the captured image based on the shaking amount of the imaging unit calculated in the step (a);
An image display program for executing The imaging unit is a camera that uses a camera lens during imaging,
The image display program according to claim 13, further causing the computer to execute a step of correcting (c) distortion generated in the captured image due to characteristics of the camera lens.   The image display program according to claim 14, wherein the step (c) limits a range of the captured image in which distortion is corrected based on the amount of shaking of the imaging unit calculated in the step (a).   The image display program according to any one of claims 13 to 15, wherein the step (b) compares the plurality of captured images captured at different timings to track the predetermined target.   The step (b) limits the range of the captured image to be compared to track the predetermined target based on the amount of shaking of the imaging unit calculated in the step (a). The image display program described.   The image display program according to any one of claims 14 to 17, wherein the distortion correction in the step (c) and the tracking in the step (b) are performed in parallel.