JP6981340B2 - Display control programs, devices, and methods - Google Patents

Description

The disclosed technique relates to a display control program, a display control device, and a display control method.

When learning work procedures and tools usage techniques, especially when the tools are dangerous, large-scale, or in a dangerous place such as aerial work or a fire site, the actual conditions are the same. It takes time and cost to prepare a place to learn technology, including ensuring safety.

In order to solve these problems, it is proposed to use Virtual Reality (VR) technology to virtually experience and train, and some products are beginning to be commercialized. Many of these are expressions in virtual space, including their own hands that operate the object to be operated, and there is a weak point in terms of presence.

Therefore, it is conceivable to attach a camera to the Head Mounted Display (HMD) and apply so-called Mixed Reality (MR), which brings the actual image into the virtual space. For example, as a conventional technique for MR, a device that accurately supplements a player moving in a wide range and presents a mixed reality feeling has been proposed. The device has a workbench with multiple markers located at known positions, a posture sensor mounted on the player to detect the player's head posture, and at least one of the multiple markers in the field of view. Equipped with a camera set to enter. Then, this device detects the head position based on the output from the attitude sensor, and corrects the detected head position signal by image processing the image signal of the camera and detecting the position of the camera. .. Further, the device generates a virtual image so as to present mixed reality at the viewpoint position corresponding to the corrected head position.

Japanese Unexamined Patent Publication No. 11-13706

However, the VR tracking sensor (posture sensor in the prior art) can acquire values such as the position and orientation of the HMD worn by the user with a certain accuracy, but in MR, it is an object captured by the camera. The misalignment between the object and the object in the virtual space gives the user a sense of discomfort. In particular, when the user holds or operates the object, there is a discrepancy between the visual sense of the object in the virtual space and the tactile sense of the actual object. In addition, when the real object is not brought into the virtual space and only the virtual object is displayed, the deviation between the position of the actual object and the position of the object in the virtual space does not have a great influence on the user.

Further, when the position of the object is detected by using the marker, for example, when the marker is well reflected in the image as shown in FIG. 26A, the position and the posture of the object are detected with high accuracy. be able to. However, for example, as shown in FIG. 26B, when only a part of the marker is shown, the position of the object cannot be detected accurately. As shown in C of FIG. 26, when the marker is located away from the camera and the size of the marker in the image is too small, and as shown in D of FIG. 26, due to blurring of the camera image, the image is included in the image. The same applies when the marker cannot be detected even if the marker is displayed. In order to compensate for this, when a large number of markers are arranged as in the prior art, it is necessary to perform calibration and arrangement management for each of the plurality of markers, which increases the processing load.

As one aspect, it is intended to alleviate the discomfort of the display when the virtual object of the object corresponding to the object that can be handled in the real space is superimposed on the image in the real space.

As one embodiment, the disclosed technique detects an object, which is arranged in association with a handleable object existing in the real space, from an image obtained by photographing the real space by a photographing device. Estimate the three-dimensional position and orientation of. In addition, the detection device detects the second index arranged in association with the object to estimate the three-dimensional position and posture of the object. Then, for comparison between the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. Based on this, the first estimation result or the second estimation result is selected. The second estimation result is the second estimation result in which the detection time by the detection device corresponds to the shooting time of the image by the photographing device. Then, based on the selected first estimation result or the second estimation result, an image obtained by superimposing a virtual object corresponding to the object on the image obtained by photographing the real space is displayed on the display device.

One aspect of the disclosed technology is that it is possible to alleviate the discomfort of the display when a virtual object corresponding to an object that can be handled in the real space is superimposed on the image in the real space.

It is a schematic diagram which shows the structure of the VR system which concerns on 1st Embodiment. It is a figure for demonstrating a handle model and an AR marker. It is a figure for demonstrating a handle model, an AR marker, and a VR tracker. It is a schematic diagram which shows an example of the arrangement of each configuration of the VR system which concerns on 1st Embodiment. It is a figure for demonstrating the handling of a handle model. It is a functional block diagram of the display control device which concerns on 1st Embodiment. It is a figure for demonstrating the virtual object in 1st Embodiment. It is a figure which shows an example of the history table held by a tracking estimation unit. It is a figure for demonstrating the calculation of a threshold value. It is a figure which shows an example of a threshold value database. It is a figure which shows the other example of a threshold database. It is a figure for demonstrating the correspondence between the estimation result of an AR marker estimation part, and the estimation result of a tracking estimation part. It is a figure for demonstrating the correspondence between the estimation result of an AR marker estimation part, and the estimation result of a tracking estimation part. It is a figure which shows an example of the VR image in 1st Embodiment. It is a block diagram which shows the schematic structure of the computer which functions as the display control device which concerns on 1st and 2nd Embodiment. It is a flowchart which shows an example of the threshold value calculation process in 1st Embodiment. It is a flowchart which shows an example of the display control processing in 1st Embodiment. It is a figure which shows an example of the VR image when the estimation result based on the unreliable AR marker is used. It is a schematic diagram which shows the structure of the VR system which concerns on 2nd Embodiment. It is a figure for demonstrating a nozzle model, an AR marker, and a VR tracker. It is a figure for demonstrating a nozzle model, an AR marker, and a VR tracker. It is a functional block diagram of the display control device which concerns on 2nd Embodiment. It is a figure for demonstrating the virtual object in 2nd Embodiment. It is a figure for demonstrating the switching of a virtual object. It is a flowchart which shows an example of the switching process in 2nd Embodiment. It is a figure for demonstrating the detection of a marker from an image.

Hereinafter, an example of the embodiment according to the disclosed technology will be described in detail with reference to the drawings.

<First Embodiment>
In the first embodiment, as an example, a case of displaying a VR image for virtually experiencing how to use a fire extinguisher will be described. An object that can be handled in real space is a model that imitates the handle of a fire extinguisher. Further, a case where a virtual three-dimensional object (hereinafter referred to as “virtual object”) of the entire fire extinguisher corresponding to the handle portion is superimposed and drawn on an image in the real space will be described.

As shown in FIG. 1, the Virtual Reality (VR) system 100 according to the first embodiment includes a display control device 10, a VR tracking device 31, a stereo camera 32, and a Head Mounted Display (HMD) 33. Further, the VR system 100 includes a handle model 34, an Augmented Reality (AR) marker 35, and a VR tracker 36.

The VR tracking device 31 can be realized by, for example, one or a plurality of laser radars. Each laser radar irradiates a laser, receives the laser reflected by each light receiving portion of the HMD 33 and the VR tracker 36, and measures tracking data including information on the irradiation direction of the laser and the distance to the light receiving portion. The VR tracking device 31 outputs the measured tracking data to the display control device 10. The VR tracking device 31 is an example of a detection device of the disclosed technology.

The stereo camera 32 is attached to the HMD 33, and each of the visible light cameras for the right eye and the left eye captures a range assuming the field of view of the wearer 40 wearing the HMD 33 at a predetermined frame rate. The stereo camera 32 outputs the image for the right eye and the image for the left eye obtained by photographing to the display control device 10. The stereo camera 32 is an example of a photographing device according to the disclosed technology.

The HMD 33 acquires the VR image for the right eye and the VR image for the left eye output from the display control device 10, and displays the stereoscopic image on the display. The VR image in the present embodiment is an image in which a virtual three-dimensional object showing a fire extinguisher, which is an object, is superimposed on an image in a real space taken by a stereo camera 32. The HMD 33 is an example of a display device of the disclosed technology.

The handle model 34 is an object that can be actually held and operated by the wearer of the HMD 33 and can be handled, and in the present embodiment, it is an object corresponding to the handle portion of the fire extinguisher. The handle model 34 is an example of an object of the disclosed technology.

The AR marker 35 is a marker of a figure registered in advance, and is arranged at a predetermined position where the positional relationship with the handle model 34 is fixed. For example, as shown in FIG. 2, the AR marker 35 is provided on the support column 37 that supports the handle model 34 and is attached to the pedestal 38. Further, even when images are taken from various angles, the pedestal 38 having a plurality of surfaces includes the AR marker 35 in good condition as shown in A of FIG. 26 within the angle of view of the stereo camera 32. A plurality of AR markers 35 can be attached. The AR marker 35 is an example of the first index of the disclosed technology.

The VR tracker 36 has a light receiving unit that receives a laser emitted from the VR tracking device 31, and is a device for detecting the position and posture of an object to which the VR tracker 36 is attached. The VR tracker 36 is arranged at a predetermined position where the positional relationship with the handle model 34 is fixed. For example, as shown in FIG. 3, the VR tracker 36 can be attached to the end of a column 37 that supports the handle model 34. The VR tracker 36 is an example of the second index of the disclosed technology.

FIG. 4 schematically shows an example of the arrangement of each configuration of the VR system 100. The HMD 33 with the stereo camera 32 is attached to the head of the wearer 40. As shown in FIG. 5, the handle model 34 is held or operated by the wearer 40. The VR tracking device 31 is located at a position where the measurement range of each laser radar includes the HMD 33 mounted on the wearer 40 and the VR tracker 36 arranged at a position corresponding to the handle model 34 held by the wearer 40. Be placed. For example, it is arranged at a position on a diagonal line sandwiching a position where the wearer 40 is expected to stand.

Functionally, as shown in FIG. 6, the display control device 10 includes an AR marker estimation unit 12, a tracking estimation unit 14, a threshold value calculation unit 15, a selection unit 16, and a display control unit 18. The AR marker estimation unit 12 is an example of the first estimation unit of the disclosed technology, the tracking estimation unit 14 is an example of the second estimation unit of the disclosed technology, and the threshold value calculation unit 15 calculates the disclosed technology. This is an example of the department. Further, the threshold database (DB) 21 and the object DB 22 are stored in the predetermined storage area of the display control device 10.

The threshold value DB 21 stores the threshold value used by the selection unit 16 described later (details will be described later).

The object DB 22 stores three-dimensional data of a virtual object to be superimposed and displayed on a real space image. In the present embodiment, the virtual object includes a virtual object indicating a fire extinguisher. More specifically, as shown in FIG. 7, the handle portion included in the virtual object 39 showing the entire fire extinguisher has a shape corresponding to the handle model 34 in the real space. Further, the object DB 22 may include an object other than the virtual object 39 indicating a fire extinguisher (for example, an object indicating a flame).

The AR marker estimation unit 12 acquires each of the right eye image and the left eye image output from the stereo camera 32, and performs image recognition processing such as pattern matching from each of the right eye image and the left eye image. , The AR marker 35, which is a pre-registered figure, is detected. The AR marker estimation unit 12 determines the three-dimensional position of the AR marker 35 in the camera coordinate system and the three-dimensional position of the AR marker 35 in the camera coordinate system based on the AR marker 35 detected from each of the right eye image and the left eye image and the binocular parallax of the stereo camera 32. Estimate the posture (angle with respect to each coordinate axis).

The tracking estimation unit 14 acquires the tracking data output from the tracking device 31. Then, the tracking estimation unit 14 estimates the three-dimensional positions and postures of the HMD 33 and the VR tracker 36 in the world coordinate system (virtual space coordinate system) based on the tracking data.

Further, since the tracking estimation unit 14 is used in the processing of the selection unit 16 described later, the tracking estimation unit 14 holds the history of the estimation results for the VR tracker 36 for a predetermined time. FIG. 8 shows an example of the history table 23 held by the tracking estimation unit 14. In the example of FIG. 8, the history table 23 can hold N estimation results. The "history number" is a number of 1, 2, ..., N assigned to each estimation result in the order of newest estimated time. The "three-dimensional position" and the "posture" are estimation results corresponding to each history number. The value of N can be set based on the frame rate of the stereo camera 32 and the maximum value of the difference in processing time between the estimation by the AR marker estimation unit 12 and the estimation by the tracking estimation unit 14, which will be described later.

The arrangement of the AR marker 35 and the VR tracker 36 with respect to the handle model 34 is known. Therefore, estimating the three-dimensional position and orientation of the AR marker 35 and the VR tracker 36 corresponds to estimating the three-dimensional position and orientation of the handle model 34. Further, the virtual object 39 showing the fire extinguisher corresponds to the handle model 34 in the real space as shown in FIG. That is, estimating the three-dimensional position and attitude of the AR marker 35 and the VR tracker 36 corresponds to estimating the three-dimensional position and attitude of the virtual object 39 indicating the fire extinguisher.

The threshold value calculation unit 15 acquires the estimation result estimated by the AR marker estimation unit 12 based on the AR marker 35 detected from the image in which the AR marker 35 is well contained, for example, as shown in FIG. 26A. .. In addition, the threshold value calculation unit 15 also acquires an estimation result estimated by the tracking estimation unit 14 based on the tracking data, which is detected when an image well containing the AR marker 35 is taken.

For example, as shown in FIG. 9, the handle model 34 in which the AR marker 35 and the VR tracker 36 are arranged in association with each other is placed at the upper left, upper right, center, lower left, and lower right positions with respect to the wearer 40, respectively. Place in. At this time, with the wearer 40 wearing the HMD 33 gazing at the center, the image taken by the stereo camera 32 arranged on the HMD 33 is arranged so that the entire AR marker 35 is included. Then, the threshold value calculation unit 15 is estimated by the tracking estimation unit 14 based on the estimation result estimated by the AR marker estimation unit 12 based on the image taken at each position and the tracking data detected at each position. Get the estimation result. In calculating the threshold value, the difference in processing time between the estimation by the AR marker estimation unit 12 and the estimation by the tracking estimation unit 14, which will be described later, is taken into consideration. Specifically, the steering wheel model 34 in which the AR marker 35 and the VR tracker 36 are arranged in association with each other is stopped at each position for a predetermined time, and an image is taken and tracking data is detected.

Then, the threshold value calculation unit 15 assigns the maximum value of the difference between the estimation result of the AR marker estimation unit 12 estimated for each position and the estimation result of the tracking estimation unit 14 to the estimation result in the selection unit 16 described later. It is calculated as a threshold value when performing. The threshold value calculation unit 15 stores the calculated threshold value in the threshold value DB 21.

FIG. 10 shows an example of the threshold value DB 21. In the example of FIG. 10, at each position (upper left, upper right, center, lower left, and lower right), the maximum value of the distance between the three-dimensional positions estimated by each of the AR marker estimation unit 12 and the tracking estimation unit 14 is set. It is stored as a "distance threshold". Similarly, at each position, the maximum value of the difference between the postures estimated by each of the AR marker estimation unit 12 and the tracking estimation unit 14 is stored as a “posture threshold”.

In the above example, considering that the fire extinguisher (handle model 34) is to be held and operated by hand, images are taken and tracking data are taken at several positions within the range of reaching out. Detection is to be performed, but the present invention is not limited to this example. The image may be taken and the tracking data may be detected at a position of 4 points or less, or may be performed at a position of 6 points or more. Further, the posture of the handle model 34 may be changed at each position. Further, the threshold value may be calculated for each region divided into N regions. FIG. 11 shows an example of the threshold value DB 21A in this case. The "area" is information for identifying each area when the virtual space is divided into a plurality of areas. Corresponding thresholds are utilized depending on which region of the virtual space the estimated position for the AR marker 35 or VR tracker 36 is contained.

The selection unit 16 selects one of the estimation results based on the comparison between the estimation result of the AR marker estimation unit 12 and the estimation result of the AR marker estimation unit 12 and the corresponding tracking estimation unit 14.

Here, the correspondence between the estimation result of the AR marker estimation unit 12 and the estimation result of the tracking estimation unit 14 will be described.

The process of estimating the position and posture of the handle model 34 based on the tracking data acquired from the tracking device 31 by the tracking estimation unit 14 can be processed in substantially real time with respect to the detection time of the tracking data. On the other hand, the process of estimating the position and orientation of the handle model 34 based on the AR marker 35 detected by the AR marker estimation unit 12 from the image taken by the stereo camera 32 requires a predetermined processing time. That is, the estimation by the AR marker estimation unit 12 and the estimation by the tracking estimation unit 14 are not synchronized. Therefore, the estimation results in which the shooting time of the image and the detection time of the tracking data are the same are associated with each other.

FIG. 12 schematically shows the loci of the estimation results estimated by each of the AR marker estimation unit 12 and the tracking estimation unit 14 at each time. In FIG. 12, the estimation result is shown in two dimensions for the sake of simplicity. The processing time from the detection of the tracking data to the acquisition of the estimation result by the tracking estimation unit 14 is 0, and the processing time from the shooting by the stereo camera 32 to the acquisition of the estimation result by the AR marker estimation unit 12 is x. In this case, as shown in FIG. 12, the estimation result of the AR marker estimation unit 12 at the time ti corresponds to the estimation result of the tracking estimation unit 14 at the time ti−x.

However, since the processing time x is not always constant, the selection unit 16 associates the estimation results with the same image shooting time and tracking data detection time as follows.

As shown in FIG. 13, the selection unit 16 determines each of the estimation results of the tracking estimation unit 14 at each of the plurality of times within the predetermined time before the time ti and the estimation result of the AR marker estimation unit 12 at the time ti. compare. Specifically, the selection unit 16 compares each of the vectors showing the estimation result of the tracking estimation unit 14 with the vector showing the estimation result of the AR marker estimation unit 12 at the time ti. Then, the estimation result of the tracking estimation unit 14 when the difference between the vectors is the minimum below the threshold value stored in the threshold value DB 21 is the estimation result of the tracking estimation unit 14 corresponding to the estimation result of the AR marker estimation unit 12 at the time ti. It is an estimation result. The difference between the vectors is, for example, the distance between the three-dimensional positions of the estimation results and the difference in the postures of the estimation results.

For example, as shown in FIGS. 26 to D, when the AR marker 35 is not detected well, the estimation of the three-dimensional position and the posture of the handle model 34 based on the AR marker 35 fails. Therefore, as described above, the selection unit 16 determines whether or not the estimation result estimated by the AR marker estimation unit 12 at time ti includes the estimation result by the corresponding tracking estimation unit 14, and the AR marker estimation unit 12 Evaluate the reliability of the estimation result by.

When the estimation result by the corresponding tracking estimation unit 14 exists, the selection unit 16 determines that the estimation result by the AR marker estimation unit 12 is reliable, and selects the estimation result by the AR marker estimation unit 12. On the other hand, when the estimation result by the corresponding tracking estimation unit 14 does not exist, the selection unit 16 determines that the estimation result by the AR marker estimation unit 12 is unreliable, and selects the estimation result by the tracking estimation unit 14. .. The selection unit 16 notifies the display control unit 18 of the selection result.

When the selection unit 16 selects the estimation result of the AR marker estimation unit 12, the estimation result of the AR marker estimation unit 12 and the estimation result of the tracking estimation unit 14 corresponding to the estimation result of the AR marker estimation unit 12 The difference between is retained as an offset value.

The display control unit 18 generates each of the VR image for the right eye and the VR image for the left eye to be displayed on the HMD 33. The method of generating each VR image is shown below. The display control unit 18 prepares a drawing area corresponding to the size of the VR image to be displayed on the HMD 33. The display control unit 18 draws the image acquired from the stereo camera 32 in the drawing area as a background image.

Further, when the display control unit 18 is notified by the selection unit 16 that the estimation result of the AR marker estimation unit 12 has been selected, the display control unit 18 reads out the three-dimensional data of the virtual object 39 indicating the fire extinguisher from the object DB 22. Then, the display control unit 18 draws a virtual object 39 showing a fire extinguisher corresponding to the three-dimensional position and orientation of the handle model 34 in the camera coordinate system estimated by the AR marker estimation unit 12 with a background image drawn. Draw on top of the area.

Further, when the display control unit 18 is notified by the selection unit 16 that the estimation result of the tracking estimation unit 14 has been selected, the display control unit 18 reads out the three-dimensional data of the virtual object 39 indicating the fire extinguisher from the object DB 22. Further, the display control unit 18 adds the offset value held by the selection unit 16 to the estimation result by the tracking estimation unit 14. Then, the display control unit 18 puts the fire extinguisher at the three-dimensional position of the world coordinate system of the handle model 34 indicated by the estimation result of the tracking estimation unit 14 to which the offset value is added, in the posture indicated by the estimation result after the offset value addition. The indicated virtual object 39 is arranged. Further, the display control unit 18 draws a background image of the virtual object 39 when the virtual space is viewed in the line-of-sight direction of the wearer 40 corresponding to the three-dimensional position and posture of the HMD 33 estimated by the tracking estimation unit 14. It is superimposed on the drawing area and drawn.

Further, the display control unit 18 extracts an image of the arm region of the wearer 40 from the image taken by the stereo camera 32, and the virtual object 39 showing the background image and the fire extinguisher draws the extracted image of the arm region. It is superimposed on the drawn drawing area and drawn.

Further, the display control unit 18 may superimpose another virtual object such as a flame stored in the object DB 22 on the drawing area and draw the object.

The display control unit 18 generates a VR image for the right eye and a VR image for the left eye by performing the above drawing for each of the right eye and the left eye. The display control unit 18 outputs the generated VR image for the right eye and the VR image for the left eye to the HMD 33.

As a result, a VR image as shown in FIG. 14, for example, is displayed on the HMD 33. In the example of FIG. 14, the fire extinguisher and the flame are virtual objects, and the other parts (wall, floor, whiteboard, desk, arm, etc.) are images of the real space taken by the stereo camera 32. ..

The display control device 10 can be realized by, for example, the computer 50 shown in FIG. The computer 50 includes a Central Processing Unit (CPU) 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. Further, the computer 50 is connected to an input / output device 54 such as an input device and a display device, a Read / Write (R / W) unit 55 that controls reading and writing of data to the storage medium 59, and a network such as the Internet. It also has a communication interface (I / F) 56. The CPU 51, the memory 52, the storage unit 53, the input / output device 54, the R / W unit 55, and the communication I / F 56 are connected to each other via the bus 57.

The storage unit 53 can be realized by a Hard Disk Drive (HDD), a Solid State Drive (SSD), a flash memory, or the like. A display control program 60 for causing the computer 50 to function as the display control device 10 is stored in the storage unit 53 as a storage medium. The display control program 60 includes an AR marker estimation process 62, a tracking estimation process 64, a threshold value calculation process 65, a selection process 66, and a display control process 68. Further, the storage unit 53 has an information storage area 70 in which information constituting each of the threshold value DB 21 and the object DB 22 is stored.

The CPU 51 reads the display control program 60 from the storage unit 53, expands the display control program 60 into the memory 52, and sequentially executes the processes included in the display control program 60. The CPU 51 operates as the AR marker estimation unit 12 shown in FIG. 6 by executing the AR marker estimation process 62. Further, the CPU 51 operates as the tracking estimation unit 14 shown in FIG. 6 by executing the tracking estimation process 64. Further, the CPU 51 operates as the threshold value calculation unit 15 shown in FIG. 6 by executing the threshold value calculation process 65. Further, the CPU 51 operates as the selection unit 16 shown in FIG. 6 by executing the selection process 66. Further, the CPU 51 operates as the display control unit 18 shown in FIG. 6 by executing the display control process 68. Further, the CPU 51 reads information from the information storage area 70 and expands the threshold value DB 21 and the object DB 22 into the memory 52. Further, the CPU 51 creates the history table 23 on the memory 52 when the tracking estimation process 64 is executed. As a result, the computer 50 that has executed the display control program 60 functions as the display control device 10. The CPU 51 that executes the program is hardware.

The function realized by the display control program 60 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an Application Specific Integrated Circuit (ASIC) or the like.

Next, the operation of the display control device 10 according to the first embodiment will be described. When the VR system 100 is activated, the VR tracking device 31 starts VR tracking and outputs tracking data, and the stereo camera 32 starts shooting and outputs an image for the right eye and an image for the left eye. Then, the display control device 10 executes the display control process shown in FIG. 17 after executing the threshold value calculation process shown in FIG. The threshold value calculation process and the display control process are examples of the display control method of the disclosed technology.

First, the threshold value calculation process shown in FIG. 16 will be described.

In step S12, the AR marker estimation unit 12 acquires an image taken by the stereo camera 32 when the handle model 34 in which the AR marker 35 and the VR tracker 36 are arranged in association with each other is arranged at each of the predetermined positions. .. At this time, with the wearer 40 wearing the HMD 33 gazing at the center, the image taken by the stereo camera 32 arranged on the HMD 33 is arranged so that the entire AR marker 35 is included. In addition, the tracking estimation unit 14 acquires the tracking data detected at each position.

Next, in step S14, the threshold value calculation unit 15 acquires the estimation result of the camera coordinate system estimated by the AR marker estimation unit 12 based on the images taken at each position, and converts it into a virtual space coordinate system. Further, the threshold value calculation unit 15 acquires the estimation result of the virtual space coordinate system estimated by the tracking estimation unit 14 based on the tracking data detected at each position.

Next, in step S16, the threshold value calculation unit 15 determines the maximum value of the difference between the estimation result of the AR marker estimation unit 12 estimated for each position and the estimation result of the tracking estimation unit 14 in the selection unit 16 described later. It is calculated as a threshold value when associating the estimation results. Then, the threshold value calculation unit 15 stores the calculated threshold value in the threshold value DB 21, and the threshold value calculation process ends.

Next, the display control process shown in FIG. 17 will be described.

In step S22, the AR marker estimation unit 12 and the display control unit 18 acquire each of the right eye image and the left eye image output from the stereo camera 32.

Next, in step S24, the display control unit 18 prepares a drawing area corresponding to the size of the VR image to be displayed on the HMD 33. The display control unit 18 draws the image acquired from the stereo camera 32 in the drawing area as a background image.

Next, in step S26, the tracking estimation unit 14 acquires the tracking data for the HMD 33 output from the tracking device 31. Then, the tracking estimation unit 14 estimates the three-dimensional position and orientation of the HMD 33 in the virtual space coordinate system based on the acquired tracking data.

Next, in step S28, the tracking estimation unit 14 acquires the tracking data for the VR tracker 36 output from the tracking device 31. Then, the tracking estimation unit 14 estimates the three-dimensional position and orientation of the VR tracker 36 in the virtual space coordinate system based on the acquired tracking data.

Next, in step S30, the tracking estimation unit 14 holds the estimation result of the three-dimensional position and the posture of the VR tracker 36 in the history table 23 as shown in FIG. 8, for example. At this time, the tracking estimation unit 14 discards the oldest estimation result held in the history table 23, and adds the estimation result estimated in step S28 to the history table 23.

Next, in step S32, the AR marker estimation unit 12 detects the AR marker 35, which is a pre-registered figure, from each of the acquired right-eye image and left-eye image by image recognition processing such as pattern matching. Determine if it has been done. If the AR marker 35 is detected, the process proceeds to step S36, and if not detected, the process proceeds to step S46.

In step S36, the AR marker estimation unit 12 determines the AR marker 35 in the camera coordinate system based on the AR marker 35 detected from each of the right eye image and the left eye image and the binocular parallax of the stereo camera 32. Estimate the 3D position and orientation.

Next, in step S38, the selection unit 16 evaluates the reliability of the estimation result by the AR marker estimation unit 12. Specifically, the selection unit 16 converts the estimation result of the camera coordinate system estimated by the AR marker estimation unit 12 in step S36 into a virtual space coordinate system. Then, the selection unit 16 compares each of the estimation results of the tracking estimation unit 14 held in the history table 23 with the estimation result (after coordinate conversion) of the AR marker estimation unit 12 estimated in step S36. The selection unit 16 tracks when the difference between each of the vectors showing the estimation result of the tracking estimation unit 14 and the vector showing the estimation result of the AR marker estimation unit 12 is the minimum below the threshold value stored in the threshold value DB 21. The estimation result of the estimation unit 14 is specified.

Next, in step S40, the selection unit 16 determines whether or not the estimation result of the AR marker estimation unit 12 is reliable. In step S38, when the estimation result of the tracking estimation unit 14 corresponding to the estimation result of the AR marker estimation unit 12 is specified, the estimation result of the AR marker estimation unit 12 is determined to be reliable, and the process is performed. The process proceeds to step S42. On the other hand, when the estimation result of the tracking estimation unit 14 corresponding to the estimation result of the AR marker estimation unit 12 is not specified, the estimation result of the AR marker estimation unit 12 is determined to be unreliable, and the process proceeds to step S46. Transition.

In step S42, the display control unit 18 reads out the three-dimensional data of the virtual object 39 indicating the fire extinguisher from the object DB 22. Then, the display control unit 18 draws a virtual object 39 showing a fire extinguisher corresponding to the three-dimensional position and orientation of the handle model 34 in the camera coordinate system estimated by the AR marker estimation unit 12 with a background image drawn. Draw on top of the area.

Next, in step S44, the selection unit 16 holds the difference between the estimation result of the AR marker estimation unit 12 and the estimation result of the corresponding tracking estimation unit 14 as an offset value.

On the other hand, in step S46, the display control unit 18 reads out the three-dimensional data of the virtual object 39 indicating the fire extinguisher from the object DB 22. Then, the display control unit 18 adds the offset value held by the selection unit 16 to the estimation result by the tracking estimation unit 14. Next, in step S48, the display control unit 18 places the posture indicated by the estimation result after the offset value addition at the three-dimensional position of the world coordinate system of the handle model 34 indicated by the estimation result of the tracking estimation unit 14 to which the offset value is added. Then, a virtual object 39 indicating a fire extinguisher is placed. Then, the display control unit 18 draws a background image of the virtual object 39 when the virtual space is viewed in the line-of-sight direction of the wearer 40 corresponding to the three-dimensional position and posture of the HMD 33 estimated by the tracking estimation unit 14. It is superimposed on the drawing area and drawn.

Next, in step S50, the display control unit 18 extracts, for example, a skin color region from each of the acquired right eye image and left eye image, and the arm region showing the arm portion of the wearer 40 is shown. To extract. Then, the display control unit 18 superimposes and draws the extracted image of the arm area on the drawing area on which the background image and the virtual object 39 indicating the fire extinguisher are drawn.

Next, in step S52, the display control unit 18 reads out the three-dimensional data of another virtual object such as a flame stored in the object DB 22, and draws the data by superimposing the three-dimensional data on the drawing area.

The display control unit 18 generates a VR image for the right eye and a VR image for the left eye by performing the above processing for each of the right eye and the left eye.

Next, in step S54, the display control unit 18 outputs the generated VR image for the right eye and the VR image for the left eye to the HMD 33. As a result, a VR image as shown in FIG. 14, for example, is displayed on the HMD 33. Then, the display control process ends.

The display control unit 18 repeatedly executes the above display control process at predetermined time intervals (for example, the same interval as the frame rate of the stereo camera) until the end of the VR system 100 is instructed.

As described above, according to the VR system 100 in the first embodiment, the display control device 10 evaluates the reliability of the estimation result based on the AR marker estimated at the time ti. Specifically, when the estimation result based on the VR tracker in which the difference between the vectors indicated by the estimation result is within the threshold value exists in the estimation result based on the VR tracker estimated at each time within the predetermined period before the time ti. In addition, it is evaluated as reliable. When the estimation result based on the AR marker is reliable, the virtual object of the object is drawn using the estimation result based on the AR marker with less positional deviation from the camera image.

On the other hand, when the estimation result based on the AR marker is unreliable, it is assumed that the virtual object of the object is drawn using the estimation result based on the AR marker. In this case, for example, as shown in FIG. 18, the virtual object of the object superimposed on the image in the real space is momentarily displaced, misaligned, or the like, and the object appears to flutter. Therefore, when the estimation result based on the AR marker is unreliable, the display control device 10 according to the present embodiment obtains the estimation result based on the VR tracker that can stably detect the position except for the positional deviation from the camera image. Use to draw a virtual object of the object. As a result, it is possible to alleviate the discomfort of the display when the virtual object of the object corresponding to the object that can be handled in the real space is superimposed on the image in the real space.

<Second Embodiment>
Next, the second embodiment will be described. In the second embodiment, a case where an object that can be handled in the real space is a model imitating the nozzle portion of the fire extinguisher, and the virtual object drawn by superimposing it on the image in the real space is also the nozzle portion of the fire extinguisher. do. In the VR system according to the second embodiment, the same parts as those of the VR system 100 according to the first embodiment are designated by the same reference numerals and detailed description thereof will be omitted.

As shown in FIG. 19, the VR system 200 according to the second embodiment includes a display control device 210, a VR tracking device 31, and an HMD 33 to which a stereo camera 32 is attached. Further, the VR system 200 includes a plurality of nozzle models 34A, 34B, 34C, a plurality of AR markers 35A, 35B, 35C, and one VR tracker 36. The number of nozzle models and AR markers is not limited to the example shown in FIG. 19, and may be two or four or more. In the following, when the nozzle models 34A, 34B, and 34C are described without distinction, the nozzle model 34A will be described as a representative. Similarly, when the AR markers 35A, 35B, and 35C are described without distinction, the AR marker 35A will be described as a representative.

The nozzle model 34A is an object that can be actually held and operated by the wearer of the HMD 33 and can be handled, and in the present embodiment, it is a rod-shaped object corresponding to the nozzle portion of the fire extinguisher. The nozzle models 34A, 34B, and 34C differ only in the AR markers 35A, 35B, and 35C arranged in association with each other, and have the same shape and the like. The nozzle model 34A is an example of an object of the disclosed technology.

The AR marker 35A is arranged at a predetermined position where the positional relationship with the nozzle model 34A is fixed. For example, as shown in FIG. 20, the AR marker 35A is attached to the tip of the nozzle model 34A. Further, similarly to the AR marker 35 of the first embodiment, a plurality of AR markers 35 can be attached to the pedestal 38 having a plurality of surfaces (see FIG. 21).

As shown in FIGS. 20 and 21, the VR tracker 36 is fixedly attached to a position such as the wrist of the wearer 40 holding the nozzle model 34A. As a result, when the wearer 40 holds the nozzle model 34A, the nozzle model 34A and the VR tracker 36 attached to the wrist or the like have a substantially fixed positional relationship.

Functionally, as shown in FIG. 22, the display control device 210 includes an AR marker estimation unit 12, a tracking estimation unit 14, a threshold value calculation unit 15, a selection unit 16, a display control unit 218, and a switching unit. Includes 219 and. Further, the threshold value DB 21 and the object DB 222 are stored in a predetermined storage area of the display control device 210.

The object DB 222 stores three-dimensional data of a virtual object indicating a nozzle of a fire extinguisher. As the virtual object indicating the nozzle, as shown in FIG. 23, virtual objects 39A, 39B, 39C having different shapes are prepared corresponding to each of the AR markers 35A, 35B, 35C. Each of the virtual objects 39A, 39B, and 39C is assigned an identification number corresponding to each of the AR markers 35A, 35B, and 35C.

When the switching unit 219 detects that the wearer 40 has changed the nozzle models 34A, 34B, 34C, the switching unit 219 has an identification number corresponding to the AR markers 35A, 35B, 35C affixed to the nozzle models 34A, 34B, 34C after the change. Is notified to the display control unit 218. Specifically, the switching unit 219 is the AR marker estimation unit 12, and when the period in which the AR marker 35A is not detected continues for a certain period, the AR markers 35A, 35B, and 35C detected next are switched to the AR. It is determined that the markers are 35A, 35B, and 35C.

When drawing the virtual object of the nozzle, the display control unit 218 reads out the three-dimensional data of the virtual objects 39A, 39B, 39C to which the identification number notified from the switching unit 219 is assigned from the object DB 222 and draws the virtual object. As a result, as shown in FIG. 24, in the real space, a virtual object of nozzles having different shapes can be displayed on the VR image simply by switching to the nozzle models 34A, 34B, 34C to which different AR markers 35A, 35B, 35C are attached. 39A, 39B, 39C are drawn.

The display control device 210 can be realized by, for example, the computer 50 shown in FIG. A display control program 260 for causing the computer 50 to function as the display control device 210 is stored in the storage unit 53 of the computer 50. The display control program 260 includes an AR marker estimation process 62, a tracking estimation process 64, a threshold value calculation process 65, a selection process 66, a display control process 268, and a switching process 269. Further, the storage unit 53 has an information storage area 70 in which information constituting each of the threshold value DB 21 and the object DB 222 is stored.

The CPU 51 reads the display control program 260 from the storage unit 53, expands the display control program 260 into the memory 52, and sequentially executes the processes included in the display control program 260. The CPU 51 operates as the display control unit 218 shown in FIG. 22 by executing the display control process 268. Further, the CPU 51 operates as the switching unit 219 shown in FIG. 22 by executing the switching process 269. Further, the CPU 51 reads information from the information storage area 70 and expands each of the threshold value DB 21 and the object DB 222 into the memory 52. Other processes are the same as those of the display control program 60 according to the first embodiment. As a result, the computer 50 that has executed the display control program 260 functions as the display control device 210.

The function realized by the display control program 260 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the display control device 210 according to the second embodiment will be described. In the second embodiment as well, the threshold value calculation process (FIG. 16) and the display control process (FIG. 17) are executed as in the first embodiment. Further, in parallel with the display control process, the switching process shown in FIG. 25 is executed.

In step S62 of the switching process shown in FIG. 25, the switching unit 219 determines whether or not the period in which the AR marker 35A is not detected by the AR marker estimation unit 12 continues for a certain period of time. If it continues, the process proceeds to step S64, and if it does not continue, the determination in this step is repeated.

In step S64, the switching unit 219 determines whether or not the AR marker 35A is detected by the AR marker estimation unit 12. If it is detected, the process proceeds to step S66, and if it is not detected, the determination in this step is repeated.

In step S66, the switching unit 219 notifies the display control unit 218 of the identification number corresponding to the detected AR markers 35A, 35B, 35C, and returns to step S62.

In the threshold value calculation process in the second embodiment, the same threshold value calculation process as in the first embodiment may be executed for any of the AR markers 35A, 35B, and 35C. Further, for each of the AR markers 35A, 35B, and 35C, the same threshold value calculation process as in the first embodiment may be executed, and the threshold value for each of the AR markers 35A, 35B, and 35C may be set.

Further, in the display control process (FIG. 17) in the second embodiment, the three-dimensional data of the virtual objects 39A, 39B, and 39C are read from the object DB 222 in steps S42 or S48. At this time, the three-dimensional data of the virtual objects 39A, 39B, 39C corresponding to the identification number notified in step S66 of the switching process may be read out.

As described above, according to the VR system 200 according to the second embodiment, one VR tracker and each of the plurality of AR markers are independently provided, the VR tracker is attached to the wearer's wrist or the like, and the VR tracker is attached by hand. Hold the nozzle model with one of the AR markers affixed. This makes it possible to draw different virtual objects according to the AR marker attached to the held nozzle model. In addition, it is not necessary to prepare VR trackers for the number of virtual objects to be drawn, and the cost can be reduced.

In the first and second embodiments, the case where the fire extinguisher or the nozzle of the fire extinguisher is drawn as a virtual object has been described, but the object to be a virtual object is not limited to this.

Further, in each of the above embodiments, the case where the AR marker is used for estimating the position and the posture of the object has been described, but the present invention is not limited to this. A marker other than the AR marker may be used, or the position and posture of the object may be estimated from the arrangement of the feature points by detecting the feature points of the object itself without using the marker.

Further, in each of the above embodiments, the case of estimating the position and the posture of the AR marker using the image taken by the stereo camera has been described, but the present invention is not limited to this. For example, based on the position, size, and tilt (distortion) of an AR marker in an image detected from an image taken with a monocular visible light camera, the actual size of the AR marker, and the focal length of the camera. The position and orientation of the AR marker may be estimated. Moreover, you may use an infrared camera. In this case, a visible light camera for taking a background image to be displayed in the VR image may be separately received.

When a stereo camera is used, it is not necessary to separately provide an image for estimating the position and orientation of the AR marker and a background image to be displayed on the VR image.

Further, in each of the above embodiments, the mode in which the display control programs 60 and 260 are stored (installed) in the storage unit 53 in advance has been described, but the present invention is not limited thereto. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

The following additional notes will be further disclosed with respect to each of the above embodiments.

(Appendix 1)
The three-dimensional position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image obtained by photographing the real space by the photographing device.
By detecting the second index arranged in association with the object by the detection device, the three-dimensional position and posture of the object are estimated.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. The first estimation result or the second estimation result is selected based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A process including displaying on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result. A display control program to be executed by a computer.

(Appendix 2)
Of the second estimation results based on the second index detected at a plurality of detection times within a predetermined time before the shooting time, the vector showing the first estimation result and the second estimation result. The display control program according to Appendix 1, wherein the second estimation result in which the difference from the vector indicating the above is the minimum within a predetermined threshold value is set as the second estimation result corresponding to the first estimation result.

(Appendix 3)
When the second estimation result corresponding to the first estimation result exists, the first estimation result is selected, and when the second estimation result corresponding to the first estimation result does not exist, the first estimation result is selected. The display control program according to Appendix 2 for selecting the second estimation result.

(Appendix 4)
When selecting the second estimation result, the difference between the first estimation result when the first estimation result is selected in the past and the second estimation result corresponding to the first estimation result. The display control program according to Appendix 3, wherein the virtual object is superimposed based on the three-dimensional position and posture of the object indicated by the value added to the selected second estimation result.

(Appendix 5)
The first estimation result obtained by estimating the threshold value based on the first index detected from the image taken so as to include the whole of the first index, and the whole of the first index Any one of Appendix 2 to Appendix 4 calculated based on the difference between the shooting time of the image captured so as to be included and the second estimation result based on the second index detected at the corresponding detection time. The display control program described in.

(Appendix 6)
The item according to any one of Supplementary note 1 to Supplementary note 5, wherein each of the plurality of different first indexes is arranged in each of the plurality of the objects, and the second index is arranged in the portion holding the object. Display control program.

(Appendix 7)
The first index is a marker arranged in a known three-dimensional position and posture with respect to the three-dimensional position and posture of the object, and is based on the position and posture of the marker on the captured image. The display control program according to any one of Supplementary note 1 to Supplementary note 6, which estimates the three-dimensional position and posture of the object.

(Appendix 8)
The display control program according to any one of Supplementary note 1 to Supplementary note 7, wherein the second index is a VR tracker, and the detection device detects tracking data including information on the direction and distance of the VR tracker.

(Appendix 9)
The third dimension position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image of the real space taken by the photographing device. 1 estimation part and
A second estimation unit that estimates the three-dimensional position and posture of the object by detecting the second index arranged in association with the object by the detection device.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. A selection unit that selects the first estimation result or the second estimation result based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A display control unit that displays on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result.
Display control unit including.

(Appendix 10)
The selection unit includes a vector showing the first estimation result among the second estimation results based on the second index detected at a plurality of detection times within a predetermined time before the shooting time, and the selection unit. The second estimation result in which the difference from the vector indicating the second estimation result is the minimum within a predetermined threshold value is described in Appendix 9 as the second estimation result corresponding to the first estimation result. Display control device.

(Appendix 11)
When the second estimation result corresponding to the first estimation result exists, the selection unit selects the first estimation result and the second estimation result corresponding to the first estimation result. The display control device according to Appendix 10, which selects the second estimation result when is not present.

(Appendix 12)
When the second estimation result is selected, the display control unit has the first estimation result when the first estimation result is selected in the past, and the first estimation result corresponding to the first estimation result. The display control device according to Appendix 11, wherein the virtual object is superimposed based on the three-dimensional position and orientation of the object indicated by the value obtained by adding the difference from the estimation result of 2 to the selected second estimation result.

(Appendix 13)
The first estimation result obtained by estimating the threshold value based on the first index detected from the image taken so as to include the whole of the first index, and the whole of the first index Addendum 10 to Addendum 12 including a calculation unit that calculates based on the difference between the shooting time of the image taken so as to be included and the second estimation result based on the second index detected at the corresponding detection time. The display control device according to any one of the items.

(Appendix 14)
The item according to any one of Supplementary note 9 to Supplementary note 13, wherein each of the plurality of different first indexes is arranged in each of the plurality of the objects, and the second index is arranged in the portion holding the object. Display control device.

(Appendix 15)
The first index is a marker arranged in a known three-dimensional position and posture with respect to the three-dimensional position and posture of the object.
The display according to any one of Supplementary note 9 to Supplementary note 14, in which the first estimation unit estimates the three-dimensional position and posture of the object based on the position and posture of the marker on the captured image. Control device.

(Appendix 16)
The display control device according to any one of Supplementary note 9 to Supplementary note 15, wherein the second index is a VR tracker, and the detection device detects tracking data including information on the direction and distance of the VR tracker.

(Appendix 17)
The three-dimensional position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image obtained by photographing the real space by the photographing device.
By detecting the second index arranged in association with the object by the detection device, the three-dimensional position and posture of the object are estimated.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. The first estimation result or the second estimation result is selected based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A process including displaying on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result. The display control method performed by the computer.

(Appendix 18)
Of the second estimation results based on the second index detected at a plurality of detection times within a predetermined time before the shooting time, the vector showing the first estimation result and the second estimation result. The display control method according to Appendix 17, wherein the second estimation result in which the difference from the vector indicating the above is the minimum within a predetermined threshold value is the second estimation result corresponding to the first estimation result.

(Appendix 19)
When the second estimation result corresponding to the first estimation result exists, the first estimation result is selected, and when the second estimation result corresponding to the first estimation result does not exist, the first estimation result is selected. The display control method according to Appendix 18 for selecting the second estimation result.

(Appendix 20)
The three-dimensional position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image obtained by photographing the real space by the photographing device.
By detecting the second index arranged in association with the object by the detection device, the three-dimensional position and posture of the object are estimated.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. The first estimation result or the second estimation result is selected based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A process including displaying on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result. A storage medium that stores a display control program to be executed by a computer.

10, 210 Display control device 12 AR marker estimation unit 14 Tracking estimation unit 15 Threshold calculation unit 16 Selection unit 18, 218 Display control unit 219 Switching unit 21 Threshold DB
22,222 Object DB
23 History table 31 VR tracking device 32 Stereo camera 33 HMD
34 Handle model 34A, 34B, 34C Nozzle model 35, 35A, 35B, 35C AR marker 36 VR tracker 39, 39A, 39B Virtual object 40 Wearer 50 Computer 51 CPU
52 Memory 53 Storage unit 59 Storage medium 60, 260 Display control program 100, 200 VR system

Claims (10)

The three-dimensional position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image obtained by photographing the real space by the photographing device.
By detecting the second index arranged in association with the object by the detection device, the three-dimensional position and posture of the object are estimated.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. The first estimation result or the second estimation result is selected based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A process including displaying on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result. A display control program to be executed by a computer. Of the second estimation results based on the second index detected at a plurality of detection times within a predetermined time before the shooting time, the vector showing the first estimation result and the second estimation result. The display control program according to claim 1, wherein the second estimation result in which the difference from the vector indicating the above is the minimum within a predetermined threshold value is the second estimation result corresponding to the first estimation result. .. When the second estimation result corresponding to the first estimation result exists, the first estimation result is selected, and when the second estimation result corresponding to the first estimation result does not exist, the first estimation result is selected. The display control program according to claim 2, wherein the second estimation result is selected. When selecting the second estimation result, the difference between the first estimation result when the first estimation result is selected in the past and the second estimation result corresponding to the first estimation result. The display control program according to claim 3, wherein the virtual object is superimposed based on the three-dimensional position and posture of the object indicated by the value added to the selected second estimation result. The first estimation result obtained by estimating the threshold value based on the first index detected from the image taken so as to include the whole of the first index, and the whole of the first index Any of claims 2 to 4 calculated based on the difference between the shooting time of the image taken to be included and the second estimation result based on the second index detected at the corresponding detection time. The display control program according to item 1. One of claims 1 to 5, wherein each of the plurality of different first indexes is arranged in each of the plurality of objects, and the second index is arranged in a portion holding the object. The display control program described in. The first index is a marker arranged in a known three-dimensional position and orientation with respect to the three-dimensional position and orientation of the object, and is based on the position and orientation of the marker on the captured image. The display control program according to any one of claims 1 to 6, which estimates the three-dimensional position and orientation of the object. The display control according to any one of claims 1 to 7, wherein the second index is a VR tracker, and the detection device detects tracking data including information on the direction and distance of the VR tracker. program. The third dimension position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image of the real space taken by the photographing device. 1 estimation part and
A second estimation unit that estimates the three-dimensional position and posture of the object by detecting the second index arranged in association with the object by the detection device.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. A selection unit that selects the first estimation result or the second estimation result based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A display control unit that displays on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result.
Display control unit including. The three-dimensional position and orientation of the object are estimated by detecting the first index arranged in association with the handleable object existing in the real space from the image obtained by photographing the real space by the photographing device.
By detecting the second index arranged in association with the object by the detection device, the three-dimensional position and posture of the object are estimated.
The detection of the first estimation result of the three-dimensional position and posture of the object based on the first index and the second estimation result of the three-dimensional position and posture of the object based on the second index. The first estimation result or the second estimation result is selected based on the comparison between the shooting time of the image by the photographing device and the second estimation result corresponding to the detection time by the apparatus.
A process including displaying on a display device an image in which a virtual object corresponding to the object is superimposed on an image obtained by capturing the real space based on the selected first estimation result or the second estimation result. The display control method performed by the computer.

Images