JP7042380B1 - Display device, program and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily create a virtual object. A display device (virtual object display device 100) includes a camera 151, a space sensor (depth sensor 153) for measuring the position and orientation of an object, and virtual marker information including the position and orientation of a virtual marker in a virtual space. A storage unit 130 for storing marker information 133) is provided. Further, the display device is in the virtual space, the space recognition unit 115 that generates the space shape data of the object from the measurement result of the space sensor, the marker recognition unit 113 that recognizes the position and orientation of the real marker imaged by the camera 151, and the marker recognition unit 113. , The image captured by the camera 151 and the virtual object generation unit 116 that generates a virtual object whose shape matches the spatial shape data and whose position and orientation with respect to the virtual marker match the position and orientation of the object with respect to the real marker. Is provided with a display control unit 117 for displaying. [Selection diagram] Fig. 1

Description

The present invention relates to display devices, programs and display methods.

Technology related to mixed reality that superimposes and displays virtual objects in virtual space (virtual three-dimensional space) on images in real space is widespread, and in addition to games, confirmation of completed images in design and trial production, etc. It is used in. When displaying in superposition, it is necessary to align the virtual object and the real space, in other words, to match the coordinate system of the virtual space and the coordinate system of the real space. For this purpose, for example, there is a method of matching the coordinate system using a marker. Specifically, first, a marker (real marker) is installed at the position of the real space corresponding to the origin of the virtual space. Next, the display device of the virtual object recognizes the marker, identifies the position of the origin of the real space and the direction of the coordinate axis, and displays the virtual object so as to match the coordinate system of the virtual space.

The display device is equipped with an acceleration sensor and an angular velocity sensor, and even if the display device moves and changes its posture, it can calculate its own position and posture in real space from the measured values of the sensors. However, the farther away from the marker position, the more an error occurs between the calculated position / posture and the actual position / posture.

The video display system described in Patent Document 1 for correcting an error detects a first submarker corresponding to a reference marker in real space, and has a plurality of first virtual positions associated with a coordinate axis (virtual object) in advance. Among them, the first positional relationship calculation unit that calculates the positional relationship between the first virtual position closest to the first submarker and the first submarker, and the second submarker arranged at a predetermined position with respect to the reference marker in the real space. A second positional relationship calculation unit that detects and calculates the positional relationship between a plurality of second virtual positions and a second submarker associated with the coordinate axes in advance at intervals narrower than the intervals of the plurality of first virtual positions, and the coordinate axes. A correction unit for correcting the relationship between the position and the posture of the image pickup device is provided.
In such an image display system, the error can be kept small by installing the first submarker and the second submarker so that the intervals are within a predetermined distance.

Japanese Unexamined Patent Publication No. 2019-066292

In the technique described in Patent Document 1, it is necessary to set a first virtual position and a second virtual position on a virtual object corresponding to a first submarker and a second submarker on an object in real space. Therefore, there is a virtual object in the virtual position corresponding to the marker, and the virtual position is set on this virtual object.

However, there is not always a virtual object corresponding to the object in which the submarker is installed (an object model in the virtual space is created). For example, at a construction site such as a plant, a marker may be placed on a temporary scaffold, but even if a virtual space model (virtual object) corresponding to plant equipment and piping is created, a model of the temporary scaffold is not created. .. Therefore, the installation positions of the reference marker, the first sub-marker, and the second sub-marker are limited to the floor or wall of the site, and it is difficult to install them at intervals sufficiently to suppress the error.

Even if a temporary scaffold is modeled and a virtual object of the scaffold is prepared, the height and position of the scaffold are not always determined in advance. In addition, the scaffolding may be assembled immediately before using the display device (video display system). Therefore, it is not practical to prepare a virtual object for scaffolding (a virtual object for which a virtual position is set) in advance.
The present invention has been made in view of such a background, and an object of the present invention is to provide a display device, a program, and a display method capable of easily creating a virtual object.

In order to solve the above-mentioned problems, the display device according to the present invention stores a camera, a spatial sensor that measures the position and orientation of an object in real space, and virtual marker information including the position and orientation of a virtual marker in virtual space. A storage unit, a space recognition unit that generates spatial shape data of the object from the measurement result of the space sensor, a marker recognition unit that recognizes the position and orientation of the actual marker imaged by the camera, and a marker recognition unit in the virtual space. A virtual object generation unit that generates a measurement virtual object based on the measurement value of the space sensor and a display control unit that displays an image captured by the camera are provided, and the virtual object generation unit is the measurement virtual object in the virtual space. The position and orientation of the measured virtual object with respect to the virtual marker in the virtual space is the position and orientation of the object with respect to the real marker in the real space so that the shape of the object matches the spatial shape data of the object. It is characterized in that the measurement virtual object is generated in the virtual space so as to coincide with.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a display device, a program, and a display method that enable easy creation of a virtual object. Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is a functional block diagram of the virtual object display device which concerns on this embodiment. It is a figure for demonstrating the virtual object which concerns on this embodiment. It is a flowchart which shows the procedure (use procedure) of the worker who uses the virtual object display device which concerns on this embodiment. It is a flowchart of the virtual object display process executed by the virtual object display device which concerns on this embodiment.

≪Overview of virtual object display device≫
Next, a virtual object display device (display device) in the embodiment (embodiment) for carrying out the present invention will be described. The virtual object display device superimposes an image of a virtual object on a real space image (captured image, captured image) and displays it. For example, at a construction site such as a plant, an image of a virtual object of equipment to be installed is displayed superimposed on an image of the installation location at the planned installation location.

The virtual object display device uses a depth sensor to measure (scan) existing objects (object surfaces) such as pipes and temporary objects that exist around the planned installation location, acquire shape data, and use it as a new virtual object. Import into virtual space. The virtual object display device can superimpose an image of a new virtual object corresponding to an existing object, an image of a virtual object to be installed, and an image of a planned installation location. By viewing such an image (image), the operator can confirm whether the equipment to be installed and the existing object do not come into contact with each other and whether the required distance can be obtained.

In order to display the image of the virtual object and the image of the site in an overlapping manner, it is necessary to match (match) the coordinate system of the virtual space in which the virtual object exists with the coordinate system of the space of the site. For this purpose, the virtual object display device aligns the two coordinate systems by acquiring the position / orientation (posture) of the actual marker installed at the site with the camera and matching it with the position / orientation of the virtual marker in the virtual space. ..

Since the virtual object display device is equipped with an inertia sensor and can detect changes in its own movement distance and posture, if it is in the vicinity of a real marker (for example, within a few meters), the image of the virtual object and the image of the site space can be displayed with a certain accuracy. Can be displayed in layers. However, the farther away from the actual marker, the more the measurement error of the inertial sensor accumulates, and the display shifts. Therefore, it is desirable to match the coordinate system again using the real marker and the virtual marker before moving and increasing the error.

The virtual object display device includes a second virtual marker set on a virtual object corresponding to an existing object captured in the virtual space, and a second real marker installed on the existing object corresponding to the second virtual marker. By matching the coordinate system with the reference, the image of the virtual object and the captured image in the real space can be superimposed and displayed with high accuracy.

In this way, the virtual object display device measures the object with the depth sensor, captures it in the virtual space, and sets the virtual object (measurement virtual object) to be captured, and the real marker installed on the object and acquired by the camera. By repeating the combination of and, the virtual object and the captured image in the real space can be displayed in an overlapping manner while changing the reference marker one after another. By changing the real marker and the virtual marker one after another, the deviation between the real space and the virtual space is eliminated, and the virtual object display device can display the virtual object and the captured image in the real space in an overlapping manner with high accuracy.

<< Configuration of virtual object display device >>
FIG. 1 is a functional block diagram of the virtual object display device 100 according to the present embodiment. The virtual object display device 100 is a computer and includes a control unit 110, a storage unit 130, an input / output unit 140, a camera 151, a touch panel display 152, a depth sensor 153, and an inertial sensor 154. The input / output unit 140 may be provided with a communication device and may be capable of transmitting / receiving data to / from other devices. Further, a media drive may be connected to the input / output unit 140 so that data can be exchanged using a recording medium.

The camera 151 is a camera that captures an image of the site including the actual marker. The touch panel display 152 is a display with a touch panel that accepts user operations. The depth sensor 153 is, for example, a ToF (Time-of-Flight) depth sensor that measures the distance to an object (object surface) in space such as a wall, floor, equipment, or installation, and measures the shape of the object in space. Can be obtained. The inertial sensor 154 includes a three-way acceleration sensor, a gyroscope (angular velocity sensor), and the like. The posture, moving distance, and moving direction of the virtual object display device 100 can be calculated from the measured values of the inertial sensor 154.

<< Virtual object display device: Storage unit >>
The storage unit 130 includes storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash memory. Model information 131, spatial shape information 132, marker information 133, and program 134 are stored in the storage unit 130. The program 134 includes a description of the virtual object display process (see FIG. 4) described later.

The model information 131 is information on a virtual object in the virtual space, and includes, for example, data such as a shape related to polygons constituting the surface of the virtual object and a position / orientation in the virtual space. As will be described later, virtual objects include manufactured virtual objects and measured virtual objects. The space shape information 132 is a measured value of the depth sensor 153, is information on a group of points constituting the surface of an object in the real space, and indicates the position of each point in the real space. The marker information 133 is information including the position and orientation of the virtual marker set on the virtual object.

≪Virtual object display device: Control unit≫
The control unit 110 includes a CPU (Central Processing Unit), and includes a model information acquisition unit 111, a marker setting unit 112, a marker recognition unit 113, a position / orientation calculation unit 114, a space recognition unit 115, and a virtual object generation unit 116. And a display control unit 117 is provided.
The model information acquisition unit 111 acquires information on a virtual object via the input / output unit 140 and stores it in the model information 131. Since the virtual object acquired by the model information acquisition unit 111 is a designed and manufactured model, it is also referred to as a manufactured virtual object below.

The marker setting unit 112 sets a virtual marker on a virtual object in the virtual space according to an instruction of a worker who is a user of the virtual object display device 100. The virtual marker is set, for example, on the floor in the virtual space near the virtual object, or on the surface of the virtual object. The information of the virtual object is stored in the model information 131.
The marker recognition unit 113 recognizes the actual marker included in the captured image of the camera 151 in the marker recognition mode, and acquires the position and orientation of the actual marker.

The position / orientation calculation unit 114 uses the position / orientation of the actual marker acquired by the marker recognition unit 113 as a reference, and adds the change in the position / orientation calculated from the measured value of the inertial sensor 154 to the position / orientation of the virtual object display device 100. Calculate the orientation (posture). In other words, the position / orientation calculation unit 114 calculates the position / orientation of the virtual object display device 100 with reference to the actual marker.
The space recognition unit 115 calculates the position of a point cloud indicating an object (object surface) in the real space based on the measured value of the depth sensor 153 in the space recognition mode, and stores it in the space shape information 132. The position of the point cloud (each point) is a position in the real space, and can be calculated based on the position / orientation (posture) of the virtual object display device 100 in the real space.

The virtual object generation unit 116 calculates the shape, position, and orientation of the object based on the position information of the point cloud in the spatial shape information 132, generates the information of the virtual object, and stores it in the model information 131. In other words, the virtual object generation unit 116 generates a virtual object so that the shape of the virtual object in the virtual space matches the position data of the point cloud indicating the object in the space shape information 132 of the object. Further, the virtual object generation unit 116 generates a virtual object so that the position / orientation of the virtual object with respect to the virtual marker in the virtual space matches the position / orientation of the object with respect to the real marker in the real space. The virtual object generated by the virtual object generation unit 116 is a virtual object generated based on the measured value of the depth sensor 153, and is also referred to as a measured virtual object below.

The display control unit 117 displays the image of the virtual object superimposed on the captured image by the camera 151. The display control unit 117 superimposes and displays an image of the virtual object in the captured image so as to match the position and orientation of the virtual object in the virtual space. Specifically, the display control unit 117 displays the image of the virtual object so that the relationship between the position and orientation of the real marker and the image of the virtual object matches the relationship between the position and orientation of the virtual marker and the virtual object in the virtual space. do. In the following, the image of the virtual object displayed overlaid on the captured image is also simply referred to as a virtual object. For example, the display control unit 117 describes that the camera 151 superimposes and displays a virtual object on the captured image.
Further, when a plurality of virtual objects (positions) overlap, the display control unit 117 does not display the overlapped portion. For example, when the manufactured virtual object and the measured virtual object overlap, the display control unit 117 does not display the overlapping portion of either one or both.

≪Virtual object≫
FIG. 2 is a diagram for explaining the virtual object 300 according to the present embodiment. The virtual object 300 is a virtual object that models a corridor straddling a turbine installed in a power plant. The virtual object 300 is a three-dimensional model of the corridor to be installed at the site, and is a manufactured virtual object. The virtual marker 350 is a virtual marker corresponding to a real marker placed on the floor where the corridor is installed, and is set on the virtual floor 330 in the virtual space. The actual marker is installed on the floor of the site where the corridor is installed so that the relationship between the position and orientation of the corridor and the actual marker and the relationship between the position and orientation of the virtual object 300 and the virtual marker 350 match. The real marker is a plate-shaped one marked (marked) with two arrows and one rectangle equivalent to the virtual marker 350, but neither the virtual marker nor the real marker is limited to such a figure. ..
Hereinafter, a procedure for using the virtual object display device 100 (see FIG. 3) using the virtual object 300 as an example and a virtual object display process (see FIG. 4) for the virtual object display device 100 will be described.

≪Procedure for using virtual object display device≫
FIG. 3 is a flowchart showing a procedure (usage procedure) of a worker who uses the virtual object display device 100 according to the present embodiment.
In step S11, the operator takes in the information of the virtual object into the virtual object display device 100. In this example, the information of the virtual object 300 and the virtual floor 330, which are the manufactured virtual objects, is taken into the virtual object display device 100 by using a network or a storage medium.

In step S12, the operator sets a virtual marker on the virtual object (manufactured virtual object or measured virtual object). In step S12 following step S11, the operator sets the virtual marker 350 on the virtual floor 330 as shown in FIG.
In step S13, the operator installs a real marker at the site so as to correspond to the position and orientation of the virtual marker set in step S12. For example, the worker installs the actual marker at the position / orientation of the floor at the site where the corridor is installed so as to correspond to the position / orientation of the virtual marker 350 on the virtual floor 330.

In step S14, the operator sets the virtual object display device 100 in the marker recognition mode and photographs the actual marker with the camera 151. Subsequently, when the operator takes a picture of the site (floor or space) with the camera 151, the virtual object display device 100 displays the virtual object (image) on the image of the site. For example, when an operator photographs a place where a corridor is planned to be installed, the virtual object display device 100 displays (an image) of a virtual object 300 (an image) which is a manufactured virtual object modeling the corridor. Further, if the object to be the measurement virtual object has been read (measured) in step S18 described later, the measurement virtual object is also displayed on the image of the site.
The operator confirms (determines) whether the position and orientation of the virtual object 300, which is a manufactured virtual object, is correct, and inputs the result to the virtual object display device 100. Although there is no actual corridor of the virtual object 300 at the site, the operator confirms it based on the positional relationship with the surrounding objects. Peripheral objects include turbines and pipes that straddle the corridor.

In step S15, the operator proceeds to step S16 if the position and orientation of the displayed virtual object 300 are correct (step S15 → OK), and returns to step S13 if they are misaligned (step S15 → NG). Returning to step S13, the operator re-installs the actual marker so as to match the virtual marker set in step S12.

In step S16, the operator moves or confirms interference. In this example, the worker moves to the floor one floor above the floor where the corridor will be installed to make sure that the handrails in the corridor do not interfere with the existing piping. This floor is located down the stairs on the right side of the corridor (virtual object 300). In the following, this floor will be referred to as the second floor (see the virtual object 340 described later). In addition, the operator looks at the virtual object 300 displayed on top of the image of the site, and confirms whether the corridor and the object (piping, etc.) at the site interfere with each other.

In step S17, the worker proceeds to step S18 if he / she wants to continue the work.
In step S18, the operator sets the virtual object display device 100 in the space recognition mode, scans the object to be the measurement virtual object (reads by the depth sensor 153), and returns to step S12. The object to be a measurement virtual object is an object for setting a virtual marker or an object for confirming interference with a corridor (virtual object 300). The worker scans the pipes near the handrails on the second floor and the corridor (read by the depth sensor 153).

When the worker scans the second floor, the virtual object display device 100 generates and displays the measured virtual object on the second floor. The virtual object 340 shown in FIG. 2 is a second floor that has been scanned and modeled as a measurement virtual object. Returning to step S12, the operator can set the virtual marker 360 on the virtual object 340 which is the second floor, and install and read the actual marker on the second floor in steps S13 and S14. Then, the virtual object display device 100 superimposes and displays the virtual object on the image of the site based on the virtual marker 360 and the real marker.

Further, when the operator scans the pipe, the virtual object display device 100 displays a measurement virtual object that models the scanned pipe in addition to the virtual object 300. When a plurality of virtual objects overlap, the virtual object display device 100 does not display the overlapping portion of one of the virtual objects. The operator can take pictures of the space where there should be a handrail that may interfere (the space displayed as an image of the handrail) from various positions and orientations, and the presence or absence of interference (virtual object that is not displayed). To confirm.

≪Virtual object display processing≫
FIG. 4 is a flowchart of a virtual object display process executed by the virtual object display device 100 according to the present embodiment.
In step S31, the model information acquisition unit 111 acquires information on the virtual object 300, which is a manufactured virtual object, and stores it in the model information 131 according to the instruction of the operator (see step S11 in FIG. 3).
In step S32, the marker setting unit 112 acquires and sets the position and orientation of the virtual marker instructed by the operator (see step S12) (stored in the marker information 133).

In step S33, the marker recognition unit 113 recognizes the actual marker included in the captured image of the camera 151, and acquires the position and orientation of the actual marker. After that, the position / orientation calculation unit 114 calculates the position / orientation of the virtual object display device 100 with reference to the actual marker. The marker recognition unit 113 recognizes the actual marker by starting the detection process of the actual marker included in the captured image when the operator switches to the marker recognition mode.

In step S34, the display control unit 117 displays the image of the virtual object superimposed on the captured image by the camera 151. If the virtual object is only the virtual object 300, which is the manufactured virtual object acquired in step S31, the display control unit 117 superimposes the image of the virtual object 300 on the captured image and displays it. If the measured virtual object (for example, the virtual object 340 on the second floor) is acquired in steps S37 and S38 described later, the display control unit 117 superimposes the measured virtual object on the captured image in addition to the virtual object 300. indicate.

In step S35, the display control unit 117 acquires the determination result (see steps S14 and S15) of whether the position / orientation of the virtual object is correct or deviated by the operator. If the display control unit 117 is correct (step S35 → OK), the process proceeds to step S36, and if the display control unit 117 is deviated (step S35 → NG), the process returns to step S33.
In step S36, the display control unit 117 ends the virtual object display process if there is an end instruction (see step S17) by the operator (step S36 → YES), and if there is no end instruction (step S36 → NO), the display control unit 117 ends. Proceed to S37.

In step S37, the space recognition unit 115 calculates the position of a point cloud indicating an object (object surface) in the real space based on the measured value of the depth sensor 153, and stores it in the space shape information 132. The objects are the second floor and piping (see step S18). Further, the space recognition unit 115 activates the depth sensor 153 when the operator switches to the space recognition mode, and measures the measurement.

In step S38, the virtual object generation unit 116 calculates the shape, position, and orientation of the object based on the position information of the point cloud in the spatial shape information 132, converts the calculation result into the information of the measurement virtual object, and model information 131. Store in. The converted measured virtual object includes a virtual object 340 (see FIG. 2).

In step S39, the display control unit 117 displays the image of the virtual object superimposed on the captured image by the camera 151. This virtual object is a virtual object of a corridor, a second floor, and a pipe. When a plurality of objects (positions) overlap, the display control unit 117 does not display the overlapped portion. For example, when the virtual object 300 in the corridor and the virtual object in the pipe overlap, the display control unit 117 does not display the overlapping portion of the virtual object 300. The process of not displaying the overlapping portions is the same in step S34.

≪Characteristics of virtual object display processing≫
The virtual object display device 100 superimposes and displays a virtual object on a captured image so that the positions and orientations match with respect to the virtual marker and the real marker. The displayed virtual object is not limited to the manufactured virtual object (virtual object acquired in step S31) prepared in advance, but also includes the measurement virtual object (see steps S37 to S38) acquired at the site.

The virtual object to which the virtual marker can be set is not limited to the manufactured virtual object, but can also be set to the measurement virtual object (see step S32). Therefore, the operator can move the site and set a virtual marker on the measurement virtual object corresponding to the moved object. A position / orientation reference based on the virtual marker and the real marker is provided at the moving destination, and the virtual object display device 100 can display the virtual object superimposed on the captured image with high accuracy. As a result, the operator can accurately confirm whether or not the object to be installed (for example, equipment such as a corridor) and the existing object (for example, piping) interfere with each other.

≪Transformation example: Alignment≫
When the position of the virtual object is deviated in the above embodiment (see steps S15 and S35 → NG), the actual marker is re-installed (see step S13) and reloaded (see steps S14 and S33). Instead of resetting the real marker, the virtual marker may be reset or the position of the virtual object may be adjusted. For example, the virtual object display device 100 can reset the virtual marker on the virtual object in the virtual space so as to match the actual marker installed in the real space, and can adjust the position of the virtual object displayed on the on-site image. You may do so. By doing so, the worker can match the coordinate system of the virtual space with the coordinate system of the real space in a short time.

≪Modification example: Interference confirmation≫
When a plurality of virtual objects (positions) overlap in the above embodiment, the display control unit 117 does not display the overlapped portion (see step S39). The display control unit 117 may display so as to emphasize the overlapping portion or the portion of a plurality of objects whose distance is equal to or less than a predetermined value. The display control unit 117 may emphasize such a portion by, for example, changing the color or displaying a pop-up window displaying the distance between the objects.

≪Other variants≫
Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and further, various modifications such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and variations thereof are included in the scope and gist of the invention described in the present specification and the like, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

100 Virtual object display device (display device)
111 Model information acquisition unit 112 Marker setting unit 113 Marker recognition unit 114 Position / orientation calculation unit 115 Space recognition unit 116 Virtual object generation unit 117 Display control unit 132 Spatial shape information (spatial shape data)
133 Marker information (virtual marker information)
151 Camera 153 Depth sensor (spatial sensor)
154 Inertia sensor 300 Virtual object (manufactured virtual object)
340 virtual object (measurement virtual object)
350,360 virtual marker

Claims (9)

With the camera
Spatial sensors that measure the position and orientation of objects in real space,
A storage unit that stores virtual marker information including the position and orientation of virtual markers in virtual space,
A space recognition unit that generates space shape data of the object from the measurement results of the space sensor,
A marker recognition unit that recognizes the position and orientation of the actual marker imaged by the camera, and
A virtual object generation unit that generates a measurement virtual object based on the measurement value of the space sensor in the virtual space,
It is equipped with a display control unit that displays the captured image of the camera.
The virtual object generation unit is
So that the shape of the measured virtual object in the virtual space matches the space shape data of the object.
In addition, the measurement virtual object is placed in the virtual space so that the position and orientation of the measurement virtual object with respect to the virtual marker in the virtual space coincide with the position and orientation of the object with respect to the real marker in the real space. A display device characterized by producing. Information related to the manufactured virtual object existing in the virtual space is stored in the storage unit.
The display control unit superimposes and displays an image of the manufactured virtual object whose position and orientation with respect to the real marker matches the position and orientation of the manufactured virtual object with respect to the virtual marker. The display device according to claim 1, wherein the display device is characterized by the above. Information related to the manufactured virtual object existing in the virtual space is stored in the storage unit.
Further, a marker setting unit for setting a second virtual marker on the measurement virtual object is provided.
The marker recognition unit recognizes the position and orientation of the second real marker installed at the position and orientation of the object corresponding to the position and orientation of the second virtual marker with respect to the measurement virtual object.
The display control unit uses an image of the manufactured virtual object whose position and orientation with respect to the second real marker coincides with the position and orientation of the manufactured virtual object with respect to the second virtual marker as an image taken by the camera. The display device according to claim 1, wherein the display device is superimposed and displayed. The display control unit is characterized in that an image of a virtual object whose position and orientation with respect to the real marker matches the position and orientation of the measurement virtual object with respect to the virtual marker is superimposed on the captured image of the camera. The display device according to any one of claims 2 to 3. The display control unit is characterized in that when the measured virtual object and the manufactured virtual object overlap in the virtual space, the image of the overlapping portion of the measured virtual object or the overlapping portion of the manufactured virtual object is not displayed. The display device according to claim 4. In the display control unit, the distance between the portion where the measurement virtual object and the manufactured virtual object overlap in the virtual space, or the portion between the measurement virtual object and the manufactured virtual object, and the distance between the two portions is set. The display device according to claim 4, wherein a portion of a predetermined value or less is highlighted and displayed. The space recognition unit generates space shape data of the second object from the measurement result of the space sensor, and generates the space shape data.
The virtual object generation unit is a virtual object that is in the virtual space and whose shape matches the space shape data, and the position and orientation with respect to the second virtual marker is the position and orientation of the second object with respect to the second real marker. Generate a second measurement virtual object that matches the position and orientation,
The display control unit captures an image of the second measurement virtual object whose position and orientation with respect to the second real marker coincides with the position and orientation of the second measurement virtual object with respect to the second virtual marker. The display device according to claim 3, wherein the display device is superimposed on an image and displayed. With the camera
Spatial sensors that measure the position and orientation of objects in real space,
A computer equipped with a storage unit that stores virtual marker information including the position and orientation of the virtual marker in the virtual space.
A space recognition unit that generates space shape data of the object from the measurement results of the space sensor,
A marker recognition unit that recognizes the position and orientation of the actual marker imaged by the camera, and
A virtual object generation unit that generates a measurement virtual object based on the measurement value of the space sensor in the virtual space,
It is equipped with a display control unit that displays the captured image of the camera.
The virtual object generation unit is
So that the shape of the measured virtual object in the virtual space matches the space shape data of the object.
In addition, the measurement virtual object is placed in the virtual space so that the position and orientation of the measurement virtual object with respect to the virtual marker in the virtual space coincide with the position and orientation of the object with respect to the real marker in the real space. A program to function as a display device to generate. With the camera
Spatial sensors that measure the position and orientation of objects in real space,
A display device including a storage unit that stores virtual marker information including the position and orientation of the virtual marker in the virtual space.
A step of generating spatial shape data of the object from the measurement result of the spatial sensor, and
The step of recognizing the position and orientation of the actual marker imaged by the camera,
A step of generating a measurement virtual object based on the measurement value of the space sensor in the virtual space,
The step of displaying the captured image of the camera is executed.
The shape of the measured virtual object in the virtual space matches the space shape data of the object, and
A display method in which the position and orientation of the measured virtual object with respect to the virtual marker in the virtual space coincide with the position and orientation of the object with respect to the real marker in the real space.

Images