JP2021047611A - Image display system - Google Patents

Abstract

To make it possible to superimpose and display a design image of a construction object on an actual construction object without placing a large number of artifacts, even in a vast construction site where the surrounding landscape is monotonous.SOLUTION: A terminal device 12 identifies a position and posture of a backhoe 8 based on positions measured by GNSS units 11(1) and 11(2) located on the backhoe 8. A terminal device 13 identifies corresponding points corresponding to three or more recognized feature points from an image taken of the backhoe 8 out of a three-dimensional shape of the backhoe 8 represented by data stored in advance. The terminal device 13 identifies a positional relationship between the terminal device 13 and the backhoe 8 based on the positions of the feature points in the image and the positions of the corresponding points in the three-dimensional shape. Based on the positional relationship and the position and posture of the backhoe 8, the terminal device 13 identifies the position and posture of the terminal device 13. Using the identified position and posture of the terminal device 13, the terminal device 13 displays an image in which the designed three-dimensional shape is superimposed on an actual image of the construction object.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for superimposing and displaying an image on a construction object.

In augmented reality (Augmented Reality) and mixed reality (Mixed Reality) systems that superimpose images in a virtual space with a common coordinate system on a real space object, display means such as a liquid crystal display It is necessary to accurately identify the position and orientation in real space. For example, Patent Document 1 is a document that discloses a technique for specifying a position and an attitude of an apparatus. Patent Document 1 describes a technique in which an index of a known position arranged in a real space is imaged by a camera, the index is detected from the captured image, and the position and orientation of the camera are measured from the position in the image of the detected index. Is described (eg, paragraph 0002).

Japanese Unexamined Patent Publication No. 2005-107966

At the construction site, workers and inspectors superimpose the design three-dimensional shape (three-dimensional shape at the time of completion) on the construction work in the real space by using augmented reality and mixed reality technology. If the displayed image can be seen, the worker or the inspector can efficiently perform the construction work and the inspection work of the construction object.

As described above, in augmented reality and mixed reality systems, it is necessary to accurately specify the position and orientation of the display means in real space. As a method for that purpose, the feature points at known positions are recognized from the image taken by the photographing means whose positional relationship does not change with the display means, and the position and orientation of the display means are specified based on the positions of the recognized feature points in the image. The method is known.

The feature points at known positions are divided into natural objects (for example, mountain peaks) and man-made objects (markers). There may be no natural objects with distinctive shapes at the construction site. For example, if the construction site is in a place where the surrounding landscape such as the sea or desert is monotonous, no characteristic natural objects can be found in any direction. In such a case, the position and orientation of the display means cannot be specified even if a natural object is photographed.

In addition, since the construction site is generally vast, it takes a lot of labor and time to arrange a large number of artificial objects (markers) that are characteristic points around the construction site in order to use artificial objects instead of natural objects. However, it is not realistic.

In view of the above circumstances, the present invention superimposes a design image of a construction object on an actual construction object without arranging a large number of artificial objects even in a vast construction site where the surrounding landscape is monotonous. Provide the means by which it is possible to do so.

The present invention includes a photographing means, a display means having a predetermined positional relationship with the photographing means, a measuring means for measuring the position and orientation of the object to be photographed by the photographing means on the earth, the photographing means and the display means. The positional relationship data indicating the positional relationship of the object, the first three-dimensional shape data representing the three-dimensional shape of the object, the three-dimensional shape in the design of the construction, and the second three-dimensional representing the position and attitude on the earth. A storage means for storing the shape data, a recognition means for the photographing means to recognize three or more feature points of the object from an image obtained by photographing the object, and a three-dimensional shape represented by the first three-dimensional shape data. The corresponding point specifying means for specifying the three or more corresponding points, which are the points corresponding to the three or more feature points recognized by the recognition means, and the positions of the three or more corresponding points in the object. The positional relationship between the object and the photographing means is specified based on the positions of the three or more feature points in the image of the object, and the object and the object and the above are based on the positional relationship and the positional relationship indicated by the positional relationship data. The positional relationship specifying means for specifying the positional relationship of the display means, the positional relationship between the object specified by the positional relationship specifying means and the display means, and the measurement means measured by the measuring means at the time of photographing the image of the object. Based on the position and orientation of the object on the earth, the position and orientation specifying means for specifying the position and orientation of the display means on the earth at the time of shooting, and the position and orientation of the display means specified by the position and orientation specifying means on the earth. Based on the design three-dimensional shape of the construction object represented by the second three-dimensional shape data and the position and orientation on the earth, the design is performed in a virtual space in which the real space and the coordinate system photographed by the photographing means are common. An image generation means for generating a display image representing a three-dimensional shape of the construction object constructed at an upper position in a design posture is provided, and the display means provides an image display system for displaying the display image. It is provided as one aspect.

According to the image display system according to the first aspect, even in a vast construction site where the surrounding landscape is monotonous, by photographing a predetermined object with a photographing means, the actual construction object can be made into the actual construction object in the design of the construction object. Images can be overlaid and displayed.

Further, the present invention is an image display system according to the first aspect, and when the measuring means is used as the first measuring means, a second method of continuously measuring a change in the position and posture of the displaying means. The position and orientation specifying means includes the position and orientation of the display means on the earth at the specified time of shooting, and the display means measured by the second measuring means from the time of shooting to the present time. As the second aspect, we propose a configuration in which the current position and orientation of the display means on the earth are specified based on the changes in the position and the attitude.

According to the image display system according to the second aspect, even during a period in which the photographing means is not photographing a predetermined object, the design image of the construction object can be superimposed and displayed on the actual construction object.

Further, the present invention is an image display system according to the first or second aspect, and the display image is an image representing a design three-dimensional shape of the construction object on an image taken by the photographing means. As a third aspect, we propose a configuration in which the image is a superposed image.

According to the image display system according to the third aspect, the user who sees the image displayed on the display means has a three-dimensional shape obtained by photographing the actual structure of the current construction, the position of the three-dimensional shape in the design of the construction, and You can intuitively grasp the deviation of the posture.

Further, the present invention is an image display system according to a first or second aspect, wherein the display means is a transmissive display means worn on a user's head, and the display image is the display means. The user sees the real space through the display means, and the user is in the virtual space where the real space and the coordinate system are common. As a fourth aspect, we propose a configuration in which the original shapes are superimposed and shown as an image.

According to the image display system according to the fourth aspect, the user wearing the transmissive display means on the head can see the display means through the display means from various positions and angles. It is possible to intuitively grasp the deviation of the position and posture of the three-dimensional shape in the design of the object.

Further, the present invention presents a first three-dimensional object in which a computer is provided with positional relationship data indicating a predetermined positional relationship between a photographing means and a display means, and a three-dimensional shape of an object whose position and orientation on the earth are measured by the measuring means. A process of acquiring shape data, a second three-dimensional shape data representing a design three-dimensional shape of a construction object and a position and orientation on the earth, and image data representing an image of the object taken by the photographing means. And the process of acquiring the position and orientation data indicating the position and orientation of the object on the earth measured by the measuring means at the time of shooting the image, and the three or more feature points of the object from the image represented by the image data. The process of recognizing, the process of specifying three or more corresponding points, which are points on the three-dimensional shape represented by the first three-dimensional shape data and corresponding to the recognized three or more feature points, and the above-mentioned The positional relationship between the object and the photographing means is specified based on the positions of the three or more corresponding points in the object and the positions of the three or more feature points in the photographed image of the object, and the positional relationship and the positional relationship are specified. The process of specifying the positional relationship between the object and the display means based on the positional relationship indicated by the data, the positional relationship between the specified object and the display means, and the position and orientation of the object on the earth indicated by the positional orientation data. Based on the above, the process of specifying the position and orientation of the display means on the earth at the time of shooting, the position and orientation of the specified display means on the earth, and the construction object represented by the second three-dimensional shape data. Based on the design three-dimensional shape and the position and orientation on the earth, the construction object constructed at the design position in the design posture in the virtual space in which the real space and the coordinate system photographed by the photographing means are common. A fifth embodiment provides a program for executing a process of generating a display image representing a three-dimensional shape and a process of displaying the display image on the display means.

According to the program according to the fifth aspect, even in a vast construction site where the surrounding landscape is monotonous, a predetermined object can be photographed by a photographing means to make an actual construction object in the design of the construction object. Images can be overlaid and displayed.

Further, the present invention is a program according to a fifth aspect, in which, when the measuring means is used as the first measuring means, the computer continuously measures the change in the position and posture of the display means. The process of acquiring the position / orientation change data indicating the change in the position and posture of the display means from the time of shooting to the present time measured by the second measuring means, and the process of acquiring the data of the change in position / orientation at the specified time of shooting on the earth. Based on the position and orientation and the change in the position and orientation of the display means from the time of shooting to the present time indicated by the change in position and orientation data, the process of specifying the position and orientation of the display means on the earth at the present time. A program for executing the above is proposed as a sixth aspect.

According to the program according to the sixth aspect, the computer can superimpose the design image of the construction object on the actual construction object even during the period when the photographing means is not photographing the predetermined object. ..

Further, the present invention is a program according to a fifth or sixth aspect, in which a three-dimensional shape in the design of the construction is applied to the image taken by the photographing means as the display image on the computer. A program for generating an image in which the represented image is superimposed is proposed as a seventh aspect.

According to the program according to the seventh aspect, the user who sees the image displayed on the display means by the computer can see the three-dimensional shape obtained by photographing the actual structure of the current construction and the position of the three-dimensional shape in the design of the construction. You can intuitively grasp the deviation of the posture.

Further, the present invention is a program according to a fifth or sixth aspect, in which the computer is transmitted through the transparent display means worn on the user's head as the display image in a real space. An image showing the user superimposing the three-dimensional shape of the construction piece constructed in the posture and shape to be constructed at the design construction position in the virtual space in which the real space and the coordinate system are common. A program for generating is proposed as an eighth aspect.

According to the program according to the eighth aspect, a user wearing a transparent display means on the head by a computer can see the display means through the display means from various positions and angles. It is possible to intuitively grasp the deviation of the position and posture of the three-dimensional shape in the design of the object.

According to the present invention, even in a vast construction site where the surrounding landscape is monotonous, a design image of the construction object can be superimposed and displayed on the actual construction object.

The figure which showed the whole structure of the image display system which concerns on one Embodiment. The figure which showed the functional structure of the two terminal devices which concerns on one Embodiment. The figure which showed the image which is displayed by the display means which concerns on one Embodiment. The figure which showed the whole structure of the image display system which concerns on one modification. The figure which showed what the user can see in the image display system which concerns on one modification.

[Embodiment]
Hereinafter, the image display system 1 according to the embodiment of the present invention will be described. The image display system 1 has a three-dimensional shape of a construction work constructed by a user such as a worker or an inspector at a construction site and a three-dimensional shape of the construction work in design (three-dimensional shape at the time of completion). It is a system that makes it possible to intuitively grasp the deviation.

FIG. 1 is a diagram showing the overall configuration of the image display system 1. Note that FIG. 1 shows, as an example, how the image display system 1 is used in the construction of a quadrangular pyramid trapezoidal construction object 9 using earth and sand with a backhoe 8.

The image display system 1 includes a GNSS (Global Navigation Satellite System) unit 11 (1) and a GNSS unit 11 (2) arranged at predetermined positions of a back hoe 8 (an example of an object photographed by the photographing means described in the claimed range). ) And the terminal device 12 (GNSS unit 11 (1)) that is arranged in the back hou 8 and specifies the position and orientation of the back hou 8 on the earth based on the position data output from the GNSS unit 11 (1) and the GNSS unit 11 (2). ) And the GNSS unit 11 (2) together with an example of the first measuring means described in the claim range), and the three-dimensional shape of the design of the construction 9 is superimposed on the photographed image of the actual construction 9. The terminal device 13 used for displaying the image is provided. The terminal device 12 and the terminal device 13 perform wireless communication. Further, the image displayed by the terminal device 13 is viewed by the user X. The user X may be either a worker, an inspector, or the like of the construction work 9.

Each of the GNSS unit 11 (1) and the GNSS unit 11 (2) receives radio waves transmitted from a plurality of artificial satellites, and measures the three-dimensional position of its own device on the earth based on the information obtained from the received radio waves. It is a device. The position measured by each of the GNSS unit 11 (1) and the GNSS unit 11 (2) is represented by, for example, latitude, longitude, and altitude. Since the GNSS unit 11 (1) and the GNSS unit 11 (2) are known GNSS units, the description of their configurations will be omitted.

The GNSS unit 11 (1) and the GNSS unit 11 (2) are arranged at different predetermined positions on the upper swing body of the backhoe 8. The positions of the GNSS unit 11 (1) and the GNSS unit 11 (2) in the back hoe 8 are the upper swivel bodies of the back hoe 8 (the portion arranged on the crawler and swiveling around the axis in the vertical direction with respect to the crawler, and the engine. , The fuel tank, the cockpit, etc., but the boom, arm, bucket, etc. whose position changes with respect to them are not included). It is a position specified by coordinates in a coordinate space (hereinafter, referred to as "object coordinate space") in which the positive axis direction and the upward direction are the positive Z-axis directions. Even if the position and orientation of the upper swivel body on the earth change, the positions of the GNSS unit 11 (1) and the GNSS unit 11 (2) in the object coordinate space do not change.

Hereinafter, unless otherwise specified, the term "backhoe 8 position" means the three-dimensional position of the origin of the object coordinate space on the earth.

Each of the GNSS unit 11 (1) and the GNSS unit 11 (2) is communicatively connected to the terminal device 12. The GNSS unit 11 (1) and the GNSS unit 11 (2) continuously measure the position of the own device on the earth, and sequentially transmit the position data indicating the measurement result to the terminal device 12.

The terminal device 12 is realized by the computer executing a process according to the program. That is, the hardware of the terminal device 12 is a communication interface for transmitting and receiving data between a memory that stores various data, a processor that performs various data processing according to a program stored in the memory, and an external device. It is a general computer equipped with.

The terminal device 13 is realized by a tablet-type computer having a built-in motion sensor and a camera executing a process according to a program. That is, the hardware of the terminal device 13 is a communication interface for transmitting and receiving data between a memory that stores various data, a processor that performs various data processing according to a program stored in the memory, and an external device. It is a general tablet computer equipped with an opaque touch display, a camera, and a motion sensor.

FIG. 2 is a diagram showing a functional configuration of the terminal device 12 and the terminal device 13. That is, the computer used as the hardware of the terminal device 12 functions as a device having the configuration of the terminal device 12 shown in FIG. 2 by performing various data processing according to the program for the terminal device 12. Further, the tablet-type computer used as the hardware of the terminal device 13 functions as a device having the configuration of the terminal device 13 shown in FIG. 2 by performing various data processing according to the program for the terminal device 13.

The functional configuration of the terminal device 12 will be described below. The storage means 121 stores various data. The storage means 121 stores in advance position data indicating the positions of the GNSS unit 11 (1) and the GNSS unit 11 (2) in the object coordinate space.

The receiving means 122 sequentially receives the position data continuously transmitted from the GNSS unit 11 (1) and the GNSS unit 11 (2). The position data received by the receiving means 122 is different from the position data stored in advance in the storage means 121, and is three-dimensional on the earth of the own device measured by each of the GNSS unit 11 (1) and the GNSS unit 11 (2). It is position data indicating a position.

The position / orientation calculating means 123 includes the positions of the GNSS unit 11 (1) and the GNSS unit 11 (2) in the object coordinate space indicated by the position data stored in advance in the storage means 121, and the receiving means 122 is the GNSS unit 11 (1). ) And the positions of the GNSS unit 11 (1) and GNSS unit 11 (2) on the earth indicated by the position data last received from the GNSS unit 11 (2), and the position and attitude of the back hou 8 on the earth are calculated. ..

The posture of the device means the degree to which the device is rotated from the reference position. There are various forms of posture expression, but in the present embodiment, the rotation angle around the X-axis clockwise (roll angle) and the rotation angle around the Y-axis clockwise (pitch angle) from the reference position of the device in the coordinate space. ), Z-axis clockwise rotation angle (yaw angle) shall represent the posture of the device. The attitude of the device on the earth is, for example, an X-axis clockwise rotation angle (roll angle) in a coordinate space in which the north direction is the X-axis positive direction, the east direction is the Y-axis positive direction, and the upward direction is the Z-axis positive direction. It is expressed by the Y-axis clockwise rotation angle (pitch angle) and the Z-axis clockwise rotation angle (yaw angle).

The transmission means 124 sequentially transmits the position / attitude data indicating the position and attitude of the backhoe 8 continuously calculated by the position / attitude calculation means 123 to the terminal device 13.

Subsequently, the functional configuration of the terminal device 13 will be described. The storage means 130 (an example of the storage means described in the claims) stores various data. In the storage means 130, first three-dimensional shape data (playing a role of indicating the position of the feature point) representing the three-dimensional shape of the portion of the upper swivel body of the back hoe 8 that can be seen from the outside, and the design of the construction body 9 The second three-dimensional shape data representing the three-dimensional shape, the position, and the posture of the above, and the positional relationship data showing a predetermined positional relationship between the photographing means 132 and the display means 139, which will be described later, are stored.

The first three-dimensional shape data represents the three-dimensional shape of the upper swing body of the backhoe 8 that does not change in the object coordinate space. The second three-dimensional shape data represents the three-dimensional shape at the time of design completion of the construction 9 that does not change in the coordinate space represented by latitude, longitude, and altitude.

The receiving means 131 sequentially receives the position / orientation data continuously transmitted from the terminal device 12. The photographing means 132 photographs an image around the terminal device 13 and generates image data representing the captured image. The recognition means 133 recognizes three or more feature points of the upper swing body of the backhoe 8 from the image taken by the photographing means 132. Since the method of recognizing the feature points by the recognition means 133 is a known image recognition method, the description thereof will be omitted.

The corresponding point specifying means 134 is a point on the three-dimensional shape represented by the first three-dimensional shape data, and identifies three or more corresponding points which are points corresponding to the three or more feature points recognized by the recognition means 133. .. Since the process performed by the corresponding point specifying means 134 is a known image process for matching between two images, the description thereof will be omitted.

The positional relationship specifying means 135 uses an image (back ho 8) used for recognizing the positions of the three or more corresponding points specified by the corresponding point specifying means 134 in the object coordinate space and the three or more feature points recognized by the recognition means 133. Based on the position in the captured image) and the positional relationship between the photographing means 132 and the display means 139 indicated by the positional relationship data stored in the storage means 130, the positional relationship between the back hoe 8 and the display means 139 is specified and specified. Generates positional relationship data indicating the positional relationship. The positional relationship data specified by the positional relationship specifying means 135 includes the position of the reference position of the display means 139 (for example, the center position of the display surface) in the object coordinate space and the posture (roll angle, pitch angle) of the display means 139 in the object coordinate space. , Yaw angle).

Since the process performed by the positional relationship specifying means 135 is a known process for identifying the position and orientation of the device having a predetermined positional relationship with the photographing means from the captured image, the description thereof will be omitted.

The position / orientation specifying means 136 includes the positional relationship between the back hou 8 and the display means 139 specified by the positional relationship specifying means 135, and the position and posture of the back hou 8 on the earth indicated by the position / orientation data received by the receiving means 131 from the terminal device 12. Based on the above, the position and orientation of the display means 139 on the earth at the time when the display means 139 takes a picture of the back hoe 8 are continuously specified, and position / orientation data indicating the specified position and attitude is generated.

The measuring means 137 (an example of the second measuring means described in the claims) continuously measures the change in the position and posture of the display means 139, and sequentially obtains the position / posture change data indicating the measurement result. Generate. The measuring means 137 is realized by a motion sensor built in a tablet-type computer, which is mainly the hardware of the terminal device 13. The motion sensor has, for example, a 3-axis acceleration sensor and a 3-axis angular velocity sensor, and continuously measures the change in the position of the display means 139 by the 3-axis acceleration sensor, and the posture of the display means 139 by the 3-axis angular velocity sensor. The change in is continuously measured.

By the way, during the period when the recognition means 133 does not recognize three or more feature points, the positional relationship specifying means 135 cannot specify the positional relationship between the backhoe 8 and the display means 139. Therefore, the position / orientation specifying means 136 cannot specify the position and orientation of the display means 139 on the earth by the above-mentioned method during the period when the recognition means 133 does not recognize three or more feature points. Therefore, the position / orientation specifying means 136 subsequently measures the position and orientation of the display means 139 finally identified by the above method on the earth during the period when the recognition means 133 does not recognize three or more feature points. By adding the changes in the position and posture of the display means 139 measured by the means 137 up to the present time, the position and the posture of the display means 139 on the earth at the present time are specified, and the position and attitude data indicating the specified position and the posture are generated. To do.

The image generation means 138 is based on the position and orientation of the display means 139 specified by the position and orientation specifying means 136 on the earth, and the three-dimensional shape of the construction body 9 represented by the second three-dimensional shape data and the position and orientation on the earth. , A work piece 9 constructed in a posture and shape to be constructed at a position to be constructed in a design in a virtual space in which the real space and the coordinate system captured by the photographing means 132 are common to the image captured by the photographing means 132. A display image in which the three-dimensional shapes of the above are superimposed is continuously generated, and display image data representing the generated display image is generated. When the position where the construction object 9 should be constructed is not within the photographing range of the photographing means 132, the image generating means 138 does not superimpose the three-dimensional shape of the construction object 9 on the image photographed by the photographing means 132.

The display means 139 displays a display image represented by the display image data generated by the image generation means 138.

FIG. 3 is a diagram showing a display image displayed by the display means 139. FIG. 3A is an example of a display image displayed by the display means 139 when the user X outside the backhoe 8 photographs the backhoe 8 using the terminal device 13. FIG. 3B is an example of a display image displayed by the display means 139 when the user X outside the backhoe 8 photographs the construction object 9 using the terminal device 13. FIG. 3C is an example of a display image displayed by the display means 139 when the user X in the backhoe 8 photographs the construction object 9 through the window of the backhoe 8 using the terminal device 13.

In the display image of FIG. 3A, the image of the outside of the back hoe 8, the numerical value indicating the position and orientation of the photographing means 132 on the earth, and the position and orientation are specified based on the image taken by the photographing means 132. The characters "measurement mode: image" are included to indicate that the position and the posture have been measured, and that the change in the position and the posture measured by the measuring means 137 has not been added. The circles drawn at the corners of the upper swing body of the backhoe 8 included in the display image of FIG. 3A indicate the positions of the feature points specified by the recognition means 133.

In the display image of FIG. 3B, the image of the construction object 9, the numerical value indicating the position and posture of the photographing means 132 on the earth, and the position and the posture are specified based on the image taken by the photographing means 132. The position and posture include the characters "measurement mode: image + motion" indicating that the position and posture are specified by adding the changes in the position and posture measured by the measuring means 137. .. Further, in the display image of FIG. 3B, a broken line representing the three-dimensional shape at the time of completion of the design of the construction object 9 is drawn on the image of the construction object 9.

The display image of FIG. 3C includes an image of the inside of the back hoe 8, an actual image of the construction object 9, numerical values indicating the position and orientation of the photographing means 132 on the earth, and the position and orientation of the photographing means 132. It is the position and posture specified based on the image taken by, and includes the characters "Measurement mode: image" indicating that the change in position and posture measured by the measuring means 137 is not added. ing. Further, in the display image of FIG. 3C, a circle indicating the position of the feature point specified by the recognition means 133 and a broken line indicating the three-dimensional shape at the time of completion of the design of the construction object 9 are drawn. ing.

By looking at the display images shown in FIGS. 3 (B) and 3 (C), the user X can determine the difference between the actual three-dimensional shape of the work piece 9 and the three-dimensional shape of the work piece 9 at the time of completion in design. It can be grasped intuitively. Therefore, the user X can efficiently perform work such as construction and inspection.

[Modification example]
The above-described embodiment is a specific example of the present invention, and can be variously modified within the scope of the technical idea of the present invention. Examples of these modifications are shown below. In addition, two or more modified examples shown below may be combined as appropriate.

(1) In the above-described embodiment, a tablet computer is used as the hardware of the terminal device 13. Instead of this, a computer that is wirelessly connected to the HMD (Head Mounted Display) for AR (Augmented Reality) and the HMD, or an HMD for AR built into the computer is used as the hardware of the terminal device 13. You may.

Further, instead of the HMD for AR, an HMD for MR (Mixed Reality) may be used. The HMD for MR has a transmissive display, and a user wearing the HMD on the head can simultaneously see the surrounding landscape seen through the display with the naked eye and the image displayed on the display.

FIG. 4 is a diagram showing the overall configuration of the image display system 2 according to this modified example. Compared with the image display system 1 shown in FIG. 1, the image display system 2 includes an MR-HMD23 which is an HMD (computer built-in type) for MR instead of the terminal device 13 which is a tablet type computer.

The functional configuration of the MR-HMD23 is the same as that of the terminal device 13 shown in FIG. 2, but the processing performed by the image generation means 138 is different from that of the terminal device 13. The image generation means 138 of the MR-HMD23 includes the position and orientation of the display means 139 specified by the position / orientation specifying means 136 on the earth, and the design three-dimensional shape and the earth of the construction 9 represented by the second three-dimensional shape data. Based on the position and orientation in the above and the positional relationship between the photographing means 132 and the display means 139 indicated by the positional relationship data stored in the storage means 130, the display means 139 mounted on the head of the user X is transmitted through the actual display means 139. Coordinates with the real space so that the user X who sees the space can see the three-dimensional shape of the construction object 9 superimposed on the position and the shape of the construction object 9 to be constructed at the position where the construction object 9 should be constructed. A display image is generated in which the three-dimensional shape of the construction object 9 represented by the second three-dimensional shape data is arranged in a virtual space having a common system.

FIG. 5 is a view showing what is visible to the user X wearing the MR-HMD23 in the image display system 2. FIG. 5A is an example showing what the user X looks like when the user X wearing the MR-HMD23 outside the backhoe 8 faces the backhoe 8. FIG. 5B is an example showing what the user X looks like when the user X wearing the MR-HMD23 outside the backhoe 8 faces the construction object 9. FIG. 5C is an example showing what the user X in the backhoe 8 can see when the user X wearing the MR-HMD23 faces the work piece 9.

In the case of the image display system 1, for example, when the user X is a worker who constructs the construction object 9 using the backhoe 8, the user X installs the terminal device 13 in a visible place in the cockpit of the backhoe 8 and actually installs the terminal device 13. While performing construction by operating the backhoe 8 while looking at the construction object 9, sometimes, in order to check how much the three-dimensional shape of the construction object 9 matches the three-dimensional shape at the time of completion of the design, the line of sight is terminal. It is necessary to move to the device 13. On the other hand, according to the image display system 2, the user X does not divert his / her line of sight from the actual construction object 9 and determines how much the three-dimensional shape of the construction object 9 is the three-dimensional shape at the time of completion of the design. You can always check.

(2) In the above-described embodiment, the position and orientation of the backhoe 8 on the earth are calculated by the terminal device 12. Instead, the terminal device 13 also serves as the terminal device 12, and the terminal device 13 receives and receives position data indicating the positions of the GNSS units 11 (1) and the GNSS unit 11 (2). The position and orientation of the backhoe 8 on the earth may be calculated based on the obtained position data.

(3) In the above-described embodiment, the construction site is assumed to be on the ground, but the construction site is not limited to the ground and may be on the water or the like. For example, when the construction site is on the water, the GNSS unit 11 (1), the GNSS unit 11 (2), and the terminal device 12 may be arranged on the ship instead of the backhoe. In this case, the user photographs the ship on which the GNSS unit 11 (1) and the GNSS unit 11 (2) and the terminal device 12 are arranged by the terminal device 13 to obtain a three-dimensional shape of the work on the water. It can be compared with the three-dimensional shape at the time of completion of the design.

(4) In the above-described embodiment, the hardware of the terminal device 12 and the terminal device 13 is a general computer. Instead, at least one of the terminal device 12 and the terminal device 13 may be configured as a dedicated device having the configuration shown in FIG.

(5) There are various GNSS methods. The method adopted for the GNSS unit 11 (1) and the GNSS unit 11 (2) may be any method as long as the accuracy required in the present invention is ensured. For example, the GNSS unit 11 (1) and the GNSS unit 11 (2) may be RTK (Real Time Kinematic) -GNSS system or D (Differential) -GNSS system GNSS unit.

(6) In the above-described embodiment, the image display system 1 uses the GNSS unit 11 (1) and the GNSS unit 11 (2) to specify the position and orientation of the backhoe 8. In the present invention, the method of specifying the position and orientation of an object is not limited to the method using two GNSS units. For example, the image display system 1 replaces the GNSS unit 11 (1) and the GNSS unit 11 (2) with a GNSS unit arranged at a predetermined position of an object, an azimuth meter, and a two-axis tilt meter orthogonal to each other on a horizontal plane. The position of the object may be specified based on the position measured by the GNSS unit, and the posture of the object may be specified based on the orientation measured by the azimuth meter and the inclination measured by the two-axis tilt meter. ..

Further, in the above modification, when the posture of the object with respect to the horizontal plane does not substantially change from the reference position, the image display system 1 does not have to be provided with an inclinometer. In that case, as the value representing the posture of the object, the value measured by the azimuth meter is used as the yaw angle, and 0 degrees (no change from the reference posture) is used as the roll angle and the pitch angle.

(7) In the above-described embodiment, the photographing means 132 is a general optical camera (a photographing means that generates a two-dimensional image by a group of image sensors that sense visible light). Instead of this, a depth sensor may be adopted as the photographing means 132, and the photographing means 132 may generate a depth image by photographing. The depth sensor method may be any of a stereo camera method, a time-of-flight method, a LIDAR method, and the like.

In this case, the photographing means 132 in which the backhoe 8 is photographed generates a depth image showing the three-dimensional shape of the backhoe 8. Then, the recognition means 133 recognizes the feature point from the three-dimensional shape of the backhoe 8 represented by the depth image generated by the photographing means 132. Then, the corresponding point specifying means 134 may specify the corresponding point corresponding to the feature point recognized by the recognition means 133 from the three-dimensional shape of the backhoe 8 represented by the first three-dimensional shape data.

Further, when the photographing means 132 generates a two-dimensional image by photographing, for example, the terminal device 13 measures the three-dimensional shape of the object from a plurality of images obtained by photographing the same object from different directions, which is called Photo to 3D. A three-dimensional shape specifying means for specifying the three-dimensional shape of the back hoe 8 by a known technique may be provided. In this case, the recognition means 133 recognizes the feature points of the three-dimensional shape of the backhoe 8 specified by the three-dimensional shape specifying means. Then, the corresponding point specifying means 134 may specify the corresponding point corresponding to the feature point recognized by the recognition means 133 from the three-dimensional shape of the backhoe 8 represented by the first three-dimensional shape data.

(8) In the above-described embodiment, the backhoe 8 shown as an example of an object to be photographed by the photographing means 132 in order to specify the position and orientation of the display means 139 on the earth is a self-propelled moving body. The object to be photographed in the present invention may be a moving object that does not run on its own or an object that is not a moving object. For example, if the position and orientation of the workbench towed by the towboat can be obtained, the workbench may be used as a photographing target for identifying the position and attitude of the display means, and the workbench does not move during construction. A construction machine or the like may be used as a photographing target. That is, any object having feature points and whose position and posture on the earth can be measured by a measuring means can be an object to be photographed in the present invention.

(9) A program for causing a computer to execute a process performed by the terminal device 13 may be provided via a network such as the Internet. Further, a program for causing the computer to execute the process performed by the terminal device 13 may be provided in a state of being recorded on a recording medium for continuously recording data.

1 ... Image display system, 2 ... Image display system, 8 ... Back ho, 9 ... Construction, 12 ... Terminal device, 13 ... Terminal device, 23 ... MR-HMD, 121 ... Storage means, 122 ... Receiving means, 123 ... Position Posture calculation means, 124 ... Transmission means, 130 ... Storage means, 131 ... Reception means, 132 ... Shooting means, 133 ... Recognition means, 134 ... Correspondence point identification means, 135 ... Positional relationship identification means, 136 ... Position / orientation identification means, 137 ... Measuring means, 138 ... Image generating means, 139 ... Displaying means.

Claims (8)

Shooting means and
A display means having a predetermined positional relationship with the photographing means,
A measuring means for measuring the position and orientation of an object to be photographed on the earth by the photographing means, and a measuring means.
Positional relationship data indicating the positional relationship between the photographing means and the display means, first three-dimensional shape data representing the three-dimensional shape of the object, the three-dimensional shape in the design of the construction, and the position and orientation on the earth. A storage means for storing the second three-dimensional shape data representing
A recognition means by which the photographing means recognizes three or more feature points of the object from an image obtained by photographing the object.
A corresponding point specifying means for specifying three or more corresponding points, which are points on the three-dimensional shape represented by the first three-dimensional shape data and corresponding to the three or more feature points recognized by the recognition means. ,
The positional relationship between the object and the photographing means is specified based on the positions of the three or more corresponding points in the object and the positions of the three or more feature points in the image obtained by photographing the object, and the positional relationship and the position are specified. A positional relationship specifying means for specifying the positional relationship between the object and the display means based on the positional relationship indicated by the relationship data, and
Based on the positional relationship between the object and the display means specified by the positional relationship specifying means, and the position and posture of the object on the earth measured by the measuring means at the time of photographing the image of the object, the photographing time point. Position and orientation specifying means for specifying the position and orientation of the display means on the earth in
Based on the position and orientation of the display means specified by the position and orientation specifying means on the earth, the design three-dimensional shape of the construction represented by the second three-dimensional shape data, and the position and orientation on the earth. It is equipped with an image generation means for generating a display image showing the three-dimensional shape of the construction object constructed at a design position in a design position in a virtual space in which the real space and the coordinate system are common to be photographed by the photographing means. ,
The display means is an image display system that displays the display image. When the measuring means is used as the first measuring means, a second measuring means for continuously measuring the change in the position and posture of the display means is provided.
The position / orientation specifying means includes the position and orientation of the display means on the earth at the specified shooting time, and the change in the position and posture of the display means measured by the second measuring means from the shooting time to the present time. The image display system according to claim 1, wherein the position and orientation of the display means on the earth at the present time are specified based on the above. The image display system according to claim 1 or 2, wherein the display image is an image obtained by superimposing an image representing the design three-dimensional shape of the construction on the image taken by the photographing means. The display means is a transparent display means worn on the user's head.
The display image is applied to the user who sees the real space through the display means in a posture and shape to be installed at a design position in a virtual space in which the real space and the coordinate system are common. The image display system according to claim 1 or 2, which is an image in which the three-dimensional shapes of the constructed works are superimposed. On the computer
Positional relationship data showing a predetermined positional relationship between the photographing means and the display means, first three-dimensional shape data showing the three-dimensional shape of an object whose position and orientation on the earth are measured by the measuring means, and design of the construction And the process of acquiring the second 3D shape data representing the 3D shape of the
A process of acquiring image data representing an image in which the object is photographed by the photographing means and position / orientation data indicating the position and orientation of the object measured by the measuring means at the time of photographing the image.
A process of recognizing three or more feature points of the object from the image represented by the image data,
A process of specifying three or more corresponding points, which are points on the three-dimensional shape represented by the first three-dimensional shape data and corresponding to the recognized three or more feature points.
The positional relationship between the object and the photographing means is specified based on the positions of the three or more corresponding points in the object and the positions of the three or more feature points in the image obtained by photographing the object, and the positional relationship and the position are specified. A process of specifying the positional relationship between the object and the display means based on the positional relationship indicated by the relationship data, and
A process of specifying the position and orientation of the display means on the earth at the time of shooting based on the positional relationship between the specified object and the display means and the position and orientation of the object on the earth indicated by the position and orientation data.
The actual image taken by the photographing means based on the position and orientation of the specified display means on the earth, the design three-dimensional shape of the construction object represented by the second three-dimensional shape data, and the position and orientation on the earth. A process of generating a display image showing the three-dimensional shape of the work constructed at a design position in a design posture in a virtual space having a common space and coordinate system.
A program for executing a process of displaying the display image on the display means. On the computer
When the measuring means is used as the first measuring means, the position of the display means from the time of shooting to the present time measured by the second measuring means that continuously measures the change in the position and posture of the display means and Processing to acquire position / posture change data indicating the change in posture,
The display at the present time is based on the position and orientation of the display means on the earth at the specified shooting time and the change in the position and posture of the display means from the shooting time to the present time indicated by the position / posture change data. The program according to claim 5, for performing a process of identifying the position and orientation of the means on the earth. On the computer
The program according to claim 5 or 6, wherein as the display image, an image obtained by superimposing an image representing a design three-dimensional shape of the construction on the image taken by the photographing means is generated. On the computer
As the display image, the user who sees the real space through the transparent display means mounted on the user's head should be constructed in a virtual space in which the real space and the coordinate system are common. The program according to claim 5 or 6, for generating an image in which a three-dimensional shape of the construction piece constructed in a posture and shape to be constructed at a position is superimposed.

Images