JP2018106661A - Inconsistency detection system, mixed reality system, program, and inconsistency detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system which can detect a geometric inconsistency in a mixed reality system.SOLUTION: The inconsistency detection system in a mixed reality system comprises a portable display device which has a see-through display unit and a photography unit which photographs a real space, where a virtual object is rendered on the display unit and a user views the same within a prescribed real space. The mixed reality system comprises: means to store three-dimensional spatial data; means to determine a user environment; and means to render the virtual object on the display unit on the basis of the user environment. First point group data is generated from a composite image which is obtained by displaying the virtual object superimposed on a naked-eye visual field image, second point group data is generated using point group data of the virtual object and point group data of the three-dimensional space in the determined user environment, and an inconsistency is detected on the basis of a result of a comparison between the first point group data and the second point group data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an inconsistency detection system and the like, and more particularly to an inconsistency detection system and the like in a mixed reality system that allows a user in a predetermined real space to experience mixed reality.

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology has been known as a technology for seamlessly combining the real world and the virtual world in real time. MR technology can make a user who experiences this experience feel as if a virtual object exists on the spot, and is attracting attention in various fields. The user can confirm the mixed reality image superimposed on the HMD by wearing an optical see-through HMD (Head Mounted Display) or a video see-through HMD, and can experience mixed reality.

  There are two known technologies for recognizing the real world as a 3D space: a method of mounting a highly accurate camera from the user's viewpoint and a method of installing a camera so as to surround the observation target space.

  As a method for mounting a highly accurate camera from the user's viewpoint, there is a method using an infrared projector and an infrared camera. For example, as a technique for mounting a highly accurate camera from the user's viewpoint, there is a technique for irradiating infrared rays and measuring the depth of an object from the distortion of the reflection pattern. In addition, in a method called Time of Flight method (TOF method), there is a method of calculating a reciprocating distance to an object by irradiating invisible light such as infrared rays and measuring a reflection result thereof. These systems have a problem that the space that can be three-dimensional is limited to the infrared reachable range and cannot be used due to the solar effect.

  In addition, as a method for photographing extremely accurate three-dimensional information, there is a method using a 3D laser scanner. Although this method can achieve high measurement accuracy, it takes a time of at least about 10 minutes and a standard image quality of about 30 minutes to perform a 360 degree measurement. Therefore, it cannot be applied to real-time usage. In addition, one 3D laser scanner is very expensive at several hundred to several tens of millions of yen, and is not suitable for mass deployment in a wide area. Due to the above characteristics, the 3D laser scanner has been used for high-precision 3D conversion over a long period of time, such as surveying in civil engineering and layout confirmation in factories. For example, Patent Document 1 realizes a technology for recognizing the real world as a 3D space by adding the color of an image captured by a camera to point cloud data created by scanning with a 3D image scanner.

  As a method of installing a camera so as to surround the observation target space, a technique called Structure-from-Motion (SfM) shown in Non-Patent Literature 1 is used to obtain a plurality of images taken from a plurality of disjoint images. A method for restoring the dimension information can be mentioned. A typical product that adopts this method is Microsoft Photosynthesis (registered trademark). Although the accuracy realized by SfM is relatively low in comparison with the accuracy required by the MR technology, it is a method that can construct a 3D model at low cost. However, the real-time property is low and cannot be applied to the realization of the MR environment as it is.

  As described above, these methods cannot simultaneously realize real-time performance and high-accuracy measurement.

JP 2013-69235 A

Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2011. Building Rome in a day. Commun. ACM 54, 10 (October 2011), 105-112. DOI = http : //dx.doi.org/10.1145/2001269.2001293

  As described above, a method for recognizing a real-world structure and environment in real time with high accuracy and reflecting it in the virtual space has not yet been established. Under such circumstances, the present applicant has proposed a mixed reality system capable of recognizing the real space environment in real time in the mixed reality environment in Japanese Patent Application No. 2006-121928.

  The mixed reality system generates a high-accuracy three-dimensional virtual space having a structure and color that matches the real space with extremely high accuracy, thereby realizing global illumination using the state of light in the real world. Realize highly accurate position tracking of users.

  Here, one of the main purposes of the mixed reality system is a seamless mixed reality in which an object in the virtual space (virtual object) and an object such as the ground or building in the real space (real object) almost exactly contact each other. It is to provide space to the user.

  In order to realize the mixed reality system as described above, it is necessary to maintain the geometric consistency in which the virtual object and the real object are correctly superimposed in real time according to the current position and movement of the HMD. . In particular, when displaying humans and animals standing on the ground or touching objects as virtual characters (virtual objects), the integrity of the boundary processing (alignment processing), that is, geometrical consistency, is virtual. It is extremely important to give reality to objects.

  However, in the optical see-through type HMD, a real object as a target for superimposing a virtual object cannot be a target for video (image) processing, and only simple optical superimposition can be performed, thereby ensuring alignment processing integrity. It is extremely difficult to do. The integrity of the alignment process (geometric consistency) depends on the accuracy of the user's position tracking, but the geometric mismatch between the virtual object and the real object depends on the accumulation of position tracking errors. It is what happens. However, the self-position estimation technique used for position tracking in a mixed reality system has a problem that it cannot self-detect (self-diagnose) a recognition error of several centimeters that causes geometric mismatch. .

  The present invention has been made to solve such problems, and in a mixed reality system using an optical see-through HMD, there is a contradiction in the geometric consistency between an object in a virtual space and an object in a real space. It is a main object of the present invention to provide a mismatch detection system capable of detecting a mismatch when an error occurs.

  In order to achieve the above object, an inconsistency detection system according to one aspect of the present invention is portable having a transmissive display unit for displaying a virtual object to a user and a photographing unit for photographing a real space. Inconsistency detection system in a mixed reality system that includes a display device and allows a user to visually recognize a virtual object superimposed on the real space through the display unit by drawing the virtual object on the display unit in a predetermined real space In the mixed reality system, three-dimensional space data including point cloud data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information, is obtained. Based on the three-dimensional spatial data storage means for storing, the data acquired from the sensor provided in the display device, and the three-dimensional spatial data A user environment determining unit that determines a user environment including a position of the display device and a visual field area visually recognized by the user through the display unit, and a drawing unit that draws a virtual object on the display unit based on the user environment. From the synthesized image obtained by superimposing the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit generated from the photographed real space image, the real object And first point cloud data generation means for generating first point cloud data that is point cloud data of the virtual object, and storage by the three-dimensional spatial data storage means in the field-of-view area determined by the user environment determination means. Using the point cloud data thus created and the point cloud data of the virtual object drawn by the drawing means. Second point cloud data generating means for generating point cloud data; and mismatch detection means for detecting mismatch based on a comparison result between the first point cloud data and the second point cloud data. It is characterized by that.

  In the present invention, it is preferable that the photographing unit acquires a real space as a stereo image, and the first point cloud data generation unit generates each of the two real space images acquired as the stereo image. The first point cloud data is generated from each composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on each real space image visually recognized by the user through the display unit.

  In the present invention, it is preferable that the first point cloud data generation unit performs projective transformation on the photographed real space image based on a positional relationship between the display unit and the imaging unit, thereby passing through the display unit. A real space image visually recognized by the user is generated.

  In the present invention, it is preferable that the inconsistency detection unit detects inconsistency when a difference between the first point cloud data and the second point cloud data exceeds a predetermined amount.

  In the present invention, it is preferable that the inconsistency detection unit includes the first point cloud data and the second point cloud data in a predetermined three-dimensional space area including a virtual object drawn based on the user environment. A mismatch is detected when the difference between the two exceeds a predetermined amount.

  In order to achieve the above object, a mixed reality system as one aspect of the present invention includes a server, a transmissive display unit for displaying a virtual object to a user, and a photographing unit for photographing a real space. And a portable display device that includes a portable display device that draws a virtual object on the display unit in a predetermined real space so that the user can visually recognize the virtual object superimposed on the real space through the display unit. 3D space data for storing 3D space data including point cloud data each having 3D position information, which is point cloud data of a real object in the predetermined real space acquired in advance. Based on the data acquired from the sensor provided in the storage device and the display device and the three-dimensional spatial data, the position of the display device and the display unit are determined. And a user environment determining means for determining a user environment including a visual field area visually recognized by the user, a drawing means for drawing a virtual object on the display unit based on the user environment, and the captured real space image. A first point cloud that is point cloud data of the real object and the virtual object from the composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit First point cloud data generating means for generating data, point cloud data stored by the three-dimensional spatial data storage means in the field of view determined by the user environment determining means, and virtual drawn by the drawing means Second point cloud data generating means for generating second point cloud data using the point cloud data of the object; Inconsistent detecting means for detecting a mismatch based on the first point group data and the comparison result of the second point group data, the means of the characterized in that it comprises the said server or said display device.

  In order to achieve the above object, a program according to one aspect of the present invention is portable having a transparent display unit for displaying a virtual object to a user and a photographing unit for photographing a real space. An inconsistency is detected in a mixed reality system that includes a display device and draws a virtual object on the display unit in a predetermined real space so that the user can visually recognize the virtual object over the real space through the display unit. The mixed reality system is a three-dimensional program including point cloud data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information. Three-dimensional spatial data storage means for storing spatial data; data acquired from sensors included in the display device; and the three-dimensional spatial data A user environment determining unit that determines a user environment including a position of the display device and a visual field area visually recognized by the user through the display unit, and a drawing unit that draws a virtual object on the display unit based on the user environment The program superimposes on the display device a virtual object drawn by the drawing means on a real space image visually recognized by the user through the display unit generated from the photographed real space image. Generating first point cloud data that is point cloud data of a real object and a virtual object from a composite image obtained by displaying, and the three-dimensional space in the field-of-view area determined by the user environment determining means Point cloud data stored by the data storage means and virtual objects drawn by the drawing means Generating second point cloud data using the point cloud data of the target, and detecting inconsistencies based on a comparison result between the first point cloud data and the second point cloud data. It is made to perform.

  In order to achieve the above object, an inconsistency detection method according to an aspect of the present invention includes a transmissive display unit for displaying a virtual object to a user and a photographing unit for photographing a real space. Inconsistency in a mixed reality system that includes a portable display device and allows a user to visually recognize a virtual object superimposed on the real space through the display unit by drawing the virtual object on the display unit in a predetermined real space In the detection method, the mixed reality system is a three-dimensional space including point cloud data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information. Based on the three-dimensional spatial data storage means for storing data, the data acquired from the sensor included in the display device, and the three-dimensional spatial data User environment determining means for determining a user environment including a position of a display device and a visual field area visually recognized by the user through the display section, and a drawing means for drawing a virtual object on the display section based on the user environment From the composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit generated from the photographed real space image, the real object and Generating first point cloud data which is point cloud data of a virtual object; point cloud data stored by the 3D spatial data storage means in the field of view determined by the user environment determining means; and the drawing Second point cloud data is generated using the point cloud data of the virtual object drawn by the means A step that is characterized by having the steps of: detecting a mismatch based on a comparison result of the first point group data and the second point group data.

  According to the present invention, in a mixed reality system using an optical see-through HMD, inconsistency can be detected when a contradiction occurs in the geometric consistency between a virtual space object and a real space object. .

1 is an overall configuration diagram of an MR system according to an embodiment of the present invention. It is a figure which shows the outline | summary of the position tracking by one Embodiment of this invention. It is a figure which shows the condition where the character of a virtual world leans against the wall of the real world in MR system by one Embodiment of this invention. It is a figure which shows the condition where the character of a virtual world leans against the wall of the real world in MR system by one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the server by one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the display apparatus by one Embodiment of this invention. It is an example of the schematic block diagram of HMD by one Embodiment of this invention. 1 is an overview of real space according to an embodiment of the present invention. It is the top view which looked at the real space of FIG. 8 from the top. It is the three-dimensional space data expressed by the point cloud data acquired in the real space of FIG. It is the three-dimensional space data expressed by the voxel created from the point cloud data of FIG. It is a functional block diagram of MR system by one Embodiment of this invention. It is the schematic explaining the production | generation of the 1st point cloud data and 2nd point cloud data by one Embodiment of this invention. It is the schematic explaining the production | generation of the 1st point cloud data and 2nd point cloud data by one Embodiment of this invention. It is a functional block diagram of the inconsistency detection system by one Embodiment of this invention. 6 is a flowchart illustrating information processing for detecting inconsistencies in an MR system according to an embodiment of the present invention. 1 is an overview of real space according to an embodiment of the present invention.

  Hereinafter, a mixed reality (MR) system that provides a mixed reality space in which a virtual space and a real space are fused to a user and a mismatch detection system in the MR system will be described with reference to the drawings. In the drawings, the same reference numerals denote the same or corresponding parts unless otherwise specified.

  One of the technical features of the MR system according to the embodiment of the present invention is that the MR system is realized in a predetermined real space (predetermined real space), and has a structure that matches the real space with extremely high accuracy. And generating a highly accurate three-dimensional virtual space having colors. Thereby, it becomes possible to realize highly accurate position tracking of the user.

  FIG. 2 shows a real world 21 in which a user is present, a virtual world 22 constructed by three-dimensional spatial data (DB), and a mixed real world (MR environment) 23 in an MR system generated by matching them. . In the MR environment 23, the user generally wears a device having a display unit such as an HMD. Various types of sensor devices are mounted on the HMD. In order to realize the MR environment 23 that recognizes the structure of the real world 21 with high accuracy and reflects it in the virtual space 22, measurement of sensors used in the past is used. Accuracy is not enough.

  Therefore, the MR system of the present embodiment performs matching between rough position information obtained from various conventionally used sensors and the high-precision virtual space 22, and the user's position in the real space 21 and the virtual space 22 and the user's position. Correct the deviation in the direction you are facing in real time. Conventionally used various sensors include a distance sensor, an image sensor, a direction sensor, a GPS sensor, a Bluetooth (registered trademark) beacon, and the like. As a result, the MR environment 23 that links the reality and the high accuracy is realized, and the position tracking of the user is realized. In such an MR environment 23, when the character 24 stands on the platform 26 of the virtual world 22, the user can visually recognize that the character 24 stands on the platform 25 of the real world 21 without a sense of incongruity. it can. The stand 26 is not displayed as a virtual object in the MR environment 23.

  As described above, in the MR system of the present embodiment, geometrical consistency (collision relationship, context, occlusion, etc.) with a real object (building, bench, etc.) when displaying a virtual object is included in the virtual object. It is extremely important in providing reality. For example, FIGS. 3 and 4 show a situation in which the character 24 of the virtual world 22 leans against the wall of the real world 21. In FIG. 3, the arm of the virtual character (virtual object) 24 is excessively buried in the wall (real object) 27. In FIG. 4, a part of the virtual character 24 is unnecessarily shaved. I'm stuck. Such a display must be prevented because it impairs the reality. However, the self-position estimation technique used for the position tracking described above cannot self-detect a three-dimensional space recognition error.

  The inconsistency detection system according to the embodiment of the present invention detects an inconsistency when a contradiction occurs in the geometric consistency between a virtual object and a real object, for example, as shown in FIGS. . A specific configuration will be described below. For convenience of explanation, an MR system including a mismatch detection system will be described first.

  FIG. 1 is an example of an overall configuration diagram of an MR system 10 according to an embodiment of the present invention. As shown in FIG. 1, the MR system 10 includes a server 100, one or more display devices 200, and one or more image acquisition devices 300. The server 100, the display device 200, and the image acquisition device 300 are connected to a network 50 such as the Internet and can communicate with each other. Note that the image acquisition device 300 is a device necessary for maintaining in real time the optical consistency in which the shadow of the real object and the virtual object are displayed without a sense of incongruity in the MR system 10. Therefore, the MR system 10 according to the embodiment of the present invention may not include the image acquisition device 300.

  The MR system 10 is assumed to be a server-client system. Preferably, the display device 200 and the image acquisition device 300 communicate only with the server 100. However, it can be configured by a system without a server such as PtoP.

  The MR system 10 allows a user in a predetermined real space to experience mixed reality. The predetermined real space is a predetermined indoor or outdoor real space, and a real object that is an object in the real world exists in the space. The real object is a structure such as a building, a bench, or a wall, and is fixed in the real space. However, a movable object may be included in the real object.

  FIG. 5 is a block diagram showing a hardware configuration of the server 100 according to the embodiment of the present invention. The server 100 includes a processing unit 101, a display unit 102, an input unit 103, a storage unit 104, and a communication unit 105. Each of these components is connected by the bus 110, but may be individually connected as necessary.

  The processing unit 101 includes a processor (for example, a CPU) that controls each unit included in the server 100, and performs various processes using the storage unit 104 as a work area. When the server 100 draws a virtual object, the processing unit 101 preferably includes a GPU that performs drawing processing separately from the CPU.

  The display unit 102 displays information to the server user, and the input unit 103 receives input from the user to the server 100, and is, for example, a touch panel, a touch pad, a keyboard, or a mouse.

  The storage unit 104 includes a hard disk, a main memory, and a buffer memory. A program is stored in the hard disk. However, the hard disk may be any non-volatile storage or non-volatile memory as long as it can store information, and may be removable. The storage unit 104 stores a program and various data that can be referred to when the program is executed. When the processing unit 101 includes a GPU, the storage unit 104 can include a video memory.

  The storage unit 104 can store data (for example, tables) and programs for various databases. Various databases are realized by the operation of the processing unit 101 and the like. For example, the server 100 may have a database server function, may include a database server, or may include or include other servers. In one example, the server 100 includes a database for three-dimensional spatial data that constructs a virtual space, and the storage unit 104 stores data and programs for the database.

  The communication unit 105 performs wired communication using an Ethernet (registered trademark) cable, mobile communication, wireless communication such as a wireless LAN, and connects to the network 50.

  The server 100 realizes various functions by executing a program, but some of these functions can also be realized by configuring an electronic circuit or the like.

  In one example, the server 100 is configured by combining a plurality of servers installed for each function or area. For example, a predetermined real space may be divided into a plurality of areas, and one server may be installed in each area, and a server that integrates these servers may be installed.

  In one example, when the server 100 draws a virtual object, the CPU writes a drawing command in the main memory, and the GPU writes drawing data in a frame buffer on the video memory with reference to the drawing command. Thereafter, the data read from the frame buffer is transmitted to the display device 200 as it is.

  FIG. 6 is a block diagram showing a hardware configuration of the display device 200 according to the embodiment of the present invention. The display device 200 includes a processing unit 201, a display unit 202, a photographing unit 203, a storage unit 204, a communication unit 205, and a sensor 206. Each of these components is connected by the bus 210, but each may be individually connected as necessary.

  The display device 200 is portable by the user, and is preferably a head-mounted image display device (HMD) that can be worn on the head. Hereinafter, in the embodiment of the present invention, the HMD 200 is used as the display device 200.

  The processing unit 201 includes a processor (for example, a CPU) that controls each unit included in the HMD 200, and performs various processes using the storage unit 204 as a work area. In one example, the HMD 200 receives a drawing command from the server 100 and performs a drawing process. In this case, the processing unit 201 includes a GPU that performs drawing processing separately from the CPU.

  The storage unit 204 includes a hard disk, a main memory, and a buffer memory. A program is stored in the hard disk. However, the hard disk may be any non-volatile storage or non-volatile memory as long as it can store information, and may be removable. The storage unit 204 stores a program and various data that can be referred to when the program is executed. When the processing unit 201 includes a GPU, the storage unit 204 can include a video memory. Further, the storage unit 204 can store data and programs for various databases. In this case, various databases are realized by the operation of the processing unit 201 and the like.

  The display unit 202 is a transmissive display that can display a virtual object to the user. That is, in the embodiment of the present invention, the HMD 200 is an optical see-through HMD. The HMD 200 allows the user wearing the HMD 200 to visually recognize the real space through the display unit 202 and, when drawing a virtual object on the display unit 202, superimposes the virtual object on the real space through the display unit 202 to the user. It can be visually recognized.

  In the embodiment of the present invention, when generating a virtual object to be drawn, the MR system 10 generates a right-eye image for visualizing the user's right eye and a left-eye image for visualizing the user's left eye. . Accordingly, the display unit 202 includes a right-eye transmissive display that displays a right-eye image and a left-eye transmissive display that displays a left-eye image. Alternatively, the display unit 202 may include a single transmissive display, and the display area of the display may include a right-eye image display area and a left-eye image display area.

  The imaging unit 203 includes a stereo camera that captures a real space, and acquires the real space as a stereo image. The imaging unit 203 stores the captured image of each frame (real world image) in the storage unit 204. In one example, the photographing unit 203 includes a right-eye camera for photographing a real space visually recognized by the user's right eye and a left-eye camera for photographing a real space visually recognized by the user's left eye. The However, the photographing unit 203 may include a monocular camera and may be configured to acquire the real space as a stereo image using a known method.

  FIG. 7 is a schematic configuration diagram according to an embodiment of the HMD 200. As shown in FIG. 7, the HMD 200 includes a display unit 202 including a right-eye transmissive display 202 a that displays a right-eye image and a left-eye transmissive display 202 b that displays a left-eye image. The HMD 200 includes a photographing unit 203 having a right-eye camera 203a installed in the vicinity of the right-eye transmissive display 202b and a left-eye camera 203b installed in the vicinity of the left-eye transmissive display.

  The right-eye transmissive display 202 a is a right-eye transmissive display, and can display a right-eye image generated by the MR system 10. In the case of a transmissive display, the real-world image is seen as it is in the area where no image is displayed. As a result, the virtual-world image (virtual object) and the real-world image (real object) are optical. Will be synthesized. The same applies to the transmissive display 203 for the left eye.

  The right-eye camera 203a is a camera for photographing a real-world image that is viewed through the right-eye transmissive display 202a. As will be described later, in the embodiment of the present invention, the MR system 10 generates a real space image (visual field image) that is visually recognized by the user through the right-eye transmissive display 202a from the image captured by the right-eye camera 203a. Therefore, the internal parameters such as the angle of view and the visual field range of the right-eye camera 203a need to be calibrated with the space displayed by the right-eye transmissive display 202a. Specifically, by applying a projection matrix calculated from internal parameters to an image obtained from the right-eye camera 203a, an image that is visually recognized (viewed) by the user through the right-eye transmissive display 203a can be obtained. . The same applies to the left-eye camera 203b.

  The sensor 206 is various sensors necessary for using the self-position estimation technology. In one example, the sensor 206 includes an acceleration sensor, a gyro sensor, an infrared depth sensor, and a camera. The infrared depth sensor is a depth sensor by infrared projection, but the same function may be realized by an RGB-D camera. The camera is a camera (monocular camera or the like) different from the stereo camera included in the photographing unit 203, but a stereo camera included in the photographing unit 203 can also be included as the sensor 206. In addition, the sensor 206 can include various other conventionally used sensors such as a GPS sensor, a Bluetooth (registered trademark) beacon, and WiFi. In another example, the image acquisition device 300 may be included as the sensor 206.

  The communication unit 205 performs wireless communication such as mobile communication and wireless LAN, and connects to the network 50. In one example, the communication unit 205 receives 3D spatial data from the server 100. In one example, the communication unit 205 transmits image data captured by the imaging unit 203 to the server 100 via the network 50.

  The image acquisition device 300 acquires a real space video (image) and transmits the acquired image data to the server 100 and the display device 200 via the network 50. In addition, the image acquisition device 300 is installed so as to surround a predetermined real space and fix a region visible to the user in the predetermined real space to a fixed point that can be photographed. In one example, the image acquisition device 300 is a fixed camera installed at a fixed point. In one example, the image acquisition device 300 acquires 30 frames of images per second and transmits them to the server 100.

  Here, the MR system 10 according to the embodiment of the present invention is an indoor space covered with a wall 36 as shown in FIG. 8 as a real-world space (predetermined real space) for providing the mixed reality space 33 to the user. A real space 31 is assumed. 8 is an overview of the real space 31, and FIG. 9 is a plan view of the real space 31 as viewed from above. In the real space 31, there are a light source 34 in the real world and a building 35 that is a real object in the real world. As shown in FIG. 8, in the real space 31, a plurality of image acquisition devices 300 are installed so that a space to be observed, that is, a space visible to the user in the real space 31 can be photographed. However, in the embodiment of the present invention, the image acquisition device 300 may not be installed.

  FIG. 10 shows an example of point cloud data acquired in the real space 31. In the embodiment of the present invention, point cloud data indicating the three-dimensional shape of a real object in the real space 31 is acquired in advance by using, for example, a high-precision 3D laser scanner (not shown). When the image acquisition device 300 is installed, the point cloud data is preferably acquired after the image acquisition device 300 is installed in the real space 31.

  As shown in FIG. 10, each point cloud data has three-dimensional coordinates (x, y, z) and is arranged in a virtual space 32 associated with the real space 31. Each of the point cloud data is colored point cloud data having color information. Colored point cloud data is obtained by obtaining color information obtained from an image taken separately from the point cloud data acquisition using a camera included in the laser scanner, according to each coordinate point ( x, y, z).

  As described above, the point cloud data can be used as a basic unit (basic component) to represent the three-dimensional shape of the real object in the virtual space 32 associated with the real space 31. In this specification, this basic unit is expressed as a three-dimensional shape element.

  In the embodiment of the present invention, for ease of explanation, the acquired point cloud data is converted into, for example, OctoMap (“OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees” in Autonomous Robots, 2013; A. Hornung,. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard (http://dx.doi.org/10.1007/s10514-012-9321-0) DOI: 10.1007 / s10514-012-9321-0.) Using a known method, it is converted into a data structure called a voxel. A voxel is a unit component in a three-dimensional space corresponding to a pixel in the two-dimensional space, and is a cube having a certain size that is identified using coordinates in the three-dimensional space.

  Hereinafter, in the embodiment of the present invention, a basic unit representing a three-dimensional shape of a real object in a virtual space 32 associated with the real space 31 is a voxel. That is, the voxel is a three-dimensional shape element. FIG. 11 shows voxels created from the point cloud data of FIG. The MR system 10 stores point cloud data and voxel data in the storage unit 104 or 204 as three-dimensional spatial data. However, a mesh (3D mesh) can also be used as the three-dimensional shape element. In this case, the MR system 10 stores point cloud data and 3D mesh data in the storage unit 104 or 204 as three-dimensional space data. The point cloud data itself can also be used as a three-dimensional shape element. In this case, the MR system stores point cloud data as three-dimensional space data.

色情報ｃは、ＲＧＢやＨＳＶ等のフォーマットで表現される。例えばＨＳＶのフォーマットで表現される場合、色情報は、色相、彩度、及び明度を有する。 In one example, the real space is divided into 1 cm ³ voxel frames, and a voxel space (a three-dimensional space expressed by voxels) is set. One voxel V has one or a plurality of color information c when viewed from the image acquisition apparatus 300 in addition to the position information x, y, z.

The color information c is expressed in a format such as RGB or HSV. For example, when expressed in the HSV format, the color information includes hue, saturation, and brightness.

  In the embodiment of the present invention, it is assumed that the virtual space 32 is limited to a region (0 ≦ X ≦ X1, 0 ≦ Y ≦ Y1, 0 ≦ Z ≦ Z1) associated with the real space 31. However, the virtual space 32 can be set to a smaller area, and each of the plurality of virtual spaces can be associated with the real space 31, or the virtual space 32 can be set to a larger area. When the virtual space 32 is set to a smaller area, the image acquisition device 300 may be installed at each of a plurality of fixed points where the user can visually recognize an area visible in the real space 31 corresponding to each virtual space 32. it can.

In one example, when the real space 31 is a wide area, it is preferable that the virtual space 32 is set for each of a plurality of areas, and the server 100 is also installed for each of the plurality of areas. For example, when the storage unit 104 of the server 100 stores voxel data, one voxel V in Expression (1) becomes 10 bytes if 1 voxel = (int16 x, int16 y, int16 z, int32 rgb). . For example, the real space is modeled as a set of 10 mm ³ voxels. When converting a wide-area space into a high-precision voxel space like a theme park, if this theme park area is divided into 10 m = 10,000 mm grids, one grid is 1,000 m ³ , which is about 1 billion voxel. If this is reduced to a space up to 5m high, it will be about 500 million voxel. That is, even if 500 million voxel is secured on the memory as it is, it can be stored in 5 GBytes, and therefore it is easy to allocate the server 100 for each grid and make it on-memory.

  FIG. 12 shows a functional block diagram of the MR system 10 according to an embodiment of the present invention. The MR system 10 includes a storage unit 11, a user environment determination unit 12, a drawing unit 13, a first point group data generation unit 14, a second point group data generation unit 15, and an inconsistency detection unit 16. Is provided.

  These functions are realized by causing the server 100 to execute the program, realized by causing the HMD 200 to execute the program, or causing the server 100 to execute the program and causing the HMD 200 to execute the program. Realized. As described above, since various functions are realized by reading a program, other means may have a part of the functions of one means. As described above, the MR system 10 is realized by providing at least one of the server 100 and the HMD 200 with various functions shown in FIG.

  The storage unit 11 is provided in both the server 100 and the HMD 200 and has a function of storing programs, data, and the like in the storage unit 104 or 204. In one example, the storage unit 11 included in the server 100 stores, in the storage unit 104, data related to the position and movement of a virtual object arranged in the virtual space 32 and three-dimensional data of the virtual object. In one example, the storage unit 11 included in the HMD 200 temporarily stores a drawing command received from the server 100 in the storage unit 204 in order to draw a virtual object. In another example, the storage means 11 performs data input / output to / from various databases.

  The storage unit 11 included in the server 100 includes a three-dimensional spatial data storage unit that stores the three-dimensional spatial data in the storage unit 104. As described above, the three-dimensional spatial data includes point cloud data and voxel data. Here, the point cloud data indicates the three-dimensional shape of the real object in the virtual space 32 associated with the real space 31 acquired in advance, and the voxel data is converted from the point cloud data using a known method. It is what is done.

  Since the point cloud data is colored point cloud data, the created voxel is a colored voxel. Each voxel has three-dimensional position information. For example, the three-dimensional position information includes the three-dimensional coordinates (x, y, z) of the cube vertex closest to the origin of the three-dimensional coordinates set in the virtual space 32.

  In one example, the server 100 has a database server function, and the server 100 appropriately outputs stored three-dimensional spatial data in response to an inquiry regarding three-dimensional position information. The storage unit 11 included in the HMD 200 can also include a three-dimensional spatial data storage unit that stores the three-dimensional spatial data in the storage unit 204.

  Based on the data acquired from the sensor 206 and the three-dimensional spatial data, the user environment determining unit 12 determines a user environment including a position of the HMD 200 and a visual field area (user visual field area) that the user visually recognizes through the display unit 202. The user environment determining means 12 realizes highly accurate position tracking of the user.

  The user environment determination unit 12 preferably uses a self-position estimation technique. In one example, the user environment determination unit 12 determines the position of the HMD 200 and the user view area by executing (1) initial alignment and (2) relative movement amount calculation.

  (1) As an initial alignment, the user environment determination unit 12 collates the real space 31 and the virtual space 32 by using data acquired from the sensor 206 and three-dimensional space data acquired in advance. . As a result, the position of the HMD 200 as the initial position and the user view area are determined.

  In one example, the user environment determination unit 12 performs initial alignment as follows. The user environment determination means 12 acquires image data with a camera, acquires shape data with a depth sensor, and determines the shape of the real object which a user can visually recognize in a user view area. On the other hand, the user environment determination unit 12 determines a provisional user environment, which is a rough user environment, using a GPS sensor, a Bluetooth (registered trademark) beacon, WiFi, or the like. After the provisional user environment is determined, three-dimensional spatial data such as voxel position information and color information visible to the user in a predetermined range of positions and directions is acquired from the storage unit 104 or the storage unit 204.

  The user environment determining means 12 includes the shape of the real object in the user view area determined from the image data and the shape data, and the shape of the real object derived from the voxel position information and color information acquired based on the provisional user environment. Are compared and the position of the HMD 200 and the user view area are determined.

  (2) As the relative movement amount calculation, the user environment determination unit 12 calculates the relative movement amount in the real space 31 from the initial position determined as described above, using the data acquired from the sensor 206. By doing so, the position of the HMD 200 and the user view area are determined. In one example, the user environment determination unit 12 calculates a relative movement amount (for example, 6 DOF coordinates) of six axes using data acquired from a gyro sensor, an acceleration sensor, or the like. At this time, the user environment determination unit 12 can also use the amount of change in the image data acquired from the camera. In one example, the user environment determination unit 12 calculates the relative movement amount of the six axes using the SLAM technology.

  As described above, by configuring the user environment determining unit 12 so as to calculate the relative movement amount after the initial alignment, the user environment can be accurately determined while reducing the information processing amount of the entire system. Is possible.

  The drawing unit 13 draws a virtual object corresponding to the user environment determined by the user environment determining unit 12 on the display unit 202 of the HMD 200. The drawing unit 13 generates and draws a right-eye image (virtual object) for visual recognition by the user's right eye and a left-eye image (virtual object) for visual recognition by the user's left eye.

  In one example, the CPU included in the server 100 writes a drawing command in the main memory and transmits it to the HMD 200. The GPU provided in the HMD 200 refers to the received drawing command, writes the drawing data to a frame buffer or the like on the video memory, and draws (renders) the content read from the frame buffer as it is on the display unit 202. In creating the drawing command, the drawing unit 13 determines the position and orientation of the virtual object displayed on the display unit 202 using the position of the HMD 200 and the user view area determined by the user environment determining unit 12.

  As described above, the drawing unit 13 is realized by the server 100 and the HMD 200 sharing and executing, but the drawing unit 13 may be realized by executing all the drawing processes in the HMD 200. Alternatively, the drawing unit 13 may be realized by executing all drawing processes in the server 100, transmitting image data to the HMD 200, and displaying the image data received by the HMD 200.

  The first point cloud data generation unit 14 generates a real object and a virtual object from a composite image obtained by superimposing and displaying the virtual object drawn by the drawing unit 13 on the real space image visually recognized by the user through the display unit 202. Generate point cloud data. In the present specification, the point cloud data is referred to as first point cloud data, but the first point cloud data is obtained from an actual rendering result using a real space image on which a virtual object is superimposed and displayed. Therefore, it can also be called an actual point cloud.

  Here, since the display unit 202 and the photographing unit 203 are not usually located at the same place, the real space image photographed by the photographing unit 203 is not a real space image visually recognized by the user through the display unit 202, that is, a visual field image. . The first point cloud data generation unit 14 generates a visual field image by performing projective transformation on the real space image photographed by the photographing unit 203 by applying a projection matrix calculated from internal parameters. The internal parameters are parameters such as a camera angle of view and a field-of-view range calculated from the positional relationship between the target photographing unit 203 and the display unit 202.

  For example, in the HMD 200 shown in FIG. 7, the right-eye camera 203a is a camera for acquiring an image of a real space that is visually recognized by the user's right eye through the right-eye transmissive display 202a. The first point cloud data generation means 14 applies a projection matrix calculated from the internal parameters to the image obtained from the right-eye camera 203a to perform projective transformation, thereby allowing the user to transmit through the right-eye transmissive display 202a. A real space image (right eye visual field image) to be visually recognized is generated. The internal parameters are parameters such as the angle of view and field of view of the camera calculated from the positional relationship between the right-eye camera 203a and the right-eye transmissive display 202a. The left-eye transmissive display 202b and the left-eye camera 203b are the same as the right-eye transmissive display 202a and the right-eye camera 203a.

  A real space image (visual field image) visually recognized by the user through the display unit 202 is a stereo image captured by the imaging unit 203. The first point cloud data generation means 14 is obtained by superimposing and displaying the virtual objects drawn by the drawing means 13 on the respective visual field images respectively generated from two real space images acquired as stereo images. Each composite image is used. The first point cloud data generation unit 14 calculates the depth (depth) of each pixel by the stereo matching method using the two visual field images synthesized (for the right eye and for the left eye) as described above, And first point cloud data including the point cloud data of the virtual object is generated. In the stereo matching method, the depth of each pixel is calculated from the parallax information of the video (image) of the right-eye camera and the left-eye camera, and converted to a point group in a three-dimensional space. Color information can also be mapped when converting to a point cloud.

  The second point cloud data generation means 15 is a virtual image drawn by the drawing means 13 and the point cloud data stored by the three-dimensional spatial data storage means in the user view area and the position of the HMD 200 determined by the user environment determination means 12. Point cloud data is generated using the point cloud data of the object. In this specification, the point cloud data is referred to as second point cloud data, but the second point cloud data is virtual data drawn by the drawing means 13 with point cloud data acquired in advance using a laser scanner. Since it is ideal point cloud data in the virtual space 32 using object point cloud data, it can also be called an ideal point cloud. The point cloud data of the virtual object can be generated from a video (image) after drawing a moving virtual video (image) of a character or the like. Alternatively, the second point cloud data generation unit 15 can use the point cloud data used for rendering the virtual object.

  Preferably, the second point group data generation unit 15 adds the point group data of the virtual object drawn by the drawing unit 13 to the point group data acquired and stored in advance by the three-dimensional space data storage unit. To generate second point cloud data.

  As described above, since the data used when generating the second point cloud data is data in the virtual space 32 in which the virtual object is superimposed on the virtual space 32, in principle, the second point cloud data is There is no geometric mismatch. In this respect, the second point cloud data is different from the first point cloud data generated from the image of the real space 31 on which the virtual object is superimposed and displayed.

  13 and 14 are schematic diagrams for explaining generation of the first point cloud data and the second point cloud data according to the embodiment of the present invention. FIG. 13 and FIG. 14 show how the user in the mixed reality space 33 visually recognizes the real object wall 36 and the virtual object 37 drawn in a superimposed manner in front of the wall 36 through the display unit 202. The first point cloud data and the second point cloud data generated in this state are shown.

  In FIG. 13, the user visually recognizes the virtual object 37 that contacts the wall 36. In this case, the geometrical consistency between the wall 36 (real object) and the virtual object 37 in the real space 31 is maintained, and as shown in FIG. 13, the generated first point cloud data and the second point cloud data The point cloud data is substantially the same.

  In FIG. 14, the user visually recognizes a virtual object 37 partially embedded in the wall 36. In this case, the geometrical consistency between the wall 36 (real object) and the virtual object 37 in the real space 31 is not achieved (consistency is inconsistent), and as shown in FIG. It is understood that there is a difference between the first point cloud data and the second point cloud data.

  The mismatch detection means 16 detects mismatch based on the comparison result of the first point cloud data and the second point cloud data as described above.

  Preferably, the inconsistency detection unit 16 performs inconsistency detection before the rendering unit 13 draws a virtual object that is a target of inconsistency detection. In this case, the first point group data generation unit 14 and the second point group data generation unit 15 similarly apply the first point group before the virtual object to be subjected to inconsistency detection is drawn by the drawing unit 13. Data and second point cloud data are generated. With this configuration, inconsistency detection can be performed before a virtual object that is not geometrically consistent is drawn on the display unit 202, that is, before the user visually recognizes the virtual object. This enables the MR system 10 to send correction information to position tracking, for example, before drawing a virtual object that is not geometrically consistent. In one example, display of a virtual object that significantly impairs reality can be prevented by stopping the drawing process of the virtual object or moving the drawing position of the object. However, each of the above-described units can also be executed when a virtual object to be subjected to inconsistency detection is drawn by the drawing unit 13 or within a very short time after drawing.

  In one example, the mismatch detection means 16 detects mismatch when the difference between the first point cloud data and the second point cloud data exceeds a predetermined amount. That is, the inconsistency detection means 16 has a quantity (point data) of point data in which point data exists on one side but no point data exists on the other side at corresponding positions of the first point cloud data and the second point cloud data. Based on the group data difference amount), the presence or absence of inconsistency is determined, and inconsistency is detected. For example, the point cloud data difference amount is calculated by deleting a portion where the first point cloud data and the second point cloud data collide at the corresponding three-dimensional position. In this case, the first point cloud data generated using the image obtained from the imaging unit 203 is data that is usually sparser than the second point cloud data generated in the virtual space 32, and There is a high possibility of having point data several times to 10 times less than the point cloud data of 2. Therefore, in this case, preferably, the point cloud data difference amount is set to be several times to 10 times larger than the size of each point of the second point cloud data after the size of each point of the first point cloud data is increased. The calculation is performed by deleting the location where the first point cloud data and the second point cloud data collide at the corresponding three-dimensional position.

  In one example, the inconsistency detection unit 16 has a predetermined amount of difference between the first point cloud data and the second point cloud data in a predetermined three-dimensional space region including the virtual object 37 drawn by the drawing unit 13. An inconsistency is detected when exceeding. With such a configuration, it is possible to detect only inconsistencies around the virtual object that give a sense of incongruity when visually recognized by the user, and to eliminate detection of inconsistencies other than around the virtual object.

  In this manner, the first point cloud data obtained from the actual rendering result using the image of the real space 31 on which the virtual object is superimposed and displayed, the second point cloud data generated on the virtual space 32, Can be detected by, for example, obtaining a difference. That is, it is possible to detect a situation in which “there should be no point cloud” or “there is a point cloud that should not be”.

  It is preferable that the HMD 200 includes the first point group data generation unit 14, the second point group data generation unit 15, and the mismatch detection unit 16. With such a configuration, it is not necessary for the HMD 200 to transmit image data via the network 50 in the mismatch detection process in the MR system 10. As a result, the information processing amount of the entire system can be reduced.

  The MR system 10 including the mismatch detection system has been described so far. However, the mismatch detection system 1 according to the embodiment of the present invention is preferably a system different from the MR system 10. In this case, the mismatch detection system 1 includes a server 100 and one or a plurality of HMDs 200. The server 100 and the HMD 200 are connected to a network 50 such as the Internet and can communicate with each other. The inconsistency detection system 1 can also be realized using a server (hardware) and a network different from the MR system 10. In this case, the mismatch detection system 1 is configured to be able to communicate with the MR system 10.

  FIG. 15 shows a functional block diagram of the inconsistency detection system 1 according to the embodiment of the present invention. The inconsistency detection system 1 includes a first point group data generation unit 14, a second point group data generation unit 15, and an inconsistency detection unit 16. It is preferable that the HMD 200 includes the first point group data generation unit 14, the second point group data generation unit 15, and the mismatch detection unit 16. In this case, these functions are realized by causing the HMD 200 to execute a program, and the HMD 200 acquires information from the server 100 as necessary. However, it may be realized by causing the server 100 to execute the program, or may be realized by causing the server 100 to execute the program and causing the HMD 200 to execute the program. In addition, since various functions are realized by reading a program in this way, some of the functions of one means may be included in another means.

  As described above, FIG. 16 is a flowchart showing information processing for detecting inconsistencies in the MR system 10 according to the embodiment of the present invention. This information processing is realized by causing the server 100 to execute the program, realized by causing the HMD 200 to execute the program, or causing the server 100 to execute the program and causing the HMD 200 to execute the program. Realized. Preferably, the information processing is realized by causing the HMD 200 to execute the program, and the HMD 200 acquires data from the server 100 as necessary. It is assumed that the user wears the HMD 200 and is in the real space 31 (mixed reality environment 33). Further, since the MR system 10 needs to sequentially grasp the user environment in order to provide the mixed reality space 33 to the user, the MR system 10 determines the user environment periodically or as necessary.

  First, it is determined whether or not there is a virtual object to be drawn by the MR system 10 in the determined user environment (step 1601). If there is no virtual object to be drawn, as long as the MR system 10 continues to provide the mixed reality environment 33 to the user (step 1607), the process returns to step 1601. In this manner, when there is no virtual object to be drawn, the configuration is such that inconsistency detection is not performed, whereby the information processing amount of the entire system can be reduced and the power consumption amount can be reduced. However, this step can be omitted.

  If there is a virtual object to be drawn, it is determined whether the physical movement amount of the HMD 200 or the movement amount of the virtual object to be drawn is equal to or greater than a predetermined threshold (step 1602). If the amount of movement of the HMD 200 or the amount of movement of the virtual object to be drawn is equal to or greater than a predetermined threshold, the process proceeds to step 1603 because the geometric consistency of the drawn virtual object with respect to the real object is likely to be lost. On the other hand, if the amount of movement of the HMD 200 or the amount of movement of the virtual object to be drawn is less than the predetermined threshold, as long as the MR system 10 continues to provide the mixed reality environment 33 to the user (step 1607), the process returns to step 1601. . In this way, when the possibility of occurrence of geometric mismatch is low, it is possible to reduce the information processing amount of the entire system and reduce the power consumption by configuring so that mismatch detection is not performed. it can. However, this step can be omitted.

  Subsequently, a real space stereo image is acquired from the imaging unit 203 of the HMD 200, and a visual field image is virtually generated by applying a projection matrix to the stereo image (step 1603). In this way, the image (image) visually recognized by the user is mechanically reproduced using the image from the photographing unit 203.

  Subsequently, in step 1604, first point cloud data that is point cloud data of the real object and the virtual object is obtained from the synthesized image obtained by superimposing and displaying the virtual object drawn on the generated visual field image. Is generated. At the same time as the generation of the first point cloud data, by adding the point cloud data in the real space 31 acquired in advance using a laser scanner to the point cloud data of the virtual object to be drawn, the ideal in the virtual space 32 Second point cloud data, which is a point cloud data, is generated.

  In this information processing, preferably, the generation of the first point cloud data and the generation of the second point cloud data are performed substantially simultaneously. However, any one of the processes may be performed in series first, Processing may be performed simultaneously in parallel. The generation process of the second point cloud data may be started before the process of Step 1603 or simultaneously with the process.

  Subsequently, a difference between the first point cloud data and the second point cloud data is extracted (step 1605). Specifically, by deleting a point where the first point cloud data and the second point cloud data collide at the corresponding three-dimensional position, point data exists on one side but point data on the other side. Extract the quantity of point data that does not exist. With this difference calculation, it is possible to recognize the presence or absence of a difference between the shape felt when viewed from the user's naked eye and the ideal shape assumed by the MR system 10.

  Subsequently, a geometric mismatch is detected using the extracted difference (step 1606). As an extracted difference, a space where there is no actual point cloud corresponding to the ideal point cloud, that is, a space where there is no point cloud that should be, and a space where there is no ideal point cloud corresponding to the actual point cloud, that is, A situation where there is a point cloud that should not be detected is detected as a geometric mismatch. In one example, a mismatch is detected when the extracted difference exceeds a predetermined amount. In another example, inconsistency is detected when a difference extracted in a predetermined three-dimensional space area including a virtual object to be drawn exceeds a predetermined amount.

  Thereafter, as long as the MR system 10 continues to provide the mixed reality environment 33 to the user (step 1607), the process returns to step 1601.

  Next, operational effects of the MR system 10 and the mismatch detection system 1 according to the embodiment of the present invention will be described. First, in the present embodiment, point cloud data indicating the three-dimensional shape of a real object in the real space 31 is acquired in advance, and the MR system 10 uses the known method to call the acquired point cloud data a voxel. The data is converted into a data structure, and point cloud data and voxel data are stored in the storage unit 104 or 204 as three-dimensional spatial data. The MR system 10 associates the real space 31 with the virtual space 32 based on the three-dimensional space data acquired and stored in advance and the data acquired from the sensor 206 of the HMD 200 worn by the user. Thereby, it becomes possible to realize highly accurate position tracking of the user.

  In the present embodiment, point cloud data that can be generated from a composite image obtained by superimposing a virtual object image in the virtual space 32 on a stereo image in the real space 31 obtained from a camera included in the HMD 200, and a high-precision laser scanner are used in advance. The point cloud data obtained by adding the point cloud data of the virtual object in the virtual space 32 to the obtained point cloud data in the real space 31 is compared. In comparison, when viewed from the viewpoint of the user who is wearing the HMD 200, the geometrical relationship between the image of the real space 31 and the image of the virtual space 32 superimposed on the transmissive display 202 is displayed on the real space 31. Detects consistency. By adopting such a configuration, an inconsistent relationship between the virtual object and the real world on the image actually seen by the user is detected, and position tracking in the MR system 10 that cannot conventionally be self-detected. Error can be detected. Thereby, for example, correction information can be sent to position tracking without displaying an inconsistent image on the screen. Note that since it is impossible to realize position tracking with no error at the current technical level, it is possible to detect contradiction from an image seen by the user and correct position information or the like according to the result. Therefore, it can be said that the technical significance of the embodiment of the present invention is great.

  Further, in this embodiment, the image obtained from the camera is not used as it is, but the depth calculation is performed on the image synthesized in the memory, and the rendering result by the MR system 10 can be verified. As a result, it is possible to detect “inconsistent rendering” by the MR system 10, that is, drawing processing that is not geometrically consistent by the MR system 10 before or after drawing the virtual object. Become.

  A real-world space (predetermined real space) in which the MR system 10 according to another embodiment of the present invention provides the user with the mixed reality environment 33 is a real space 41 that is an outdoor space as shown in FIG. Also good. The real space 41 is the same as the real space 31 except that there is no ceiling and a pair of horizontal walls, and that the light source 22 including sunlight is outside the real space, not inside the real space. Also in this case, the virtual space 42 is considered to be limited to a region (0 ≦ X ≦ X1, 0 ≦ Y ≦ Y1, 0 ≦ Z ≦ Z1) associated with the real space 41.

  In another embodiment of the present invention, by using a table that associates the position information of the voxel and the position information of each pixel of the image acquired by the image acquisition apparatus 300, the color information of the voxel is obtained by the image acquisition apparatus 300. Is determined from the color information of the pixel of the image obtained by the above and updated. By using the table in this way, the color information obtained from the image acquisition device 300 can be reflected in each voxel in real time. Thereby, in the mixed reality environment in which the shape of the real space is reproduced with high accuracy, the color of the real space 31 can be recognized with high accuracy and in real time and reflected in the virtual space 32.

  In this case, the voxel color information reflected in real time can be used as the voxel color information used by the user environment determination unit 12.

  In another embodiment of the present invention, as described above, each voxel reflecting the color of the real space 31 is used as virtual indirect illumination (light source) of a virtual object to be drawn. As a result, global illumination using the light state of the real space 31 can be realized, and a virtual object having a color and a shadow very close to the environment of the real space 31 can be drawn.

  In another embodiment of the present invention, the real space 31 can be used as if it were the virtual space 32 in the real space 31. For example, the game developer can handle the real space in the same manner as the game space (virtual space) in the game engine.

  In the process or operation described above, a process or operation can be freely performed at a certain step as long as there is no inconsistency in the process or operation such as using data that should not be used at that step. Can be changed. Moreover, each Example described above is the illustration for demonstrating this invention, and this invention is not limited to these Examples. The present invention can be implemented in various forms without departing from the gist thereof.

DESCRIPTION OF SYMBOLS 1 Mismatch detection system 10 Mixed reality (MR) system 11 Memory | storage means 17 User environment acquisition means 18 Drawing means 21 Light source 22 Building 23 Shadow 24 Character (virtual object)
25, 26 units 27 walls 31, 41 Real world (real space)
32, 42 Virtual world (virtual space)
33, 43 Mixed reality world (mixed reality environment)
34 Light source 35 Building 36 Wall 37 Virtual object 50 Network 100 Server 101, 201 Processing unit 102, 202 Display unit 103 Input unit 104, 204 Storage unit 105, 205 Communication unit 110, 210 Bus 200 Display device 202a Right-eye transmissive display 202b Left-eye transmissive display 203 Imaging unit 203a Right-eye camera 203b Left-eye camera 206 Sensor 300 Image acquisition device

Claims (8)

A portable display device having a transmissive display unit for displaying a virtual object to a user and a photographing unit for photographing a real space, and rendering the virtual object on the display unit in a predetermined real space A mismatch detection system in a mixed reality system that allows a user to visually recognize a virtual object by superimposing it on real space through the display unit,
The mixed reality system is:
Three-dimensional spatial data storage means for storing three-dimensional spatial data including point group data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information;
User environment determining means for determining a user environment including a position of the display device and a visual field area visually recognized by the user through the display unit based on the data acquired from the sensor included in the display device and the three-dimensional spatial data;
Drawing means for drawing a virtual object on the display unit based on the user environment,
From the composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit generated from the captured real space image, the real object and the virtual object are displayed. First point cloud data generating means for generating first point cloud data which is point cloud data;
Second point cloud data using the point cloud data stored by the three-dimensional spatial data storage means in the field of view determined by the user environment determining means and the point cloud data of the virtual object drawn by the drawing means Second point cloud data generating means for generating
Mismatch detection means for detecting mismatch based on a comparison result between the first point cloud data and the second point cloud data;
An inconsistency detection system comprising: The imaging unit acquires real space as a stereo image,
The first point cloud data generation means includes a virtual object drawn by the drawing means on each real space image visually recognized by the user through the display unit generated from the two real space images acquired as the stereo image. The mismatch detection system according to claim 1, wherein the first point cloud data is generated from each composite image obtained by superimposing and displaying each of the images.   The first point cloud data generation means performs a projective transformation on the photographed real space image based on a positional relationship between the display unit and the imaging unit, thereby generating a real space image visually recognized by the user through the display unit. The mismatch detection system according to claim 1, wherein the mismatch detection system is generated.   4. The mismatch detection unit according to claim 1, wherein the mismatch detection unit detects mismatch when a difference between the first point cloud data and the second point cloud data exceeds a predetermined amount. 5. Inconsistency detection system.   The mismatch detection means has a difference between the first point cloud data and the second point cloud data exceeding a predetermined amount in a predetermined three-dimensional space region including a virtual object drawn based on the user environment. The mismatch detection system according to any one of claims 1 to 3, wherein a mismatch is detected in a case. A portable display device having a server, a transmissive display unit for displaying a virtual object to a user, and a photographing unit for photographing a real space, and displaying the virtual object in a predetermined real space; A mixed reality system that allows a user to visually recognize a virtual object superimposed on real space through the display unit,
Three-dimensional spatial data storage means for storing three-dimensional spatial data including point group data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information;
User environment determining means for determining a user environment including a position of the display device and a visual field area visually recognized by the user through the display unit based on the data acquired from the sensor included in the display device and the three-dimensional spatial data;
Drawing means for drawing a virtual object on the display unit based on the user environment;
From the composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit generated from the captured real space image, the real object and the virtual object are displayed. First point cloud data generating means for generating first point cloud data which is point cloud data;
Second point cloud data using the point cloud data stored by the three-dimensional spatial data storage means in the field of view determined by the user environment determining means and the point cloud data of the virtual object drawn by the drawing means Second point cloud data generating means for generating
Mismatch detection means for detecting mismatch based on a comparison result between the first point cloud data and the second point cloud data;
A mixed reality system in which the server or the display device includes the means described above. A portable display device having a transmissive display unit for displaying a virtual object to a user and a photographing unit for photographing a real space, and rendering the virtual object on the display unit in a predetermined real space A program for detecting inconsistencies in a mixed reality system that allows a user to visually recognize a virtual object through the display unit and
The mixed reality system is:
Three-dimensional spatial data storage means for storing three-dimensional spatial data including point group data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information;
User environment determining means for determining a user environment including a position of the display device and a visual field area visually recognized by the user through the display unit based on the data acquired from the sensor included in the display device and the three-dimensional spatial data;
Drawing means for drawing a virtual object on the display unit based on the user environment,
The program is stored in the display device.
From the composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit generated from the captured real space image, the real object and the virtual object are displayed. Generating first point cloud data which is point cloud data;
Second point cloud data using the point cloud data stored by the three-dimensional spatial data storage means in the field of view determined by the user environment determining means and the point cloud data of the virtual object drawn by the drawing means A step of generating
Detecting a mismatch based on a comparison result of the first point cloud data and the second point cloud data;
A program that executes A portable display device having a transmissive display unit for displaying a virtual object to a user and a photographing unit for photographing a real space, and rendering the virtual object on the display unit in a predetermined real space A mismatch detection method in a mixed reality system in which a virtual object is superimposed on a real space through the display unit and visually recognized by a user,
The mixed reality system is:
Three-dimensional spatial data storage means for storing three-dimensional spatial data including point group data of real objects in the predetermined real space acquired in advance, each of which includes point cloud data having three-dimensional position information;
User environment determining means for determining a user environment including a position of the display device and a visual field area visually recognized by the user through the display unit based on the data acquired from the sensor included in the display device and the three-dimensional spatial data;
Drawing means for drawing a virtual object on the display unit based on the user environment,
From the composite image obtained by superimposing and displaying the virtual object drawn by the drawing means on the real space image visually recognized by the user through the display unit generated from the captured real space image, the real object and the virtual object are displayed. Generating first point cloud data which is point cloud data;
Second point cloud data using the point cloud data stored by the three-dimensional spatial data storage means in the field of view determined by the user environment determining means and the point cloud data of the virtual object drawn by the drawing means A step of generating
Detecting a mismatch based on a comparison result of the first point cloud data and the second point cloud data;
Having a method.

Images