JP6134874B1 - System for creating a mixed reality environment - Google Patents

Abstract

A system capable of recognizing an environment in real space in a mixed reality environment in real time is provided. A system according to the present invention includes a portable display device having a server, a display unit for displaying a virtual object for a user, and a photographing unit for photographing a real space, and a predetermined real space. A system for drawing a virtual object by superimposing a virtual object on a real space or a captured image of a real space that a user visually recognizes through a display unit Each of the three-dimensional spatial data storage means, the table storage means, the color information determination means, the color information update means, the user environment determination means, the virtual illumination information generation means, and the drawing means is a server. Or a display apparatus is provided. [Selection] Figure 1

Description

  The present invention relates to a system, and more particularly to a system for recognizing a real space environment in real time in a mixed reality environment.

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology has been known as a technology for seamlessly combining the real world and the virtual world in real time. The present technology can make a user who experiences this experience as if a virtual object exists on the spot. Mixed reality technology is attracting attention in various fields. In order to realize the MR technology, the user can confirm the mixed reality image superimposed and displayed on the HMD by wearing an optical see-through HMD (Head Mounted Display) or a video see-through HMD.

  There are two known technologies for recognizing the real world as a 3D space: a method of mounting a highly accurate camera from the user's viewpoint and a method of installing a camera so as to surround the observation target space.

  As a method for mounting a highly accurate camera from the user's viewpoint, there is a method using an infrared projector and an infrared camera. For example, as a technique for mounting a highly accurate camera from the user's viewpoint, there is a technique for irradiating infrared rays and measuring the depth of an object from the distortion of the reflection pattern. In addition, in a method called Time of Flight method (TOF method), there is a method of calculating a reciprocal distance to an object by irradiating invisible light such as infrared rays and measuring the reflection result. These systems have a problem that the space that can be three-dimensional is limited to the infrared reachable range and cannot be used due to the solar effect.

  In addition, as a method for photographing extremely accurate three-dimensional information, there is a method using a 3D laser scanner. Although this method can achieve high measurement accuracy, it takes a time of at least about 10 minutes and a standard image quality of about 30 minutes to perform a 360 degree measurement. Therefore, it cannot be applied to real-time usage. In addition, one 3D laser scanner is very expensive at several hundred to several tens of millions of yen, and is not suitable for mass deployment in a wide area. Due to the above characteristics, the 3D laser scanner has been used for high-precision 3D conversion over a long period of time, such as surveying in civil engineering and layout confirmation in factories. For example, Patent Document 1 realizes a technology for recognizing the real world as a 3D space by adding the color of an image captured by a camera to point cloud data created by scanning with a 3D image scanner.

  As a method of installing the camera so as to surround the observation target space, a technique called Structure-from-Motion (SfM) shown in Non-Patent Document 1 is used to obtain the original 3 A method for restoring the dimension information is mentioned. A representative product that employs this method is Microsoft Photosynth (registered trademark). Although the accuracy realized by Sfm is relatively low in comparison with the accuracy required by the MR technology, it is a method that can construct a 3D model at low cost. However, the real-time property is low and cannot be applied to the realization of the MR environment as it is.

  As mentioned above, these methods cannot achieve real-time performance and high-accuracy measurement at the same time, so it is possible to capture the background of buildings, etc. with high accuracy, and to recognize changes in indirect lighting due to movement of the sun and movement of people. Difficult to do.

JP 2013-69235 A

Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2011. Building Rome in a day. Commun. ACM 54, 10 (October 2011), 105-112. DOI = http : //dx.doi.org/10.1145/2001269.2001293

  As described above, a method for recognizing a real-world structure and environment in real time with high accuracy and reflecting it in the virtual space has not yet been established. This is a problem in realization of high-quality global illumination of a virtual object to be drawn and position tracking of a user (HMD) in an MR environment, for example.

  The present invention has been made to solve such a problem, and an object thereof is to provide a system capable of recognizing a real space environment in real time in a mixed reality environment.

  In order to achieve the above object, a system according to one aspect of the present invention includes a portable display device having a server, a display unit for displaying a virtual object to a user, and a photographing unit for photographing a real space. And an image acquisition device that acquires images from a plurality of fixed points capable of capturing an area in a predetermined real space, and a virtual object for a real space or a captured image of the real space that the user visually recognizes through the display unit Are three-dimensional spatial data created based on point cloud data of real objects in the predetermined real space acquired in advance, each of which is a three-dimensional position. 3D spatial data storage means for storing 3D spatial data composed of 3D shape elements having information, and each of the images acquired by the image acquisition device. Table storage means for storing a table in which position information of each pixel is associated with three-dimensional position information of one or more of the three-dimensional shape elements indicated by each pixel, and 1 with which the three-dimensional shape element is associated Or color information determination means for determining color information of the three-dimensional shape element based on color information of a plurality of the pixels, and based on a change in color information of each pixel of the image acquired by the image acquisition device. Color information updating means for updating color information of a three-dimensional shape element, user environment determining means for determining a position of the display device and a photographing direction of the photographing unit, color information and three-dimensional position information of the three-dimensional shape element Based on the virtual illumination information generating means for generating virtual illumination information for the virtual object to be drawn, the position of the display device, and the imaging of the imaging unit. Direction, and based on the virtual illumination information, and drawing means for drawing the virtual object on the display unit, the means of the provided in the server or the display device, characterized in that.

  According to the present invention configured as described above, the position information of the three-dimensional shape elements (for example, meshes and voxels) constituting the three-dimensional space data, and the images acquired by the image acquisition device (for example, a fixed camera), respectively. By using a table in which pixel position information is associated, the color information of the three-dimensional shape element is determined from the color information of the pixel of the image acquired by the image acquisition device and updated. By using the table in this way, the color information obtained from the image acquisition device can be reflected in real time on each of the three-dimensional shape elements.

  In the present invention, it is preferable that the virtual illumination information generating means is provided on each surface of a virtual polyhedron that accommodates the rendered virtual object based on the color information and the three-dimensional position information of the three-dimensional shape element. Virtual illumination information is generated as virtual illumination information for the rendered virtual object.

  According to the present invention configured as described above, a virtual polyhedron accommodating the virtual object is assumed for the virtual object to be drawn, and the color (light) state on each surface of the polyhedron is realized. It is determined from each of the three-dimensional shape elements reflecting the color of the space. Then, the determined color state on each surface is used as virtual indirect illumination of the virtual object to be drawn. This makes it possible to draw a virtual object having colors and shadows that are closer to the real space environment.

  In the present invention, it is preferable that the display device further includes a position / orientation sensor, and the user environment determining means determines the position of the display device and the image capturing direction of the image capturing unit acquired by the position / orientation sensor. The three-dimensional shape element that can be photographed by the photographing unit at a position and direction within a predetermined range from the provisional user environment is acquired from the three-dimensional spatial data storage means, and the color of the acquired three-dimensional shape element The position of the display device and the photographing direction of the photographing unit are determined based on the information and the three-dimensional position information and the photographed real space photographed image.

  According to the present invention configured as described above, the approximate position of the display device determined using the position and orientation sensor and the shooting direction of the shooting unit are set as a temporary user environment, and the shooting unit can be shot around the temporary user environment. The position of the display device and the photographing direction of the photographing unit are determined by comparing and collating the color information and three-dimensional position information of the three-dimensional shape element in a certain area with the photographed image of the photographing unit. Here, normally, the position of the display device corresponds to the position of the user, and the shooting direction of the shooting unit corresponds to the direction in which the user faces.

  In this way, the rough position information obtained from the conventional position and orientation sensor is matched with the high-accuracy virtual space using the three-dimensional space data, and the position of the user in the real space and the virtual space and the user's orientation It is possible to correct the deviation in the direction in which it is present in real time. As a result, it is possible to realize an MR environment that links the reality with high accuracy, and it is possible to realize position tracking.

  In the present invention, it is preferable that the three-dimensional shape element is a mesh composed of polygons created based on point cloud data of real objects in the predetermined real space acquired in advance.

  According to the present invention configured as described above, a mesh can be used as a three-dimensional shape element.

  In order to achieve the above object, a display device according to an aspect of the present invention includes a display unit for displaying a virtual object to a user and a photographing unit for photographing a real space. A portable display device for rendering a virtual object superimposed on a real space or a captured image of the real space that is visually recognized by the user through the points of the real object in a predetermined real space acquired in advance 3D space data storage means for storing 3D space data created based on group data, each of which is composed of 3D shape elements each having 3D position information; Position information of each pixel of an image acquired from a plurality of fixed points capable of photographing an area in real space, and one or a plurality of the three-dimensional shapes indicated by each pixel Table storage means for storing a table associated with the three-dimensional position information of the element, and color information of the three-dimensional shape element based on color information of one or a plurality of the pixels associated with the three-dimensional shape element Color information determining means for determining, color information updating means for updating color information of the three-dimensional shape element based on a change in color information of each pixel of the acquired image, a position of the display device, and the photographing User environment determining means for determining the shooting direction of the part, virtual illumination information generating means for generating virtual illumination information for the rendered virtual object based on the color information and three-dimensional position information of the three-dimensional shape element, Drawing means for drawing the virtual object on the display unit based on the position of the display device, the shooting direction of the shooting unit, and the virtual illumination information It comprises, characterized in that.

  According to the present invention configured as described above, the technology realized by the above system can be realized by a display device.

  In order to achieve the above object, a system according to one aspect of the present invention includes a server, a display unit for displaying a three-dimensional virtual object for a user, a photographing unit for photographing a real space, and A portable display device having a position and orientation sensor, and a system for rendering a virtual object superimposed on a real space or a captured image of the real space visually recognized by the user through the display unit, 3D space data created based on the acquired point cloud data of a real object in a predetermined real space, each of which is composed of 3D shape elements having color information and 3D position information Temporary user sets the position of the display device and the shooting direction of the shooting unit acquired by the three-dimensional space data storage means for storing the dimension space data and the position and orientation sensor. The three-dimensional shape element that can be photographed by the photographing unit at a position and direction within a predetermined range from the provisional user environment is acquired from the three-dimensional spatial data storage means, and the color of the acquired three-dimensional shape element Based on the information, the three-dimensional position information, and the photographed real space photographed image, the user environment determining means for determining the position of the display device and the photographing direction of the photographing unit is the server or the The display device is provided.

  According to the present invention configured as described above, the approximate position of the display device determined using the position and orientation sensor and the shooting direction of the shooting unit are set as a temporary user environment, and the shooting unit can be shot around the temporary user environment. The position of the display device and the photographing direction of the photographing unit are determined by comparing and collating the color information and three-dimensional position information of the three-dimensional shape element in a certain area with the photographed image of the photographing unit.

  In this way, the rough position information obtained from the conventional position and orientation sensor is matched with the high-accuracy virtual space using the three-dimensional space data, and the position of the user in the real space and the virtual space and the user's orientation It is possible to correct the deviation in the direction in which it is present in real time. As a result, it is possible to realize an MR environment that links the reality with high accuracy, and it is possible to realize position tracking.

  In order to achieve the above object, a method according to one aspect of the present invention is applied to a real space or a captured image of a real space that is visually recognized by a user through a display unit of a portable display device in a predetermined real space. A method of superimposing and drawing a virtual object, the step of acquiring an image from each of a plurality of fixed points capable of photographing an area in the predetermined real space, and in the predetermined real space acquired in advance Color information of a 3D shape element having 3D position information constituting 3D space data created based on point cloud data of a real object, corresponding to the 3D shape element of the acquired image Determining the color information of the three-dimensional shape element based on the color information of the pixel at the position, and the third order based on the change in the color information of each pixel of the acquired image. Based on the step of updating the color information of the shape element, the step of determining the position of the display device and the photographing direction of the photographing unit provided in the display device, the color information of the three-dimensional shape element and the three-dimensional position information, Generating virtual illumination information for the virtual object to be drawn, drawing the virtual object on the display unit based on the position of the display device, the shooting direction of the shooting unit, and the virtual lighting information; It is characterized by having.

  In order to achieve the above object, a program according to one aspect of the present invention is portable and includes a server, a display unit for displaying a virtual object for a user, and a photographing unit for photographing a real space. In a system including a display device and an image acquisition device that acquires images from a plurality of fixed points that can capture an area in a predetermined real space, the real space or a captured image of the real space that is visually recognized by the user through the display unit A program for superimposing and drawing a virtual object on the server, the program acquiring an image from the display device on the server, and a real object in the predetermined real space acquired in advance Color information of a 3D shape element having 3D position information constituting 3D space data created based on the point cloud data of Determining color information of the three-dimensional shape element based on color information of a pixel at a position corresponding to the three-dimensional shape element of the image acquired by the image acquisition device; and Updating the color information of the three-dimensional shape element based on changes in the color information of each pixel; determining the position of the display device and the photographing direction of the photographing unit; and the color of the three-dimensional shape element Generating virtual illumination information for the rendered virtual object based on the information and the three-dimensional position information; and displaying the display based on the position of the display device, the photographing direction of the photographing unit, and the virtual illumination information. And a step of drawing the virtual object in a section.

  In order to achieve the above object, a method according to one aspect of the present invention is applied to a real space or a captured image of a real space that is visually recognized by a user through a display unit of a portable display device in a predetermined real space. And creating a mixed reality environment for rendering a three-dimensional virtual object superimposed on the basis of the point cloud data of the real object in the predetermined real space. Each of a step of creating three-dimensional space data composed of three-dimensional shape elements having, a step of acquiring images from a plurality of fixed points capable of photographing a region in the predetermined real space, and the acquired images Creating a table associating the position information of each pixel with the three-dimensional position information of one or more of the three-dimensional shape elements indicated by each pixel; And a step of determining the color information of the three-dimensional shape element based on color information of one or more of the pixels serial three-dimensional shape element is associated, characterized in that.

  According to the present invention configured as described above, three-dimensional space data representing a predetermined real space including a real object is created in a virtual space, an image is acquired from a fixed camera or the like, and the three-dimensional space data is configured. A table in which the positional information of the three-dimensional shape element to be associated with the positional information of each pixel of the image acquired by a fixed camera or the like is created, and the color information of the three-dimensional shape element is determined using the table.

  As a result, the color state (light state) of the real space can be reflected in the color information of the three-dimensional shape element in real time and with high accuracy. As a result, the color state of the real space can be reflected in real time and high. It is possible to realize a mixed reality environment that reflects the accuracy.

  In the present invention, it is preferable that the method further includes a step of updating the color information of the three-dimensional shape element based on a change in color information of each pixel of the acquired image.

  In order to achieve the above object, a system according to an aspect of the present invention is provided for a real space or a captured image of a real space that a user visually recognizes through a display unit of a portable display device in a predetermined real space. A system for creating a mixed reality environment for superimposing and rendering a three-dimensional virtual object, each of which is based on the point cloud data of the real object in the predetermined real space. 3D space data creation means for creating 3D space data composed of 3D shape elements having information, and image acquisition means for obtaining images from a plurality of fixed points capable of photographing the region in the predetermined real space. And the position information of each pixel of the acquired image and the three-dimensional position information of one or more of the three-dimensional shape elements indicated by each pixel Table creating means for creating a table, and color information determining means for determining color information of the three-dimensional shape element based on color information of one or a plurality of the pixels associated with the three-dimensional shape element. It is characterized by that.

  In order to achieve the above object, a data structure as one aspect of the present invention is a real space or a captured image of a real space that is visually recognized by a user through a display unit of a portable display device in a predetermined real space. A data structure for creating a mixed reality environment for superimposing and drawing a three-dimensional virtual object, and acquiring images from a plurality of fixed points capable of photographing an area in the predetermined real space. 3D constituting 3D space data created based on 2D coordinate data indicating the position information of each pixel of the acquired image and the point cloud data of the real object in the predetermined real space And a three-dimensional coordinate data indicating three-dimensional position information of the three-dimensional shape element associated with each of the pixels.

  According to the present invention configured as described above, the data structure includes the position information of the three-dimensional shape element constituting the three-dimensional space data representing the predetermined real space including the real object in the virtual space, and the image acquisition device ( For example, it includes a table that associates position information of each pixel of an image acquired by a fixed camera. By using such a table, it is possible to determine and update the color information of the three-dimensional shape element from the color information of the pixels of the image acquired by a fixed camera or the like.

  According to the present invention, the real space environment can be recognized in real time in the mixed reality environment.

1 is an overall configuration diagram of a system according to an embodiment of the present invention. It is a figure which shows the outline | summary of the global illumination using the light state of the real world by one Embodiment of this invention. It is a figure which shows the outline | summary of the position tracking by one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the server by one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the display apparatus by one Embodiment of this invention. 1 is an overview of real space according to an embodiment of the present invention. It is the top view which looked at real space of Drawing 6a from the top. It is the three-dimensional space data expressed by the point cloud data acquired in the real space of FIG. 6a. It is the three-dimensional space data expressed by the voxel created from the point cloud data of FIG. 6c. It is a functional block diagram of the system by one Embodiment of this invention. It is a figure explaining the correlation with the positional information of each pixel of the image acquired by the fixed camera, and the three-dimensional positional information of the voxel which each pixel shows by one Embodiment of this invention. It is a figure explaining the direction which the user is facing, and the imaging direction of the imaging | photography part 203 by one Embodiment of this invention. FIG. 3 is a diagram illustrating a method for creating a mixed reality environment that reflects an environment in real space in a virtual space in real time according to an embodiment of the present invention; FIG. 11 is a diagram illustrating a method for providing global illumination using a light state in real space and high-accuracy position tracking of a user, in which the user can experience mixed reality in the mixed reality environment created in FIG. 10. is there. 1 is an overview of real space according to an embodiment of the present invention.

  Hereinafter, a system for providing a mixed reality space in which a virtual space and a real space are fused to a user according to an embodiment of the present invention will be described with reference to the drawings.

  One of the technical features of the system according to the embodiment of the present invention is to generate a high-precision three-dimensional virtual space having a structure and color that match the real space with extremely high accuracy. As a result, (1) it is possible to realize global illumination using light conditions in the real world, and (2) it is possible to realize highly accurate position tracking of the user.

  (1) An outline of global illumination using light conditions in the real world will be described. The system according to the embodiment of the present invention assumes an operation mode of an MR environment where computer graphics (CG) occupies only a small part on the screen, such as a game character. For example, in FIG. 2, when a character 24 that is a virtual object (that is, CG) that is the existence of a virtual space enters a shadow 23 generated from a building 22 in the real space by the light source 21 in the real space, the darkness of the place The brightness of the character 24 image is changed according to the (brightness). The system according to the embodiment of the present invention calculates the lightness of the character image from the lightness of the color information of the three-dimensional space data (for example, 3D mesh and voxel) of the building and the floor, and renders the light in the real world. The character display that matches the state of is realized.

  (2) An outline of highly accurate position tracking of the user will be described. FIG. 3 shows a real world 31 in which a user is present, a virtual world 32 constructed by three-dimensional space data (DB), and a mixed real world (MR environment) 33 generated by matching them. In the MR environment 33, a user generally wears a device having a display unit such as an HMD. Various sensor devices are installed in the HMD. In order to realize the MR environment 33 that recognizes the structure of the real world 31 with high accuracy and reflects it in the virtual space 32, measurement of sensors conventionally used is performed. Accuracy is not enough.

  In view of this, the system according to the embodiment of the present invention provides rough position information obtained from various sensors conventionally used (for example, a distance sensor, an image sensor, a direction sensor, a GPS sensor, a Bluetooth (registered trademark) beacon) and high-precision Matching with the virtual space 32 is performed, and the deviation of the position of the user and the direction in which the user is facing in the real space 31 and the virtual space 32 is corrected in real time. As a result, an MR environment 33 that links the reality with high accuracy is realized, and position tracking is realized. In such an MR environment 33, when the character 24 stands on the virtual object platform 26 in the virtual world 32, the user visually recognizes that the character 24 stands on the platform 25 in the real world 31 without a sense of incongruity. can do.

  The system according to the embodiment of the present invention as described above is realized in a predetermined real space (predetermined real space). A system according to an embodiment of the present invention recognizes the state of a real object such as a building or a bench in a predetermined real space, generates a high-precision three-dimensional space, and allows a user in the predetermined real space to In order to experience mixed reality, it is configured to include a recognition phase in which the state of light in real space is recognized and mapped to a high-precision three-dimensional space. A specific configuration will be described below.

[System configuration]
FIG. 1 is an example of an overall configuration diagram of a system 10 according to an embodiment of the present invention. As shown in FIG. 1, the system 10 includes a server 100, one or more display devices 200, and one or more image acquisition devices 300. The server 100, the display device 200, and the image acquisition device 300 are connected to a network 50 such as the Internet and can communicate with each other.

  The system 10 is assumed to be a server-client system. Preferably, the display device 200 and the image acquisition device 300 communicate only with the server 100. However, it can be configured by a system without a server such as PtoP.

  The system 10 can allow a user in a predetermined real space to experience mixed reality. The predetermined real space is a predetermined indoor or outdoor real space, and a real object that is an object in the real world exists in the space. The real object is a structure such as a building, a bench, or a wall, and is fixed in the real space. However, a movable object may be included in the real object.

  FIG. 4 is a block diagram showing a hardware configuration of the server 100 according to the embodiment of the present invention. The server 100 includes a processing unit 101, a display unit 102, an input unit 103, a storage unit 104, and a communication unit 105. Each of these components is connected by the bus 110, but may be individually connected as necessary.

  The processing unit 101 includes a processor (for example, a CPU) that controls each unit included in the server 100, and performs various processes using the storage unit 104 as a work area. When the server 100 draws a virtual object, the processing unit 101 preferably includes a GPU that performs drawing processing separately from the CPU. The display unit 102 has a function of displaying information to the server user. The input unit 103 has a function of receiving input from the server user, such as a keyboard and a mouse.

  The storage unit 104 includes a hard disk, a main memory, and a buffer memory. A program is stored in the hard disk. However, the hard disk may be any non-volatile storage or non-volatile memory as long as it can store information, and may be removable. The storage unit 104 stores a program and various data that can be referred to when the program is executed. When the processing unit 101 includes a GPU, the storage unit 104 can include a video memory.

  In one example, the program is one or a plurality of programs for displaying / moving a virtual object through the display device 200 according to the position / orientation / action of the user, and various data are virtual such as a 3D character to be drawn. Contains object data.

  The storage unit 104 can store data (for example, tables) and programs for various databases. Various databases are realized by the operation of the processing unit 101 and the like. For example, the server 100 may have a database server function, may include a database server, or may include or include other servers.

  The communication unit 105 performs wired communication using an Ethernet (registered trademark) cable, mobile communication, wireless communication such as a wireless LAN, and connects to the network 50.

  The server 100 realizes various functions by executing a program, but some of these functions can also be realized by configuring an electronic circuit or the like.

  In one example, when the server 100 draws, the CPU writes a drawing command to the main memory, and the GPU refers to the drawing command and writes the drawing data to the frame buffer on the video memory. Thereafter, the data read from the frame buffer is transmitted to the display device 200 as it is.

  In one example, the server 100 is configured by combining a plurality of servers installed for each function or area. For example, a predetermined real space may be divided into a plurality of areas, and one server may be installed in each area, and a server that integrates these servers may be installed.

  FIG. 5 is a block diagram showing a hardware configuration of the display device 200 according to the embodiment of the present invention. The display device 200 includes a processing unit 201, a display unit 202, a photographing unit 203, a storage unit 204, a communication unit 205, and a position / orientation sensor 206. Each of these components is connected by the bus 210, but each may be individually connected as necessary.

  The processing unit 201 includes a processor (for example, a CPU) that controls each unit included in the display device 200, and performs various processes using the storage unit 204 as a work area. When the display device 200 performs rendering, the processing unit 201 preferably includes a GPU that performs rendering processing separately from the CPU. In one example, the display device 200 receives a drawing command from the server 100 and performs a drawing process.

  The storage unit 204 includes a hard disk, a main memory, and a buffer memory. A program is stored in the hard disk. However, the hard disk may be any non-volatile storage or non-volatile memory as long as it can store information, and may be removable. The storage unit 204 stores a program and various types of data that can be referred to when the program is executed. Furthermore, the storage unit 204 can store data and programs for various databases. Various databases are realized by the operation of the processing unit 201 and the like. When the processing unit 201 includes a GPU, the storage unit 204 can include a video memory.

  The display unit 202 is a transmissive display, a transflective display, or a non-transmissive display that displays a virtual object to the user.

  The imaging unit 203 captures a real space, and stores the captured image of each frame (real world image) in the storage unit 204, for example. The area captured by the imaging unit 203 is preferably the same as the field of view of the user carrying or wearing the display device 200.

  The display device 200 is portable by the user, and is preferably a head-mounted image display device (HMD) that can be worn on the head. Hereinafter, in the present embodiment, the HMD 200 is used as the display device 200. The HMD 200 includes an optical see-through HMD and a video see-through HMD.

  The optical see-through type HMD includes a display unit 202 of a transmissive display, allows a user wearing the HMD to visually recognize the real space through the display unit 202, and draws a virtual object superimposed on the display unit 202. Provide mixed reality images.

  The video see-through type HMD includes a display unit 202 of a non-transparent display, allows a user wearing the HMD to visually recognize a captured image of real space through the display unit 202, and superimposes a virtual object on the display unit 202 for drawing. To provide a mixed reality image.

  However, the display device 200 may be a display device (Hand Held Display) of a type that is held by a user's hand.

  The position and orientation sensor 206 includes a GPS, a gyro sensor, and an accelerometer, and determines a rough position of the display device and a direction in which the photographing unit 203 of the display device photographs (or a direction in which the display unit 202 faces) in real time. Various other sensors conventionally used may be included.

  The communication unit 205 performs wireless communication such as mobile communication and wireless LAN, and connects to the network 50. In one example, the communication unit 205 transmits image data captured by the imaging unit 203 to the server 100 via the network 50.

  The image acquisition device 300 acquires a real space video (image) and transmits the acquired image data to the server 100 and the display device 200 via the network 50. In addition, the image acquisition device 300 is installed so as to surround a predetermined real space and fix a region visible to the user in the predetermined real space to a fixed point that can be photographed.

  In one example, the image acquisition device 300 is a fixed camera installed at a fixed point. In one example, the image acquisition device 300 acquires 30 frames of images per second and transmits them to the server 100. Hereinafter, in this embodiment, the fixed camera 300 is used as the image acquisition device 300.

[First Embodiment]
As a real-world space (predetermined real space) in which the system 10 according to the embodiment of the present invention provides the mixed reality environment 43 to the user, a real space 41 that is an indoor space covered with a wall 27 as shown in FIG. Is assumed. 6A is an overview of the real space 41, and FIG. 6B is a plan view of the real space 41 as viewed from above. In the real space 41, there are a light source 21 in the real world and a building 22 that is a real object in the real world. As shown in FIG. 6, in the real space 41, a plurality of fixed cameras 300 are arranged so that a space to be observed, that is, a space visible to the user in the real space 41 can be photographed.

  In the embodiment of the present invention, point cloud data indicating the three-dimensional shape of a real object in the real space 41 is acquired in advance by using, for example, a high-precision laser scanner (not shown). The acquisition of the point cloud data is preferably performed after the fixed camera 300 is installed in the real space 41, and the image data of the fixed camera 300 is preferably acquired simultaneously with the acquisition of the point cloud data. However, the fixed camera 300 may be installed after the point cloud data is acquired using a laser scanner. FIG. 6 c shows an example of three-dimensional space data represented by point cloud data acquired in the real space 41.

  As shown in FIG. 6 c, each of the point group data has three-dimensional coordinates (x, y, z) and is arranged in a virtual space 42 associated with the real space 41. Each of the point cloud data is colored point cloud data having color information. Colored point cloud data is obtained by obtaining color information obtained from an image taken separately from the point cloud data acquisition using a camera included in the laser scanner, according to each coordinate point ( x, y, z).

  In this way, three-dimensional space data is constructed by using point cloud data as a basic unit (basic component) as a representation of the three-dimensional shape of a real object in the virtual space 42 associated with the real space 41. Can do. In this specification, this basic unit is expressed as a three-dimensional shape element.

  In the embodiment of the present invention, for ease of explanation, the acquired point cloud data is converted into, for example, OctoMap (“OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees” in Autonomous Robots, 2013; A. Hornung,. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard (http://dx.doi.org/10.1007/s10514-012-9321-0) DOI: 10.1007 / s10514-012-9321-0.) Using a known method, it is converted into a data structure called a voxel. A voxel is a unit component in a three-dimensional space corresponding to a pixel in the two-dimensional space, and is a cube having a certain size that is identified using coordinates in the three-dimensional space.

  In the first embodiment, hereinafter, it is assumed that the three-dimensional space data representing the three-dimensional shape of the real object in the virtual space 42 associated with the real space 41 is composed of voxels. In this case, the voxel is a three-dimensional shape element of the three-dimensional space data. FIG. 6d shows three-dimensional spatial data represented by voxels created from the point cloud data of FIG. 6c. However, as described later, a mesh (3D mesh) can be used as the three-dimensional shape element, or the point cloud data itself can be used as the three-dimensional shape element.

色情報ｃは、ＲＧＢやＨＳＶ等のフォーマットで表現される。例えばＨＳＶのフォーマットで表現される場合、色情報は、色相、彩度、及び明度を有する。 In the embodiment of the present invention, for example, a real space is divided into 1 cm ³ voxel frames, and a voxel space (a three-dimensional space expressed by voxels) is set. One voxel V has one or more color information c when viewed from the fixed camera 300 in addition to the position information x, y, z.

The color information c is expressed in a format such as RGB or HSV. For example, when expressed in the HSV format, the color information includes hue, saturation, and brightness.

  In the embodiment of the present invention, the virtual space 42 is considered to be limited to a region (0 ≦ X ≦ X1, 0 ≦ Y ≦ Y1, 0 ≦ Z ≦ Z1) associated with the real space 41. . However, the virtual space 42 can be set to a smaller area, and a plurality of virtual spaces can be associated with the real space 41, respectively, or the virtual space 42 can be set to a larger area. When the virtual space 42 is set to a smaller area, the fixed camera 300 is installed at each of a plurality of fixed points that can shoot an area visible to the user in the real space 41 corresponding to each virtual space 42.

In one example, when the real space 41 is a wide area, it is preferable that the virtual space 42 is set for each of a plurality of areas, and the server 100 is also installed for each of the plurality of areas. For example, when the storage unit 104 of the server 100 stores voxel data, one voxel V in Expression (1) becomes 10 bytes if 1 voxel = (int16 x, int16 y, int16 z, int32 rgb). . When the real space is modeled as a set of 10 mm ³ voxels, 1000 mm ³ = 1 million voxel, which is about 10 Mbytes. When converting a broad space as a theme park with high accuracy voxel space, the theme park area, when divided into a grid of 10 m = 10,000 mm, 1 grid, so 10,000 m ^3, a 1 billion voxel . If this is reduced to a space up to 5m high, it will be about 500 million voxel. That is, even if 500 million voxel is secured in memory as it is, it can be stored in 5 GByte, so it is easy to allocate a server for each grid and make it on-memory.

  FIG. 7 shows a functional block diagram of a system according to an embodiment of the present invention. The system 10 includes a storage unit 11, a three-dimensional space data generation unit 12, a table generation unit 13, a color information determination unit 14, a color information update unit 15, a virtual illumination information generation unit 16, and a user environment acquisition unit. 17 and the drawing means 18.

  These functions are realized by causing the server 100 to execute the program, realized by causing the HMD 200 to execute the program, or causing the server 100 to execute the program and causing the HMD 200 to execute the program. Realized. As described above, since various functions are realized by reading a program, other means may have a part of the functions of one means. As described above, this system is realized by providing at least one of the server 100 and the HMD 200 with the various functions illustrated in FIG. 7.

  The storage unit 11 is provided in both the server 100 and the HMD 200 and has a function of storing programs, data, and the like in the storage unit 104 or 204. In one example, the storage unit 11 included in the server 100 stores, in the storage unit 104, data related to the position and movement of a virtual object placed in the virtual space 42, and three-dimensional data of the virtual object, and the storage unit 11 included in the HMD 200. Temporarily stores the drawing command received from the server 100 in the storage unit 204 and performs a drawing process. In another example, the storage means 11 performs data input / output to / from various databases.

  The three-dimensional space data creation unit 12 converts the point cloud data indicating the three-dimensional shape of the real object into voxels in the virtual space 42 associated with the real space 41, thereby obtaining three-dimensional space data (three-dimensional voxel data). Create

  Here, since the point cloud data is colored point cloud data, the created voxel is a colored voxel. Each voxel has three-dimensional position information. For example, the three-dimensional position information includes the three-dimensional coordinates (x, y, z) of the cube vertex closest to the origin of the three-dimensional coordinates set in the virtual space.

  It is preferable that the server 100 is provided with the three-dimensional spatial data creation means 12. However, when the three-dimensional spatial data is created in advance and stored in the storage unit 104 or the like, the system 10 may not include the three-dimensional spatial data creation unit 12.

  The storage unit 11 included in the server 100 includes a three-dimensional spatial data storage unit that stores the created three-dimensional spatial data in the storage unit 104. In one example, the server 100 has a database server function, and the server 100 appropriately outputs stored three-dimensional spatial data in response to an inquiry regarding three-dimensional position information. However, the storage unit 11 included in the HMD 200 can also include a three-dimensional spatial data storage unit that stores the three-dimensional spatial data in the storage unit 204.

  The table creation means 13 creates a table by associating the position information of each pixel of the image acquired by the fixed camera 300 with the three-dimensional position information of one or a plurality of voxels indicated by each pixel. This will be described with reference to FIG.

  As shown in FIG. 8, any one pixel (x, y) of an image acquired by one fixed camera 300 a of the fixed cameras 300 projects a minute area of the real object in the real space. The projected micro area corresponds to a plurality of voxels (x, y, z) in the voxel space. However, any one pixel may correspond to one voxel.

  Here, as described above, each voxel has color information. The fixed camera 300 is fixed to a predetermined point, and the three-dimensional position information of the fixed camera 300 in the virtual space 42 is stored as three-dimensional space data. For example, as shown in FIG. 6c, by acquiring in advance point cloud data indicating the three-dimensional shape of a real object called the fixed camera 300, the three-dimensional coordinates of the fixed camera 300 are stored as three-dimensional space data.

  Therefore, by comparing the three-dimensional position information and color information of the voxel (point cloud data) with the image data captured by the fixed camera 300, the position information (two-dimensional coordinates) of each pixel of the image data, The position information (three-dimensional coordinates) of the voxel corresponding to the minute area of the real object indicated by the pixel can be associated. In this way, the table creation unit 13 creates a table by associating the position information of the pixels of the image captured by the fixed camera 300 with the three-dimensional position information of the voxel indicated by each pixel. In one example, the association is performed using similarity between the three-dimensional position information and color information of the voxel in the three-dimensional space data and the position information and color information of the image data captured by the fixed camera 300.

  In addition, by performing such association, the three-dimensional position information of the fixed camera 300 and the shooting direction (for example, 6 DOF coordinates) can be determined with high accuracy.

  The table creation means 13 is preferably provided in the server 100. However, when table data is created in advance and stored in the storage unit 104 or the like, the system 10 may not include the table creation unit 13.

  The storage unit 11 provided in the server 100 further includes a table storage unit that stores the created table in the storage unit 104. The stored table can be used, for example, as a table referenced by the database. In another example, the storage unit 11 included in the HMD 200 further includes a table storage unit that stores the created table in the storage unit 204.

  The color information determination unit 14 determines the color information of each voxel indicating the three-dimensional shape of the real object in the real space 41 by using the created table. Here, the color information of the colored voxel created from the colored point cloud data is used only in the above table creation.

  When a certain voxel (position information) is associated with only one pixel (position information) of an image captured by one fixed camera 300, the color information determination unit 14 displays the color information of the pixel. Determined as voxel color information.

  When a certain voxel is associated with each pixel of each image captured by a plurality of fixed cameras 300, the color information determination unit 14 displays a fixed camera that reflects the minute area of the real object corresponding to the voxel most greatly. The color information of the pixel associated with the voxel of the image acquired from 300 is determined as the color information of the voxel. However, in this case, the color information of any one of the pixels associated with the voxel in the table created by the table creating unit 13 may be determined as the color information of the voxel.

  As described above, the system 10 according to an embodiment of the present invention uses voxel color information as indirect illumination for a virtual object to be drawn in global illumination. The color information determination unit 14 is preferably provided in the server 100, and the color information of the voxel determined by the color information determination unit 14 is stored in the storage unit 104 in association with each voxel.

  In one example, the voxel color information is stored together with the three-dimensional position information as one element of the three-dimensional spatial data as shown in Expression (1). In another example, voxel color information is stored as table data in a database.

  In another example, when a certain voxel is associated with each pixel of each image captured by a plurality of fixed cameras 300, the color information obtained by averaging the color information of each pixel is used as the color information of the voxel. May be determined as Or you may determine the color information weighted with respect to the color information of each pixel as the color information of this voxel.

  The color information update means 15 is a voxel color information determined by the color information determination means 14 using the color information of the pixel when a change in the color information of the pixel of the image acquired by the fixed camera 300 satisfies a predetermined condition. Is updated (newly determined) using the latest pixel color information.

  In one example, when the change in the color information of the pixel satisfies the predetermined condition, the color information of the latest pixel is within a certain distance range (distance in the HSV color space or the like) from the color information of the original pixel. is there. In another example, when a change in color information of a pixel satisfies a predetermined condition, an average of the color information of images in the past 30 frames for each pixel is a certain distance range from the color information of the original pixel. This is the case.

  In this way, if the color information changes so as to satisfy a certain condition, it can be determined that the change is due to the light state, and the color information of the voxel is reflected in the voxel by reflecting the latest pixel color information at the time of the determination. Can be updated. On the other hand, for example, when a person who passes the camera appears in an image acquired by the fixed camera 300, the color information update unit 15 determines that the color in the pixel in which the person is reflected is not a color change of a certain distance, and the color of the voxel. Do not update information.

  The color information update unit 15 is preferably provided in the same device as the color information determination unit 14, and is preferably provided in the server 100.

  The virtual illumination information generating means 16 generates virtual illumination information for a virtual object to be drawn based on the voxel color information and its three-dimensional position information in order to draw a shadow in consideration of indirect illumination. Since it is based on the voxel color information, when the voxel color information is updated, the virtual illumination information is also updated. The virtual illumination information generation unit 16 is preferably provided in the same device as the color information update unit 15, and is preferably provided in the server 100.

  In one example, based on the voxel color information and its three-dimensional position information, the virtual illumination information on each surface of the virtual polyhedron containing the rendered virtual object is used as virtual illumination information for the rendered virtual object. Generate. Here, each surface of the virtual polyhedron is used as an indirect illumination (light source) of each surface as in the image-based lighting (IBL) method. As described above, according to the embodiment of the present invention, the texture obtained from the real space is mapped in real time to the real object serving as the light source of the ambient light (indirect illumination), so that the IBL to the moving virtual object is also possible. It becomes. Note that when a virtual object is drawn by conventional IBL, it is necessary to install a camera for acquiring ambient light in all directions from the drawing position. Therefore, a virtual object such as a game character can be moved freely. The application of IBL was difficult.

  The user environment determination unit 17 determines the position of the HMD 200 attached to the user and the direction in which the image capturing unit 203 captures images. The server environment determining unit 17 is preferably provided in the server 100.

  Specifically, the user environment determination unit 17 acquires the rough position of the HMD 200 and the shooting direction of the shooting unit 203 by the position / orientation sensor 206, and uses this as the provisional user environment. Subsequently, the user environment determination unit 17 acquires from the storage unit 104 or the storage unit 204 the voxel color information and three-dimensional position information that can be visually recognized by the user at a position and direction within a predetermined range from the provisional user environment. Subsequently, the user environment determination unit 17 compares the captured image of the real space 41 captured by the capturing unit 203 with the acquired position information and color information of the voxel, thereby comparing the position of the HMD 200 and the capturing unit. A shooting direction 203 is determined.

  In one example, the position of the HMD 200 and the photographing direction of the photographing unit 203 are determined by matching the color information pattern of the photographed image with the acquired color information pattern of the voxel. In the embodiment of the present invention, the position of the HMD 200 corresponds to the position of the user, and the shooting direction of the shooting unit 203 corresponds to the direction in which the user is facing.

  FIG. 9 is a diagram illustrating the direction in which the user is facing and the shooting direction of the shooting unit 203. When the user is at the position shown in FIG. 9, the photographed image of the real space 41 photographed by the photographing unit 203 is as shown in the lower diagram of FIG. 9. The color information pattern of the photographed image, the position of the HMD 200 acquired by the position / orientation sensor 206, the direction in which the photographing unit 203 shoots at a position within a certain range from the position, and the voxel in the direction within the certain range from the direction. The color information pattern is compared and compared. Thereby, the position of the HMD 200 and the shooting direction of the shooting unit 203 can be determined.

  The drawing unit 18 draws the virtual object on the display unit 202 of the HMD 200. In one example, the CPU included in the server 100 writes a drawing command in the main memory and transmits it to the HMD 200. The GPU included in the HMD 200 refers to the received drawing command, writes drawing data to a frame buffer or the like on the video memory, and draws the content read from the frame buffer on the display unit 202 as it is.

  In creating the drawing command, the drawing unit 18 uses the position of the HMD 200 determined by the user environment determining unit 17 and the shooting direction of the shooting unit 203 to determine the position and orientation of the virtual object displayed on the display unit 202. decide. In creating the drawing command, the drawing unit 18 adds a shadow to the virtual object using the virtual illumination information.

  As described above, it is preferable that the drawing unit 18 is executed by the server 100 and the HMD 200 being shared. However, the server 100 may include the drawing unit 18, and after executing all drawing processes in the server 100, the image data may be transmitted to the HMD 200 and the image data received by the HMD 200 may be displayed. Alternatively, the HMD 200 may include the drawing unit 18 and the HMD 200 may be configured to execute all drawing processes.

  Next, the generation phase will be described. FIG. 10 shows a method of creating a mixed reality environment 43 that reflects the environment of the real space 41 in the virtual space 42 in real time according to an embodiment of the present invention. In the real space 41, as shown in FIG. 6a, fixed cameras 300 are installed in advance at a plurality of fixed points that can shoot an area in the real space 41 that can be visually recognized by the user.

  First, by using a laser scanner (not shown), colored point cloud data indicating the three-dimensional shape of a real object in the real space 41 is acquired in advance, and the colored point cloud data is converted into voxels to obtain a three-dimensional data. Spatial data (three-dimensional voxel data) is created (step 1001).

  Subsequently, an image is acquired from the fixed camera 300 installed in advance (step 1002). However, since the fixed camera 300 can acquire an image after installation, it may be performed before step 1001.

  Subsequently, by comparing the three-dimensional position information and color information of the voxel with the image data captured by the fixed camera 300, the position information (two-dimensional coordinates) of each pixel of the image data and the pixel are A table in which position information (three-dimensional coordinates) of one or a plurality of voxels corresponding to the minute area of the real object to be shown is associated is created (step 1003).

  Using the table created as described above, the color information of each voxel indicating the three-dimensional shape of the real object in the real space 41 is determined (step 1004). In this step, one voxel associated with only one pixel of an image photographed by one fixed camera 300 determines the color information of the pixel as the color information of the voxel. In this step, one voxel associated with each pixel of each image captured by the plurality of fixed cameras 300 is acquired from the fixed camera 300 that displays the largest minute area of the real object corresponding to the voxel. The color information of the pixel associated with the voxel in the image is determined as the color information of the voxel.

  However, one voxel associated with each pixel of each image captured by the plurality of fixed cameras 300 may determine color information obtained by averaging the color information of each pixel as the color information of the voxel.

  Subsequently, in each pixel of the image acquired by the plurality of fixed cameras 300, it is determined whether or not the color information change satisfies a predetermined condition (step 1005). For example, the predetermined condition is whether the average of the color information of the past 30 frames for each pixel is within a certain distance range (distance in HSV color space) from the color information of the original pixel. It is a condition.

  For pixels satisfying the predetermined condition, the color information of the voxel that refers to the color information of the pixel is updated (determined) when determining the color information of the voxel using the color information of the latest pixel (step 1006). For pixels that do not satisfy the predetermined condition, the voxel color information that refers to the color information of the pixel is not updated (determined) when determining the color information of the voxel. Thereafter, as long as the mixed reality environment 43 is continuously created (step 1007), the process returns to step 1005.

  Next, the recognition phase will be described. FIG. 11 is a diagram illustrating global illumination using the light state of the real space 41 and the user's experience in the mixed reality environment 43 created in FIG. 10 according to the embodiment of the present invention. Shows how to provide highly accurate position tracking. It is assumed that the user wears the HMD 200 and is in the real space 41 (mixed reality environment 43). It should be noted that the color information of each voxel is determined (updated) in step 1004 or 1006 in FIG. 10 at the start of this step.

  By determining the position of the HMD 200 and the shooting direction of the shooting unit 203, the user environment, that is, the position of the user and the direction in which the user is facing are determined (step 1101). In this step, the rough position of the HMD 200 and the shooting direction of the shooting unit 203 are acquired using the position and orientation sensor 206 as a provisional user environment. Further, in this step, color information and three-dimensional position information of voxels that can be photographed by the photographing unit 203 in the position and photographing direction within a predetermined range are obtained from the provisional user environment. Further, in this step, the position of the HMD 200 and the shooting direction of the shooting unit 203 are compared by comparing the captured image of the real space shot by the shooting unit 203 with the acquired voxel position information and color information. To decide.

  Subsequently, it is determined whether or not there is a virtual object to be drawn in the direction in which the photographing unit 203 shoots (step 1102). Here, when a virtual object is arranged at a location corresponding to the direction in which the image capturing unit 203 captures an image in the virtual space 42 constituting the mixed reality environment 43, there is a virtual object to be drawn.

  When there is a virtual object to be drawn, the virtual illumination information on each surface of the virtual polyhedron containing the virtual object to be drawn is drawn by the IBL technique using the voxel color information and its three-dimensional position information. It is generated as virtual illumination information for the virtual object to be executed (step 1103).

  However, virtual illumination information for a virtual object to be drawn may be generated by a ray tracing technique using the voxel color information and its three-dimensional position information.

  Subsequently, the position and orientation are determined using the position of the HMD 200 determined in step 1101 and the shooting direction of the shooting unit 203, and shading (color) is added using the virtual illumination information generated in step 1104. The virtual object is drawn on the display unit 202 of the HMD 200 (step 1104). If there is no virtual object to be drawn, steps 1103 and 1104 are not performed.

  Subsequent step 1105 is the same processing as step 1005 in FIG. 10, and step 1106 is the same processing as step 1006 in FIG. Thereafter, as long as the mixed reality environment (global illumination and position tracking) continues to be provided (step 1107), the process returns to step 1101.

  As described above, in the embodiment of the present invention, colored point cloud data indicating the three-dimensional shape of a real object in the real space 41 is acquired in advance by using a laser scanner. However, since the measurement by the laser scanner takes time, the measurement is only performed once first unless the position / shape of the real object is changed.

  Next, the operational effects of the system 10 according to the first embodiment will be described. In the present embodiment, by using a table that associates the position information of the voxel and the position information of each pixel of the image acquired by the fixed camera 300, the color information of the voxel is obtained from the image acquired by the fixed camera 300. Determine and update from pixel color information. By using the table in this way, the color information obtained from the fixed camera 300 can be reflected on each of the voxels in real time. Thereby, in the mixed reality environment in which the shape of the real space is reproduced with high accuracy, the color of the real space 41 can be recognized with high accuracy and in real time and reflected in the virtual space.

  In the present embodiment, each voxel reflecting the color of the real space 41 is used as virtual indirect illumination (light source) of a virtual object to be drawn. As a result, global illumination using the light state of the real space 41 can be realized, and a virtual object having a color and a shadow very close to the environment of the real space 41 can be drawn.

  In the present embodiment, the real space 41 can be used as if it were the virtual space 42 in the real space 41. For example, the game developer can handle the real space in the same manner as the game space (virtual space) in the game engine.

  In the present embodiment, a virtual polyhedron that accommodates the virtual object is assumed for the virtual object to be drawn, and the color (light) state of each surface of the polyhedron reflects the color of the real space 41. Decide from each of the voxels that were made. Then, the determined color state on each surface is used as virtual indirect illumination of the virtual object to be drawn. As a result, it is possible to draw a virtual object having colors and shadows that are closer to the environment of the real space 41.

  In the present embodiment, the approximate position of the HMD 200 determined using the position / orientation sensor 206 and the shooting direction of the shooting unit 203 are set as a provisional user environment, and the shooting unit 203 is in a shootable region around the temporary user environment. The position of the HMD 200 and the photographing direction of the photographing unit 203 are determined by comparing the voxel color information and three-dimensional position information with the photographed image of the photographing unit 203 and collating them. In this way, the rough position information obtained from the conventional position and orientation sensor 206 is matched with the high-precision virtual space 42 using the three-dimensional space data, and the position of the user in the real space 41 and the virtual space 42 A shift in the direction in which the user is facing can be corrected in real time. As a result, it is possible to realize the MR environment 43 that links the reality and the high accuracy, and it is possible to realize position tracking.

[Second Embodiment]
In this embodiment, a mesh is used as a three-dimensional shape element. As in the case of the first embodiment, the obtained colored point cloud data is converted into a data structure called a mesh. For example, polygons having each point cloud as a vertex are formed from the obtained colored point cloud data using a known method, and converted to a mesh formed from one or a plurality of polygons. However, since the basic idea is the same as in the case of the voxel of the first embodiment, the description will focus on differences from the first embodiment.

  A table created by the table creation means 13 in this embodiment will be described. As described above, the three-dimensional position information of the fixed camera 300 and the shooting direction can be determined with high accuracy. A projection matrix that converts the color information of the pixel obtained from the fixed camera 300 into the color information of the point group by comparing the acquired point group data with color and each pixel of the image obtained from the fixed camera 300 ( Transformation matrix). As a known method in this regard, there is a method using optical flow or similarity of color information.

  The table creation means 13 creates the above projection matrix. The matrix is a table for converting a coordinate system of an image obtained from the fixed camera 300 into a coordinate system in a three-dimensional space. In one example, a table for converting the coordinate system of an image obtained from the fixed camera 300 into UV coordinates of a mesh in a three-dimensional space is created.

  A specific example regarding the projection matrix will be described below. When the position information of the fixed camera 100 can be obtained, the projection matrix can be set as follows, and the pixels of the image of the fixed camera 100 can be mapped to the color where the object is located in the three-dimensional space. become. Let P be the projection matrix. P is a 3x4 matrix configured as A (R | t), A is a matrix obtained from the angle of view, resolution, image center, etc. of the fixed camera 100 and is a fixed value depending on the device specifications. It is called a parameter matrix. R is a 3x3 rotation matrix and t is a translation vector. A 3x4 matrix with R | t arranged is called an external parameter matrix. A corresponds to the specification value of the fixed camera 100. R | t is the rotation and position in the world coordinate system, and can be obtained by collating the point cloud data and the image of the fixed camera 100 in this embodiment. This makes it possible to convert the image coordinate m to the world coordinate X using the normal projective transformation formula m = P · X.

  The color information determination unit 14 determines the color information of each mesh by converting the color information of the pixels of the image acquired from the fixed camera 300 using a conversion matrix. It is preferable that the image referred to by the color information of each mesh, that is, the fixed camera 300 that is the source of the color information of each mesh, is determined in advance.

  The color information update unit 15 determines the mesh color information determined by the color information determination unit 14 using the color information of the pixel when the change in the color information of the pixel of the image acquired by the fixed camera 300 satisfies a predetermined condition. Is updated (newly determined) using the latest pixel color information.

  In steps 1004 to 1006 and steps 1105 to 1106, the same processing as described above is performed.

  As described above, the color information of the mesh of the present embodiment is obtained by converting the color information of the pixels of the image obtained from the fixed camera 300 using the conversion matrix. Thus, it should be noted that having different colors within one mesh differs from the first embodiment having the same color information in each voxel.

[Third Embodiment]
A real space 51 that is an outdoor space as shown in FIG. 12 is assumed as a real-world space (predetermined real space) in which the system 10 according to the embodiment of the present invention provides the mixed reality environment 53 to the user. The real space 51 is the same as the real space 41 except that there is no ceiling and a pair of horizontal walls, and that the light source 22 including sunlight is outside the real space, not inside the real space. Also in the real space 51, the fixed camera 300 is arranged at each of a plurality of fixed points that can shoot an area in the real space 51 that can be visually recognized by the user. In order to simplify the description, in this embodiment, voxels are used as the three-dimensional shape elements, but mesh and point cloud data can also be used as the three-dimensional shape elements. Hereinafter, a description will be given focusing on differences from the first embodiment.

  Also in the present embodiment, the virtual space 52 is considered to be limited to an area (0 ≦ X ≦ X1, 0 ≦ Y ≦ Y1, 0 ≦ Z ≦ Z1) associated with the real space 51.

  In the present embodiment, unlike the first embodiment, since it is an outdoor space, there is an area where the fixed camera 300 does not project a real object.

  When the real object does not exist in at least a part of the image acquired by the fixed camera 300, the table creation unit 13 determines the three-dimensional position information and the shooting direction of the fixed camera 300 for pixels that do not display the real object in the image. Is used to relate to the position (x, y, z) on the boundary of the virtual space 52. In this case, assuming that there is, for example, a large mesh (or voxel) at the position on the boundary, the table may be created using the mesh by the method described in the first or second embodiment.

  The color information determination unit 14 determines each color information corresponding to the position information of the boundary at the boundary of the virtual space 42 by using the created table.

  The color information update unit 15 is configured to update the virtual space 42 determined by the color information determination unit 14 using the color information of the pixel when the change in the color information of the pixel of the image acquired by the fixed camera 300 satisfies a predetermined condition. Color information (such as a mesh) at a position on the boundary is updated (newly determined) using the color information of the latest pixel.

  In steps 1004 to 1006 and steps 1105 to 1106, the same processing as described above is performed.

[System Architecture]
As described above, the system architecture has the functions shown in FIG. 7, but the system architecture according to one embodiment can be composed of the following six modules. The six modules are voxel converter (Point Cloud to Voxel Converter), voxel DB, 3D address mapping table generator (hereinafter referred to as “generator”), 3D address mapping table (hereinafter referred to as “table”), position It consists of a tracker and ambient lighting parameters. By modularizing in this way, some modules can be changed and applied to the system.

  The voxel converter converts Point Cloud Data acquired in advance and stored in the cloud into a set of voxels without gaps using a technique such as Octomap. This conversion into voxels makes it easy to use as 3D information used directly in the game, and makes it easy to uniquely identify a voxel using coordinates, so that the format is suitable for color update.

  The voxel converter has a function corresponding to the three-dimensional spatial data creation means 12. Since the embodiment of the present invention does not depend on the method of using voxels, a mesh converter that converts point cloud data into a set of meshes without gaps is used as a component of the module instead of the voxel converter. Can do.

  The voxel DB divides the real world into fixed sections, converts all the contents of the sections into voxels, and makes them identifiable by position information (x, y, z). A high-precision voxel space capable of mapping a physical space with a minimum unit of voxels based on a certain minimum size (for example, 1 cm) in the real world is preferable.

  The voxel DB has a function corresponding to a database that stores the three-dimensional spatial data created by the three-dimensional spatial data creation means 12. Note that the embodiment of the present invention does not depend on the method of using voxels, and the voxel DB may store point cloud data directly or store a set of UV coordinates of a mesh in a three-dimensional space. May be.

  In this space, the generator maps each pixel of the video data and the colored voxel generated from the point cloud data using the position information and the color information of the voxel. Note that the embodiment of the present invention does not depend on a method using voxels. For example, mapping may be performed using position information and color information of one point in point cloud data. The function of the generator is a table for converting the coordinate system of the image obtained from the fixed camera into the coordinate system in the three-dimensional space by collating the image obtained from the fixed camera with the point cloud data or voxel data. Is to be generated. The generator has a function corresponding to the table creation means 13.

ここで、ｃｉｄは、固定カメラの識別し、ｃｘ、ｃｙは固定カメラのピクセル、ｖｘ、ｖｙ、ｖｚは、マッピング先となるボクセルの３次元座標を示す。なお、本発明の実施形態はボクセルを使用する方法に依存しておらず、ｖｘ、ｖｙ、ｖｚは、点群データの座標でもよいし、３次元空間上の特定のメッシュのＵＶ座標でもよい。テーブルは、テーブル作成手段１３により作成されたテーブルに対応する機能を有するものである。 The table is a table that statically associates each pixel of the fixed camera with n voxels in the three-dimensional space. This table is a set of mapping entries E shown below.

Here, cid identifies a fixed camera, cx, cy indicate pixels of the fixed camera, and vx, vy, vz indicate three-dimensional coordinates of voxels to be mapped. The embodiment of the present invention does not depend on the method of using voxels, and vx, vy, and vz may be coordinates of point cloud data or UV coordinates of a specific mesh in a three-dimensional space. The table has a function corresponding to the table created by the table creating means 13.

  The position tracker is a module that grasps the user's visual field by keeping track of the current position of the user and the direction of the line of sight. The position tracker has a function corresponding to the user environment determining means.

  The environmental lighting parameter is obtained as x, y, z corresponding to the query to the voxel DB for the position of the character to be displayed in the real world, and the global illumination around the location is calculated from the color information of the surrounding voxels. It is a module to do. The environmental illumination parameter has a function corresponding to the virtual illumination information generation means 16.

  In the process or operation described above, a process or operation can be freely performed at a certain step as long as there is no inconsistency in the process or operation such as using data that should not be used at that step. Can be changed. Moreover, each Example described above is the illustration for demonstrating this invention, and this invention is not limited to these Examples. The present invention can be implemented in various forms without departing from the gist thereof.

DESCRIPTION OF SYMBOLS 10 System 11 Memory | storage means 12 3D spatial data creation means 13 Table creation means 14 Color information determination means 15 Color information update means 16 Virtual illumination information generation means 17 User environment acquisition means 18 Drawing means 21 Light source 22 Building 23 Shadow 24 Character 25, 26 units 27 walls 31, 41, 51 Real world (real space)
32, 42, 52 Virtual world (virtual space)
33, 43, 53 Mixed reality world (mixed reality environment)
50 network 100 server 101, 201 processing unit 102, 202 display unit 103 input unit 104, 204 storage unit 105, 205 communication unit 110, 210 bus 200 display device 203 photographing unit 206 position and orientation sensor 300 fixed camera

Claims (2)

Method for creating a mixed reality environment for rendering a three-dimensional virtual object superimposed on a real space or a captured image of the real space that is visually recognized by a user through a display unit of a portable display device in a predetermined real space Because
Creating three-dimensional space data composed of three-dimensional shape elements each having three-dimensional position information based on point cloud data of real objects in the predetermined real space;
Acquiring images respectively from a plurality of fixed points capable of photographing an area in the predetermined real space;
Creating a table associating position information of each pixel of the acquired image with three-dimensional position information of one or a plurality of the three-dimensional shape elements indicated by each pixel;
Determining color information of the three-dimensional shape element for rendering the virtual object based on color information of one or a plurality of the pixels associated with the three-dimensional shape element;
Updating color information of the three-dimensional shape element for drawing the virtual object based on a change in color information of each pixel of the acquired image;
Having a method. To create a mixed reality environment for superimposing and drawing a three-dimensional virtual object on a real space or a captured image of a real space that is visually recognized by a user through a display unit of a portable display device in a predetermined real space System,
Three-dimensional spatial data creating means for creating three-dimensional spatial data composed of three-dimensional shape elements each having three-dimensional position information based on point cloud data of a real object in the predetermined real space;
Image acquisition means for acquiring images from a plurality of fixed points capable of capturing an area in the predetermined real space;
Table creation means for creating a table in which position information of each pixel of the acquired image is associated with three-dimensional position information of one or more of the three-dimensional shape elements indicated by each pixel;
Color information determining means for determining color information of the three-dimensional shape element for drawing the virtual object based on color information of one or a plurality of the pixels associated with the three-dimensional shape element;
Color information updating means for updating color information of the three-dimensional shape element for drawing the virtual object based on a change in color information of each pixel of the acquired image;
A system comprising:

Images