WO2017007254A1

WO2017007254A1 - Device and method for generating and displaying 3d map

Info

Publication number: WO2017007254A1
Application number: PCT/KR2016/007359
Authority: WO
Inventors: 도락주; 백인식; 이건용
Original assignee: 고려대학교 산학협력단
Priority date: 2015-07-08
Filing date: 2016-07-07
Publication date: 2017-01-12

Abstract

A device for generating and displaying a 3D map, according to one embodiment of the present invention, comprises: an acquisition unit for acquiring pose information including position and rotation information with regard to photographing locations, of which 3D information for the 3D space is obtained, on the basis of a global coordinate system; a generation unit for generating, based on the acquired pose information, a plurality of local coordinates for the 3D space according to at least one of spatial proximity and time proximity of the photographing locations; and a display unit for generating and displaying a 3D map by, if an observer of virtual reality is located within the 3D space, searching for a photographing location corresponding to the location of the observer by using the plurality of local coordinates, and using 3D information obtained in the corresponding photographing location.

Description

3D map generation and display device and method

Embodiments of the present invention relate to an apparatus and method for generating and displaying a 3D map.

Recently, the performance of PCs and smart devices has been advanced, and devices that can enjoy realistic images in three dimensions such as HMD (Head Mount Display) are commercially available. There is a growing interest in dimensional maps.

Existing two-dimensional map serviced by portal has limitation that only rotational movement based on x, y, z axis (3 degree of freedom) is possible. However, in 3D map, not only rotational movement about each axis, but also viewpoint movement in 3D space is possible, so there is an advantage that you can change the viewpoint with 6 degrees of freedom.

However, due to the limitations of the technology of generating such 3D maps in the real environment, current 3D maps are usually 3D models created by humans through 3D modeling programs such as computer-aided design (CAD) tools. It is a general opinion of the industry that the reality is less in terms of color, resolution, and the like compared to the three-dimensional model using the photograph taken at. Or, as shown in the road view provided by Daum Kakao Co., Ltd., it is limited to the form of showing 2D panoramic pictures at the place where the pictures were taken.

Therefore, there is a need for a system capable of easily acquiring three-dimensional spatial information in a real environment, a three-dimensional map generation method using the acquired data, and a service method for providing the generated three-dimensional map to the user. need.

According to an embodiment of the present invention, a three-dimensional map generation capable of finding and capturing a photographing point corresponding to the location information of an observer (avatar) in a virtual reality space and generating and displaying a three-dimensional map using the three-dimensional information acquired at the photographing point And a display device and a method thereof.

The problem to be solved by the present invention is not limited to the problem (s) mentioned above, and other object (s) not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, a 3D map generation and display apparatus includes pose information including position and rotation information about photographing points from which 3D information of a 3D space is acquired based on a global coordinate system. Acquisition unit for obtaining; A generation unit configured to generate a plurality of local coordinate systems with respect to the three-dimensional space according to at least one of the spatial proximity and the temporal proximity of the photographing points based on the acquired pose information; And when the observer of the virtual reality is located in the three-dimensional space, finds a corresponding photographing point corresponding to the point where the observer is located by using the plurality of local coordinate systems, and uses the three-dimensional information acquired from the corresponding photographing point to determine the photographing point. And a display unit for generating and displaying the dimensional map.

According to an embodiment of the present invention, the 3D map generation and display device includes at least one of an error between estimation of pose information associated with the plurality of generated local coordinate systems and an error and distortion of 3D information acquired at the selected photographing points. A calculator for calculating a defined grouping index; And comparing the calculated grouping index with a specific threshold to generate the plurality of local coordinate systems into a group area coordinate system and an independent area coordinate system, and manage the generated group area coordinate system and the independent area coordinate system in one global coordinate system. The management unit may further include.

3D map generation and display device according to an embodiment of the present invention includes a calculation unit for calculating the spatial and temporal proximity between the pose information associated with the generated plurality of local coordinate system; And

Comparing the calculated spatial and temporal adjacencies with a specific threshold, the plurality of local coordinate systems are divided into a group area coordinate system and an independent area coordinate system, and the generated group area coordinate system and the independent area coordinate system are generated in one global coordinate system. The management unit may further include a management unit.

The generation unit may further perform an operation of selecting photographing points satisfying a preset condition among the photographing points, and generating the plurality of local coordinate systems for the 3D space with respect to the selected photographing points. The preset condition may be a condition for selecting photographing points obtained by obtaining at least a predetermined number of three-dimensional information, or may be a condition for selecting photographing points having a predetermined rank or more.

The pause information may be obtained using a SLAM (Simultaneous Localization and Mapping) algorithm or location information input by a user.

When the observer is located at a point other than the photographing point in the three-dimensional space, the display unit may have a long axis of a bounding volume having a specific shape passing through the center of the observer and parallel to the observer's line of sight. Set the size of the bounding volume until the predetermined number of shooting points is included in the bounding volume, and find the corresponding shooting point included in the bounding volume, and use the 3D information acquired from the corresponding shooting point. To generate and display the 3D map.

The display unit, when there are a plurality of shooting points included in the bounding volume, based on the shooting time of the three-dimensional information acquired at each of the plurality of shooting points, the three-dimensional of the existing position of the three-dimensional information acquired from each of the plurality of shooting points 3D information having a minimum time difference from the information may be selected, and the 3D map may be generated and displayed using the selected 3D information.

The display unit corresponds to the entire 3D space based on the global coordinate system when the long axis length of the bounding volume is greater than a preset distance as the size of the bounding volume is enlarged when the 3D map is generated and displayed. By generating and displaying a 3D map, two 3D maps can be automatically switched and displayed according to the position of the observer.

The display unit rotates the bounding volume by using the gaze vector of the observer's gaze when the observer rotates to generate the three-dimensional map by using the three-dimensional information acquired at the corresponding photographing point included in the bounding volume. And the gaze vector includes a vector corresponding to each of the modes of roll, pitch, and yaw in the case of a gaze of 3 degrees of freedom, and the bounding volume in the case of gaze of 1 degree of freedom. It may include a vector corresponding to the long axis of the.

The display unit sets a plurality of cost functions based on each of a plurality of vectors from the photographing points to the point where the observer is located when the viewer is located at a point other than the photographing point in the three-dimensional space. The apparatus may find a corresponding photographing point for minimizing the value of the set cost function, and generate and display the 3D map using the 3D information acquired from the corresponding photographing point.

The display unit 3D information acquired at each of the plurality of shooting points based on the shooting time of the three-dimensional information obtained at each of the plurality of shooting points, when there are a plurality of corresponding shooting points to minimize the value of the set cost function 3D information having a minimum time difference from the 3D information of the existing location may be selected, and the 3D map may be generated and displayed using the selected 3D information.

When the display unit generates and displays the 3D map, when the length of each of the vectors from the photographing points to the viewer is greater than a preset length, the display unit corresponds to the entire 3D space based on the global coordinate system. By generating and displaying a 3D map, two 3D maps can be automatically switched and displayed according to the position of the observer.

The display unit generates the three-dimensional map by using the three-dimensional information obtained from another photographing point nearest to the photographing point, regardless of the position of the observer, when the obstacle includes the 3D information acquired at the photographing point. And it is displayed, it is preferable that the obstacle is not included in the three-dimensional information obtained from the other shooting point.

The display unit may load data related to a three-dimensional map that is stored in advance in correspondence with a location where the observer is located, and display the three-dimensional map using the data whenever the observer moves.

The display unit reproduces the 3D map in the form of a video when the viewer is located within a predetermined distance from one of the photographing points, and the observer moves during reproduction of the 3D map in the form of a video, If it is located outside a predetermined distance from one can further perform the operation of stopping the playback of the three-dimensional map of the video form.

The display unit sets a specific threshold distance value when playing the 3D map in the video form, and generates the 3D information generated by using the 3D information acquired at the photographing points within the threshold distance value from the observer's position. The 3D map can be played repeatedly in chronological order.

When the display unit reproduces the 3D map in the form of a video, the photographing point closest to the observer's position among photographing points included in a circle of a critical distance centering on the observer's position and having a radius of a critical distance as a radius. Search in reverse chronological order on the basis of the shooting time of to set the shooting point farthest from the observer's position as the interval repeating start point, and search in chronological order based on the shooting time of the shooting point nearest to the observer's position. After setting the photographing point farthest from the position of the interval repeating end point, the three-dimensional information taken at the interval repeating end point from the three-dimensional map generated using the three-dimensional information photographed at the interval repeating start point Sequence table up to the 3D map created using It can be repeatedly played back.

According to an embodiment of the present invention, a method of generating and displaying a 3D map includes pose information including position and direction information about photographing points from which 3D information of a 3D space is acquired based on a global coordinate system. Obtaining a; Generating a plurality of local coordinate systems with respect to the three-dimensional space based on at least one of spatial proximity of the photographing points and temporal proximity of photographing points corresponding to the photographing points, based on the acquired pose information; And when the observer of the virtual reality is located in the three-dimensional space, finds a corresponding photographing point corresponding to the point where the observer is located by using the plurality of local coordinate systems, and uses the three-dimensional information acquired from the corresponding photographing point to determine the photographing point. Generating and displaying the dimensional map.

In the method of generating and displaying a 3D map according to an embodiment of the present disclosure, the generating of the plurality of local coordinate systems may further include selecting photographing points satisfying a preset condition from among the photographing points. At least one of the adjacency and the temporal adjacency is calculated with respect to the selected photographing points, and the preset condition is a condition for selecting photographing points for acquiring a predetermined number or more of 3D information, or the number of acquired 3D information is equal to or greater than a predetermined rank. It may be a condition for selecting photographing points.

The generating and displaying of the 3D map may include a long axis of a bounding volume having a specific shape passing through the center of the observer when the observer is located at a point other than the photographing point in the 3D space. And parallel to the observer's line of sight; Finding a corresponding shooting point included in the bounding volume by enlarging or reducing the size of the bounding volume until a predetermined number of shooting points is included in the bounding volume; And generating and displaying the 3D map using the 3D information acquired at the corresponding photographing point.

The generating and displaying of the 3D map may be performed by rotating the bounding volume by using a gaze vector of the observer's gaze, and using the 3D information acquired at a corresponding photographing point included in the bounding volume. And generating and displaying a map, wherein the line of sight vector corresponds to a mode of roll, pitch, and yaw when the line of sight has a degree of freedom of 3, such as a head mounted display (HMD). In the case of a line of sight 1 having a degree of freedom, such as an overhead view, a vector corresponding to the long axis of the bounding volume may be included.

The generating and displaying of the 3D map may include: when the observer is located at a point other than the shooting point in the three-dimensional space, each of the plurality of vectors from the shooting points to the point where the observer is located. Establishing a plurality of cost functions based on the number of cost functions; Finding a corresponding photographing point to minimize the value of the set cost function; And generating and displaying the 3D map using the 3D information acquired at the corresponding photographing point.

Specific details of other embodiments are included in the detailed description and the accompanying drawings.

According to an embodiment of the present invention, a photographing point corresponding to the location information of an observer (avatar) in the virtual reality space may be found, and a 3D map may be generated and displayed using the 3D information acquired at the photographing point.

1 is a block diagram illustrating a three-dimensional map generation and display device according to an embodiment of the present invention.

2 to 5 are diagrams for explaining an embodiment of generating and displaying a 3D map when using a bounding volume.

6 to 8 are diagrams for explaining another embodiment of generating and displaying a 3D map when using a cost function.

FIG. 9 is a diagram illustrating an example of changing a virtual reality environment in accordance with the movement of an observer.

10 is a diagram illustrating an example of a photo for creating a virtual reality environment and an example of a 3D screen viewed from a user's point of view.

FIG. 11 is a diagram for explaining an example of importing and displaying specific 3D information regardless of the actual position of the observer in the virtual reality.

FIG. 12 is a diagram showing that the space is divided, which information acquisition point should be selected at which position, and previously stored and stored.

13 is a flowchart illustrating a 3D map generation and display method according to an embodiment of the present invention.

Advantages and / or features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only the present embodiments to make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention;

Referring to FIG. 1, the 3D map generation and display device 100 according to an exemplary embodiment of the present invention may include an acquisition unit 110, a generation unit 120, a display unit 130, a calculation unit 140, and a management unit ( 150, and the controller 160.

The acquirer 110 acquires pose information including position and rotation information about photographing points from which 3D information of a 3D space is acquired based on a global coordinate system.

Here, the global coordinate system refers to a reference coordinate system for the entire 3D space, and the 3D information includes a 3D image, a 3D video, etc. photographed at photographing points, and can capture a 3D space 360 degrees. It may be generated using a plurality of two-dimensional images generated by a plurality of cameras.

The 3D video refers to video data in which RGB information and depth information of the video exist in every image frame constituting the video. At the time of executing the 3D video, the system obtains RGB and depth information of a corresponding time or index for each image frame, and displays the combined information. The combination of the two is implemented by distorting the shape of the RGB image using depth information. If the original RGB image and depth information were captured by a normal camera, the method of distortion distorts the height for each pixel in the plane, and is displayed in the height map format recognized as standard in 3D graphics. If the original RGB image and depth information were captured by a 360-degree view camera, it would be displayed in the form of a distorted sphere moved in the opposite direction of the sphere's center vector or the sphere's center vector according to the depth information corresponding to the respective RGB data of the sphere. do. The figure showing the distortion of the sphere is shown in FIG.

In addition, the 3D information may be generated using a plurality of 2D images generated by one or more cameras capable of capturing a portion of the 3D space. In addition, it may be generated by combining one or more images generated by one or more cameras capable of capturing a portion of the three-dimensional space and one or more lidars capable of laser scanning of the three-dimensional space.

In addition, acquisition of pose information may be performed using a SLAM (Simultaneous Localization and Mapping) algorithm and other location estimation algorithms, or may be performed by location information input by a user.

Preferably, the pose information may be generated using a lidar, an inertial sensor, an indoor GPS, or a combination thereof capable of laser scanning of a three-dimensional space. In addition, the rotation information included in the pose information may include a rolling angle, pitch angle, and yaw angle with respect to the 3D information.

Meanwhile, according to an embodiment of the present invention, a robot equipped with a device including a lidar, an inertial sensor and two 360 degree cameras (upper and lower) moves through a predetermined three-dimensional space, The two-dimensional images of the dimensional space may be acquired (processed into three-dimensional information after being acquired), and the pose information of the two-dimensional images may be obtained through the lidar. However, the present invention is not limited thereto, and images and pose information about a 3D space may be obtained through various methods.

The generation unit 120 generates a plurality of local coordinate systems with respect to the three-dimensional space by calculating at least one of the spatial proximity of the photographing points and the temporal proximity of the photographing points corresponding to the photographing points based on pose information. .

For example, the generation unit 120 may generate two or more local coordinate systems by giving priority to photographing points that are temporally and spatially adjacent to the photographing points. Here, the local coordinate system means a reference coordinate system for each area existing in the three-dimensional space. In such a local coordinate system, the coordinates of the shooting points may be displayed based on the origin of the corresponding local coordinate system. When the observer is located in a specific local coordinate system, only the 3D information of the shooting points belonging to the local coordinate system may be displayed to the observer. Can be.

Preferably, the origin of the local coordinate system may be designated as an inertial sensor mounted on the robot moved to obtain 3D information and a point on the path of the origin of the lidar, but may be randomly determined as an arbitrary point. have.

In addition, the generation unit 120 may further perform an operation of selecting photographing points satisfying a preset condition among the photographing points.

For example, the generation unit 120 may select a shooting point for acquiring a predetermined number or more of 3D information as a preset condition (ie, a shooting point for acquiring 10 or more pieces of 3D information) or the number of the acquired 3D information is greater than or equal to a predetermined rank. After setting the conditions for selecting the shooting points (ie, the shooting points included in the top 20 on the basis of the number of 3D information acquired at each of the shooting points), the shooting points satisfying the set conditions are selected, and the selected A plurality of local coordinate systems for the 3D space may be generated by calculating at least one of the spatial adjacency of the photographing points and the temporal adjacency of the photographing points corresponding to the selected photographing points.

When the observer of the virtual reality is located in the three-dimensional space, the display unit 130 finds a corresponding photographing point corresponding to the spot where the observer is located by using the plurality of local coordinate systems, and acquires the three-dimensional image obtained from the corresponding photographing point. Create and display 3D maps using information.

In this case, the observer may be located at a point other than the photographing point in the three-dimensional space. In this case, the display unit 130 may generate and display a 3D map corresponding to the position of the observer using a bounding volume in one embodiment, and in another embodiment, the cost function using the cost function. A 3D map corresponding to the position of the observer may be generated and displayed.

In one embodiment, when the observer is located at a point other than the photographing point in the 3D space, the long axis of the bounding volume having a specific shape passes through the center of the observer. Set to be parallel to the observer's line of sight, and expand or reduce the size of the bounding volume until a predetermined number of shooting points are included in the bounding volume to find a corresponding shooting point included in the bounding volume. The 3D map may be generated and displayed using the 3D information obtained from.

In this case, when there are a plurality of shooting points included in the bounding volume, the display unit 130 of the three-dimensional information acquired at each of the plurality of shooting points based on the shooting time of the three-dimensional information acquired at each of the plurality of shooting points. The 3D information having a minimum time difference from the 3D information of the existing location may be selected, and the 3D map may be generated and displayed using the selected 3D information.

For example, the display unit 130 may provide a user interface that selects and displays one piece of three-dimensional information when a plurality of pieces of three-dimensional information exist in one place (shooting point). There may be several pieces of 3D information photographed at a distance below the threshold from the position of the observer of the virtual position, for example, at a 3D point within 2 m from the observer. At this time, the shooting time between the plurality of three-dimensional information may be different. When the 3D information 1 is an image photographed during the day and the 3D information 2 is an image photographed at night, the brightness image information of the two 3D information may be different even in the same space due to the influence of light. In addition, because the shooting time is different, the surrounding objects, such as people passing by, furniture arrangement may be different, and thus there may be a lot of difference in image information. Due to this difference, image information may change rapidly and frequently when an observer in a virtual space passes through the space. As a result, the virtual reality user may feel uncomfortable.

In order to prevent this, in the present embodiment, when there is a plurality of 3D information, the 3D information photographed at the same (or similar) time as the 3D information shown at the location of the existing user is preferentially selected and displayed. We propose a priority algorithm that uses 3D information taken at different times only when there is no 3D information. In addition, the user may be informed that various pieces of 3D information exist at the corresponding location, and may provide an interface for the user to select it.

On the other hand, when the display unit 130 generates and displays the 3D map, when the length of the long axis of the bounding volume is larger than a preset distance as the size of the bounding volume is enlarged, the display unit 130 may display the 3D map based on the global coordinate system. By generating and displaying a 3D map corresponding to the entire dimensional space, two 3D maps can be automatically switched and displayed according to the position of the observer. That is, the display unit 130 generates and displays a 3D map corresponding to the entire 3D space based on the global coordinate system according to the position of the observer, or 3 corresponding to the photographing point of the point where the observer is located based on the local coordinate system. Create and display dimensional maps.

In addition, when the observer rotates, the display unit 130 rotates the bounding volume by using a gaze vector of the observer's gaze, and uses the 3D information acquired at a corresponding photographing point included in the bounding volume. Create and display 3D maps. Here, the gaze vector includes a vector corresponding to each mode of roll, pitch, and yaw when the gaze vector has a degree of freedom of 3, such as a head mounted display (HMD), and includes an overhead view ( In the case of a line of sight 1 having a degree of freedom, such as an overhead view, a vector corresponding to the long axis of the bounding volume may be included.

The above-described embodiment will be described in more detail with reference to FIGS. 2 to 5 as follows.

Bounding volume is any invisible space that is closed to a closed surface in virtual space in computer graphics. In the game, the bounding volume refers to the shape of a virtual object that acts like a button that starts the action of the main character's dialogue when the user's avatar enters a specific bounding volume. The method of an embodiment using the bounding volume is as follows.

First, a bounding volume having a specific shape is set as shown in FIG. 2. For example, the bounding volume may be set as a bounding box having a rectangular parallelepiped shape (FIG. 2), a bounding ellipsoid having a three-dimensional ellipse shape, and the like. In this embodiment, it is set as a bounding box. At this time, the longest axis of the bounding box is called the long axis, and the long axis passes through the center of the observer's head and is set to be parallel to the observer's line of sight.

Next, the size of the bounding box is enlarged or reduced to change its ratio. In this case, as shown in FIG. 3, when the bounding box is enlarged at the same ratio K in all directions with respect to the center, the number of information acquisition points (shooting points) included in the bounding box is set to M (M Is one or more natural numbers). In this case, the ratio K is changed to any value larger than 0, and the change is repeated until the number of information acquisition points included in the bounding box (volume) is equal to M. In this process, several search algorithms such as binary search and linear search can be used.

Next, as shown in FIG. 4, a three-dimensional map corresponding to a space reconstructed with three-dimensional information of the information acquisition point d included in the bounding box is displayed. At this time, the bounding box can be rotated using only rotation about a specific axis. With such constraints, for example, when the user (observer) moves on the floor, the long axis of the bounding box may be perpendicular to the floor, so that appropriate three-dimensional information may not be retrieved.

Next, as shown in FIG. 5, the observer removes the three-dimensional information currently displayed when leaving the position.

In another embodiment, when the observer is located at a point other than the photographing point in the three-dimensional space, the display unit 140 may display a plurality of vectors from the photographing points to the point where the observer is located. Set a plurality of cost functions based on the plurality of cost functions, find a corresponding photographing point for the minimum or maximum value of the set cost function, and generate and display the 3D map using the 3D information acquired from the corresponding photographing point. Can be.

In another embodiment, when setting the cost function, the display unit 130 may consider not only the point where the observer is located, but also the observer's gaze, the moving direction, and the moving speed. In this case, the display unit 130, when there are a plurality of corresponding shooting points to maximize the value of the set cost function, each of the plurality of shooting points based on the shooting time of the three-dimensional information obtained from each of the plurality of shooting points Three-dimensional information having a minimum time difference from the three-dimensional information of the existing position among the three-dimensional information acquired at, may be selected, and the three-dimensional map may be generated and displayed using the selected three-dimensional information.

In addition, the display unit 130 generates and displays the 3D map, when the length of each of the vectors from the photographing points to the viewer is greater than a preset length, based on the global coordinate system. By generating and displaying a three-dimensional map corresponding to the entire space, two three-dimensional maps can be automatically switched and displayed according to the position of the observer.

Another embodiment described above will be described in more detail with reference to FIGS. 6 to 9. The method of another embodiment using the cost function is as follows.

First, when the position of the observer is called P1 in FIG. 6 and the positions of the information acquisition points (shooting points) are respectively a, b, c, and d, the position of the observer from the information acquisition points (shooting points) is reached. The observer point vectors V1, V2, V3, and V4, which are vectors, are defined.

Next, the magnitude of the observer point vector Vn is set as a cost function. Next, find 'b', which is an information acquisition point for V2 that minimizes the result of the cost function, that is, the size of the observer point vector Vn is the smallest. In this case, when a plurality of information acquisition points are to be found, 'd' where the result value of the cost function is smaller than 'b' may be found. Meanwhile, in another embodiment, when setting the cost function, the gaze vector for the observer's gaze may be further considered. For example, in another embodiment, a plurality of cost functions are set on the basis of the gaze vector for the observer's gaze and each of the vectors from the shooting points to the observer, and the corresponding shooting point is set such that the value of the set cost function is maximized. You can find it.

In this case, when the observer gaze vector is represented by V1 and the vectors from the photographing points to the observer are represented by V2, the cost function may be expressed as in Equation (1).

[Equation 1]

That is, according to Equation 1, the cost function is set such that the result value of the cost function fn (v1, v2) is directly proportional to the dot product of V1 and V2, and the result value of the cost function is inversely proportional to the magnitude of V2. Can be. The proportional relationship does not need to be linear, and an exponential function, a log function, and the like can be used, but it is preferable that the singular function (a sign of a slope is constant).

Next, as shown in FIG. 7, the 3D space is restored (created and displayed 3D map corresponding to b) using the selected information acquisition point 'b'.

Next, when the observer moves away from the current position as shown in FIG. 8, the 3D map of the 3D space restored to the corresponding information acquisition point 'b' is not shown and removed from the memory.

In this case, when the information acquisition point 'd' is selected as the observer moves, the 3D information of the information acquisition point 'd' may be restored. For example, as the observer moves, the surrounding virtual reality environment may change from FIG. 9A to FIG. 9B. In FIG. 9, (a) shows a movement of a character (observer) in a virtual reality environment, and (b) shows a 3D environment that is different from the previous environment.

As described above, if the process of restoring and then removing the 3D information at the information acquisition point is repeated, since the environment around the observer is continuously changed, the observer may feel that the environment is moving forward or backward.

On the other hand, if the process of restoring and then removing the 3D information at the information acquisition point is applied to all spaces, as the user manipulates the avatar of the virtual space, the virtual space is not local space due to continuous 3D information update. You may feel that the whole is restored in three dimensions.

That is, when the 3D map is continuously updated and displayed with the 3D information corresponding to the avatar's position as the avatar moves, the avatar sees the local 3D information at a point in time, but the 3D information continues as the avatar moves. It feels like you're navigating a large 3D space because it is updated to.

An example for explaining this embodiment in more detail is as follows.

In the global space O, the information acquisition points A and B exist, and a non-moving object K exists in the coordinate of a in the local coordinate system with the A point as the origin, and in the local coordinate system with the B point as the origin, When present in the coordinates, the position of the object K has different coordinates in the local coordinate system based on the point A and the local coordinate system based on the point B, but has the same coordinates based on the global coordinate system.

In the present invention, since the information acquisition point A and B are managed as the local coordinate system, respectively, and the two local coordinate systems are managed through one global coordinate system, as the avatar moves from the A point to the B point, the A point is the origin. Even if the object K displayed on the three-dimensional map of the local coordinate system is updated and displayed on the three-dimensional map of the local coordinate system with the point B as the origin, only the observation point of the object is changed and the position is managed without change. Accordingly, even if the avatar moves from point A to point B, the position of the object K is displayed on the 3D map unchanged, and the user can feel that the 3D map is displayed continuously on the screen.

Since the 3D map is continuously updated and displayed even when the avatar moves as described above, the present invention can be regarded as an advanced technology than the 2D street view technology of displaying discontinuous images according to the movement of the position on the map.

FIG. 10 illustrates an example of a photo for creating a virtual reality environment and an example of a 3D screen viewed from a user's point of view. In FIG. 10, (a) represents an original RGB photograph, and (b) restores a 3D image (three-dimensional information). Depth image for (c) represents a 3D screen seen from the user's point of view. In the case of FIG. 10 (c), since (a) the original RGB photograph and (b) the depth image were generated by being projected inside the spherical three-dimensional space, (a) the original RGB photograph was displayed in a distorted direction toward the center of the sphere. Able to know.

In this case, (c) the shape of the 3D screen viewed from the user's point of view may be a three-dimensional shape surrounding the character in the virtual reality, may be a plane shape visible only to a part of the field of view, or may be a shape of a curved surface surrounding the entire field of view. It may be.

When the obstacle includes the 3D information acquired at the photographing point, the display unit 130 uses the 3D information acquired at another photographing point closest to the photographing point regardless of the position of the observer. Create and display 3D maps. In this case, it is preferable that the obstacle is not included in the 3D information acquired at the other photographing point.

As such, the display unit 130 may retrieve and display specific 3D information regardless of the actual position of the observer in the virtual reality. Such a function is called a location fake function in the present invention. Here's why you need a fake feature: In the case of the present invention, the surrounding environment is automatically changed according to the movement of the observer. In this case, there is a problem that may occur, which is a case where you want to observe specific three-dimensional information up close.

Referring to FIG. 11, it is assumed that an observer in a virtual reality observes an object C at a long distance as shown in FIG. 11A. As shown in (b) of FIG. 11, when moving to the B position, the object C can be observed closer and moved to the position B. An image of another object may exist between the observer and the C object. For example, a person may have passed between a photographer and a C object at the time of 3D information collection. In this case, although the distance from the object C is close, the shape of the object C can not be confirmed. If we photographed C from B, it might have been better than C's image from A, but we can't see C near, because we don't see C from B.

In order to overcome this problem, a function (preferably called a hold function in the present invention) is required to prevent 3D information replacement due to movement. If you activate this hold function, you can prevent 3D information replacement by movement. When the present hold function is used for the previous problem, as shown in FIG. 11 (b), even when the user at the position A drives the hold function and goes to the position B, the 3D information is not updated. At this time, the three-dimensional shape shown is a three-dimensional shape reconstructed at the position of A, unlike the three-dimensional information reconstructed with the imaging three-dimensional information at the position of B, the object C can be observed. This hold function can be driven by input button, UI on the screen, or automatically executed.

Here, in the technology of automatically executing the hold function, the video content may provide an interface that prevents the 3D information update even if the viewer moves until the playback of the video is completed. If there is no such hold function, an observer may move even though the video content is not finished. When the user plans the content to watch the video to the end, the user can force the user to watch the video content by executing the hold function when the video starts playing and ending the hold function when the video ends.

In another embodiment, the display unit 130 reproduces the 3D map in the form of a video when the observer is located within a predetermined distance from one of the photographing points, and the observer moves during the reproduction of the 3D map in the form of a video to capture the spot. If it is located outside a predetermined distance from one of these may further perform the operation of stopping the playback of the three-dimensional map of the video form. In this case, the 3D map in the form of a video means a form of 3D information in which 3D information on a 3D map changes with time, and color information (RGB + Alpha) of the 3D information changes with time.

In another exemplary embodiment, the display unit 130 may set a specific threshold distance value R_repeat when playing a video, and may repeatedly play only images photographed at photographing points within a threshold distance value from the observer's position.

At this time, the interval repetition of the video is searched in reverse chronological order based on the recording time of the closest point to the observer's position among the recording points included in the circle of the critical distance centered on the position of the observer and the critical distance value is the radius. Section retrieval point that is farthest from the observer's position as the start point of interval repeating, and search in time order based on the recording time of the point that is closest to the observer's position. Can be performed. That is, the images acquired at the plurality of photographing points are acquired according to the photographing motion lines. In this embodiment, the photographing points where the images are acquired first in time on the photographing motion lines within a critical distance circle are regarded as the interval repeating start points. Using the shooting point where the image was acquired as the end point of the interval repeat, in order from the earliest acquired image (image acquired at the interval repeat start point) to the last acquired image (image acquired at the interval repeat end point) within the corresponding critical distance circle. By playing repeatedly, video section repetition is realized.

For example, when the observer is located near the 140th photographing point, the display unit 130 may be located among the photographing points included in a critical distance circle centered on the observer's position and having a radius of a predetermined threshold distance value (for example, 5 m). Set the first 119th shooting point recorded as the interval repeat start point, set the last 151th shooting point recorded as the interval repeat end point, and then sequentially repeat the images recorded at each of the 119th to 151th shooting points. By repeating the video section repetition, the video section repeating can be performed. Through such video section repeating, it is possible to provide more realistic 3D information to the user.

Meanwhile, in another embodiment, the observer moves during the video section repetition to change the section repeat start point and the section repeat end point. In this case, the display unit 130 immediately before the observer moves instead of performing the video segment repetition based on the new segment repeat start point and the new segment repeat end point according to the new observer position so that the video jump does not occur according to the observer's movement. The image photographed at the photographing point immediately after the photographing point corresponding to the photographing point of the image which has been reproduced may be reproduced. Here, the video jump means moving to another image frame which is suddenly discontinuous by a user's input or the like while playing a continuous image frame.

For example, while the observer is located at the 140th shooting point, the 119th shooting point is set as the interval repeat start point, and the 151th shooting point is set as the interval repeat end point, the viewer is playing the 148th shooting point while the image of the 140th shooting point is being played. When the section repeat start point is changed to the 126th photographing point and the section repeat end point is changed to the 155th photographing point, the display unit 130 plays an image of the 126th photographing point (the changed section repeating start point) or the 148th photographing point ( By reproducing the image of the 141st photographing point without reproducing the image of the changed observer position), the moving image is naturally reproduced regardless of the observer's movement.

However, when the observer's moving speed is very fast, the video jump may be allowed when the critical distance circle centered on the observer's position after the movement and the critical distance circle centered on the observer's position before the movement do not overlap at all.

Meanwhile, the display unit 130 loads three-dimensional information about a three-dimensional map that is stored in advance corresponding to the point where the observer is located, and displays the three-dimensional map by using the three-dimensional information whenever the observer moves. can do.

In detail, the display unit 130 may provide a function of pre-calculating and storing in the form of 3D information before execution of a program which 3D information that the observer in the virtual reality should bring up and display in the program. By utilizing this function, the computational burden to select a lot of 3D information every time is reduced, which helps to increase real-time. This is implemented in the following way. When the observer in the virtual reality is at an arbitrary position, the three-dimensional information to be retrieved from the position is calculated before the program execution. In addition, if manual work is required, the three-dimensional information to be modified manually. Arbitrary points thus calculated are set as regions. When the program is executed, when the observer in the virtual reality enters the relevant area, the 3D information to be displayed is selected by using the result of the previously calculated and stored operation without performing the operation of finding the necessary 3D information. For example, FIG. 12 divides the space, calculates in advance which information acquisition point should be selected at which position according to the position (pos) of the observer's movement paths, and segments the three-dimensional space to capture the three-dimensional information. It is an expression of the storage.

The calculation unit 140 calculates a grouping index for a plurality of local coordinate systems generated by the generation unit 120. Here, the grouping indicator may include an estimation error between two or more pose information associated with the local coordinate system, a distortion due to the influence of light when matching three-dimensional information acquired at the selected photographing points into one, and each It may be defined as one value that quantifies the error of the 3D information (distance information) acquired from the pose information.

In this case, according to an embodiment of the present invention, the grouping indicator may be defined as a weighted sum of a plurality of estimation errors as shown in Equation 2 below.

[Equation 2]

In this case, W is a weight and a sum of weights from W1 to Wn is 1, and e is an error variable of three-dimensional information, and each of e1, e2, and e3 is an error variable representing different uncertainties. To estimate the error variable

On the other hand, as an error variable of a single information acquisition point (shooting point), variance of position estimation, standard deviation, Mahalanobis distance, etc. may be used, and as a measure of similarity between a plurality of information acquisition points. As error variables, covariance of an image and an energy function comparing each other with 3D feature point information may be used.

The management unit 150 generates the plurality of local coordinate systems by dividing the plurality of local coordinate systems into a group area coordinate system and an independent area coordinate system by comparing the calculated grouping index with a specific threshold value, and generates the generated group area coordinate system and the independent area coordinate system by one. Can be managed in the global coordinate system.

For example, when the grouping index between two or more pose information associated with the local coordinate system is less than the threshold value, the two or more pose information is generated as a group area coordinate system. When the grouping index is greater than or equal to the threshold value, the two or more pose information is generated. Each of these can be generated by dividing it into an independent local coordinate system.

In this way, if the grouping indicator is set and the 3D information corresponding to the pose information whose grouping index is less than the threshold is regarded as similar 3D information, and the corresponding pose information is generated as a group area coordinate system instead of an independent area coordinate system, it is unnecessary. Since it is possible to prevent the update of the 3D map, it is possible to reduce the volume of the 3D information used to generate the 3D map.

For example, when three-dimensional information corresponding to a plurality of pose information is very similar to each other, even if the avatar moves and the three-dimensional map is updated using the new three-dimensional information, the user sees little change on the screen. In such a case, since it is not necessary to update the 3D map by using all the similar 3D information, among the corresponding 3D information, the pose information corresponding to the plurality of similar 3D information is managed by grouping coordinates. By updating the three-dimensional map using only one, unnecessary updating can be prevented.

On the other hand, with respect to the two or more shooting points corresponding to the two or more pose information managed by the group area coordinate system, when the observer enters any one of the shooting points, the shooting points corresponding to the group area coordinate system After loading all the 3D information about the 3D map can be generated and displayed at once.

Further, in another embodiment, after calculating spatial and temporal proximity to two or more shooting points, and comparing them with a specific threshold, a plurality of local coordinate systems corresponding to the corresponding shooting points are converted into a group area coordinate system and an independent area coordinate system. It is also possible to distinguish and generate and manage the generated group area coordinate system and the independent area coordinate system in one global coordinate system. In this case, when managing the shooting points in the local coordinate system, the coordinates of the shooting points are managed based on the origin point (0,0) of the individual local coordinate system, and in case of managing each local coordinate system in the global coordinate system, The coordinates of the origin of the local coordinate system are newly set based on the origin of the global coordinate system, and the coordinates of the photographing points are set and managed based on the coordinates of the origin of each of the newly set local coordinate systems.

The controller 160 is a three-dimensional map generation and display device 100 according to an embodiment of the present invention, that is, the acquisition unit 110, the generation unit 120, the display unit 130, the calculation unit ( 140), the operation of the manager 150 may be controlled overall.

13 is a flowchart illustrating a 3D map generation and display method according to an embodiment of the present invention. The 3D map generation and display method may be performed by the 3D map generation and display apparatus 100 of FIG. 1.

Referring to FIG. 13, in operation 1310, the 3D map generating and displaying apparatus includes a pose including position and rotation information about photographing points obtained by obtaining 3D information on a 3D space based on a global coordinate system. (pose) Acquire information.

Next, in operation 1320, the 3D map generating and display apparatus calculates at least one of the spatial proximity of the photographing points and the temporal adjacency of the photographing points corresponding to the selected photographing points, thereby obtaining a plurality of the 3D maps. Create a local coordinate system.

Meanwhile, in another embodiment, before the step 1320, the step of selecting shooting points satisfying a predetermined condition among the shooting points may be performed, and in this case, the step 1320 may be performed based on the selected shooting points. have.

Next, in operation 1330, when the observer of the virtual reality is located in the 3D space, the 3D map generation and display apparatus uses the plurality of local coordinate systems to capture a corresponding photographing point corresponding to the position where the observer is located. Find it.

Next, in operation 1340, the 3D map generation and display device generates and displays a 3D map using the 3D information acquired at the corresponding photographing point.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. The computer readable medium may include program instructions, local data files, local data structures, or the like, alone or in combination. The media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

While specific embodiments of the present invention have been described so far, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims below, but also by the equivalents of the claims.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, the spirit of the present invention should be understood only by the claims set forth below, and all equivalent or equivalent modifications thereof will belong to the scope of the present invention.

Claims

An acquisition unit for obtaining pose information including position and rotation information about photographing points from which three-dimensional information of the three-dimensional space is acquired based on the global coordinate system;

A generation unit configured to generate a plurality of local coordinate systems with respect to the three-dimensional space according to at least one of the spatial proximity and the temporal proximity of the photographing points based on the acquired pose information; And

When the observer of the virtual reality is located in the three-dimensional space, using the plurality of local coordinate systems to find the corresponding shooting point corresponding to the location where the observer is located, three-dimensional by using the three-dimensional information acquired from the corresponding shooting point Display unit for creating and displaying map

3D map generation and display device comprising a.
The method of claim 1,

A calculator configured to calculate a grouping index defined by at least one of an estimation error between pose information associated with the generated plurality of local coordinate systems and an error and distortion of three-dimensional information acquired at the selected photographing points; And

Comparing the calculated grouping indicators with a specific threshold value, the plurality of local coordinate systems are divided into a group area coordinate system and an independent area coordinate system, and the generated group area coordinate system and the independent area coordinate system are managed in one global coordinate system. Management

3D map generating and display device further comprising.
The method of claim 1,

A calculator configured to calculate spatial and temporal proximity between pose information associated with the generated plurality of local coordinate systems; And

Comparing the calculated spatial and temporal adjacencies with a specific threshold, the plurality of local coordinate systems are divided into a group area coordinate system and an independent area coordinate system, and the generated group area coordinate system and the independent area coordinate system are generated in one global coordinate system. Managing department

3D map generating and display device further comprising.
The method of claim 1,

The generation unit

Selecting photographing points satisfying a predetermined condition among the photographing points, and generating a plurality of local coordinate systems for the 3D space with respect to the selected photographing points,

And the preset condition is a condition for selecting photographing points obtained by obtaining at least a predetermined number of three-dimensional information, or a condition for selecting photographing points having a predetermined rank or more.
The method of claim 1,

The pose information is

3D map generation and display device, characterized in that obtained by using a SLAM (Simultaneous Localization and Mapping) algorithm or location information input by the user.
The method of claim 1,

The display unit

When the observer is located at a point other than the photographing point in the three-dimensional space, the long axis of the bounding volume having a specific shape is set to pass through the center of the observer and parallel to the observer's line of sight, The size of the bounding volume is increased or reduced until a predetermined number of shooting points are included in the bounding volume to find a corresponding shooting point included in the bounding volume, and the 3D information is obtained by using the 3D information acquired from the corresponding shooting point. 3D map generation and display device, characterized in that for generating and displaying the dimensional map.
The method of claim 6,

The display unit

When there are a plurality of shooting points included in the bounding volume, three-dimensional information of the existing position and the minimum of the three-dimensional information acquired at each of the plurality of shooting points based on the shooting time of the three-dimensional information acquired at each of the plurality of shooting points And selecting 3D information having a time difference of and generating and displaying the 3D map using the selected 3D information.
The method of claim 6,

The display unit

When generating and displaying the 3D map, when the length of the long axis of the bounding volume is greater than a preset distance as the size of the bounding volume is enlarged, a 3D corresponding to the entire 3D space based on the global coordinate system And generating and displaying a map to automatically switch and display two three-dimensional maps according to the position of the observer.
The method of claim 6,

The display unit

When the observer rotates, the bounding volume is axially rotated using a gaze vector of the observer's gaze to generate and display the three-dimensional map by using three-dimensional information acquired at a corresponding photographing point included in the bounding volume. ,

The eye vector is

If the line of sight has 3 degrees of freedom, it includes a vector corresponding to each mode of the roll, pitch, and yaw; if the line of sight has a degree of freedom of 1, it includes a vector corresponding to the long axis of the bounding volume. 3D map generating and display device characterized in that.
The method of claim 1,

The display unit

When the observer is located at a point other than the photographing point in the three-dimensional space, a plurality of cost functions are set based on each of a plurality of vectors from the photographing points to the point where the observer is located, 3. A 3D map generation and display apparatus, wherein the 3D map is generated and displayed by finding a corresponding photographing point for minimizing a set cost function and using the 3D information acquired from the corresponding photographing point.
The method of claim 1,

The display unit

When the observer is located at a point other than the photographing point in the three-dimensional space, a plurality of cost functions are set based on the eye vector of the observer's gaze and each of the vectors from the photographing points to the observer. And find a corresponding photographing point to maximize the value of the set cost function, and generate and display the 3D map using the 3D information acquired from the corresponding photographing point. Device.
The method of claim 10,

The display unit

In the case where there are a plurality of corresponding photographing points for minimizing the value of the set cost function, the existing position among the three-dimensional information acquired at each of the plurality of photographing points based on the photographing time of the three-dimensional information acquired at each of the plurality of photographing points And selecting 3D information having a minimum time difference from the 3D information, and generating and displaying the 3D map using the selected 3D information.
The method of claim 10,

The display unit

When generating and displaying the 3D map, when the length of each of the vectors from the photographing points to the viewer is greater than a preset length, a 3D map corresponding to the entire 3D space based on the global coordinate system 3D map generation and display device, characterized in that by automatically generating and displaying, by switching the two three-dimensional map in accordance with the position of the observer.
The method of claim 1,

The display unit

When the obstacle includes the 3D information acquired at the corresponding photographing point, the 3D map is generated and displayed using the 3D information acquired at another photographing point closest to the corresponding photographing point regardless of the position of the observer. And the obstacles are not included in the 3D information acquired at the other photographing point.
The method of claim 1,

The display unit

3D, characterized in that for loading the three-dimensional information about the three-dimensional map that is stored in advance corresponding to the point where the observer is located, and displaying the three-dimensional map using the three-dimensional information each time the observer moves. Map generation and display device.
The method of claim 1,

The display unit

If the observer is located within a predetermined distance from one of the shooting points, the viewer plays the 3D map in the form of a moving picture, and the observer moves during the playing of the 3D map in the moving picture form and takes a predetermined distance from one of the shooting points. 3. The apparatus for generating and displaying 3D maps, further comprising: stopping the reproduction of the 3D map in the form of a video when located outside a distance.
The method of claim 16,

The display unit

When playing the 3D map in the form of a video,

A specific threshold distance value is set, and the 3D map generated using the 3D information acquired at the photographing points within the threshold distance value from the observer's position is repeatedly played in a time sequence. 3D map generation and display device.
The method of claim 16,

The display unit

When playing the 3D map in the form of a video,

Caused by the distance of the observer centered on the position of the observer and the radius of the critical distance value from the position of the observer from the position of the observer by searching in reverse chronological order based on the recording time of the closest shooting point from the position of the observer Set the farthest shooting point as the start of the interval repeat,

After searching in chronological order based on the shooting time of the shooting point nearest to the observer's position, and set the shooting point farthest from the observer's position as the interval repeat end point,

And sequentially playing from the three-dimensional map generated using the three-dimensional information photographed at the section repeat start point to the three-dimensional map generated using the three-dimensional information photographed at the section repeat end point. 3D map generation and display device.
Acquiring pose information including position and rotation information about photographing points from which three-dimensional information about the three-dimensional space is acquired based on the global coordinate system;

Generating a plurality of local coordinate systems with respect to the three-dimensional space according to at least one of spatial and temporal proximity of the photographing points based on the acquired pose information; And

When the observer of the virtual reality is located in the three-dimensional space, using the plurality of local coordinate systems to find the corresponding shooting point corresponding to the location where the observer is located, three-dimensional by using the three-dimensional information acquired from the corresponding shooting point Steps to create and display a map

3D map generation and display method comprising a.
The method of claim 19,

The generating of the plurality of local coordinate systems further includes selecting photographing points satisfying a preset condition among the photographing points.

At least one of the spatial adjacency and the temporal adjacency is calculated for the selected imaging points,

The preset condition is a condition for selecting photographing points obtained by obtaining more than a predetermined number of three-dimensional information, or a condition for selecting photographing points having a predetermined rank or more.
The method of claim 19,

Creating and displaying the 3D map

When the observer is located at a point other than the photographing point in the three-dimensional space, setting a long axis of a bounding volume having a specific shape to pass through the center of the observer and be parallel to the observer's line of sight ;

Finding a corresponding shooting point included in the bounding volume by enlarging or reducing the size of the bounding volume until a predetermined number of shooting points is included in the bounding volume; And

Generating and displaying the 3D map using the 3D information acquired at the corresponding photographing point

3D map generation and display method comprising a.
The method of claim 21,

Creating and displaying the 3D map

Axially rotating the bounding volume using a gaze vector of the observer's gaze, and generating and displaying the 3D map using 3D information acquired at a corresponding photographing point included in the bounding volume.

More,

The eye vector is

In the case of 3 degrees of freedom, such as HMD (Head Mounted Display), it includes a vector corresponding to each mode of roll, pitch, and yaw, and freedom, such as overhead view. In the case of the eye of FIG. 1, the 3D map generation and display method includes a vector corresponding to the long axis of the bounding volume.
The method of claim 19,

Creating and displaying the 3D map

When the observer is located at a point other than the photographing point in the three-dimensional space, setting a plurality of cost functions based on each of the plurality of vectors from the photographing points to the point where the observer is located;

Finding a corresponding photographing point to minimize the value of the set cost function; And

Generating and displaying the 3D map using the 3D information acquired at the corresponding photographing point

3D map generation and display method comprising a.