JP7138361B2 - User Pose Estimation Method and Apparatus Using 3D Virtual Space Model - Google Patents

Description

The technical field relates to the generation and utilization of spatial maps, and more particularly to a method and apparatus for estimating user poses in real space using a 3D virtual space model.

A method of estimating a user pose using a spatial map includes a method of using a terrain value (geometry), a method of using an image value, and a method of using a mixture of a terrain value and an image value.
At this time, LiDAR or a depth measuring device with a similar operating principle is used to acquire point cloud information, or a camera or a video measuring device with a similar operating principle is used to capture images. acquire information, acquire color-point cloud information using a Kinect or similar depth-to-image measurement device, or use a combination of these to A spatial map can be constructed to represent

The video information, depth information, and depth-video link information for the real space are called "spatial information."
The user pose is estimated by comparing the user information acquired by the user device in real space with the spatial map.

Here, “user information” is information including images acquired by the user device in the real space. Also, "pose" is a concept that includes both position and orientation. Therefore, it can be said that the “user pose” is information including position information in which the image information is acquired with respect to the real space and direction information in which the image information is acquired.
However, user pose estimation using a spatial map in the prior art has the following problems.

First, spatial maps can be sensitive to the pose from which spatial information is obtained. Therefore, if the spatial map is sensitive to the pose from which spatial information is obtained, the precision of user pose estimation will be degraded. For example, when spatial information is acquired for all theoretically possible poses to form a spatial map, highly accurate user poses can be estimated.

However, it is practically impossible to acquire spatial information for all poses in the real space. When the spatial information of a large number of poses is obtained from the real space with an even distribution to construct a spatial map, the sensitivity of user pose estimation is reduced by the distribution of the poses from which the spatial information is obtained. However, in such a case, system load problems such as acquisition time of spatial information, capacity of spatial information, and processing time may occur.

On the other hand, when spatial information is acquired with a small number of poses while considering the system load problem, the spatial map cannot sufficiently represent the real space. Furthermore, if the route for obtaining the spatial map changes, the reliability of the spatial map will decrease, and the real space will not be robustly represented. A spatial map that does not robustly represent the real space leads to a decrease in accuracy of user pose estimation.
Second, a discontinuous spatial map can reduce the accuracy of user pose estimation. FIG. 1 is a diagram showing an example of a discontinuous spatial map composed of point group information.

As shown in FIG. 1, when constructing a spatial map using point cloud information, it may not be possible to acquire point cloud information densely depending on the acquisition range and route of the spatial information. If point cloud information cannot be acquired densely, a spatial map without continuity is generated, which leads to a decrease in accuracy of user pose estimation.
Third, the accuracy of user pose estimation may decrease due to the difference between the time when the spatial information for constructing the spatial map is obtained and the time when the user information is obtained.

2 and 3 are illustrative diagrams showing changes in space over time.
FIG. 2 shows an example in which the light or illumination changes over time.
More specifically, (a), (b), and (c) of FIG. 2 show examples in which the amount of illumination and the amount of light entering from the outside changes over time in the same space.
Moreover, (a) and (b) of FIG. 3 show an example in which an object changes with the flow of time in the same space.

Although nothing is placed on the table 210 in FIG. 3(a), an object is placed on the table 220 in FIG. 3(b).
For example, for the space shown in FIG. 2, spatial information for constructing a spatial map may be obtained from (a) and user information may be obtained from (c). For the space shown in FIG. 3, spatial information for constructing a spatial map may be obtained from (a), and user information may be obtained from (b).

As described above, even in the same space, the video information may not match due to the difference between the time when the spatial information is acquired and the time when the user information is acquired. Therefore, even in the same space, the accuracy of user pose estimation is degraded due to the difference between the time when the spatial information is obtained and the time when the user information is obtained.
In the real space, changes in light or illumination, changes in dynamic movements of people, changes in objects or interiors, etc. occur with the passage of time. When using a spatial map that has not been updated with such changes, the similarity with user information is reduced, which leads to a reduction in accuracy of user pose estimation.
Therefore, there is a need for a method that solves the conventional problems encountered when estimating user pose based on spatial maps.

In order to solve the above-described problems, the present invention estimates a user pose by utilizing a three-dimensional virtual space model configured based on space information acquired in the real space and user information acquired by the user. A method and apparatus are provided.

A method for estimating a user pose for a 3D space according to an exemplary embodiment includes acquiring spatial information including depth information and image information for a 3D space using a depth measurement device and an image acquisition device; constructing depth-image association information based on the depth-image association information, and constructing a 3D virtual space model corresponding to the 3D space based on the depth-image association information; receiving user information including video; generating correspondence information corresponding to the user information in the three-dimensional virtual space model; calculating a similarity between the correspondence information and the user information; estimating a user pose based on the degrees.

The step of constructing the 3D virtual space model includes dividing a background region related to the structure of the 3D space and a non-background region corresponding to an object placed in the 3D space, using image information for the 3D space. and constructing the three-dimensional virtual space model using the background region.

Generating the correspondence information includes dividing a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in the image included in the user information. , processing the user information using a background area of an image included in the user information; and generating corresponding information corresponding to the user information processed in the 3D virtual space model. good.

The step of calculating the degree of similarity may include the steps of: regenerating the correspondence information in a direction to increase the degree of similarity; and recalculating the degree of similarity based on the regenerated correspondence information.

The step of calculating the similarity includes extracting a comparison target region for comparing the user information and the correspondence information, and a comparison target region extracted from the user information and a comparison target extracted from the correspondence information. The method may include determining a common area in the area, and respectively regenerating the user information and the corresponding information based on the common area.

Calculating the similarity may include obtaining additional user information about the user device surroundings, and calculating a similarity based on the user information and the additional user information.

In the step of estimating the user pose, when user additional information, which is additional information used for estimating the user pose by the user device, is acquired, the user information or the additional user information is Estimating the user pose using side information may be included.

Acquiring the additional user information may include transmitting guidance information to the user device for acquiring additional user information based on the 3D virtual space model.

The guidance information may include a user information acquisition pose for preset feature points in the 3D virtual space model, and the step of acquiring the additional user information may be repeatedly performed in a direction of increasing the similarity. .
A method for estimating a user pose including information on a user's position and orientation with respect to a 3D space according to another embodiment comprises receiving user information including an image captured in the 3D space; confirming a 3D virtual space model constructed based on space information including depth information and image information for the 3D space; generating correspondence information corresponding to the user information in the 3D virtual space model; calculating a similarity between the corresponding information and the user information; and estimating a user pose based on the similarity.

A user pose estimation apparatus for a 3D space according to an embodiment includes a spatial information acquisition unit that acquires spatial information including depth information and video information for a 3D space, and depth-video link information based on the spatial information. a virtual space model generation unit configured to generate a 3D virtual space model corresponding to the 3D space based on the depth-image linkage information; and user information including the video acquired by the user device in the 3D space. and generating correspondence information corresponding to the user information in the three-dimensional virtual space model, calculating a similarity between the correspondence information and the user information, and based on the similarity A control unit including at least one processor configured to estimate the user pose.

The space model generation unit divides image information for the 3D space into a background region related to the structure of the 3D space and a non-background region corresponding to an object placed in the 3D space, and divides the background region into may be used to construct the three-dimensional virtual space model.

The controller distinguishes between a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in the image included in the user information, The user information may be processed using the background area of the included image to generate corresponding information corresponding to the user information processed within the 3D virtual space model.

According to another embodiment, an apparatus for estimating a user pose including user position and orientation information in a 3D space is based on spatial information including depth information and image information in the 3D space. a virtual space model providing unit that provides a constructed 3D virtual space model; a user information receiving unit that receives user information including images acquired by a user device in the 3D space; A control comprising at least one processor configured to generate correspondence information corresponding to user information, calculate a similarity between the correspondence information and the user information, and estimate the user pose based on the similarity. have a department.

A user pose estimation apparatus for a three-dimensional space according to another embodiment includes a user information generating unit that generates user information including an image for a three-dimensional space, a user information that is transmitted to a user pose estimation server, and a three-dimensional virtual space. controlling operations of a communication unit for receiving information about the user pose estimated by the model from the server, the user information generating unit and the communication unit, and transmitting the information about the user pose to a currently executing application or driving system; a controller including at least one processor configured to:

By using a 3D virtual space model as a spatial map, the embodiment of the present invention can construct a robust 3D virtual space model for the acquisition path of spatial information, and perform user pose estimation precision based on the spatial information acquisition pose. degree sensitivity can be reduced.

In addition, the 3D virtual space model according to the embodiment of the present invention can be configured similarly to the real space, and can reduce the acquisition time of spatial information, the volume of spatial information, the processing time of data, and the like.
In addition, it is possible to provide a user pose estimation method that is robust against changes in the real space over time.
Also, embodiments of the present invention can be utilized when estimating user poses in mixed reality.
Furthermore, accurate user pose estimation can reduce discomfort between the real space and the virtual space, and increase the user's immersion in mixed reality. Therefore, embodiments of the present invention can contribute to the commercialization and development of mixed reality-related technologies.

FIG. 4 is a diagram showing an example of a discontinuous spatial map constructed using point group information; FIG. 4 is an exemplary diagram showing changes in space due to the passage of time; FIG. 10 is another illustrative diagram showing changes in space due to passage of time; FIG. 4 is a diagram showing an example of a three-dimensional virtual space model in one embodiment of the present invention; FIG. 4 is a diagram for explaining an example of generating a 3D virtual space model in one embodiment; 1 is a diagram for explaining a user pose estimation system using a 3D virtual space model in one embodiment; FIG. 1 is a diagram for explaining the configuration of a user pose estimation device for a three-dimensional space in one embodiment; FIG. 1 is a diagram for explaining the configuration of a user device in one embodiment; FIG. FIG. 10 is an exemplary diagram for explaining the concept of poses in one embodiment; 4 is a flow chart for explaining a user pose estimation method for a 3D space in one embodiment. 9 is a flowchart for explaining a method of estimating a user pose for a 3D space according to another embodiment; FIG. 10 is a diagram for explaining an example of a method for additionally acquiring a user pose in one embodiment;

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and the contents described in the accompanying drawings, but the present invention is not restricted or limited by the embodiments.

The terminology used herein is for the purpose of describing embodiments only and is not intended to be limiting of the invention. Singular forms used herein also include plural forms unless the context clearly dictates otherwise. As used herein, "comprises" and/or "comprising" means that a stated component, step, act, and/or element includes one or more other components, steps, It does not exclude operations and/or the presence or addition of elements.

As used herein, "embodiment,""example,""aspect,""exemplary," etc. are used to indicate that any aspect or design described is preferred or advantageous over other aspects or designs. should not be construed as
Also, the term "or" means an inclusive or rather than an exclusive or. That is, unless stated otherwise or clear from context, the phrase "x utilizes a or b" means any one of the natural inclusive permutations.

Also, terms such as first and second used in the specification and claims are used to describe various components, but the components are not limited by the terms. must not. The terms are only used to distinguish one component from another.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the meaning commonly understood by one of ordinary skill in the art to which this invention belongs. will be used as Also, commonly used pre-defined terms should not be idealized or overly interpreted unless explicitly specifically defined.

In describing the present invention, if it is determined that a specific description of related known functions or configurations may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Further, the terminology used herein is a term to adequately describe the embodiments of the present invention, which may be subject to user, operator intent, or conventions in the field to which the present invention pertains. may differ depending on Therefore, terms used herein should be defined based on the general content of the specification.

FIG. 4 is a diagram showing an example of a three-dimensional virtual space model in one embodiment of the present invention.
FIG. 5 is a diagram for explaining an example of generating a 3D virtual space model in one embodiment.
Referring to FIG. 4, a typical 3D virtual space model refers to a model for a real space constructed using depth-image link information such as obj, x3d, and the like. For example, the 3D virtual space model is generated by Korean Patent No. 10-1835434 (title of invention: projection image generation method and apparatus, mapping method between image pixels and depth values), such as obj, x3d, etc. model or "TeeVR model".

At this time, as shown in FIG. 5, the 3D virtual space model according to an embodiment is composed only of the background region by dividing the background region and the non-background region to construct the 3D virtual space model. good.
In FIG. 5, (a) is the image information included in the spatial information, (b) is the image excluding the non-background area, and (c) is an example of generating the image information by expanding the background area. showing.
For example, the background area may refer to the structure of the building itself that forms a three-dimensional space, or may be a structure attached to the building such as a door or window. Therefore, in the image information, the background area may be defined as an area associated with the structure of the 3D space.

In FIG. 5(a), areas corresponding to various objects (desks, bookshelves, chairs, etc.) located in the indoor space correspond to non-background areas. FIG. 5(b) shows a non-background area from which various objects (areas displayed in white) located in the indoor space are removed.
The 3D virtual space model is a concept that includes both indoor and outdoor spaces, and may be an independent indoor space, an independent outdoor space, or a space in which the indoor and outdoor spaces are connected. A model such as obj, x3d, etc. (person, object, etc.) may be added to the 3D virtual space model, and the 3D virtual space model is a concept including the 3D virtual space model to which the model is added. good too. On the other hand, it is also possible to use a 2D virtual space model by lowering the dimension of the space map instead of the 3D space map.

The three-dimensional space model may use a pre-constructed model such as obj or x3d, may be constructed by newly acquiring spatial information, or may be constructed by updating a pre-constructed model. can be used. A three-dimensional space model may be determined to be similar to real space.

FIG. 6 is a diagram for explaining a user pose estimation system using a 3D virtual space model in one embodiment.
Referring to FIG. 6, a user pose estimation system using a 3D virtual space model comprises a user device 610 and a user pose estimation device 620. As shown in FIG. User pose estimator 620 according to an embodiment may be provided in a server (not shown) or may be provided in user device 610 .

User device 610 may acquire user information 611 in physical space 601 and transmit user information 611 to user device 610 .
The user pose estimation device 620 may estimate the user pose using the 3D virtual space model 630 and the user information 611 recorded in the storage system 602 inside or outside the device.

The user pose estimation device 620 can accurately estimate the user pose by comparing the correspondence information 621 that has a high probability of corresponding to the user pose in the three-dimensional virtual space model 630 with the user information 611 .

FIG. 7 is a diagram for explaining the configuration of a user pose estimation device for three-dimensional space in one embodiment.
Referring to FIG. 7, a user pose estimation apparatus 620 for 3D space according to an embodiment includes a virtual space model provider 730, a controller 740, and a user information receiver 750. FIG. Also, the user pose estimation device 620 may further include a spatial information acquisition unit 710 and a virtual space model generation unit 720 . Furthermore, the user pose estimation device 620 may further comprise a user information requester 760 .
The spatial information acquisition unit 710 acquires spatial information including depth information and image information regarding a three-dimensional space. For example, spatial information may be obtained using a depth measurement device and a video measurement device.

A path through which the field of view (FoV) of a measurement device, such as a depth measurement device or a video measurement device, that acquires spatial information for constructing a 3D virtual space model can secure the real space. If spatial information is acquired with , the 3D virtual space model will be configured to resemble the real space, and the acquisition time of spatial information, the amount of spatial information, and the processing time of data can be reduced. Done and efficient.

The image information may be in a form that can be expressed as a two-dimensional image in a three-dimensional space with two-degree-of-freedom basis vectors. It may be in a form in which an infrared filter is attached to express the three-dimensional string information two-dimensionally.

Depth information is in the form of points that can be represented by basis vectors with 3 degrees of freedom, and may be obtained using a depth measuring device using two or more images taken at different locations. may be estimated. Examples of the former include depth information obtained using LiDAR, SONAR, infrared (InfraRed), and TOF (Time Of Flight) rangefinders, while examples of the latter include stereo There are depth information acquired using cameras, multi-cameras, omnidirectional stereo cameras, etc. On the other hand, if a device such as Kinect, JUMP, PrimeSense, Project Beyond is used, depth information and image information can be obtained at the same time.

For example, in one embodiment of the present invention, in addition to depth information obtained using a depth measuring device, new depth information may be estimated and used by interpolation. More specifically, three or more pieces of depth information are selected from a plurality of acquired pieces of depth information to form a polygonal (including triangular) mesh (Mesh). New depth information is estimated and added by interpolation.

Alternatively, the acquired depth information and image information according to an embodiment of the present invention may be acquired simultaneously using an integrated sensor system. When using multiple measurement devices, a calibration process may be required to determine the coordinate relationships between the sensors.

An inertial measurement unit (IMU) or the like may be additionally used in the process of acquiring spatial information, and distance information (odometry) may be used when sensors are mounted on a tire-type mobile robot for measurement. If the physical space is wider than the viewing angle of the measurement device, the sensor may be rotated, moved, or a combination thereof to obtain spatial information. At this time, the three-dimensional poses from which the individual spatial information is acquired may be different. (VIO: Visual Inertial Odometry), visual odometry (VO: Visual Odometry), and other methods may be utilized.

On the other hand, the configuration of the spatial information may differ depending on the type of measurement device. For example, if the measurement device consists of only a single camera, the pre-measurement information consists of camera image information, and if it is a single camera, the relative distance between pixels is calculated using the corresponding image information. , it is possible to estimate the absolute distance between pixels in the case of multiple cameras. In particular, in the case of a single camera without extracting feature points, the pixel depth can be predicted by utilizing the accumulated image information. Pixel depth can also be predicted using image information.

Furthermore, when information such as additional depth information, inertial information, etc. are utilized together, spatial information processing can be tailored to the unique characteristics of each measurement device. For example, if inertial information can be acquired by an inertial measurement device, it can be used to improve the performance of SRAM, or can be used as prediction information for image acquisition poses during image information processing. can be made more easily. In addition, the acceleration value or angular velocity value of inertial information can be used to estimate the actual moving distance, which is corrected for the scale of depth information extracted from a single camera or multiple cameras. can also be used for

The virtual space model generator 720 constructs depth-image linkage information based on spatial information, and generates a 3D virtual space model corresponding to the 3D space based on the depth-image linkage information.

For example, when generating a 3D virtual space model of an indoor space, the spatial information acquisition unit 710 may acquire an indoor space image, which is spatial information for the indoor space. At this time, the indoor space images may be images captured at various positions inside the indoor space.

At this time, the virtual space model generation unit 720 divides the background region, which is the region corresponding to the structure of the indoor space, and the non-background region, which is the region corresponding to the object located in the indoor space or the moving person. good.
The virtual space model generation unit 720 may divide the background region and the non-background region based on the pixel values of the images forming the indoor space image.

The background area is not complete data because it is partly hidden by other elements, but it is inferred that it has similarity to the non-hidden part, and the non-hidden part is used as hole filling or inpayment. It may correspond to a part that can be reconstructed by an inpainting technique. In addition, the background area may hide other objects such as large billboards and information desks inside the building, but the edges of the objects in question match the image and terrain in all data. or can be a portion that can be matched by another matching process.

The virtual space model generating unit 720 may generate at least one extended indoor space image by affixing a background region to a non-background region in the indoor space image. For example, in (b) of FIG. 5, the portion expressed in white from which the non-background area has been removed may be confirmed as the background area.

The virtual space model generation unit 720 determines that when an edge included in the background area is cut off at the boundary between the background area and the non-background area, the extended line of the edge crosses the boundary between the background area and the non-background area and enters the non-background area. Based on the connection inference, an augmented image may be generated.

At this time, one or more indoor space images are designated as background complementary images in addition to the specific indoor space image, and the information of the background complementary image is used for the area corresponding to the non-background area of the specific indoor space image. can be reduced by

The virtual space model generation unit 720 may generate depth-image linkage information based on at least one extended indoor space image and terrain information including depth value information about the indoor space. The depth-image link information may be information in which the depth value of the indoor space corresponding to at least one pixel of the extended indoor space image is matched.

The virtual space model generation unit 720 generates an image including information on acquisition positions and acquisition angles of at least one extended indoor space image and terrain information, as well as at least one extended indoor space image and terrain information. The acquisition pose and the depth acquisition pose may be further utilized to generate depth-image association information.

The virtual space model generation unit 720 generates a 3D virtual space model for a real 3D space using at least one extended indoor space image, terrain information, and depth-image linkage information.
The virtual space model providing unit 730 provides a 3D virtual space model constructed based on space information including depth information and image information regarding a 3D space when user pose estimation is required.

At this time, the user pose estimation may be performed after executing an application installed on the user device 610 or the user pose estimation device 620 . The virtual space model providing unit 730 may provide the 3D virtual space model to an application running on the user device 610 or the user pose estimation device 620 or a driving system of the corresponding device.

Controller 740 may include at least one processor. At this time, the control unit 740 may be connected to at least one computer-readable storage medium (one or more computer-readable storage media) in which instructions or programs are recorded.

Therefore, the control unit 740 generates correspondence information corresponding to the user information in the 3D virtual space model, calculates the similarity between the correspondence information and the user information, and estimates the user pose based on the similarity. configured with at least one processor.

User pose estimation according to an embodiment may be performed by learning a 3D virtual space model using deep learning or a neural network.

Learning may be divided into reinforcement learning, supervised learning, and unsupervised learning according to the form of the learning problem. A huge amount of training data (training test) may be required in the learning stage, and the training data consists of data including image information and data including poses from which the data is obtained. In order to increase the amount of training data, the two types of data may be modified by adding noise. Convolutional Neural Networks (CNN) or various neural networks, in whole or in part, may be used. For deep learning performance or speed enhancement, one or more GPUs may be used and parallel computation may be performed. The result of deep learning may be expressed by a scalar, vector, probability, etc. By using this result, the user pose expected as the pose from which the user information was acquired may be estimated. Depending on the input, the video information of the user information may be used, and the user additional information may be used together. When using user additional information, layers may be added to the neural network, functions may be changed, the number of parameters may be adjusted, and their values may be changed. A computer language such as Python, C language, MATLAB, or a combination thereof may be used to construct the neural network.

When user information is sequentially acquired, a particle filter, EKF. Techniques such as EIF, UKF, etc. may be utilized to estimate the user pose. If inertial information or distance information is acquired as user additional information, the estimated user pose may be corrected. A value of the particle filter may be converged as a specific pose according to the sequentially acquired user information, and the converged point may be estimated as the user pose. A weight may be added when estimating the user pose, and the user pose may be determined from a number of convergent poses.

A user pose may be estimated by fusing a pose estimated by deep learning and a pose estimated by a particle filter or the like. For example, a particle filter may be run around the pose estimated by deep learning to estimate the user pose, or conversely, deep learning may be used around the pose converged by the particle filter to estimate the user pose. You can A weight may be added when estimating the user pose, and the user pose may be determined from a number of convergent poses.

The degree of similarity means the degree of similarity between the corresponding information generated by the 3D virtual space model and the user information. The pose of the 3D virtual space model for which the correspondence information is generated may be estimated as the user pose for which the user information is acquired. The similarity may be represented by a scalar, vector, covariance matrix, etc., and may be represented by euclidean distance, Manhattan distance, mahalanobis distance, structure structural similarity (SSIM), normalized information distance (NID), minimum mean square error (MMSE), entropy, and the like.

Similarity calculation and user pose estimation will be described in more detail with reference to FIG.

At this time, the 3D virtual space model divides the background area related to the structure of the 3D space and the non-background area corresponding to the object placed in the 3D space in the image information for the 3D space. It may be constructed.

The user information receiver 750 receives user information including images captured by the user device in a 3D space.
The user information is information including image information, includes one or more image measurement devices, and may be obtained using a depth measurement device or an additional device. If the viewing angle of the measurement device is too narrow to obtain sufficient user information, the user information may be obtained by rotating, moving, or combining the measurement device. The user information may be acquired by single or multiple video sensors (cameras) and may be acquired in pin-hole model, fisheye, or panoramic format. Single video information, multiple video information, or permutations of video information may be obtained. Image information, depth information, depth-image linkage information, or the like may be configured using the acquired user information.

For example, image information can be obtained using a single image measurement device, and depth information can be calculated using the sequentially obtained image information, thereby forming depth-image link information. can do.
For example, if multiple image measuring devices are used, depth information can be calculated by utilizing the relationship between the image information acquired by each image measuring device and the image measuring device. Information can be configured. The relationship with the image measuring devices may be calibration information between image measuring devices or conversion information (homography matrix) between image information obtained by each image measuring device.

For example, when using at least one or more image measurement devices and at least one or more depth measurement devices, depth-image linkage information may be constructed using calibration information between the two devices. Depth information may be extracted from image information using deep learning. A neural network may be constructed and a convolutional neural network may be used. A large amount of data may be required for training and testing, neural networks may consist of linear functions, non-linear functions, many layers, etc., and deep learning results may be probabilistic, scalar, vector, etc. may be expressed. Iterative learning may be performed and may require parameter tuning. Depth-image link information may be configured using depth information extracted by deep learning. Processed image information may be used to process image information. For example, operations such as changing the brightness or saturation of an image or converting a panoramic image into a rectified image are performed. you can

The user information requesting unit 760 may send guidance information to the user device 610 when it is necessary to obtain additional user information. Guidance information will be described in detail with reference to FIG.

FIG. 8 is a diagram for explaining the configuration of a user device in one embodiment.
Referring to FIG. 8, the user device 610 comprises a user information generation section 810, a communication section 820 and a control section 830. FIG. User device 610 may further comprise a user interface portion 840 that includes a display, input means, and output means for interfacing with a user.

The user information generator 810 generates user information including an image for a 3D space. Therefore, the user information generator 810 may include at least one of the image measuring device and the depth measuring device.

The communication unit 820 transmits user information to the user pose estimation server and receives information about the user pose estimated by the 3D virtual space model from the server.
At this time, the user pose estimation server may be the user pose estimation device 620 shown in FIG. 7 or another server that provides the user pose estimation service.

Control unit 830 includes at least one processor configured to control the operation of user information generator 810 and communication unit 820 and to communicate information regarding user poses to a currently executing application or driving system.
FIG. 9 is an exemplary diagram for explaining the concept of poses in one embodiment.
Spatial information used to construct a 3D virtual space model may be considered as discontinuous information obtained in partial poses in real space. Here, pose is a concept that includes both position and orientation. As an example, in two dimensions, the pose may be represented by x, y, the position of the measuring device, and the angle a of the measuring device.

The example shown in FIG. 9 is a square plane with a width and length of 1 m, the measuring device moves in the range of 0 to 1 m with respect to the x-axis and the y-axis at intervals of 10 cm, and the rotation angle is 0 to 360 degrees. shows an example of rotation by 10 degrees in the range of .
At this time, the number of possible global poses is 11×11×37, that is, 4,477 types. Similarly, in three dimensions, the pose may be represented by the position of the sensor x, y, z and the angles roll, pitch and yaw of the measurement device.

In a regular hexahedral space of 1 m in width, length and height, the sensor moves in the range of 0 to 1 m with respect to the x, y and z axes at 10 cm intervals, and the rotation angle is in the range of 0 to 360 degrees. , the number of possible global poses is 11.times.11.times.11.times.37.times.37.times.19, ie, there are about 34 million kinds of cases.

Discontinuous information can be made to look like continuous information by reducing the interval of movement and angle of rotation of the measuring device, but the number of possible poses should increase exponentially, and the volume of real space becomes much larger than 1 m ³ , it is practically impossible to acquire spatial information in all possible poses.

For this reason, in the step of acquiring spatial information, data is acquired in some poses that can sufficiently include the real space, and depth-video link information is constructed based on the acquired spatial information. By constructing a three-dimensional virtual space model by doing so, it is possible to expand the spatial information acquired in some poses.

The 3D virtual space model may be configured based on the spatial information acquired in some poses, but in order to utilize the depth-image linkage information configured from the spatial information, Any pose can generate corresponding information that is similar to spatial information or constructed depth-to-video linkage information acquired at the pose in the same physical space.

That is, even for a pose for which spatial information is not acquired, if the 3D virtual space model is used, it is possible to generate depth-video link information or corresponding information similar to spatial information in the pose. The generated correspondence information may be considered similar to the depth-image linkage information composed of the spatial information acquired at the same pose in the physical space. The three-dimensional virtual space model can convert spatial information, which is discontinuous information, into corresponding information, which is continuous information.

FIG. 10 is a flowchart for explaining a user pose estimation method for 3D space in one embodiment.
The method shown in FIG. 10 may be performed by the user pose estimator 620 shown in FIG.
In step S1010, the apparatus acquires spatial information including depth information and image information for a 3D space using a depth measurement device and an image acquisition device.
In step S1020, the apparatus constructs depth-image association information based on the spatial information, and builds a 3D virtual space model corresponding to the 3D space based on the depth-image association information.
At step S1030, the apparatus receives user information including images captured by the user device in 3D space. At this time, the user information may further include spatial depth information corresponding to the captured image.
At step S1040, the device generates corresponding information corresponding to the user information within the 3D virtual space model.

By using a 3D virtual space model, it is possible to generate correspondence information similar to spatial information or depth-video link information acquired in a pose for which no spatial information is acquired.

Correspondence information may be represented by depth information, image information, or depth-image link information. Corresponding information may be generated with poses expressed as basis vectors with three degrees of freedom in the three-dimensional virtual space model.

For example, if the height of the user information acquisition pose does not change, the corresponding information may be generated with a pose represented by basis vectors with two degrees of freedom within the three-dimensional virtual space model. Corresponding information may be generated through processes such as viewing angle, image information conversion, and depth information conversion.

At this time, the step of generating correspondence information (S1040) distinguishes between the background area related to the structure of the 3D space and the non-background area corresponding to the object placed in the 3D space in the image included in the user information. processing the user information using a background area of an image included in the user information; and generating corresponding information corresponding to the processed user information in the three-dimensional virtual space model. good.

The real space when the user acquires user information in the real space that is the background of the 3D virtual space model may not be the same as the time when the space information was acquired to construct the 3D virtual space model. , objects, interiors, etc. may change.

Therefore, the user information may be divided into a background portion and a non-background portion, and the non-background portion may be removed from the user information, or the user information may be converted using the background portion. User information may be manipulated and used to remove effects due to lighting, light, and the like. In the process of comparing the user information with the corresponding information generated by the 3D space model, the form of the user information or the corresponding information may be converted and compared.

At step S1050, the device calculates the similarity between the correspondence information and the user information.
At this time, calculating the similarity may include regenerating the correspondence information in a direction of increasing the similarity and recalculating the similarity based on the regenerated correspondence information. At this time, ways to increase the degree of similarity include reacquiring user information, regenerating correspondence information corresponding to user information, and using additional information in addition to user information.

In order to increase the degree of similarity, the step of calculating the degree of similarity S1050 is the step of extracting a comparison target region for comparing the user information and the correspondence information. A step of determining a common region in the comparison target region, and a step of respectively regenerating the user information and the corresponding information based on the common region may be included.

For example, the correspondence information used in the comparison process is removed from the correspondence information according to a predetermined criterion, such as a distorted area due to structural simplification, and by removing the area corresponding to the non-background portion of the user information. can be regenerated. In addition, the user information used in the comparison process may be regenerated by removing the non-background portion as well as the area corresponding to the distorted area of the corresponding information.
The method of calculating the similarity between the correspondence information generated by the 3D virtual space model and the user information obtained by the user includes a method of comparing video information of the correspondence information and video information of the user information, and a depth of the correspondence information. A method of comparing information and depth information of user information, a method of comparing depth-video linkage information, or the like may be used.

At this time, since the size (scale) of the correspondence information and the user information may differ, normalization may be required or relative comparison may be required.
On the other hand, comparison of video information may require conversion of the video information to make the format of each video information similar. For example, there may be conversion between a Panorama Image and a Rectified Image, normalization of the size of image information, and conversion of a viewing angle.

Conversely, it is also possible to transform a stationary image into a panoramic format and use it. Feature points of the image information may be found from the two pieces of image information using techniques such as RANSAC, SIFT, FAST, SURF, or a combination thereof, and pairs of similar feature points may be connected. Feature points may be edges, straight lines, line segments, corners, circles, ellipses, etc., or combinations thereof, and may have different scales, rotations, and the like. The similarity of image information may be calculated by a method such as Feature Matching, SSIM (Structural Similarity), NID (Normalized Information Distance), Homography Matrix.

A number of pixel coordinates connected by feature point matching may be used to calculate a homography matrix, which may be used to calculate the difference (error) between two image information. SSIM is a method of calculating similarity between two images, and NID is a probabilistic calculation method.

If the depth information can be extracted from the user information, the degree of similarity between the corresponding information and the depth information may be compared. Depth information may be expressed as 3D point cloud information (PCD), depth map, mesh, etc., and requires a process of unifying the two depth information formats. may be Depth information may be compared pixel-to-pixel (per point) and may be compared considering the surrounding area. Depth information may be newly estimated and compared by interpolation, or may be calculated by adding a weighted value.

If the depth-video linkage information can be configured by the user information, the corresponding information may be compared with the depth-video linkage information. Each degree of similarity may be calculated by comparing the depth information and the image information, and the overall degree of similarity may be calculated. Further, the depth-image linkage information may be compared in a composite manner, and a method of calculating the similarity of the depth information and the image information may be combined and executed.

Since the point in time when the spatial information for constructing the 3D virtual space model is acquired may be different from the point in time when the user information is acquired, the correspondence information and the user information may differ even for the same pose. be. Therefore, robust feature points may be compared between the correspondence information and the user information. For example, a background portion and a non-background portion may be separated from the corresponding information and the user information, and the similarity may be calculated using the background portion. may be generated to calculate the degree of similarity with the background portion of the user information. The degree of similarity may be calculated by removing the light source effect for illumination or light from the correspondence information and the user information, or the degree of similarity may be calculated by comparing features that are robust against the light source effect.

At this time, calculating the similarity S1050 may include obtaining additional user information about the user device surroundings and calculating the similarity based on the user information and the additional user information. Guidance information may be utilized as shown in FIG. 12 to obtain additional user information.

In step S1060, the apparatus identifies candidate correspondence information whose similarity is greater than or equal to a preset value, and estimates a pose matching the candidate correspondence information as a user pose.

As the degree of similarity increases, it may be considered that the pose of the 3D virtual space model for which the corresponding information is generated is the same as the pose for which the user information is acquired. Alternatively, if the similarity is higher than a reference value (threshold), it may be assumed that the poses reconstructed by acquiring two pieces of data are substantially the same, and the reference value may differ depending on the environment in the real space. Alternatively, a pose selected by determining which pose has the highest similarity to the user pose among corresponding information generated from a large number of candidate poses may be considered as the user pose.

The step of generating correspondence information and calculating the degree of similarity may be performed once to estimate the user pose, or may be performed repeatedly. The iterative run may re-estimate precisely around the chosen pose, re-estimate randomly for the entire region, and re-estimate new poses with added weights. may be selected. Such steps may be repeated a predetermined number of times, and may be repeated until the similarity is greater than or equal to a reference value or until the repeated and estimated poses converge. Optimization techniques may be used to increase the similarity.

The correspondence information may be regenerated so as to increase the similarity, and the regenerated correspondence information is regenerated with a pose expected as the user pose based on the relationship between the pose for which the existing correspondence information was generated and the similarity. may be After regenerating the correspondence information, the similarity is calculated, and if necessary, the process of regenerating the correspondence information and calculating the similarity may be repeated.

Using user-additional information such as inertial and range information increases the similarity, so corresponding information may be generated and regenerated at the expected pose. After that, the similarity between the corresponding information and the user information may be calculated, and if necessary, the user additional information may be used to regenerate the corresponding information, and the similarity calculation process may be repeated.

At this time, the user additional information is information supporting user pose estimation in addition to the image information obtained by the user, and may include inertial information (IMU), distance information (odometry), and the like. For example, when inertial information can be acquired by an inertial measurement device, it is used as prediction information for an image acquisition pose when processing image information, thereby facilitating correction of an image acquisition pose. be able to.

Therefore, in step S1050 of calculating the similarity or in step S1060 of estimating the user pose, when user additional information, which is additional information used for estimating the user pose, is acquired by the user device, The method may include estimating the user pose using the additional user information together with the additional user information.

At this time, the acceleration value or angular velocity value of the inertial information may be used to estimate the actual moving distance, and the scale of the depth information extracted from the single or multiple image measurement devices may be corrected. You can use it to

The distance information may be distance information predicted using VO (Visual Odometry) and VIO (Visual Inertial Odometry) constructed based on image information obtained by the user, and a measuring device is mounted on the tire-type mobile robot. When the user information is acquired by the mobile robot, the distance information may be the distance information of the mobile robot. Accordingly, if the inertial information is utilized, it becomes possible to utilize it for correcting the distance information extracted by the above method.

When a sensor is attached to a tire type mobile robot instead of a user to acquire user information, the user may operate the mobile robot or the mobile robot may run autonomously. may be obtained. A mobile robot pose can be considered as a user pose, and if the coordinate transformation relationship between the mobile robot and the user's field of view is recognized or coordinate transformation is possible, the mobile robot pose can be transformed into the user pose. .

The mobile robot may acquire not only user information including images but also distance information (odometry) of the mobile robot as user additional information. Distance information may be utilized to correct user pose. A relative expected pose of the mobile robot may be predicted using the sequentially obtained distance information, and information such as a covariance matrix may be calculated using methods such as EKF, EIF, UKF, or similar methods. This information may be updated to correct the user pose.
When using a mobile robot, related algorithms such as mobile robot motion, driving, steering, locomotion, data acquisition, recording, and processing may be executed on a robot operating system (ROS).

Spatial information, depth-video link information, 3D virtual space model, user information, user additional information, etc. may be recorded and processed in an external server.
A three-dimensional virtual space model may be constructed by constructing depth-image link information simultaneously with acquisition of spatial information, and a user pose may be estimated in real time at the same time with acquisition of user information, causing latency. , or may be processed after acquisition of the user pose is complete.

As long as a three-dimensional virtual space model is constructed, it is not necessary to acquire additional spatial information, and additional spatial information may be acquired for a part of the space. The constructed 3D virtual space model may be used when additional spatial information is not obtained, and part or the entire constructed 3D virtual space model is updated when additional spatial information is obtained. can be rebuilt and used.

The user information may be acquired first and then the space information may be acquired to construct the 3D virtual space model to estimate the user pose. User information may be obtained to estimate a user pose.
The present invention may be implemented in a system in which a sensor system and computer are fused, or may be implemented in an independent sensor system and computer.

When acquiring user information, the pose of each measuring device may differ from the pose of the entire user sensor system, but transformation is possible using the coordinate transformation relationship between each measuring device and the sensor system. For example, the center or appropriate position of the user sensor system may be assumed as the user pose, or the user pose may be assumed relative to the user sensor system. In this case, one may either know the required calibration information or the relative pose from the user sensor system to the user pose, or assume some value.

FIG. 11 is a flow chart for explaining a user pose estimation method for a 3D space in another embodiment.
The method shown in FIG. 11 may be performed by the user pose estimator 620 shown in FIG.
At step 1110, the device receives user information including images captured in a 3D space.
In step 1120, the apparatus confirms a 3D virtual space model constructed based on spatial information including depth information and image information for the 3D space. At this time, the 3D virtual space model may be provided by the virtual space model providing unit 730 of FIG.

At step 1130, the device generates corresponding information corresponding to the user information within the 3D virtual space model.
At step S1140, the device calculates the similarity between the corresponding information and the user information.
At step S1150, the device estimates the user pose based on the similarity. At this time, the user pose may be, for example, the pose of the corresponding information that has the highest degree of similarity with the user information.

FIG. 12 is a diagram for explaining an example of an additional user pose acquisition method in one embodiment.
Additional user information may be acquired to improve the similarity, the 3D virtual space model may be utilized to guide the user with additional user information acquisition poses, and the user may perform additional user information acquisition poses with the guided poses. User information may be obtained.

Therefore, in the description of FIG. 10, obtaining additional user information may include transmitting guidance information for obtaining additional user information to the user device 610 based on the 3D virtual space model.
At this time, the guidance information may include user information acquisition poses for preset feature points in the 3D virtual space model, and the step of acquiring additional user information may be repeatedly performed in the direction of increasing the degree of similarity.

For example, as shown in FIG. 12, in the case of a long corridor with many similar environments, additional user information acquisition poses may be guided in consideration of feature points in the 3D virtual space model.
In FIG. 12, the additional user information acquisition pose is a pose for sequentially acquiring images for feature points 1, 2, and 3, or a pose for any one of feature points 1, 2, and 3. you can

The apparatus described above may be realized by hardware components, software components, and/or a combination of hardware and software components. For example, the devices and components described in the embodiments may include, for example, processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microcontrollers, It may be implemented using one or more general purpose or special purpose computers, such as a processor or various devices capable of executing instructions and responding to instructions. A processing unit may run an operating system (OS) and one or more software applications that run on the OS. The processing unit may also access, store, manipulate, process, and generate data in response to executing software. For convenience of understanding, one processing device may be described as being used, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. You can understand. For example, a processing unit may include multiple processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include computer programs, code, instructions, or a combination of one or more of these, to configure a processor to operate at its discretion or to independently or collectively instruct a processor. You can Software and/or data may be any kind of machine, component, physical device, virtual device, computer storage medium or device, or to be interpreted on or to provide instructions or data to a processing device. It may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed over computer systems connected by a network so that they are stored and executed in a distributed fashion. Software and data may be stored on one or more computer-readable media.

The method according to the embodiments may be embodied in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable media may include program instructions, data files, data structures, etc. singly or in combination. The program instructions recorded on the medium may be those specially designed and constructed for an embodiment, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. , and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that is executed by a computer, such as using an interpreter, as well as machine language code, such as that generated by a compiler. The hardware devices described above may be configured to act as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations based on the above description. For example, the techniques described may be performed in a different order than in the manner described and/or components such as systems, structures, devices, circuits, etc. described may be performed in a manner different from the manner described. Appropriate results may be achieved when combined or combined, opposed or substituted by other elements or equivalents.
Accordingly, different embodiments that are equivalent to the claims should still fall within the scope of the appended claims.

Claims (25)

acquiring spatial information including depth information and image information for a 3D space using a depth measuring device and an image acquiring device;
constructing depth-image interaction information based on the spatial information, and constructing a 3D virtual space model corresponding to the 3D space based on the depth-image interaction information;
receiving user information including images captured by a user device in the three-dimensional space;
generating correspondence information corresponding to the user information in the three-dimensional virtual space model;
calculating a similarity between the corresponding information and the user information; and estimating a user pose including position information and direction information of the user device acquiring an image in the 3D space based on the similarity. A user pose estimation method for a three-dimensional space, comprising: The step of constructing the three-dimensional virtual space model includes:
A background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space are separated from image information for the 3D space, and the background area is used for the 3D space. characterized by constructing a virtual space model,
A user pose estimation method for a three-dimensional space according to claim 1. The step of generating the correspondence information includes:
separating a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in the image included in the user information;
processing the user information using a background area of an image included in the user information; and generating corresponding information corresponding to the processed user information in the three-dimensional virtual space model. A user pose estimation method for a three-dimensional space according to Item 1. The step of calculating the degree of similarity includes:
The user pose estimation method for a 3D space according to claim 1, comprising the steps of: regenerating the correspondence information in the direction of increasing the similarity; and recalculating the similarity based on the regenerated correspondence information. . The step of calculating the degree of similarity includes:
extracting a comparison target region for comparing the user information and the corresponding information;
determining a common area from the comparison area extracted from the user information and the comparison area extracted from the correspondence information; and regenerating the user information and the correspondence information based on the common area. 2. The method of estimating a user pose for a three-dimensional space according to claim 1, comprising: The step of calculating the degree of similarity includes:
2. The method of claim 1, comprising: obtaining additional user information about the user device surroundings; and calculating a similarity based on the user information and the additional user information. Estimating the user pose includes:
When user additional information, which is additional information used for estimating the user pose, is acquired by the user device, the user pose is obtained by using the user additional information together with the user information or the additional user information. including estimating
A user pose estimation method for a three-dimensional space according to claim 6. Obtaining the additional user information includes:
transmitting guidance information for obtaining additional user information to the user device based on the three-dimensional virtual space model;
A user pose estimation method for a three-dimensional space according to claim 6. The guidance information includes user information acquisition poses for preset feature points in the three-dimensional virtual space model,
The step of acquiring the additional user information is repeatedly performed in the direction of increasing the similarity,
A user pose estimation method for a three-dimensional space according to claim 8. A method of estimating a user pose including position and orientation information of a user device with respect to a three-dimensional space, comprising:
receiving user information including an image captured in the three-dimensional space;
confirming a 3D virtual space model constructed based on spatial information including depth information and image information for the 3D space;
generating correspondence information corresponding to the user information in the three-dimensional virtual space model;
A user pose estimation method for a three-dimensional space, comprising: calculating a similarity between the corresponding information and the user information; and estimating a user pose based on the similarity. The 3D virtual space model divides a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in image information for the 3D space, and characterized by being constructed using the area,
A user pose estimation method for a three-dimensional space according to claim 10. The step of generating the correspondence information includes:
separating a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in the image included in the user information;
processing the user information using a background area of an image included in the user information; and generating corresponding information corresponding to the processed user information in the three-dimensional virtual space model. Item 11. User pose estimation method for three-dimensional space according to item 10. The step of calculating the degree of similarity includes:
11. The user pose estimation method for a 3D space according to claim 10, comprising the steps of: regenerating the correspondence information in a direction of increasing the similarity; and recalculating the similarity based on the regenerated correspondence information. . The step of calculating the degree of similarity includes:
extracting a comparison target region for comparing the user information and the corresponding information;
determining a common area from the comparison area extracted from the user information and the comparison area extracted from the correspondence information; and regenerating the user information and the correspondence information based on the common area. 11. A user pose estimation method for a three-dimensional space according to claim 10, comprising: The step of calculating the degree of similarity includes:
11. The user pose estimation method for a three-dimensional space according to claim 10, comprising: obtaining additional user information about a user device surroundings; and calculating a similarity based on the user information and the additional user information. Estimating the user pose includes:
When user additional information, which is additional information used for estimating the user pose, is acquired by the user device, the user pose is obtained by using the user additional information together with the user information or the additional user information. including estimating
A user pose estimation method for a three-dimensional space according to claim 15. Obtaining the additional user information includes:
transmitting guidance information for obtaining additional user information to the user device based on the three-dimensional virtual space model;
A user pose estimation method for a three-dimensional space according to claim 15. The guidance information includes user information acquisition poses for preset feature points in the three-dimensional virtual space model,
The step of acquiring the additional user information is repeatedly performed in the direction of increasing the similarity,
A user pose estimation method for a three-dimensional space according to claim 17. a spatial information acquisition unit that acquires spatial information including depth information and video information for a three-dimensional space;
a virtual space model generating unit that configures depth-video link information based on the spatial information and generates a 3D virtual space model corresponding to the 3D space based on the depth-video link information;
a user information receiving unit for receiving user information including an image acquired by a user device in the three-dimensional space; and generating correspondence information corresponding to the user information in the three-dimensional virtual space model, A user pose estimation apparatus for a three-dimensional space, comprising: a controller including at least one processor configured to calculate a similarity measure with user information and estimate a user pose based on said similarity measure. The virtual space model generation unit
A background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space are separated from image information for the 3D space, and the background area is used for the 3D space. characterized by constructing a virtual space model,
A user pose estimation apparatus for a three-dimensional space according to claim 19. The control unit
distinguishing between a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in the image included in the user information; processing the user information using the background area to generate corresponding information corresponding to the user information processed in the three-dimensional virtual space model;
A user pose estimation apparatus for a three-dimensional space according to claim 19. An apparatus for estimating a user pose including position and orientation information of a user device with respect to a three-dimensional space, comprising:
a virtual space model providing unit that provides a 3D virtual space model constructed based on space information including depth information and image information for the 3D space;
a user information receiving unit that receives user information including an image acquired by the user device in the three-dimensional space; and a corresponding information that corresponds to the user information in the three-dimensional virtual space model, A user pose estimation apparatus for a three-dimensional space, comprising: a controller including at least one processor configured to calculate a similarity with said user information and estimate said user pose based on said similarity. The 3D virtual space model divides a background area related to the structure of the 3D space and a non-background area corresponding to an object placed in the 3D space in image information for the 3D space, and characterized by being constructed using the area,
User pose estimation apparatus for a three-dimensional space according to claim 22. a user information generation unit that generates user information including an image for a three-dimensional space;
a communication unit that transmits the user information to a user pose estimation server and receives information about the user pose estimated by the three-dimensional virtual space model from the server; and controls operations of the user information generation unit and the communication unit, A user pose estimation apparatus for three-dimensional space, comprising: a control unit including at least one processor configured to communicate information about the user pose to a currently executing application or driving system. The 3D virtual space model is generated based on spatial information including depth information and image information for the 3D space, and the image information for the 3D space includes a background region related to the structure of the 3D space and the A non-background area corresponding to an object placed in a three-dimensional space is separated and constructed using the background area,
User pose estimation apparatus for a three-dimensional space according to claim 24.

Images