KR101869605B1 - Three-Dimensional Space Modeling and Data Lightening Method using the Plane Information - Google Patents

Abstract

Dimensional space modeling and data weighting method using plane information is provided. A three-dimensional space modeling method according to an embodiment of the present invention acquires a color image and a depth image, estimates a plane using the depth image, and then calculates three-dimensional spatial information using a color image, a depth image, . Thus, spatial reconstruction error is minimized by utilizing the structural information of the planar surface, so that it is possible to construct indoor three-dimensional spatial information of a building in which a large-scale three-dimensional space restoration technology is a key.

Description

TECHNICAL FIELD [0001] The present invention relates to a three-dimensional space modeling method and data lightening method using planar information,

The present invention relates to a three-dimensional imaging technique, and more particularly, to a method and system for modeling a three-dimensional space and lightening data.

The importance of indoor location information is increasing due to the increase in complexity due to the enlargement of the building and the increase in the number of buildings. Also, the existing text-based or 2D-based drawing such as location tracking of moving objects and CCTV diffusion for rescue / It is difficult to express complex geographical information. Therefore, it is important to construct indoor and outdoor 3D spatial information that can complement the geographical information.

In the past, 3D modeling of the terrain and the exterior of the building using aerial photographs or aerial data has been important. In recent years, 3D modeling as well as building exterior are important. 3D spatial modeling using depth information sensors (Lada, ToF, structured light, stereo) data is a core technology for constructing 3D spatial information.

1. Cumulative error problem

It is very important to accurately estimate the attitude of the sensor that acquires the data for accurate three-dimensional spatial restoration. However, as in the conventional SLAM (Simultaneously Localization and Mapping) method, a small error occurs every time the sensor attitude is estimated, and the corresponding error accumulates as time passes.

For this reason, even if one closed space is restored, the space is shifted as shown in Fig. In order to solve this problem, the bundle adjustment method, which is a global optimization method, is applied, but there is a limit to minimize the cumulative error as shown in FIG.

2. Space objectification problem

In order to utilize planar information in 3D modeling or to lighten 3D model, it is most important to objectize spatial information. Most of the previous methods used automatic objectification of structure for one scene or limited area, 3 is a result of user input based spatial objectification.

However, it is difficult to apply a single scene or a restricted area to a 3D space restoration technique of a large space, and the user input method has a disadvantage that time and manpower are consumed inefficiently.

SUMMARY OF THE INVENTION The present invention has been conceived to solve the above-mentioned problems, and it is an object of the present invention to provide a sensor attitude Dimensional spatial modeling method and system for minimizing estimation errors and restoration cumulative errors and reducing the point cloud data capacity constituting a three-dimensional space.

According to an aspect of the present invention, there is provided a three-dimensional space modeling method comprising: obtaining a color image and a depth image; Estimating a plane using a depth image; And generating three-dimensional spatial information using the color image, the depth image, and the plane information.

In the plane estimation step, the plane can be estimated using the normal vector value generated from the depth image.

According to another aspect of the present invention, there is provided a three-dimensional space modeling method, comprising the steps of: fusing similar planes among independently estimated planes in each frame; Dimensional space information can be generated using the information.

Then, the similar planes can be determined with reference to at least one of the angular difference between planes, the difference in the distance between planes, and the presence or absence of overlapping areas between planes.

Further, the normal vector of the fused plane may be the average of the normal vectors of the fused planes.

The three-dimensional space modeling method according to an embodiment of the present invention may further include estimating a posture of a sensor for acquiring an image from a depth image and aligning depth images acquired at different points of time .

The 3D spatial modeling method according to an embodiment of the present invention may further include extracting feature points from the aligned depth images, calculating a descriptor, and comparing the calculated descriptors to identify similar scenes have.

The fusion step may refer to at least one of the attitude information of the sensor and the similar scene information to fuse similar planes.

In addition, the method of three-dimensional space modeling according to an embodiment of the present invention may further include correcting the three-dimensional spatial information so that the planes whose angular differences are within a specific range are orthogonal.

According to another aspect of the present invention, there is provided a three-dimensional space modeling system including: an input unit for obtaining a color image and a depth image; And a processor for estimating a plane using the depth image and generating three-dimensional spatial information using the color image, the depth image, and the plane information.

According to another aspect of the present invention, there is provided a three-dimensional space modeling method comprising: obtaining a color image and a depth image on a frame basis; Comparing the depth images to obtain a similar scene; Estimating a plane using a depth image; Referring to similar scene information, fusing similar planes among independently estimated planes in each frame; And generating three-dimensional spatial information using the color image, the depth image, and the fused plane information.

According to another aspect of the present invention, there is provided a three-dimensional space modeling system including: an input unit for obtaining a color image and a depth image on a frame basis; And the depth image, compares the similar planes among the independently estimated planes in each frame by referring to the similar scene information, estimates the planar image using the depth image, And a processor for generating three-dimensional spatial information using the fused plane information.

As described above, according to the embodiments of the present invention, spatial restoration error is minimized by utilizing the structure information of the plane, so that it is possible to construct indoor three-dimensional spatial information of a building in which a large-scale three-dimensional space restoration technology is a key.

In addition, according to the embodiments of the present invention, it is possible to automatically estimate and manage the planar structure without user input, thereby reducing inefficient manpower consumption and time consumption.

In addition, according to the embodiments of the present invention, the amount of data of the three-dimensional spatial model can be drastically reduced by utilizing the polygon generation technique based on the objectification information of the space.

1 shows a mobile 3D space restoration result,
2 shows a result of three-dimensional spatial optimization using a bundle adjustment method,
FIG. 3 shows a result of spatial objectification based on user input,
Figure 4 is a flow chart provided in the description of a three-dimensional spatial modeling and data lightening method according to an embodiment of the present invention;
5 illustrates depth data-based sensor attitude estimation,
6 is a conceptual view of a scene recognition,
7 is a diagram showing a normal vector and a normal image,
FIG. 8 is a view showing a planar expansion condition,
9 is a diagram depicting a scene recognition based plane extension,
FIG. 10 is a graph showing the results of three-dimensional spatial restoration based on a planar structure,
11 is a block diagram of a three-dimensional spatial modeling and data lightening system according to another embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1. Three-Dimensional Spatial Modeling and Data Weight Reduction Using Planar Information

4 is a flow chart provided in the description of a three-dimensional space modeling and data weighting method according to an embodiment of the present invention. The three-dimensional space modeling and data weighting method according to the embodiment of the present invention includes six steps.

1) Sensor attitude estimation (S110): Estimation of sensor attitude using depth image acquired by sequence

2) Scene Recognition (S120): The similarity of images captured in the sensor posture of each frame is compared

3) Plane estimation (S130): Plane estimation using normal vector values generated from depth information

4) Planar fusion (S140): Using the sensor attitude information, scene recognition information, and plane information,

5) Planar Structure-based 3-D Spatial Modeling (S150): Minimizing 3-D spatial restoration error by performing global optimization method using fused plane information, sensor attitude information, color image and depth image

6) Three-dimensional spatial model correction and weight reduction (S160): The restored three-dimensional space is lightened by simple polygon data by using point cloud information

In the following, each step will be described in detail one by one.

2. Sensor posture estimation (S110)

And estimates the attitude of the sensor using the depth image information (S100) acquired by the sequence. As shown in FIG. 5, the estimated sensor attitude aligns the depth images obtained at different viewpoints to the same position.

3. Scene recognition (S120)

We propose a scene recognition method that can secure robustness against viewpoint and illumination change and guarantee computation efficiency.

To do this, a descriptor (for example, SIFT, SURF, or ORB) can be calculated from any three regions to extract feature points from the database image and describe them. At this time, an integral image is used for fast descriptor calculation.

For matching with a large number of database images, the descriptors are rearranged in a tree form. Therefore, not all descriptors are compared at the recognition stage, but hierarchical comparisons are made at each stage in the tree structure. FIG. 6 shows a conceptual view of a scene recognition technique.

4. Plane estimation (S130)

In an embodiment of the present invention, the plane is represented by a normal vector. The normal vector corresponding to each pixel is generated by comparing the depth information in the template as shown in FIG. 7, and it is assumed that pixels having a similar norm are grouped into one plane.

The well-known region growing algorithm is applied to group each pixel. When the normal vector between pixels differs by more than a certain angle or the depth between pixels differs by a certain distance, the region stops expanding assuming another plane or another object.

5. Planar fusion (S140)

Errors are continuously accumulated when estimating the camera attitude, and the structure of the three-dimensional space is shifted. If the planes of the displaced part are the same plane, the two planes that are displaced must be estimated as one plane in the real three-dimensional coordinate system.

In order to utilize the planar structure as one constraint information, it is necessary to move the plane generated independently in each frame to the three-dimensional coordinate system, and to label the similar planes. For the fusion of similar planes, three measurements were used in this task (see FIG. 8).

I) Angle difference between planes using a normal vector,

II) distance between planes,

III) The presence or absence of overlapping areas between planes.

The normal vector of the fused plane uses an average value of two normal vectors. The attitude information of the sensor is used to move the plane information of each frame to the three-dimensional coordinate system. However, the positional difference between the same planes may be large due to the cumulative error caused by the sensor's attitude estimation. In this case, it is determined that the same plane is used by using the scene graft information as shown in FIG.

6. Three-Dimensional Space Modeling Based on Planar Structure (S150)

In the embodiment of the present invention, the assumption that the planes having the same label are located at the same position in the three-dimensional position is applied as an additional error function to optimize the posture of the sensor in the direction of making the plane of the same label have the same norm do.

7. Three-Dimensional Spatial Model Calibration and Weight Reduction (S160)

Even if the three-dimensional space is optimized using the plane information, the points in the plane are not seen straightly due to the error of the depth image but are scattered by the constant noise.

To solve this problem, the plane noise is minimized by using the normal vector of each plane estimated from the plane expansion. In addition, if all the planes differ from each other within a specific angle range (for example, 80 ° to 100 °), the two planes are assumed to be orthogonal and the estimated planes are corrected.

FIG. 10 shows a result of a three-dimensional spatial axis alignment example as a final result from a mobile three-dimensional spatial restoration result. A plane polygon for data lightening is composed of 4 points by obtaining the maximum value and the minimum value located on the in-plane axis excluding the normal axis of each plane.

8. Three-Dimensional Spatial Modeling and Data Lightening System

11 is a block diagram of a three-dimensional spatial modeling and data lightening system according to another embodiment of the present invention. 11, the three-dimensional space modeling and data weighting system according to the embodiment of the present invention includes an image input unit 210, a processor 220, a storage unit 230, and an output unit 240 .

The image input unit 210 receives the color image and the depth from the camera (RGB camera + Z camera) in real time, and applies the input image to the processor 220.

The image input to the image input unit 210 may not be a real-time image. That is, the technical idea of the present invention can also be applied to a case where a color / depth image stored in an external device / network is input to the image input unit 210 and then applied to the processor 220.

The processor 220 includes a GPU and a CPU for executing the three-dimensional space modeling and data weighting method described above. The storage unit 230 provides storage space necessary for the processor 220 to perform three-dimensional space modeling and data weighting.

The output unit 240 includes a display for displaying three-dimensional spatial information performed by the processor 220 and a communication interface for communicating to an external device / network.

9. Variations

It is needless to say that the technical idea of the present invention can also be applied to a computer-readable recording medium having a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention may be embodied in computer-readable code form recorded on a computer-readable recording medium. The computer-readable recording medium is any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. In addition, the computer readable code or program stored in the computer readable recording medium may be transmitted through a network connected between the computers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

210:
220: Processor
230:
240: Output section

Claims (12)

A three-dimensional space modeling system, comprising: obtaining a color image and a depth image;
A three-dimensional space modeling system comprising: estimating a plane using a depth image;
A three-dimensional space modeling system comprising: generating three-dimensional spatial information using a color image, a depth image, and planar information; And
And correcting the three-dimensional spatial information so that the three-dimensional space modeling system is orthogonal to the planes whose angular difference is within a specific range.
The method according to claim 1,
The plane estimating step includes:
Dimensional space modeling method, wherein a plane is estimated using a normal vector value generated from a depth image.
The method according to claim 1,
Wherein the three-dimensional space modeling system further comprises fusing similar planes among the independently estimated planes in each frame,
The three-dimensional spatial information generating step includes:
And generating three-dimensional spatial information using the fused plane information.
The method of claim 3,
Similar planes,
Plane angle difference, a difference in plane-to-plane distance, and the presence or absence of an overlapping area between planes.
The method of claim 4,
Wherein the normal vector of the fused plane is an average of normal vectors of the fused planes.
The method of claim 3,
The three-dimensional space modeling system further includes a step of estimating a posture of a sensor acquiring an image from the depth image, and aligning the depth images acquired at different points of time.
The method of claim 6,
The three-dimensional space modeling system may further include extracting feature points from the aligned depth images, calculating descriptors, and comparing the calculated descriptors to identify similar scenes.
The method of claim 7,
In the fusion step,
Wherein the similar planes are fused by referring to at least one of the attitude information of the sensor and the similar scene information.
delete An input unit for acquiring a color image and a depth image;
And a processor for estimating a plane using the depth image and generating three-dimensional spatial information using the color image, the depth image, and the plane information,
The processor,
And corrects the three-dimensional spatial information so that the planes whose angular differences are within a specific range are orthogonal to each other.
A three-dimensional space modeling system comprising: a step of acquiring a color image and a depth image on a frame basis;
The three-dimensional space modeling system compares depth images and grasps similar scenes;
A three-dimensional space modeling system comprising: estimating a plane using a depth image;
The three-dimensional space modeling system refers to similar scene information and fuses similar planes among the independently estimated planes in each frame;
A three-dimensional space modeling system, comprising: generating three-dimensional spatial information using a color image, a depth image, and fused plane information; And
And correcting the three-dimensional spatial information so that the three-dimensional space modeling system is orthogonal to the planes whose angular difference is within a specific range.
An input unit for acquiring a color image and a depth image on a frame basis; And
Depth images are compared to obtain a similar scene, a plane is estimated using a depth image, and similar planes among independently estimated planes are fused with reference to similar scene information, and color images, depth images, and fusion And generating three-dimensional spatial information using the extracted plane information,
The processor,
And corrects the three-dimensional spatial information so that the planes whose angular differences are within a specific range are orthogonal to each other.

Images