KR20240028867A - Ai-based spatial structure estimation method using indoor floor point cloud, device and recording medium for performing the method - Google Patents

Abstract

The present invention is an AI-based spatial structure estimation device using an indoor floor point cloud, comprising: a processor; Includes a memory connected to the processor, wherein the memory scans the indoor space using a camera, generates an entire point cloud from the entire indoor space, extracts a floor cloud from the entire point cloud, and An indoor floor point that converts a cloud into a two-dimensional binary image and stores program instructions executed by the processor to detect the corner through a CNN model that learns by classifying the two-dimensional binary image into corners and lines. This is about an AI-based spatial structure estimation device using the cloud.
As a result, the corner of the floor can be accurately detected based on the CNN model from the entire point cloud of the indoor space.

Description

AI-based spatial structure estimation method using indoor floor point cloud and devices and recording media for performing the same {AI-BASED SPATIAL STRUCTURE ESTIMATION METHOD USING INDOOR FLOOR POINT CLOUD, DEVICE AND RECORDING MEDIUM FOR PERFORMING THE METHOD}

The present invention relates to an AI-based spatial structure estimation method using an indoor floor point cloud and a device and recording medium for performing the same. More specifically, it relates to a three-dimensional space of an indoor space by detecting corners of the floor using a CNN model. It is about techniques for estimating and modeling structures.

Due to the increase in non-face-to-face activities, interest in the three-dimensional virtual space Metaverse, where both reality and virtuality can coexist in all aspects of our lives, including economy, society, and culture, is increasing. In order for users to easily immerse themselves in the metaverse environment, modeling a realistic 3D virtual space is very important.

Recently, research on reconstructing and modeling 3D space from images captured with cameras rather than manually is being actively conducted. This estimates depth information using the homography of images and extracts a 3D point cloud based on this.

Since the point cloud obtained in this way has texture and 3D coordinate information, it has the advantage of being able to automatically build a realistic virtual space to which complex information (texture, lighting, materials, etc.) of the 3D real world space is applied.

However, point clouds have scattered and noisy characteristics during the extraction process, and information is lost in parts that were not captured. In particular, the point cloud representing the indoor space contains information about complex and diverse objects, and this information occludes the floor and walls.

When modeling real-world buildings, floor information is a very important clue. The floor surface actually matches the drawing, and knowing this is equivalent to knowing the structure of the entire space. In particular, the corners of the floor serve to connect architectural structures (walls, ceilings, etc.) and serve as the basis for modeling buildings.

However, in the case of point clouds, there is a lot of noise, and when noise is present, it has the disadvantage of not being able to detect accurate corners.

KR 10-1740259 B1

Accordingly, the technical problem of the present invention was conceived in this regard, and the purpose of the present invention is to accurately detect the corners of the floor surface of an indoor space and estimate the structure of the entire space by using an AI-based spatial structure estimation method using an indoor floor point cloud; The aim is to provide a device and recording medium to accomplish this.

An AI-based spatial structure estimation device using an indoor floor point cloud according to an embodiment for realizing the object of the present invention described above is an AI-based spatial structure estimation device using an indoor floor point cloud, comprising: a processor; It includes a memory connected to the processor, wherein the memory scans an indoor space using a camera, generates an entire point cloud from the indoor space, extracts a floor point cloud from the entire point cloud, and extracts a floor point cloud from the entire point cloud. Program instructions executed by the processor are stored to convert a surface point cloud into a two-dimensional binary image and detect the corner through a CNN model that learns by classifying the two-dimensional binary image into corners and lines.

In an embodiment of the present invention, the entire point cloud may be generated by generating a partial point cloud from the indoor space and matching the partial point cloud based on color image information or depth image information.

In an embodiment of the present invention, the floor point cloud may be extracted by removing a portion that exceeds a threshold corresponding to the floor from the entire point cloud.

In an embodiment of the present invention, detecting the corner may mean recovering the missing part of the two-dimensional binary image through a hole-filling algorithm and then detecting it through a CNN model.

In an embodiment of the present invention, the structure of the indoor space may be visualized by erecting a vertical wall based on the coordinates of the center of the bounding box connecting the detected corners.

An AI-based spatial structure estimation method using an indoor floor point cloud according to an embodiment for realizing another object of the present invention described above includes the steps of scanning an indoor space using a camera; Generating an entire point cloud from the entire indoor space; extracting a floor point cloud from the entire point cloud; Converting the floor cloud into a two-dimensional binary image; and detecting the corner through a CNN model that learns by classifying the two-dimensional binary image into corners and lines.

In another embodiment of the present invention, generating the entire point cloud includes generating a partial point cloud from the indoor space; And it may also include generating the partial point cloud by matching it based on color image information or depth image information.

In another embodiment of the present invention, the step of extracting the floor point cloud may be a step of extracting the entire point cloud by removing a portion that exceeds a threshold corresponding to the floor.

In another embodiment of the present invention, the step of detecting the corner may be a step of recovering the missing part of the two-dimensional binary image through a hole-filling algorithm and then detecting it through the CNN model.

In another embodiment of the present invention, the step of visualizing the structure of the indoor space by erecting a vertical wall based on the coordinates of the center of the bounding box connecting the detected corners may be included.

A computer-readable storage medium on which a computer program according to an embodiment for realizing the above-described other object of the present invention is recorded, a computer program for performing an AI-based spatial structure estimation method using the indoor floor point cloud. It may be recorded.

The present invention can accurately detect the corners of the floor based on a CNN model from the entire point cloud of the indoor space.

Additionally, based on this, the three-dimensional spatial structure of the indoor space is estimated and modeled.

Therefore, it is possible to efficiently build a metaverse space similar to the real world.

Figure 1 is a flowchart of an AI-based spatial structure estimation method using an indoor floor point cloud according to the present invention.
Figure 2 is an example of an AI-based spatial structure estimation method using an indoor floor point cloud according to the present invention.
Figure 3 is a process for generating an entire point cloud from a camera image according to the present invention.
Figure 4 is an example of generating partial and full point clouds according to the present invention.
Figure 5 is an example of detecting corners of a floor surface according to the present invention.
Figure 6 is an example of detecting corners and lines from a two-dimensional binary floor surface according to the present invention.
Figure 7 is an example comparing the floor corner detection of the prior art and the present invention.
Figure 8 is an example of estimating indoor space from floor corner detection according to the present invention.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

Figure 1 is a flowchart of an AI-based spatial structure estimation method using an indoor floor point cloud according to the present invention, and Figure 2 is an example corresponding to the flowchart. 1 and 2, first, the indoor space is photographed and scanned with the RGB-D camera 200 (S101). Afterwards, continuous color images 310 and depth images 330 are extracted (S103). Through this, a frame-by-frame partial point cloud 400 is generated (S105). Afterwards, the entire point cloud 500 is generated through registration of the partial point cloud 400 based on a 3D reconstruction algorithm (S107). The process of generating the entire point cloud 500 will be described in more detail with reference to FIGS. 3 and 4 below.

Figure 3 is a process for generating an entire point cloud 500 from a camera image according to the present invention. Referring to FIG. 4, when the number k of the color image 310 and the depth image 330 extracted through the RGB-D camera 200 are given as input, n frames (in this experiment, N = 100 is fixed) are input. Divide to extract m partial point clouds.

The ORB feature point matching algorithm between n frames is performed on a continuous set of color and depth image pairs. Through this, feature points in consecutive frames are matched and 5-Point RANSAC (Radom Sample Consensus) is performed to roughly estimate alignment between previous frames. Based on this, RGB-D odometry is calculated to track the camera trajectory for m partial point clouds, and a pose graph for multi-directional registration is constructed.

When problems such as loop closing occur, a process is performed to optimize the positions of all nodes to minimize errors through all nodes and edge information where errors have accumulated. The loss function for pose graph optimization can be expressed as Equation 1.

Here, x _i and x _j are the location information of the node in the current graph, and through this, the relative position between the two nodes can be obtained.

The difference between the position value (observed value) of the next frame node and the position value (predicted value) of the previous frame node between two nodes is defined as e _ij (x _i x _j ). Ω _ij refers to the information matrix of the i-th node and j-th node, and is the node that can minimize the sum of the loss function ( ) to extract the set.

Through this, a partial point cloud (400) containing voxels of a certain size and RGB information is extracted. The process of extracting the entire point cloud 500 from the extracted partial point cloud 400 will be described with reference to FIG. 4.

Figure 4 is an example of generating a partial and full point cloud 500 according to the present invention. Referring to FIG. 5, the entire point cloud 500 of the indoor space is generated based on the partial point cloud 400 and the pose graph generated in FIG. a. Approximate alignment is estimated through RGB-D odometry between partial point clouds 400, and registered in the global space using the RANSAC algorithm.

Based on this, partial point clouds 400 with similar color image information and depth image information are matched, and a rigid body transformation for registration is calculated by detecting the connection between the following sequences in the continuous partial point cloud 400 sequence. In the present invention, the Colored-IC P (Iterative Closest Point) algorithm is used to accurately and quickly detect the connection relationship between partial point clouds 400 using color image information. Figure 5b is the result of generating the entire point cloud 500 through registration of the partial point cloud 400. The steps (S109 to S115) after generating the entire point cloud 500 (S107) will be described below with reference to FIGS. 5 and 6.

Figure 5 is an example of detecting corners of a floor surface according to the present invention. Referring to FIG. 5, first, the floor point cloud 610 is extracted from the entire point cloud 500 (S109).

More specifically, in order to extract the floor point cloud 610 from the entire point cloud 500, the point cloud corresponding to the floor is extracted using the RANSAC algorithm. Outlier values that occur at this time (if they are not on a plane corresponding to the floor surface) are removed through threshold adjustment. Figure 5a shows the result of extracting the floor point cloud 610 from the entire indoor space point cloud 500.

Afterwards, the extracted floor point cloud 610 is converted into a two-dimensional binary image 620 through ortho projection (S111). A two-dimensional binary image means that the part that is lost due to occlusion by an object is displayed alone, and the other part is displayed to be filled.

Figure 5b shows the result of converting the floor point cloud 610 into a two-dimensional binary image 620. In the image 620, the point cloud is lost in the area 621 due to noise and occlusion by objects in the RGB-D camera, and the pixel value is also lost when converted to a two-dimensional binary image.

To solve this problem, the present invention restores the lost area 621 through a hole filling algorithm (S113). Figure 5c shows the restored two-dimensional binary image.

Then, as shown in FIG. 5, the corner 641 is detected by classifying it into two labels 640 (corner 641 and line 642) through the CNN model (S115).

Figure 6 is an example of detecting corners and lines from a two-dimensional binary floor surface according to the present invention. In order to detect the corners of the floor, the present invention detects the floor with AI learned through a CNN model. For example, more than 2,000 two-dimensional binary floor plan images classified into two labels (corner, line) are trained through a CNN model, and data augmentation techniques (Flip, Rotate, Sheer, etc.) are used to improve the reliability of the learning data. Raise.

Figure 7 is an example comparing the floor corner detection of the prior art and the present invention. Referring to Figure 7, Figure 7a is a restored two-dimensional binary image, Figure 7b shows corner detection using the Harris Corner algorithm, and Figure 7c shows corner detection using the SIFT algorithm. As shown in Figures 7b and 7c, it can be seen that irregular pixel values corresponding to the boundaries of the resulting image are detected as corners.

On the other hand, corner detection using the CNN model of the present invention produces accurate corner detection results even in irregular pixels, as shown in Figure 7d.

To evaluate the corner detection performance of the ball invention, F1-Score, precision, and recall values are used. F1-Score means the harmonic mean of recall and precision and can be expressed in Equation 2. The performance indicators extracted through this are shown in [Table 1].

The higher these three evaluation indicators are, the more accurately the corner of the floor has been detected.

The CNN model shows high classification performance in detecting floor corners with a precision of 90%, recall rate of 85%, and F1-Score of 87%.

Figure 8 is an example of estimating indoor space from floor corner detection according to the present invention. Referring to Figure 8, corners are detected from the binary floor surface extracted from the entire indoor point cloud (Figure 8a) (Figure 8b).

Then, as shown in Figure 8c, a vertical wall is built based on the center coordinates of the bounding box recognized as a corner to finally visualize the indoor space structure.

This AI-based spatial structure estimation method using an indoor floor point cloud can be implemented as an application or in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination.

Additionally, the program may be stored in a memory connected to the processor and executed by the processor.

The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention, or may be known and usable by those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc.

Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

Although the above has been described with reference to the embodiments, those skilled in the art can make various modifications and changes to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand.

The present invention overcomes the problems that existing methods for modeling a metaverse space similar to the real world consume time and a lot of cost. This can increase user convenience in modeling a metaverse space similar to the real world. Therefore, it can be usefully applied to the metaverse, which will be the next-generation social media, streaming, and gaming platform.

200: RGB-D camera 310: color image
330: depth image 400: partial point cloud
500: Entire point cloud 610: Floor point cloud
620: 2D binary image 621: Vanishing area
641: corner 642: line

Claims (11)

An AI-based spatial structure estimation device using indoor floor point cloud,
processor;
Including a memory connected to the processor,
The memory is,
Scan the indoor space using a camera,
Generate an entire point cloud from the indoor space,
Extract the floor point cloud from the entire point cloud,
Converting the floor point cloud into a two-dimensional binary image,
An AI-based spatial structure estimation device using an indoor floor point cloud that stores program instructions executed by the processor to detect the corners through a CNN model that classifies and learns the two-dimensional binary image into corners and lines.
According to paragraph 1,
The entire point cloud is,
Generate a partial point cloud from the indoor space,
An AI-based spatial structure estimation device using an indoor floor point cloud that is generated by matching the partial point cloud based on color image information or depth image information.
According to paragraph 1,
The floor point cloud is,
An AI-based spatial structure estimation device using an indoor floor point cloud, which is extracted by removing the part that exceeds the threshold corresponding to the floor from the entire point cloud.
According to paragraph 1,
Detecting the corner is an AI-based spatial structure estimation device using an indoor floor point cloud, which involves recovering the missing part of the two-dimensional binary image through a hole-filling algorithm and then detecting it through a CNN model.
According to paragraph 1,
An AI-based spatial structure estimation device using an indoor floor point cloud that visualizes the structure of the indoor space by erecting vertical walls based on the coordinates of the center of the bounding box connecting the detected corners.
Scanning an indoor space using a camera;
Generating an entire point cloud from the entire indoor space;
Extracting a floor point cloud from the entire point cloud;
Converting the floor cloud into a two-dimensional binary image; and
An AI-based spatial structure estimation method using an indoor floor point cloud, comprising the step of classifying the two-dimensional binary image into corners and lines and detecting the corner through a learning CNN model.
According to clause 6,
The step of generating the entire point cloud is,
generating a partial point cloud from the indoor space; and
An AI-based spatial structure estimation method using an indoor floor point cloud, comprising generating the partial point cloud by matching it with color image information or depth image information.
According to clause 6,
The step of extracting the floor point cloud is,
An AI-based spatial structure estimation method using an indoor floor point cloud, which is a step of extracting the part that exceeds the threshold corresponding to the floor from the entire point cloud.
According to clause 6,
The step of detecting the corner is,
An AI-based spatial structure estimation method using an indoor floor point cloud, which is a step of recovering the missing part of the two-dimensional binary image through a hole-filling algorithm and then detecting it through the CNN model.
According to clause 6,
An AI-based spatial structure estimation method using an indoor floor point cloud, comprising the step of visualizing the structure of the indoor space by erecting a vertical wall based on the coordinates of the center of the bounding box connecting the detected corners.
A computer-readable storage medium recording a computer program for performing an AI-based spatial structure estimation method using the indoor floor point cloud according to any one of claims 6 to 10.

Images