KR102224272B1 - METHOD AND DEVICE FOR DETECTING PLANAR SURFACE USING DEPTH IMAGE, and Non-Transitory COMPUTER READABLE RECORDING MEDIUM - Google Patents

Abstract

An embodiment of the present invention includes the steps of generating a local plane for each pixel of a depth image; Detecting a feature of the generated local plane; Calculating a similarity between adjacent local planes using the detected features of the local planes; And if the similarity is greater than or equal to a predetermined similarity, grouping the adjacent local planes and detecting them as the same plane.

Description

Planar detection method and device through depth image, and non-transitory computer-readable recording medium {METHOD AND DEVICE FOR DETECTING PLANAR SURFACE USING DEPTH IMAGE, and Non-Transitory COMPUTER READABLE RECORDING MEDIUM}

The present invention relates to a method and apparatus for detecting a plane through a depth image, and to a non-transitory computer-readable recording medium. In detail, the present invention relates to a method of detecting each planar region in a depth image.

The depth image representing the distance of the captured image is currently being actively studied in the fields of SLAM (Simultaneous Localization And Mapping: Simultaneous Localization And Mapping), object recognition, detection, and tracking.

A plane is an important geometric feature in an image, and plane detection of an image is a subject covered in various computer vision fields such as scene understanding, image reconstruction, and object recognition. So far, various methods have been developed using random sample consensus (RANSAC) and Hough Transform to detect planes in images. Several methods developed so far have in common that they are based on depth information of an image, and a depth image or a 3D point set made of depth information obtained through a stereo image or a range finder is used for plane detection.

However, RANSAC is effective when a single plane in a 3D space is not plural, and has a limitation that it is not effective in a planar model of a 3D space including a plurality of planes.

Variations that can estimate multiple planes, such as sequential RANSAC (Sequential RANSAC), and multi-RANSAC, have been introduced, but they still do not provide an optimal solution, or the number of planes must be specified by the user in advance. It is inconvenient to do so, and is not particularly reliable in 3D information containing a lot of noise.

In addition, conventional plane detection methods have a disadvantage in that they are sensitive to noise in depth measurement because they detect a plane by using all pixels with known depth as candidates, and because only a depth value is used.

KR 10-2016-0086941 A1

An object of the present invention is to provide a method and apparatus capable of estimating a plane through a depth value of an area consisting of a plane in a depth screen, and a non-transitory computer-readable recording medium.

In addition, the present invention is to provide a plane detection method and apparatus and a non-transitory computer-readable recording medium for efficiently and accurately estimating a plane model of a 3D space from a depth image.

In addition, an object of the present invention is to provide a plane detection method and apparatus and a non-transitory computer-readable recording medium capable of solving a problem of a large time complexity in a conventional plane detection method.

In addition, an object of the present invention is to provide a plane detection method and apparatus and a non-transitory computer-readable recording medium capable of solving a problem sensitive to noise in depth measurement by using only a depth value in a conventional method.

In addition, an object of the present invention is to provide a plane detection method and apparatus and a non-transitory computer-readable recording medium that detects a plane more precisely than a conventional method.

An embodiment of the present invention includes the steps of generating a local plane for each pixel of a depth image; Detecting a feature of the generated local plane; Calculating a similarity between adjacent local planes using the detected features of the local planes; And if the similarity is greater than or equal to a predetermined similarity, grouping the adjacent local planes and detecting them as the same plane.

In another aspect, the generating of the local plane includes: generating a center transform coordinate of a three-dimensional coordinate system with a camera as an origin for one pixel in the depth image; Generating first to fourth peripheral transformed coordinates of a three-dimensional coordinate system with a camera as an origin for a neighboring pixel of one pixel in the depth image; And generating a plane including the center transform coordinates and the first to fourth peripheral transform coordinates as a local plane of the one pixel.

In another aspect, the peripheral pixel is a first vertical peripheral pixel that is a k-th pixel in a vertical downward direction from the one pixel (k is a natural number of 1 or more), and a second vertical transformation that is a k-th pixel in a vertical upward direction from the one pixel. A pixel, a first horizontal peripheral pixel that is a k-th pixel in a horizontal right direction from the one pixel, and a second horizontal peripheral pixel that is a k-th pixel in a horizontal left direction from the one pixel, and the first vertical peripheral pixel is the first 1 is generated as a peripheral transform coordinate, the second vertical peripheral pixel is generated as the second peripheral transform coordinate, the first horizontal peripheral pixel is generated as the third peripheral transform coordinate, and the second horizontal peripheral pixel is the A method of detecting a plane through a depth image generated by the fourth peripheral transform coordinate may be provided.

In yet another aspect, detecting a feature of the local plane comprises: generating a first local vector of the local plane; Generating a second local vector of the local plane; Generating a normal vector of the local plane by using the first local vector and the second local vector; And generating a feature distance of the local plane, which is a distance between the local plane and the camera, using the center transform coordinate and the normal vector.

In another aspect, the first local vector is a vector in a first direction passing through the center transformation coordinate and passing through the first peripheral transformation coordinate from the second peripheral transformation coordinate, and the second local vector is the center transformation coordinate. A method for detecting a plane through a depth image externally generated from the fourth peripheral transform coordinate and passing through the third peripheral transform coordinate, and the normal vector is an externally generated depth image of the first local vector and the second local vector. Can provide.

In another aspect, the calculating of the similarity between adjacent local planes includes: calculating a first similarity using a normal vector of a local plane of one pixel and a local plane of another pixel; And calculating a second similarity using the local plane name of the one pixel and the feature distance of the local plane of the other pixel.

In another aspect, the first similarity is an angle between a local plane of the one pixel and a normal vector of a local plane of the other pixel, and the second similarity is a feature distance of the local plane of the one pixel and the other pixel. A method of detecting a plane through a depth image, which is a difference between a feature distance of a local plane, can be provided.

In another aspect, if the similarity is greater than or equal to a predetermined similarity, grouping the adjacent local planes and detecting the same plane may include determining whether the first similarity is greater than or equal to a predetermined value; Determining whether the second similarity is less than or equal to a predetermined value if the first similarity is greater than or equal to a predetermined value; Grouping the one pixel and the other pixel if the second similarity is less than or equal to a predetermined value; Determining whether there are other pixels in the vicinity that have not been compared with the one pixel; And detecting the grouped pixels as the same plane when there are no other surrounding pixels that are not compared with the one pixel.

In another aspect, the generating of the local plane includes: generating a center transform coordinate of a three-dimensional coordinate system with a camera as an origin for one pixel in the depth image; Generating peripheral transformed coordinates of a three-dimensional coordinate system with a camera as an origin for neighboring pixels in an N*N rectangular area centered on one pixel in the depth image; And generating a plane including the center transform coordinates and the peripheral transform coordinates as a local plane of the one pixel.

In yet another aspect, detecting a feature of the local plane comprises: generating a first average local vector of the local plane; Generating a second average local vector of the local plane; Generating a normal vector of the local plane using the first and second average local vectors; And generating a feature distance of the local plane using the center transform coordinate and the normal vector, wherein the first average local vector is a sum of a plurality of first local vectors of the local plane, and the first 2 The average local vector may provide a plane detection method through a depth image that is a sum of a plurality of second local vectors of the local plane.

In yet another aspect, at least one memory for storing instructions; And at least one processor; wherein the instructions are executable by the processor to cause the processor to perform operations, wherein the operations are:

A local plane is generated for each of the depth image pixels, a feature of the generated local plane is detected, and a similarity between adjacent local planes is calculated using the detected features of the local plane, and if the similarity is greater than or equal to a predetermined similarity, It is possible to provide a plane detection apparatus including grouping the adjacent local planes and detecting the same plane.

In yet another aspect, a non-transitory computer-readable medium for storing the instructions may be provided.

According to an embodiment of the present invention, a plane can be quickly and accurately estimated through depth information in a depth image.

Further, in the embodiment of the present invention, precision is improved by detecting a planar area in units of pixels.

In addition, according to an embodiment of the present invention, the data processing speed is improved by calculating a planar factor in units of a local plane and grouping adjacent local planes having similar plane factors into the same plane to detect regions of each plane.

In addition, according to the exemplary embodiment of the present invention, even when a depth image contains a lot of noise, reliable plane estimation is possible.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood from the following description.

1 is a block diagram showing a plane detection apparatus according to an embodiment of the present invention.
2 is an exemplary diagram of a plane detection method according to an embodiment of the present invention.
3 is a flowchart illustrating a method of generating a local plane according to an exemplary embodiment.
4 shows that a point on the camera coordinate system is projected onto the image plane.
5 shows a state in which pixels in a depth image are transformed into a 3D coordinate system.
6 is a flowchart illustrating a method of detecting a feature of a local plane according to an exemplary embodiment.
7 is a three-dimensional coordinate system for describing a normal vector that is a feature of the local plane of FIG. 6.
8 is a flowchart illustrating a method of calculating a similarity between local planes according to an exemplary embodiment.
9 is a flowchart illustrating a method of detecting a plane through grouping according to a degree of similarity between local planes according to an exemplary embodiment.
10 is a flowchart illustrating a method of generating a local plane according to another exemplary embodiment.
11 is a flowchart illustrating a method of detecting a feature of a local plane according to another exemplary embodiment.
12 is an example of a plurality of local vectors in a local plane for obtaining an average local vector of the local plane according to another embodiment.
13 shows a result of plane detection according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used for the purpose of distinguishing one constituent element from other constituent elements rather than a limiting meaning. In addition, expressions in the singular include plural expressions unless the context clearly indicates otherwise. In addition, terms such as include or have means that the features or components described in the specification are present, and do not preclude the possibility of adding one or more other features or components in advance. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, and thus the present invention is not necessarily limited to what is shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted. .

1 is a block diagram showing a plane detection apparatus according to an embodiment of the present invention.

The plane detection device 100 may provide image data to the data receiving device 200.

The plane detection device 100 and the data receiving device 200 are desktop computers, notebook (ie, laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart". It may include any of a wide variety of devices including pads, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and the like.

In some implementations, the plane detection device 100 and the data receiving device 200 may be equipped with a configuration 10 for wireless communication.

In addition, the data receiving apparatus 200 may receive image-processed image data through a computer-readable medium.

The computer-readable medium may include any type of medium or device capable of moving image data processed from the plane detection apparatus 100 to the data receiving apparatus 200. For example, the computer-readable medium may include a communication medium, such as a transmission channel, which enables the plane detection device 100 to directly transmit image data to the data receiving device 200 in real time.

The image-processed image data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the data receiving apparatus 200. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from the plane detection device 100 to the data receiving device 200. In some examples, the image-processed image data may be output from the output interface 130 to a computer-readable storage medium, such as a non-transitory computer-readable storage medium, that is, a data storage device. Similarly, image data may be accessed from the storage device by the input interface 220 of the data receiving apparatus 200. The storage device can be any of a variety of distributed or distributed digital storage media, such as a hard drive, Blu-ray disks, DVDs, CD-ROMs, flash memory, volatile or nonvolatile memory, or any other suitable digital storage media for storing image data. It may include any of the non-transitory data storage media that are accessed locally. In a further example, the storage device may correspond to a file server or other intermediate storage device that may store image data generated by the plane detection apparatus 100.

The data receiving apparatus 200 may access the stored image data from the storage device through streaming or download.

In the example of FIG. 1, the plane detection apparatus 100 may include an image source 110 and an image processing unit 120. In addition, the plane detection device 100 may further include an output interface 130.

The data receiving device 200 may include an input interface 220 and a data processing unit 210.

In another example, the plane detection apparatus 100 and the data processing unit 210 may include different components.

For example, the plane detection apparatus 100 may receive an image from an external video source, such as an external camera, and the external camera may be a depth image photographing device that generates a depth image.

The image source 110 of the plane detection apparatus 100 is a depth image photographing device, such as a camera, an archive including a previously photographed depth image, and/or an interface for receiving a depth image from a depth image content provider. It may also include.

In some implementations, the depth image capturing device may provide a depth image in which depth information of a scene is expressed in an integer data type of 16 bits per pixel (pixel). The number of bits for representing one pixel of the depth image may be changed. The depth image capturing device may provide a depth image having a value proportional or inversely proportional to the distance by measuring a distance from the depth image capturing device to an object and a background using infrared rays or the like.

The pixel value of the depth image may be, for example, not color information of RGB, but depth information of an integer of mm unit (but not limited thereto).

Each of the plane detection device 100 and the data receiving device 200 includes one or more memories and one or more microprocessors, digital signal processors (DSPs), custom integrated circuits (ASICs), field programmable gate arrays ( FPGAs), individual logic circuits, software, hardware, firmware, or any combination thereof.

The memory contains instructions (eg, executable instructions) such as computer readable instructions or processor readable instructions. Instructions may include one or more instructions executable by a computer, such as by each of one or more processors.

For example, the one or more instructions may be executable by one or more processors to perform operations including processing the depth image to cause the one or more processors to detect a plane of the depth image.

In detail, the image processing unit 120 may include one or more memories 121 for storing instructions and at least one processor 122 for executing the instructions.

The processor 122 of the image processing unit 120 may be configured to apply techniques for detecting a plane in the depth image.

In some implementations, the processor 122 of the image processing unit 120 may be configured to apply techniques for grouping similar planes in the depth image.

In some implementations, the processor 122 of the image processing unit 120 may be configured to apply techniques for encoding a depth image.

In another embodiment, the processor 122 of the image processing unit 120 may be configured to apply a technique of encoding and decoding a depth image.

The data processing unit 220 may be configured to transmit, display, and analyze the depth image data from the image processing unit 120 to an external device.

In some implementations, the data processing unit 220 may be configured to decode the encoded depth image data from the image processing unit 120.

Although not shown in FIG. 1, in some embodiments, the plane detection device 100 may be a device integrated with an encoding device. In some embodiments, the plane detection device 100 and the data processing device 200 may be integrated devices. For example, the plane detection apparatus 100 may be configured to encode a depth screen and decode the encoded depth screen.

2 is an exemplary diagram of a plane detection method according to an embodiment of the present invention.

Referring to FIG. 2, in the plane detection method according to an embodiment of the present invention, the processor 122 of the plane detection apparatus 100 generates a local plane for each pixel of a depth image (S100), and the plane detection apparatus The processor 122 of (100) detects a feature of a local plane (S200), and the processor 122 of the plane detection apparatus 100 calculates a similarity between adjacent local planes using the detected feature of the local plane. In operation S300 and the processor 122 of the plane detection apparatus 100, if the adjacent local planes are greater than or equal to a predetermined degree of similarity, grouping them and detecting them as the same plane (S400). Each step of the plane detection method according to an embodiment of the present invention will be described in more detail with reference to FIGS. 2 to 9.

3 is a flowchart illustrating a method of generating a local plane according to an exemplary embodiment, and FIG. 4 is a diagram illustrating that a point on a camera coordinate system is projected onto an image plane, and FIG. 5 is a diagram illustrating pixels in a depth image as a 3D coordinate system. It shows the transformed form.

Referring to FIG. 3, in a method of generating a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 calculates a center transform coordinate of a 3D coordinate system with a camera as an origin for a pixel in a depth image. It may include the step of generating (S110). More specifically, referring to FIG. 4, an arbitrary point P'of an object in the camera coordinate system has (x`, y`, z`) coordinates, where P`(x`, y`) is of the image plane. It is projected to a point with P(x, y) coordinates, and the z` value becomes the depth value at the P(x, y) coordinates. In addition, the depth value may be expressed in units of each pixel. Specifically, a screen captured when capturing a depth image is projected onto an image plane in which points in the capturing area are virtual planes. And if a point P'≡(x', y', z') in the camera coordinate system is projected to P = (x, y), which is a point in the image plane with the center as the origin, the depth value at the pixel containing P When this is d, the relationship between P'and P satisfies Equation 1.

[Equation 1]

x`=dx/f

y`=dy/f

z`=d

In Equation 1, f is the focal length of the camera and is defined as the distance between the origin and the image plane in the camera coordinate system. As an example, referring to FIG. 5, the processor 122 converts one pixel (P(x,y)) in the depth image into a center transformed coordinate (Pc(x,y)) which is a three-dimensional coordinate system using Equation 1 Can be converted.

In the method of generating a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 transforms the first to fourth surroundings of a 3D coordinate system with a camera as an origin for a neighboring pixel of a pixel in a depth image. It may include generating coordinates (S120). More specifically, the surrounding pixels of one pixel may mean a pixel located adjacent to the one pixel. The processor 122 may generate four neighboring pixels of one pixel P(x,y) as first to fourth peripheral transform coordinates. As an example, the peripheral pixel is a first vertical peripheral pixel that is a k-th pixel vertically downward from one pixel (k is a natural number of 1 or more), a second vertical peripheral pixel that is a k-th pixel vertically upward from one pixel, and one pixel. A first horizontal peripheral pixel that is a k-th pixel in a horizontal right direction, and a second horizontal peripheral pixel that is a k-th pixel in a horizontal left direction from one pixel may be included. The processor 122 uses a first vertical peripheral pixel as a first peripheral transformed coordinate, a second vertical peripheral pixel as a second peripheral transformed coordinate, a first horizontal peripheral pixel as a third peripheral transformed coordinate, and a second horizontal peripheral pixel as a second peripheral transformed coordinate. It can be converted into a fourth peripheral transform coordinate.

In the method of generating a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 generates a plane including a center transform coordinate and first to fourth peripheral transform coordinates as a local plane of one pixel. It may include (S130).

6 is a flowchart illustrating a method of detecting a feature of a local plane according to an exemplary embodiment, and FIG. 7 is a three-dimensional coordinate system for describing a normal vector that is a feature of the local plane of FIG. 6.

)를 생성하는 단계(S210)를 포함할 수 있다. 보다 구체적으로, 국소 평면의 제1 국소 벡터(

)는 중심 변환 좌표(Pc(x,y))를 지나며 제2 주변 변환 좌표(Pc(x,y+k))에서 제1 주변 변환 좌표(Pc(x,y-k))를 지나는 제1 방향의 벡터 일 수 있다. 일 예로, 제1 방향은 수직 하측 방향 또는 수직 하측 방향에 가까운 방향일 수 있다. 제1 국소 벡터(

)는 Pc(x,y-k)-Pc(x, y+k)로 나타낼 수 있다. Referring to FIG. 6, in a method of detecting a feature of a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 uses a first local vector of the local plane (

) May include generating (S210). More specifically, the first local vector of the local plane (

) Passes through the center transform coordinate (Pc(x,y)) and passes from the second peripheral transform coordinate (Pc(x,y+k)) to the first peripheral transform coordinate (Pc(x, yk)) in the first direction. It can be a vector. For example, the first direction may be a vertical downward direction or a direction close to a vertical downward direction. The first local vector (

) Can be represented as Pc(x,yk)-Pc(x, y+k).

)를 생성하는 단계(S220)를 포함할 수 있다. 보다 구체적으로, 국소 평면의 제2 국소 벡터(

)는 중심 변환 좌표(Pc(x,y))를 지나며 제4 주변 변환 좌표(Pc(x-k,y))에서 제3 주변 변환 좌표(Pc(x+k,y))를 지나는 제2 방향의 벡터 일 수 있다. 일 예로, 제2 방향은 수평 우측 방향 또는 수평 우측 방향에 가까운 방향일 수 있다. 제2 국소 벡터(

)는 Pc(x+k,y)-Pc(x-k, y)로 나타낼 수 있다.In a method of detecting a feature of a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 may use a second local vector (

) May include generating (S220). More specifically, the second local vector of the local plane (

) Passes through the center transform coordinate (Pc(x,y)) and passes from the fourth peripheral transform coordinate (Pc(xk,y)) to the third peripheral transform coordinate (Pc(x+k,y)) in the second direction. It can be a vector. For example, the second direction may be a horizontal right direction or a direction close to a horizontal right direction. The second local vector (

) May be represented as Pc(x+k,y)-Pc(xk, y).

) 및 제2 국소 벡터(

)를 이용하여 일 화소(P)에 대한 국소 평면의 법선 벡터(n(x,y))를 생성하는 단계(S230)를 생성하는 단계를 포함할 수 있다. 보다 구체적으로, 법선 벡터(n(x,y))는 수학식 2에 따라 제1 국소 벡터(

)와 제2 국소 벡터(

)의 외적으로 구할 수 있다. In a method of detecting a feature of a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 may use a first local vector (

) And the second local vector (

) To generate a local plane normal vector (n(x,y)) for one pixel P (S230). More specifically, the normal vector (n(x,y)) is the first local vector (

) And the second local vector (

) Can be obtained externally.

[Equation 2]

)와 제1 국소 벡터(

)의 곱으로 순서를 달리하여 구할 수 있다.It is not limited thereto, and the normal vector (n(x,y)) is the second local vector (

) And the first local vector (

It can be obtained in a different order by multiplying by ).

)를 생성하는 단계(S240)를 포함할 수 있다. 보다 구체적으로, 중심 변환 좌표(Pc(x,y))의 국소 평면의 법선 벡터(n(x,y))의 성분을 (

)이라고 했을 때, 중심 변환 좌표(Pc(x,y))를 지나고 법선 벡터(n(x,y))를 가지는 평면은 수학식 3과 같이 표현 할 수 있다.In the method of detecting a feature of a local plane according to an embodiment, the processor 122 of the plane detection apparatus 100 uses a center transform coordinate (Pc(x,y)) and a normal vector (n(x,y)). And the feature distance of the local plane, which is the distance between the local plane and the camera (

) May include the step of generating (S240). More specifically, the component of the local plane normal vector (n(x,y)) of the center transform coordinate (Pc(x,y)) is (

), a plane passing through the center transform coordinate (Pc(x,y)) and having a normal vector (n(x,y)) can be expressed as Equation 3.

[Equation 3]

)는 수학식 4를 통해 계산될 수 있다. At this time, the feature distance, which is the distance from the camera on the local plane (

) Can be calculated through Equation 4.

[Equation 4]

8 is a flowchart illustrating a method of calculating a similarity between local planes according to an exemplary embodiment.

Referring to FIG. 8, in a method of calculating the similarity between local planes according to an embodiment, the processor 122 of the plane detection apparatus 100 uses a local plane of one pixel and a normal vector of a local plane of another pixel. 1 It may include the step of calculating the similarity (U) (S310). More specifically, the first degree of similarity (U) can be determined by the angle between the normal vector (n(x,y)) of the local plane of one pixel and the normal vector (m(x,y)) of the local plane of another pixel. have. For example, the first similarity U may be determined according to Equation 5.

[Equation 5]

In addition, in a method of calculating the similarity between local planes according to an embodiment, the processor 122 of the plane detection apparatus 100 uses a second similarity (D) using a feature distance between a local plane of one pixel and a local plane of another pixel. ) It may include a calculating step (S320). More specifically, the second similarity D may be determined by a difference between a feature distance Dn of a local plane of one pixel and a feature distance Dm of a local plane of another pixel. For example, the second similarity D may be determined according to Equation 6.

[Equation 6]

9 is a flowchart illustrating a method of detecting a plane through grouping according to a degree of similarity between local planes according to an exemplary embodiment.

Referring to FIG. 9, in the method for detecting a plane through grouping according to the similarity between local planes according to an embodiment, the processor 122 of the plane detection apparatus 100 determines whether the first similarity U is greater than or equal to a predetermined value. It may include a step (S410). More specifically, the first similarity U may be set in advance according to the level, and may be adjusted through programming of the memory 121.

In addition, in the plane detection method through grouping according to the similarity between local planes according to an embodiment, when the processor 122 of the plane detection apparatus 100 satisfies the first similarity U, the second similarity D is predetermined. It may include determining whether the value is less than or equal to (S420, S430). More specifically, the first similarity U may be satisfied when the first similarity U is greater than or equal to a predetermined value. If it is determined that the first similarity U is not satisfied, the processor 122 may perform steps S420 and S470 of comparing one pixel with other surrounding pixels that have not been compared. That is, the processor 122 may determine whether the first similarity U of one pixel and another pixel is equal to or greater than a predetermined value (S470 and S410).

In addition, in the method of detecting a plane through grouping according to the similarity between local planes according to an embodiment, when the processor 122 of the plane detection apparatus 100 satisfies the second similarity (D), one pixel and the other pixel are grouped. It may include steps S440 and S450. More specifically, satisfying the second degree of similarity (D) may be a case where the second degree of similarity (D) is less than or equal to a predetermined value. The second similarity D may be set in advance according to the level, and may be adjusted through programming of the memory 121. If it is determined that the second similarity D is not satisfied, the processor 122 may perform steps (S440, S470) of comparing one pixel with other surrounding pixels that have not been compared. That is, the processor 122 may determine whether the first similarity U of one pixel and another pixel is equal to or greater than a predetermined value (S470 and S410).

In addition, in the method for detecting a plane through grouping according to the degree of similarity between local planes according to an exemplary embodiment, the processor 122 of the plane detecting apparatus 100 determines whether there is another pixel in the vicinity that has not been compared with one pixel. It may include (S460). The processor 122 may perform steps (S460, S470) of comparing one pixel with other surrounding pixels that have not been compared with other pixels in the surrounding area if there is another pixel in the vicinity that has not been compared with one pixel. That is, the processor 122 may determine whether the first similarity U of one pixel and another pixel is equal to or greater than a predetermined value (S470 and S410).

In addition, in the plane detection method through grouping according to the degree of similarity between local planes according to an embodiment, the processor 122 of the plane detection apparatus 100 is grouped if there is no other surrounding pixels that are not compared with one pixel. It may include the step (S480) of detecting the pixels in the same plane. According to an embodiment of the present invention, a plane can be quickly and accurately estimated through depth information in a depth image.

Accordingly, according to an exemplary embodiment of the present invention, precision is improved by detecting a planar area in units of pixels. In addition, according to an embodiment of the present invention, the data processing speed is improved by calculating a planar factor in units of a local plane and grouping adjacent local planes having similar plane factors into the same plane to detect regions of each plane. In addition, according to the exemplary embodiment of the present invention, even when a depth image contains a lot of noise, reliable plane estimation is possible.

In some embodiments, the processor 122 may group pixels satisfying the first and second similarities of a local plane among a plurality of pixels in the depth image according to the above-described process to detect a plane, and A plurality of planes may be detected. The detection of a plurality of planes in the depth image may be a case in which the first and second similarities between the detected planes are not satisfied. That is, the detected plurality of planes are planes that are not grouped together into one plane. The detected planes may transfer information of each plane area to other applications for use in other applications. Here, the plane information may be, for example, the number of detected planes, the position of the upper left corner in the depth image of each plane, and the height and width of the plane in the depth image. The processor 122 may provide information on a plane in response to a request from such other software. [0038] In more detail, a plurality of planes within one depth image may be described. An average value of the feature distances of all pixels in the corresponding plane may be generated by using feature distance information of each of the local planes of a pixel included in the plane. For example, the processor 122 may generate a first average value for the feature distances of the first pixels in the first plane, and may generate a second average value for the feature distances of the second pixels in the second plane. . The processor 122 may store difference information between the first and second average values. Herein, the difference information between the first and second average values may be one of plane distribution information provided by the processor 122 to the user. The embodiment may provide distribution information between the objects when a plurality of objects having planes on a plurality of photographing planes exist in a depth image. Accordingly, when objects having a level difference from each other exist in the photographing area, the corresponding level difference information can be detected. For example, when photographing a desk arranged in a space, the height of the desk can be estimated by detecting a difference in height between the upper surface of the desk and the floor surface of the space. In addition, it can be used as basic data to obtain various information about an object by estimating the step information of an object having a plane.

Each step of a plane detection method according to another embodiment of the present invention will be described in more detail with reference to FIGS. 2 and 10 to 13. Other embodiments of the present invention will focus on a method of generating a local plane having a difference and a method of detecting a feature of the local plane, except for the same description as in the embodiment of the present invention.

10 is a flowchart illustrating a method of generating a local plane according to another exemplary embodiment.

Referring to FIG. 10, in a method of generating a local plane according to another embodiment, the processor 122 of the plane detection apparatus 100 uses a camera for one pixel in a depth image as an origin. It may include a step of generating (S1110). As an example, referring to FIG. 5, the processor 122 converts one pixel (P(x,y)) in the depth image into a center transformed coordinate (Pc(x,y)) which is a three-dimensional coordinate system using Equation 1 Can be converted.

In addition, in a method of generating a local plane according to another exemplary embodiment, the processor 122 of the plane detection apparatus 100 uses an N*N (N is 3, 5, 7, …) rectangle centered on one pixel in the depth image. It may include generating (S1120) peripheral transformed coordinates of a 3D coordinate system with a camera as an origin of the surrounding pixels of the region. Accordingly, another embodiment makes it possible to obtain an accurate local plane from a noisy depth image. More specifically, the processor 122 defines pixels located outside the N*N rectangular area around one pixel (P(x,y)) as surrounding pixels, and generates the surrounding pixels as a plurality of peripheral transform coordinates. I can. As an example, the plurality of peripheral transform coordinates may be 4 (N-1).

In addition, in the method of generating a local plane according to another embodiment, the processor 122 of the plane detection apparatus 100 generates a plane including the center transform coordinates and the peripheral transform coordinates as a local plane of one pixel (S1130). ) Can be included.

FIG. 11 is a flowchart illustrating a method of detecting a feature of a local plane according to another embodiment, and FIG. 12 is a diagram illustrating a plurality of local vectors in a local plane for obtaining an average local vector of the local plane according to another embodiment. Yes.

)를 생성하는 단계(S1210)을 포함할 수 있다. 보다 구체적으로, 제1 평균 국소 벡터(

)는 국소 평면의 복수의 제1 국소 벡터의 평균 일 수 있다. 복수의 제1 국소 벡터는 주변 변환 좌표들 중 상측에 위치한 주변 변환 좌표들과 하측에 위치한 주변 변환 좌표들이 각각 쌍을 이루어 상측 주변 변환 좌표에서 하측 주변 변환 좌표로 향한 수직 성분의 벡터들일 수 있다. 이에 제한되는 것은 아니고, 복수의 제1 국소 벡터는 주변 변환 좌표들 중 하측에 위치한 주변 변환 좌표들과 상측에 위치한 주변 변환 좌표들이 각각 쌍을 이루어 하측 주변 변환 좌표에서 상측 주변 변환 좌표로 향한 수직 성분의 벡터들일 수 있다. 또한, 복수의 제1 국소 벡터 중 임의의 제1 국소 벡터(

)는 수학식 7에 따라 결정될 수 있다.Referring to FIG. 11, in a method of detecting a feature of a local plane according to another embodiment, the processor 122 of the plane detection apparatus 100 uses a first average local vector of the local plane (

) May include the step of generating (S1210). More specifically, the first average local vector (

) May be an average of a plurality of first local vectors in the local plane. The plurality of first local vectors may be vectors of vertical components from the upper peripheral transform coordinates to the lower peripheral transform coordinates by pairing the upper peripheral transform coordinates and the lower peripheral transform coordinates among the peripheral transform coordinates. It is not limited thereto, and the plurality of first local vectors is a vertical component from the lower peripheral transform coordinates to the upper peripheral transform coordinates by pairing the lower peripheral transform coordinates and the upper peripheral transform coordinates among the peripheral transform coordinates. May be vectors of. In addition, any first local vector from among the plurality of first local vectors (

) May be determined according to Equation 7.

[Equation 7]

)를 결정할 수 있다. 제1 평균 국소 벡터(

)는 수학식 8에 따라 복수의 제1 국소 벡터들을 합하여 각 벡터가 가지는 방향의 평균 방향을 가지도록 생성될 수 있다.As an example, referring to FIG. 12, if N=3, the processor 122 has three first local vectors according to Equation 7 (

) Can be determined. First mean local vector (

) May be generated to have an average direction of a direction of each vector by summing a plurality of first local vectors according to Equation 8.

[Equation 8]

)를 생성하는 단계(S1220)을 포함할 수 있다. 보다 구체적으로, 제2 평균 국소 벡터(

)는 국소 평면의 복수의 제2 국소 벡터의 평균 일 수 있다. 복수의 제2 국소 벡터는 주변 변환 좌표들 중 좌측에 위치한 주변 변환 좌표들과 우측에 위치한 주변 변환 좌표들이 각각 쌍을 이루어 좌측 주변 변환 좌표에서 우측 주변 변환 좌표로 향한 수평 성분의 벡터들일 수 있다. 이에 제한되는 것은 아니고, 복수의 제2 국소 벡터는 주변 변환 좌표들 중 우측에 위치한 주변 변환 좌표들과 좌측에 위치한 주변 변환 좌표들이 각각 쌍을 이루어 우측 주변 변환 좌표에서 좌측 주변 변환 좌표로 향한 수평 성분의 벡터들일 수 있다. 또한, 복수의 제2 국소 벡터 중 임의의 제2 국소 벡터(

)는 수학식 7에 따라 결정될 수 있다. 일 예로, 도 12를 참조하면, 프로세서(122)는 N=3이면 수학식 7에 따라 3개의 제2 국소 벡터(

)를 결정할 수 있다. 제2 평균 국소 벡터(

)는 수학식 8에 따라 복수의 제2 국소 벡터들을 합하여 각 벡터가 가지는 방향의 평균 방향을 가지도록 생성될 수 있다.In addition, in a method of detecting a feature of a local plane according to another exemplary embodiment, the processor 122 of the plane detection apparatus 100 uses the second average local vector of the local plane (

) May include generating (S1220). More specifically, the second average local vector (

) May be an average of a plurality of second local vectors in the local plane. The plurality of second local vectors may be vectors of a horizontal component from a left peripheral transform coordinate to a right peripheral transform coordinate by forming a pair of peripheral transform coordinates located at the left and the surrounding transform coordinates located at the right among the surrounding transform coordinates. It is not limited thereto, and the plurality of second local vectors is a horizontal component of the peripheral transformation coordinates located at the right and the surrounding transformation coordinates located at the left of the surrounding transformation coordinates, respectively, forming a pair from the surrounding transformation coordinates on the right to the transformation coordinates on the left May be vectors of. In addition, any second local vector from among the plurality of second local vectors (

) May be determined according to Equation 7. As an example, referring to FIG. 12, if N=3, the processor 122 has three second local vectors according to Equation 7 (

) Can be determined. Second mean local vector (

) May be generated to have an average direction of a direction of each vector by summing a plurality of second local vectors according to Equation 8.

) 및 제2 평균 국소 벡터(

)를 이용하여 일 화소(P)에 대한 국소 평면의 법선 벡터(

)를 생성하는 단계(S1230)를 생성하는 단계를 포함할 수 있다. 보다 구체적으로, 법선 벡터(

)는 수학식 9에 따라 제1 평균 국소 벡터(

)와 제2 평균 국소 벡터(

)의 외적으로 구할 수 있다. In addition, in a method of detecting a feature of a local plane according to another embodiment, the processor 122 of the plane detection apparatus 100 uses a first average local vector (

) And the second mean local vector (

) Using the local plane normal vector (

) May include the step of generating the generating step (S1230). More specifically, the normal vector (

) Is the first average local vector (

) And the second mean local vector (

) Can be obtained externally.

[Equation 9]

)는 제2 평균 국소 벡터(

)와 제1 평균 국소 벡터(

)의 곱으로 순서를 달리하여 구할 수 있다.It is not limited thereto, and the normal vector (

) Is the second mean local vector (

) And the first mean local vector (

It can be obtained in a different order by multiplying by ).

)를 이용하여 국소 평면과 카메라의 거리인 국소 평면의 특징 거리(

)를 생성하는 단계(S1240)를 포함할 수 있다. 보다 구체적으로, 중심 변환 좌표(Pc(x,y))의 국소 평면의 법선 벡터(

)의 성분을 (

)이라고 했을 때, 중심 변환 좌표(Pc(x,y))를 지나고 법선 벡터(n(x,y))를 가지는 평면은 수학식 3과 같이 표현 할 수 있다. 이 때, 국소 평면의 카메라로부터의 거리인 특징 거리(

)는 수학식 4를 통해 계산될 수 있다. In addition, in a method of detecting a feature of a local plane according to another embodiment, the processor 122 of the plane detection apparatus 100 performs a center transform coordinate (Pc(x,y)) and a normal vector (

) Using the feature distance of the local plane, the distance between the local plane and the camera (

) May include generating (S1240). More specifically, the normal vector of the local plane of the center transform coordinate (Pc(x,y)) (

) Of the component (

), a plane passing through the center transform coordinate (Pc(x,y)) and having a normal vector (n(x,y)) can be expressed as Equation 3. At this time, the feature distance, which is the distance from the camera on the local plane (

) Can be calculated through Equation 4.

13 shows a result of plane detection according to an embodiment of the present invention.

Referring to FIG. 13, the present invention enables detection of a plane in a depth image with very high accuracy as shown in a result using the plane detection device 100. Accordingly, in the embodiment of the present invention, precision is improved by detecting a planar area on a pixel-by-pixel basis. In addition, according to an embodiment of the present invention, the data processing speed is improved by calculating a planar factor in units of a local plane and grouping adjacent local planes having similar plane factors into the same plane to detect regions of each plane. In addition, according to the exemplary embodiment of the present invention, even when a depth image contains a lot of noise, reliable plane estimation is possible.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium includes, singly or in combination, program instructions, data files, data structures, etc. executed by the processor 122 so that the processor 122 calculates Equations 1 to 9 as described above. can do. The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. medium), and a hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be changed to one or more software modules to perform the processing according to the present invention, and vice versa.

Specific implementations described in the present invention are examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections. It may be referred to as a connection, or circuit connections. In addition, if there is no specific mention such as “essential” or “important”, it may not be a necessary component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, the spirit of the present invention described in the claims to be described later if one of ordinary skill in the relevant technical field or those of ordinary skill in the relevant technical field And it will be understood that various modifications and changes can be made to the present invention within a range not departing from the technical field. Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

100: plane detection device
110: video source
120: image processing unit
121: memory
122: processor
130: output interface
200: data receiving device
210: data processing unit
220: input interface

Claims (12)

Generating a local plane including one pixel among pixels in the depth image converted into a 3D coordinate system and surrounding pixels in a rectangular area around the one pixel;
In the rectangular area, a first average local vector of a plurality of vertical vectors passing through an upper peripheral pixel and a lower peripheral pixel centered on the one pixel, and a peripheral pixel located on the left side and a right side of the one pixel. Detecting a normal vector of the local plane according to an external product of a second average local vector of a plurality of horizontal vectors passing through a neighboring pixel, detecting a plane having the coordinates of the one pixel and the normal vector, and from the camera according to Equation 1 Detecting a feature distance that is a distance to the plane;
A difference between a normal vector in the local plane and a normal vector of another local plane including the one pixel and another pixel adjacent to the one pixel according to Equation 2, and a feature distance in the local plane and a feature distance in the other local plane Calculating a degree of similarity between the local plane and the other local plane on the basis of;
Grouping the local plane and the other local plane with each other if the similarity is greater than or equal to a predetermined similarity;
Determining the presence or absence of other surrounding pixels that are not compared with the one pixel, and if the other pixels do not exist, detecting already grouped local planes as an area of one plane;
Detecting a plurality of grouped planes that are not grouped from each other; And
Estimating step difference information of objects in the depth image by comparing average values of feature distances of each of the plurality of grouped planes;
Plane detection method through a depth image comprising a.
Equation 1

Equation 2

In Equation 1, (xc, yc) is a coordinate value of the one pixel, zc is a depth value of the one pixel, (an, bn, cn) is a component of the normal vector, and in Equation 2, n is a local The plane's normal vector and m is the other local plane's normal vector. The method of claim 1,
The surrounding pixels are pixels located outside the rectangular area N*N (N is an odd number of 3 or more) around the one pixel.
Plane detection method through depth image. delete delete delete delete The method of claim 1,
The inter-angle is a first degree of similarity, the difference between the feature distances is a second degree of similarity,
It is determined whether the first similarity is greater than or equal to a predetermined value,
If the first similarity is greater than or equal to a predetermined value, it is determined whether the second similarity is less than or equal to a predetermined value,
If the second similarity is less than or equal to a predetermined value, the local plane and the other local plane are grouped.
Plane detection method through depth image. delete delete delete At least one memory for storing instructions; And
At least one processor; Including,
The instructions are executable by the processor to cause the processor to perform operations,
The above actions are:
A local plane including one pixel of the pixels in the depth image converted to the 3D coordinate system and the neighboring pixels in a rectangular area around the one pixel is generated,
In the rectangular area, a first average local vector of a plurality of vertical vectors passing through an upper peripheral pixel and a lower peripheral pixel centered on the one pixel, and a peripheral pixel located on the left side and a right side of the one pixel. Detecting a normal vector of the local plane according to an external product of a second average local vector of a plurality of horizontal vectors passing through a neighboring pixel, detecting a plane having the coordinates of the one pixel and the normal vector, and from the camera according to Equation 3 Detect the feature distance of the plane of,
A difference between a normal vector in the local plane and a normal vector of another local plane including the one pixel and another pixel adjacent to the one pixel according to Equation 4, and a feature distance in the local plane and a feature distance in the other local plane Calculate the similarity between the local plane and the other local plane based on,
If the similarity is greater than or equal to a predetermined similarity, the local plane and the other local plane are grouped with each other,
The presence or absence of other pixels in the surrounding area not compared with the one pixel is determined, and if the other pixels do not exist, the grouped local planes are detected as an area of one plane,
Detects a plurality of grouped planes that are not grouped between each other,
Comparing the average value of the feature distances of each of the plurality of grouped planes to estimate step information of the objects in the depth image
Containing
Plane detection device.
Equation 3

Equation 4

In Equation 3, (xc, yc) is a coordinate value of the one pixel, zc is a depth value of the one pixel, (an, bn, cn) is a component of the normal vector, and in Equation 4, n is a local The plane's normal vector and m is the other local plane's normal vector. A non-transitory computer-readable recording medium storing the instructions according to claim 11.

Images