KR20160034756A - Bottom detection method and apparatus using the normal information based depth image - Google Patents

Abstract

The present invention relates to a method which detects a bottom area by using normal vectors of each pixel from a depth image captured from a three-dimensional camera. The method comprises the following steps: (a) receiving the depth image; (b) extracting a three-dimensional coordinate value of the pixel; (c) calculating the normal vectors of the pixels; (d) selecting a normal vector of the bottom; (e) rotation-converting the same; and (f) detecting the bottom area. After the step (f), the normal vector of the bottom is corrected by using the detected bottom pixel, and the steps (e) and (f) can be repeatedly performed.

Description

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to a depth detection apparatus and a method thereof,

The present invention relates to an image processing technique, and more particularly, to an apparatus and method for detecting a bottom region using a normal vector of each pixel in a depth image photographed by a three-dimensional camera.

Detection of the bottom region plays a very important role in tracking the motion of the object using depth image information obtained from the 3D camera. Therefore, it is necessary to quickly and accurately detect the bottom area of the input image.

As a conventional method for detecting such a floor area, in Patent Application No. 10-2013-0073736, image information photographed by a three-dimensional camera is received and at least three points adjacent to a minimum value of a height coordinate value among image information are selected , And a method of detecting the bottom area by calculating the coefficients of the plane equation passing through the selected point is proposed.

However, in such a method, when three points for detecting the bottom region are mistakenly detected, there is a problem that it is difficult to accurately detect the floor because the error of the floor becomes large.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide a method and an apparatus for searching a depth region by using spatial coordinate information of each pixel extracted from depth image information and normal vector information calculated therefrom, And to provide a method and an apparatus for accurately detecting the same.

According to another aspect of the present invention, there is provided a floor detecting apparatus including: a video input unit receiving a depth image of a building taken by a 3D camera; A coordinate extraction unit for extracting three-dimensional coordinate values of all the pixels in the input depth image; A normal vector calculation unit for calculating a normal vector at all pixels from the extracted coordinate values; A bottom normal selection unit for selecting a bottom normal vector from the set of the calculated normal vectors; A coordinate transformation unit for performing a rotation transformation such that the normal vector direction of the selected bottom coincides with the direction of the height axis (z-axis) of the three-dimensional coordinate system; And a bottom detector for detecting a bottom area using the three-dimensional coordinate value and the normal vector of each pixel according to the rotation transformation, and a bottom normal calculation unit for calculating a bottom normal vector using the detected bottom pixel, And may further include a correction unit.

In the bottom normal selection unit, the normal vector of the normal direction among the normal vectors may be selected as the normal vector of the bottom, as the normal vector of pixels having the height value (z value) of the three-dimensional coordinate values equal to or smaller than the threshold value th1 .

In the bottom detecting unit, a difference between a normal vector of the selected pixel and a bottom normal vector is a pixel having a z value equal to or less than a threshold value th2, It is possible to detect a pixel that is equal to or smaller than the threshold value th3.

According to another aspect of the present invention, there is provided a floor detection method including: (a) receiving a depth image of a building taken by a three-dimensional camera; (b) extracting three-dimensional coordinate values of all the pixels in the input depth image; (c) calculating a normal vector at every pixel from the extracted coordinate values; (d) selecting a bottom normal vector from the set of calculated normal vectors; (e) rotationally transforming the normal vector direction of the selected bottom to coincide with the height direction (z-axis) direction on the three-dimensional coordinate; And (f) detecting a bottom region using a three-dimensional coordinate value and a normal vector of each pixel according to the rotation transformation, wherein, after the step (f) The vector may be corrected, and the steps (e) and (f) may be repeated.

In the step (d), the normal vector of the normal direction among the normal vectors may be selected as the normal vector of the bottom, as the normal vector of pixels having the height value (z value) of the three-dimensional coordinate value or less.

In the step (f), a pixel having a z value equal to or less than a threshold value th2, the difference between the z value of each pixel according to the rotation transformation, and the minimum value, A pixel whose difference is equal to or less than the threshold value th3 can be detected as the bottom.

Further, the floor detection method according to the present invention can be implemented by executing a program stored in a computer-readable recording medium.

According to the present invention, the floor area is automatically detected through the image processing using the depth image information, so that passive operation such as floor designation by the user's input is unnecessary when the 3D camera is installed and the direction of the camera is changed It is possible to detect the bottom area adaptively even if the depth information is changed due to the change of the environment or the like.

In addition, the bottom normal is estimated using both the spatial coordinate information and the normal vector information extracted from the depth image information, and the rotation transformation is repeated, so that a more accurate bottom region can be detected using only one image.

1 is a block diagram showing the configuration of a floor detecting apparatus according to an embodiment of the present invention,
2 is a diagram illustrating a color image and a depth image photographed from the three-dimensional camera,
3 is a conceptual diagram for explaining the normal vector selection process of the floor,
FIG. 4 is a diagram illustrating a three-dimensional coordinate transformation of a depth image,
5 is a flowchart illustrating a bottom detection method according to an embodiment of the present invention.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear, and the same reference numerals will be used throughout the specification to refer to the same or like parts. use.

The terms used in this specification are used to appropriately express the preferred embodiment of the present invention, and this may vary depending on the user, the intention of the operator, or the practice of the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing the structure of a floor detecting apparatus using depth vector-based normal vectors according to an embodiment of the present invention. 1, the floor detection apparatus according to the present invention includes an image input unit 10, a coordinate extraction unit 20, a normal vector calculation unit 30, a floor normal selection unit 40, a coordinate transformation unit 50, And a floor detecting unit 60. [

The image input unit 10 receives a color image and a depth image captured by a three-dimensional camera installed in an area where monitoring is required, and provides image information on the color image and the depth image to the coordinate extraction unit 20. 2 shows an example of a color image and a depth image captured by a three-dimensional camera. At this time, the depth image includes distance information separated from the three-dimensional camera for each pixel of the two-dimensional image.

The coordinate extraction unit 20 extracts three-dimensional coordinates of each pixel from the depth image provided from the image input unit 10 and provides the three-dimensional coordinates to the normal vector calculation unit 30 and the bottom normal selection unit 40. Here, the three-dimensional coordinate system is an orthogonal coordinate system composed of the x-axis, the y-axis, and the z-axis. As shown in Fig. 2, the image plane is set as the zx plane and the y- . Accordingly, the three-dimensional coordinates (P _{i, j} ) at any pixel (i, j) can be expressed by (x, y, z).

The normal vector calculation unit 30 calculates a normal vector in all the pixels using the three-dimensional coordinates of each pixel extracted by the coordinate extraction unit 20 and supplies the normal vector to the bottom normal selection unit 40 and the bottom normal selection unit 40. [ And provides it to the detection unit 60.

Here, the normal vector (N _{i, j)} in any pixel (i, j) may be calculated by the cross product by using the three-dimensional coordinates of the adjacent pixel in accordance with equation (1) below.

[Equation 1]

_{N i, j = (P (} i + 1), j - P i, j) × (P i, (j + 1) - P i, j)

In addition, the normal vector calculation unit 30 may calculate the normal vectors at all the pixels, and then normalize the normal vectors by adding all the normal vectors of the pixels adjacent to the normal vector of each pixel according to the following Equation (2) have. At this time, it is preferable to exclude the portion where the depth value of the adjacent pixel is 0 when adding the silver.

&Quot; (2) "

_{N'i, j = Normalize (N} (i + 1), (j + 1) + N (i + 1), j + N (i + 1), (j-1) + N i, (j + 1 _{) + N i, j + N} i, (j-1) + N (i-1), (j + 1) + N (i-1), j + N (i-1), (j-1) )

As a result, the normal vector in each pixel becomes a unit vector of size 1, and the amount of computation in the image processing step thereafter can be reduced, and the normal vector can be processed more smoothly in the object in the image. On the other hand, acceleration using a parallel program such as a GPU can be used for smoothing and calculating the normal vector.

Based on the three-dimensional coordinates of each pixel extracted by the coordinate extraction unit 20 and the normal vector of each pixel calculated by the normal vector calculation unit 30, the bottom normal selection unit 40 selects a bottom normal among the normal vectors of all pixels, The normal vector having the greatest property is selected as the bottom normal vector, and the rotation vector is provided to the rotation conversion unit 50.

In the case of a floor existing in an image, there is a high possibility that a height value (z value) among the three-dimensional coordinate values is generally composed of pixels having a small value. This is because the surveillance camera is installed on the ceiling or above the wall to shoot the surveillance zone, and the bottom of the surveillance image is generally located downward because the surveillance camera is directed downward at a certain angle.

In addition, the normal vector at the bottom indicates the direction perpendicular to the floor. The normal vector of all objects parallel to the floor, such as a desk surface, also points in the same direction as the normal vector at the bottom. If so, as shown in Fig. 3, the bottom normal vector is likely to be the normal vector of the most normal direction among all the normal vectors. However, in this case, it is preferable to exclude the normal vector of the wall or ceiling .

Considering these two factors of the bottom property, the normal vector of the pixel estimated as the bottom among the normal vectors of all the pixels is selected.

Specifically, if the normal vector of each pixel is moved so as to be the starting point, it corresponds to one point on the spherical surface having the radius of 1 with the origin as the center. At this time, if we find the place where the normal vector is gathered on the spherical surface, that is, the mode is found, a vector approximating the normal vector can be found. To find these modes simply, we divide the spheres of radius 1 into zones and use the number of pixels belonging to each zone to find the zone to which the bottom norm belongs.

In simple terms, a three-dimensional space containing a sphere is divided into square cells and a cell containing the largest number of normal vectors is used as an approximated normal vector candidate. Cells that are likely to be wall surfaces are excluded. At this time, the assumption is that the camera is not tilted to the left or right by more than 45 degrees. The value obtained by normalizing the center of the obtained cell is taken as a representative value of the normal vector.

When calculating, we exclude the top 3% pixels on the image. p3 can be set to about 20 as a parameter, and it is unlikely that such a pixel is bottomed out, and even if the bottom is excluded, it does not cause an error in floor detection. However, in this case, it should be assumed that the camera is not looking up or upside down.

The coordinate transformation unit 50 performs rotation transformation so that the normal vector direction of the floor estimated by the bottom normal selection unit 40 coincides with the direction of the height axis (z-axis) of the three-dimensional coordinate system to transform the coordinates of each pixel, And provides it to the bottom detector 60. That is, as shown in FIG. 4, the bottom surface on the extracted three-dimensional coordinate forms a certain angle not parallel to the xy plane due to the installation angle of the camera, the difference in distance between the camera and the floor, The direction of the normal vector at the bottom does not coincide with the z-axis direction. Therefore, coordinate conversion is required to coincide the normal vector direction and the z-axis direction of the bottom so that the bottom area can be easily detected.

At this time, when the rotation transformation matrix is R, the matrix R can be calculated by the following equation (3), and the coordinates (P ? _{, j} ) of each transformed pixel are calculated according to equation (4).

&Quot; (3) "

^{[N T (1,0,0) T (} N × (1,0,0)) T] R = [(0,0,1) T (1,0,0) T (0,1,0) ^T ]

^{R = [N T (1,0,0)} T (N × (1,0,0)) T] -1 [(0,0,1) T (1,0,0) T (0,1, 0) ^T ]

Where N is the normal vector of the selected bottom, [] is the 3 * 3 matrix, [] ^-1 is the inverse matrix, and ^T is the transpose matrix.

&Quot; (4) "

P _{i, j} = R * P _{i, j}

The bottom detecting unit 60 detects the bottom area based on the coordinate values of the pixels converted by the rotation converting unit 50 and the normal vectors of the pixels calculated by the normal vector calculating unit 30.

That is, the bottom region can be detected by selecting a pixel whose z value is small in the rotation-transformed coordinate values of all the pixels and has a normal vector direction similar to the normal vector direction of the bottom. For example, a pixel satisfying the following expression (5) can be detected as a bottom area.

&Quot; (5) "

_{(P'i, j z -.} P'min _z) * (P'i, j - P'min _z) - p4 * (N ij - N) * (N ij - N) <p5

Here, P'i _{, j} . and _z- N _ij with each pixel (i, j) rotating the transformed z coordinate values and a normal vector, _{P'_z min} is the minimum value of the z coordinate values of the whole pixels in the rotated coordinate conversion in, N is Is the normal vector of the estimated floor. And, p4 and p5 represent preset parameter values.

The bottom normal correction unit 70 calculates the normal vector N 'of the corrected bottom using the detected bottom pixel and provides it to the rotation conversion unit 50.

For example, the bottom normal correction unit 70 calculates a plane equation from the coordinate information of the detected bottom pixel. At this time, if a plane equation is ax + by + cz + d = 0, a, b, c are calculated to minimize the following expression (6).

&Quot; (6) &quot;

_{Sumation (x, y, z)} in P (ax + by + cz + 1) 2

Where Sumation represents the sum of the equations for all bottom pixels.

From the values of a, b and c calculated from Equation (6), the corrected bottom normal vector N 'can be obtained through the normalization process, and the rotation transformation and the floor detection process are repeated using the normal vector. Through such a repetition routine, the floor area can be detected more accurately.

Hereinafter, a floor detection method using depth vector-based normal vectors according to an embodiment of the present invention will be described with reference to FIG. However, the parts overlapping with those described in the embodiment of the floor detection image processing apparatus will be omitted.

In order to detect the floor using the depth vector based normal vector according to the present invention, the depth image of the inside of the building photographed by the 3D camera is input to the image input unit 10 first. The input depth information includes depth information. Meanwhile, as a three-dimensional camera, a kinect developed by Microsoft Corporation can be used.

The inputted depth image is provided to the coordinate extracting unit 20, and the coordinate extracting unit extracts the three-dimensional coordinates of each pixel (S10). At this time, the coordinate system uses an orthogonal coordinate system, and three-dimensional coordinates can be formed on the x, y, and z axes by using the depth information at each pixel. In the embodiment of the present invention, the image plane is set as a zy plane and the y-axis is extracted as a three-dimensional coordinate by reflecting the depth information.

The extracted three-dimensional coordinates are calculated by the normal vector calculation unit 30, and the normal vectors at the respective pixels are calculated through the outer product using the three-dimensional coordinates of the adjacent pixels (S30). Thereafter, the calculated normal vector smoothing and normalization process can be further performed to reduce the calculation amount and correct floor detection.

When all the normal vectors at each pixel are calculated, a normal vector having the most properties of the double bottom is selected (S30). For example, in the case of a bottom existing in an image, a height value (z value) among three-dimensional coordinate values is composed of pixels, and a normal vector in a pixel indicates a direction perpendicular to the bottom surface, You can choose a vector.

Thus, a pixel having a normal vector in the same direction as the normal vector of the selected bottom may include not only the bottom but also a desk surface parallel to the bottom surface. Therefore, additional work is required to detect a pixel composed of only the bottom surface Do.

Then, the rotation transformation is performed so that the direction of the normal vector of the selected bottom coincides with the z-axis direction which is the height axis on the coordinate axis (S40). Thus, the bottom surface is parallel to the xy plane and the coordinate values of each pixel are transformed.

Then, the floor is detected using the transformed coordinate value and the normal vector stepped up (S50). That is, after the rotation transformation, the bottom surface is parallel to the xy plane. If a pixel whose z-value is close to the minimum value is selected as the bottom plane, and the direction of the normal vector of the selected pixel coincides with the z- The floor can be detected.

On the other hand, the bottom normal vector is corrected to detect the finer bottom region (S60), and the rotation transformation step S40 and the bottom detection step S50 can be repeated based on the corrected normal vector.

Thus, by automatically detecting the floor surface using the depth image information input from the 3D camera, manual operation such as floor design is not required when installing the 3D camera, and the depth information is changed according to the environment change, The floor surface can be detected adaptively when the direction of the floor surface is changed.

Meanwhile, the floor detection method using the depth image-based normal vector described above can be implemented by a software program and recorded on a computer-readable recording medium.

For example, the recording medium may be a hard disk, a flash memory, a RAM, a ROM, or the like embedded in each reproduction apparatus, or an external optical disk such as a CD-R or a CD-RW, a compact flash card, a smart media, have.

The functional operations and implementations described in the present specification may be embodied in digital electronic circuitry, computer software, firmware, or hardware, or a combination of any of the foregoing. The implementations described in the present disclosure may be implemented as one or more computer program products, that is, one or more modules associated with computer program instructions encoded on a program storage medium of the type for control of, or for execution by, Lt; / RTI &gt;

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, This is possible. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

10: image input unit 20:
30: normal vector calculation unit 40: bottom normal selection unit
50: rotation converting section 60: bottom detecting section

Claims (9)

(a) receiving a depth image of a building taken by a 3D camera;
(b) extracting three-dimensional coordinate values of all the pixels in the input depth image;
(c) calculating a normal vector at every pixel from the extracted coordinate values;
(d) selecting a bottom normal vector from the set of calculated normal vectors;
(e) rotationally transforming the normal vector direction of the selected bottom to coincide with the height direction (z-axis) direction on the three-dimensional coordinate; And
(f) detecting a bottom region using a three-dimensional coordinate value and a normal vector of each pixel according to the rotation transformation;
Based depth vector-based normal vector. The method according to claim 1,
Wherein the normal vector of the bottom is corrected using the detected bottom pixel after the step (f), and the steps (e) and (f) are repeatedly performed. Way. The method of claim 1, wherein the step (d)
Wherein a normal vector in a most diagonal direction of the normal vector is selected as a normal vector at the bottom as a normal vector of a pixel having a height value (z value) of the three-dimensional coordinate values equal to or less than a threshold value th1. A floor detection method using vector. 4. The method of claim 3, wherein step (f)
Wherein a difference between a normal vector of a pixel and a bottom normal vector of the selected pixel is equal to or less than a threshold th3 and a difference between a minimum value and a z- And a pixel having a depth less than or equal to a predetermined depth is detected as a bottom. A video input unit receiving a depth image of a building taken by a 3D camera;
A coordinate extraction unit for extracting three-dimensional coordinate values of all the pixels in the input depth image;
A normal vector calculation unit for calculating a normal vector at all pixels from the extracted coordinate values;
A bottom normal selection unit for selecting a bottom normal vector from the set of the calculated normal vectors;
A coordinate transformation unit for performing a rotation transformation such that the normal vector direction of the selected bottom coincides with the direction of the height axis (z-axis) of the three-dimensional coordinate system; And
A bottom detector for detecting a bottom area using a three-dimensional coordinate value and a normal vector of each pixel according to the rotation transformation;
Based depth vector-based normal vector. 6. The method of claim 5,
And a bottom normal correction unit for calculating a normal vector of the corrected floor using the detected bottom pixel. 6. The apparatus of claim 5, wherein the bottom normal selection unit comprises:
Wherein a normal vector of a direction in the most positive direction of the normal vector is selected as a normal vector of a bottom as a normal vector of pixels having a height value (z value) of the three-dimensional coordinate values equal to or less than a threshold value. Floor detecting device. 7. The apparatus according to claim 6,
Wherein a difference between a normal vector of a pixel and a bottom normal vector of the selected pixel is equal to or less than a threshold th3 and a difference between a minimum value and a z- Wherein the pixel is detected as a bottom pixel based on a depth image-based normal vector. A computer-readable recording medium on which a program for executing the method according to any one of claims 1 to 4 is recorded.

Images