KR20210108283A - A method of improving the quality of 3D images acquired from RGB-depth camera - Google Patents

Abstract

The present invention relates to a method for improving the quality of a three-dimensional image obtained from a depth image camera, which improves the quality by processing a three-dimensional image obtained from a texture depth (RGB-depth) camera on a multi-view camera system. The method of the present invention comprises the steps of: (a) obtaining a multi-view depth image and a texture image; (b) generating a binary image with respect to a brightness value of the texture image, and generating a mask image from the generated binary image; (c) applying the mask image to the texture image, and filtering the same; (d) performing primary correction by adding an alpha channel to the filtered texture image; (e) filtering a depth image with the mask image; (f) applying time-average filtering to the filtered depth image to be corrected; (g) secondly correcting the firstly corrected texture image using a histogram; (h) generating point cloud data from a corrected multi-view depth image, and removing surface noise for correction; and (i) configuring a curved surface from the corrected point cloud data. In accordance with the present invention, a more accurate three-dimensional image can be obtained by relieving noise in a depth image, noise in illumination and a boundary surface between adjacent color images, noise in point cloud data, noise in surface data, etc.

Description

{ A method of improving the quality of 3D images acquired from RGB-depth camera }

The present invention improves the quality by processing a 3D image acquired through a texture depth (RGB-Depth) camera on a multi-view camera system, but improves noise in depth images, compensates for lighting between adjacent texture images (texture, color), and It relates to a method for improving the quality of a 3D image obtained from a depth image camera, which performs processes such as interpolation of interfaces, noise improvement of point cloud data converted from a depth image, and noise improvement of mesh data reconstructed from point cloud data.

In general, the importance of 3D space and 3D object detection and recognition technology is emerging in the fields of computer vision, robotics, and augmented reality. In particular, since it is possible to acquire RGB images and depth images in real time through an image sensor using Microsoft's Kinect method, many changes have been made in object detection, tracking, and recognition research [non-patented] Documents 1 and 2].

The entire process of recognizing a 3D object in RGB-D (RGB image and depth image) is shown in FIG. 1 . That is, as shown in FIG. 1 , a depth frame is acquired from a depth camera, and a texture image is acquired from an RGB camera (or texture camera). Then, processing is performed on two-dimensional images such as depth images and texture images, and a point cloud (or point cloud, point cloud) is extracted from these images. Then, the process of generating a mesh from the point cloud is performed. Through this process, a 3D object composed of a 3D mesh is finally extracted.

However, in the process shown in FIG. 1 , the following problems should be solved.

First, there is a problem of illumination noise caused by structured light used when acquiring a depth image, and a problem of boundary noise generated when synthesizing an image by DIBR.

That is, the structured light image-based depth imaging camera is composed of a normal camera and a structured light irradiation device having a certain visual deviation. A red (660 [nm]) visible ray laser or an infrared (780 [nm]) laser is commonly used as a structured light generator to be distinguished from ambient light, and it is a device that converts a point type laser into a structural type. As a representative method, a method of projecting an infrared laser onto a pattern engraved on glass using an optical diffraction element is used. Such an infrared structured light method may include a large error in the depth image acquisition result due to phenomena such as reflection, blurring, frequency overlap, and the like of structured light projected on an object and ambient lighting noise.

In addition, boundary noise generated during image synthesis using DIBR (Depth Image Based Rendering) is noise generated when pixels originally belonging to the foreground region are scattered into the background. This is caused by blurring in Such boundary noise acts as a major cause of deterioration of the 3D image quality.

In addition to the causes of illumination noise and boundary noise of the depth image, the noise in the above situation fluidly acts on all depth image frames acquired in the time axis from the RGB-Depth camera. This is a time-base noise problem.

Next, there is the problem of lighting mismatch caused by the multi-view camera. The biggest factor causing the lighting mismatch is the location of the different cameras in the multi-view camera system. According to different viewpoints of the camera, lighting environments between viewpoint images are also different from each other, and even the same object may have different configurations and a degree of reflected light present in the acquired image depending on viewpoints. As such, illumination mismatches occurring in the same object between viewpoints are referred to as local illumination mismatches. The lighting mismatch phenomenon lowers the correlation between images of adjacent viewpoints, resulting in deterioration of quality in restoring images of adjacent viewpoints to surface data.

Also, there is a problem of surface noise removal on point cloud data. That is, the time-of-flight measurement method using infrared rays, the structured light imaging method, etc. have disadvantages in that the measurement precision varies depending on the distance between the camera and the photographed object and the shape of the object. All noises act in a complex way, and for this reason, it requires a high degree of difficulty in specifying the noise of the surface and boundary when the depth image is converted to point cloud data.

In addition, there is a noise problem that occurs on the plane data when the object is composed of a polygonal mesh or the like from the point cloud data.

Therefore, there is a need for a 3D image generation technology that solves the above problems.

W. Lee, N. Park, W. Woo, “Depth-assisted 3D Object Detection for Augmented Reality,” International Conference on Artificial reality and Telexistence, pp. 126-132, 2011. Park, Y., Lepetit, V., Woo, W., "Texture-less object tracking with online training an RGB-D camera", Int. Symp. Mixed and Augmented Reality (ISMAR), pp. 121-126 (2011)

SUMMARY OF THE INVENTION An object of the present invention is to solve the above problems, and a 3D object is generated from depth and texture images acquired through a texture-depth (RGB-D) camera on a multi-view camera system, but in the depth image Depth image, which performs processes such as noise improvement, illumination compensation between adjacent color images (textures) and interpolation of boundary surfaces, noise improvement of point cloud data converted from depth images, and noise improvement of mesh data restored from point cloud data. An object of the present invention is to provide a method for improving the quality of a 3D image obtained from a camera.

In order to achieve the above object, the present invention relates to a method for improving the quality of a 3D image obtained from a depth image camera, comprising the steps of: (a) obtaining a multi-view depth image and a texture image; (b) generating a binary image based on the brightness value of the texture image, and generating a mask image from the generated binary image; (c) filtering by applying the mask image to the texture image; (d) performing primary correction by adding an alpha channel to the filtered texture image; (e) filtering the depth image with the mask image; (f) correcting the filtered depth image by applying time-average filtering; (g) secondarily correcting the firstly corrected texture image using the histogram; (h) generating point cloud data from the corrected multi-view depth image and correcting it by removing surface noise; and, (i) constructing a curved surface from the corrected point cloud data.

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, in the step (b), the binary image is characterized in that noise of the binary image is removed by repeating the close operation during the morphological operation on the binary image. .

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, in the step (c), the brightness value of the mask area is maintained in the texture image, and the brightness value of the area outside the mask is set to the smallest characterized in that

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, in the step (d), an alpha channel of transparency is added to the filtered texture image, and an alpha value of a region over a specific distance in the texture image is set to 0, but the region greater than a specific distance is an region in which the depth value of the region of the corresponding depth image is greater than or equal to a preset threshold distance.

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, the present invention maintains the depth value of the mask region in the depth image and sets the maximum depth value of the region outside the mask in the step (e). characterized in that

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, the present invention is characterized in that, in the step (f), time-average filtering is performed only on the filtered region corresponding to the mask region in the depth image. do.

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, in the step (f), the pixel values of the same position of the current frame of the depth image and at least one frame of the previous time are averaged and correcting the pixel value of the depth image with an average value.

In addition, in the method for improving the quality of a 3D image obtained from a depth image camera, the present invention is characterized in that the texture image is corrected by performing a histogram specification method in step (g).

In addition, the present invention provides a method for improving the quality of a 3D image obtained from a depth image camera, in the step (h), clustering is performed on the point cloud data, calculating a normal vector of each point, and a normal vector between adjacent points. It is characterized in that a point (hereinafter, a boundary point) having a higher correlation with a neighboring point than a predetermined reference value is detected through the dot product, and a point outside the boundary region is deleted with respect to a boundary formed by the detected boundary points.

In addition, the present invention provides a method for improving the quality of a 3D image obtained from a depth image camera, in the step (i), when constructing a curved surface, correcting it using the distance between vertices, but with respect to each surface of the curved surface A face is formed only when one of at least two sides facing the face is less than or equal to the threshold value, and the minimum (min) and maximum (max) values are calculated for the x, y, and z coordinates of the vertices constituting each face. It is characterized in that the output is limited by setting a threshold value in the volume of the AABB box.

In addition, the present invention relates to a computer-readable recording medium in which a program for performing a method for improving the quality of a 3D image obtained from a depth image camera is recorded.

As described above, according to the method for improving the quality of a 3D image obtained from a depth image camera according to the present invention, noise in a depth image, illumination and interface noise between adjacent color images, noise of point cloud data, noise of plane data, etc. By improving, the effect of obtaining a more accurate 3D image is obtained.

1 is a flowchart illustrating a method of generating a 3D object from a texture and depth image according to the prior art;
Figure 2 is a diagram showing the configuration of the entire system for implementing the present invention.
3 is a flowchart illustrating a method for improving the quality of a 3D image obtained from a depth image camera according to an embodiment of the present invention.
4 is a diagram illustrating a process of generating a mask image and masking a texture image according to an embodiment of the present invention;
5 is an exemplary view of an image from which lighting noise is removed by applying a mask according to an embodiment of the present invention;
6 is an exemplary view of an image from which boundary noise is removed by applying a mask according to an embodiment of the present invention;
7 is a graph illustrating a change in the value of one pixel on the time axis before and after time average filtering according to an embodiment of the present invention.
8 is a graph showing a change in the value of the 160th column before and after filtering according to an embodiment of the present invention.
9 is an exemplary view of a color image or a texture image to which illumination compensation is applied according to an embodiment of the present invention.
10 is a detailed flowchart illustrating a step of generating and correcting point cloud data according to an embodiment of the present invention.
11 is a diagram illustrating a method for specifying surface noise of point cloud data according to an embodiment of the present invention.
12 is a detailed flowchart illustrating a face data generation step according to an embodiment of the present invention.
13 is an exemplary diagram illustrating a noise type of surface data according to an embodiment of the present invention.
14 is an exemplary diagram of a vertex image constituting a mesh according to an embodiment of the present invention.
15 is an exemplary view of the creation of an AABB box according to an embodiment of the present invention.
16 is an exemplary view of a noise plane specific image through volume calculation of an AABB box according to an embodiment of the present invention.
17 is an exemplary view of improving noise of the surface data of FIG. 13 according to an embodiment of the present invention;

Hereinafter, specific contents for carrying out the present invention will be described with reference to the drawings.

In addition, in demonstrating this invention, the same part is attached|subjected with the same code|symbol, and the repetition description is abbreviate|omitted.

First, examples of the configuration of the entire system for implementing the present invention will be described with reference to FIG. 2 .

As shown in FIG. 2 , the method for improving the quality of a 3D image obtained from a depth image camera according to the present invention receives a multi-viewpoint depth and texture image 60 photographed by a camera system 20 as an input to obtain a 3D object or 3D image. It may be implemented as a program system on the computer terminal 30 for generating a dimensional image. That is, the method for improving the quality of a 3D image may be configured as a program and installed in the computer terminal 30 to be executed. A program installed in the computer terminal 30 may operate as one program system 40 .

Meanwhile, as another embodiment, the method for improving the quality of a 3D image may be implemented with a single electronic circuit, such as an ASIC (application specific semiconductor), in addition to being configured as a program and operated in a general-purpose computer. Alternatively, it may be developed as a dedicated computer terminal 30 that exclusively processes only matching point clouds in multi-view depth and color images. This will be referred to as the 3D image quality improvement system 40 . Other possible forms may also be implemented.

On the other hand, the camera system 20 is composed of a plurality of RGB-D cameras or texture-depth cameras 21 that take pictures of the object 10 from different viewpoints.

Each RGB-D camera 21 also includes a depth camera and a texture camera (or RGB camera, color camera). The texture camera is a conventional RGB camera, and acquires a texture image (or a color image, a color image) or a texture image 62 of the object 10 .

Also, the depth camera is a camera that measures the depth of the object 10 , and outputs a depth image or a depth image 61 by measuring depth information. In particular, the depth camera is a structured light image-based depth imaging camera having a constant visual deviation from a normal camera. That is, a laser of a certain pattern is projected onto an object, and a distorted pattern of the reflected light is detected to measure the depth. Preferably, the structured light uses a red (660 [nm]) visible light laser or an infrared (780 [nm]) laser. As an example, structured light may be formed by projecting an infrared laser onto a pattern engraved on glass using an optical diffraction element.

The multi-viewpoint depth image 61 and the texture image 62 photographed by the camera system 20 are directly input to and stored in the computer terminal 30 , and are processed by the 3D image quality improvement system 40 . Alternatively, the multi-viewpoint depth image 61 and the texture image 62 are pre-stored in the storage medium of the computer terminal 30 and read the depth-texture image 60 stored by the 3D image quality improvement system 40 . may be entered.

An image is made up of consecutive frames in time. For example, if the frame of the current time t is called the current frame, the frame of the immediately preceding time t-1 is called the previous frame, and the frame of t+1 is called the next frame. Meanwhile, each frame has a texture image (or color image) and a depth image (or depth information).

In particular, the number of multi-view RGB-D cameras 21 is photographed at different viewpoints with respect to the object 10, and at a specific time t, multi-view depth and texture images 61 and 62 as many as the number of cameras are obtained. .

On the other hand, the depth image 61 and the texture image 62 are composed of consecutive frames in time. One frame has one image. Also, the images 61 and 62 may have one frame (or image). That is, the images 61 and 62 correspond to a single image.

Generating a 3D image or object from a multi-viewpoint depth image and a texture image means detecting each depth/texture frame (or image), but unless there is a need for special distinction below, the term image or image to mix

Next, a method for improving the quality of a 3D image obtained from a depth image camera according to an embodiment of the present invention will be described with reference to FIG. 3 .

First, a multi-view depth image and a texture image (or color image) are obtained from the camera system 20 ( S10 ). That is, the multi-view depth and texture images are images shot from a plurality of RGB-D cameras 21 installed from different viewpoints.

Preferably, the texture image (or color image, color image) is an RGB image. The depth image and the texture image captured by each RGB-D camera 21 are images captured at the same viewpoint. In addition, the depth image is a depth image acquired based on structured light.

The number of multi-views is M (at least 2 or more), and all M texture images and depth images of each view are input. In addition, each image is input as consecutive frames in time.

Next, a mask image is generated by generating a binary image of the texture image (S20). That is, a binary image of the texture image is generated and corrected, and a mask image is generated using the corrected binary image. Specifically, a threshold value is set based on a brightness value in a texture image, and a binary image is generated by binarizing the texture image with the set threshold value. As an example, a threshold value or a brightness value in the threshold range 126 or 126 to 255 is set. That is, a region greater than or equal to the threshold value of 126 or included in the threshold range of 126 to 255 is binarized.

In addition, the binary image is corrected so that noise of the binary image is removed by repeating a closing operation during the morphological operation on the binary image. That is, areas (holes in binary images, etc.) lost by reflected light on the skin during the binarization process are restored through repetition of the closing operation during the morphology operation.

In addition, a mask image or a mask image is generated using the corrected binary image. That is, after binarization, a mask image is generated by inverting the binary image. The mask image has 255 on the inside of the object's outline and 0 on the outside. On the other hand, such a binarization method or a method of setting a mask image is only an example, and may have various other forms.

In this step, the mask image or the process of generating the mask image has the meaning of separating the background and the foreground from the texture image. An example of the mask image generated in this way is shown in the “Mask Generation” image of FIG. 4 .

Next, the original texture image (or the original color image) is filtered (masked) with the mask image to exclude the mask area from the texture image (S31).

That is, by masking the texture image (or the original texture image) with a mask image, the mask region in the texture image maintains the original image, and the region other than the mask region in the texture image sets the smallest brightness value. That is, the area outside the mask is set to 0.

Next, an alpha channel of transparency is added to the masked texture image, and an alpha value of an area over a specific distance is set to 0 (S32).

First, an alpha channel of transparency is added to the texture image. That is, the A channel of transparency is added to the texture image in addition to the RGB channel. In this case, the default value of the alpha channel of transparency is set to 1 (completely opaque). Transparency has a value from 0 to 1, when it is 0, it is completely transparent, when it is 1, it is completely opaque.

Align between depth image and texture image. That is, the positions of the pixels are matched to each other. The depth map and texture image captured by the RGB-D camera have different degrees of distortion depending on the resolution and the lens aperture of the camera being photographed, so it means matching them.

In a specific region or pixel of the depth image, if the depth value or Z value is greater than or equal to a preset threshold distance, the alpha channel value (or alpha value) of the texture image corresponding to the region or pixel is set to fully transparent or 0 do. The object is close to the critical distance, and the background is far from the critical distance. Therefore, the object can be extracted through the critical distance. That is, the alpha channel of the background area is set to be transparent.

As another embodiment, a depth binary image (or depth mask image) is generated by a threshold distance from the depth image, and the alpha channel image is masked with the binary image or the depth mask image.

Previously, the correction process ( S20 , S31 , and S32 ) of the texture image is illustrated in FIG. 4 . By the process of FIG. 4 , lighting noise and boundary noise are removed from the texture image. In addition, images from which illumination noise and boundary noise are removed by applying a mask are exemplified in FIGS. 5 and 6, respectively.

It is illustrated in the first image and the third image of FIG. 5 . That is, a small image in the lower right corner of the first image of FIG. 5 is a depth image. When the texture image is aligned and mapped to the depth image of FIG. 5 , it is output like the first large image of FIG. 5 . At this time, since the threshold value (threshold distance) according to the distance is not applied to the first image of FIG. 5 , the partition of the background is output together. If only objects within 1M are output in the first image of FIG. 5 , an image from which partitions are removed is output as shown in the second image of FIG. 5 .

In addition, if only the color image mask is applied to the second image of FIG. 5 , there is data in the depth information, but the transparency becomes 0 because the mask is applied in the texture image. This is output. The mask can be equally applied in the depth information map. In this case, noise data is not simply made transparent, but noise data is deleted from the actual depth information.

That is, both applying the mask to the depth image and applying the mask to the texture image are performed. When applied to color images, noise becomes transparent, and when applied to depth information, noise data is deleted.

Meanwhile, in the case of a multi-view image, if there are M multi-view depth and texture images, the above process is performed for each of the M images.

Next, preprocessing such as median blurring is performed on the depth image (S41).

Object padding is a process that performed an erosion operation. All three methods are available: a method of finding and padding an outer line to make the boundary thicker and clipping, a method of eroding inward with an erode method, and a masking method. In the present invention, the masking method is unified.

Median blurring equalizes data. The median method uses a method of taking an average value using a 3×3 window.

Next, before the depth image, filtering is performed with the mask image obtained in step S20 (42).

That is, in the depth image, the mask area (area of the depth image corresponding to the mask area of the mask image) is maintained and the area outside the mask is excluded. Preferably, the depth value of the region outside the mask in the depth image is set to be the largest.

Next, temporal average filtering is applied to the depth image (S50). In particular, time average filtering is performed only on the filtered depth image. In other words, time average filtering is performed only in the mask region in the depth image.

Preferably, at least M (preferably 4) frames (time t-4, t-3, t-2, t-1, etc.) of the previous time for the current frame (time t frame) of the depth image frame) and take the average value. In particular, the pixel values of the same position are averaged.

In this case, the current frame is also included. For example, filtering is performed using a total of five depth images with reference to four previous frames in the current frame. Each pixel value of the final depth image is determined as an average value of each pixel of the five depth images.

Through the above process, it is possible to remove time axis noise that is flexibly applied to all depth image frames acquired in the time axis from the RGB-D camera 21 .

7 shows the change of the value of one pixel on the time axis before and after the time average filtering, respectively, and FIG. 8 shows the change of the value of the 160th column before and after filtering.

Next, the texture image is corrected using the histogram (S60).

Since multi-view texture images are obtained by different camera positions, there is a problem in that the correlation of objects between texture images at each viewpoint is deteriorated due to mismatch of lighting according to each position. Therefore, for multi-view texture images, illumination compensation and interpolation between adjacent images should be performed.

To solve this problem, the histogram matching method is applied. This is used to improve the contrast (contrast) of some areas of the image by including the histogram of a specific shape in the histogram of the generated digital image. A color image or texture image of an RGB-D camera on a multi-view camera system is acquired, and a cumulative histogram of the image is matched with a cumulative histogram of a predetermined reference image. In this way, the texture quality can be improved by compensating for the mismatch of the lighting component of the texture image.

First, a histogram of an input color image or texture image is generated (S61).

Next, the histogram of the texture image is smoothed (S62). That is, the function T of the normalized cumulative frequency is obtained for smoothing the histogram of the color image, and then the following transformation equation is obtained. Here, P is the pixel value of the original image, and q is the smoothing value.

[Equation 1]

The methods described here are collectively referred to as histogram specification, histogram matching, histogram specification, and the like. That is, when there is a reference image and the pixel values of the reference image are the same, simple cumulative addition is performed. That is, the pixel value is stored in the form of the number of pixels. As an example, brightness 1 is the number of pixels having brightness 1, and brightness 2 is the number of pixels having brightness 2.

Next, a histogram (or a uniformly distributed histogram) of the smoothed color image (texture image) is obtained (S63). A uniformly distributed histogram of the input color image (Texture) is obtained by smoothing the input color image (Texture) based on the conversion equation. That is, the histogram of the smoothed color image is a uniformly distributed histogram.

Next, a normalized cumulative frequency function G of the color image histogram of an adjacent view is obtained, a transformation equation having an inverse transform function is obtained, and then the color image of an adjacent view is smoothed (S64). Here, Z is the brightness value of the color image histogram of an adjacent view, and v is the smoothing value.

[Equation 2]

At this time, the definition of the adjacent view image is made in pairs such as left for front, right for front, rear for right, and rear for left, and proceeds individually.

Next, a uniformly distributed histogram is obtained by performing smoothing of color images of adjacent viewpoints (S65).

Next, the inverse transform function is obtained by de-smoothing the smoothed adjacent view color image histogram (S66). Here, the inverse transform function becomes the actual lookup table.

[Equation 3]

Next, the histogram of the smoothed original color image is converted into a histogram of the color image of an adjacent view by using the inverse transform function obtained in the previous step (S46) (S67).

[Equation 4]

The previous process is histogram matching and specification method. This is a method of matching the shape of an adjacent image histogram to a reference histogram using the generated lookup table. The histogram specification is applied sequentially to each camera to make the entire image uniform.

Next, point cloud data is generated from multi-view depth images (M depth images), and surface noise is removed ( S70 ). The entire process of generating and correcting the point cloud data is shown in FIG. 10 .

First, point cloud data is generated from the depth image. A conventional method is applied to a method of generating point cloud data from a multi-view image. Therefore, a specific method of generating the point cloud data is omitted. In this case, point cloud data is generated from the depth image (M depth images) of the multi-viewpoint.

In this case, the measurement precision of the time-of-flight measurement method using infrared rays, the structured light imaging method, etc. varies depending on the distance between the camera and the photographed object and the shape of the object. Therefore, surface noise occurs when generating point cloud data by such noise.

In the related art, the image noise removal method includes a data clustering method for specifying a normal and a correlation with surrounding data in addition to averaging and determining a median.

go. To remove noise from the point cloud data, clustering is performed to classify similar data among adjacent data, and then the normal vector of each point is found.

me. A point having a low correlation with a neighboring point is calculated through the dot product of a normal vector between neighboring points. Through this process, only points with high correlation with the surrounding point cloud remain. This point becomes the boundary point.

all. In this process, the external point cloud removal method is a process to determine the points that have low correlation with the surrounding points even though the original object, such as arms and legs, is highly curved and is not noise data. me. Set the point that is not removed in the manner mentioned in the boundary point boundary (Pn).

La. That is, it is checked whether the points of the original point cloud data exist outside the boundary area. Delete points outside the boundary. The boundary inspection method is as follows.

Find the unit vector between the target point and the closest boundary point (point) from the original point cloud data. The normal vector of the boundary point and the dot product are performed. If this value is positive, it is determined to be a point within the boundary and restored.

11 shows a method for specifying surface noise of point cloud data. In Figure 11, the removal of the point P _ori1 part comprises a point P _ori2 part.

Through the process of K and I in the previous process, only the points with high correlation with the surrounding points remain. It is determined that the remaining points are points with a low probability of being noise, and these points are set as boundaries.

Since the point cloud is located on the surface of the object, the noise is also located near the surface of the object. Therefore, if a surface is defined, there is a high probability that the point outside the defined surface is noise.

Therefore, in order to determine whether the i-th point (P _i ) is noise, the dot product of the direction of the normal vector of the boundary point set through the steps A and B above and the unit vector toward this boundary point necessary. At this time, if the result of the dot product is positive, it is judged not to be noise because it is a point located inside the boundary, and if it is negative, it is judged to be noise. _Taking the situation of FIG. 11 as an example, when P _{ori2 located inside the boundary is dotted with P 3} , which is the normal vector of the boundary point, it becomes positive, and the dot product result for _{P ori1 located outside the boundary becomes negative.} It proceeds in order from A to D.

Next, the curved surface is reconstructed from the point cloud data and the reconstructed curved surface is corrected (S80). Detailed steps of the reconstruction and correction steps are shown in FIG. 11 .

First, a curved surface or a surface is reconstructed from the point cloud data (S81).

That is, converting the input object pointcloud data into an explicit expression form such as a polygon mesh, a parametric surface, or a zero-set of a spatial function is a curved surface. This is called surface reconstruction. A form of noise that is not improved in the above process may be expressed as shown in FIG. 11 depending on the type.

13A shows continuous surfaces adjacent to each other on the x and y coordinates but having a large z-axis distance in the process of reconstructing the curved surface. B shows a case in which illumination noise in the depth image remains, and C shows a case in which boundary noise in the depth image remains.

Specifically, only a surface having a predetermined distance or less is output using the distance between vertices with respect to the reconstructed surface (S82). That is, in the case of A of FIG. 13, it is solved by calculating the distance between vertices constituting the surface and outputting only the surface having a predetermined distance or less. That is, as shown in FIG. 14 , the surface is output only when the distance of P1 P2, which is the vertex constituting the surface, is less than or equal to the threshold value or the distance of P3 P4 is less than or equal to the threshold value. That is, only when one of the at least two opposing sides of the surface is equal to or less than the threshold value, the surface is configured as a surface.

[Equation 5]

Here, pi.x, pi.y, and pi.z are three-dimensional coordinates of the vertex Pi.

In the above equation, vertices P1 and P2 and vertices P3 and P4 illustrate two distances between vertices. It constitutes a face only when one of at least two opposing sides of the face is equal to or less than a threshold value.

Next, in the case of B and C of FIG. 13, the threshold value is applied to the volume of the AABB box created after calculating the minimum (min) and maximum (max) values for the x, y, and z coordinates of the vertices constituting each face. Set to limit the output. Since the noise is elongated in a specific direction, noise can be specified as shown in FIGS. 15 and 16 .

In the case of FIG. 15, various noises, including boundary noise, are specified according to the volume of the surface, but there is a disadvantage in that it is not possible to distinguish the surface of an originally bulky area such as a neck. Therefore, it calculates the normal vector for each face and treats as an exception when its direction is adjacent to a negative number on the Y-axis.

FIG. 17 shows an image in which noise is improved in the original curved data of FIG. 13 .

In the above, the invention made by the present inventors has been described in detail according to the above embodiments, but the present invention is not limited to the above embodiments, and various modifications can be made without departing from the gist of the present invention.

10: hologram data 20: camera system
21 : RGB-D camera
30: computer terminal 40: quality improvement system
60: multi-view texture-depth image 61: depth image
62: texture image

Claims (11)

A method for improving the quality of a 3D image obtained from a depth image camera, the method comprising:
(a) obtaining a multi-view depth image and a texture image;
(b) generating a binary image based on the brightness value of the texture image, and generating a mask image from the generated binary image;
(c) filtering by applying the mask image to the texture image;
(d) performing primary correction by adding an alpha channel to the filtered texture image;
(e) filtering the depth image with the mask image;
(f) correcting the filtered depth image by applying time-average filtering;
(g) secondarily correcting the firstly corrected texture image using the histogram;
(h) generating point cloud data from the corrected multi-view depth image and correcting it by removing surface noise; and,
(i) A method for improving the quality of a 3D image obtained from a depth image camera, comprising the step of constructing a curved surface from the corrected point cloud data.
According to claim 1,
In the step (b), the method for improving the quality of a 3D image obtained from a depth image camera, characterized in that the binary image noise is removed by repeating the close operation during the morphological operation on the binary image.
According to claim 1,
In the step (c), the method for improving the quality of a 3D image obtained from a depth image camera, characterized in that maintaining the brightness value of the mask region in the texture image and setting the brightness value of the region outside the mask to the smallest value.
According to claim 1,
In step (d), an alpha channel of transparency is added to the filtered texture image, and the alpha value of an area over a certain distance in the texture image is set to 0, but the area over a certain distance has a depth value of the area of the corresponding depth image. A method for improving the quality of a 3D image obtained from a depth imaging camera, characterized in that the region is greater than or equal to a preset threshold distance.
According to claim 1,
In the step (e), the depth value of the mask region is maintained in the depth image, and the depth value of the region outside the mask is set to be the largest.
According to claim 1,
In the step (f), the method for improving the quality of a 3D image obtained from a depth image camera, characterized in that the time-average filtering is performed only on the filtered region corresponding to the mask region in the depth image.
According to claim 1,
In the step (f), the pixel values of the current frame of the depth image and at least one frame of the previous time are averaged, and the pixel values of the depth image are corrected with the average value. A method for improving the quality of a 3D image obtained from a video camera.
According to claim 1,
In the step (g), a method for improving the quality of a 3D image obtained from a depth image camera, characterized in that correcting the texture image by performing a histogram specification method.
According to claim 1,
In the step (h), clustering is performed on the point cloud data, a normal vector of each point is calculated, and a point having a correlation with a neighboring point higher than a predetermined reference value through a normal vector dot product between adjacent points (hereinafter, a boundary point). ) and deleting points outside the boundary region with respect to the boundary formed by the detected boundary points.
According to claim 1,
In step (i), when composing the curved surface, it is configured by correcting using the distance between the vertices, but with respect to each surface of the curved surface, only when one of the at least two sides opposite to the surface is less than or equal to a threshold value, Depth image, characterized in that the output is limited by calculating the minimum (min) and maximum (max) values for the x, y, and z coordinates of the vertices constituting each face and setting a threshold value in the volume of the AABB box created A method for improving the quality of a 3D image acquired from a camera.
A computer-readable recording medium recording a program for performing the method for improving the quality of a 3D image obtained from the depth image camera of any one of claims 1 to 10.

Images