KR101826711B1 - Method for Calibrating Depth Map of ToF camera - Google Patents

Abstract

The present invention provides a correction method for a depth map of a ToF camera, capable of effectively removing a noise from a depth map generated by a reflectivity of an image. According to an embodiment of the present invention, the method for correcting a depth map of a time of flight (ToF) depth camera comprises the following steps of: obtaining the depth map from a plurality of ToF depth cameras; performing a pre-processing with respect to the obtained depth map; deriving a noise region of the depth map obtained from the plurality of ToF depth cameras; and warping depth information obtained from the ToF cameras located at the left and right sides to the ToF depth camera located at the center, and improving the noise region.

Description

[0001] The present invention relates to a method of calibrating a depth map of a ToF camera,

The present invention relates to a depth map correction method, and more particularly, to a method for correcting a depth map by utilizing an exposure time of an asynchronous ToF camera.

In recent years, studies have been actively conducted to process three-dimensional images based on depth information using a time-of-flight (ToF) camera. The ToF camera is an active sensor that receives energy emitted from an infrared (IR) sensor having a wavelength of about 850 nm at an optical sensor and calculates phase angle or parallax to obtain distance information to the reflected surface.

However, the following problems are encountered when using a ToF camera. First, since the shape of the waveform is not an ideal waveform due to sine waves, a lot of noise occurs, and objects with high reflectance such as mirrors, monitors, glasses, etc. are difficult to obtain accurate depth information.

In addition, objects with low light reflectance such as black objects and hair are more likely to acquire inaccurate depth information, noise is generated because the amount of light reflected near the outline of the object is small, It has a relatively short acquisition range of up to 10 m since it calculates the phase angle or parallax of the wavelength.

In addition, since the infrared sensor is mainly used, there is a problem that it can not be used outdoors where there are many infrared rays. To solve this problem, a method of calibrating a depth map is being studied.

The ToF camera produces errors due to intrinsic problems caused by changes in radiation measurement, geometry, and illumination, as well as low resolution, such as depth distortion errors, integration time related errors, temperature related errors, built-in pixel related errors, . Figure 1 shows several errors in the depth map obtained by the ToF depth sensor. 1 (a) shows an amplitude map (b) shows a distance image, (c) shows a dark surface, (d) shows a mismatch boundary, (e) shows a bright surface and (f) shows a mismatch boundary.

Most of the depth map correction algorithms focus on minimizing optical noise, and adaptive sampling and Gaussian smoothing methods are used to improve the depth map information. However, the noise generated by the reflectance of the ToF camera is processed There is a difficulty.

An object of the present invention is to provide a depth map correction method capable of effectively removing noise of a depth map caused by reflectance of an image in forming a depth map using a ToF depth camera.

An embodiment provides a depth map correction method of a ToF (Time of Flight) depth camera, comprising: obtaining a depth map in a plurality of ToF depth cameras; Performing a preprocessing on the obtained depth map; Deriving a noise region of a depth map obtained from a plurality of ToF depth cameras; And warping the depth information obtained from the left and right ToF cameras with a ToF depth camera disposed in the middle to improve the noise region.

And, the ToF depth camera is three cameras separated by a certain distance in the horizontal direction, and each camera can acquire the depth map at different exposure times.

The step of performing preprocessing on the obtained depth map is characterized by performing at least one of camera parameter acquisition processing, camera calibration processing, and depth correction.

The step of deriving the noise region of the depth map obtained by the plurality of ToF depth cameras is characterized by searching the noise of the depth map using the amplitude image information. Here, the amplitude image information indicates how much light is reflected by the ToF depth camera, and the amplitude image information has different light intensities depending on the amplitude of the surface.

The step of performing the preprocessing on the obtained depth map includes a process of performing noise elimination filtering. In order to remove the noise caused by the non-ideal waveform used in the ToF depth camera and the noise generated at the boundary between the objects, Distribution weighted median filtering can be performed.

According to the embodiment of the present invention, it is possible to eliminate a main error inherent in a bright or dark surface by using a ToF depth camera having a different exposure time and a distance error correction method using a weighted median filter, A distance map can be obtained.

Figure 1 shows several errors of the depth map obtained by the ToF depth sensor
2 is a view showing a camera system according to the depth map correction method of the embodiment
3 is a diagram illustrating a system for depth map correction according to an embodiment;
4 is a flowchart illustrating a depth map correction method according to an embodiment.
5 is a graph showing an asymmetric normal distribution with a parameter a
6 is a diagram illustrating an example of using an asymmetric normal distribution weighted median filter;
7 is a view showing a depth map correction result of the ToF camera according to the embodiment
FIG. 8 is a diagram illustrating a three-dimensional surface using a zoomed-
9 is a diagram showing the result of error search
Fig. 10 is a diagram showing the result of improved noise

The embodiments of the present invention will be described in detail with reference to the accompanying drawings, but the present invention is not limited to these embodiments. In describing the present invention, a detailed description of well-known functions or constructions may be omitted for the sake of clarity of the present invention.

2 is a view showing a camera system according to the depth map correction method of the embodiment.

Referring to FIG. 2, the camera system for performing the depth map correction method of the embodiment horizontally arranges a plurality of ToF cameras having different exposure times at predetermined intervals. In the embodiment, three ToF cameras are preferably arranged. The ToF 1 (1) and ToF 3 (3) cameras disposed on the left and right are cameras arranged to correct the noise of the ToF 2 (2) camera disposed in the middle.

When the ToF cameras operate in the same area, the illumination may interfere with other ToF cameras, so that the frequencies of the ToF cameras are preferably set to be different from each other. In addition, it is preferable that the integration time of each ToF camera is set differently.

3 is a diagram illustrating a system for depth map correction according to an embodiment.

Referring to FIG. 3, the system for correcting depth maps according to the embodiment may include an image acquiring unit 10, a preprocessing unit 20, a noise searching unit 30, and a depth map improving unit 40.

The image acquiring unit 10 acquires a depth map image from a camera of the same generation as shown in FIG. 2, and obtains a depth map image of a multi-view point on one surface.

The preprocessing unit 20 performs at least one of a camera parameter acquisition process, a camera calibration process, and a noise removal filtering process before the correction of the depth map image.

The noise searching unit 30 predicts a noise region in a depth map obtained from three ToF cameras. The depth map improving unit 40 warps the depth information obtained from the left and right ToF cameras to a ToF camera at an intermediate point, And the like.

4 is a flowchart illustrating a depth map correction method according to an embodiment.

Referring to FIG. 4, a depth map correction method according to an embodiment includes a step S10 of obtaining a depth map in a plurality of ToF cameras, a step S20 of performing a preprocessing on the obtained depth map, (S30) of acquiring a noise region of the depth map, and (S40) improving the noise region by warping depth information obtained from the left and right ToF cameras with an intermediate point ToF camera. Next, each step will be described in detail.

Step S10 of obtaining a depth map from a plurality of ToF cameras is a process of obtaining a depth map of an image to be obtained through a camera of a generation set with different frequencies and exposure times. Through the generations of cameras, depth maps with different viewpoints can be acquired.

The step S20 of performing the preprocessing on the obtained depth map is as follows. The ToF camera mainly uses the pinhole camera model, which is parameterized by implicit and external variables. Camera parameters are the key information for three-dimensional image processing and applications, and implicit parameters include intrinsic features of the camera, such as focal length.

The external parameters may correspond to the direction and position of the camera. In the embodiment, the camera parameters are evaluated by a camera correction method using a chess board. The distance map through the ToF depth camera does not provide intensity information beyond distance information, so an amplitude map can be used for camera calibration. The integration time can be adjusted in the process of obtaining the amplitude map. The longer the integration time is, the higher the accuracy of the amplitude measurement can be. Also, since most ToF depth cameras exhibit nonlinear lens distortion such as radial distortion, the lens distortion can be corrected by the correction method.

And, the process of performing the preprocessing on the obtained depth map may include a process of performing noise removal filtering.

In the embodiment, an asymmetric normal distribution weighted median filter is used in order to remove the noise generated at the boundary of the object due to the fact that the waveform used in the ToF camera is not ideal. The asymmetric normal distribution is a continuous distribution of probability, which means a normal distribution with asymmetry.

Median filtering is a nonlinear method used to remove noise from an image to preserve image boundaries. The purpose of the filtering is to replace the pixel value with the median of the neighboring values of that pixel. The weighted median filter is a nonlinear digital filter that takes into account the window length of 2N + 1, and the pixels may be weighted in the histograms.

The ToF depth camera derives a distance map with an inherent error, which in the example uses an intermediate value filter with asymmetric normal distribution weights to correct for this error. In fact, as many data do not follow a normal distribution, asymmetric normal distributions can be used as weights.

5 is a graph showing an asymmetric normal distribution with a parameter a.

The? Parameter of the probability density function of the asymmetric normal distribution can be defined by Equation 1 as follows.

Here, φ (x) is a standard normal probability density function and φ (x) can be expressed by the following equation (2).

In Equation (1),? (X) represents a cumulative distribution function and can be expressed by Equation (3).

Here, erf is an error function, and the error function is an S-shaped singular function appearing in probability, which is expressed by Equation (4) below.

And, the asymmetric normal distribution weight histogram can be defined by the following Equation (5).

Figure 6 is an illustration of an example of using an asymmetric normal distribution weighted median filter; Referring to FIG. 6, negative asymmetry represents the amount of distribution concentrated to the right, and positive asymmetry represents the amount of distribution concentrated to the left. By performing the processing by the asymmetric normal distribution weighted median filter as in the embodiment, the error region can be found and correction for the error region can be performed.

Next, the step (S30) of acquiring the noise region of the depth map obtained from the plurality of ToF cameras is as follows. Ambiguity of depth information on bright or dark surfaces is a major issue in error generation. Because bright surfaces do not absorb light well and are mainly reflected, bright surfaces reflect a lot of light waves, while dark surfaces have a larger amount of light absorption than bright surfaces. This results in a lot of noise when you get a dark or bright surface ToF depth map, where the distance map is affected much by these surfaces.

The ToF depth camera provides an amplitude map that shows how light is light, making the surface dark or matt when the amplitude of the surface is too low, and brightening the surface if the amplitude of the surface is too high. Hence, the amplitude map can be utilized to search for error regions and the error search function can be defined by Equation (6).

Where TD and TS represent the threshold of the dark surface and the threshold of the bright surface, respectively, and each amplitude map for detecting the error region can be calculated through this Equation (6).

Next, step S40 of improving the noise region by warping the depth information obtained from the left and right ToF cameras with the midpoint ToF camera is as follows. The displacement map by the ToF depth camera which performed the integration for a relatively long time shows more accurate results in the distance measurement in the dark or the non-light area. However, the displacement map performing the integration in a relatively short time, .

It is assumed in the embodiment that the displacement map of the ToF 1 camera has accurate information on a bright surface, and the ToF 3 camera has accurate information on a dark, matte surface. In the searched area, the correct depth information can be warped from the ToF 1 and ToF 3 cameras to the ToF 2 camera. First, assuming that m ₁ and m _c are pixels corresponding to the viewpoint images of ToF 1 and ToF 2, the pixel point m ₁ based on the pinhole camera model can be defined by the following formula using the camera parameters.

Then, in order to find the position of the pixel corresponding to m _c at the viewpoint of the ToF 2 camera, the world coordinate M is scanned in the direction of ToF using the camera parameter, which can be expressed as shown in equation (8).

Through the above process, the points of ToF 1 and ToF 3 can be warped to the point of ToF 2.

FIG. 7 is a view showing the depth map correction result of the ToF camera according to the embodiment. Referring to FIG. 7, three ToF depth cameras are used in all cases, and a case of using the filtered distance map as in the embodiment and a case of using other methods are compared. (a) is an original depth map taken at ToF 2, (b) is a depth map when only an intermediate value filter is used, (c) is a depth map when an asymmetric normal distribution weighted median filter is used Represents a map. In the case of applying the depth map correction of the embodiment, it can be seen that the visible noise of the distance map is reduced because the boundary can be preserved without using unintended artifacts.

8 shows a three-dimensional surface structure using a zoomed-in depth map. Referring to FIG. 8, (a) shows that the surface of the distance map varies greatly, but it can be seen that when the asymmetric normal distribution weighted median filter is used as in the present invention, the variation of the depth map is reduced.

9A and 9B show the results of error detection. In FIG. 9A, the distance map (b) shows the error map, the amplitude map shows the error map, the red area shows the bright surface and the black area shows the dark area . FIG. 10 also shows the result of error detection, wherein (a) shows a color image (b), and (c) a distance map using a ToF 2 camera shows a corrected distance map.

In order to verify the error correction method of the embodiment using three ToF depth cameras, the actual image obtained by the ToF depth camera can be used, and in the corrected depth map (c) as shown in FIG. 10, And the noise level of the input signal is significantly reduced.

The embodiment has proposed a distance error correction method using a ToF depth camera and a weighted median filter having different exposure times as described above. Three ToF depth cameras can be used to eliminate key inherent errors on bright or dark surfaces and achieve more accurate distance maps.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood that various modifications and applications other than those described above are possible. For example, each component specifically shown in the embodiments of the present invention can be modified and implemented. It is to be understood that all changes and modifications that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Claims (6)

A method for correcting a depth map of a ToF (Time of Flight) depth camera,
Disposing three ToF depth cameras that acquire the depth map at different exposure times so as to be spaced apart by a certain distance in the horizontal direction;
Obtaining a depth map from a plurality of ToF depth cameras;
Performing a preprocessing on the obtained depth map;
Deriving a noise region of a depth map obtained from a plurality of ToF depth cameras; And
And warping the depth information obtained from the left and right ToF cameras with an intermediate positioned ToF depth camera to improve the noise region. delete The method according to claim 1,
The step of performing preprocessing on the obtained depth map comprises:
A depth map correction method of a ToF camera that performs at least one of acquisition processing of camera parameters, camera correction processing, and depth correction. The method according to claim 1,
Deriving the noise region of the depth map obtained from the plurality of ToF depth cameras is a method of correcting the depth map of the ToF camera that searches the noise of the depth map using the amplitude image information. 5. The method of claim 4,
Wherein the amplitude image information indicates how much light is reflected by the ToF depth camera, and the amplitude image information has different light intensities depending on the amplitude of the surface. The method according to claim 1,
The step of performing the preprocessing on the obtained depth map includes a process of performing noise elimination filtering. In order to remove the noise caused by the non-ideal waveform used in the ToF depth camera and the noise generated at the boundary between the objects, A depth map correction method of a ToF camera performing median filtering.

Images