KR101919879B1 - Apparatus and method for correcting depth information image based on user's interaction information - Google Patents

Abstract

The present invention relates to an apparatus and method for correcting a depth information image based on user interaction information, and more particularly, to an apparatus and method for correcting a depth information image based on user interaction information, correcting the depth information image by receiving interaction information about an error part of the depth information image by the simple drawing of a user. The apparatus for correcting a depth information image based on user interaction information according to the present invention includes: a receiving unit for receiving a depth information image estimated by at least one depth estimation apparatus and a color information image photographed by at least one color photographing apparatus; an input unit for inputting the interaction information into the received color information image; a divided image generating unit for generating a divided image for distinguishing an object from a background using the inputted interaction information and the received color information image; an interpolation unit for interpolating the inputted interaction information using an interpolation method; and a correcting unit for correcting the estimated depth information image using the interpolated interaction information, the color information image, the divided image, and a global objective function.

Description

Technical Field [0001] The present invention relates to an apparatus and method for correcting a depth information image based on user interaction information,

The present invention relates to a user interaction information based depth information image correction apparatus and method for correcting a depth information image by receiving interaction information on a portion where an error exists in a depth information image as a simple drawing of a user.

The depth information image contains the distance information from the camera to the object used in acquiring the image. Depth information images are used in many fields such as pedestrian detection, object recognition, and 3D movie. The method of extracting the existing depth information image is divided into two major areas. There is a passive method of estimating a depth information image by estimating corresponding points in images obtained from a plurality of cameras and an active method using phase or time difference of reflected signals using an active sensor. The passive method requires a lot of cost in estimating the correspondence point using several cameras and there is a limitation on the estimation performance. Typical examples of the active type include a lidar and a time-of-flight type camera, but they operate only in an indoor environment or have a very high cost. In recent years, due to the breakthrough of deep-running technology, a learning-based method of estimating depth information from images obtained from a single camera has been studied by various researchers.

The depth information images obtained in the above methods still have many problems in terms of the accuracy of the depth information. In particular, passive methods using multiple cameras and recent deep-run-based single-image-depth-estimation methods often cause inaccurate depth information to be estimated due to estimation errors. Therefore, in applications requiring high- There are still limitations to use.

Korean Registered Patent No. 10-1626072 (Registered on May 25, 2015)

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above problems, and it is conceived to propose a new method of correcting an estimation error existing in a depth information image using only a minimum input provided by a user.

According to an aspect of the present invention, there is provided an apparatus for correcting user interaction information based depth information, comprising: a depth information image estimated by at least one depth estimating apparatus; A receiving unit for receiving the color information image; An input unit for inputting interaction information into the received color information image; A divided image generating unit for generating a divided image for distinguishing an object and a background using the inputted interaction information and the received color information image; An interpolation unit interpolating the input interaction information using an interpolation method; And a correcting unit correcting the estimated depth information image using the interpolated interaction information, the color information image, the divided image, and the global objective function. .

In the present invention, the correction unit may define a global objective function for correction using the interpolated interaction information and the estimated depth information image, find the optimal solution of the global objective function, Can be corrected.

In the present invention, the global objective function may include at least one of the interpolated interaction information and the estimated depth information image. And a prior knowledge term using at least one of the segmented image and the color information image.

In the present invention, the interpolator may use at least one of the depth information image, the coordinate map corresponding to the depth information image, the index function, the relevance function, and the segmented image.

Preferably, if the global objective function is used, an optimization method that uses an approximation process to convert the global objective function into a one-dimensional linear system may be used.

According to another aspect of the present invention, there is provided a method for correcting user interaction information based depth information, the method comprising the steps of: receiving a depth information image estimated by at least one depth estimating apparatus and a color information image picked up by at least one color photographing apparatus; ; Inputting interaction information to the received color information image; Generating a divided image including an object image and a background image using the input interaction information and the received color information image; Interpolating the input interaction information using an interpolation method; Correcting the estimated depth information image using the interpolated interaction information, the color information image, the divided image, and the global objective function; .

In the present invention, the correcting step may compare the interpolated interaction information and the estimated depth information image, and may correct the estimated depth information image as a result of the comparison.

 The interpolated interaction information may include at least one of interpolated depth interaction information and the segmented image.

 Wherein the global objective function comprises at least one of the interpolated interaction information and the estimated depth information image; And prior knowledge using at least one of the segmented image and the color information image; . ≪ / RTI >

The correcting step may correct the depth-estimated depth image by defining a global objective function as the interpolated interaction information and the estimated depth information image and obtaining an optimal solution of the defined global objective function .

In the present invention, the interpolator may use at least one of the depth information image, the coordinate map corresponding to the depth information image, the index function, the relevance function, and the segmented image.

In addition, the present invention discloses a computer program stored in a computer-readable recording medium for causing a computer to execute the user interaction information-based depth information image correction method.

The present invention has the effect of correcting the depth information image by receiving the interaction information for the portion where the error exists in the depth information image as a simple drawing of the user.

1 is a block diagram of a user interaction information-based depth information image correction apparatus according to an exemplary embodiment of the present invention.
2 is a diagram illustrating an image in which interaction information is input to an input color information image according to an exemplary embodiment of the present invention.
FIG. 3 is a diagram illustrating an image in which a divided image generator generates a divided image according to an embodiment of the present invention. Referring to FIG.
4 is a diagram illustrating an image in which the receiver receives a depth information image estimated by at least one depth estimation apparatus according to an embodiment of the present invention.
FIG. 5 is a flowchart illustrating a method of decoding a depth information image of FIG. 4 by using a pseudo inverse matrix operator provided by MATLAB, Fig.
FIG. 6 is a view showing a depth information image obtained by correcting the depth information image of FIG. 4 using a correction device of a user information-based depth information image.
7 is a view showing a depth information image estimated by a time-of-flight camera having a small final estimation error.
8 is a diagram illustrating an image in which interaction information is input to an input color information image according to an exemplary embodiment of the present invention.
9 is a diagram illustrating an image in which the receiver receives a depth information image estimated by at least one depth estimation apparatus according to an embodiment of the present invention.
10 is a view showing a depth information image obtained by correcting the depth information image of FIG. 9 using a correction device of a user information-based depth information image.
11 is a view showing a depth information image estimated by a time-of-flight camera having a small final estimation error.
12 is a flowchart illustrating a method for correcting a user interaction information based depth information image according to an embodiment of the present invention and a pseudo inverse matrix And the correction execution time when it is used as a method of loosening by the operator.
13 is a flowchart of a method of correcting a depth information image based on user interaction information according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Hereinafter, a configuration of a user interaction information-based depth information image correction apparatus according to an embodiment of the present invention will be described in detail with reference to related drawings.

1 is a block diagram of a user interaction information-based depth information image correction apparatus according to an exemplary embodiment of the present invention.

The user interaction information based depth information image correction apparatus includes a receiving unit 110, an input unit 120, a divided image generating unit 130, an interpolation unit 140, and a correction unit 150.

Wherein the user interaction-information-based depth information image correction device receives the depth information image estimated by at least one depth estimation device and the color information image photographed by at least one color photographing device, Generates a divided image including an object image and a background image as object-background interaction information, generates interpolated interaction information by interpolating the inputted interaction information using an interpolation method, and outputs the interpolated interaction information Is used to correct the depth information image.

The receiving unit 110 receives a depth information image estimated by at least one depth estimation apparatus and a color information image photographed by at least one color photographing apparatus.

The depth estimating apparatus may be a device such as a time-of-flight camera. The color photographing apparatus may be a camera such as a digital camera.

The input unit 120 inputs the interaction information to the captured color information image.

The interaction information may include depth interaction information, object interaction information, and background interaction information. For example, the depth interaction information may be information input by the user in green, the object interaction information may be information input by the user in red, and the background interaction information may include information input by the user in blue Lt; / RTI > The object interaction information and the background interaction information may be used to generate an object segment image before performing image correction.

The depth interaction information may be an interaction information indicating an area corresponding to a depth information image to be corrected by a user, in a color information image. The object interaction information may be an interaction information indicating an area in which the user interaction information based depth information image correction apparatus desires to recognize the object as a color information image. The background interaction information may be an interaction information indicating an area that the user interaction information-based depth information image correction device desires to recognize as a background, on the color information image.

Since the object interaction information and the background interaction information are used to clearly distinguish the object to be corrected, the object interaction information and the background interaction information may include approximate coordinate information about the object and the background, respectively. Since the depth interaction information is used to correct the existing depth information to the depth information desired by the user, the depth interaction information can be composed of the approximate coordinate information of the part to be corrected and the user depth information corresponding to the corresponding coordinate.

The divided image generator 130 generates a divided image for distinguishing the object and the background. The divided image generating unit 130 generates a divided image that distinguishes an object and a background using object interaction information, background interaction information, and color information image received from the receiving unit 110. The object interaction information may be information input by a user in red, and may include coordinate information on an object targeted for correction. The background interaction information may be information input by the user in a blue color, and may include coordinate information about a background around an object aiming at correction. The segmented image can distinguish the corrected target object from the background by the object interaction and the background interaction.

The process of generating the divided image is described in detail in Equation (22), Equation (23), and Equation (24).

The interpolator 140 interpolates the input depth interaction information using an interpolation method to generate interpolated interaction information. The depth interaction information may be information input by a user in green, and may include coordinates of a part to be corrected and corresponding post-correction depth information.

The interpolation unit 140 generates the interpolated interaction information using the interpolation method according to Equations (4), (5), (6), (7), (8) and (9) have. The interpolation method generates the interpolated interaction information using the depth interaction information, the received color information image, and the object-background divided image generated by the divided image generating unit 130.

The interpolated interaction information may include first interpolation interaction information and second interpolation interaction information. The first interpolation interaction information and the second interpolation interaction information may include information corresponding to the size of the received depth information image. The first interpolation interaction information and the second interpolation interaction information may be information corresponding to interpolated depth interaction information and interpolated indexing interaction information, respectively.

The correction unit 150 corrects the depth estimation information image using the color information image of the reception unit 110, the divided image of the divided image generation unit 130, and the interpolated interaction information of the interpolation unit 140, do. The process of correcting the depth estimation image by defining the global objective function by the correction unit 150 may be performed using Equation 10, Equation 11, Equation 12, Equation 13, Equation 14, Equation 15, (16), (17), (18), (19), (20) and (21).

The global objective function consists of a data term using the interpolated interaction information and the received depth estimation image and a dictionary term for the term. The data term may be a term that corrects the received depth estimation image using the interpolated interaction information, and the prior knowledge term may be a term that exists to smooth out the corrected depth information image.

2 is a diagram illustrating an image in which an input unit 120 according to an embodiment of the present invention inputs interaction information to a color information image.

The input unit 120 inputs the interaction information to the captured color information image.

The interaction information may include depth interaction information 210, object interaction information 230, and background interaction information 220. For example, the depth interaction information 210 may be information input by a user in green, and may include coordinates of a part to be corrected and corresponding post-correction depth information. The object interaction information 230 may be information input by a user in red, and may include coordinate information on an object targeted for correction. The background interaction information 220 may be information input by a user in a blue color, and may include coordinate information about a background around the object aiming at correction. The object interaction information 230 and the background interaction information 220 may be used to generate an object segmented image before performing image correction.

The depth interaction information 210 may be an interaction information indicating an area corresponding to a depth information image to be corrected by a user in a color information image. The object interaction information 230 may be an interaction information indicating an area that the user interaction information-based depth information image correction device desires to recognize as an object, on the color information image. The background interaction information 220 may be an interaction information indicating an area that the user interaction information-based depth information image correction device desires to recognize as a background, on the color information image. The segmented image generating unit 130 generates a segmented image including the object image and the background image in Equations 18, 19, and 20 using the object interaction information 230 and the background interaction information 220, As shown in FIG.

3 is a diagram illustrating an image in which a divided image is generated by the divided image generating unit 130 according to an embodiment of the present invention.

FIG. 3 is a diagram illustrating an image generated by the divided image generating unit 130 that generates a divided image that distinguishes an object 310 from a background 320 using the object-object interaction information.

The divided image generating unit 130 generates a divided image that distinguishes the object 310 and the background 320 from each other. The process of generating the divided image including the object image 310 and the background image 320 by the divided image generating unit 130 is described in detail in Equations (18), (19), and (20).

4 is a diagram illustrating an image in which the receiver 110 receives a depth information image estimated by at least one depth estimation apparatus according to an embodiment of the present invention.

The receiving unit 110 may be a device for receiving a depth information image estimated by at least one depth estimation apparatus and a color information image photographed by at least one color photographing apparatus.

The depth information image of the at least one depth estimating apparatus is obtained by a depth information image obtained by correcting the depth information image of FIG. 6 using a correction method of the user interaction information based depth information image, When the depth information is estimated using v2, the depth information is not clear.

When the user wants to correct the depth information image, the area corresponding to the depth interaction information 410, which is the interaction information input to the color information image, is the area corresponding to the depth interaction information 510 of FIG. 5 or the depth It can be seen that the depth information is not clear when compared with an area corresponding to the interaction information 610 or an area corresponding to the depth interaction information 710 of FIG. When the area corresponding to the background interaction information is compared with the area corresponding to the background interaction information in Fig. 5 or the area corresponding to the background interaction information in Fig. 6 or the area corresponding to the background interaction information in Fig. 7, It can be seen that the depth information is not clear.

In order to correct the image estimated by the at least one depth estimating device, the color information image may be used.

5 is a view showing a depth information image corrected using a method of extracting a large-scale linear system derived from the process of correcting the depth information image of FIG. 4 through optimization of Equation (10) by a pseudoinverse matrix operator provided by MATLAB to be.

The area corresponding to the depth interaction information 510 may be one that does not differ greatly from the depth information of the area corresponding to the depth interaction information 610 of Fig. 6 or the depth information of the depth interactions information 710 of Fig. 7 Able to know. In addition, it can be seen that the area corresponding to the background interaction information does not greatly differ from the depth information of the area corresponding to the background interaction information of Fig. 6 or the area corresponding to the background interaction information of Fig.

FIG. 6 is a view showing a depth information image obtained by correcting the depth information image of FIG. 4 using a correction device of a user information-based depth information image.

Wherein the user interaction-information-based depth information image correction device receives the depth information image estimated by at least one depth estimation device and the color information image photographed by at least one color photographing device, Information of the depth information, the received color information image, and the object-background image, interpolating the input interaction information by interpolation to generate interpolated interaction information, And corrects the depth information image.

The region corresponding to the depth interaction information 610 may be one that does not differ greatly from the depth information of the region corresponding to the depth interaction information 510 of FIG. 5 or the depth information of the depth interaction information 710 of FIG. 7 Able to know. In addition, the area corresponding to the background interaction information is not significantly different from the depth information of the area corresponding to the background interaction information in Fig. 5 or the area corresponding to the background interaction information in Fig.

When the large linear system derived in the correction process is solved, the method of solving the problem by using the subsystem used in the present invention can guarantee a correction rate much faster than the method using the pseudoinverse operator of the MATLAB. The improvement of the speed of correcting the depth information image using the present invention is shown in FIG.

FIG. 12 shows the depth information image corrected in the environment of 3.3 GHz, 4 cores CPU and 16 GB RAM.

In the correction process, it is confirmed that the corrected depth information image using the correction method of the depth information image and the user interaction information based depth information image provided by the MATLAB pseudoinverse operator provides almost similar correction results. However, referring to FIG. 12, it can be seen that the correction execution time of the corrected depth information image is significantly faster by using the correction method of the depth information image based on the user interaction information.

FIG. 7 is a view showing a depth information image estimated by a time-of-flight camera (Microsoft Kinect v2) with a small final estimation error.

The time-of-flight camera is a camera that measures the time taken to reflect light by shooting light. Using the Time-of-Flight camera, it is possible to estimate a depth information image with a small final estimation error, but it may be limited to operate only in an indoor environment or require a very high cost.

When the area corresponding to the depth interaction information 710 is viewed, the depth information of the area corresponding to the depth interaction information 510 of FIG. 5 or the depth information of the area corresponding to the depth interaction information 610 of FIG. 6 Able to know. In addition, the area corresponding to the background interaction information is not significantly different from the depth information of the area corresponding to the background interaction information of Fig. 5 or the area corresponding to the background interaction information of Fig.

FIG. 8 is a diagram illustrating an image in which an input unit 120 according to an embodiment of the present invention inputs interaction information to a color information image.

The input unit 120 inputs the interaction information to the captured color information image.

The interaction information may include depth interaction information 830, object interaction information 820, and background interaction information 810. For example, the depth interaction information 830 may be information input by a user in green, and may include coordinates of a desired portion to be corrected and corresponding post-correction depth information. The object interaction information 820 may be information input by the user in red, and may include coordinate information on an object targeted for correction. The background interaction information 810 may be information input by a user in a blue color and may include coordinate information on a background around an object aiming at correction. The object interaction information 820 and the background interaction information 810 may be used to generate an object segmented image before performing image correction.

The depth interaction information 830 may be an interaction information indicating an area corresponding to a depth information image to be corrected by a user in a color information image. The object interaction information 820 may be an interaction information indicating an area that the user interaction information-based depth information image correction device desires to recognize as an object, on the color information image. The background interaction information 810 may be an interaction information indicating an area that the user interaction information-based depth information image correction device desires to recognize as a background, on the color information image. The segmented image generating unit 130 generates the segmented image including the object image and the background image in Equations 18, 19, and 20 using the object interaction information 820 and the background interaction information 810, As shown in FIG.

9 is a diagram illustrating an image in which the receiver 110 receives a depth information image estimated by at least one depth estimation apparatus according to an embodiment of the present invention.

The receiving unit 110 may be a device for receiving a depth information image estimated by at least one depth estimation apparatus and a color information image photographed by at least one color photographing apparatus.

The depth information image of the at least one depth estimating device is obtained by using the depth information image corrected using the user interaction information depth information image correction method or the MS Kinect v2 shown in FIG. The depth information is not clear when compared with the image obtained by estimating the depth information image. In particular, as shown in FIG. 9 or 11, when estimating the depth of a transparent glass or mirror, a very large estimation error can be calculated using MS Kinect v2.

When a region corresponding to the depth interaction information 910, which is the interaction information inputted to the color information image, is viewed by the user in order to correct the depth information image, the region corresponding to the depth interaction information 1010 of FIG. 10 or the depth It can be seen that the depth information is not clear when compared with the area corresponding to the interaction information 1110. In addition, when viewing the area corresponding to the background interaction information, it is understood that the depth information is not clear when compared with the area corresponding to the background interaction information in Fig. 10 or the area corresponding to the background interaction information in Fig.

10 is a view showing a depth information image obtained by correcting the depth information image of FIG. 9 using a correction device of a user information-based depth information image.

It can be seen that there is no significant difference from the depth information of the area corresponding to the depth interaction information 1110 of FIG. 11 when viewing the area corresponding to the depth interaction information 1010. In addition, it can be seen that the area corresponding to the background interaction information does not greatly differ from the depth information of the area corresponding to the background interaction information of Fig. FIG. 10 shows that a similar result to the depth information image estimated using the MS Kinect v2 can be obtained by using the user interaction information-based depth information image correcting device.

The user interaction information based depth information image correcting apparatus of the present invention can perform accurate depth correction even when correction is performed on transparent glasses or mirrors in which the estimation error is largely calculated. It can be seen that there is no significant difference from the depth information of the area corresponding to the depth interaction information 1010 of FIG. 10 when viewing the area corresponding to the depth interaction information 1110. In addition, the region corresponding to the background interaction information is not significantly different from the depth information of the region corresponding to the background interaction information of Fig.

12 is a flowchart illustrating a method for correcting a user interaction information based depth information image according to an embodiment of the present invention and a pseudo inverse matrix Which is used as a correction method to be eliminated by an operator.

When the image size is 625 x 468, the correction execution time of 1210 is 2.11 seconds when the correction is performed by using the pseudoinverse operator of MATLAB 1210. On the other hand, when the correction method of the depth information image based on the user interaction information is used 1220 ) Is 0.1 second.

When the image size is 900 X 680, the correction execution time of 1210 is 2.62 seconds when the correction is performed by using the pseudoinverse operator of MATLAB 1210. On the other hand, when the correction method of the depth information image based on the user interaction information is used 1220 ) Is 0.29 seconds.

When the image size is 1200 X 900, the correction execution time of 1210 is 24.6 seconds when the correction is performed by using the pseudoinverse operator of MATLAB 1210. On the other hand, when the correction method of the depth information image based on the user interaction information is used 1220 ) Is 0.59 seconds.

When the image size is 900 X 680 and the image size is 1200 X 900, the correction execution time is 2.62 seconds and 24.6 seconds respectively by optimizing with the pseudoinverse operator of MATLAB. Therefore, when the image resolution is higher than HD (1280 X 720), the speed can be significantly slowed by using the correction method that is optimized with the pseudoinverse of MATLAB. 13 is a flowchart of a method of correcting a depth information image based on user interaction information according to an embodiment of the present invention.

The user interaction information-based depth information image correction method according to an embodiment of the present invention includes the following steps that are processed in a time-series manner in the user interaction information-based depth information image correction apparatus.

In S110, the receiving unit 110 receives the depth information image and the color information image.

In S120, the input unit 120 inputs the interaction information to the captured color information image.

In step S130, the divided image generating unit 130 generates a divided image including an object image and a background image.

In step S140, the interpolating unit 140 generates interlace information interpolated using the interpolated depth interworking information.

In step S150, the correcting unit 150 defines a global objective function and corrects the depth information image using the received depth-estimated image, the color information image, the interpolated interaction information, and the generated divided image.

: 픽셀 p를 중심으로 상하좌우 픽셀, q: 픽셀 p를 중심으로 상하좌우 픽셀 중 하나의 픽셀,

: 수학식 2와 같은 픽셀 p와 q간의 연관성,

: 입력 영상에서의 픽셀 p의 값,

: 결과 영상에서의 픽셀 p의 값,

: 결과 영상에서의 픽셀 q의 값,

: 영향 파라미터이다.Here, p: arbitrary pixel,

: Upper and lower left and right pixels around pixel p, q: One pixel among pixels vertically and horizontally around pixel p,

: The relationship between pixels p and q as in equation (2)

: The value of the pixel p in the input image,

: The value of the pixel p in the resultant image,

: The value of the pixel q in the resultant image,

: Influence parameter.

를 구함으로써 최적의 결과 영상을 얻을 수 있다. 수학식 1의

에 대한 미분값이 0이 될 때 최적의

를 연산함으로 최소화 과정을 수행할 수 있다. 수학식 1, 수학식 2, 수학식 3 및 수학식 4는 통상적인 WLS 문제에 관련된 수학식이고, 수학식 5, 수학식 6 및 수학식 7은 본 발명에서 WLS 문제를 효율적으로 푸는 방법에 관련된 수학식이다.Equation (1) is a Weighted Least Squares (WLS) problem that is widely used in the image processing field. In the present invention, image correction and image separation processes are defined as WLS problems and image processing operations are performed. Minimizing Equation 1 for a given input image

The optimum result image can be obtained. In Equation (1)

Lt; RTI ID = 0.0 > 0 < / RTI >

The minimization process can be performed. Equations (1), (2), (3) and (4) are mathematical formulas related to the conventional WLS problem, and Equations (5), (6) and (7) are related to a method of efficiently solving the WLS problem in the present invention Is a mathematical expression.

)은 입력 영상 중 임의의 픽셀 p의 값(

)과 유사해지도록 영향을 받는다. 사전 지식 항으로 인하여 최소화 과정을 수행한 후, 결과 영상 중 임의의 픽셀 p의 값(

)은 결과 영상 중 픽셀 p의 상하좌우 픽셀(

)의 영향을 받는다. 이 때, 픽셀 p와 q간의 연관성(

)이 크면 클수록 최소화 과정 후 결과 영상 중 임의의 픽셀 p의 값(

)은 데이터 항에 비해 사전 지식 항에 더욱 많은 영향을 받고 상하좌우에 위치한 픽셀의 값(

)과 유사한 결과를 얻는다. 예를 들어, 깊이 보정의 과정 중 영상 내 임의의 물체는 매우 유사한 컬러 속성 때문에 물체 내 픽셀 p와 q간의 연관성(

) 커지고, 최소화 과정을 수행한 후 사전 지식 항의 영향력에 따라 유사한 깊이 정보를 가지도록 보정된다.The first term of Equation (1) is a data term, and the second term is a prior knowledge term. After performing the minimization process due to the data term, the value of arbitrary pixel p (

) Is the value of an arbitrary pixel p in the input image

). ≪ / RTI > After performing the minimization process due to the pre-knowledge term, the value of arbitrary pixel p (

) Are the pixels in the top, bottom, left, and right of the pixel p

). At this time, the correlation between the pixels p and q

) Is larger, the value of arbitrary pixel p (

) Is more influenced by the prior knowledge term than the data term, and the value of pixels located in the upper,

). For example, during the process of depth correction, any object in an image may be associated with a pixel p and q in the object

), And after performing the minimization process, it is corrected to have similar depth information according to the influence of the prior knowledge term.

The influence parameter is a constant value that adjusts the ratio of the influence of the data term to the influence of the prior knowledge term. The larger the influence parameter, the greater the influence of the prior knowledge term during the minimization process.

: 두 픽셀 p와 q간의 연관성,

: 수신된 컬러 정보 영상에서의 픽셀 p의 빨강색 값,

: 수신된 컬러 정보 영상에서의 픽셀 q의 빨강색 값,

: 수신된 컬러 정보 영상에서의 픽셀 p의 초록색 값,

: 수신된 컬러 정보 영상에서의 픽셀 q의 초록색 값,

: 수신된 컬러 정보 영상에서의 픽셀 p의 파랑색 값,

: 수신된 컬러 정보 영상에서의 픽셀 q의 파랑색 값,

: 컬러 특징값에 대한 정규화 파라미터이다.From here,

: The relationship between two pixels p and q,

: The red value of the pixel p in the received color information image,

: The red value of the pixel q in the received color information image,

: The green value of pixel p in the received color information image,

: The green color value of the pixel q in the received color information image,

: The blue color value of the pixel p in the received color information image,

: The blue color value of the pixel q in the received color information image,

: Normalization parameter for color feature values.

 The correlation between the two pixels p and q becomes larger as the value of the pixel p in the color information image is similar to the value of the pixel q in the color information image. Equation (2) may be used in Equation (1). Equation (2) may be related to Equations (4), (5) and (7).

대각행렬(N = 입력영상의 높이 x 입력영상의 너비), W: 수학식 4와 같은

연관성 행렬, f: 입력 영상에 대응되는

벡터, u: 결과 영상에 대응되는

벡터,

: 영향 파라미터이다.In this case, D:

Diagonal matrix (N = height of input image x width of input image), W:

Correlation matrix, f: corresponding to the input image

Vector, u: corresponding to the result image

vector,

: Influence parameter.

의 역행렬을 계산해야 하고, 이 과정은 N의 크기가 커질수록 기하급수적으로 시간 소비가 증가된다. 또한, 수학식 4의 연관성 행렬의 특징 때문에 행렬

는 각 행마다 5개의 원소를 가진다.Equation (3) is a linear system derived from the minimization process of Equation (1). To obtain the vector u corresponding to the resultant image in the linear system of Equation (3)

, And this process increases exponentially with time as N becomes larger. Also, because of the feature of the relation matrix of equation (4)

Has five elements in each row.

대각 행렬 D의 p번째 행, q번째 열에 있는 값, W(p,q):

연관성 행렬 W의 p번째 행, q번째 열에 있는 값, p: 임의의 픽셀, q: 임의의 픽셀,

: 픽셀 p를 중심으로 상하좌우 픽셀, j: 픽셀 p의 상하좌우 픽셀 중 하나,

: 수학식 2와 같은 연관성,

: 영향 파라미터이다Here, D (p, q):

The value in the pth row and qth column of the diagonal matrix D, W (p, q):

The value in the pth row, qth column of the relevance matrix W, p: arbitrary pixel, q: arbitrary pixel,

: Upper, lower, left and right pixels around pixel p, j: one of upper, lower, left, and right pixels of pixel p,

: ≪ / RTI > such as Equation (2)

: It is an influence parameter

 When the association matrix is defined as shown in Equation (4), the association matrix W is a matrix having four elements for each row.

: 픽셀 p를 중심으로 좌우 픽셀,

: 픽셀 p를 중심으로 상하 픽셀, q: 픽셀 p를 중심으로 좌우 픽셀 또는 상하 픽셀 중 하나,

: 행으로 분리된 입력 영상 중 픽셀 p의 값,

: 열로 분리된 입력 영상 중 픽셀 p의 값,

: 행으로 분리된 결과 영상 중 픽셀 p의 값,

: 행으로 분리된 결과 영상 중 픽셀 q의 값,

: 열로 분리된 결과 영상 중 픽셀 p의 값,

: 열로 분리된 결과 영상 중 픽셀 q의 값,

: 수학식 2와 같은 픽셀 p와 q간의 연관성,

: 영향 파라미터이다.Here, p: arbitrary pixel,

: Left and right pixels around pixel p,

: Upper and lower pixel around pixel p, q: one of left and right pixel or upper and lower pixel around pixel p,

: The value of the pixel p in the input image separated by the row,

: The value of the pixel p in the input image separated by the column,

: The value of the pixel p in the result image separated by the row,

: The value of the pixel q in the result image separated by the row,

: The value of the pixel p in the result image divided into columns,

: The value of the pixel q in the result image divided into columns,

: The relationship between pixels p and q as in equation (2)

: Influence parameter.

크기의 희소행렬이 포함된 수학식 3의 대규모 선형 시스템을 풀어야 하는데, 본 발명에서 사용되는 분리된 서브 시스템의 근사화 방법을 사용하면 대규모 선형시스템을 분리하여 효율적인 시간 내에 전역해를 구할 수 있다.To minimize the WLS problem of Equation 1

It is necessary to solve the large-scale linear system of Equation (3) including the sparse matrix of the size. By using the method of approximating the separated subsystems used in the present invention, it is possible to separate the large-scale linear system and obtain the global solution in an efficient time.

 Equation (5) is a mathematical expression that divides Equation (1) into an X-axis subsystem (first equation) and a Y-axis subsystem (second equation).

는 각각 2차원 입력 영상 f를 행(X축)마다 분리시켰다는 의미를 내포하고 있고, 그 결과 f는 입력 영상의 높이 개수만큼의 분리된 행으로 분리된다. 마찬가지로, 수학식 5의

는 각각 2차원 입력 영상 f를 열(Y축)마다 분리시켰다는 의미를 내포하고 있고, 그 결과 f는 입력 영상의 너비 개수만큼의 분리된 열으로 분리된다. 따라서, 높이 개수만큼의 분리된 행마다 또는 너비 개수만큼의 분리된 열마다 최소화 과정을 수행하면,

와

에 대응되는 분리된 결과 영상의 높이 개수만큼의

와 너비 개수만큼의

를 구할 수 있다.Equation 5

Implies that the two-dimensional input image f has been separated for each row (X axis), and the result f is divided into a plurality of rows separated by the height of the input image. Similarly, in Equation 5

Dimensional input image f is divided into columns (Y-axis), and as a result, f is divided into a number of columns separated by the width of the input image. Therefore, if the minimization process is performed for each of the rows separated by the number of the height or for the columns separated by the number of the width,

Wow

Corresponding to the number of heights of the separated result image corresponding to

And the number of widths

Can be obtained.

 The linear system derived from the minimization process of Equation (5) can be expressed by Equation (6).

 The method of approximating the discrete subsystems of Equations (5) and (6) can be performed by the sparse interpolation method of Equations (8) and (9), the depth estimation image correction process of Equations (14), It is used to solve the WLS problem in an efficient time in the object background separation process. Using the above method, the WLS problem of Equation (1) can be solved for the time complexity of O (height X width).

: 수학식 7과 같은 너비x너비 대각행렬,

: 수학식 7와 같은 너비x너비 연관성 행렬,

: 행으로 분리된 입력 영상에 대응되는 너비x1 벡터,

: 행으로 분리된 결과 영상에 대응되는 너비x1 벡터,

: 수학식 7과 같은 높이x높이 대각행렬,

: 수학식 7와 같은 높이x높이 연관성 행렬,

: 열로 분리된 입력 영상에 대응되는 높이x1 벡터,

: 열로 분리된 결과 영상에 대응되는 높이x1 벡터,

: 영향 파라미터이다.From here,

: Width x width diagonal matrix as in Equation 7,

: Width x width relevance matrix as in Equation 7,

: A width x1 vector corresponding to an input image separated by a row,

: A width x1 vector corresponding to a result image separated by a row,

: Height x height diagonal matrix as in Equation 7,

: Height-x-height relation matrix as shown in equation (7)

: Height x1 vector corresponding to the input image separated by the column,

: Height x1 vector corresponding to the result image separated into columns,

: Influence parameter.

와

를 얻기 위해서

와

의 역행렬을 계산할 수 있다.Equation (6) is a linear system derived from the minimization process of Equation (5). In the linear system of Equation (6), the vectors corresponding to the separated rows and columns of the resulting image

Wow

To get

Wow

Can be calculated.

와

는 수학식 7의 연관성 행렬 특징 때문에 각 행마다 3개의 원소를 가지는 특성을 가진 Tridiagonal matrix가 된다. Tridiagonal matrix가 포함된 선형 시스템은 Gaussian Elimination 방법을 사용하여 간단하고 빠르게 풀 수 있다.The matrix of equation (6)

Wow

Is a Tridiagonal matrix having three elements for each row due to the relevance matrix characteristic of Equation (7). A linear system containing a tridiagonal matrix can be solved simply and quickly using the Gaussian Elimination method.

크기의 행렬에 대한 역행렬을 구해야 하지만, 수학식 6과 같은 본 발명에서 사용하는 분리된 서브 시스템에서는 너비x너비 또는 높이x높이 크기의 행렬에 대한 역행렬을 구하면 되므로 효율적인 시간 내에 연산 처리 과정을 마칠 수 있다. 또한, 기존

크기의 행렬은 행마다 5개의 원소를 가졌고 본 발명에서의 높이x높이 행렬과 너비x너비 행렬은 행마다 3개의 원소를 가지는 특성을 고려했기 때문에 수행 시간은 더욱 단축된다. 여기에서 N=높이x너비를 의미한다.In a linear system derived from the existing WLS problem such as Equation 3

However, in the separate subsystem used in the present invention as shown in Equation (6), since the inverse matrix of the matrix having the width x width or the height x height can be obtained, the computation process can be completed in an efficient time have. In addition,

The matrix of size has five elements per row, and the execution time is further shortened because the height x height matrix and the width x width matrix in the present invention take into account the characteristic of having three elements per row. Where N = height x width.

The method of approximating the discrete subsystems of Equations (5) and (6) can be performed by the sparse interpolation method of Equations (8) and (9), the depth estimation image correction process of Equations (14), It is used to solve the WLS problem in an efficient time in the object background separation process.

Using this method, a large linear system of equation (3) can be solved for the time complexity of O (height X width).

: 너비x너비 대각 행렬

의 p번째 행, q번째 열에 있는 값,

(p,q): 너비x너비 연관성 행렬

의 p번째 행, q번째 열에 있는 값,

: 높이x높이 대각 행렬

의 p번째 행, q번째 열에 있는 값,

(p,q): 높이x높이 연관성 행렬

의 p번째 행, q번째 열에 있는 값, p: 임의의 픽셀, q: 임의의 픽셀,

: 픽셀 p를 중심으로 좌우 픽셀,

: 픽셀 p를 중심으로 상하 픽셀, j: 상하 또는 좌우 픽셀 중 하나,

: 수학식 2와 같은 연관성,

: 영향 파라미터이다.From here,

: Width x width diagonal matrix

The value in the pth row, the qth column of < RTI ID = 0.0 >

(p, q): width x width association matrix

The value in the pth row, the qth column of < RTI ID = 0.0 >

: Height x height diagonal matrix

The value in the pth row, the qth column of < RTI ID = 0.0 >

(p, q): height x height association matrix

The value in the pth row, the qth column of the pixel, p: an arbitrary pixel, q: an arbitrary pixel,

: Left and right pixels around pixel p,

: Upper and lower pixel around pixel p, j: one of upper and lower or left and right pixels,

: ≪ / RTI > such as Equation (2)

: Influence parameter.

,

는 각 행마다 2개의 원소를 가지는 행렬이 된다.When a relevance matrix is defined as in Equation (7), a relevance matrix

,

Is a matrix having two elements for each row.

: 픽셀 p를 중심으로 상하좌우 픽셀, q: 픽셀 p를 중심으로 상하좌우 픽셀 중 하나의 픽셀,

: 깊이 인터랙션 정보(830)에서 픽셀 p의 값, hp: 깊이 인터랙션 정보(830)의 색인 함수,

: 수학식 10과 같은 픽셀 p와 픽셀 q의 연관성을 나타내는 연관성 함수,

: 영향 파라미터, u1,p: 제1 보간 인터랙션 정보에서 픽셀 p의 값, u1,q: 제1 보간 인터랙션 정보에서 픽셀 q의 값이다.Here, p: arbitrary pixel,

: Upper and lower left and right pixels around pixel p, q: One pixel among pixels vertically and horizontally around pixel p,

: The value of the pixel p in the depth interaction information 830, hp: the index function of the depth interaction information 830,

: A correlation function representing the association between pixel p and pixel q as in Equation 10,

: Influence parameter, u1, p: value of pixel p in the first interpolation interaction information, and u1, q: value of pixel q in the first interpolation interaction information.

Equations (8), (9), and (13) are mathematical expressions related to the process of generating interpolation interaction information by using the sparse interpolation method, and the generated interpolation interaction information is used in the correction process of the depth estimation image . The interpolation interaction information generated by the sparse interpolation method is information on the depth interaction information 830, and the deep interaction information 830 is interpolated by interpolating the depth interaction information 830 to the entire object for the purpose of correction It implies meaning. Since the depth interaction information 830 is sparse compared to the size of the input image, it is not suitable to use the depth interaction information 830 during the correction process. Accordingly, the depth interaction information 830 is processed into the appropriate interpolation interaction information through the sparse interpolation process.

)를 보간시킨다.Equation (8) and Equation (9) are mathematical expressions that define the sparse interpolation method as the WLS problem of Equation (1), where Equation (8)

).

)는 깊이 인터랙션 정보(830)에서 픽셀 p의 사용자 입력 깊이값를 의미하며, 상기 픽셀이 깊이 인터랙션 정보에 해당하는 좌표 맵에 포함된다면 상기 픽셀의 색인 함수(

)는 1의 값을 지니고 포함되지 않는다면 상기 픽셀의 색인 함수(

)는 0의 값을 지닌다. Depth Interaction Information (

Denotes a user input depth value of the pixel p in the depth interaction information 830. If the pixel is included in the coordinate map corresponding to the depth interaction information,

) Has a value of 1, and if not included, the index function of the pixel

) Has a value of zero.

The first term in Equation (4) is a data term. When the value of the pixel p in the first interpolation interaction information is obtained, the value of the pixel p may be referred to in the index function and the interaction information. The value of the pixel p in the first interpolation interaction information may be proportional to a value obtained by multiplying the index function and the value of the pixel p in the interaction information.

The second term in Equation (4) is a prior knowledge item. The value of the pixel p in the first interpolation interaction information and the value of the pixel q in the first interpolation interaction information, which is one of the upper, The larger the association function, which indicates the association between the pixel p and the pixel q, becomes more similar.

The value of the pixel p in the first interpolation interaction information and the value of the pixel q in the first interpolation interaction information may be information obtained by solving the linear system as shown in Equation (7). The value of the pixel p in the first interpolation interaction information may be the value of the pixel p in the first interpolation interaction information obtained by solving Expression (7), and the value of the pixel q in the first interpolation interaction information may be obtained by solving Expression (7) 1 < / RTI > in the interpolation interaction information.

The influence parameter is a parameter which means that when the corrected image at pixel p is obtained by the optimization of Equation (4), it is influenced by the first term or is influenced by the second term. The optimization of Equation (4) can be implemented from a linear system expressed by differentiating Equation (4) with respect to the value (u1, p) of the pixel P in the first interpolation interaction information.

Interpolation is used to generate the first interpolation interaction information that constitutes the data term. First, a user generates a coordinate map in an image corresponding to the interaction information using the interaction information input from the received depth information image. The interaction information may be depth interaction information. The first interpolation interaction information of Equation (7) and the second interpolation interaction information of Equation (8) can be obtained using the input interaction information and the coordinate map in the video corresponding to the interaction information, and using the first interpolation interaction information The value of the pixel p and the value of the pixel q in the first interpolation interaction information can be obtained in the first interpolation interaction information and the value of the pixel p in the second interpolation interaction information can be obtained using the second interpolation interaction information, The value of the pixel q can be obtained from the interaction information.

The association function indicating the association between the pixel p and the pixel q may be calculated by using the similarity between the value of the pixel p and the value of the pixel q in the color information image and the similarity between the value of the pixel q and the object- And the value of the pixel q.

The first term in Equation (4) is a data term, and the second term in Equation (4) is a prior knowledge term. Equation (4) represents a global objective function including the data term and the prior knowledge term It can be used.

In the present invention, as shown in Equation (1), the WLS problem of Equation (8) can be approximated to a separate subsystem, as in the case where the global solution is obtained in an efficient time using the method of approximating the separated subsystem of Equation Respectively. By using the above method, the first interpolation interaction information of Equation (8) can be obtained for the time complexity of O (height X width).

: 픽셀 p를 중심으로 상하좌우 픽셀, q: 픽셀 p를 중심으로 상하좌우 픽셀 중 하나의 픽셀,

: 색인 함수,

: 수학식 10과 같은 픽셀 p와 픽셀 q의 연관성을 나타내는 연관성 함수,

: 영향 파라미터, u2,p: 제2 보간 인터랙션 정보에서 픽셀 p의 값, u2,q: 제2 보간 인터랙션 정보에서 픽셀 q의 값이다.Here, p: arbitrary pixel,

: Upper and lower left and right pixels around pixel p, q: One pixel among pixels vertically and horizontally around pixel p,

: Index function,

: A correlation function representing the association between pixel p and pixel q as in Equation 10,

: U2, p: value of pixel p in the second interpolation interaction information, u2, q: value of pixel q in the second interpolation interaction information.

Equations (8), (9), and (13) are mathematical expressions related to the process of generating interpolation interaction information by using the sparse interpolation method, and the generated interpolation interaction information is used in the correction process of the depth estimation image . The interpolation interaction information generated by the sparse interpolation method is information on the depth interaction information 830, and the deep interaction information 830 is interpolated by interpolating the depth interaction information 830 to the entire object for the purpose of correction It implies meaning. Since the depth interaction information 830 is sparse compared to the size of the input image, it is not suitable to use the depth interaction information 830 during the correction process. Accordingly, the depth interaction information 830 is processed into the appropriate interpolation interaction information through the sparse interpolation process.

)를 보간시킨다.Equation (8) and Equation (9) are mathematical expressions in which the sparse interpolation is defined as the WLS problem of Equation (1), and Equation (9)

).

)는 1의 값을 지니고, 포함되지 않는다면 임의의 픽셀 p의 색인 함수(

)는 0의 값을 지닌다.If an arbitrary pixel is included in the coordinate map corresponding to the depth interaction information 830, the index function of arbitrary pixel p (

) Has a value of 1, and if it does not contain the index function of any pixel p (

) Has a value of zero.

The value of the pixel p in the second interpolation interaction information and the value of the pixel q in the second interpolation interaction information may be information that can be obtained by solving the linear system such as Equation (8). The value of the pixel p in the second interpolation interaction information may be the value of the pixel p in the second interpolation interaction information obtained by solving Expression (8), and the value of the pixel q in the second interpolation interaction information may be obtained by solving Expression 2 < / RTI > interpolation interaction information.

The influence parameter is a parameter that indicates whether the image is affected by the first term or influenced by the second term when the corrected image in the pixel p is obtained by the optimization of Equation (5).

Interpolation is used to generate the second interpolation interaction information that constitutes the data term. First, a user generates a coordinate map in an image corresponding to the interaction information using the interaction information input from the received depth information image. The interaction information may be depth interaction information. The first interpolation interaction information of Equation (11) and the second interpolation interaction information of Equation (12) can be obtained using the input interaction information and the coordinate map in the video corresponding to the interaction information, and using the first interpolation interaction information The value of the pixel p and the value of the pixel q in the first interpolation interaction information can be obtained in the first interpolation interaction information and the value of the pixel p in the second interpolation interaction information can be obtained from the value of the pixel p using the second interpolation interaction information, The value of the pixel q in the information can be obtained. The depth interaction information is coordinates corresponding to the input interaction information when the user inputs the interaction information. If the pixel is included in the coordinate map corresponding to the depth interaction information, the index function is 1, The index function is 0 if it is not included in the coordinate map corresponding to the interaction information.

The association function indicating the association between the pixel p and the pixel q may be calculated by the similarity between the value of the pixel p and the value of the pixel q in the color information image and the similarity of the value of the pixel q in the object- And the product of the value of the pixel p and the value of the pixel q.

The first term in Equation (9) is a data term, and the second term in Equation (5) is a prior knowledge term. Equation (9) represents a global objective function including the data term and the prior knowledge term It can be used.

In the present invention, as shown in Equation (1), the WLS problem of Equation (9) can be approximated to a disjoint subsystem as a global solution is obtained in an efficient time by using the method of approximating the separated subsystem of Equation Respectively. By using the above method, the second interpolation interaction information of Equation (9) can be obtained for the time complexity of O (height X width).

: 두 픽셀 p와 q간의 연관성,

: 픽셀 p의 빨강색 값,

: 픽셀 q의 빨강색 값,

: 픽셀 p의 초록색 값,

: 픽셀 q의 초록색 값,

: 픽셀 p의 파랑색 값,

: 픽셀 q의 파랑색 값,

: 컬러 특징값에 대한 정규화 파라미터,

: 분할 영상 생성부(130)에서 생성된 객체-배경 분할 영상의 픽셀 p의 값,

: 객체-배경 분할 영상의 픽셀 q의 값,

: 분할 특징값에 대한 정규화 파라미터 이다.From here,

: The relationship between two pixels p and q,

: The red value of the pixel p,

: Red value of pixel q,

: The green value of pixel p,

: Green value of pixel q,

: The blue color value of the pixel p,

: Blue color value of pixel q,

: A normalization parameter for a color feature value,

: The value of the pixel p of the object-background divided image generated by the divided image generating unit 130,

: The value of the pixel q of the object-background divided image,

Is a normalization parameter for the partition feature value.

 The correlation between the two pixels p and q becomes larger as the value of the pixel p in the color information image is similar to the value of the pixel q in the color information image. The correlation between the two pixels p and q becomes larger as the value of the pixel p and the value of the pixel q in the object-background divided image are similar. Equation (10) may be used in Equations (8) and (9). Equation (10) may be related to Equations (12), (14) and (16).

대각행렬(N = 입력영상의 높이 x 입력영상의 너비), W: 수학식 12과 같은

연관성 행렬, g: 깊이 인터랙션 정보값에 대응되는

벡터, 1: 모든 원소가 1의 값을 가지는

벡터, H: 수학식 12과 같은 색인 함수에 대응되는

대각행렬, u1: 제1 보간 인터랙션 정보에 대응되는

벡터, u1: 제2 보간 인터랙션 정보에 대응되는

벡터,

: 영향 파라미터이다.Here, D: as in Equation 12

Diagonal matrix (N = height of input image x width of input image), W:

Association matrix, g: depth corresponding to the depth interaction information value

Vector, 1: all elements have a value of 1

Vector, H: corresponding to the index function as shown in equation (12)

Diagonal matrix u1: corresponding to the first interpolation interaction information

Vector, u1: corresponding to the second interpolation interaction information

vector,

: Influence parameter.

The first and second interpolation interaction information satisfying the equations (4) and (5) can be obtained by solving the linear system of Equation (7). The first and second interpolation interaction information may be information on which the received depth interaction information 830 is processed. In the present invention, the large-scale linear system of Equation (3) can be obtained by using the approximate method of the separated sub-linear system of Equation (6) The approximate method of solving the problem is solved. Using this method, a large linear system of equation (11) can be solved for the time complexity of O (height X width).

대각 행렬 D의 p번째 행, q번째 열에 있는 값, W(p,q):

연관성 행렬 W의 p번째 행, q번째 열에 있는 값,

: 수학식 10과 같은 연관성,

: 픽셀 p를 중심으로 상하좌우 픽셀, j: 픽셀 p의 상하좌우 픽셀 중 하나, H(p,q): 색인 대각 행렬 H의 p번째 행, q번째 열에 있는 값,

: 픽셀 p에서의 색인 함수,

:영향 파라미터이다. 수학식 12와 같이 연관성 행렬을 정의했을 때, 연관성 행렬 W는 각 행마다 4개의 원소를 가지는 행렬이 된다. Here, p: arbitrary pixel, q: arbitrary pixel, D (p, q):

The value in the pth row and qth column of the diagonal matrix D, W (p, q):

The values in the pth row, qth column of the relevance matrix W,

: ≪ / RTI > such as Equation 10,

(P, q): the value in the pth row, qth column of the index diagonal matrix H, H (p, q)

: The index function at pixel p,

: Influence parameter. When the association matrix is defined as in Equation (12), the association matrix W is a matrix having four elements for each row.

벡터, u1: 제 1보간 인터랙션 정보에 대응되는

벡터, u2: 제 2보간 인터랙션 정보에 대응되는

벡터, D: 수학식 12과 같은

대각 행렬, W: 수학식 12과 같은

연관성 행렬, H: 수학식 12과 같은

색인 대각 함수, g: 깊이 인터랙션 정보값에 대응되는

벡터, 1 모든 원소가 1의 값을 가지는

벡터,

: 영향 파라미터, 연산자./는 같은 크기의 벡터의 원소간에 나눗셈을 의미하는 연산자이다.Here, u: corresponding to the interpolation interaction information

Vector, u1: corresponding to the first interpolation interaction information

Vector, u2: corresponding to the second interpolation interaction information

Vector, D: < RTI ID = 0.0 >

Diagonal matrix, W:

H, < / RTI >< RTI ID = 0.0 >

Index diagonal function, g: depth corresponding to the depth interaction information value

Vector, 1 All elements have a value of 1

vector,

: Influence parameter, operator ./ is an operator that means division between elements of the same size vector.

The pixel value of the interpolation interaction information may be obtained by dividing the first interpolation interaction information by the second interpolation interaction information.

In the present invention, the large-scale linear system of Equation (3) can be approximated by using the method of approximating the separated sub-linear system of Equation (6) System approximation method. By using the above method, interpolation interaction information can be provided in time complexity of O (height X width).

: 픽셀 p를 중심으로 상하좌우 픽셀, q: 픽셀 p를 중심으로 상하좌우 픽셀 중 하나의 픽셀,

: 보정된 깊이 정보 영상에서 픽셀 p의 값,

: 보정된 깊이 정보 영상에서 픽셀 q의 값, up: 보간 인터랙션 정보,

: 깊이 정보 영상에서 픽셀 p의 값,

: 픽셀p와 픽셀 q의 연관성을 나타내는 연관성 함수,

: 수학식 15과 같은 제1 영향 파라미터,

: 제2 영향 파라미터,

: 제3 영향 파라미터이다.Here, p: arbitrary pixel,

: Upper and lower left and right pixels around pixel p, q: One pixel among pixels vertically and horizontally around pixel p,

: The value of the pixel p in the corrected depth information image,

: The value of the pixel q in the corrected depth information image, up: the interpolation interaction information,

: The value of the pixel p in the depth information image,

: A correlation function indicating the association of pixel p with pixel q,

: A first influence parameter as shown in equation (15)

: Second influence parameter,

: This is the third influence parameter.

The process of obtaining the corrected depth information image using the depth estimation information image and the interpolation interaction information is defined as a WLS problem as shown in Equation (10). The interpolation information is obtained by using the coordinate map in the video corresponding to the interaction information and the interaction information in the process of obtaining the interpolation interaction information, ≪ / RTI >

The first influence parameter, the second influence parameter, and the third influence parameter are parameters indicating how the first, second, and third terms in Equation (10) have an influence.

The first term in Equation (10) is a data term and includes the value of the pixel p in the interpolation interaction information. In Equation (10), the second term is a data term and includes the value of the pixel p in the depth information image. In Equation (10), the third term is a prior knowledge term and includes the value of the pixel p in the corrected depth information image. According to the third term in Equation (10), the corrected depth information image may be generated as an overall smoothed image.

If the arbitrary pixel is a pixel neighboring the depth interaction information, the value of the first influence parameter will be larger than the value of the second influence parameter, and any pixel adjacent to the depth interaction information will be highly affected by the interpolation interaction information .

If the arbitrary pixel is not a pixel neighboring the depth interaction information, the value of the first influence parameter will be smaller than the value of the second influence parameter, and any pixel not neighboring the depth interaction information will affect the depth information image You may not receive much.

In the present invention, the WLS problem of Equation (1) can be expressed as Equation (16) and Equation (17) as if the WLS problem of Equation 1 is obtained in an efficient time by using the method of approximating the separated subsystem of Equation And solved using a method of approximating a separate subsystem. By using the above method, it is possible to provide a depth information image corrected for time complexity (height X width).

: 제1 영향 파라미터, u2,p: 제2 보간 인터랙션 정보에서 픽셀 p의 값이다. From here,

: First influence parameter, u2, p: value of pixel p in the second interpolation interaction information.

The first influence parameter may be a value obtained by dividing 1000 by 1 (-64 x value of the pixel p in the second interpolation interaction information x constant) multiplied by e.

If the value of the second interpolation interaction information is equal to or greater than a predetermined value, the first influence parameter has a value close to 1000, and the influence of the first term in the data item of the expression (14) becomes larger.

: x축에서 픽셀 p의 두 이웃 픽셀, q: 픽셀 p를 중심으로 좌우 픽셀 중 하나의 픽셀,

: 행으로 분리된 보정된 깊이 정보 영상 중 픽셀 p의 값,

: 행으로 분리된 보정된 깊이 정보 영상 중 픽셀 q의 값,

: 행으로 분리된 보간 인터랙션 정보 중 픽셀 p의 값,

: 행으로 분리된 깊이 정보 영상 중 픽셀 p의 값,

: 수학식 10과 같은 픽셀p와 픽셀 q의 연관성,

: 제1 영향 파라미터,

: 제2 영향 파라미터,

: 제3 영향 파라미터이다.Here, p: arbitrary pixel,

: two neighboring pixels of pixel p in the x-axis, q: one of the left and right pixels around pixel p,

: The value of the pixel p in the corrected depth information image separated by the row,

: The value of the pixel q in the corrected depth information image separated by the row,

: The value of the pixel p among the row-separated interpolation interaction information,

: The value of the pixel p in the depth information image separated by the row,

: The relationship between the pixel p and the pixel q as shown in Equation 10,

: First influence parameter,

: Second influence parameter,

: This is the third influence parameter.

크기의 희소행렬을 포함하는 2차원 대규모 선형 시스템을 풀어야 하는데, 본 발명에서는 수학식 1의 WLS 문제를 수학식 5와 같은 분리된 서브 시스템의 근사화 방법을 사용하였다. 수학식 16와 수학식 17은 각각 수학식 14에 대한 분리된 X축 서브 시스템, 분리된 Y축 서브 시스템이다.To optimize the WLS problem of Equation 14

Dimensional linear system including a sparse matrix having a size of 1 to 5 is solved. In the present invention, the WLS problem of Equation (1) is used to approximate a separate subsystem such as Equation (5). Equations (16) and (17) are separate X-axis subsystems, separate Y-axis subsystems for (14), respectively.

는 높이x너비 크기의 보간 인터랙션 정보를 높이 개수만큼의 분리된 임의의 행 중 픽셀 p의 값을 의미한다.In Equation (16)

Means the value of the pixel p among the arbitrary rows separated by the height number of the interpolation interaction information of the height x width size.

는 높이x너비 크기의 깊이 정보 영상을 높이 개수만큼의 분리된 임의의 행 중 픽셀 p의 값을 의미한다.In Equation (16)

Denotes a value of a pixel p among arbitrary rows separated by a height number of the depth information image of the height x width size.

는 높이x너비 크기의 보정된 깊이 정보 영상 중 높이 개수만큼의 분리된 임의의 행 중 픽셀 p의 값을 의미한다.In Equation (16)

Means a value of a pixel p among arbitrary rows separated by the height of the corrected depth information image of the height x width size.

: y축에서 픽셀 p의 두 이웃 픽셀, q: 픽셀 p를 중심으로 상하 픽셀 중 하나의 픽셀,

: 열로 분리된 보정된 깊이 정보 영상 중 픽셀 p의 값,

: 열로 분리된 보정된 깊이 정보 영상 중 픽셀 q의 값,

: 열로 분리된 보간 인터랙션 정보 중 픽셀 p의 값,

: 열로 분리된 깊이 정보 영상 중 픽셀 p의 값,

: 수학식 10과 같은 픽셀p와 픽셀 q의 연관성,

: 제1 영향 파라미터,

: 제2 영향 파라미터,

: 제3 영향 파라미터이다.Here, p: arbitrary pixel,

: Two neighboring pixels of the pixel p in the y-axis, q: One pixel of the upper and lower pixels around the pixel p,

: The value of the pixel p in the corrected depth information image separated by the column,

: The value of the pixel q in the corrected depth information image separated by the column,

: The value of the pixel p among the interpolation information of the columns,

: The value of the pixel p in the depth information image separated by the column,

: The relationship between the pixel p and the pixel q as shown in Equation 10,

: First influence parameter,

: Second influence parameter,

: This is the third influence parameter.

크기의 희소행렬을 포함하는 2차원 대규모 선형 시스템을 풀어야 하는데, 본 발명에서는 수학식 1의 WLS 문제를 수학식 5와 같은 분리된 서브 시스템의 근사화 방법을 사용하였다. 수학식 16와 수학식 17은 각각 수학식 14에 대한 분리된 X축 서브 시스템, 분리된 Y축 서브 시스템이다.To optimize the WLS problem of Equation 14

Dimensional linear system including a sparse matrix having a size of 1 to 5 is solved. In the present invention, the WLS problem of Equation (1) is used to approximate a separate subsystem such as Equation (5). Equations (16) and (17) are separate X-axis subsystems, separate Y-axis subsystems for (14), respectively.

는 높이x너비 크기의 보간 인터랙션 정보를 너비 개수만큼의 분리된 임의의 열 중 픽셀 p의 값을 의미한다.In Equation 17

Means the value of the pixel p among arbitrary columns separated by the number of widths of the interpolation interaction information of the height x width size.

는 높이x너비 크기의 깊이 정보 영상을 너비 개수만큼의 분리된 임의의 열 중 픽셀 p의 값을 의미한다.In Equation 17

Means the value of the pixel p among the arbitrary columns separated by the number of widths in the depth information image of the height x width size.

는 높이x너비 크기의 보정된 깊이 정보 영상 중 너비 개수만큼의 분리된 임의의 열 중 픽셀 p의 값을 의미한다.In Equation 17

Denotes a value of a pixel p among arbitrary columns separated by the number of widths from the corrected depth information image of height x width size.

: 수학식 20와 같은 너비x너비 대각 행렬,

: 행으로 분리된 보정된 깊이 정보 영상에 대응되는 너비x1 벡터,

: 행으로 분리된 보간 인터랙션 정보에 대응되는 너비x1 벡터,

: 수학식 21과 같은 너비x너비 연관성 행렬,

: 행으로 분리된 깊이 영상 정보에 대응되는 너비x1 벡터,

: 제1 영향 파라미터,

: 제2 영향 파라미터,

: 제3 영향 파라미터이다.From here,

: Width x width diagonal matrix as in Equation 20,

: A width x1 vector corresponding to the corrected depth information image separated by a row,

: A width x1 vector corresponding to the interpolation interaction information separated by a row,

: Width x width relevance matrix as shown in equation 21,

: A width x1 vector corresponding to depth image information separated by a row,

: First influence parameter,

: Second influence parameter,

: This is the third influence parameter.

는

에서 너비x너비 크기로 줄어들었다. 또한, 행렬

의 각 행마다 3개의 원소를 가지는 특성 덕분에 역행렬 연산 과정 중 수행 속도를 더욱 효율적으로 단축시킨다. 그 결과 위 선형 시스템을 이용한 깊이 추정 영상의 보정은 O(높이x너비)의 시간 복잡도로 단축된다.Equation (18) is a linear system derived from the optimization process of Equation (16), and thanks to the method of approximating a separate subsystem used in the present invention,

The

To width x width. Also,

The performance of the inverse matrix computation process is more efficiently shortened owing to the characteristics having three elements in each row. As a result, the correction of the depth estimation image using the linear system is shortened to the time complexity of O (height x width).

 The diagonal matrix may include interpolation interaction information and depth image information. The similarity matrix may be composed of an association function.

: 열로 분리된 보정된 깊이 정보 영상에 대응되는 높이x1 벡터,

: 열로 분리된 보간 인터랙션 정보에 대응되는 높이x1 벡터,

: 수학식 21과 같은 높이x높이 연관성 행렬,

: 열로 분리된 깊이 영상 정보에 대응되는 높이x1 벡터,

: 제1 영향 파라미터,

: 제2 영향 파라미터,

: 제3 영향 파라미터이다.Where: height x height diagonal matrix as in equation 20,

: A height x1 vector corresponding to the corrected depth information image separated into columns,

: Height x1 vector corresponding to the interpolation interaction information separated by the column,

: Height-x-height correlation matrix as shown in equation (21)

: Height x1 vector corresponding to the depth image information separated by the row,

: First influence parameter,

: Second influence parameter,

: This is the third influence parameter.

는

에서 높이x높이 크기로 줄어들었다.Equation (19) is a linear system derived from the optimization process of Equation (17), and thanks to the method of approximating a separate subsystem used in the present invention,

The

To the height x height.

의 각 행마다 3개의 원소를 가지는 특성 덕분에 역행렬 연산 과정 중 수행 속도를 더욱 효율적으로 단축시킨다. 그 결과 위 선형 시스템를 이용한 깊이 추정 영상의 보정은 O(높이x너비)의 시간복잡도로 단축된다.Also,

The performance of the inverse matrix computation process is more efficiently shortened owing to the characteristics having three elements in each row. As a result, the correction of the depth estimation image using the linear system is shortened to the time complexity of O (height x width).

The diagonal matrix may include interpolation interaction information and depth image information. The similarity matrix may be composed of an association function.

: 높이x높이 대각 행렬

의 p번째 행, q번째 열에 있는 값,

: 너비x너비 대각 행렬

의 p번째 행, q번째 열에 있는 값, p: 임의의 픽셀, q: 임의의 픽셀,

: 행으로 분리된 보간 인터랙션 정보 중 픽셀 p의 값,

: 열로 분리된 보간 인터랙션 정보 중 픽셀 p의 값,

: 행으로 분리된 깊이 정보 영상 중 픽셀 p의 값,

: 열로 분리된 깊이 정보 영상 중 픽셀 p의 값,

: 수학식 10과 같은 픽셀 p와 픽셀 j 간의 연관성,

: 픽셀 p의 좌우 픽셀,

: 픽셀 p의 상하 픽셀,

: 행으로 분리된 제1 영향 파라미터,

: 열로 분리된 제1 영향 파라미터,

: 제2 영향 파라미터,

: 제3 영향 파라미터이다.From here,

: Height x height diagonal matrix

The value in the pth row, the qth column of < RTI ID = 0.0 >

: Width x width diagonal matrix

The value in the pth row, the qth column of the pixel, p: an arbitrary pixel, q: an arbitrary pixel,

: The value of the pixel p among the row-separated interpolation interaction information,

: The value of the pixel p among the interpolation information of the columns,

: The value of the pixel p in the depth information image separated by the row,

: The value of the pixel p in the depth information image separated by the column,

: The relationship between the pixel p and the pixel j as shown in Equation 10,

: The left and right pixels of pixel p,

: Upper and lower pixel of pixel p,

: A first influence parameter separated by a row,

: A first effect parameter separated by heat,

: Second influence parameter,

: This is the third influence parameter.

: 분리된 X축 서브시스템에 대한 유사성 행렬

에서 좌표 (p,q)에 해당하는 값,

: 분리된 Y축 서브시스템에 대한 유사성 행렬

에서 좌표 (p,q)에 해당하는 값, p: 임의의 p번째 행, q: 임의의 q번째 열,

(p): x축에서 픽셀 p의 두 이웃 픽셀,

(p): y축에서 픽셀 p의 두 이웃 픽셀,

: 픽셀 p와 픽셀 q의 연관성을 나타내는 연관성 함수이다. From here,

: Similarity matrix for discrete X-axis subsystems

A value corresponding to the coordinates (p, q)

: Similarity matrix for discrete Y-axis subsystems

(P, q), p: arbitrary p-th row, q: arbitrary q-th column,

(p): two neighboring pixels of pixel p in the x-axis,

(p): two neighboring pixels of pixel p in the y-axis,

Is an associative function that indicates the association of pixel p with pixel q.

Two neighboring pixels of the pixel p in the x-axis may refer to the left and right pixels with respect to the pixel p. Two neighboring pixels of the pixel p in the y-axis may refer to the upper and lower pixels based on the pixel p. The association function indicating the association between the pixel p and the pixel q may be defined using Equation (12).

라플라시안 행렬, I:

단위 행렬,

:영향 파라미터, m: 임의의 픽셀, h1: 수학식 19와 같은 색인 함수, h2: 수학식 19와 같은 색인 함수이다. Here, A: as in Equation 20

Laplacian matrix, I:

Unit matrix,

: An influence parameter, m: arbitrary pixel, h1: an index function as shown in equation (19), and h2: an index function as shown in equation (19).

과

는 분할 영상 생성시에 국한되어 사용되며, 상기 색인 함수

와는 다르다.The index function

and

Is used only when a divided image is generated, and the index function

.

Equation (22) is a mathematical expression for generating an object-background divided image in the divided image generating unit 130. Equation (22) is a linear system such as Equation (3) that is derived during the optimization process after defining the object-background division process as a WLS problem such as Equation (1). In the present invention, a large-scale linear system of Equation (3) is solved efficiently by a method of approximating the large-scale linear system of Equation (6) into a separate subsystem of Equation (6).

(m): 임의의 픽셀 m에 대한 색인 함수의 요소,

(m): 임의의 픽셀 m에 대한 색인 함수의 요소이다.Here, m: arbitrary pixel,

(m): an element of the index function for any pixel m,

(m) is an element of the index function for any pixel m.

과

는 분할 영상 생성시에 국한되어 사용되며,상기 색인 함수

와는 다르다.The index function

and

Is used only when a divided image is generated, and the index function

.

라플라시안 행렬A에서 좌표 (m,n)에 해당하는 값,

: 컬러 영상에서 픽셀 m에 해당하는 값,

: 컬러 영상에서 픽셀 l에 해당하는 값,

: 컬러 영상에서 픽셀 n에 해당하는 값,

: 컬러 특징 값에 대한 정규화 파라미터, N(m): 픽셀 m을 중심으로 상하좌우 픽셀, l: 픽셀 m을 중심으로 상하좌우 픽셀 중 하나의 픽셀, n: 픽셀 m을 중심으로 상하좌우 픽셀 중 하나의 픽셀이다.Here, m: arbitrary mth row, n: arbitrary nth column, A (m, n):

A value corresponding to the coordinate (m, n) in the Laplacian matrix A,

: A value corresponding to a pixel m in a color image,

: A value corresponding to a pixel l in a color image,

: A value corresponding to pixel n in a color image,

: Normalized parameter for color feature value, N (m): Upper and lower left and right pixel around pixel m, l: One pixel of upper and lower left and right pixel around pixel m, n: One of upper, / RTI >

Here, the computer program for executing the correction method of the depth information image based on the user interaction information according to another embodiment of the present invention may be a computer program stored in the medium for executing the underwater guided weapon defense method according to the present invention .

It is to be understood that the present invention is not limited to these embodiments, and all elements constituting the embodiment of the present invention described above are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them.

In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer to implement an embodiment of the present invention. As the recording medium of the computer program, a magnetic recording medium, an optical recording medium, or the like can be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Commonly used terms, such as predefined terms, should be interpreted to be consistent with the contextual meanings of the related art, and are not to be construed as ideal or overly formal, unless expressly defined to the contrary.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate and not to limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (15)

A receiving unit for receiving a depth information image estimated by at least one depth estimating apparatus and a color information image picked up by at least one color photographing apparatus;
An input unit for inputting interaction information into the received color information image;
A divided image generating unit for generating a divided image for distinguishing an object and a background using the inputted interaction information and the received color information image;
An interpolation unit interpolating the input interaction information using an interpolation method; And
A correcting unit correcting the estimated depth information image using the interpolated interaction information, the color information image, the divided image, and the global objective function; Lt; / RTI >
The global objective function may be expressed as:
A data item using at least one of the interpolated interaction information and the estimated depth information image; And
Wherein the depth information comprises at least one of the segmented image and the color information image. The apparatus of claim 1, wherein the correcting unit
Defining the global objective function for correction using the interpolated interaction information and the estimated depth information image and correcting the estimated depth information image by finding a solution of the global objective function, User Interaction Information Based Depth Information Image Correction Device. 3. The method of claim 2, wherein the interpolated interaction information comprises:
The interpolated depth interaction information, and the segmented image. ≪ RTI ID = 0.0 > [100] < / RTI > delete The apparatus of claim 1, wherein the correcting unit
Wherein the depth information image is replaced with the identified divided image, wherein the divided image is mapped to the depth information image, and the depth information image is replaced with the identified divided image. The apparatus of claim 1, wherein the interpolator
Wherein at least one of the depth information image, the coordinate map corresponding to the depth information image, the index function, the association function, and the segmented image is used. 2. The method of claim 1, wherein if the global objective function is used,
And an optimization method using an approximation process is used to convert the global objective function into a one-dimensional linear system. Receiving a depth information image estimated by at least one depth estimation device and a color information image photographed by at least one color photographing device;
Inputting interaction information to the received color information image;
Generating a divided image for distinguishing an object and a background using the input interaction information and the received color information image;
Interpolating the input interaction information using an interpolation method;
Correcting the estimated depth information image using the interpolated interaction information, the color information image, the divided image, and the global objective function; Lt; / RTI >
The global objective function may be expressed as:
A data item using at least one of the interpolated interaction information and the estimated depth information image; And
A prior knowledge item using at least one of the divided image and the color information image; And generating a depth information image based on the user interaction information. 9. The method of claim 8, wherein the correcting comprises:
Defining the global objective function for correction using the interpolated interaction information and the estimated depth information image and correcting the estimated depth information image by finding a solution of the global objective function, User Interaction Information Based Depth Information Image Correction Method. 10. The method of claim 9, wherein the interpolated interaction information
The interpolated depth interaction information, and the segmented image. ≪ Desc / Clms Page number 26 > delete 9. The method of claim 8, wherein the correcting comprises:
Wherein the depth information image is replaced with the segmented image, wherein the segmented image is mapped to the depth information image, and the depth information image is replaced with the segmented image. 9. The method of claim 8, wherein interpolating the input interaction information comprises:
Wherein the interpolation information interpolation unit interpolates the input interaction information using at least one of the depth information image, the coordinate map corresponding to the depth information image, the index function, the association function, Way. 9. The method of claim 8, wherein if the global objective function is used,
And an optimization method using an approximation process is used to convert the global objective function into a one-dimensional linear system. A computer-readable medium having stored thereon a computer-readable medium for executing the user interaction-based depth information image correction method according to any one of claims 8 to 10, 12, 13 and 14, program.

Images