KR20170120567A - Method and apparatus for extracting depth information from an image - Google Patents

Abstract

In one embodiment of the present invention, there is provided a method of acquiring a first image of an object using a first iris, acquiring a second image of the object using a second iris, Obtaining a first blur patch for a first patch based on a first point spread function (PSF), and obtaining a second blur patch for a first patch based on a second point spread function (PSF) Acquiring a third blur patch through interpolation of the obtained first blur patch and second blur patch respectively, setting a second patch on the second image corresponding to the first patch, A method and apparatus are disclosed for extracting depth information from an image comprising the step of chasing depth information based on each of a blur patch, a second blur patch or a third blur patch and a second patch.

Description

Method and apparatus for extracting depth information from an image

The present invention relates to a method and apparatus for extracting depth information from an image, and more particularly to extracting depth information based on images photographed using different diaphragms of a double-iris system, To a method and apparatus for extracting depth information from an image selectively using point spread functions having standard deviation.

CMOS image sensor (CIS), which can be included in an object image pickup device widely used in recent years due to expansion of digital cameras, is an apparatus for converting an external optical image signal into an electric image signal, -Coupled Device), it is possible to operate at a low voltage and the power consumption is low. In addition, the CMOS image sensor (CIS) is used in various fields such as digital cameras as well as various portable device devices because of advantages advantageous in integration.

In a general object image pickup apparatus, it is required to use a filter for blocking infrared rays. A human eye recognizes an object by reflecting and absorbing infrared rays and ultraviolet rays and transmitting only visible rays through a light component, while an optical instrument such as a digital camera transmits all the light to capture an object image. In an optical device such as an object imaging device, ultraviolet light can be selectively blocked by an ultraviolet shielding coating applied to the lens. However, the use of an IR cut-off filter is required to selectively block infrared (IR) from optical devices. For example, a single-lens double-iris camera system may include an iris filter for selectively blocking infrared rays in addition to a diaphragm for adjusting the amount of light such as visible light.

In order to acquire depth information or depth information of an object, various techniques such as comparing or analyzing multiview images or selectively comparing IR channel images with color channel images obtained by selectively controlling IR blocking filters Can be utilized.

According to an embodiment of the present invention, in extracting depth information based on images photographed using different diaphragms of a double diaphragm system, depth diffusion functions having different standard deviations are selectively used to extract depth information Can be extracted.

Extraction algorithms for distance information or depth information of an existing object are designed to perform computations for each of the point spread functions, so that the computation amount is excessive, and according to the method and apparatus according to an embodiment of the present invention, The depth information can be extracted from the image by selectively using the diffusion functions.

The single-lens double-iris camera system can extract distance information of an object or depth information of an object. Conventional distance information extraction algorithms are inefficient because a considerable amount of computation is required. According to the embodiment of the present invention, there is an effect that the amount of computation can be greatly reduced as compared with the existing distance information extraction algorithm, and at the same time, the accuracy is not greatly lost.

Figure 1 is a schematic diagram of a single lens double-iris camera system.
2 is a conceptual diagram of a conventional distance information extraction algorithm.
3 is a flowchart illustrating a method for extracting depth information from an image in accordance with an embodiment of the present invention.
4 is a conceptual diagram of an algorithm for extracting depth information from an image in accordance with an embodiment of the present invention.
5 is a graph for explaining an interpolation method according to an embodiment of the present invention.
6 is a schematic diagram of an interpolation method according to an embodiment of the present invention.
7 is a block diagram of an apparatus for extracting depth information from an image in accordance with an embodiment of the present invention.

Best Mode for Carrying Out the Invention

A method for extracting depth information from an image in accordance with an embodiment of the present invention includes obtaining a first image for an object using a first iris and for obtaining a second image for an object using a second iris, Obtaining a first blur patch for a first patch set based on a first point spread function (PSF), acquiring a first blur patch for a first patch based on a first point spread function (PSF) Acquiring a second blur patch for a first patch set based on a second blur patch (PSF), acquiring a third blur patch through interpolation of the obtained first blur patch and second blur patch respectively, Setting the second patch corresponding to the first patch, and estimating the depth information based on each of the first blur patch, the second blur patch or the third blur patch, and the second patch.

The first patch of the predetermined area set on the first image according to an embodiment of the present invention may be an image area set to a predetermined area so that the edge of the object included in the first image is included.

Each of the first point spread function PSF and the second point spread function PSF according to an exemplary embodiment of the present invention has N different standard deviation values when the depth resolution is N (N is a natural number) / RTI > may be any of the point spread functions having < RTI ID = 0.0 >

The step of acquiring the third blur patch through interpolation of the obtained first blur patch and second blur patch, respectively, according to an embodiment of the present invention includes acquiring a first blur patch (PSF) and a second point spread function ) To obtain a third blur patch.

An apparatus for extracting depth information from an image in accordance with an embodiment of the present invention includes a first image acquisition unit for acquiring a first image for an object using a first iris and a second image for an object using a second iris, A patch setting unit for setting a first patch of a predetermined area on the first image and setting a second patch on the second image in correspondence with the first patch, a first point spread function (PSF) A blur patch acquiring unit acquiring a first blur patch for the first patch based on the second point spread function (PSF) and acquiring a second blur patch for the first patch based on the second point spread function (PSF) And a depth estimation unit for estimating depth information based on the interpolation unit interpolating the second blur patch, the first blur patch, the second blur patch or the third blur patch and the set second patch, and the third blur patch Interpolation And it may be obtained by the blur patch obtaining unit based on the interpolation performed by the.

The first patch of the predetermined area set on the first image according to an embodiment of the present invention may be an image area set to a predetermined area so that the edge of the object included in the first image is included.

 Each of the first point spread function PSF and the second point spread function PSF according to an exemplary embodiment of the present invention includes N point spread functions having N different standard deviation values when the depth resolution is N, , And N may be a natural number.

The interpolator according to an exemplary embodiment of the present invention may perform interpolation using a weighted average of a first point spread function (PSF) and a second point spread function (PSF).

 According to another aspect of the present invention, there is provided a computer-readable recording medium storing a program for causing a computer to execute the above-described method.

DETAILED DESCRIPTION OF THE INVENTION

The terms used in this specification will be briefly described and the present invention will be described in detail.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. Also, in certain cases, there may be a term selected arbitrarily by the applicant, in which case the meaning thereof will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term, not on the name of a simple term, but on the entire contents of the present invention.

When an element is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements, without departing from the spirit or scope of the present invention. Also, the terms "part," " module, "and the like described in the specification mean units for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Figure 1 is a schematic diagram of a single lens double-iris camera system.

The user can obtain a color channel image and an infrared channel image using the single lens double iris camera system 100 as shown in FIG. The lens barrel 140 may include a diaphragm 110 for adjusting the amount of visible light to be transmitted, a light blocking filter 120 for selectively blocking infrared rays, and at least one lens 130. The aperture and the light passing through the lens can be sensed in a CMOS image sensor (CIS) 150. Such a CMOS image sensor 150 may be connected to other electronic modules, for example, via a substrate such as a PCB.

A color channel image and an infrared channel image can be obtained according to the amount and amount of light detected by the CMOS image sensor. The color channel image may be referred to as a color image for an object, and the infrared channel image may be referred to as an infrared image. An image acquired with a small aperture aperture may have less blur than an image acquired with a large aperture aperture, depending on the object distance from the focal distance.

The distance information of the object can be obtained by acquiring the color channel image and the infrared channel image, extracting the edge of the object from the obtained images, and performing the depth estimation on the extracted edge. The depth information of the object in the image can be obtained by comparing and analyzing the color channel image and the infrared channel image. In other words, the distance information of the object can be obtained based on the edge of the color channel image and the infrared channel image. Conventionally, there is a distance information extraction algorithm for obtaining distance information of an object, but there is a problem in that a considerable calculation amount is consumed.

2 is a conceptual diagram of a conventional distance information extraction algorithm.

A blurred image can be obtained through an operation with a point spread function having a different standard deviation with respect to a predetermined area (for example, a box area) in the infrared image 10. The predetermined area may be referred to as a patch or the like. Such an operation may include a convolution operation. For example, for a depth resolution of 10, each of the point spread functions having ten different standard deviation values (e.g.,? 0 to? 9) can be convoluted with a predetermined area in the infrared image 10 have. For each of the blurred patches obtained through operation with the point spread functions having ten different standard deviation values (e.g., 0 to 9) The similarity and distance values (e.g., PSF index) may be obtained. The distance information of the object can be estimated using the obtained similarity and distance values. As described above, the conventional distance information extraction algorithm has to be operated on all of the point spread functions having different standard deviation values, so that the calculation amount is increased and the complexity is increased, so that the distance information extraction efficiency may be lowered.

3 is a flowchart illustrating a method for extracting depth information from an image in accordance with an embodiment of the present invention.

A method for extracting depth information from an image in accordance with an embodiment of the present invention includes obtaining a first image for an object using a first iris and for obtaining a second image for an object using a second iris, (S200) of setting a first patch of a predetermined area on the first image (S200); acquiring a first blur patch for the first patch based on a first point spread function (PSF) (S300) of acquiring a second blur patch for the first patch based on the second point spread function (PSF), and acquiring a second blur patch for the third patch through the interpolation of the obtained first blur patch and the second blur patch (S500) of setting a second patch corresponding to the first patch on the second image (S500); acquiring a blur patch, a second blur patch or a third blur patch, and a second patch And estimating the depth information based on the depth information S600). The first patch of the predetermined area set on the first image may be an image area set to a predetermined area so that the edge of the object included in the first image is included. A method for extracting depth information from an image according to an embodiment of the present invention will be described below with reference to FIG.

4 is a conceptual diagram of an algorithm for extracting depth information from an image in accordance with an embodiment of the present invention.

A first image for the object may be obtained using the first diaphragm in accordance with an embodiment of the present invention. The first diaphragm may be an infrared (IR) cutoff filter for selectively blocking infrared rays. The first image may be an infrared image, but is not necessarily limited thereto. Also, a second image for the object can be obtained using the second aperture. The second diaphragm may be a diaphragm normally used in a camera system or the like. Also, for convenience of explanation, an infrared image is referred to as an example of a first image, and the color channel image is described as an example of a second image, but the present invention is not limited thereto. In other words, the method according to one embodiment of the present invention can be generally used in a dual aperture (DA) system capable of taking an image using different diaphragms.

A first patch 11 of a predetermined area may be set on the first image 10. [ The first patch 11 of a predetermined area set on the first image 10 may be an image area set to a predetermined area so that at least one edge in the first image 10 is included.

The first blur patch 1 for the first patch 11 can be obtained based on the first point spread function (PSF). The first blur patch 1 can be obtained through a combination of the first point spread function PSF having the first standard deviation value (e.g.,? 0) and the first patch 11. The combination of the first point spread function (PSF) and the first patch 11 may include a convolution operation (C1) or the like. Further, the second blur patch 2 for the first patch 11 can be obtained based on the second point spread function (PSF). The second blur patch 2 can be obtained through the combination of the first point patch 11 and the second point spread function PSF having the second standard deviation value (e.g.,? 2). The combination of the second point spread function (PSF) and the first patch 11 may include a convolution operation (C2) or the like. A blur patch obtained through combination with a point spread function (PSF) may be referred to as a basis.

According to an embodiment of the present invention, the third blur patch 3 can be obtained through interpolation of the obtained first blur patch 1 and second blur patch 2, respectively. In other words, the third blur patch 3 can be obtained through the interpolation process I1 of the convolution operation (C1 and C2) results. As mentioned earlier, as compared to the conventional distance estimation method of performing convolution for each of all the point spread functions having ten different standard deviation values (e.g., 0 to 9), according to an embodiment of the present invention Since the depth information extraction method performs a convolution operation with respect to a part of the point spread functions (e.g.,? 0,? 2,? 5 and? 9) with the first patch 11, the amount of computation can be greatly reduced, Can be reduced. As shown in FIG. 4, for example, each of the first point spread function PSF and the second point spread function PSF has a point spread having N different standard deviation values when the depth resolution is N, Functions. ≪ / RTI > At this time, N may be a natural number.

According to an embodiment of the present invention, the second patch 21 may be set on the second image 20 in correspondence with the first patch 11. The position and area in the second image 20 of the second patch 21 may be the same patch as at least one of the position and the area of the first patch 11. [

Depth information can be estimated based on each of the first blur patch, the second blur patch or the third blur patch and the second patch. As shown in FIG. 4, the depth information can be estimated through the similarity analysis between each blur patch and the second patch. For example, for each of the blur patches (e.g., the first blur patch, the second blur patch, or the third blur patch), a comparison is made between the patches through the comparison with the second patch 21 in the second image 20 The similarity and distance values (e.g., PSF index) may be obtained. The acquired similarity and distance values (e.g., PSF index) may be included in the depth information.

Interpolation (I1) of the obtained first blur patch and second blur patch according to an embodiment of the present invention may include interpolation as a weighted average of a first point spread function and a second point spread function.

5 is a graph for explaining an interpolation method according to an embodiment of the present invention.

The PSF according to an embodiment of the present invention can be defined by the following equation (1).

x and y represent the positional coordinates on the plane of the PSF, and σk represents the standard deviation. Interpolation can be performed through the weighted average of the first point spread function f1 and the second point spread function f2. In other words, the interpolation function f3 is a weighted average of the first point spread function f1 and the second point spread function f2, and can be defined by the following equation (2).

α and β denote the coefficients of the weighted average. For example, as shown in FIG. 5B, when the first point spread function f1 has a first standard deviation value (e.g.,? A) and the second point spread function f2 has a second standard deviation value (e.g.,? B The interpolation function f3 that can be obtained by the weighted average of the first point spread function f1 and the second point spread function f2 can be obtained by multiplying the first standard deviation value a by the weight coefficient, And a point spread function f4 having a standard deviation value? C between two standard deviation values? B. In other words, it is possible to obtain a result approximated to the conventional convolution operation result by interpolation according to the adjustment of the weighting coefficient according to an embodiment of the present invention. Therefore, according to the embodiment of the present invention, the amount of computation can be greatly reduced compared to the conventional art.

6 is a schematic diagram of an interpolation method according to an embodiment of the present invention.

An n-th blur patch is acquired using a point spread function having an n-th standard deviation value (? N) with respect to the first patch (11) on an infrared image, and a point having an n + 1 standard deviation value It is possible to acquire the (n + 1) -th blur patch using the diffusion function. At least one blur patch Pk between the n-th blur patch and the (n + 1) -th blur patch includes a point spread function having an n-th standard deviation value? N, a convolution operation result of the first patch 11, Can be obtained as a weighted average of a point spread function having a +1 standard deviation value (? N + 1) and a convolution operation result of the first patch (11). In other words, a blur patch Bk (n) obtained through an operation with a point spread function f4 having a standard deviation value sigk between the nth standard deviation value sigma n and the n + 1 standard deviation value sigma n + Can be obtained in approximation to the blur patch Pk described above. The above-described interpolation process related to FIG. 6 can be expressed by the following equations (3) to (6).

7 is a block diagram of an apparatus for extracting depth information from an image in accordance with an embodiment of the present invention.

An apparatus 700 for extracting depth information from an image in accordance with an embodiment of the present invention includes means for obtaining a first image for an object using a first iris and a second image for an object using a second iris, A patch setting unit 720 that sets a first patch of a predetermined area on the first image and sets a second patch on the second image in accordance with the first patch, Acquiring a first blur patch for a first patch set based on a diffusion function (PSF) and acquiring a second blur patch for a first patch set based on a second point spread function (PSF) An interpolation section 740 for interpolating the obtained first blur patch and the second blur patch, respectively, and depth information 730 based on the first blur patch, the second blur patch or the third blur patch, A depth estimator 750 for estimating a depth It can hamhal. The third blur patch may be acquired by the blur patch acquisition section 730 based on the interpolation performed by the interpolation section 740. [

The first patch of the predetermined area set on the first image according to an embodiment of the present invention may be an image area set to a predetermined area so that the edge of the object included in the first image is included.

Each of the first point spread function PSF and the second point spread function PSF according to an embodiment of the present invention has N different standard deviation values when the depth resolution is N (N is a natural number) / RTI > may be any of the point spread functions having < RTI ID = 0.0 > In other words, only some of the point spread functions having N different standard deviation values can be combined with the first patch. Such a combination may include a convolution operation.

Interpolator 740 may perform interpolation as a weighted average of the first blur patch and the second blur patch.

The contents of the above-described method can be applied in connection with the apparatus according to an embodiment of the present invention. Therefore, the description of the same contents as those of the above-described method with respect to the apparatus is omitted.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

Claims (9)

A method for extracting depth information from an image,
Obtaining a first image for the object using the first aperture and acquiring a second image for the object using the second aperture;
Setting a first patch of a predetermined area on the first image;
Acquiring a first blur patch for the set first patch based on a first point spread function (PSF), and obtaining a second blur patch for the second patch based on the second point spread function (PSF) Obtaining a blur patch;
Acquiring a third blur patch through interpolation of the obtained first blur patch and second blur patch, respectively;
Setting a second patch on the second image corresponding to the first patch; And
Estimating depth information based on each of the first blur patch, the second blur patch or the third blur patch and the set second patch. The method according to claim 1,
Wherein the first patch of the predetermined area set on the first image is an image area set to a predetermined area such that an edge of the object included in the first image is included. The method according to claim 1,
Wherein each of the first point spread function PSF and the second point spread function PSF is one of a point spread function having N different standard deviation values when the depth resolution is N, Is a natural number. The method according to claim 1,
Wherein acquiring the third blur patch through interpolation of the obtained first blur patch and second blur patch, respectively,
Interpolating the first point spread function (PSF) and the second point spread function (PSF) to obtain a third blur patch. An apparatus for extracting depth information from an image,
An image acquiring unit acquiring a first image for the object using the first iris and acquiring a second image for the object using the second iris;
A patch setting unit that sets a first patch of a predetermined area on the first image and sets a second patch on the second image in accordance with the first patch;
Acquiring a first blur patch for the set first patch based on a first point spread function (PSF), acquiring a second blur patch for the set first patch based on a second point spread function (PSF) A blur patch acquisition unit;
An interpolator interpolating the obtained first blur patch and the second blur patch; And
And a depth estimation unit for estimating depth information based on each of the first blur patch, the second blur patch or the third blur patch and the set second patch,
And the third blur patch is obtained by the blur patch acquisition section based on interpolation performed by the interpolation section. 6. The method of claim 5,
Wherein the first patch of the predetermined area set on the first image is an image area set to a predetermined area so that an edge of the object included in the first image is included. 6. The method of claim 5,
Wherein each of the first point spread function PSF and the second point spread function PSF is one of a point spread function having N different standard deviation values when the depth resolution is N, Is a natural number. 6. The method of claim 5,
Wherein the interpolator performs interpolation as a weighted average of the first point spread function (PSF) and the second point spread function (PSF). A computer-readable recording medium on which a program for implementing the method of any one of claims 1 to 4 is recorded.

Images