WO2016117716A1

WO2016117716A1 - Method and apparatus for extracting depth information from image

Info

Publication number: WO2016117716A1
Application number: PCT/KR2015/000556
Authority: WO
Inventors: 이영민; 박현상; 경종민
Original assignee: 재단법인 다차원 스마트 아이티 융합시스템 연구단
Priority date: 2015-01-20
Filing date: 2015-01-20
Publication date: 2016-07-28
Also published as: KR20170120567A

Abstract

As an embodiment of the present invention, a method and apparatus for extracting depth information from an image is disclosed, the method comprising the steps of: acquiring a first image of an object using a first aperture and acquiring a second image of the object using a second aperture; configuring a first patch having a predetermined area on the first image; acquiring a first blurred patch for the first patch on the basis of a first point spread function (PSF) and acquiring a second blurred patch for the first patch on the basis of a second point spread function (PSF); acquiring a third blurred patch through interpolation of the acquired first and second blurred patches; configuring a second patch on the second image such that the second patch corresponds to the first patch; and estimating depth information on the basis of each of the first, second, and third blurred patches and the second patch.

Description

Method and apparatus for extracting depth information from an image

The present invention relates to a method and apparatus for extracting depth information from an image, and more particularly to extracting depth information based on images photographed using different apertures of a dual aperture system. A method and apparatus for extracting depth information from an image using selectively point spreading functions having standard deviations.

CMOS Image Sensor (CIS), which can be included in object image capturing devices that are used in recent years due to the spread of digital cameras, is a device that converts an external optical image signal into an electrical image signal. Compared to the coupled device, it can operate at low voltage and consumes less power. In addition, the CMOS image sensor (CIS) has been used in various fields such as not only digital cameras but also various portable device devices due to the advantage of integration.

In general object image pickup devices, the use of a filter to block infrared rays is required. The human eye perceives an object by reflecting or absorbing infrared rays and ultraviolet rays and transmitting only visible light in the light component, whereas optical devices such as digital cameras can capture object images by transmitting all light. In optical devices such as object image pickup devices, ultraviolet light can be selectively blocked by a sunscreen coating applied to the lens. However, the use of an IR cut-off filter is required to selectively block infrared rays (IR) in optical devices. For example, the single lens dual aperture camera system may include an IR blocking filter for selectively blocking infrared rays in addition to the aperture for adjusting the amount of light such as visible light.

In order to obtain depth information or depth information about an object, various techniques such as comparing or analyzing multiview images or selectively controlling an IR cut filter to compare or analyze an infrared channel image and a color channel image acquired Can be utilized.

According to an embodiment of the present invention, in extracting depth information based on images photographed using different apertures of a dual aperture system, depth information from an image is selectively used by using point spreading functions having different standard deviations. Can be extracted.

Algorithms for extracting the distance information or depth information of the existing object are to operate on all of the point spreading functions, and according to the method and apparatus according to the exemplary embodiment of the present invention, the calculations have different standard deviations. Depth information may be extracted from the image by selectively using the diffusion functions.

Using a single lens dual aperture camera system, distance information of an object or depth information of an object may be extracted. Existing distance information extraction algorithms are inefficient because they require a significant amount of computation. According to an embodiment of the present invention, the amount of computation can be greatly reduced compared to the existing distance information extraction algorithm, and at the same time, the accuracy is not greatly reduced.

1 is a schematic diagram of a single lens dual aperture camera system.

2 is a conceptual diagram of a conventional distance information extraction algorithm.

3 is a flowchart illustrating a method for extracting depth information from an image according to an embodiment of the present invention.

4 is a conceptual diagram of an algorithm for extracting depth information from an image according to an embodiment of the present invention.

5 is a graph illustrating an interpolation method according to an embodiment of the present invention.

6 is a schematic diagram of an interpolation method according to an embodiment of the present invention.

7 is a block diagram of an apparatus for extracting depth information from an image according to an embodiment of the present invention.

A method for extracting depth information from an image according to an exemplary embodiment of the present invention may include obtaining a first image of an object using a first aperture and using a second aperture to obtain a second image of the object. Obtaining a first blur patch for the set first patch based on the first point spreading function (PSF), and obtaining a second point spreading function on the first image. Obtaining a second blur patch for the first patch set based on the (PSF), obtaining a third blur patch through interpolation of the obtained first blur patch and the second blur patch, respectively, on the second image Setting a second patch corresponding to the first patch, and estimating depth information based on each of the first blur patch, the second blur patch, or the third blur patch, and the second patch.

The first patch of a predetermined area set on the first image according to an embodiment of the present invention may be an image area set to a predetermined area to include an edge of an object included in the first image.

Each of the first point spread function (PSF) and the second point spread function (PSF) according to an embodiment of the present invention may have N different standard deviation values when the depth resolution is N (N is a natural number). It may be any one of the point spreading function.

Acquiring the third blur patch through interpolation of the first blur patch and the second blur patch, respectively, according to an embodiment of the present invention, may include a first point spread function PSF and a second point spread function PSF. Interpolating with a weighted average of) to obtain a third blur patch.

An apparatus for extracting depth information from an image according to an exemplary embodiment of the present disclosure may be configured to obtain a first image of an object using a first aperture, and a second image of an object using a second aperture. An image acquisition unit for acquiring an image, a patch setting unit for setting a first patch having a predetermined area on the first image, and setting a second patch corresponding to the first patch on the second image, and a first point spreading function A blur patch acquisition unit obtaining a first blur patch for the first patch based on the first blur patch, and obtaining a second blur patch for the first patch based on the second point spreading function (PSF), respectively. And an interpolation unit that interpolates the second blur patch, the first blur patch, the second blur patch, or the third blur patch, and a depth estimation unit for estimating depth information based on the set second patch, wherein the third blur patch includes: Interpolation And it may be obtained by the blur patch obtaining unit based on the interpolation performed by the.

Each of the first point spread function (PSF) and the second point spread function (PSF) according to an embodiment of the present invention has point spread functions having N different standard deviation values when the depth resolution is N. Any one of which, N may be a natural number.

The interpolator according to an embodiment of the present invention may perform interpolation as a weighted average of the first point spread function PSF and the second point spread function PSF.

Meanwhile, according to an embodiment of the present invention, a computer-readable recording medium having recorded thereon a program for executing the above-described method on a computer can be provided.

Terms used herein will be briefly described and the present invention will be described in detail.

The terms used in the present invention have been selected as widely used general terms as possible in consideration of the functions in the present invention, but this may vary according to the intention or precedent of the person skilled in the art, the emergence of new technologies and the like. In addition, in certain cases, there is also a term arbitrarily selected by the applicant, in which case the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present invention should be defined based on the meanings of the terms and the contents throughout the present invention, rather than the names of the simple terms.

When any part of the specification is to "include" any component, this means that it may further include other components, except to exclude other components unless otherwise stated. In addition, the terms "... unit", "module", etc. described in the specification mean a unit for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software. .

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a single lens dual aperture camera system.

The user may acquire a color channel image and an infrared channel image using the single lens dual aperture camera system 100 as shown in FIG. 1. The barrel 140 may include an aperture 110 for adjusting the amount of visible light to be transmitted, a writing line blocking filter 120 for selectively blocking infrared rays, and at least one lens 130. Light passing through the iris and the lens may be sensed by the CMOS image sensor 150. The CMOS image sensor 150 may be connected to other electronic modules through, for example, a PCB.

The color channel image and the infrared channel image may be acquired according to the component and amount of light detected by the CMOS image sensor. The color channel image may be referred to as the color image for the object, and the infrared channel image may be referred to as the infrared image. An image acquired by the aperture having a small aperture may have a smaller degree of blurring according to the object distance from the focal length than an image obtained by the aperture having a large aperture.

The distance information of the object may be obtained by obtaining a color channel image and an infrared channel image, extracting an edge of the object from the obtained images, and performing depth estimation on the extracted edge. Depth information of the object in the image may be obtained by comparing and analyzing the color channel image and the infrared channel image. In other words, distance information of the object may be obtained based on the edges of the color channel image and the infrared channel image. Although a distance information extraction algorithm for obtaining distance information of an object has existed in the past, there is a problem in that a considerable amount of computation is consumed.

The blurred image may be obtained through calculation with point spread functions having different standard deviations with respect to a predetermined area (eg, a box area) in the infrared image 10. The predetermined area may be referred to as a patch or the like. Such an operation may include a convolution operation. For example, for depth resolution 10, each of the point spreading functions having ten different standard deviation values (e.g., sigma 0 to sigma 9) may be subjected to a convolution operation with a predetermined area in infrared image 10. have. For each of the blurred patches obtained through operation with point spreading functions having ten different standard deviation values (e.g., sigma 0 to sigma 9), the comparison of the images with a predetermined area in the color image 20 Similarity and distance values (eg, PSF index) can be obtained. Distance information of the object may be estimated using the obtained similarity and distance value. As described above, the conventional distance information extraction algorithm increases computational complexity in terms of the need to perform operations on all of the point spreading functions having different standard deviation values, thereby increasing the complexity and thus reducing the efficiency of distance information extraction.

A method for extracting depth information from an image according to an exemplary embodiment of the present invention may include obtaining a first image of an object using a first aperture and using a second aperture to obtain a second image of the object. Acquiring (S100), setting a first patch of a predetermined area on the first image (S200), and a first blur patch for the first patch based on a first point spread function (PSF) And obtaining a second blur patch for the first patch based on the second point spread function (PSF) (S300), respectively, by interpolating the obtained first blur patch and the second blur patch. Acquiring a blur patch (S400), setting a second patch corresponding to the first patch on the second image (S500), and each of the first blur patch, the second blur patch, or the third blur patch and the second patch. Estimating depth information based on S600) may be included. The first patch of the predetermined area set on the first image may be an image area set to a predetermined area to include an edge of an object included in the first image. A method for extracting depth information from an image according to an embodiment of the present invention will be described with reference to FIG. 4 below.

According to an embodiment of the present invention, a first image of an object may be obtained using the first aperture. The first aperture may be an infrared (IR) blocking filter or the like for selectively blocking infrared rays. The first image may be an infrared image, but is not necessarily limited thereto. In addition, a second image of the object may be acquired using the second aperture. The second aperture may be an aperture generally used in a camera system or the like. In addition, for convenience of description, the infrared image is referred to as an example of the first image, and the color channel image is referred to as an example of the second image, but is not necessarily limited thereto. In other words, the method according to an embodiment of the present invention may be generally used in a dual aperture (DA) system capable of capturing an image using different apertures.

A first patch 11 of a predetermined area may be set on the first image 10. The first patch 11 having a predetermined area set on the first image 10 may be an image area set to a predetermined area to include at least one edge in the first image 10.

The first blur patch 1 for the first patch 11 may be obtained based on the first point spread function (PSF). The first blur patch 1 may be obtained through a combination of the first point spread function PSF having the first standard deviation value (eg, σ 0) and the first patch 11. The combination of the first point spread function PSF and the first patch 11 may include a convolution operation C1. In addition, a second blur patch 2 for the first patch 11 may be obtained based on the second point spread function PSF. The second blur patch 2 can be obtained through a combination of the first point 11 and the second point spread function PSF having a second standard deviation value (eg, sigma 2). The combination of the second point spread function PSF and the first patch 11 may include a convolution operation C2. The blur patch obtained through the combination with the point spread function (PSF) may be referred to as a basis.

According to an embodiment of the present invention, the third blur patch 3 may be obtained through interpolation of the obtained first blur patch 1 and the second blur patch 2, respectively. In other words, the third blur patch 3 may be obtained through the interpolation process I1 of the results of the convolutional operations C1 and C2. As mentioned in the foregoing example, compared to the conventional distance estimation method of performing convolution for each of all the point spread functions having ten different standard deviation values (e.g., sigma 0 to sigma 9), according to an embodiment of the present invention The depth information extraction method can greatly reduce the amount of computation in that it performs a convolution operation with the first patch 11 on some of the point spread functions (eg, sigma 0, sigma 2, sigma 5 and sigma 9), and the complexity of the operation Can be reduced. As shown in FIG. 4, for example, each of the first point spread function PSF and the second point spread function PSF has a point spread having N different standard deviation values when the depth resolution is N. It can be any one of the functions. In this case, N may be a natural number.

According to an embodiment of the present invention, the second patch 21 may be set on the second image 20 to correspond to the first patch 11. The location and area in the second image 20 of the second patch 21 may be a patch that is the same as at least one of the location and area of the first patch 11.

Depth information may be estimated based on each of the first blur patch, the second blur patch, or the third blur patch and the second patch. As shown in FIG. 4, depth information may be estimated through similarity analysis between each of the blur patches and the second patch. For example, for each of the blur patches (eg, the first blur patch, the second blur patch, or the third blur patch, etc.), the comparison between the patches through the comparison with the second patch 21 in the second image 20 is possible. Similarity and distance values (eg, PSF index) can be obtained. The acquired similarity and distance value (eg, PSF index) may be included in the depth information.

An interpolation process I1 of the first blur patch and the second blur patch, respectively, according to an embodiment of the present invention may include interpolation as a weighted average of the first point spread function and the second point spread function.

Point spread function (PSF) according to an embodiment of the present invention can be defined by the following equation (1).

Equation 1

x, y represents the on-plane position coordinates of the PSF, and σ k represents the standard deviation. Interpolation may be performed through a weighted average of the first point spread function f1 and the second point spread function f2. In other words, the interpolation function f3 is a weighted average of the first point spread function f1 and the second point spread function f2, and can be defined by the following equation (2).

Equation 2

α, β represent coefficients of the coefficient of weighted average. For example, as in FIG. 5B, the first point spread function f1 has a first standard deviation value (eg, σa), and the second point spread function f2 has a second standard deviation value (eg, σb). ), The interpolation function f3, which can be obtained as the weighted average of the first point spreading function f1 and the second point spreading function f2, is determined by the first standard deviation value? It can be set to approximate a point spread function f4 having a standard deviation value sigma c between two standard deviation values sigma b. In other words, it is possible to obtain a result approximated with a conventional convolution operation result by interpolation according to the adjustment of the weighting coefficient according to an embodiment of the present invention. Therefore, according to one embodiment of the present invention, the amount of computation can be greatly reduced compared to the conventional art.

Obtain an nth blur patch using a point spread function having an n th standard deviation value σ n for the first patch 11 on the infrared image, and a point having an n + 1 th standard deviation value σ n + 1 The spreading function may be used to obtain an n + 1th blur patch. The at least one blur patch Pk between the n th blur patch and the n th +1 th blur patch is a result of the convolution operation of the point spread function having the n th standard deviation value σ n and the first patch 11 and the n th blur patch. It can be obtained as a weighted average of a point spread function having a +1 standard deviation value sigma n + 1 and the result of the convolution operation of the first patch 11. In other words, the blur patch Bk obtainable through operation with the point spread function f4 having the standard deviation value σk between the nth standard deviation value σn and the n + 1th standard deviation value σn + 1 ) May be obtained approximately similar to the above-described blur patch Pk. The above-described interpolation process related to FIG. 6 may be represented by Equations 3 to 6 below.

Equation 3

Equation 4

Equation 5

Equation 6

The apparatus 700 for extracting depth information from an image according to an embodiment of the present invention acquires a first image of an object using a first aperture, and a second image of the object using a second aperture. An image acquisition unit 710 for acquiring a second image, a patch setting unit 720 for setting a first patch having a predetermined area on a first image, and setting a second patch corresponding to the first patch on a second image, and a first point A blur patch acquisition unit that obtains a first blur patch for the first patch set based on the spread function PSF, and obtains a second blur patch for the first patch set based on the second point spread function PSF. 730, depth information based on the obtained interpolation unit 740 for interpolating the first blur patch and the second blur patch, the first blur patch, the second blur patch, or the third blur patch, respectively, and the set second patch; Depth Estimation (750) It can hamhal. The third blur patch may be obtained by the blur patch acquirer 730 based on the interpolation performed by the interpolator 740.

Each of the first point spread function (PSF) and the second point spread function (PSF) according to an embodiment of the present invention has N different standard deviation values when the depth resolution is N (N is a natural number). It may be any one of the point spreading function. In other words, only some of the point spreading functions having N different standard deviation values may be combined with the first patch. Such combination may include convolution operations.

The interpolator 740 may perform interpolation as a weighted average of the first blur patch and the second blur patch.

With regard to the apparatus according to an embodiment of the present invention, the above-described method may be applied. Therefore, with respect to the apparatus, the description of the same contents as those of the above-described method is omitted.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims

In the method for extracting depth information from an image,

Acquiring a first image of the object using the first aperture and acquiring a second image of the object using the second aperture;

Setting a first patch of a predetermined area on the first image;

Obtain a first blur patch for the set first patch based on a first point spread function (PSF), and generate a second blur patch for the set first patch based on a second point spread function (PSF) Obtaining a blur patch;

Obtaining a third blur patch through interpolation of the obtained first blur patch and the second blur patch;

Setting a second patch corresponding to the first patch on the second image; And

Estimating depth information based on each of the first blur patch, the second blur patch, or the third blur patch, and the set second patch.
The method of claim 1,

And a first patch of a predetermined area set on the first image is an image area set to a predetermined area to include an edge of an object included in the first image.
The method of claim 1,

Each of the first point spread function PSF and the second point spread function PSF is any one of N point spread functions having N different standard deviation values when the depth resolution is N, and N Is a natural number.
The method of claim 1,

Acquiring a third blur patch through interpolation of the obtained first blur patch and the second blur patch,

Interpolating as a weighted average of the first point spread function (PSF) and the second point spread function (PSF) to obtain a third blur patch.
An apparatus for extracting depth information from an image, the apparatus comprising:

An image acquisition unit for acquiring a first image of the object using the first aperture and acquiring a second image of the object using the second aperture;

A patch setting unit configured to set a first patch having a predetermined area on the first image and to set a second patch corresponding to the first patch on the second image;

Obtain a first blur patch for the set first patch based on a first point spread function (PSF) and obtain a second blur patch for the set first patch based on a second point spread function (PSF) Blur patch acquisition unit;

An interpolator configured to interpolate the obtained first and second blur patches; And

A depth estimation unit for estimating depth information based on each of the first blur patch, the second blur patch, or the third blur patch, and the set second patch;

The third blur patch is obtained by the blur patch acquisition unit based on the interpolation performed by the interpolation unit.
The method of claim 5, wherein

And a first patch of a predetermined area set on the first image is an image area set to a predetermined area to include an edge of an object included in the first image.
The method of claim 5, wherein

Each of the first point spread function PSF and the second point spread function PSF is any one of N point spread functions having N different standard deviation values when the depth resolution is N, and N Is a device that is a natural number.
The method of claim 5, wherein

And the interpolation unit performs interpolation as a weighted average of the first point spread function (PSF) and the second point spread function (PSF).
A computer-readable recording medium having recorded thereon a program for implementing the method of any one of claims 1 to 4.