WO2016006901A1

WO2016006901A1 - Method and apparatus for extracting depth information from image

Info

Publication number: WO2016006901A1
Application number: PCT/KR2015/006977
Authority: WO
Inventors: 박현상
Original assignee: 재단법인 다차원 스마트 아이티 융합시스템 연구단
Priority date: 2014-07-08
Filing date: 2015-07-07
Publication date: 2016-01-14
Also published as: KR101600183B1; KR20160005984A

Abstract

A method for extracting depth information from an image comprises the steps of: extracting a first patch data including data of each of a plurality of pixels corresponding to a predetermined region on the basis of a certain pixel position of a first image; receiving an input of the first patch data in a raster scan order; processing the first patch data that are input in the raster scan order using one-dimensional horizontal point spread function (PSF) and a plurality of filters, to which the one-dimensional horizontal PSF is applied, in order to generate a plurality of blurred patches; extracting second patch data which corresponds to the first patch data and includes data of each of a plurality of pixels corresponding to the predetermined region on the basis of a certain pixel position of a second image which is distinctive from the first image; comparing similarities between the second patch data and each of the plurality of blurred patches; and determining the depth of the certain pixel position on the basis of the comparison result.

Description

Method and apparatus for extracting depth information from an image

The present invention relates to an apparatus and method for extracting depth information from an image, and more particularly, to a technique for extracting depth information from an image by applying a one-dimensional point spread function (PSF).

Techniques for extracting depth information from an existing image determine the depth at a particular pixel location using a two-dimensional PSF based on the first image and a second image that is distinct from the first image. For example, referring to FIG. 1, which illustrates a method of extracting depth information from an existing image, the apparatus for extracting depth information may correspond to a plurality of preset areas corresponding to a predetermined pixel position of the first image 110. After extracting the first patch data including the data of each of the pixels of the plurality of pixels, by applying a two-dimensional PSF using a plurality of filters 120 to the extracted first patch data, a plurality of blurred patches (blurred patches) Generate 130. Here, the plurality of filters 120 is a filter applying the 2D PSF, and a result value of applying the 2D PSF to the first patch data may mean a brightness value of the pixel.

For example, the apparatus for extracting the existing depth information extracts first patch data corresponding to the MxM region based on a specific pixel position (x, y) of the first image 110 and then extracts the first extracted data. By applying the 2D PSF to the patch data, a plurality of blur patches 130 including MxM pixels may be generated. In this case, the first patch data is data corresponding to each of the plurality of pixels corresponding to the MxM area around the specific pixel position (x, y), and the plurality of pixels are (x + i, y + i),- M / 2 ≦ i, j ≦ M / 2. In addition, since the plurality of blur patches 130 are generated by the N plurality of filters 120 corresponding to the N depths, the plurality of blur patches 130 may be configured of N MxM pixels.

Thereafter, the apparatus for extracting the existing depth information includes a second data including data of each of a plurality of pixels corresponding to a predetermined area centered on a specific pixel position of the second image 140 corresponding to the first patch data. By extracting the patch data and comparing the similarity between the second patch data and each of the plurality of blur patches 130, the depth at a specific pixel position is determined. For example, the apparatus for extracting the existing depth information extracts the second patch data corresponding to the MxM area around the specific pixel position (x, y) of the second image 140 so as to correspond to the first patch data. Thus, by comparing the similarity with the plurality of blur patches 130, the depth at a specific pixel position may be determined.

However, since the conventional technique for extracting depth information applies a two-dimensional PSF to the first patch data, the complexity of the operation is increased, so that the time required for generating the plurality of blur patches 130 is long. have.

Accordingly, in the present specification, by using a one-dimensional PSF, by reducing the complexity of the operation, a technique for reducing the time required for the process of generating a plurality of blur patches 130.

Embodiments of the present invention provide a method, apparatus, and system that reduce complexity of computation and reduce computation time by using a one-dimensional PSF in the process of generating a plurality of blur patches.

At this time, embodiments of the present invention by applying the one-dimensional PSF in any one of the order of the one-dimensional horizontal PSF from the one-dimensional vertical PSF or the order of the one-dimensional vertical PSF from the one-dimensional horizontal PSF, Provided are a method, an apparatus, and a system for substituting data of a plurality of pixels corresponding to a predetermined area around a specific pixel position of a single image with brightness values.

The method of extracting depth information from an image according to an embodiment of the present invention extracts first patch data including data of each of a plurality of pixels corresponding to a predetermined area around a specific pixel position of a first image. step; Receiving the first patch data in a raster scan order; The first patch input to the raster scan order using a plurality of filters applying a one-dimensional horizontal point spread function (PSF) and a one-dimensional vertical PSF to generate a plurality of blurred patches Processing the data; Extracting second patch data including data of each of a plurality of pixels corresponding to the preset area around the specific pixel position of the second image, which is different from the first image, corresponding to the first patch data Doing; Comparing the similarity between each of the second patch data and the plurality of blur patches; And determining a depth at the specific pixel position based on the comparison result.

The receiving of the first patch data in the raster scan order may include receiving data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction.

The processing of the first patch data may include applying the one-dimensional horizontal PSF to data of each of a plurality of pixels included in the first patch data input in the horizontal direction, for each of the plurality of filters. step; Storing data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF is applied to a line memory included in each of the plurality of filters; And for each of the plurality of filters, applying the one-dimensional vertical PSF to data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF applied to the line memory is applied. It may include.

The receiving of the first patch data in the raster scan order may include receiving data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction.

The processing of the first patch data may include applying the one-dimensional vertical PSF to data of each of a plurality of pixels included in the first patch data input in the vertical direction, for each of the plurality of filters. step; Storing data of each of a plurality of pixels included in the first patch data to which the one-dimensional vertical PSF is applied to a line memory included in each of the plurality of filters; And for each of the plurality of filters, applying the one-dimensional horizontal PSF to data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical PSF applied to the line memory is applied. It may include.

Comparing the similarity between the second patch data and each of the plurality of blur patches may include a Sum of Absolute Difference (SAD) technique, a Sum of Squared Difference (SSD) technique, a Mean of Absolute Difference (MAD) technique, or a SASE (SASE) technique. And comparing the similarity between each of the second patch data and each of the plurality of blur patches using at least one of a Sum of Squared Error technique.

Comparing the similarity between each of the second patch data and the plurality of blur patches includes obtaining a difference value between each of the second patch data and each of the plurality of blur patches. Based on the above method, determining a depth at the specific pixel position may include selecting a smallest difference value among difference values between the second patch data and each of the plurality of blur patches; And determining the smallest difference value as a depth at the specific pixel position.

An apparatus for extracting depth information from an image according to an embodiment of the present invention extracts first patch data including data of each of a plurality of pixels corresponding to a predetermined area around a specific pixel position of a first image. A first patch data extracting unit; An input unit configured to receive the first patch data in a raster scan order; The first patch input to the raster scan order using a plurality of filters applying a one-dimensional horizontal point spread function (PSF) and a one-dimensional vertical PSF to generate a plurality of blurred patches A processing unit for processing data; Extracting second patch data including data of each of a plurality of pixels corresponding to the preset area around the specific pixel position of the second image, which is different from the first image, corresponding to the first patch data A second patch data extracting unit; A comparison unit comparing the similarity between each of the second patch data and the plurality of blur patches; And a determination unit that determines a depth at the specific pixel position based on the comparison result.

The input unit may receive data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction.

The processing unit applies the one-dimensional horizontal PSF to each of the plurality of pixels included in the first patch data input in the horizontal direction, for each of the plurality of filters, and the one-dimensional horizontal PSF. Is stored in a line memory included in each of the plurality of filters, and the data of each of the plurality of pixels included in the first patch data to which the first patch data is applied is stored in each of the plurality of filters. The one-dimensional vertical PSF may be applied to data of a plurality of pixels included in the first patch data to which the stored one-dimensional horizontal PSF is applied.

The input unit may receive data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction.

The processing unit applies the one-dimensional vertical direction PSF to each of the plurality of pixels included in the first patch data input in the vertical direction, for each of the plurality of filters, and applies the one-dimensional vertical direction to the data. The data stored in each of the plurality of pixels included in the first patch data to which the PSF is applied is stored in a line memory included in each of the plurality of filters, and for each of the plurality of filters, the 1 stored in the line memory. The one-dimensional horizontal PSF may be applied to data of a plurality of pixels included in the first patch data to which the dimensional vertical PSF is applied.

The comparison unit uses the second patch data using at least one of a sum of absolute difference (SAD) technique, a sum of squared difference (SSD) technique, a mean of absolute difference (MAD) technique, or a sum of squared error (SASE) technique. And similarity between each of the plurality of blur patches.

Embodiments of the present invention can provide a method, apparatus and system for reducing the complexity of computation and reducing the computation time by using a one-dimensional PSF in the process of generating a plurality of blur patches.

At this time, embodiments of the present invention by applying the one-dimensional PSF in any one of the order of the one-dimensional horizontal PSF from the one-dimensional vertical PSF or the order of the one-dimensional vertical PSF from the one-dimensional horizontal PSF, It is possible to provide a method, an apparatus, and a system for replacing data of a plurality of pixels corresponding to a predetermined area centering on a specific pixel position of a single image with brightness values.

1 is a conceptual diagram illustrating a method of extracting depth information from an existing image.

2 is a conceptual diagram illustrating a method of extracting depth information from an image according to an embodiment of the present invention.

3 is a diagram illustrating a first patch data input process and a one-dimensional PSF application process according to an embodiment of the present invention.

4 is a diagram illustrating a first patch data input process and a 1D PSF application process according to another embodiment of the present invention.

5 is a diagram illustrating a one-dimensional horizontal PSF and a one-dimensional vertical PSF according to an embodiment of the present invention.

6 is a flowchart illustrating a method of extracting depth information from an image according to an embodiment of the present invention.

7 is a block diagram illustrating an apparatus for extracting depth information from an image according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

Referring to FIG. 2, an apparatus for extracting depth information from an image according to an embodiment of the present invention uses a one-dimensional PSF based on a first image 210 and a second image 220 distinguished from the first image. To determine the depth at a particular pixel location. In this case, the first image 210 may be an image clearly captured at all depths using a small aperture, and the second image 220 may be a small aperture used in the process of acquiring the first image 210. Compared to a depth at which the focus is achieved by using a relatively large aperture, the image may be sharper as the distance from the depth is in focus. In addition, the one-dimensional PSF is a one-dimensional function indicating how the data of the pixel, which can be represented by the brightness value, has a spatial distribution, and may mean a one-dimensional vertical PSF and a one-dimensional horizontal PSF.

In detail, the apparatus for extracting depth information from an image extracts first patch data including data of each of a plurality of pixels corresponding to a predetermined area around a specific pixel position of the first image 210 and then extracts the first patch data. Received first patch data in a raster scan order. In this case, the raster scan order refers to a method in which data of each of the plurality of pixels included in the first patch data is sequentially input from left to right and from top to bottom.

The apparatus for extracting depth information from the image then uses a plurality of filters 240 that apply a one-dimensional horizontal PSF and a one-dimensional vertical PSF to generate a plurality of blur patches 230. The first patch data input to the master scan order is processed.

In this case, since the apparatus for extracting depth information from the image receives data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction, for each of the plurality of filters 240, One-dimensional horizontal PSF may be applied to data of each of the plurality of pixels included in the first patch data input in the horizontal direction, and may be stored in the line memory 250 included in each of the plurality of filters 240. Next, the apparatus for extracting depth information from the image reads data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF applied to the one-dimensional horizontal PSF stored in the line memory 250 is applied in the vertical direction, thereby the one-dimensional vertical PSF. Can be applied.

For example, an apparatus for extracting depth information from an image may include data of each of a plurality of pixels included in first patch data corresponding to an MxM region, based on a specific pixel position (x, y) of the first image 210. In the case of applying a one-dimensional PSF having k tabs, the line is applied by applying a one-dimensional horizontal PSF to each of k-1 rows of data of each of the plurality of pixels included in the first patch data input in the horizontal direction. May be stored in the memory 250. Next, the apparatus for extracting the depth information from the image applies the one-dimensional horizontal PSF to the last row of the data of each of the plurality of pixels included in the first patch data currently input in the horizontal direction, and the line memory 250 The one-dimensional vertical PSF may be applied by combining with k-1 rows of data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF to be stored is applied.

On the other hand, since the apparatus for extracting depth information from the image may receive data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction, For example, one-dimensional vertical PSF may be applied to data of each of the plurality of pixels included in the first patch data input in the vertical direction, and stored in the line memory 250 included in each of the plurality of filters 240. . Next, the apparatus for extracting depth information from the image reads horizontally the data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical PSF to which the one-dimensional vertical PSF is stored in the line memory 250 is applied. You can also apply

For example, an apparatus for extracting depth information from an image may include data of each of a plurality of pixels included in first patch data corresponding to an MxM region, based on a specific pixel position (x, y) of the first image 210. In the case of applying a one-dimensional PSF having k tabs, the line is applied by applying a one-dimensional vertical PSF to each of the k-1 columns of data of each of the plurality of pixels included in the first patch data input in the vertical direction. May be stored in the memory 250. Next, the apparatus for extracting depth information from the image applies the one-dimensional vertical PSF to the last column of each of the plurality of pixels included in the first patch data currently input in the vertical direction, and applies the line memory 250 to the line memory 250. The one-dimensional horizontal PSF may be applied by combining the data of k-1 columns of the plurality of pixels included in the first patch data to which the stored one-dimensional vertical PSF is applied.

In this case, since the one-dimensional PSF application process is performed in each of the plurality of filters 240, a plurality of blur patches 230 corresponding to the number of the plurality of filters 240 may be generated. Here, the plurality of filters 240 are filters applying one-dimensional vertical PSF and one-dimensional horizontal PSF, and a result value of applying the one-dimensional PSF to the first patch data may mean a brightness value of the pixel. Detailed description thereof will be described with reference to FIGS. 3 and 4.

Thereafter, the apparatus for extracting depth information from the image includes a plurality of regions corresponding to a predetermined area of a specific pixel position of the second image 220, which is distinguished from the first image 210, corresponding to the first patch data. By extracting the second patch data including the data of each of the pixels, and comparing the similarity between the second patch data and each of the plurality of blur patches 230, the depth at a particular pixel position is determined. In this case, the apparatus for extracting the depth information from the image may use various techniques in the process of comparing the similarity between the second patch data and each of the plurality of blur patches 230. For example, an apparatus for extracting depth information from an image may include at least one of a Sum of Absolute Difference (SAD) technique, a Sum of Squared Difference (SSD) technique, a Mean of Absolute Difference (MAD) technique, or a Sum of Squared Error (SASE) technique. By comparing the similarity between each of the second patch data and each of the plurality of blur patches 230 by using any one, based on the comparison result, at a specific pixel position of the first image 210 or the second image 220. Depth can be determined. As a more specific example, the apparatus for extracting depth information from an image obtains a difference value between each of the second patch data and the plurality of blur patches 230, and the second patch data and the plurality of blur patches 230. By selecting the smallest difference value among the difference values between each, the smallest difference value can be determined as the depth at a specific pixel position.

As such, the apparatus for extracting depth information from an image according to an embodiment of the present invention may generate one-dimensional PSFs (one-dimensional horizontal PSF and one-dimensional vertical PSF) in the process of generating a plurality of blur patches 230. By using, it is possible to reduce the complexity of the calculation and to shorten the computation time.

3 is a diagram illustrating a first patch data input process and a one-dimensional PSF application process according to an embodiment of the present invention. Hereinafter, the first patch data input process and the 1D PSF application process according to an embodiment of the present invention receive data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction, 1 After applying the dimensional horizontal PSF, a description will be given of the case of applying the one-dimensional vertical PSF.

Referring to FIG. 3, an apparatus for extracting depth information from an image according to an embodiment of the present invention inputs data of each of a plurality of pixels included in the first patch data 310 in a horizontal direction to a vertical direction. I can receive it. For example, an apparatus for extracting depth information from an image may configure a plurality of rows using data of each of a plurality of pixels included in the first patch data 310, and when input of data constituting one row is completed, You can receive input from the data that makes up the next row. For example, the apparatus for extracting depth information from an image receives data included in the first row 320 in the order of left to right, and when input of data included in the first row 320 is completed, The data included in the second row 330 may be input in the order of left to right.

The data of each of the pixels included in the first patch data 310 input as described above may be stored in the line memory included in each of the plurality of filters by applying a one-dimensional horizontal PSF to each of the plurality of filters. Can be. For example, the data of each of the pixels included in the first patch data 310 configured by configuring the plurality of rows may be stored in the line memory by applying one-dimensional horizontal PSF to each row. More specifically, an apparatus for extracting depth information from an image may be applied to data included in the first row 320 and stored in the line memory by applying the one-dimensional horizontal PSF to the second row 330. One-dimensional horizontal PSF is applied to the data to be stored in the line memory.

Thereafter, the apparatus for extracting depth information from the image may apply the one-dimensional vertical PSF to the data of each of the plurality of pixels included in the first patch data 310 to which the one-dimensional horizontal PSF to which the one-dimensional horizontal PSF is stored is applied. have. For example, the one-dimensional horizontal PSF may be applied to each of the plurality of rows, and thus, the one-dimensional vertical PSF may be applied to the data of each of the pixels included in the first patch data 310 stored in the line memory. More specifically, the apparatus for extracting depth information from an image may include data included in the first row 320 to which the one-dimensional horizontal PSF is applied and second row 330 to which the one-dimensional horizontal PSF is applied. One-dimensional vertical PSF may be applied to the data to generate a blur patch including brightness values corresponding to data of each of the pixels included in the first patch data 310.

As described above, the apparatus for extracting depth information from an image according to an embodiment of the present invention applies the one-dimensional PSF in the order of the one-dimensional vertical PSF to the one-dimensional vertical PSF, thereby focusing on the specific pixel position of the first image. Data of each of the plurality of pixels corresponding to the preset area may be replaced with a brightness value without omission. In this case, since the one-dimensional PSF application process is performed in each of the plurality of filters, a plurality of blur patches corresponding to the number of the plurality of filters may be generated.

4 is a diagram illustrating a first patch data input process and a 1D PSF application process according to another embodiment of the present invention. Hereinafter, the first patch data input process and the 1D PSF application process according to another embodiment of the present invention receive data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction, A case in which one-dimensional vertical PSF is applied and then one-dimensional horizontal PSF is applied will be described.

Referring to FIG. 4, an apparatus for extracting depth information from an image according to another exemplary embodiment of the present invention may include data of each of a plurality of pixels included in the first patch data 410 in a vertical direction from a horizontal direction. You can receive input. For example, the apparatus for extracting depth information from an image may configure a plurality of columns of data of each of the plurality of pixels included in the first patch data 410, and when input of data constituting one column is completed, In this case, you can receive input of the data that makes up the next column. For example, the apparatus for extracting depth information from an image receives data included in the first column 420 in the order of the top to the bottom, and receives data included in the second column 430 from the top to the bottom. In this order, the data included in the third column 440 may be input in the order of the top to the bottom.

Data of each of the pixels included in the first patch data 410 input as described above may be stored in the line memory included in each of the plurality of filters by applying a one-dimensional vertical PSF to each of the plurality of filters. Can be. For example, data of each of the plurality of pixels included in the first patch data 410 input by configuring a plurality of columns may be stored in the line memory by applying one-dimensional vertical PSF to each column. For example, the apparatus for extracting depth information from an image may be applied to the data included in the first column 420 and stored in the line memory by applying a one-dimensional vertical PSF to the data included in the second column 430. After applying the one-dimensional vertical PSF to the data and storing it in the line memory, the one-dimensional vertical PSF may be applied to the data included in the third column 440 and stored in the line memory.

Thereafter, the apparatus for extracting the depth information from the image may apply the one-dimensional horizontal PSF to the data of each of the plurality of pixels included in the first patch data 410 to which the one-dimensional vertical PSF applied in the line memory is applied. have. For example, the one-dimensional vertical PSF may be applied to each of the plurality of columns so that the one-dimensional horizontal PSF may be applied to the data of each of the pixels included in the first patch data 410 stored in the line memory. For example, the apparatus for extracting depth information from an image includes data included in the first column 420 to which the one-dimensional vertical PSF is applied and data included in the second column 430 to which the one-dimensional vertical PSF is applied. Data corresponding to the data of each of the plurality of pixels included in the first patch data 410 by applying the one-dimensional horizontal PSF to the data and the data included in the third column 440 to which the one-dimensional vertical PSF is applied. A blur patch consisting of brightness values can be generated.

As described above, the apparatus for extracting depth information from an image according to another embodiment of the present invention applies the one-dimensional PSF in the order of the one-dimensional vertical PSF to the one-dimensional horizontal PSF, thereby centering the specific pixel position of the first image. As a result, data of each of the pixels corresponding to the preset area may be replaced with a brightness value without omission. In this case, since the one-dimensional PSF application process is performed in each of the plurality of filters, a plurality of blur patches corresponding to the number of the plurality of filters may be generated.

Referring to FIG. 5, the one-dimensional horizontal PSF 510 and the one-dimensional vertical PSF 520 according to an embodiment of the present invention may determine the brightness weight of the horizontal component and the brightness weight of the vertical component from the separable two-dimensional PSF. By separating each, it can be obtained.

For example, the one-dimensional horizontal PSF 510 and the one-dimensional vertical PSF 520 separate the brightness weights of the horizontal component and the brightness weights of the vertical component from the Gaussian filter 530 which is the two-dimensional PSF, respectively. Can be obtained. However, the one-dimensional horizontal PSF 510 and the one-dimensional vertical PSF 520 according to an embodiment of the present invention are not limited thereto, but are not limited thereto. It can be obtained by separating the brightness weights and the brightness weights of the vertical components, respectively.

Referring to FIG. 6, an apparatus for extracting depth information from an image according to an embodiment of the present invention may include a first data including data of each of a plurality of pixels corresponding to a predetermined area around a specific pixel position of a first image. One patch data is extracted (610).

In operation 620, the apparatus for extracting depth information from the image receives the first patch data in a raster scan order. At this time, the apparatus for extracting the depth information from the image may receive the data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction. In addition, the apparatus for extracting depth information from the image may receive data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction.

The apparatus for extracting depth information from an image then uses a plurality of filters to apply a one-dimensional horizontal point spread function (PSF) and a one-dimensional vertical PSF to generate a plurality of blurred patches. The first patch data input to the raster scan order is processed (630).

In this case, when data of each of the plurality of pixels included in the first patch data is input in the order from the horizontal direction to the vertical direction, the apparatus for extracting depth information from the image may be horizontally applied to each of the plurality of filters. One-dimensional horizontal PSF is applied to data of each of the plurality of pixels included in the first patch data input, and data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF is applied is plural. Stored in a line memory included in each of the filters, and for each of the plurality of filters, in each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF applied in the line memory is applied. By applying the one-dimensional vertical PSF, the first patch data can be processed.

In addition, when data of each of the plurality of pixels included in the first patch data is input in the order from the vertical direction to the horizontal direction, the apparatus for extracting depth information from the image is input in the vertical direction with respect to each of the plurality of filters. One-dimensional vertical PSF is applied to data of each of the plurality of pixels included in the first patch data, and data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical PSF is applied is a plurality of filters. Stored in the line memory included in each of the plurality of filters, and for each of the plurality of filters, the one-dimensional horizontal PSF in the data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical PSF stored in the line memory is applied. By applying, the first patch data can be processed.

Next, the apparatus for extracting depth information from the image may extract data of each of a plurality of pixels corresponding to a predetermined area around a specific pixel position of a second image, which is different from the first image, corresponding to the first patch data. In operation 640, the second patch data including the extracted second patch data is included.

The apparatus for extracting depth information from the image then compares the similarity between each of the second patch data and the plurality of blur patches (650). In this case, the apparatus for extracting the depth information from the image is at least any one of a Sum of Absolute Difference (SAD) technique, a Sum of Squared Difference (SSD) technique, a Mean of Absolute Difference (MAD) technique, or a Sum of Squared Error (SASE) technique. One may be used to compare the similarity between each of the second patch data and the plurality of blur patches.

The apparatus for extracting depth information from the image then determines 660 the depth at a particular pixel location based on the comparison result.

For example, an apparatus for extracting depth information from an image obtains a difference value between each of the second patch data and the plurality of blur patches, and obtains a smallest value of the difference value between each of the second patch data and the plurality of blur patches. By selecting the difference value, the smallest difference value selected can be determined as the depth at a specific pixel position.

Referring to FIG. 7, an apparatus for extracting depth information from an image according to an exemplary embodiment may include a first patch data extractor 710, an input unit 720, a processor 730, and a second patch data extractor ( 740, a comparator 750, and a determiner 760.

The first patch data extractor 710 extracts first patch data including data of each of a plurality of pixels corresponding to a predetermined area centering on a specific pixel position of the first image.

The input unit 720 receives the first patch data in a raster scan order.

In this case, the input unit 720 receives data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction. In addition, the input unit 720 may receive data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction.

The processor 730 is input to the raster scan order using a plurality of filters applying one-dimensional horizontal point spread function (PSF) and one-dimensional vertical PSF to generate a plurality of blurred patches. Process the first patch data.

In this case, when the input unit 720 receives data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction, the processor 730 may receive the horizontal direction with respect to each of the plurality of filters. One-dimensional horizontal PSF is applied to data of each of the plurality of pixels included in the first patch data input to the first patch data, and data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF is applied is plural. Data of each of the plurality of pixels stored in a line memory included in each of the filters of the plurality of filters and included in the first patch data to which the one-dimensional horizontal PSF stored in the line memory is applied to each of the plurality of filters. By applying the one-dimensional vertical PSF to the first patch data, the first patch data can be processed.

In addition, when the processor 730 receives data of each of the plurality of pixels included in the first patch data from the input unit 720 in the order from the vertical direction to the horizontal direction, the processing unit 730 may be configured in the vertical direction with respect to each of the plurality of filters. A one-dimensional vertical PSF is applied to data of each of the plurality of pixels included in the input first patch data, and the data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical PSF is applied is plural. One-dimensional horizontal direction is stored in the line memory included in each of the filters, and for each of the plurality of filters, data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical direction PSF stored in the line memory is applied. By applying the PSF, the first patch data can be processed.

The second patch data extractor 740 includes data of each of a plurality of pixels corresponding to a predetermined area centered on a specific pixel position of a second image, which is different from the first image, corresponding to the first patch data. Extract the second patch data.

The comparison unit 750 compares the similarity between each of the second patch data and the plurality of blur patches.

In this case, the comparator 750 uses at least one of a Sum of Absolute Difference (SAD) technique, a Sum of Squared Difference (SSD) technique, a Mean of Absolute Difference (MAD) technique, or a Sum of Squared Error (SASE) technique. The similarity between each of the second patch data and the plurality of blur patches may be compared.

The determination unit 760 determines the depth at the specific pixel position based on the result.

For example, the comparison unit 750 obtains a difference value between each of the second patch data and the plurality of blur patches, and the determination unit 760 determines a difference value between each of the second patch data and the plurality of blur patches. By selecting the smallest difference value among these, the smallest difference value selected can be determined as the depth at a specific pixel position.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims

Extracting first patch data including data of each of a plurality of pixels corresponding to a predetermined area centering on a specific pixel position of the first image;

Receiving the first patch data in a raster scan order;

The first patch input to the raster scan order using a plurality of filters applying a one-dimensional horizontal point spread function (PSF) and a one-dimensional vertical PSF to generate a plurality of blurred patches Processing the data;

Extracting second patch data including data of each of a plurality of pixels corresponding to the preset area around the specific pixel position of the second image, which is different from the first image, corresponding to the first patch data Doing;

Comparing the similarity between each of the second patch data and the plurality of blur patches; And

Determining a depth at the specific pixel position based on the comparison result

Extracting depth information from an image comprising a.
The method of claim 1,

Receiving the first patch data in the raster scan order

Receiving data of each of the plurality of pixels included in the first patch data in an order from a horizontal direction to a vertical direction

Extracting depth information from an image comprising a.
The method of claim 2,

Processing the first patch data

For each of the plurality of filters, applying the one-dimensional horizontal PSF to data of each of a plurality of pixels included in the first patch data input in the horizontal direction;

Storing data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF is applied to a line memory included in each of the plurality of filters; And

For each of the plurality of filters, applying the one-dimensional vertical PSF to data of each of the plurality of pixels included in the first patch data to which the one-dimensional horizontal PSF applied to the line memory is applied.

Extracting depth information from an image comprising a.
The method of claim 1,

Receiving the first patch data in the raster scan order

Receiving data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction

Extracting depth information from an image comprising a.
The method of claim 4, wherein

Processing the first patch data

For each of the plurality of filters, applying the one-dimensional vertical PSF to data of each of the plurality of pixels included in the first patch data input in the vertical direction;

Storing data of each of a plurality of pixels included in the first patch data to which the one-dimensional vertical PSF is applied to a line memory included in each of the plurality of filters; And

For each of the plurality of filters, applying the one-dimensional horizontal PSF to data of each of the plurality of pixels included in the first patch data to which the one-dimensional vertical PSF applied to the line memory is applied.

Extracting depth information from an image comprising a.
The method of claim 1,

Comparing the similarity between the second patch data and each of the plurality of blur patches

The second patch data and the plurality of patches using at least one of a Sum of Absolute Difference (SAD) technique, a Sum of Squared Difference (SSD) technique, a Mean of Absolute Difference (MAD) technique, or a Sum of Squared Error (SASE) technique. Comparing depth of similarity between each of the blur patches.
The method of claim 6,

Comparing the similarity between the second patch data and each of the plurality of blur patches

Obtaining a difference value between each of the second patch data and the plurality of blur patches

Including,

Based on the comparison result, determining the depth at the specific pixel position

Selecting the smallest difference value among the difference values between each of the second patch data and the plurality of blur patches; And

Determining the smallest difference as the depth at the particular pixel location

Extracting depth information from an image comprising a.
A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 1 to 7.
A first patch data extracting unit configured to extract first patch data including data of each of a plurality of pixels corresponding to a predetermined area centering on a specific pixel position of the first image;

An input unit configured to receive the first patch data in a raster scan order;

The first patch input to the raster scan order using a plurality of filters applying a one-dimensional horizontal point spread function (PSF) and a one-dimensional vertical PSF to generate a plurality of blurred patches A processing unit for processing data;

Extracting second patch data including data of each of a plurality of pixels corresponding to the preset area around the specific pixel position of the second image, which is different from the first image, corresponding to the first patch data A second patch data extracting unit;

A comparison unit comparing the similarity between each of the second patch data and the plurality of blur patches; And

A determination unit that determines a depth at the specific pixel position based on the comparison result

Apparatus for extracting depth information from an image comprising a.
The method of claim 9,

The input unit

An apparatus for extracting depth information from an image that receives data of each of the plurality of pixels included in the first patch data in the order from the horizontal direction to the vertical direction.
The method of claim 10,

The processing unit

For each of the plurality of filters, the one-dimensional horizontal PSF is applied to data of each of the plurality of pixels included in the first patch data input in the horizontal direction, and the one-dimensional horizontal PSF is applied. Storing data of each of the plurality of pixels included in the first patch data in a line memory included in each of the plurality of filters, and for each of the plurality of filters, the 1 stored in the line memory And extracting depth information from an image of applying the one-dimensional vertical PSF to data of each of the plurality of pixels included in the first patch data to which the dimensional horizontal PSF is applied.
The method of claim 9,

The input unit

And extracting depth information from an image that receives data of each of the plurality of pixels included in the first patch data in the order from the vertical direction to the horizontal direction.
The method of claim 12,

The processing unit

For each of the plurality of filters, the one-dimensional vertical PSF is applied to data of each of the plurality of pixels included in the first patch data input in the vertical direction, and the one-dimensional vertical PSF is applied. Storing the data of each of the plurality of pixels included in the first patch data in a line memory included in each of the plurality of filters, and for each of the plurality of filters, the one-dimensional vertical PSF stored in the line memory And extracting depth information from an image of applying the one-dimensional horizontal PSF to data of each of the plurality of pixels included in the first patch data.
The method of claim 9,

The comparison unit

The second patch data and the plurality of patches using at least one of a Sum of Absolute Difference (SAD) technique, a Sum of Squared Difference (SSD) technique, a Mean of Absolute Difference (MAD) technique, or a Sum of Squared Error (SASE) technique. Apparatus for extracting depth information from an image comparing the similarity between each of the blur patches of.