KR101764504B1 - Multi aperture camera for adaptively selecting blur channel and opertation method thereof - Google Patents

Multi aperture camera for adaptively selecting blur channel and opertation method thereof Download PDF

Info

Publication number
KR101764504B1
KR101764504B1 KR1020150179279A KR20150179279A KR101764504B1 KR 101764504 B1 KR101764504 B1 KR 101764504B1 KR 1020150179279 A KR1020150179279 A KR 1020150179279A KR 20150179279 A KR20150179279 A KR 20150179279A KR 101764504 B1 KR101764504 B1 KR 101764504B1
Authority
KR
South Korea
Prior art keywords
blur
image
channel
sharp
edge
Prior art date
Application number
KR1020150179279A
Other languages
Korean (ko)
Other versions
KR20170071216A (en
Inventor
김경호
이영민
박현상
경종민
Original Assignee
재단법인 다차원 스마트 아이티 융합시스템 연구단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 재단법인 다차원 스마트 아이티 융합시스템 연구단 filed Critical 재단법인 다차원 스마트 아이티 융합시스템 연구단
Priority to KR1020150179279A priority Critical patent/KR101764504B1/en
Publication of KR20170071216A publication Critical patent/KR20170071216A/en
Application granted granted Critical
Publication of KR101764504B1 publication Critical patent/KR101764504B1/en

Links

Images

Classifications

    • H04N13/02—
    • H04N13/0003—
    • H04N13/0022—
    • H04N13/0203—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/30—Transforming light or analogous information into electric information

Abstract

A multi-aperture camera for adaptively selecting a blur channel includes a sharp aperture for introducing an optical signal of a sharp channel; A blur aperture for introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and for introducing an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel; An image sensor for processing the optical signal of the sharp channel to generate a sharp image and processing the optical signal of the first blur channel and the optical signal of the second blur channel to generate a first blur image and a second blur image, an image sensor; A blur channel selection unit for adaptively selecting one of edge areas of the first blur image corresponding to edge areas of the sharp image or edge areas of the second blur image corresponding to edge areas of the sharp image part; And a depth determination unit for determining a depth of the edge area based on a change in blur size between an edge area of the sharp image and the selected edge area.

Description

TECHNICAL FIELD [0001] The present invention relates to a multi-aperture camera for adaptively selecting a blur channel and a method for operating the same. [0002] MULTI APERTURE CAMERA FOR ADAPTIVELY SELECTING BLUR CHANNEL AND OPERATING METHOD THEREOF [

The following embodiments relate to a multi-aperture camera and an operation method thereof. More specifically, in order to determine a depth in an image, a blur channel corresponding to a blur image, in which a sharp image and a blur size are compared, As shown in FIG.

Conventional depth determining techniques determine depth in an image based on a blur size change between a sharp image generated from an optical signal of a sharp channel and a blur image generated from an optical signal of a blur channel.

Here, the conventional technique for determining the depth is to process only one optical signal of a blur channel in the process of generating one blur image for comparison with a sharp image. Therefore, under the color of one blur channel, When the contrast between regions is poor, there is a problem that the wrong depth is determined.

Therefore, the following embodiments are directed to a technique for solving the problem that a wrong depth is determined by adaptively selecting and using a plurality of blur images generated from optical signals of a plurality of blur channels with an image compared with a sharp image I suggest.

One embodiment provides a multi-aperture camera and method of operation that determine an accurate depth by adaptively selecting and using a plurality of blur images generated from an optical signal of each of a plurality of blur channels with an image to be compared with a sharp image to provide.

In particular, one embodiment may include calculating a correlation for each sharp image of each of the plurality of blur images, and calculating a correlation for each of the blur images to be compared with the sharp image of the plurality of blur images based on the calculated correlation And a method of operating the multi-aperture camera.

Another embodiment is a multi-aperture image processing method that selects a blur image to be compared with a sharp image of a plurality of blur images based on a calculated contrast value, by calculating a contrast value for a subject of each of the plurality of blur images Camera and method of operation thereof.

In particular, one embodiment provides a multi-aperture camera and method of operating the same that adaptively selects any one of a plurality of blurred images for each edge region based on an edge region searched in a sharp image.

According to one embodiment, a multi-aperture camera that adaptively selects a blur channel includes a sharp aperture for introducing a sharp channel optical signal; A blur aperture for introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and for introducing an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel; An image sensor for processing the optical signal of the sharp channel to generate a sharp image and processing the optical signal of the first blur channel and the optical signal of the second blur channel to generate a first blur image and a second blur image, an image sensor; A blur channel selection unit for adaptively selecting one of edge areas of the first blur image corresponding to edge areas of the sharp image or edge areas of the second blur image corresponding to edge areas of the sharp image part; And a depth determination unit for determining a depth of the edge area based on a change in blur size between an edge area of the sharp image and the selected edge area.

Wherein the blur channel selection unit calculates a correlation between an edge area of the first blurred image and an edge area of the sharp image of each of the edge areas of the second blurred image, An edge region of the first blur image or an edge region of the second blur image can be selected.

The blur channel selector may select any edge region having a higher correlation with respect to the edge region of the sharp image among the edge region of the first blur image or the edge region of the second blur image.

Wherein the blur channel selection unit calculates a contrast value for an object of each of an edge region of the first blur image and an edge region of the second blur image and calculates a contrast value of the edge region of the first blur image based on the calculated contrast value Edge region of the second blurred image or an edge region of the second blurred image.

The blur channel selector may select any edge region having a higher contrast value for the subject among the edge region of the first blur image or the edge region of the second blur image.

Wherein the blur channel selector searches for an edge area of the sharp image and acquires an edge area of the first blur image corresponding to an edge area of the sharp image of the first blur image, And obtain an edge area of the second blur image corresponding to an edge area of the image.

According to one embodiment, a method of operating a multi-aperture camera that adaptively selects a blur channel includes the steps of: introducing an optical signal of a sharp channel through a sharp aperture; Introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel respectively through a blur aperture; ; A method of processing an optical signal of the sharp channel, an optical signal of the first blur channel, and an optical signal of the second blur channel, respectively, using an image sensor to generate a sharp image, a first blur image, ≪ / RTI > A blur channel selection unit is used to adapt any one of the edge areas of the first blurred image corresponding to the edge area of the sharp image or the edge area of the second blurred image corresponding to the edge area of the sharp image Selecting as a target; And determining a depth for the edge region based on a change in a blur size between an edge region of the sharp image and the selected edge region using a depth determining unit.

Wherein the step of adaptively selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image comprises the step of selecting an edge region of the first blur image and an edge region of the edge region of the second blur image, Calculating a correlation for an edge region of the sharp image; And selecting an edge region of either the edge region of the first blurred image or the edge region of the second blurred image based on the calculated correlation.

Wherein the step of selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image comprises the step of selecting the edge region of the first blur image or the edge region of the sharp image It may be a step of selecting any one edge region having a higher correlation with respect to the edge region.

Wherein the step of adaptively selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image comprises the step of selecting an edge region of the first blur image and an edge region of the edge region of the second blur image, Calculating a contrast value for the subject; And selecting an edge region of either the edge region of the first blurred image or the edge region of the second blurred image based on the calculated contrast value.

Wherein the step of selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image comprises selecting an edge region of the first blur image or an edge region of the second blur image And selecting one of the edge regions having a higher contrast value.

Wherein adaptively selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image comprises: searching an edge region of the sharp image; Obtaining an edge region of the first blur image corresponding to an edge region of the sharp image of the first blur image; And obtaining an edge area of the second blur image corresponding to an edge area of the sharp image of the second blur image.

According to one embodiment, a multi-aperture camera that adaptively selects a blur channel includes a sharp aperture for introducing a sharp channel optical signal; A blur aperture for introducing optical signals of each of three or more blur channels of a different wavelength band than the sharp channel; An image sensor processing the optical signal of the sharp channel to produce a sharp image, and processing an optical signal of each of the three or more blur channels to produce three or more blur images; A blur channel selection unit adapted to adaptively select any one edge region of each of the three or more blur images corresponding to edge regions of the sharp image; And a depth determination unit for determining a depth of the edge area based on a change in a blur size between an edge area of the sharp image and the selected edge area,

Wherein the blur channel selector calculates a correlation of edge areas of each of the three or more blurred images with respect to an edge area of the sharp image and calculates a correlation of edge areas of each of the three or more blurred images based on the calculated correlation One of the edge areas can be selected.

Wherein the blur channel selection unit calculates a contrast value of an edge area of each of the three or more blur images with respect to a subject and calculates a contrast value of edge areas of each of the three or more blur images based on the calculated contrast value Any one edge region can be selected.

According to one embodiment, a multi-aperture camera that adaptively selects a blur channel includes a sharp aperture for introducing a sharp channel optical signal; A blur aperture for introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and for introducing an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel; An image sensor for processing the optical signal of the sharp channel to generate a sharp image and processing the optical signal of the first blur channel and the optical signal of the second blur channel to generate a first blur image and a second blur image, an image sensor; At least one of a correlation of each of the first blur image and the second blur image with respect to the sharp image or a contrast value of a subject of each of the first blur image and the second blur image A blur channel selecting unit adapted to adaptively select any one of the first blur image and the second blur image based on the first blur image and the second blur image; And a depth determination unit for determining a depth of the image based on a change in a blur size between the sharp image and the selected one of the images.

One embodiment provides a multi-aperture camera and method of operation that determine an accurate depth by adaptively selecting and using a plurality of blur images generated from an optical signal of each of a plurality of blur channels with an image to be compared with a sharp image .

Accordingly, the multi-aperture camera according to one embodiment can solve the problem that the wrong depth is determined.

In particular, one embodiment selects a blur image to be compared to a sharp image of a plurality of blur images based on the calculated correlation, by calculating a correlation for each sharp image of the plurality of blur images A multi-aperture camera and a method of operation thereof.

Another embodiment is a multi-aperture image processing method that selects a blur image to be compared with a sharp image of a plurality of blur images based on a calculated contrast value, by calculating a contrast value for a subject of each of the plurality of blur images Camera and method of operation thereof.

In particular, one embodiment can provide a multi-aperture camera and method of operating the same that adaptively select any one of a plurality of blur images for each edge area based on edge areas searched in a sharp image.

1 illustrates a multi-aperture camera that adaptively selects a blur channel according to one embodiment.
FIGS. 2A and 2B are views for explaining a process of adaptively selecting a blur channel according to an exemplary embodiment and determining a depth.
FIGS. 3A and 3B are diagrams for explaining a process of adaptively selecting a blur channel according to an edge region according to an embodiment and determining a depth.
4 is a flowchart illustrating an operation method of a multi-aperture camera that adaptively selects a blur channel according to an exemplary embodiment.
FIG. 5 is a flowchart specifically illustrating a step of adaptively selecting any one of the edge regions shown in FIG. 4 according to an embodiment.
FIG. 6 is a flowchart specifically illustrating a step of adaptively selecting any one of the edge areas shown in FIG. 4 according to another embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Also, terminologies used herein are terms used to properly represent preferred embodiments of the present invention, which may vary depending on the user, intent of the operator, or custom in the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

1 illustrates a multi-aperture camera that adaptively selects a blur channel according to one embodiment.

Referring to FIG. 1, a multi-aperture camera according to an exemplary embodiment includes a sharp aperture 110, a blur aperture 120, an image sensor 130, a blur channel selection unit 140, and a depth determination unit 150, . The blur channel selecting unit 140 and the depth determining unit 150 may be implemented as separate modules. However, the blur channel selecting unit 140 and the depth determining unit 150 are not limited thereto, And may be provided in the image sensor 130.

The sharp aperture 110 introduces the optical signal of the sharp channel.

The blur apertureter 120 introduces an optical signal of a first blur channel of a different wavelength band from the sharp channel and introduces an optical signal of a second blur channel of a wavelength band different from that of the sharp channel and the first blur channel.

For example, the sharp aperture 110 receives an optical signal of a B channel, which is one of an R channel, a G channel, and a B channel, and the blur aperture 120 is an R channel, a G channel, The optical signal of the R channel and the optical signal of the G channel, which are the other channels except for the B channel which is the sharp channel, can be respectively introduced. In this case, the sharp channel may be a B channel, the first blur channel may be an R channel, and the second blur channel may be a G channel.

In another example, the sharp aperture 110 introduces an optical signal of an IR channel, and the blur aperture 120 may introduce optical signals of any two of the R, G, and B channels. In this case, the sharp channel may be an IR channel, the first blur channel may be a B channel, and the second blur channel may be an R channel.

However, the Sharp channel, the first blur channel, and the second blur channel are not limited thereto, and may be set to various wavelength bands different from each other.

In addition, although the sharp aperture 110 and the blur aperture 120 are shown as being formed on one filter in the drawing, they may be separately formed in each of the plurality of filters as required.

In addition, a lens 111 may be formed on the lower or upper portion of the sharp aperture 110 and the blur aperture 120.

The image sensor 130 processes the optical signals of the sharp channel to generate a sharp image, and processes the optical signals of the first blur channel and the optical signals of the second blur channel, respectively, to generate a first blur image and a second blur image can do.

For example, the image sensor 130 processes the optical signal of the B channel, which is a sharp channel, to generate a sharp image, processes the optical signal of the R channel, which is the first blur channel, to generate a first blur image, A second blur image can be generated by processing an optical signal of a G channel that is a blur channel.

The blur channel selection unit 140 adaptively selects any one of the first blur image and the second blur image.

For example, the blur channel selection unit 140 selects the correlation between the first blur image and the sharp image and the correlation between the second blur image and the sharp image , And then selects either the first blur image or the second blur image based on the calculated correlation. More specifically, for example, the blur channel selector 140 may select any one of the first blur image or the second blur image having a higher correlation to the sharp image.

In another example, the blur channel selection unit 140 selects a contrast value for a subject of each of the first blur image and the second blur image (a contrast value in which at least some regions for the subject in the first blur image are distinguished from other regions, A contrast value in which at least a part of the area for the subject in the second blur image is distinguished from the other area), and then selects either the first blur image or the second blur image based on the calculated contrast value . More specifically, the blur channel selector 140 can select any one of the first blur image or the second blur image having a higher contrast value for the object.

Here, various conventional algorithms can be used for calculating the correlation or contrast value. The detailed description thereof will be omitted because it goes beyond the technical idea of the present invention.

Therefore, the depth determining unit 150 determines the depth in the image based on the blur size change between any one image selected by the blur channel selecting unit 140 and the sharp image. For example, the depth determining unit 150 may generate a plurality of blurring images corresponding to the sharp image by applying a plurality of point spread functions (PSFs) to the sharp image and convoluting the plurality of blurring images The depth of the image can be determined based on the similarity between the blur size included in each of the blurred images and the blur size included in any one of the images selected by the blurred channel selecting unit 140. [

At this time, various depth determination algorithms can be used as well as an algorithm for applying a plurality of PSFs, as a technique for determining a depth based on a blur size change between a selected blur image and a sharp image.

A detailed description of the process of adaptively selecting any one of the first blur image and the second blur image to be compared with the sharp image will be described with reference to FIGS. 2A through 2B.

In addition, the blur channel selection unit 140 according to an exemplary embodiment may generate a blurred image based on the first blurred image or the blurred image based on the edge region (an area of a predetermined range searched to include the edge of the subject) It is possible to adaptively select any one edge area to be compared with the edge area of the sharp image in the two blur images. In this case, any one edge region to be compared with the edge region of the sharp image for depth determination may be selected differently for each of the edge regions of the first blur image or the edge region of the second blur image.

Specifically, the blur channel selection unit 140 searches for an edge region including the edge of the subject in the sharp image (for example, after detecting the edge of the subject in the sharp image, Of the second blurred image corresponding to the edge area of the sharp blurred image corresponding to the edge area of the sharp blurred image corresponding to the edge area of the sharp blurred image corresponding to the edge area of the sharp blurred image corresponding to the edge area of the sharp blurred image corresponding to the edge area of the sharp blurred image, Area can be obtained.

Therefore, the blur channel selection unit 140 adaptively selects one of the edge regions of the first blur image or the edge region of the second blur image.

For example, the blur channel selection unit 140 selects the edge area of the first blurred image and the edge area of the sharp image of the second blurred image, The edge area of the second blur image and the edge area of the sharp image) of the first blur image and the edge area of the second blur image based on the calculated correlation, One of the edge areas can be selected. More specifically, the blur channel selection unit 140 selects one edge region having a higher correlation with respect to the edge region of the sharp image of the edge region of the first blur image or the edge region of the second blur image .

Alternatively, the blur channel selection unit 140 may select a contrast value for the subject of each of the edge region of the first blur image and the edge region of the second blur image (at least a portion of the edge region of the first blur image, A contrast value distinguishable from the other area, and a contrast value in which at least a part of the edge area of the second blur image is distinguished from other areas), and then, based on the calculated contrast value, Edge region of the first blur image or an edge region of the second blur image. More specifically, for example, the blur channel selection unit 140 can select any one of the edge regions of the first blurred image or the edge region of the second blurred image having a higher contrast value for the subject.

Accordingly, the depth determining unit 150 can determine the depth of the edge region based on the blur size change between any one edge region selected by the blur channel selecting unit 140 and the edge region of the sharp image. For example, the depth determination unit 150 generates a plurality of blur edge areas corresponding to edge areas of a sharp image by applying a plurality of PSFs to a sharp image and performing convolution, The depth for the edge region can be determined based on the blur size and the similarity between the blur sizes included in any one edge region selected by the blur channel selecting unit 140. [

Similarly, various depth determination algorithms can be used as well as an algorithm for applying a plurality of PSFs, as a technique for determining a depth based on a blur size change between a selected one edge region and a sharp image edge region.

A detailed description of a process of adaptively selecting one edge region to be compared with the edge region of the sharp image of the first blur image or the second blur image in each edge region will be described with reference to FIGS. 3A to 3B do.

As described above, the blur channel selection unit 140 adaptively selects one of the edge regions of the first blur image or the edge region of the second blur image. However, the present invention is not limited thereto, By introducing three or more blur channels of each optical signal in the putter 120 and using the image sensor 130 to generate three or more blur images, the blur channel selector 140 may select three or more blur images It is also possible to adaptively select any one edge region of each edge region. In this case, the operation of the blur channel selection unit 140 may be performed in the same manner as the operation of selecting any one of the edge regions of the two blur images described above.

FIGS. 2A and 2B are views for explaining a process of adaptively selecting a blur channel according to an exemplary embodiment and determining a depth.

Referring to FIG. 2A, a multi-aperture camera according to an embodiment processes an optical signal of a B channel, which is a sharp channel, to generate a sharp image 210 using an image sensor, The optical signal and the optical signal of the G channel, which is the second blur channel, can be processed to generate the first blur image 220 and the second blur image 230, respectively. However, the Sharp channel, the first blur channel, and the second blur channel may be various wavelength bands different from each other.

In this case, since the first blur channel and the second blur channel have different wavelength bands, the first blur image 220 and the second blur image, in which the optical signals of the first blur channel and the optical signals of the second blur channel are processed and generated, The degree of contrast with respect to the subject in each blurred image 230 may be different from each other.

 For example, if the subject has R color and B color as shown in FIG. 2A, the R color area and the B color area of the subject are remarkably contrasted in the sharp image 210 and the first blur image 220 , The second blurred image 230 may be poorly contrasted. If the sharp image 210 and the second blur image 230 are compared to determine the depth, an incorrect depth, such as the first depth map 240, can be determined.

That is, as shown in FIG. 2B, the image intensity (contrast degree) 260 in the first blur image 220 is significantly different in the R color area and the B color area of the subject, whereas the second blur image The image intensity 270 in the subject 230 may be poorly different in the R color area and the B color area of the subject.

Accordingly, the multi-aperture camera calculates the correlation of each of the first blur image 220 and the second blur image 230 with respect to the sharp image 210 using the blur channel selection unit, ) Or the second blur image 230 with a higher correlation.

The multi-aperture camera also calculates the contrast value for the subject of each of the first blur image 220 and the second blur image 230 using the blur channel selection unit, The first blur image 220 having the higher contrast value among the blur images 230 may be selected.

Accordingly, the multi-aperture camera can determine the correct depth, such as the second depth map 250, by comparing the selected first blur image 220 with the sharp image 210 using the depth determining unit.

FIGS. 3A and 3B are diagrams for explaining a process of adaptively selecting a blur channel according to an edge region according to an embodiment and determining a depth.

3A, a multi-aperture camera according to an embodiment processes an optical signal of a B channel, which is a sharp channel, by using an image sensor to generate a sharp image 310, The optical signal and the optical signal of the G channel that is the second blur channel may be processed to generate the first blur image 320 and the second blur image 330, respectively. However, the Sharp channel, the first blur channel, and the second blur channel may be various wavelength bands different from each other.

In this case, since the first blur channel and the second blur channel have different wavelength bands, the first blur image 320 and the second blur image, in which the optical signal of the first blur channel and the optical signal of the second blur channel are processed and generated, The degree of contrast with respect to the subject in each blurred image 330 may be different from each other.

Particularly, when the subject has three or more colors, the degree of contrast of three or more color areas of the subject area may be different from each other in the first blur image 320 and the second blur image 330, respectively.

For example, if the subject has R color, B color, and G color, as shown in FIG. 3A, the R color area and the B color area of the subject are distinguished from the sharp image 310 and the first blur image 320, And may be poorly contrasted in the second blur image 330. [ On the other hand, the G color area and the B color area of the subject are remarkably contrasted in the sharp image 310 and the second blur image 330, and can be poorly contrasted in the first blur image 320. [

3B, the image intensity 360 in the first blur image 320 is significantly different in the R color area and the B color area of the subject, and is poor in the B color area and the G color area The image intensity 370 in the second blur image 330 is poorly different in the R color area and the B color area of the subject and may be significantly different in the B color area and the G color area .

Accordingly, the multi-aperture camera uses the blur channel selection unit to detect the edge areas 311 and 312 of the first blurred image 320 or the second blurred image 330 for each of the edge areas 311 and 312, One of the edge regions to be compared with the edge regions 311 and 312 can be adaptively selected.

Specifically, the multi-aperture camera searches for the edge region 311 in the sharp image 310 and then detects the first blur image 320 corresponding to the edge region 311 of the sharp image 310 of the first blur image 320 Acquires an edge region 321 of the second blurred image 320 and acquires an edge region 331 of the second blurred image 330 corresponding to the edge region 311 of the sharp image 310 of the second blurred image 330 can do.

Accordingly, the multi-aperture camera calculates the correlation between the edge area 321 of the first blur image 320 and the edge area 331 of the second blur image 330 using the blur channel selector, The edge region 321 of the first blur image 320 having the higher correlation among the edge region 321 of the first blur image 320 or the edge region 331 of the second blur image 330 can be selected .

The multi-aperture camera also calculates a contrast value for the subject in each of the edge region 321 of the first blur image 320 and the edge region 331 of the second blur image 330 using the blur channel selection unit The edge area 321 of the first blur image 320 or the edge area 321 of the first blur image 320 having the higher contrast value of the edge area 331 of the second blur image 330 You can also choose.

Thus, the multi-aperture camera compares the edge region 321 of the first blur image 320 thus selected with the edge region 311 of the sharp image 310 using the depth determination unit, The exact depth for the edge region 341 can be determined.

The multi-aperture camera then searches for the next edge region 312 in the sharp image 310 and then searches for the next edge region 312 corresponding to the next edge region 312 of the sharp image 310 of the first blur image 320 The next edge region 322 of the image 320 is obtained and the next edge region 312 of the second blur image 330 corresponding to the next edge region 312 of the sharp image 310 of the second blur image 330 332).

Thus, the multi-aperture camera uses the blur channel selector to calculate the correlation of each of the next edge region 322 of the first blur image 320 and the next edge region 332 of the second blur image 330 A next edge region 322 of the first blur image 320 or a next edge region 332 of the second blur image 330 having a higher correlation of the next edge region 332 of the second blur image 330 ) Can be selected.

The multi-aperture camera also uses the blur channel selector to compare the contrast value of the subject in each of the next edge region 322 of the first blur image 320 and the next edge region 332 of the second blur image 330 Of the second blurred image 330 having the higher contrast value of the next edge region 322 of the first blurred image 320 or the next edge region 332 of the second blurred image 330, Area 332 may be selected.

Accordingly, the multi-aperture camera compares the next edge region 332 of the selected second blurred image 330 with the next edge region 312 of the sharp image 310 using the depth determination unit, (340) to determine the correct depth for the edge region (351), such as the second depth map (350).

Although it has been described above that the edge regions of the different blur images 320 and 330 are adaptively selected for each of the two edge regions in the sharp image 310, the present invention is not limited thereto and the number of edge regions And the position can be adaptively adjusted.

4 is a flowchart illustrating an operation method of a multi-aperture camera that adaptively selects a blur channel according to an exemplary embodiment.

Referring to FIG. 4, a multi-aperture camera according to an exemplary embodiment introduces an optical signal of a sharp channel through a sharp aperture (410).

Next, the multi-aperture camera transmits the optical signals of the first blur channel of the sharp channel and the sharp channel, and the optical signals of the second blur channel of the other blur zone to the first blur channel through the blur aperture Respectively (420).

Then, the multi-aperture camera uses the blur channel selector to select one of the edge areas of the first blurred image corresponding to the edge area of the sharp image or the edge area of the second blurred image corresponding to the edge area of the sharp image The edge region is adaptively selected (430). A detailed description thereof will be described with reference to Figs. 5 and 6. Fig.

The multi-aperture camera then uses the depth determination section to determine the depth for the edge region based on the change in blur size between the edge region of the sharp image and one selected edge region (440) .

For example, a multi-aperture camera applies a plurality of point spread functions (PSFs) to an edge region of a sharp image using a depth determination unit to convolute a plurality of blur edges corresponding to an edge region of a sharp image After creating the regions, the depth for the edge region can be determined based on the degree of similarity between the blur size included in each of the plurality of blur edge regions and the blur size included in any one selected edge region.

FIG. 5 is a flowchart specifically illustrating a step of adaptively selecting any one of the edge regions shown in FIG. 4 according to an embodiment.

Referring to FIG. 5, a multi-aperture camera according to an exemplary embodiment may search for an edge area of a sharp image using a blur channel selector (510).

Then, the multi-aperture camera obtains an edge area of the first blur image corresponding to the edge area of the sharp image in the first blur image using the blur channel selector, and corresponds to the edge area of the sharp image in the second blur image The edge area of the second blur image may be obtained (520).

The multi-aperture camera may then use the blur channel selector to calculate the correlation for the edge region of the sharp image of each of the edge region of the first blur image and the edge region of the second blur image (530 ).

The multi-aperture camera may then use the blur channel selector to select an edge region of either the edge region of the first blur image or the edge region of the second blur image based on the calculated correlation (540) .

For example, the multi-aperture camera may select any one edge region that has a higher correlation to the edge region of the first blur image or the edge region of the sharp image of the edge region of the second blur image.

FIG. 6 is a flowchart specifically illustrating a step of adaptively selecting any one of the edge areas shown in FIG. 4 according to another embodiment.

Referring to FIG. 6, a multi-aperture camera according to another embodiment may search for an edge area of a sharp image using a blur channel selector (610).

Then, the multi-aperture camera obtains an edge area of the first blur image corresponding to the edge area of the sharp image in the first blur image using the blur channel selector, and corresponds to the edge area of the sharp image in the second blur image (620) the edge area of the second blur image.

The multi-aperture camera may then calculate 630 a contrast value for the subject of each of the edge region of the first blur image and the edge region of the second blur image using the blur channel selector.

Thereafter, the multi-aperture camera may select an edge region of either the edge region of the first blur image or the edge region of the second blur image based on the calculated contrast value using the blur channel selector (640) .

For example, the multi-aperture camera may select any one of the edge regions of the first blur image or the edge region of the second blur image that has a higher contrast value for the subject.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (16)

A multi-aperture camera for adaptively selecting a blur channel,
A sharp aperture for introducing the optical signal of the sharp channel;
A blur aperture for introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and for introducing an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel;
An image sensor for processing the optical signal of the sharp channel to generate a sharp image and processing the optical signal of the first blur channel and the optical signal of the second blur channel to generate a first blur image and a second blur image, an image sensor;
A blur channel selection unit for adaptively selecting one of edge areas of the first blur image corresponding to edge areas of the sharp image or edge areas of the second blur image corresponding to edge areas of the sharp image part; And
And a depth determination unit for determining a depth of the edge area based on a change in blur size between an edge area of the sharp image and the selected edge area,
Lt; / RTI >
The blur channel selector
Calculating a correlation of an edge region of the first blurred image and an edge region of the sharp image of each of the edge regions of the second blurred image based on the calculated correlation, Edge area or an edge area of the second blur image, which edge area has a higher correlation with respect to the edge area of the sharp image. delete delete A multi-aperture camera for adaptively selecting a blur channel,
A sharp aperture for introducing the optical signal of the sharp channel;
A blur aperture for introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and for introducing an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel;
An image sensor for processing the optical signal of the sharp channel to generate a sharp image and processing the optical signal of the first blur channel and the optical signal of the second blur channel to generate a first blur image and a second blur image, an image sensor;
A blur channel selection unit for adaptively selecting one of edge areas of the first blur image corresponding to edge areas of the sharp image or edge areas of the second blur image corresponding to edge areas of the sharp image part; And
And a depth determination unit for determining a depth of the edge area based on a change in blur size between an edge area of the sharp image and the selected edge area,
Lt; / RTI >
The blur channel selector
Calculating a contrast value for an object of each of an edge region of the first blurred image and an edge region of the second blurred image based on the calculated contrast value, And selects one edge region having a higher contrast value for the subject among the edge regions of the two blur image. delete The method according to claim 1,
The blur channel selector
Searching an edge region of the sharp image to obtain an edge region of the first blurred image corresponding to an edge region of the sharp image of the first blurred image, And obtains an edge area of the corresponding second blur image. A method of operating a multi-aperture camera adaptively selecting a blur channel,
Introducing an optical signal of a sharp channel through a sharp aperture;
Introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel respectively through a blur aperture; ;
A method of processing an optical signal of the sharp channel, an optical signal of the first blur channel, and an optical signal of the second blur channel, respectively, using an image sensor to generate a sharp image, a first blur image, ≪ / RTI >
A blur channel selection unit is used to adapt any one of the edge areas of the first blurred image corresponding to the edge area of the sharp image or the edge area of the second blurred image corresponding to the edge area of the sharp image Selecting as a target; And
Determining a depth for the edge region based on a change in blur size between an edge region of the sharp image and the selected edge region using a depth determination unit
Lt; / RTI >
Wherein the step of adaptively selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image
Calculating a correlation for an edge region of the first blurred image and an edge region of the sharp image of each of the edge regions of the second blurred image; And
Selecting one of the edge regions of the first blurred image or the edge region of the second blurred image that has a higher correlation to the edge region of the sharp image based on the calculated correlation
Lt; RTI ID = 0.0 > a < / RTI > multi-aperture camera. delete delete A method of operating a multi-aperture camera adaptively selecting a blur channel,
Introducing an optical signal of a sharp channel through a sharp aperture;
Introducing an optical signal of a first blur channel of a wavelength band different from that of the sharp channel and an optical signal of a second blur channel of a wavelength band different from the sharp channel and the first blur channel respectively through a blur aperture; ;
A method of processing an optical signal of the sharp channel, an optical signal of the first blur channel, and an optical signal of the second blur channel, respectively, using an image sensor to generate a sharp image, a first blur image, ≪ / RTI >
A blur channel selection unit is used to adapt any one of the edge areas of the first blurred image corresponding to the edge area of the sharp image or the edge area of the second blurred image corresponding to the edge area of the sharp image Selecting as a target; And
Determining a depth for the edge region based on a change in blur size between an edge region of the sharp image and the selected edge region using a depth determination unit
Lt; / RTI >
Wherein the step of adaptively selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image
Calculating a contrast value for a subject of each of an edge region of the first blurred image and an edge region of the second blurred image; And
Selecting one of the edge areas of the first blur image or the edge area of the second blur image based on the calculated contrast value and having a higher contrast value for the subject
Lt; RTI ID = 0.0 > a < / RTI > multi-aperture camera. delete 8. The method of claim 7,
Wherein the step of adaptively selecting an edge region of either the edge region of the first blur image or the edge region of the second blur image
Searching an edge region of the sharp image;
Obtaining an edge region of the first blur image corresponding to an edge region of the sharp image of the first blur image; And
Obtaining an edge area of the second blur image corresponding to an edge area of the sharp image of the second blur image
Lt; RTI ID = 0.0 > a < / RTI > multi-aperture camera. A multi-aperture camera for adaptively selecting a blur channel,
A sharp aperture for introducing the optical signal of the sharp channel;
A blur aperture for introducing optical signals of each of three or more blur channels of a different wavelength band than the sharp channel;
An image sensor processing the optical signal of the sharp channel to produce a sharp image, and processing an optical signal of each of the three or more blur channels to produce three or more blur images;
A blur channel selection unit adapted to adaptively select any one edge region of each of the three or more blur images corresponding to edge regions of the sharp image; And
And a depth determination unit for determining a depth of the edge area based on a change in blur size between an edge area of the sharp image and the selected edge area,
Lt; / RTI >
The blur channel selector
Calculating a correlation of an edge region of each of the three or more blurred images with respect to an edge region of the sharp image and calculating a correlation of edge regions of each of the three or more blurred images based on the calculated correlation, A multi-aperture camera in which one edge region having the highest correlation with respect to an edge region of a sharp image is selected. delete A multi-aperture camera for adaptively selecting a blur channel,
A sharp aperture for introducing the optical signal of the sharp channel;
A blur aperture for introducing optical signals of each of three or more blur channels of a different wavelength band than the sharp channel;
An image sensor processing the optical signal of the sharp channel to produce a sharp image, and processing an optical signal of each of the three or more blur channels to produce three or more blur images;
A blur channel selection unit adapted to adaptively select any one edge region of each of the three or more blur images corresponding to edge regions of the sharp image; And
And a depth determination unit for determining a depth of the edge area based on a change in blur size between an edge area of the sharp image and the selected edge area,
Lt; / RTI >
The blur channel selector
Calculating a contrast value of an edge region of each of the three or more blurred images relative to the subject and calculating a contrast value of the edge region of each of the three or more blurred images based on the calculated contrast value, Wherein one edge region having a high contrast value is selected. delete
KR1020150179279A 2015-12-15 2015-12-15 Multi aperture camera for adaptively selecting blur channel and opertation method thereof KR101764504B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150179279A KR101764504B1 (en) 2015-12-15 2015-12-15 Multi aperture camera for adaptively selecting blur channel and opertation method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150179279A KR101764504B1 (en) 2015-12-15 2015-12-15 Multi aperture camera for adaptively selecting blur channel and opertation method thereof

Publications (2)

Publication Number Publication Date
KR20170071216A KR20170071216A (en) 2017-06-23
KR101764504B1 true KR101764504B1 (en) 2017-08-03

Family

ID=59283500

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150179279A KR101764504B1 (en) 2015-12-15 2015-12-15 Multi aperture camera for adaptively selecting blur channel and opertation method thereof

Country Status (1)

Country Link
KR (1) KR101764504B1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190105357A (en) * 2018-03-05 2019-09-17 재단법인 다차원 스마트 아이티 융합시스템 연구단 Endomicroscopy using single lens camera and operating method thereof

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101575964B1 (en) * 2014-07-01 2015-12-11 재단법인 다차원 스마트 아이티 융합시스템 연구단 Sensor array included in dual aperture camera

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101575964B1 (en) * 2014-07-01 2015-12-11 재단법인 다차원 스마트 아이티 융합시스템 연구단 Sensor array included in dual aperture camera

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190105357A (en) * 2018-03-05 2019-09-17 재단법인 다차원 스마트 아이티 융합시스템 연구단 Endomicroscopy using single lens camera and operating method thereof
KR102060880B1 (en) * 2018-03-05 2020-02-11 재단법인 다차원 스마트 아이티 융합시스템 연구단 Endomicroscopy using single lens camera and operating method thereof

Also Published As

Publication number Publication date
KR20170071216A (en) 2017-06-23

Similar Documents

Publication Publication Date Title
US9749614B2 (en) Image capturing system obtaining scene depth information and focusing method thereof
KR101672669B1 (en) Multi aperture camera system using disparity
US10212332B2 (en) Image sensor, calculation method, and electronic device for autofocus
US9083877B2 (en) Image processing apparatus, focus detection apparatus, and non-transitory computer-readable storage medium
JP5374119B2 (en) Distance information acquisition device, imaging device, and program
KR101415872B1 (en) Method and apparatus for auto focusing of image capturing
US20090136148A1 (en) Digital auto-focusing apparatus and method
US20190066335A1 (en) Method of calibrating a direction of a pan, tilt, zoom, camera with respect to a fixed camera, and a system in which such a calibration is carried out
US8411195B2 (en) Focus direction detection confidence system and method
JP2015136470A5 (en)
US11347133B2 (en) Image capturing apparatus, image processing apparatus, control method, and storage medium
JP2017208606A5 (en)
CN106973199B (en) Multi-aperture camera system for improving depth accuracy by using focusing distance scanning
US10051263B2 (en) Multi-aperture camera system using scan line processing
US20080226159A1 (en) Method and System For Calculating Depth Information of Object in Image
KR101764504B1 (en) Multi aperture camera for adaptively selecting blur channel and opertation method thereof
JP2017040726A5 (en) Focus detection apparatus and control method thereof
US20150304632A1 (en) Image processing method, image processing apparatus, image pickup apparatus, and non-transitory computer-readable storage medium
KR102135372B1 (en) Structured light system
KR102193285B1 (en) Method for automatic correction of astigmatism
KR101737260B1 (en) Camera system for extracting depth from images of different depth of field and opertation method thereof
JP7034781B2 (en) Image processing equipment, image processing methods, and programs
KR102405635B1 (en) focus control apparatus and focus measure method
KR101775273B1 (en) Method and system for acquiring correlation coefficient between images
KR101733419B1 (en) Searching Method of Auto Focusing

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E701 Decision to grant or registration of patent right