JP6887223B2 - Plenoptic Forbidden Camera - Google Patents

Description

The fields of this disclosure relate to light field imaging. In particular, the present disclosure relates to techniques for recording color light field images.

More specifically, the present disclosure relates to a color filter array (CFA) intended to be attached to a sensor of a Plen optic camera.

This item is intended to introduce to the reader various aspects of the technology that may be relevant to the various aspects of the disclosure described and / or claimed below. This discussion is believed to be useful in providing background information to the reader to aid in a better understanding of the various aspects of this disclosure. However, it should be understood that those statements should be read from this point of view, not as a prior art approval.

Traditional image capture devices render a 3D scene onto a 2D sensor. During operation, conventional capture devices capture a two-dimensional (2D) image that represents the amount of light that reaches each point on a sensor (or photodetector) within the device. However, this 2D image does not contain information about the directional distribution of light rays reaching the sensor (which can be called a light field). For example, depth is lost during capture. Therefore, conventional capture devices do not retain most of the information about the light distribution from the scene.

Lightfield capture devices (also referred to as "lightfield data acquisition devices") have been designed to measure the four-dimensional (4D) lightfield of a scene by capturing light from different perspectives of the scene. Thus, by measuring the amount of light traveling along each beam of light across the sensor, such devices can post-process additional light information (bundles of light) to provide new imaging applications. Information on the directional distribution of The information collected / obtained by the light field capture device is called light field data. A light field capture device is defined herein as any device capable of capturing light field data.

The processing of light field data, in particular but without limitation, is to generate a refocused image of the scene, to generate each view of the scene, to generate a depth map of the scene, to extend depth of focus (EDOF). Includes extending depth of field) generating images, generating stereoscopic images, and / or any combination thereof.

Among some existing groups of light field capture devices, the "plenoptic device" or "plenoptic camera" has a microlens array located in the image focus field of the main lens in front of the sensor. .. One microimage is projected on the sensor for each microlens. In the following description, the area of the sensor for which one (or more) microimages are projected is the "microimage area" or "sensor microimage" or "sensor microimage" or "exposure section". It is called "or" projection section ". The resulting raw image of the scene is the sum of the microimages of one sensor and / or all the pixels of the sensors belonging to the other. In that case, the angular information of the light field is given by the relative position of the pixels in the microimage area with respect to their center. Based on this raw image, the extraction of an image of the captured scene from a particular point of view (also called "demultiplicating") can be performed. The demultiplexing process can be considered as a data conversion from a 2D low image to a 4D light field. The resulting demultiplexed lightfield data can be represented by a matrix of views in which all views are aligned horizontally and vertically.

For color detection, most sensor pixels record only the amount of visible photons that pass through them, regardless of their color. It is known from background art to mount a color filter array (CFA) on top of a sensor to obtain color 4D light field data. For example, a Bayer filter that covers 2x2 pixels and has three different colors Red (R), Green (G), and Blue (B) placed in the order of RGGB in a 2x2 matrix. , Widely used as CFA. However, one of ordinary skill in the art will appreciate that the expression "color filter array" used in the following description refers to all alternative CFAs of the art, not just Bayer filters.

One of the main drawbacks of this technique is that at least half of the photons are blocked by the CFA, which reduces the optical sensitivity of the sensor. In addition, chromatic aberration occurs at the edges of the sensor microimage and can therefore affect the quality of the rendered image. In an attempt to solve these problems, US Patent Application Publication No. 2014/0146201 (A1) (Patent Document 1) discloses a CFA that covers only a part of a sensor microimage. However, it is insufficient to improve the optical sensitivity of the sensor.

Therefore, it is desirable to provide a sensor that indicates an improvement over the background technology.

In particular, it is desirable to provide such a sensor that features better light sensitivity while maintaining satisfactory color sensitivity and limiting the risk of the appearance of chromatic aberration.

U.S. Patent Application Publication No. 2014/0146201 (A1)

References herein to "one embodiment," "embodiment," and "exemplary embodiment" are made in any way, although the described embodiments may have specific mechanisms, structures, or properties. Indicates that the embodiments do not necessarily have to have that particular mechanism, structure or properties. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular mechanism, structure, or property is described in relation to an embodiment, such mechanism, structure, in connection with another embodiment, whether explicitly described or not. Alternatively, it is suggested that acting on the property is within the knowledge of one of ordinary skill in the art.

In one particular embodiment of the technique, a sensor intended to be attached to a Plenoptic camera is disclosed. The sensor has at least one microimage area intended to detect the microimage refracted by the microlens array. The microimage area is at least partially covered by the color filter array. The color saturation of the color filter array decreases as it moves away from the center of gravity of the microimage area.

In the following description, the expression "partially covered" refers to the position of the CFA compared to the sensor microimage (microimage area). In one embodiment, such CFA covers the entire sensor microimage. In other embodiments, the CFA covers only a portion of the sensor microimage. That part includes or does not include the center of gravity of the microimage area. Based on the distance between the CFA point to be considered and the centroid of the sensor microimage, the color saturation of this point decreases as this distance increases.

The reduced color saturation of the CFA makes it possible to capture more photons than the background technology CFA when approaching the boundaries of the sensor microimage. Therefore, such sensors are characterized by improved light sensitivity. In addition, since the pixels at the center of gravity are illuminated by photons that have less aberration than the photons that pass through the pupil boundary of the main lens, such sensors further maintain their color sensitivity at saturation levels while maintaining chromatic aberration. Allows you to limit the risk of emergence.

In one embodiment, the centroid of the color filter array corresponds to the centroid of the microimage area.

Due to the chromatic aberration phenomenon, the pixels at the center of gravity of the sensor microimage provide sharper detail and color than the pixels located at the boundaries of the sensor microimage. Thus, positioning the CFA at the center of gravity of the sensor microimage with respect to CFA surfaces with equal values makes it possible to improve the quality of the rendered microlens image.

According to the embodiment, the color saturation of the color filter array _{decreases from the center of gravity (xi, j} , y _{i, j} ) of the micro image area toward the boundary of the micro image area.

In one embodiment, the color saturation of the color filter array varies from 1 at the center of gravity of the microimage area to 0 at the boundary of the microimage area.

In one embodiment, the CFA is a Bayer filter and the color components are given in the HSV color space as in Equation 1 below:

このとき、（ｘ，ｙ）は、マイクロレンズ（ｉ，ｊ）の下にあるピクセルの座標であり、（ｘ_ｉ，ｊ，ｙ_ｉ，ｊ）は、マイクロイメージエリアの重心の座標であり、ｐは、２つの隣接したマイクロイメージエリアの夫々の重心の間の距離である。

At this time, (x, y) is the coordinates of the pixels under the microlens (i, j), and (x _{i, j} , y _{i, j} ) is the coordinates of the center of gravity of the micro image area. p is the distance between the respective centers of gravity of the two adjacent microimage areas.

Such a smooth change in saturation from the center of gravity (xi _{, j} , y _{i, j} ) of the microimage area to the boundary prevents the staircase effect in color saturation of the refocused image.

In one embodiment, the microimage area is only partially covered by the color filter array.

According to this particular CFA layout, the ratio corresponding to the number of recorded photons compared to the total number of photons increases as the number of pixels covered by the CFA decreases. Therefore, the sensor is light sensitive compared to the CFA that covers the entire sensor microimage.

In one embodiment, the color filter array covers only the 4x4 pixel area at the center of gravity of the microimage area.

According to this particular layout, each microlens image covers 50.26 pixels and only 16 pixels receive less light. The sensor microimage collects a reduced number of photons from 16 pixels and all visible photons from the remaining pixels. Therefore, such a sensor captured more than 84% of photons. This is a significant improvement over the prior art CFA. As an example, for a typical Bayer pattern covering all pixels, this ratio is about 50%.

In one particular embodiment of the technique, a method of determining _{the color component Rx} of at least one refocused image of a microimage area partially covered by a color filter array is disclosed. Such a method
Refocusing been image R _a of the micro-image area of the pixel covered by the color filter _array, a step of determining the _b, color saturation of the color filter array, the center of gravity (x _i of the micro image area _{, J} , y _{i, j} ) Decreased as you move away from the steps and
A step of determining a _{refocused image RT} of pixels in the microimage area that is not covered by the color filter array.
It has a step of determining the color component R _x by combining the refocused images R _{a and b} and the refocused image R _T.
The step of determining the color component R _x introduces a weight to the refocused _{image RT.}

The introduction of such weights has the advantage of distinguishing photons that are largely unaffected by the photon aberration of the main lens from photons that are subject to greater photoaberration. Those skilled in the art will appreciate that the expression "color component" refers to any color component of the CFA implemented. For example, when considering a sensor covered by a Bayer filter, the determined color component can be a red, green or blue component.

In one particular embodiment of the technique, the present disclosure relates to a computer program product that is downloadable from a communication network and / or recorded on a computer-readable and / or processor-executable medium. Related. Such computer program products have program code instructions that implement the method.

In one particular embodiment of the technique, the present disclosure can be performed by a processor and is non-transient computer readable in which a computer program product containing program code instructions that implement the method is recorded. Related to various carrier media.

Although not explicitly stated, the present embodiment may be used in any combination or partial combination.

The present disclosure may be better understood with reference to the following description and drawings given as an example without limiting the scope of protection.
It is the schematic which shows the general Bayer filter. It is the schematic which shows the HSV color space. It is a diagram showing the pixel response of various color filters. It is the schematic which shows the Plen optic camera. It is the schematic which shows the light field data recorded by the sensor of a Plen optic camera. It is the schematic which shows the Plen optic camera which has W> P. It is the schematic which shows the Plen optic camera which has W <P. It is a schematic diagram of the chromatic aberration phenomenon. It is the schematic which shows the Plen optic camera. FIG. 6 is a schematic representation of light field data recorded by a Plenoptic camera sensor for various ray wavelengths. FIG. 6 is a schematic representation of light field data recorded by a sensor covered by a Bayer filter in a Plenoptic camera. FIG. 5 is a schematic representation of light field data recorded by a sensor in a Plenoptic camera according to an embodiment of the present disclosure. FIG. 6 is a schematic representation of light field data recorded by a sensor partially covered by a CFA. FIG. 5 is a flow chart of a series of steps performed when performing a method according to an embodiment of the present disclosure. It is a schematic block diagram which shows the apparatus which determines the color component of the refocused image according to one Embodiment of this disclosure. The components in the figure are not necessarily in their actual size and are instead highlighted to illustrate the principles of the present disclosure.

General concepts and specific details of certain embodiments of the present disclosure are shown in the following description and in FIGS. 1-13 to provide a complete understanding of such embodiments. It will be appreciated by those skilled in the art that the present disclosure may have additional embodiments or may be implemented without any of the details described in the description below. There will be.

[1 General concept]
[1.1 Color filter array]
Color images are typically formed using three color components per pixel. Common components are red, green, and blue, which correspond slightly to the color sensitivity of the eye. Unfortunately, most sensors record visible photons entering a pixel, regardless of the photon wavelength. A color filter array (CFA) is commonly attached to the sensor to make the gray level sensor a color-enabled sensor. The most common CFA consists of 2x2 pixels and is a so-called Bayer pattern (filter) that is replicated across the sensors as shown in FIG.

Images recorded using CFA are incomplete as only one color is recorded per pixel. Calculations of two other colors of a given pixel are described in US Pat. No. 5,506,619 (A) entitled "Adaptive color plan interpolation in single sensor color electronic camera" in the case of the Bayer pattern. As described above, it is performed by interpolation using the surrounding colors according to the so-called demosaiking algorithm. Such algorithms can affect image quality and require heavy computational load.

[1.2 Thoughts on RGB and HSV color space]
RGB (Red-Green-Blue) is a color space used to characterize the color of a pixel with three color components. These three values can be converted to other color spaces, such as the Hue Saturation Value (HSV) color space. The HSV space is from pure color (given hue, saturation and lightness set to maximum) to white (same hue, saturation set to 0, and maximum), as shown in FIG. Allows you to define a color gradient up to (brightness set in).

The conversion from RGB color space to HSV space is performed by the following equation (Equation 2):

ＨＳＶ色空間からＲＧＢ色空間への変換は、次の数３によって実行される：

The conversion from the HSV color space to the RGB color space is performed by the following equation 3:

このとき、｛Ｈ｝はＨ′の小数部分を表す。トリプレット（Ｐ，Ｖ，Ｑ）から、Ｈ′よりも小さい最大整数
（外１）

に応じて、値（Ｒ，Ｇ，Ｂ）が定義される。Ｉ＝０の場合に（Ｒ，Ｇ，Ｂ）＝（Ｖ，Ｔ，Ｐ）、Ｉ＝１の場合に（Ｒ，Ｇ，Ｂ）＝（Ｑ，Ｖ，Ｐ）、Ｉ＝２の場合に（Ｒ，Ｇ，Ｂ）＝（Ｐ，Ｖ，Ｔ）、Ｉ＝３の場合に（Ｒ，Ｇ，Ｂ）＝（Ｐ，Ｑ，Ｖ）、Ｉ＝４の場合に（Ｒ，Ｇ，Ｂ）＝（Ｔ，Ｐ，Ｖ）、Ｉ＝５の場合に（Ｒ，Ｇ，Ｂ）＝（Ｖ，Ｐ，Ｑ）。

At this time, {H} represents the fractional part of H'. From triplet (P, V, Q), maximum integer smaller than H'(outside 1)

Values (R, G, B) are defined accordingly. When I = 0, (R, G, B) = (V, T, P), when I = 1, (R, G, B) = (Q, V, P), when I = 2. When (R, G, B) = (P, V, T), I = 3, (R, G, B) = (P, Q, V), when I = 4, (R, G, B) ) = (T, P, V), I = 5, (R, G, B) = (V, P, Q).

The above equation assumes that the three color components are real numbers between 0 and 1. In return, the components S and V are real numbers between 0 and 1, and H is a real number between 0 and 360 degrees.

As pointed out earlier, an important drawback of CFA is that some photons are blocked by color filters, resulting in reduced sensor sensitivity. For example, according to the Bayer pattern, at least half of the photons are lost in the filter. FIG. 3 shows how many photons are collected relative to the total number of photons entering a pixel for various color filters in Kodak's sensor. The efficiency of the sensor is greatly influenced by the color filter.

[1.3 Description of Plenoptic Camera]
FIG. 4 represents an overview Plenoptic camera 1 having a main lens 2, a microlens 3, and a sensor 4. The main lens 2 receives the light from the scene captured in its object field and renders the light through the microlens array 3. The microlens array 3 is located in the image field of the main lens. In one embodiment, the microlens array 3 includes a plurality of circular microlenses arranged in a two-dimensional (2D) array. In other embodiments, such microlenses have various shapes, eg, elliptical shapes, without departing from the scope of the present disclosure. Each microlens has a lens characteristic that directs the light of the corresponding microimage to a dedicated area on the sensor 4, i.e., the sensor microimage 5.

In one embodiment, some spacer is placed between the microlens array 3 and the sensor 4 around each lens to prevent light from one lens from overlapping with light from another lens on the sensor side.

[1.4 4D light field data]
The image captured by the sensor 4 consists of a set of 2D small images arranged within the 2D image. Each small image is generated by the microlenses (i, j) in the microlens array 3. FIG. 5 shows an example of an image recorded by the sensor 4. Each microlens (i, j) produces a microimage represented by a circle (the shape of the small image depends on the shape of the microlens, which is usually circular). Pixel coordinates are labeled as (x, y). p is the distance between two consecutive microimages 5. The microlens (i, j) is selected so that p is greater than the pixel size δ. The microlens image area 5 is referred to by their coordinates (i, j). Some pixels (x, y) may not receive any light from any microlens (i, j). Such pixels (X) are discarded. In fact, the space between the microlenses is masked to prevent photons from exiting the microlens (no masking is required if the microlens has a quadrangular shape). _{The center (xi, j} , y _{i, j} ) of the microlens image (i, j) is located at the sensor 4 in _{coordinates (xi, j} , y _{i, j).} θ is the angle between the square grid of pixels and the square grid of microlenses. (X _{i, j} , y _{i, j} ) can be calculated by the following equation 1 (Equation 4) with the pixel coordinates of the microlens image ( _{0 , 0} ) as (zx 0, 0, y 0, 0). :

この式は、マイクロレンズアレイ３が正方格子に従って配置されると仮定する。なお、本開示は、この格子に制限されず、六方格子又は不規則格子に等しく当てはまる。

This equation assumes that the Microlens Arrays 3 are arranged according to a square grid. It should be noted that the present disclosure is not limited to this grid and applies equally to hexagonal grids or irregular grids.

FIG. 5 further shows that objects from the scene are visible in several adjacent microlens images (black dots). The distance between two consecutive views of an object is w, and this distance is also called the word "disparity".

[1.5 Optical properties of light field camera]
The distances p and w introduced in subsection 1.4 above are given in pixels. They are converted to physical unit distances P and W (meters), respectively, by multiplying by the pixel size δ:

W = δw and P = δp

Their distance depends on the characteristics of the light field camera.

6 and 7 represent a schematic light field, assuming a perfect thin lens model. The main lens has a focal length F and an aperture Φ. The microlens array has a microlens having a focal length f. The pitch of the microlens array is φ. The microlens array is located at a distance D from the main lens and a distance d from the sensor. The object (not visible in the figure) is located at a distance z from the main lens (left). This object is focused by the main lens at a distance z'from the main lens (right). 6 and 7 represent the cases of D> z'and D <z', respectively. In either case, the microlens image can be focused according to d and f.

The parallax W differs depending on the distance z of the object.

[1.6 Image Refocusing]
The main characteristic of a light field camera is the possibility of calculating a 2D refocused image if the refocusing distance is freely adjustable. The 4D light field data is projected into the 2D image by simply shifting and zooming the microlens image and then adding them to the 2D image. The amount of shift controls the refocusing distance. The projection of 4D lightfield pixels (x, y, i, j) onto the refocused 2D image coordinates (X, Y) is defined by Equation 2 (Equation 5):

このとき、ｓは、２Ｄのリフォーカシングされたイメージのサイズを制御し、ｇは、リフォーカシングされたイメージの焦点調整距離を制御する。

At this time, s controls the size of the 2D refocused image, and g controls the focus adjustment distance of the refocused image.

[1.7 Chromatic Aberration Problem]
As represented by FIG. 8, the chromatic aberration problem stems from the imperfections of the lens. This prevents focusing all colors of the point source in the same image plane.

When examining the effects of chromatic aberration in a Plenoptic camera system as represented by FIGS. 9 and 10, it can be observed that wavelength-dependent fluctuations in the convergence plane change to wavelength-dependent parallax W fluctuations as well. ..

In this regard, one of ordinary skill in the art will notice that the photons entering at the boundary of the main lens 2 are affected by chromatic aberration much larger than the photons entering at the center of the pupil. At this time, the photons passing through the center and the boundary of the pupil of the main lens are the photons collected at _{the center (xi, j} , y _{i, j) and the boundary of the sensor microimage 5, respectively.} Thus, the photons collected at the boundary of the sensor microimage 5 are much more affected by chromatic aberration than the photons collected near _{their center (xi, j} , y _{i, j).}

That is, the image quality of each microlens image _{changes from the center (xi, j} , y _{i, j} ) to the boundary. The pixel in the center of the sensor microimage 5 provides sharper detail and color than the pixel located at the boundary of the sensor microimage 5.

[1.8 CFA attached to Plenoptic camera]
The CFA is placed on the sensor to record a color 4D light field. The sensor records a 4D light field. For example, suppose the CFA consists of M × M pixels at coordinates (a, b) with (a, b) ∈ [0, M] ^2. A 4D light field plus a color parameter is a 5D light field (x, y, i, j, ca _{, b} ). At this time, c _{a and b} are the colors of the CFA coordinates (a and b). FIG. 11 represents a Bayer pattern (M = 2) set on a sensor that records a 4D light field.

Refocusing of the 5D light field is performed by Equation 2, but is applied independently of _{the CFA colors ca and b, respectively.} In this way, M ² refocused images _{Ra and b} were obtained. The images are then combined to calculate the _{three color components R red} , R _green and R _{blue of the refocused RGB image.}

For example, in the case of the Bayer pattern, M = 2, c ₀ , ₀ = red (Red), c ₁ , ₁ = blue (blue), c ₀ , ₁ = c ₁ , ₀ = green (Green). The three color components of the _{refocused RGB image are equal to R red} = R ₀ , ₀ , R green = (R ₀ , ₁ + R ₁ , ₀ ) / 2 and R _blue = R ₁ , ₁ .

[2 Description of First Embodiment of the present Disclosure]
FIG. 12 represents a sensor 4 according to an embodiment of the present disclosure. The sensor 4 can be attached to the Plenoptic camera 1 and has a plurality of microimage areas 5. Each of the microimage areas 5 is covered by CFA6.

The sensor 4 is a CCD (Charge-Coupled Device) type using CMOS (Complementary Metal Oxide Semiconductor) technology. If you are skilled in the art, such a sensor 4 may be a sensor based on a neuromorphic spike (eg, an artificial retina), or some other type of optical sensor known from background technology. It will be understood that may have as an alternative.

The CFA 6 represented by FIG. 12 is a Bayer filter. It should be noted that those skilled in the art will substitute such CFA6 for RGBE filters, CYYM filters, or any other type of CFA known from the background art, without departing from the scope of the present disclosure. It will be understood that it may be possessed by.

In one embodiment of the present disclosure, the color saturation of CFA 6 decreases from the center (X) of the sensor microimage 5 towards the boundary. The upper part of FIG. 12 represents such a decrease in color saturation. Color saturation, the center _{(x i, j, y i} , j) , the well center _{(x i, j, y i} , j) away 1,2 and 3 pixels from the pixels respectively located, 100% It is set to 50%, 25% and 12.5%.

In another embodiment of the present disclosure, as represented by the lower part of FIG. 12, color saturation is given according to the function f as in Equation 6. The function f returns a value between [0.0,1.0] and drops from f (d = 0) to f (p / 2):

この特定の実施形態において、色の飽和は、センサマイクロイメージ５の中心（ｘ_ｉ，ｊ，ｙ_ｉ，ｊ）での１からその境界での０まで線形に変化する。色成分は、次の数７のように、ＨＳＶ色空間において与えられる：

In this particular embodiment, the color saturation varies linearly from 1 at the _{center (xi, j} , y _{i, j) of the sensor microimage 5 to 0 at its boundary.} The color components are given in the HSV color space, as in Equation 7 below:

マイクロイメージエリア５の中間（ｘ_ｉ，ｊ，ｙ_ｉ，ｊ）から境界への飽和のそのような滑らかな変化は、リフォーカシングされた画像の色飽和における階段効果を防ぐ。

Such a smooth change in saturation from the middle (xi _{, j} , y _{i, j} ) of the microimage area 5 to the boundary prevents the staircase effect in color saturation of the refocused image.

This reduction in saturation in the CFA allows more photons to be captured than a typical Bayer filter and thus improves the photosensitivity of the sensor 4. In addition, such a sensor 4 is satisfactory because the pixels in the center (xi _{, j} , y _{i, j} ) are illuminated by photons that have less aberration than the photons that pass through the pupil boundary of the main lens. It makes it possible to limit the risk of the appearance of chromatic aberration while maintaining a high color sensitivity.

[3 Description of the Second Embodiment of the Present Disclosure]
According to the second embodiment of the present disclosure, the CFA 6 is applied only to a part of the sensor microimage 5. FIG. 13 represents a 4D light field captured by a Plenoptic camera 1 equipped with a square grid and a microlens array 3 having a pitch φ = 8δ (8 pixels). The Bayer pattern is duplicated only at the 4x4 pixels in _{the center (xi, j} , y _{i, j) of each sensor microimage 5.} The diameter p of the microlens image is equal to p = φ (D + d) / D. Since (D + d) / D is extremely close to 1.0, it may be considered that the diameter of the microlens image is given by the pitch of the microlens array.

This particular design allows the sensor 4 to be more sensitive to light as compared to the CFA 6 which covers the entire sensor microimage 5.

As an example, according to the second embodiment, the sensor 4 having a constant color saturation of CFA 6 represented by FIG. 13 is tested under operating conditions. For such a sensor 4, each microlens image covers 50.26 pixels, and only 16 pixels (ie, the 4x4 pixels covered by the Bayer pattern) receive less light. The sensor microimage 5 collects half the photons from 16 pixels and all visible photons from the remaining pixels. The number of recorded photons compared to the total number of photons is {(50.26-16 x 0.5) /50.26} x 100 = 84%. This ratio is greater than about 50% of the Bayer pattern covering all pixels.

For those skilled in the art, the total number of photons in the case of the sensor 4 according to the second embodiment, in which the color saturation of the CFA _{decreases from the center (xi, j} , y _{i, j) of the sensor microimage 5 toward the boundary.} It will be appreciated that the ratio corresponding to the number of recorded photons compared to is considerably higher, further improving the optical sensitivity of the sensor 4.

[4 Calculation of refocused RGB image]
The following paragraph relates to the sensor 4 according to the second embodiment of the present disclosure. Those skilled in the art will appreciate that such implementation may be applied to the sensor 4 according to the first embodiment of the present disclosure without departing from the scope of the present disclosure.

Than this, considering the sensor 4 according to the second embodiment of the present disclosure, CFA pattern of M ² pieces of corresponding to the filter M ² pieces of refocusing image R _a, and _b, covered by the color filter 6 It is possible to calculate with a single refocused image _{RT corresponding to no pixels.}

Several methods to convert ^{M 2} pieces of refocusing is image _{R a, b} and refocusing is image _{R T,} 3 one color component _R red refocusing by _image, the _{R green} and _{R blue} Can be used for. For example, in the case of the Bayer pattern, M = 2, c ₀ , ₀ = red (Red), c ₁ , ₁ = blue (blue), c ₀ , ₁ = c ₁ , ₀ = green (Green). Three color components refocusing been RGB _{image, R red = (R 0,} 0 + R T) / 2, R gree n = (R 0, 1/2 + R 1, 0/2 + R T) / 2 and _{R blue} It can be calculated by = (R ₁ , ₁ + _{RT) / 2.} Obviously, other combinations may be used without departing from the scope of this disclosure.

As represented by FIG. 14, the method according to one embodiment of the present disclosure is:
Step 7 to determine the _{refocused images Ra, b} of the microimage area pixels (i, j) covered by the color filter array 6 and
Step 8 to determine the _{refocused image RT} of the microimage area pixels (i, j) not covered by the color filter array 6 and
It has step 9 of determining the color component R _x by combining the refocused images R _{a and b} and the refocused image _RT. Step 9 of determining R _x introduces a weight (W) into the refocused _{image RT.}

Those skilled in the art will appreciate that the expression "color component" refers to any color component of the CFA implemented. For example, when considering a sensor covered by a Bayer filter, the determined color components are red R _r , green R _g , or blue R _b component.

The refocused image _RT has lower image quality than _{the other refocused images R a, b} because it collects photons more affected by the photoaberration of the main lens 2. To take this image characteristic into account, _{the values of RT} and _{Ra, b} are mixed according to the local contrast. For _example, the weight of _{R a, b} for _{R T} (W) _{is, R a,} lowered around locally identified Tesukucha in _b.

_{The introduction of such a weight W of RT to Ra, b} into the combination of R _{a, b} and _RT causes photons that are less affected by the photon aberration of the main lens to undergo greater photoaberration. Has the advantage of distinguishing from.

FIG. 15 represents an example of a device 10 that determines _{the color component Rx} of at least one refocused image of a microimage area 5 partially covered by a color filter array 6 according to an embodiment of the present disclosure. It is a schematic block diagram. Such a device 10 includes a processor 11, a storage unit 12, an interface unit 13 and a sensor 4 connected by a bus 14. Of course, the components of the computer device 10 may be connected by a connection other than the bus connection by the bus 14.

The processor 11 controls the operation of the device 10. The storage unit 12 includes at least one program executed by the processor 11, data of the Plenoptic image, parameters used in the calculation performed by the processor 11, intermediate data of the calculation executed by the processor 11, and the like. Stores various data. Processor 11 may be formed by any known suitable hardware or software, or by a combination of hardware and software. For example, the processor 11 may be formed by dedicated hardware such as a processing unit, or by a programmable processing unit such as a CPU that executes a program stored in its memory.

The storage unit 12 may be formed by any suitable storage or means capable of storing programs, data, or the like in a computer-readable manner. Examples of the storage unit 12 include semiconductor memory devices and non-transitory computer readable storage media such as magnetic, optical, or optical magnetic recording media loaded on read and write units. The program tells processor 11 at least one refocused image of the microimage area 5 partially covered by the color filter array 6 according to embodiments of the present disclosure as described above with reference to FIG. The process for determining the color component R _x of is executed.

The interface unit 13 provides an interface between the device 10 and the external device. The interface unit 13 may communicate with an external device via cable or wireless communication. In this embodiment, the external device may be a Plenoptic camera. In this case, the Plen optic image can be input from the Plen optic camera to the device 1 through the interface unit 13 and then stored in the storage unit 12.

The device 10 and the Plenoptic camera may communicate with each other via cable or wireless communication.

Even if only one processor 11 is shown in FIG. 15, those skilled in the art would appreciate such processors as various modules and units that embody the functionality performed by the device 10 according to embodiments of the present disclosure. May have. For example:
A module that determines the _{refocused images Ra, b} of the microimage pixels (i, j) covered by the color filter array 6;
A module that determines the _{refocused image RT} of microimage pixels (i, j) that are not covered by the color filter array 6;
A module that determines the color component R _x by combining the refocused images R _{a and b} and the refocused image _RT.

These modules may also be embodied in several processors 9 that communicate and cooperate with each other.

As will be apparent to those skilled in the art, aspects of this principle may be embodied as a system, method or computer readable medium. However, aspects of this Principle may be in the form of a fully hardware embodiment, a fully software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware embodiments. Can be taken.

When this principle is implemented by one or more hardware components, the hardware components are processors that are integrated circuits such as central processing units and / or microprocessors, and / or application-specific integrated circuits ( ASIC), and / or application set processor (ASIP), and / or graphics processing unit (GPU), and / or physical processing unit (PPU), and / or floating point unit, and / or network processor, and / Or have an audio processor and / or a multi-core processor. Further, the hardware component may further have a baseband processor (eg, having a memory unit and firmware) and / or a wireless electronic circuit (which may have an antenna) capable of transmitting and receiving radio signals. .. In one embodiment, the hardware components are such as ISO / IEC18092 / ECMA-340, ISO / IEC21481 / ECMA-352, GSMA, StoLPaN, ETSI / SCP (smart card platform), global platform (ie, secure element). Follow one or more standards. In a variant, the hardware component is a radio frequency identification (RFID) tag. In one embodiment, the hardware components are Bluetooth® communication and / or Wi-Fi® communication and / or Zigbee® communication and / or USB communication and / or firewire. It has a circuit that enables communication and / or NFC (near field) communication.

Furthermore, aspects of this principle can take the form of computer readable storage media. Any combination of one or more computer-readable storage media may be utilized.

Thus, for example, any flowchart, flow diagram, state transition diagram, pseudocode, and the like are substantially represented in a computer-readable storage medium, and thus, by a computer or processor, such a computer or processor Represents various processes that can be performed, whether explicitly stated or not.

のように、ＨＳＶ色空間において与えられ、このとき、（ｘ，ｙ）は、マイクロレンズ（ｉ，ｊ）の下にあるピクセルの座標であり、（ｘ _ｉ，ｊ ，ｙ _ｉ，ｊ ）は、前記マイクロイメージエリアの重心の座標であり、ｐは、２つの隣接したマイクロイメージエリアの夫々の重心（ｘ _ｉ，ｊ ，ｙ _ｉ，ｊ ）の間の距離である、
付記４に記載のセンサ。
［付記６］
前記マイクロイメージエリアは、前記カラーフィルタアレイによって部分的にのみ覆われる、
付記１乃至５のうちいずれか一項に記載のセンサ。
［付記７］
前記カラーフィルタアレイは、前記マイクロイメージエリアの重心にある４×４のピクセルエリアのみを覆う、
付記６に記載のセンサ。
［付記８］
カラーフィルタアレイによって部分的に覆われているマイクロイメージエリアの少なくとも１つのリフォーカシングされたイメージの色成分Ｒ _ｘ を決定する方法であって、
前記カラーフィルタアレイによって覆われている前記マイクロイメージエリアのピクセルのリフォーカシングされたイメージＲ _ａ，ｂ を決定するステップであり、前記カラーフィルタアレイの色飽和は、前記マイクロイメージエリアの重心から遠ざかる場合に低下する、ステップと、
前記カラーフィルタアレイによって覆われていない前記マイクロイメージエリアのピクセルのリフォーカシングされたイメージＲ _Ｔ を決定するステップと、
前記リフォーカシングされたイメージＲ _ａ，ｂ と前記リフォーカシングされたイメージＲ _Ｔ とを結合することで前記色成分Ｒ _ｘ を決定するステップと
を有し、
前記色成分Ｒ _ｘ を決定するステップは、前記リフォーカシングされたイメージＲ _Ｔ に対して重みを導入する、方法。
［付記９］
センサを有し、該センサは、マイクロレンズアレイによって屈折されるマイクロイメージを検知することを対象とした少なくとも１つのマイクロイメージエリアを有し、
前記マイクロイメージエリアは、カラーフィルタアレイによって少なくとも部分的に覆われており、
前記カラーフィルタアレイの色飽和は、前記マイクロイメージエリアの重心から遠ざかる場合に低下する、
ライトフィールドデータ収集デバイス。
［付記１０］
通信ネットワークからダウンロード可能であり、且つ／あるいは、コンピュータによって読み出し可能な及び／又はプロセッサによって実行可能な媒体において記録されているコンピュータプログラムであって、
付記８に記載の方法を実施するプログラムコード命令を含むコンピュータプログラム。
［付記１１］
プロセッサによって実行されることが可能であり、付記８に記載の方法を実施するプログラムコード命令を含むコンピュータプログラムが記録されている非一時的なコンピュータ読み出し可能なキャリア媒体。 The present disclosure has been described with reference to one or more examples, but as will be appreciated by those skilled in the art, without departing from the scope of this disclosure and / or the appended claims. Changes may be made in the details.
[Appendix 1]
It is a sensor intended to be attached to a Plenoptic camera.
It has at least one microimage area intended to detect the microimage refracted by the microlens array.
The microimage area is at least partially covered by a color filter array.
The color saturation of the color filter array decreases as it moves away from the center of gravity of the microimage area.
Sensor.
[Appendix 2]
The center of gravity of the color filter array corresponds to the center of gravity of the micro image area.
The sensor according to Appendix 1.
[Appendix 3]
The color saturation of the color filter array decreases from the center of gravity of the microimage area toward the boundary of the microimage area.
The sensor according to Appendix 1 or 2.
[Appendix 4]
The color saturation of the color filter array varies from 1 at the center of gravity of the microimage area to 0 at the boundary of the microimage area.
The sensor according to any one of Appendix 1 to 3.
[Appendix 5]
The color filter array is a Bayer filter, and the color components are

Given in the HSV color space, where (x, y) is the coordinates of the pixels under the microlens (i, j), (x _{i, j} , y _{i, j} ) is , The coordinates of the center of gravity of the microimage area, where p is the distance between the respective _{centers of gravity (xi, j} , _{yi, j) of the two adjacent microimage areas.}
The sensor according to Appendix 4.
[Appendix 6]
The microimage area is only partially covered by the color filter array.
The sensor according to any one of Appendix 1 to 5.
[Appendix 7]
The color filter array covers only the 4x4 pixel area at the center of gravity of the microimage area.
The sensor according to Appendix 6.
[Appendix 8]
A method of determining _{the color component R x} of at least one refocused image of a microimage area partially covered by a color filter array.
_{A step of determining the refocused images Ra, b} of the pixels of the microimage area covered by the color filter array, where the color saturation of the color filter array moves away from the center of gravity of the microimage area. Steps down to
A step of determining a _{refocused image RT} of pixels in the microimage area that is not covered by the color filter array.
A step of determining the color component R _x by combining the refocused images R _{a and b} and the refocused image R _T.
Have,
The step of determining the color component R _x is a method of introducing weights into the refocused _{image RT.}
[Appendix 9]
Having a sensor, the sensor has at least one microimage area intended to detect the microimage refracted by the microlens array.
The microimage area is at least partially covered by a color filter array.
The color saturation of the color filter array decreases as it moves away from the center of gravity of the microimage area.
Lightfield data acquisition device.
[Appendix 10]
A computer program that is downloadable from a communication network and / or recorded on a computer-readable and / or processor-executable medium.
A computer program that includes program code instructions that implement the method described in Appendix 8.
[Appendix 11]
A non-temporary computer-readable carrier medium on which a computer program containing program code instructions that can be executed by a processor and implements the method described in Appendix 8 is recorded.

Claims (11)

It is a sensor intended to be attached to a Plenoptic camera.
It has at least one microimage area intended to detect the microimage refracted by the microlens array.
The microimage area is at least partially covered by a color filter array.
The color saturation in terms of color components of the color filter array decreases as the micro image centroid of the area until the point of the color components of the color filter array, the distance is increased,
Sensor. The center of gravity of the color filter array corresponds to the center of gravity of the micro image area.
The sensor according to claim 1. The color saturation of the color filter array decreases from the center of gravity of the microimage area toward the boundary of the microimage area.
The sensor according to claim 1 or 2. The color saturation of the color filter array varies from 1 at the center of gravity of the microimage area to 0 at the boundary of the microimage area.
The sensor according to any one of claims 1 to 3. The color filter array is a Bayer filter, and the color components are

Given in the HSV color space, where (x, y) is the coordinates of the pixels under the microlens (i, j), (x _{i, j} , y _{i, j} ) is , The coordinates of the center of gravity of the microimage area, where p is the distance between the respective _{centers of gravity (xi, j} , _{yi, j) of the two adjacent microimage areas.}
The sensor according to claim 4. The microimage area is only partially covered by the color filter array .
The microimage area is larger than the area of the color filter array.
The sensor according to any one of claims 1 to 5. The color filter array is not covered only 4 pixels area × 4 in the center of gravity of the micro image area,
The microimage area is larger than the area of the color filter array.
The sensor according to claim 6. A method of determining _{the color component R x} of at least one refocused image of a microimage area partially covered by a color filter array.
_{It is a step of determining the refocused images Ra, b} of the pixels of the microimage area covered by the color filter array, and the color saturation in terms of the color components of the color filter array is the microimage. From the center of gravity of the area to the point of the color component of the color filter array , the step decreases as the distance increases.
A step of determining a _{refocused image RT} of pixels in the microimage area that is not covered by the color filter array.
It has a step of determining the color component R _x by combining the refocused images R _{a and b} and the refocused image R _T.
The step of determining the color component R _x is a method of introducing weights into the refocused _{image RT.} Having a sensor, the sensor has at least one microimage area intended to detect the microimage refracted by the microlens array.
The microimage area is at least partially covered by a color filter array.
The color saturation in terms of color components of the color filter array decreases as the micro image centroid of the area until the point of the color components of the color filter array, the distance is increased,
Lightfield data acquisition device. A computer program that is downloadable from a communication network and / or recorded on a computer-readable and / or processor-executable medium.
A computer program comprising a program code instruction that implements the method of claim 8. A non-transitory computer-readable carrier medium on which a computer program containing program code instructions that can be executed by a processor and that implements the method of claim 8 is recorded.

Images