JP2017188729A - Information processing device, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable chromatic aberration to be properly corrected for acquired light field data even when a lens to be used upon acquiring the light field data is an unknown lens.SOLUTION: An information processing device comprises: acquisition means for acquiring light field data; separation means for separating the light field data for each of a plurality of channels having different wavelength; derivation means for deriving an adjustment parameter for matching the positions and angles of light beams recorded in each light field data from a correspondence relation between a plurality of pieces of light field data obtained by separation; and adjustment means for adjusting the position and angle of a light beam recorded in at least one piece of light field data of the plurality of pieces of light field data on the basis of the adjustment parameter derived by the derivation means.SELECTED DRAWING: Figure 13

Description

  The present invention relates to correction of chromatic aberration in light field data.

  In recent years, by adding a new optical element to the optical system, it is possible to obtain information on the angle and intensity of light (light field data), and to adjust the focus position by image processing (refocus) later. A technology called “tational photography” has been developed.

  If this technique is used, since focus adjustment can be performed after imaging, there is an advantage that failure of focus adjustment during imaging can be compensated by image processing. Further, by changing the image processing method, it is possible to obtain a plurality of images focused on an arbitrary subject in the image from one captured image, and there is an advantage that the number of imaging can be reduced.

  As an apparatus for acquiring light field data, there is a plenoptic camera in which a microlens array is arranged in the vicinity of an image sensor of a digital camera constituted by an optical system and an image sensor. In ordinary digital cameras, degradation of image quality due to chromatic aberration of the lens becomes a problem, but the same applies to plenoptic cameras. As a method for reducing chromatic aberration in a plenoptic camera, a method is disclosed in which correction is performed at the time of image generation based on aberration characteristics obtained in advance (Patent Document 1).

Special table 2008-515110 gazette

  However, the method disclosed in Patent Document 1 needs to obtain aberration characteristics in advance based on lens design information and measurement results used when acquiring light field data. Therefore, when an unknown lens is used by exchanging lenses in an imaging apparatus that acquires light field data, such as a plenoptic camera, the method disclosed in Patent Document 1 corrects chromatic aberration. Cannot be done properly.

  An information processing apparatus according to the present invention includes an acquisition unit that acquires light field data, a separation unit that separates light field data for each of a plurality of channels having different wavelengths, and correspondence between the plurality of light field data obtained by separation. From the relationship, a derivation means for deriving an adjustment parameter for making the position and angle of the light beam recorded in each light field data coincide with each other, and based on the adjustment parameter derived by the derivation means, at least of the plurality of light field data Adjusting means for adjusting the position and angle of the light beam recorded in one light field data.

  According to the present invention, even when a lens used when acquiring light field data is an unknown lens, chromatic aberration can be appropriately corrected for the acquired light field data.

It is a block diagram showing an example of composition of an imaging device of a 1st embodiment. It is a figure which shows an example of a structure of an imaging part. It is a figure which shows typically an example of the optical system in 1st Embodiment. It is a figure for demonstrating the expression method of light field data. It is a figure for demonstrating propagation of the light ray in space, and refraction by a lens. It is a figure which shows an example of the light field of the light ray which propagates through an optical system. It is a figure which shows an example of a to-be-photographed object and an optical system. It is a figure which shows typically a mode that light field data change according to propagation of a light ray. It is a figure for demonstrating chromatic aberration. It is a figure for demonstrating the shift | offset | difference by the chromatic aberration in a light field. 3 is a block diagram illustrating an example of a software configuration of the imaging apparatus according to the first embodiment. FIG. It is a figure which shows the process in which the plane on light field data is detected. 3 is a flowchart illustrating an example of a processing flow of the imaging apparatus according to the first embodiment. It is a block diagram showing an example of the software configuration of the imaging device of the second embodiment. 10 is a flowchart illustrating an example of a processing flow of the imaging apparatus according to the second embodiment.

[Embodiment 1]
<Overall configuration of imaging device>
FIG. 1 is a block diagram illustrating an example of a configuration of an information processing apparatus (hereinafter also referred to as an imaging apparatus) according to the first embodiment.
As illustrated in FIG. 1, the imaging apparatus according to the present embodiment includes a CPU 101, a ROM 102, a RAM 103, an operation unit 104, a display control unit 105, a media interface (hereinafter, media I / F) 107, a character generation 109, and an encoder unit 111. A digital signal processing unit 112 and an imaging unit 113. Note that the imaging unit 113 may be installed outside the imaging apparatus.

  The imaging unit 113 is, for example, a plenoptic camera, and acquires light field data using an optical system and an imaging device. Details of the imaging unit 113 will be described later.

  The CPU 101 sequentially reads and interprets instructions stored in the ROM 102 and RAM 103 and executes processing according to the result. In addition, the ROM 102 and the RAM 103 provide the CPU 101 with programs, data, work areas, and the like necessary for executing processing.

  The bus 110 functions as a path for each component to exchange data and processing instructions with each other.

  The operation unit 104 corresponds to a button, a mode dial, and the like, and receives a user instruction input via these buttons.

  The character generation 109 generates character data, graphic data, and the like.

  The display unit 106 is, for example, a liquid crystal display, and displays captured image data and character data received from the character generation 109 and the display control unit 105. Further, the display unit 106 may have a touch screen function. In this case, the operation unit 104 can receive a user instruction input via the display unit 106.

  The digital signal processing unit 112 adjusts the luminance value of the image data acquired by the imaging unit 113 and interpolates defective pixels. These processes are performed before image processing (the flow shown in FIG. 12) described later.

  The encoder unit 111 encodes the light field data generated as a result of the chromatic aberration correction processing in the image processing.

  The media I / F 107 is an interface for connecting the medium (for example, hard disk, memory card, CF card, SD card, USB memory) 108 and the bus 110. Light field data encoded by the encoder 111 is output via the media I / F 107. In addition to the medium 108, a PC (personal computer) or the like may be connected to the media I / F 107.

  In addition, although the component of an apparatus exists besides the above, since it is not the main point of this invention, description is abbreviate | omitted.

<Configuration of imaging unit>
FIG. 2 is a diagram illustrating an example of the configuration of the imaging unit 113. The configuration of the imaging unit 113 will be described with reference to FIG.

  The lens 201 and the lens 202 constitute an imaging optical system. The light emitted from the subject passes through the diaphragm 203 and the shutter 204 by the imaging optical system, and then forms an image on a microlens array 205 including a plurality of microlenses (hereinafter simply referred to as a microlens 205). Further, the light beam refracted by the micro lens 205 passes through the IR cut filter 206, the low pass filter 207, and the color filter 208 and then reaches the image sensor 209. In this embodiment, a color image is acquired using three colors of RGB as a color filter.

  The image sensor 209 and the A / D converter 210 correspond to an image sensor such as a CMOS image sensor. The image sensor 209 is arranged in a two-dimensional lattice and converts incident light rays into an electrical signal. The A / D conversion unit 210 converts the information on the light beam converted into an electric signal into a digital signal.

  The control unit 212 performs imaging by controlling the image sensor 209 and the shutter 204 in accordance with an instruction of the CPU 101 input via the bus 110. Further, the control unit 212 controls the lens 201 and the lens 202 to change the focus position and the focal length to a state designated by the user. In addition, the control unit 212 controls the diaphragm 203 to change the aperture to a state designated by the user.

  Image information converted into a digital signal is stored in the buffer 211 and becomes light field data. The light field data is sent to the CPU 101 via the bus 110.

  FIG. 3 is a diagram schematically illustrating an example of the optical system according to the first embodiment. As shown in FIG. 3A, the optical system has, as the main lens 300, a thick convex lens composed of one or more lenses corresponding to the lens 201 and the lens 202, and a microlens on the focal point thereof. An array 301 (hereinafter simply referred to as a microlens 301) is included. On the focal point of the microlens 301, as shown in FIG. Light field data is obtained from the position of the microlens through which the incident light beam is specified and the angle at which the light enters the microlens, which is specified from the pixel position of the image sensor 302. The micro lens 301 is installed on the focal point of the main lens 300 when the main lens 300 is focused at infinity. You may change the relative position of the microlens 301 and the image pick-up element 302, and the main lens 300 by focus adjustment.

  Note that the configuration of the imaging unit shown here is a simplified example for explanation, and any configuration having a function of acquiring a light field for light of a plurality of wavelengths through an optical system having chromatic aberration. But it doesn't matter.

<Expression method of light field>
A light field expression method used in the description of this embodiment will be described. FIG. 4 is a diagram for explaining a light field expression method. In the present embodiment, the discussion proceeds on the assumption of paraxial approximation. In this embodiment, it is assumed that the optical axis of the optical system is in the z direction as shown in FIG. A light ray passing through a surface having a constant z coordinate is expressed by four-dimensional coordinates using the position (x, y) of the passing point and the inclination (u, v) of the light ray. Note that u represents the inclination of the light beam in the x direction, and v represents the inclination of the light beam in the y direction. FIG. 4B shows three light rays A, B, and C on the xz plane in real space. As shown in FIG. 4B, the inclination of the light beam is represented by u = Δx / Δz. FIG. 4C is a diagram showing light rays A, B, and C on the xu plane (z = 0) in the light field space. Since A and B have the same inclination, u coordinates are equal. On the other hand, since B and C have the same passing position on the z = 0 plane, their x coordinates are equal.

Next, how the light ray changes in the light field space due to propagation in the space and refraction by the lens will be described. Here, for the sake of convenience, the light field is considered in two dimensions, position x and inclination u. FIG. 5 is a diagram for explaining propagation of light in space and refraction by a lens. FIG. 5A is a diagram showing light rays propagating in real space on the xz plane. Here, the light field coordinates of the light beam at z = z ₀ are (x ₀ , u ₀ ), and the light field coordinates of the light beam at z = z ₁ are (x ₁ , u ₁ ). Since the light beam goes straight, the relationship between the light field coordinates at z = z ₀ and z = z ₁ is as shown in Equation 1.

FIG. 5B is a diagram showing light rays refracted by the lens on the xz plane. Here, the light field coordinates of the light beam before refraction at z = z ₀ are (x ₀ , u ₀ ), and the light field coordinates of the light beam after refraction are (x ₁ , u ₁ ). The relationship between the light field coordinates before and after refraction using the focal length f due to the properties of the optical system is expressed by Equation 2.

  The above relationship holds true for the position y and the inclination v of the light beam.

<Example of light field of propagating light beam>
An example of a light field of rays propagating through the optical system is shown in FIG. FIG. 6A is a diagram showing light rays on the xz plane in real space. The optical axis is on the straight line of x = 0, and the lens is placed at the position of z = 0. In FIG. 6A, light rays A, B, C, and D are emitted from the positions where z = −3 and x = ± 1. FIG. 6B shows the light field coordinates of each ray when the rays A, B, C, and D reach z = 3 after propagating from z = -3 to z = 0 and being refracted. It is shown. The light field coordinates of each ray at z = -3 are (1, 1), (1, -1), (-1, 1), and (-1, -1), respectively. After each light beam is refracted by the lens, it reaches a position where z = 3 and x = ± 1. It can be seen that propagation and refraction in the space are shear on the light field coordinates as expressed by Equation 1 and Equation 2, respectively.

FIG. 7 is a diagram illustrating an example of a subject and an optical system. Light rays emitted from subjects 701, 702, and 703 placed at z = z ₀ , z ₁ , and z ₂ pass through a lens 704 placed at z = z ₃ , and a microlens placed at z = z ₄ It reaches an array 705 (hereinafter simply referred to as a microlens 705). Light rays emitted from each subject are recorded as light field data by the image sensor 706. FIG. 8 is a diagram schematically showing how the light field changes in accordance with the propagation of light rays in the example shown in FIG. First, at z = z ₀ , the light field is entirely black reflecting the color of the subject 701. In the upper left, upper center, upper right, lower left, and lower center of FIG. 8, light field data is shown when recording is possible at each position of z = z _{0 to} z ₃ . When z = z ₁ , a light beam emitted from the subject 702 is added to the light field. At this time, the portion of the light beam emitted from the subject 701 that overlaps the subject 702 is shielded by occlusion. When z = z ₀ , the light field of light rays emitted from the subjects 701 and 702 by propagation undergoes shear transformation as shown in Equation 1, and the light field of light rays emitted from the subject 703 is superimposed thereon. At z = z ₃ , the light field of rays emitted from all objects by propagation undergoes further shear transformation. When refraction occurs at z = z ₃ , the light field undergoes a shear transformation as shown in Equation 2. With propagation from z = z ₃ to z = z ₄ , the light field again undergoes a shear transformation as in Equation 1. As a result, a light field as shown in the lower right diagram of FIG. 8 is recorded by the microlens 705 and the image sensor 706. Note that the lower right diagram of FIG. 8 shows a light field recorded with z = z ₂ in focus.

<Chromatic aberration in the light field>
When an optical system is composed of a material having a refractive index dispersion such as glass, the characteristics of the optical system change depending on the wavelength of light, that is, the color. Due to this phenomenon, as shown in FIG. 9, the focal length and the position of the main plane change for each wavelength. FIG. 9 is a diagram for explaining chromatic aberration. Light L which is emitted from the object 9 to propagate space, results refracted by an optical system 902 having distributed, is separated into a light beam L _B of the green light L _G and blue. Then, they are offset from one another and the light beam L _B of the green light L _G and blue. Here, the plane 903 is an object-side principal plane to the wavelength of L _B. Plane 904 is the object side principal plane to the wavelength of L _G. The planar 905 is the image side principal plane to the wavelength of L _B. Plane 906, an image-side principal plane to the wavelength of L _G. The shift of the image forming position due to the wavelength dispersion of the optical system so that L _G and L _B are shifted is called chromatic aberration. FIG. 10 is a diagram for explaining a shift due to chromatic aberration in the light field. In FIG. 10 (a), is recorded by the microlens 907 and the image sensor 908, light field data L _G is shown. In FIG. 10 (b), are recorded by the microlens 907 and the image sensor 908, light field data L _B is shown. As shown in FIG. 10, are offset from one another and L _G and L _B. The imaging apparatus according to the present embodiment corrects the light field data so that such a shift due to chromatic aberration is eliminated. For example, imaging apparatus, in the example shown in FIG. 10, so as to overlap and the L _G and L _B, adjusting at least one of the light field data L _G and L _B. In the present embodiment adjusts the light field data L _B based on the L _G. Such adjustments, return the L _B having reached the microlenses 907 to the surface 901 on the object side space characteristics for a wavelength of L _B, be propagated again L _B to the microlens 907 in characteristic to the wavelength of L _G be equivalent to. Therefore, such adjustment can be performed by combining the conversion according to Expressions 1 and 2 and the inverse conversion thereof. Therefore, if the coordinates of the point on the light field data of the L _G wavelength are (x _G , u _G ) and the coordinates of the point on the light field data of the L _B wavelength are (x _B , u _B ), The correspondence between them can be expressed by linear transformation as shown in Equation 3.

The same relationship holds for the position y and the inclination v of the light beam in the vertical direction with respect to (x, v). Therefore, when the coordinates of the point on the light field data of the L _G wavelength are (y _G , v _G ) and the coordinates of the point on the light field data of the L _B wavelength are (y _B , v _B ), 4 holds.

  In this embodiment, chromatic aberration is corrected by obtaining such a linear conversion from the correspondence between the light field data of each color.

<Configuration and processing of image processing unit>
FIG. 11 is a block diagram illustrating an example of a software configuration of the imaging apparatus according to the first embodiment. A software configuration of the imaging apparatus will be described with reference to FIG.

  The imaging apparatus of this embodiment includes a channel separation unit 1101, an image generation unit 1102, a feature point detection unit 1103, a feature amount calculation unit 1104, a plane detection unit 1105, a correspondence search unit 1106, an adjustment parameter derivation unit 1107, and light field data adjustment. Part 1108 and a channel integration part 1109. In this embodiment, the CPU 101 loads the program stored in the ROM 102 into the RAM 103 and executes the loaded program, so that each unit shown in FIG. 11 functions.

  The channel separation unit 1101 acquires light field data via the bus 110 and separates it into channels of each color. It is assumed that the acquired light field data is converted in advance according to the light field coordinates on the microlens 205 based on the design information and configuration information of the imaging unit 113.

  The image generation unit 1102 generates the two-dimensional image data along the xy plane by cutting out the plane of u = 0 and v = 0 from the light field data separated for each color channel. FIG. 12 is a diagram illustrating a process of detecting a plane on the light field data. In FIG. 12, the light field data is represented in two dimensions of position x and inclination u for convenience. In FIG. 12A, a region indicated by the image data generated by the image generation unit 1102 is indicated by a black rectangle with respect to the xu plane of the light field data.

  The feature point detection unit 1103 extracts feature points from the image data generated by the image generation unit 1102 using Harris corner detection or the like. In FIG. 12B, examples of detected feature points are indicated by black rectangles with respect to the xu plane of the light field data. In this embodiment, Harris corner detection is used as a feature point extraction method. However, any method can be used as long as it can detect feature points such as DoG (Difference of Gaussian) and FAST (Features from Accelerated Segment Test). I do not care.

  The feature amount calculation unit 1104 calculates a feature amount for identifying each feature point detected by the feature point detection unit 1103 based on the image data generated by the image generation unit 1102. In the present embodiment, the feature amount calculation unit 1104 calculates a feature amount using pixel values of blocks around the feature point.

The plane detection unit 1105 detects a plane on the light field data passing through each feature point detected by the feature point detection unit 1103. Here, the plane on the light field data is a two-dimensional linear subspace formed by a group of rays emitted from the same point on the subject. The distribution of values on the xy plane in the vicinity of this plane is different at (u, v), reflecting the property that the intensity of light rays emitted from the same point on the subject changes gently with respect to the angle of the light rays. Is a similar pattern. Therefore, the detection of the plane on the light field data creates a block in the vicinity of the feature point on the xy plane where u = 0 and v = 0 where the feature point is detected, and the sum of squares of differences with this block is minimized. The position (x, y) is searched for on a different (u, v) xy plane. When the plane detection unit 1105 performs a search on the xy plane where u ≠ 0 and v ≠ 0, the image generation unit 1102 cuts out a plane where u ≠ 0 and v ≠ 0 from the light field data, and xy. Two-dimensional image data along a plane is generated. Then, the plane detection unit 1105 performs the search for the region (search range) indicated by the two-dimensional image data. In FIG. 12C, an example of a block used for searching is indicated by a black rectangle. Further, in FIG. 12D, an example of the search range is indicated by a black rectangle. On the xy plane _where (u, v) = (u _n , v _m ), when the sum of squared differences is minimum at a position where (x, y) = (x _n , y _m ), the plane (light field The relationship shown in Equation 5 is obtained as information for constraining the plane on the data.

This is obtained for a plurality of (u _n , v _m ) as expressed by Equation 6. Note that N in Equation 6 is the total number of samplings in the u direction. M is the total number of samplings in the v direction.

  In FIG. 12E, the relationship between a plurality of x and u obtained by the search is indicated by a black rectangle. Rays emitted in the direction of (U, V) from the position (X, Y) are converted as shown in Equation 7 before being recorded at the point (x, y, u, v) on the light field data. receive. a, b, c, and d are constants. Expression 7 is a conversion expression when it is assumed that the optical axis is at the center of the lens (lenses 201 and 202).

Equation 7 represents a linear subsection formed by a group of light rays emitted from the same point on the subject, that is, a plane on the light field data, if it is assumed that it holds for an arbitrary direction (U, V). The constraint condition of (x, y, u, v) that should be satisfied by the light ray group emitted in an arbitrary direction (U, V) from the position (X, Y) on a certain subject is expressed by U, V from Equation 7. And are obtained as equations 8 and 9.

When p = d, q = −b, r = (bc−ad) x ′, and s = (bc−ad) y ′, the constraints shown in equations 8 and 9 are rearranged to obtain equations 10 and 11.

p, q, r, and s are parameters that determine the plane on the light field data. The plane detection unit 1105 obtains p, q, r, and s as a least-squares solution that minimizes Equation 12 using the relationship shown in Equation 5. At this time, the plane detection unit 1105 uses a constraint condition as shown in Expression 13 so that p, q, r, and s do not become zero.

The least square solution is obtained as an eigenvector corresponding to the minimum eigenvalue of the matrix F expressed by Equation 14. In FIG. 12F, an example of a linear subspace obtained by the least square method, that is, a plane on the light field data is shown by a solid line.

However,

  In the present embodiment, the plane parameters on the light field data are obtained by the least squares solution using all the relationships of Expression 5. However, robust estimation such as RANSAC may be used in order to improve robustness with respect to a loss or erroneous correspondence of a linear subspace due to occlusion or bignetting. In the present embodiment, the block search is performed so that the sum of squared differences is minimized, and the plane constraint condition on the light field data is obtained. However, the correspondence between the light field data of each color may be obtained by other methods such as block search based on normalized cross-correlation and search by gradient method. Further, in this embodiment, the constraint condition of the plane on the light field data is obtained by the correspondence search based on the block on the xy plane, and the plane parameter on the light field data is obtained by the least square method. However, any method may be used as long as the plane on the light field data can be specified. For example, a method of directly optimizing the plane parameters on the light field data so as to maximize the similarity on the xy plane between different uvs, or a method of expanding the Hough transform to four dimensions may be used.

  The correspondence search unit 1106 associates a plane (a plane on the light field data) between the light field data of different channels based on the feature points. The association is performed by searching for a feature point having the most similar feature amount with respect to a feature point of a reference channel from feature points of different channels. In the present embodiment, normalized cross-correlation is used as the similarity between feature quantities. Further, in a general optical system, the deformation of the light field data due to chromatic aberration is slight, so the search target is limited by the distance of the feature points and the distance of the plane (plane on the light field data). In this embodiment, the reference channel is the G component, and the B and R component channel feature points are associated with the reference channel. In this embodiment, matching based on normalized cross-correlation of blocks is used. However, matching based on SIFT (Scale-Invariant Feature Transform) features, matching based on the sum of squared differences, and matching with feature points that are geometrically nearby. Any method may be used.

  The adjustment parameter derivation unit 1107 is expressed by Equations 3 and 4 for the G channel that is the reference channel and the other channels (B channel and R channel) based on the plane relationship associated by the correspondence search unit 1106. Find the relationship of coordinates on the light field. Here, assuming that the coordinates on the reference channel are (x, u) and the coordinates on the other channels are (x ′, u ′), the relationship expressed by Expression 3 and Expression 4 is expressed by Expression 15.

Since (x, u) and (x ′, u ′) are located on a linear subspace formed by a group of rays emitted from the same point on the subject, that is, on a plane on the light field data, respectively, 17 satisfies the constraint condition. Note that p ′, q ′, r ′, and s ′ are parameters that determine linear subspaces in other channels (B channel and R channel).

From the equations 15, 16, and 17, the condition shown in the equation 18 is obtained.

In the present embodiment, h ′, i ′, j ′, and k ′ are calculated by the least square method using the relationship shown in Expression 18. The calculated h ′, i ′, j ′, and k ′ are adjustment parameters between parameters representing the linear subspace. Therefore, in the present embodiment, h, i, j, and k that are adjustment parameters between light field coordinates are derived from h ′, i ′, j ′, and k ′ based on Expression 18. λ is an arbitrary scalar introduced reflecting the indeterminacy regarding the scales of Equations 16 and 17. Therefore, the adjustment parameter deriving unit 1107 deletes λ in advance when calculating h ′, i ′, j ′, k ′. In this embodiment, each row is linearly combined so that the left side of Equation 18 is λ, and λ is obtained and deleted. When the linear combination coefficients are α, β, and γ, the linear combination of each row of Expression 18 is expressed by Expression 19.

The condition of the coefficient in which the left side of Equation 19 is λ is expressed by Equation 20.

In order to stably calculate λ, α, β, and γ that satisfy Equation 20 and minimize Equation 21 are obtained.

When γ is eliminated from Expression 21 using Expression 20, and a coefficient for minimizing Expression 21 is obtained under the condition that the partial differentiation by α and β is 0, Expression 22 is obtained.

Substituting Equations 19 and 22 into Equation 18 yields Equation 23.

However,

  Further, by similarly deriving (y, v) of the light field coordinates, Expression 24 is obtained.

However,

Since the relationship between Expression 23 and Expression 24 is obtained for each correspondence between planes on the light field data between channels, h ′, i ′, j ′, k ′ is obtained by the least square method using a plurality of correspondences. . Here, there are N corresponding, parameters representing n-th plane (light field data on the plane), for the reference channel is expressed as p _n, q _n, r _n, s _n, other channels for _{p'n, q'n, r'n} , expressed as _s'n. At this time, h ′, i ′, j ′, and k ′ to be obtained are expressed by Expression 25. Note that N in Expression 25 represents the total number of feature points detected in step S1204.

However,

H ′, i ′, j ′, k ′ satisfying Expression 25 is obtained from the relationship that the partial differentiation by h ′, i ′, j ′, k ′ becomes 0 when the objective function on the right side is the minimum value. And is calculated by Equation 26.

Adjustment parameters h, i, j, and k are derived from h ′, i ′, j ′, and k ′ calculated as described above, as shown in Expression 27. In the present embodiment, the adjustment parameter is obtained by the least square solution using the correspondence of all the planes (planes on the light field data). However, in order to improve the robustness against miscorrespondence, robust estimation such as RANSAC is used. It doesn't matter.

  Based on the adjustment parameter derived by the adjustment parameter deriving unit 1107, the light field data adjustment unit 1108 adjusts the light field data of each channel so as to overlap the light field data of the reference channel. In this embodiment, expressions such as “superimpose on the light field data of the reference channel” are used. However, in practice, the position and angle of the light beam indicated by the light field data of the reference channel are matched with the position and angle of the light beam indicated by the other light field data.

  First, the light field data adjustment unit 1108 corresponds to the sampling coordinates (x, y, u, v) on the reference channel and corresponding coordinates (x ′, y ′, u ′, v ′) on another channel. Is calculated based on Equation 28. Next, the light field data adjustment unit 1108 generates a value at coordinates (x ′, y ′, u ′, v ′) on another channel by interpolation. The light field data adjustment unit 1108 arranges the value generated by the interpolation at the sampling position on the reference channel. In this way, the light field data adjustment unit 1108 linearly converts and adjusts the light field data of each channel.

  The channel integration unit 1109 integrates the light field data of the B and R channels adjusted by the light field data adjustment unit 1108 and the light field data of the reference G channel to generate color light field data. . The generated light field data is output via the bus 110.

  FIG. 13 is a flowchart illustrating an example of a processing flow of the imaging apparatus according to the first embodiment. A processing flow of the imaging apparatus according to the present embodiment will be described with reference to FIG.

  In step S1201, the channel separation unit 1101 acquires light field data.

  In step S1202, the channel separation unit 1101 separates one new channel from the light field data acquired in step S1201. At this time, the channel separation unit 1101 selects the G channel as a new channel. Then, every time the process of step S1202 is repeatedly executed, the channel separation unit 1101 switches the new channel between the B channel and the R channel. That is, channel separation section 1101 separates channels in the order of G channel, B channel, and R channel.

  In step S1203, the image generation unit 1102 cuts out data with the ray angles u = 0 and v = 0 from the light field data of the channel extracted in step S1202, and generates two-dimensional image data.

  In step S1204, the feature point detection unit 1103 extracts feature points from the image data generated in step S1203. In step S1205, the feature amount calculation unit 1104 calculates a feature amount indicating the feature point feature extracted in step S1204. In step S1206, the plane detection unit 1105 detects a plane on the light field data passing through the feature point detected in step S1205.

  In step S1207, it is determined whether the data being processed is light field data of the G channel. If it is G-channel light field data (YES in step S1207), the imaging apparatus proceeds to step S1208. If it is not the G channel (NO in step S1207), the imaging apparatus proceeds to the process in step S1209.

  In step S1208, the correspondence search unit 1106 holds G channel feature values, feature point positions, and plane information on the light field data. Thereafter, the imaging apparatus returns to the process of step S1202.

  In step S1209, the correspondence search unit 1106 associates the plane on the light field data of the G channel with the plane on the other channel light field data. This association is performed based on the G channel information held in step S1208 and the other channel information calculated in steps S1204 to S1206.

  In step S1210, the adjustment parameter deriving unit 1107 derives an adjustment parameter based on the correspondence of the plane obtained by the correspondence searching unit 1106 in step S1209. Here, an adjustment parameter is derived for superimposing the light field data of the channel being processed on the light field data of the G channel.

  In step S1211, the light field data adjustment unit 1108 adjusts the light field data of the channel being processed so as to overlap the light field data of the G channel based on the adjustment parameter derived in step S1210. The adjusted light field data is held by the channel integration unit 1109.

  In step S1212, the imaging apparatus determines whether all channels have been processed. If processed (YES in step S1212), the imaging apparatus proceeds to the process in step S1213. If not processed (NO in step S1212), the imaging apparatus returns to the process in step S1202.

  In step S1213, the channel integration unit 1109 integrates the B channel and R channel light field data adjusted in step S1211 and the G channel light field data to generate color light field data.

  In step S1214, the channel integration unit 1109 outputs the light field data generated in step S1213.

  In this embodiment, assuming that there is no error in the optical axis position of the optical system, correction of chromatic aberration in the light field is expressed by two-dimensional linear conversion common to the xu plane and the yv plane. However, you may express using other transformations, such as the four-dimensional affine transformation with respect to xyuv space. In this case, the method described in the present embodiment is extended to estimate a four-dimensional affine transformation. As a result, it is possible to cope with an error in the optical axis position. In this embodiment, an example of a light field having three wavelengths of RGB has been described. However, the present invention may be applied to a light field having two wavelengths or a light field having more than three wavelengths. In the present embodiment, an imaging apparatus that adjusts light field data of other channels (R and B channels) with the G channel as a reference is taken as an example. However, it is not always necessary to provide a reference channel, and the light field data of all channels may be superimposed by adjusting the light field data of all channels.

  As described above, according to the present embodiment, it is possible to correct chromatic aberration in the light field without requiring lens information. In other words, according to the present embodiment, even when the lens used when acquiring the light field data is an unknown lens, it is possible to appropriately correct the chromatic aberration with respect to the acquired light field data. In this embodiment, the adjustment parameter is derived from the correspondence between the plane on the light field data of the reference channel and the plane on the other channel light field data. Therefore, for example, in the process of step S1204, the adjustment parameter can be appropriately derived even when the number of extracted feature points is not sufficient. Therefore, the accuracy of correcting chromatic aberration can be maintained even in such a case.

[Embodiment 2]
In the first embodiment, correction of chromatic aberration is performed by superimposing light field data of different channels on the basis of correspondence of a plane (a plane on the light field data). In the second embodiment, feature points are extracted from light field data, and light field data of different channels are superimposed on each other based on the correspondence of the feature points, thereby correcting chromatic aberration. In this case, since it is not necessary to detect a plane on the light field data, the calculation cost can be reduced.

  FIG. 14 is a block diagram illustrating an example of a software configuration of the imaging apparatus according to the second embodiment. The imaging apparatus of the present embodiment includes a channel separation unit 1101, a feature point detection unit 1401, a feature amount calculation unit 1402, a correspondence search unit 1403, an adjustment parameter derivation unit 1404, a light field data adjustment unit 1108, and a channel integration unit 1109. . The channel separation unit 1101, the light field data adjustment unit 1108, and the channel integration unit 1109 are the same as those in the first embodiment. In this embodiment, the feature point detection unit 1401, the feature amount calculation unit 1402, the correspondence search unit 1403, and the adjustment parameter derivation unit 1404 derive adjustment parameters for superimposing light field data of different channels on each other. Hereinafter, a different part from 1st Embodiment is demonstrated.

  The feature point detection unit 1401 uses, as a feature point, a point having a nonuniform pixel value distribution in all directions in the light field space from the light field data of each channel acquired from the channel separation unit 1101. To detect. In the present embodiment, as an example, Harris corner detection for each of the xu plane and the yv plane is used. Such feature points are detected in an area where light rays emitted from a textured subject are shielded by occlusion on the light field. In this embodiment, Harris corner detection is used as the feature point extraction method. However, any method may be used as long as feature points such as DoG and FAST can be detected.

  The feature amount calculation unit 1402 calculates a feature amount for identifying each feature point detected by the feature point detection unit 1401 based on the light field data of each channel acquired from the channel separation unit 1101. In the present embodiment, the feature amount calculation unit 1402 calculates a feature amount using pixel values of blocks around the feature point.

  The correspondence search unit 1403 associates feature points between light field data of different channels. The association is performed by searching for a feature point having the most similar feature amount with respect to a feature point of a reference channel from feature points of different channels. In the present embodiment, normalized cross-correlation is used as the similarity between feature quantities. Further, in a general optical system, light field data is hardly deformed due to chromatic aberration, and the search target is limited by the distance between feature points. In this embodiment, the reference channel is the G component, and the B and R component channel feature points are associated with the reference channel. In this embodiment, matching based on normalized cross-correlation of blocks is used. However, any method such as matching based on SIFT feature values, matching based on a sum of squares of differences, or matching with feature points that are geometrically adjacent to each other is used. It doesn't matter.

The adjustment parameter deriving unit 1404 is based on the relationship between the feature points associated by the correspondence searching unit 1403, using Equations 3 and 4 for the G channel that is the reference channel and the other channels (B channel and R channel). Find the relationship of coordinates in the light field space as shown. Adjustment parameters representing the relationship of coordinates are derived by solving Equations 29 and 30 by the least square method. In this _{case, (x n, y n,} u n, v n) is the n-th feature point coordinates on the reference channel _{(x'n, y'n, u'} n, v'n) otherwise Are the coordinates of the nth feature point on the channel.

In the present embodiment, the adjustment parameter is obtained by the least square solution using the correspondence of all the feature points, but robust estimation such as RANSAC may be used in order to improve the robustness against erroneous correspondence.

  The light field data adjustment unit 1108 adjusts the light field data based on the adjustment parameters (h, i, j, k) as in the first embodiment.

  FIG. 15 is a flowchart illustrating an example of a processing flow of the imaging apparatus according to the second embodiment. A processing flow of the imaging apparatus according to the present embodiment will be described with reference to FIG. Note that the processes in steps S1501, S1502, S1505, and S1509 to S1512 are the same as the processes in steps S1201, S1202, S1207, and S1211 to S1214 in the first embodiment, and thus description thereof is omitted.

  In step S1503, the feature point detection unit 1401 detects feature points from the light field data of the channel separated in step S1201.

  In step S1504, the feature amount calculation unit 1402 calculates a feature amount based on the feature points detected in step S1501.

  In step S1506, the correspondence search unit 1403 stores the information on the feature amount and feature point position of the G channel.

In step S1507, the correspondence searching unit 1403 associates the feature points of the G channel and other channels based on the information on the G channel held in step S1506 and the information on the other channels calculated in steps S1503 to S1504.
In step S1508, the adjustment parameter deriving unit 1404 derives an adjustment parameter for adjusting the light field data of the channel being processed so as to overlap the G channel based on the correspondence of the feature points obtained in step S1507.

  In the present embodiment, correction of chromatic aberration in the light field is expressed by two-dimensional linear transformation common to the xu plane and the yv plane, assuming that there is no error in the optical axis position of the optical system. However, you may express using other transformations, such as the four-dimensional affine transformation with respect to xyuv space. In this case, the method described in the present embodiment is extended to estimate a four-dimensional affine transformation. As a result, it is possible to cope with an error in the optical axis position.

  As described above, according to the present embodiment, the same effects as those of the first embodiment can be obtained. Further, according to the present embodiment, it is not necessary to detect a plane on the light field data, so that the calculation cost can be reduced.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1101 Channel separation unit 1107 Adjustment parameter derivation unit 1108 Light field data adjustment unit

Claims (15)

An acquisition means for acquiring light field data;
Separating means for separating the light field data for each of a plurality of channels having different wavelengths;
Deriving means for deriving adjustment parameters for making the positions and angles of the light rays recorded in each light field data coincide with each other from the correspondence between the plurality of light field data obtained by separation,
Adjusting means for adjusting a position and an angle of a light beam recorded in at least one light field data among the plurality of light field data based on the adjustment parameter derived by the deriving means. Information processing device. The adjusting means includes
2. The information processing apparatus according to claim 1, wherein among the plurality of light field data, one light field data is used as a reference, and a position and an angle of a light beam recorded in the other light field data are adjusted based on the adjustment parameter. . The derivation means includes
Detection that detects a feature point from image data generated from the one light field data serving as a reference, and further detects a linear subspace formed by a group of rays emitted from a point on the subject corresponding to the feature point Means,
Correspondence search means for searching a linear subspace corresponding to the linear subspace detected by the detection means from image data generated from the other light field data;
The information processing apparatus according to claim 2, further comprising: an adjustment parameter deriving unit that derives the adjustment parameter from a correspondence relationship between the linear subspace detected by the detection unit and the linear subspace searched by the correspondence search unit. The detecting means detects a plurality of feature points from image data generated from the one light field data serving as a reference, and further detects a linear subspace corresponding to the feature points for each of the plurality of feature points. And
The correspondence search unit searches for a linear subspace corresponding to the linear subspace for each of the plurality of linear subspaces detected by the detection unit from image data generated from the other light field data,
The adjustment parameter derivation unit derives the adjustment parameter based on a correspondence relationship obtained with each of the plurality of linear subspaces detected by the detection unit and with the linear subspace searched by the correspondence search unit. 3. The information processing apparatus according to 3. The information processing apparatus according to claim 4, wherein the adjustment parameter deriving unit derives the adjustment parameter using a least square method from the correspondence obtained for each of the plurality of linear subspaces detected by the detection unit. The derivation means includes
Detecting means for detecting feature points from the image data generated from the light field data;
From the correspondence between feature points detected from image data generated from the one light field data serving as a reference and feature points detected from image data generated from the other light field data, the adjustment parameter The information processing apparatus according to claim 2, further comprising adjustment parameter deriving means for deriving. The derivation means further includes correspondence search means,
The detection means detects a plurality of feature points from image data generated from the one light field data serving as a reference,
The correspondence search means, for each of a plurality of feature points detected from image data generated from the one light field data serving as a reference, finds a feature point corresponding to the feature point from the other light field data. Search from the generated image data,
The adjustment parameter derivation unit derives the adjustment parameter based on a correspondence relationship obtained with each of the plurality of feature points detected by the detection unit and with the feature points searched by the correspondence search unit. The information processing apparatus described. The information processing apparatus according to claim 7, wherein the adjustment parameter derivation unit derives the adjustment parameter using a least square method from the correspondence obtained for each of the plurality of feature points detected by the detection unit. The information processing apparatus according to claim 1, wherein the adjustment by the adjustment unit is performed by two-dimensional linear conversion using the adjustment parameter. The information processing apparatus according to any one of claims 1 to 8, wherein the adjustment by the adjustment unit is performed by four-dimensional affine transformation using the adjustment parameter. The acquisition means has a microlens array including a plurality of microlenses, acquires light field data obtained by imaging with a plenoptic camera,
The light field data includes at least a position of a microlens through which a light beam passes and an angle at which the light beam enters the microlens, which are specified from the pixel position, for each pixel position. The information processing apparatus according to claim 9. An acquisition means for acquiring light field data;
Separating means for separating the light field data for each of a plurality of channels having different wavelengths;
Deriving means for deriving adjustment parameters for matching the positions and angles of the light rays recorded in each light field data from the correspondence relationship between the plurality of light field data obtained by separation. An information processing apparatus characterized by the above. An acquisition step for acquiring light field data;
A separation step of separating the light field data for each of a plurality of channels having different wavelengths;
A derivation step for deriving an adjustment parameter for matching the position and angle of the light beam recorded in each light field data from the correspondence between the plurality of light field data obtained by separation, and
An adjustment step of adjusting a position and an angle of a light beam recorded in at least one light field data among the plurality of light field data based on the adjustment parameter derived in the derivation step. Information processing method. An acquisition step for acquiring light field data;
A separation step of separating the light field data for each of a plurality of channels having different wavelengths;
A derivation step for deriving an adjustment parameter for making the position and angle of the light beam recorded in each light field data coincide with each other from the correspondence relationship between the plurality of light field data obtained by separation. An information processing method characterized by the above.   A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 12.

Images