JP2019208214A - Image processing apparatus, image processing method, program, and imaging apparatus - Google Patents

Abstract

To provide an image processing apparatus capable of preventing miscorrection of an image caused by blur in image processing of a refocused image.SOLUTION: The image processing apparatus (an imaging apparatus 100) includes: data acquisition means for acquiring light field data generated by imaging means having a pupil dividing function of an exit pupil of an imaging optical system forming an optical image of an object; image processing means for performing predetermined image processing of image data; image generation means for generating the image data subjected to the predetermined image processing by using the light field data; and control means for performing the predetermined image processing on the generated image data by controlling the image processing means to generate light field data corrected in accordance with the predetermined image processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that processes an image based on refocusable light field data and an imaging apparatus to which the image processing apparatus is applied.

  2. Description of the Related Art In recent years, imaging devices such as digital cameras having a function of performing image processing on a specific area of a photographed subject image are known. For example, Patent Document 1 discloses a technique capable of performing a color conversion process in which an area having color information specified by a user is similarly converted to a color specified by the user.

  By the way, there is known a technique capable of obtaining images in focus at different distances in the field of field by one shooting. In Patent Literature 2, image data (re-field data) that can be acquired at an arbitrary focal plane by synthesizing an imaging signal (light field data) obtained by imaging a light beam that has passed through different areas of the exit pupil of the imaging optical system with an imaging device. A technique for generating a (focus image) by one shooting is disclosed.

JP 2006-186594 A JP 2007-4471 A

  In the imaging apparatus described in Patent Literature 1, when blur occurs in a subject area having a color to be converted, an intended color conversion processing result may not be obtained due to color mixing that occurs in the blur area. . Such a problem is not limited to the above-described color conversion process, and the same problem occurs in a scene where image processing using information unique to a subject or different image processing is applied to each subject region.

  On the other hand, in the imaging apparatus described in Patent Document 2, when generating a plurality of refocused images in which the focus position is changed from the same scene and performing the same image processing on them uniformly, the image processing is performed every time the image is combined. It will be carried out, and the processing becomes complicated.

  Accordingly, an object of the present invention is to avoid erroneous correction of an image due to image processing caused by blur and to perform image processing efficiently when image processing is performed on an image such as a refocus image. An image processing apparatus is provided.

  In order to solve the above problems, according to the present invention, an image processing apparatus acquires light field data generated by an imaging unit having a pupil division function of an exit pupil of an imaging optical system that forms an optical image of a subject. Means, image processing means for performing predetermined image processing on image data, image generation means for generating image data to be subjected to predetermined image processing using light field data, and controlling the image processing means to generate Control means for generating light field data corrected according to the predetermined image processing by performing predetermined image processing on the processed image data.

  According to the present invention, in the image processing of the refocus image, it is possible to correct the light field data corresponding to the image processing, and it is possible to prevent erroneous correction of the image due to various image processing caused by blurring. An image processing apparatus can be provided.

1 is a block diagram illustrating a configuration of an imaging apparatus to which an image processing apparatus according to a first embodiment of the present invention is applied. The figure which shows the example of the arrangement structure of the pixel which has a color filter of an image pick-up element typically The figure which shows the structural example of the pixel which has a division | segmentation pixel. The figure which shows notionally the relationship between a division | segmentation pixel and the exit pupil area | region of an imaging optical system The figure for demonstrating the image composition by write field data The figure which shows the flowchart of imaging | photography operation | movement of the imaging device to which the image processing apparatus concerning 1st Example of this invention is applied. FIG. 3 is a block diagram showing a configuration of an image pickup apparatus to which an image processing apparatus according to a second embodiment of the present invention is applied. A diagram for explaining a parallax map The figure for demonstrating the synthesized image produced | generated according to a parallax map The figure which shows the flowchart of imaging | photography operation | movement of the imaging device to which the image processing apparatus concerning the 2nd Example of this invention is applied. The figure for demonstrating the relationship between an image sensor and shading shape

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  A first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is a block diagram showing a configuration of an image pickup apparatus 100 to which an image processing apparatus according to the first embodiment of the present invention is applied. In FIG. 1, reference numeral 101 denotes an image pickup optical system that guides light from an incident subject to the image pickup element 102 through a plurality of lens groups and a diaphragm (not shown). The light beam that has passed through the imaging optical system 101 forms an optical image on the image sensor 102. The image sensor 102 has a pixel array in which pixels having R (red), G (green), and B (blue) color filters are arranged according to a Bayer array, and each pixel includes at least a pair of photoelectric conversion units. Yes. Details of the image sensor 102 and the pixels will be described later.

  The A / D conversion unit 103 converts the analog image signal output from the image sensor 102 into a digital signal. The pupil division image generation unit 104 generates a pair of pupil division image data from the signal output from the A / D conversion unit 103. The detailed operation of the pupil division image generation unit 104 will also be described later. The system control unit 105 controls the entire imaging apparatus according to the state inside the imaging apparatus in operations such as shooting and recording. The D / A conversion unit 106 converts the digital image signal into an analog signal and outputs the analog signal to the display unit 107.

  The display unit 107 is a liquid crystal display such as an LCD, and displays image data stored in the EVF or the recording medium 111, various setting screens, and the like. The image composition unit 108 generates a composite image from a plurality of images. Detailed operation of the image composition unit 108 will be described later. The color conversion unit 109 performs color conversion processing for converting a predetermined color in the image into another predetermined color. Details of the color conversion unit 109 will be described later.

  The memory 110 is a memory for storing an operation control program inside the camera, images, calculation data, and various parameters, and stores in advance information such as parameters used for various image processing, lookup table information, and shading characteristics. . The recording medium 111 is a recording medium such as an SD card or CompactFlash, and records captured image data.

  Next, the configuration of the image sensor 102 in this embodiment will be described with reference to FIGS. FIG. 2 is an enlarged view of a part of the light receiving region of the image sensor 102. As shown in FIG. 2, pixels having any of RGB color filters 200 are arranged in the light receiving region of the image sensor 102 according to a Bayer array.

  Next, the pixel configuration of the image sensor 102 will be described with reference to FIG.

  FIG. 3A is a view of one of the pixels of the image sensor 102 observed from above. FIG. 3B is a cross-sectional view of the pixel along the line segment P-P ′ shown in FIG. In the figure, similar parts are denoted by the same reference numerals.

  In FIG. 3, reference numeral 301 denotes a microlens having a pupil division function, which condenses the light beam output from the imaging optical system. The color filter 200 transmits only light of a specific wavelength out of the light flux that has passed through the microlens 301. Reference numerals 302, 303, 304, and 305 denote photoelectric conversion units that can photoelectrically convert light beams received through the same microlens and individually output photoelectric conversion signals (pupil division). Note that when simply described as a pixel in this embodiment, it means a unit pixel that is a set unit of the above-described elements 301 to 305 and 200 that is configured under one microlens of the image sensor 102. Further, the photoelectric conversion unit corresponding to the position of the photoelectric conversion unit 302 in the pixel is divided into pixels A and 303, the photoelectric conversion unit corresponding to the position of 303 is divided into pixels B, and the photoelectric conversion unit corresponding to the position of 304 is divided into pixels C and 305. The photoelectric conversion units corresponding to the positions are denoted as divided pixels D, respectively.

  In this embodiment, an example in which four photoelectric conversion units are configured for one pixel will be described. However, the number of photoelectric conversion units configured in one pixel is not limited thereto. The photoelectric conversion unit configured in one pixel may be configured to be able to independently receive light beams that have passed through different regions of the exit pupil of the imaging optical system.

  In this embodiment, the configuration using the RGB Bayer array as the pixel array of the image sensor 102 has been described. However, the spectral sensitivity characteristic of the color filter is not limited to this. Further, in this embodiment, an example in which one color filter is configured for one microlens is shown, but a color filter having different spectral sensitivity characteristics may be configured for each divided pixel under the microlens.

Next, a detailed operation of image generation by the pupil division image generation unit 104 will be described.
The pupil divided image generation unit 104 acquires signals (referred to as light field data) obtained from the divided pixel A, divided pixel B, divided pixel C, and divided pixel D, respectively, and first signals are obtained from the signals obtained by the divided pixel A group. A second image signal is generated from an image signal (pupil divided image data) from a signal obtained by the divided pixel B group. In addition, a third image signal is generated from a signal obtained from the divided pixel group C, and a fourth image signal is generated from a signal obtained from the divided pixel group D.

  FIG. 4 is a diagram for explaining the relationship between the exit pupil of the imaging optical system 101 and the divided pixels. In the figure, the same parts as those in FIG. 3 are denoted by the same reference numerals. In addition, in order to simplify the description, the description will be given focusing on the divided pixel A and the divided pixel B. Therefore, the following description can also be applied to combinations of divided pixels having the same relationship as the positional relationship shown for divided pixels A and B.

  In this embodiment, the light output from the exit pupil 401 of the imaging optical system is pupil-divided by the microlens 301, the divided pixel A group 302 is a region 402, and the divided pixel B group 303 is a light beam that has passed through the region 403. Each is designed to receive light. Therefore, the first image signal and the second image signal become a pair of pupil division image data having parallax in the pupil division image direction. Each of the pair of pupil divided images has a depth of field that is two steps deeper than the F value of the imaging optical system 101.

Next, a detailed operation of the color conversion unit 109 will be described.
The color conversion unit 109 interpolates missing color information for each pixel position from the surrounding pixels in the input image, and generates color information Pix (R, G, B) corresponding to each pixel position. The color conversion unit 109 uses a lookup table to convert color information Pix (R, G, B) corresponding to each pixel position into a predetermined conversion target color Pix (R ′, G ′, B ′). The color conversion processing using the look-up table can be realized by using the technique disclosed in Patent Document 1, for example, although detailed description is omitted here.

  The technique disclosed in Patent Document 1 is color conversion processing in the YUV space, but the pixel value in the YUV space after color conversion is returned to the RGB space by performing inverse conversion processing by matrix operation. It is easy. The color conversion unit 109 performs color conversion processing using the lookup table on the entire image, and corrects the color to be converted in the image to the conversion target color. In this embodiment, the case where the conversion target color and the conversion target color are determined according to the shooting mode has been described. However, the user can directly specify the conversion target color and the conversion target color using an operation unit (not shown). It may be.

  In the present embodiment, an example in which image processing is performed on the entire image has been described, but the present invention is not limited to this. For example, when a shooting mode that performs image processing on a specific subject is set, the region where the specific subject exists in the image is estimated by image analysis or the like, and the image processing target region is limited. May be.

  Next, the detailed operation of the image composition unit 108 will be described with reference to FIG. The image synthesis unit 108 synthesizes the pupil-divided images and generates image data (refocus image) that can be acquired on an arbitrary focal plane (virtual imaging plane). FIG. 5 is a diagram for explaining the operation of the image composition processing. Here, for the sake of simplicity of explanation, the description will be made with attention paid to the divided pixel A and the divided pixel B. The following description can also be applied to combinations of divided pixels having the same relationship as the positional relationship shown for divided pixel A and divided pixel B.

  FIG. 5A shows the relationship between the incident light and the focal plane in a certain area of the image sensor 102. The light beam that has passed through the exit pupil regions 402 and 403 described with reference to FIG. Further, the divided pixels 501, 503, 505, 507, and 509 receive light beams that have passed through the exit pupil region 402, respectively.

  FIG. 5B is a diagram schematically showing the light flux received by each of the divided pixels in FIG. By adding the divided pixels 501 and 502, 503 and 504, 505 and 506, 507 and 508, 509 and 510, respectively, under the same microlens, pixel signals of the imaged image indicated by the same type of line segment are obtained. I can do it.

  FIG. 5D shows an example in which the signal group obtained by the light flux that has passed through the exit pupil region 403 is shifted by one pixel and added. Since each of the divided pixels has the light ray information shown in FIG. 5A, the signal obtained by the shift addition is a pixel signal of an imaged image indicated by different types of line segments. This can be treated as equivalent to a signal that can be acquired on the virtual imaging surface 1 as shown in FIG.

  FIG. 5F is an example in which the pixel signal group obtained by the light beam that has passed through the exit pupil region 403 is shifted by −1 pixel and added, and is treated as equivalent to a signal that can be acquired on the virtual imaging plane 2. I can do it.

In the present embodiment, the image composition processing unit 108 performs addition with a shift amount of 0, that is, adds the pixel values of a pair of divided pixels existing under the same microlens, thereby synthesizing a composite image equivalent to data captured by a conventional imaging device Is generated. An expression for generating the composite image ImgAB in the present embodiment is shown below. In Equation 1 below, ImgA and ImgB represent a pair of input images, and x and y represent horizontal and vertical coordinates.
ImgAB (x, y) = ImgA (x, y) + ImgB (x, y) Equation 1
The formula shown here is only an example, and the formula may be modified as appropriate according to the number and characteristics of the divided pixels to be configured.

  Next, with reference to FIG. 6, the photographing operation of the image pickup apparatus 100 to which the image processing apparatus according to this embodiment is applied will be described. FIG. 6 is a diagram illustrating a flowchart of the shooting operation of the imaging apparatus 100 to which the image processing apparatus according to the present embodiment is applied. This operation is realized by controlling each part of the imaging apparatus 100 by the system control unit 105 loading and executing an operation control program stored in the memory 110.

  The imaging apparatus 100 repeats the color conversion photographing process while a power switch (not shown) is ON (“Y” in step S600). When color conversion photographing is started (“Y” in step S600), first, an image signal of a subject generated by the image sensor 102 and the A / D conversion unit 103 is acquired in step S601. Next, in step S602, the pupil divided image generation unit 104 generates a pair of pupil divided image data from the image signals of the divided pixel A group and the divided pixel B group. The pupil division image data generated in step S602 is sequentially input to the color conversion processing unit 109.

  Next, in step S603, the first pupil division image data output in step S602 is based on the conversion target color and the conversion target color determined by the color conversion processing unit 109 according to the shooting mode. Color conversion processing is performed. Next, in step S604, the second pupil divided image data output in step S602 is based on the conversion target color and the conversion target color determined by the color conversion processing unit 109 according to the shooting mode. Color conversion processing is performed.

  Next, in step S605, the third pupil divided image data output in step S602 is based on the conversion target color and the conversion target color determined by the color conversion processing unit 109 according to the shooting mode. Color conversion processing is performed. In step S606, the fourth pupil divided image data output in step S602 is subjected to color conversion based on the conversion target color and the conversion target color determined by the color conversion processing unit 109 according to the shooting mode. Processing is performed.

  Next, in step S607, it is determined whether or not a shutter button (not shown) has been pressed. If the shutter button has been pressed, the process proceeds to step S608 to record a captured image. If the shutter button has not been pressed, the process proceeds to the display process in step S609.

  In step S608, the divided pixel unit data (light field data) included in the first pupil divided image data and the second pupil divided image data on which the color conversion processing has been performed is stored in the recording medium 111, and in step S609. Proceed to display processing. In step S609, image composition processing is performed by the image composition processing unit 108, and the output composite image is displayed on the display unit 107. When the display process in step S609 ends, the process returns to step S600. If the power switch is turned on, an image signal of the next frame is acquired in step S601, and the above-described series of operations is repeatedly executed.

  In this embodiment, the color conversion processing for converting the color to be converted into the conversion target color is used as an example of the image processing. However, the scope of the present invention is not limited to this, and information specific to the subject is used. Any image processing can be applied as long as it uses the image processing.

  In this embodiment, the correction data for each divided pixel is recorded on the recording medium as it is. However, the pupil divided image after color conversion is synthesized by the image synthesis unit 108 and the synthesized image is recorded. good.

  In this embodiment, an example in which a pair of pupil-divided images is sequentially processed using a single color conversion unit has been described. However, a second color conversion unit is configured for the imaging apparatus 100 to perform color conversion processing. By proceeding in parallel, the processing time for color conversion can be shortened. Further, a larger number of color conversion means may be configured according to the number of pupil divided images.

  As described above, if image processing is performed on each of the pupil-divided image data generated by receiving light flux groups that have passed through different regions on the exit pupil, the problem of image processing due to blurring will be caused. It can be solved.

  Next, a second embodiment of the present invention will be described with reference to FIGS.

  FIG. 7 is a block diagram showing a configuration of an imaging apparatus 700 to which the image processing apparatus according to the second embodiment of the present invention is applied. In the figure, the same parts as those in FIG. 1 are denoted by the same reference numerals.

  In FIG. 1, the imaging apparatus 700 is the same as the imaging apparatus 100 of the embodiment shown in FIG. 1 except that an inverse conversion unit 701, a parallax detection unit 702, and a synthesis parameter determination unit 703 are added. In the present embodiment, the light that has been corrected through the color conversion processing by the color conversion section 109 in the imaging apparatus of the present embodiment, and the added inverse conversion section 701, parallax detection section 702, and synthesis parameter determination section 703. Only the description of whether field data can be obtained will be given.

  First, the parallax detection unit 702 will be described. The parallax detection unit 702 detects the amount of parallax from the pair of input pupil divided image data (image signals). Correlation between detection areas set while shifting the reference image using one of the pair of pupil-divided images input to the parallax detection unit 702 as a standard image and the other as a reference image is obtained. Although a detailed description of the method for obtaining the correlation will not be given, for example, a known technique such as SAD or SSD which is a pattern matching (template matching) technique may be used.

  By repeating the calculation while shifting the calculation for obtaining the correlation in units of pixels in the parallax direction, it is possible to obtain the shift amount up to the position with the highest correlation for each pixel. A parallax map in which the parallax amount for the same subject is converted into a pixel value is generated using the shift amount up to the position with the highest correlation.

  FIG. 8 is a diagram illustrating a parallax map. Here, for the sake of simplicity of explanation, the description will be made with attention paid to the divided pixel A and the divided pixel B. The following description can also be applied to combinations of divided pixels having the same relationship as the positional relationship shown in divided pixel A and divided pixel B.

  FIG. 8A is a pupil divided image acquired by the divided pixel A group, and is an image handled as a reference image in the parallax detection unit 702. FIG. 8B is a pupil divided image acquired by the divided pixel group B, and is an image handled as a reference image in the parallax detection unit 702. Reference numerals 801 and 802 denote subjects, and the subject 801 exists closer to the imaging device than the subject 802. FIG. 8C is a parallax map generated from the standard image and the reference image, and the parallax is mapped as a pixel value. The parallax amount of the subject 801 is 0, and the parallax amount of the subject 802 is 100.

Next, the operation of the synthesis parameter determination unit 703 will be described.
An image that is in focus on a predetermined subject can be obtained by performing image synthesis processing described later on each of the possible values of the parallax amount. However, it takes time to perform image composition processing for all possible amounts of parallax, which is not practical. Therefore, the synthesis parameter determination unit 703 determines parameters for efficiently generating a composite image based on the distribution of the amount of parallax in the parallax map.

First, the synthesis parameter determination unit 703 obtains the distribution of the parallax amount by taking statistics of the pixel values in the parallax map (ParallaxMap). Then, the amount of parallax (pixel value) appearing in the distribution is determined. For example, when the pixel value of an area with a parallax map is as shown in the following ParallaxMap, the parallax amount (Param1) that appears in the distribution is 50, 10, 20, 100.
ParallaxMap = {50,50,50,10,10,50,50,20,20,20,100}
Param1 = {50,10,20,100}

Next, the synthesis parameter determination unit 703 rearranges the parallax amount (Param1) appearing in the distribution according to a predetermined condition, and determines the priority.
For example, rearrangement is performed in ascending order of the absolute value of the parallax amount to determine the priority.
Param2 = {10,20,50,100}

Next, parameters are extracted in order of priority for the number of times of predetermined synthesis processing. Here, assuming that color conversion processing can be performed three times per frame, the parameter Param3 in consideration of the priority and the processing time is as follows.
Param3 = {10,20,50}

  The synthesis parameter (Param3) determined in this way is a synthesis parameter capable of preferentially processing a subject close to the focal plane at the time of shooting. Param3 is effective, for example, when a minute parallax is generated in the subject area when image processing is performed on the subject that is in focus at the time of shooting.

Further, as another priority determination condition, the priorities may be determined by rearranging in descending order of frequency of appearance in the distribution.
Param4 = {50,20,10,100}

When parameters are extracted in order of priority by the number of times of synthesis processing determined in advance for Param4, the result is as follows.
Param5 = {50,20,10}

  The synthesis parameter (Param5) determined in this way is a synthesis parameter with a higher priority for the subject occupying the largest area in the composition at the time of shooting. Param5 is effective when image processing is performed on the entire screen because, for example, when there are a plurality of subjects with different distances in the object field, processing is performed preferentially from the highest proportion in the captured image. is there.

  Further, as a method for determining the shift amount other than the distribution of the parallax amount, the depth of field is obtained according to the lens information (F value, focal length) that is the shooting parameter, and the shift amount is determined based on the depth of field. You may do it.

  In the case where a discrete shift amount is determined in advance, if there is a pixel having a disparity amount between discretely determined parallaxes, a result obtained by image processing described later may be subjected to linear interpolation processing. .

  As a method for determining the shift amount other than the distribution of the parallax amount, an arbitrary shift amount may be determined using a technique for recognizing the subject such as image analysis or person detection according to the shooting mode. For example, in the shooting mode in which correction is performed for a person, a person area may be extracted from a reference image by image analysis, and the amount of parallax corresponding to the person area may be used as a composition shift amount. Also, the shift amount and priority may be arbitrarily designated by the user through an operation unit (not shown).

  The method for determining the synthesis parameter has been described above. In any case, it is only necessary to determine a synthesis parameter that is in focus on the subject to be subjected to color conversion processing in the later-described synthesis image generation.

Next, the operation of the image composition unit 108 in this embodiment will be described.
In the present embodiment, the image composition unit 108 generates a composite image based on the pair of pupil divided images and the shift amount (virtual imaging plane) determined by the composite parameter determination unit 703. Here, for the sake of simplicity of explanation, the description will be made with attention paid to the divided pixel A and the divided pixel B. The following description can also be applied to combinations of divided pixels having the same relationship as the positional relationship shown for divided pixel A and divided pixel B.

An expression for synthesizing pixel values of each coordinate of the composite image ImgAB is shown below.
ImgAB (x, y) = ImgA (x, y) + ImgB (x + shift, y) Equation 2

  ImgA and ImgB are the base image before color conversion and the reference image before color conversion, x and y are pixel coordinates in the horizontal and vertical directions, respectively, and the shift amount determined by the synthesis parameter determination unit 703 is Shift Yes. That is, the composite image is an image obtained by adding and synthesizing a pair of pupil-divided images by shifting the shift pixels in the parallax direction.

  FIG. 9 shows a composite image synthesized with a shift amount 0 and a composite image synthesized with a shift amount 100 with respect to the pair of pupil-divided images shown in FIG. As a synthesized image synthesized with the shift amount 0, an image focused on the subject 901 can be obtained, and as an image synthesized with the shift amount 100, an image focused on the subject 902 can be obtained. That is, in order to synthesize an image focused on an arbitrary area on the parallax map, it is only necessary to shift (combine) and add by the pixel value of the arbitrary area on the parallax map. Note that Expression 2 shown here is merely an example, and the expression may be modified as appropriate according to the number of divided pixels and characteristics.

  Next, the operation of the inverse transform unit 701 will be described. Here, for the sake of simplicity of explanation, the description will be made with attention paid to the divided pixel A and the divided pixel B. The following description can also be applied to combinations of divided pixels having the same relationship as the positional relationship shown in divided pixel A and divided pixel B.

The inverse conversion unit 701 corrects the pixel values (light field data) of the divided pixels based on the composite image that has been subjected to the color conversion processing by the color conversion processing unit. In an area where the correlation is high, that is, an in-focus area, the following Expression 3 is established.
ImgA (x, y) ≒ ImgB (x + shift, y) ・ ・ ・ Equation 3

From Equation 2 and Equation 3, the following Equation 4 and Equation 5 hold.
ImgAB (x, y) ≒ ImgA (x, y) + ImgA (x, y) Equation 4
ImgAB (x, y) ≒ ImgB (x + shift, y) + ImgB (x + shift, y)

Equations 4 and 5 can be modified to derive pixel values of the divided pixel value ImgB (x + shift, y) and the divided pixel ImgA (x, y).
ImgB (x + shift, y) ≒ ImgAB (x, y) / 2 Equation 6
ImgA (x, y) ≒ ImgAB (x, y) / 2 Equation 7

  The pixel values of the respective divided pixels thus obtained are overwritten on ImgA (x, y) and ImgB (x + shift, y).

  Here, for the sake of explanation, an example in which the inverse transformation process is performed using a simple expression is shown, but the pupil-divided image has different shapes of shading depending on the position of the image sensor as shown in FIG.

  Therefore, the shading characteristic of each divided pixel configured in the image sensor 102 is held in the memory 110 in advance, and gain is obtained at a ratio according to the shading characteristic according to the coordinates of each divided pixel calculated by Expression 6 and Expression 7. To hang. Then, ImgA (x, y) and ImgB (x + shift, y) are more natural correction results.

  In addition, the above shows an example in which the value of each divided pixel is obtained by inversely transforming the composite image that has undergone color conversion. However, since the parallax between the pixels has already been determined, another method may be used. Can be realized. For example, color conversion may be performed only on the base image, and the pixel value of the base image after color conversion may be overwritten on the pixel corresponding to the parallax amount of the reference image.

  Alternatively, the same color conversion table may be called up in association with the coordinates of the reference image and the coordinates of the reference image shifted by the corresponding amount of parallax.

  Further, the equation may be modified as appropriate according to the number of divided pixels configured in pixel units and other characteristics.

  Next, with reference to FIG. 10, the shooting operation of the imaging apparatus 700 will be described. FIG. 10 is a flowchart illustrating the photographing operation of the imaging apparatus to which the image processing apparatus according to the present embodiment is applied. This operation is realized by the system control unit 105 controlling each unit of the imaging apparatus 100 by loading and executing a program stored in the memory 110. In FIG. 10, parts that perform the same operations as those in the first embodiment of the present invention are denoted by the same reference numerals.

  The imaging apparatus 700 repeats the color conversion imaging process while a power switch (not shown) is ON (“Y” in step S600). Steps S600 to S602 are the same processes as those in the first embodiment, and thus description thereof is omitted here.

  In step S1003, the parallax amount of the pair of pupil divided images input by the parallax detection unit 702 is detected, and a parallax map is generated. In step S1004, the parallax map acquired in step S1003 is input to the synthesis parameter determination unit 703, and parameters for image synthesis are determined based on the parallax map as described above. In step S1005, the image composition unit 108 performs image composition processing based on the image composition parameters and the pair of pupil-divided images.

  Next, in step S1006, the color conversion process according to the shooting mode is performed on the composite image generated in step S1005 by the color conversion unit 109, and the composite image data subjected to the color conversion process is sent to the inverse conversion unit 701. Is output.

  In step S1007, the inverse conversion unit 701 performs an inverse conversion process based on the color-converted composite image and the parallax map, and the divided pixel data (light field data) of the in-focus area in the composite image. The pixel value of is corrected.

Next, in step S1008, it is determined whether or not the processing has been completed for each of the shift amounts determined by the composite image parameter determined in step S1004. If an unprocessed synthesis parameter (shift amount) remains, the process returns to step S1005 to perform image synthesis processing according to the unprocessed synthesis parameter. On the other hand, if the processing for all the synthesis parameters (shift amounts) has been completed, the process proceeds to step S607.
Since the processing after step S607 is the same as that of the first embodiment, description thereof is omitted here.

  As described above, inverse conversion is performed such that color conversion processing is performed on the composite image, and pixel values after color conversion are converted into pixel values in divided pixel units at the in-focus position of the composite image. Processing is performed, and writing is performed back to the corresponding divided pixel position. As a result, the color conversion processing result can be reflected on the divided pixels. That is, if the image processing result for the subject in focus is appropriately returned to the original divided pixel position, the light field data correction processing for each subject can be realized.

  According to the present invention described above, in the image processing of the refocus image, it is possible to correct the light field data corresponding to the image processing, and it is possible to prevent erroneous correction of various image processing due to blurring. An image processing apparatus can be provided. The above embodiment is an example in which the present invention is applied to an imaging apparatus. However, the present invention is not limited to this, and a multi-lens camera, a PC, or the like that captures subject images having parallax with a plurality of imaging elements. It goes without saying that the present invention can also be applied to other information processing apparatuses.

  Moreover, the processing of the above-described embodiment may provide a system or apparatus with a storage medium that records a program code of software that embodies each function. The functions of the above-described embodiments can be realized by the computer (or CPU or MPU) of the system or apparatus reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying such a program code, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, or the like can be used. Alternatively, a CD-ROM, CD-R, magnetic tape, nonvolatile memory card, ROM, or the like can be used.

  The functions of the above-described embodiments are not only realized by executing the program code read by the computer. Including the case where the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. It is.

  Furthermore, the program code read from the storage medium may be written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Is also included.

Claims (1)

In the image processing apparatus,
An acquisition unit that acquires light field data generated by an imaging unit having a pupil division function of an exit pupil of an imaging optical system that forms an optical image of a subject;
Image processing means for performing predetermined image processing on image data;
Image generation means for generating image data to be subjected to the predetermined image processing using the light field data;
Control means for generating the light field data corrected according to the predetermined image processing by controlling the image processing means and performing the predetermined image processing on the generated image data;
An image processing apparatus comprising:

Images