JP2017108243A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device and an image processing method, capable of successfully estimating a point spread function and the like.SOLUTION: An image processing device comprises: a first region detecting unit which detects a first region from an image; a second region detecting unit which detects a second region similar to the first region from the image; an amount-of-characteristics calculating unit which calculates the amount of characteristics in at least either of the first region and the second region; a region selecting unit which selects the first region or the second region on the basis of the amount of characteristics calculated by the amount-of-characteristics calculating unit; and a point spread function estimating unit which estimates a point spread function from the first region or the second region selected by the region selecting unit.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus and an image processing method.

  There has been proposed a technique for correcting image quality degradation caused by a focus error or camera shake using a point spread function (PSF) representing blur or blur in an image.

  For example, Patent Document 1 discloses the following technique for correcting image blur and blur using a technique called blind deconvolution. Specifically, first, the main subject region is determined from the captured image. The main subject area may be selected by the user, or may be automatically selected using the sharpness of the image. Next, the PSF is estimated from the main subject area using the blind deconvolution technique. Then, the estimated PSF and the input image are deconvoluted to correct image blur and blur.

  Patent Document 2 discloses a technique for dividing an image into regions, estimating PSF and calculating PSF reliability in each region, and improving the accuracy of PSF based on these. . Specifically, the reliability is calculated for each region based on the number of saturated pixels and gradient information, and the accuracy of the PSF is compensated for by using the PSF estimated from the surrounding region for regions with low reliability. To improve. Note that the degree of reliability is calculated based on the number of saturated pixels and gradient information. When there are many saturated pixels or when there is little gradient information, there is a high possibility that the subject is a flat subject, which is necessary for PSF estimation. This is because sufficient information cannot be acquired sufficiently.

JP 2007-306548 A JP 2012-155456 A

G.R.Ayers and J.C.Dainty, “Iterative blind deconvolution method and its applications,” Optics Letters, vol. 13 (7), pp, 547-549, july 1988. Xu L, Jia J (2010) Two-phase kernel estimation for robust motion deblurring. In: Proceedings of European Conference on Computer Vision, pp 157-170.

  However, with the conventional technology, there has been a case where correction of blur / blur cannot always be performed satisfactorily.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of satisfactorily estimating a point spread function and the like.

  According to one aspect of the present invention, a first region detection unit that detects a first region from an image, and a second region detection unit that detects a second region similar to the first region from the image. A feature amount calculating unit that calculates a feature amount in at least one of the first region and the second region, and the first amount based on the feature amount calculated by the feature amount calculating unit. A region selection unit that selects the region or the second region, and a point spread function estimation unit that estimates a point spread function from the first region or the second region selected by the region selection unit An image processing apparatus is provided.

  According to another aspect of the present invention, a first region detection unit that detects a first region from an image, and a second region that detects a second region similar to the first region from the image. An area detector, a feature quantity calculator for calculating a feature quantity in at least one of the first area and the second area, and a first point spread function is estimated from the first area. A point spread function estimating unit that estimates a second point spread function from the second region, and a composition ratio determining unit that determines a composition ratio based on the feature amount calculated by the feature amount calculating unit, Combining the first point spread function and the second point spread function based on the combination ratio; or information based on the first point spread function and information based on the second point spread function; And a synthesizing unit for synthesizing based on the synthesis ratio The image processing apparatus is provided, characterized in that.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing method which can estimate a point spread function etc. favorably can be provided.

1 is a block diagram illustrating a configuration of an image processing device according to a first embodiment of the present invention. It is a block diagram which shows the structure of the image processing part of the image processing apparatus by 1st Embodiment of this invention. It is a block diagram which shows the structure of the blurring / blur correction | amendment part of the image processing apparatus by 1st Embodiment of this invention. 4 is a flowchart illustrating blur / blur correction processing performed in the image processing apparatus according to the first embodiment of the present invention. It is a block diagram which shows the structure of the feature-value calculation part of the image processing apparatus by 1st Embodiment of this invention. It is a graph which shows the relationship between the ratio of a saturated pixel, and a gain. It is a block diagram which shows the structure of the blurring / blur correction | amendment part of the image processing apparatus by 2nd Embodiment of this invention. It is a flowchart which shows the blurring-blur correction process performed in the image processing apparatus by 2nd Embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
An image processing apparatus and an image processing method according to a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the present embodiment. The image processing apparatus 111 according to the present embodiment is an imaging apparatus, for example. More specifically, the image processing apparatus 111 according to the present embodiment is an electronic camera, that is, a digital camera. Note that the image processing apparatus 111 is not limited to an imaging apparatus, and may be an electronic device that does not have an imaging function.

  The image processing apparatus 111 according to this embodiment includes an optical system 101, an image sensor 102, an A / D conversion unit 103, an image processing unit 104, a recording unit 105, a control unit 106, a distance measuring unit 107, an image memory 108, and an operation unit 109. And a display unit 110. The optical system 101 includes a focus lens, a diaphragm, a shutter, and the like.

  The optical system 101 focuses a subject by driving a focus lens (not shown), and adjusts an exposure amount by controlling a diaphragm (not shown) and a shutter (not shown). The optical system 101 may be detachable from the image processing apparatus 111 or may be detachable. The imaging element 102 is for converting an optical image of a subject formed in the optical system 101 into an electrical signal by photoelectric conversion, and is, for example, a CCD image sensor or a CMOS image sensor. The A / D conversion unit 103 is for digitizing an analog image signal output from the image sensor 102.

  The image processing unit 104 performs synchronization processing, white balance correction processing, color conversion processing, edge enhancement processing, gamma processing, blurring / blurring correction processing, and the like on the image signal digitized by the A / D conversion unit 103. Apply. The image processing unit 104 also executes processing for detecting the main subject, motion vector detection processing, and the like. An image signal (image, image information) that has been subjected to predetermined processing by the image processing unit 104 is output to the recording unit 105. The recording unit 105 converts the image output from the image processing unit 104 into an image of a predetermined format such as a JPEG format, and records the image converted into the predetermined format on a recording medium.

  The control unit 106 governs the overall operation of the image processing apparatus 111 according to the present embodiment. For example, the control unit 106 calculates a predetermined evaluation value based on the photographed image, and determines parameters for image processing performed by the image processing unit 104. A distance measuring unit (distance information acquiring unit) 107 acquires distance information to the subject and generates a distance map. A distance map is a two-dimensional representation of distance information to a subject in a captured image in pixel units.

  The image memory 108 is for storing information necessary for image processing in the image processing unit 104. The operation unit 109 is for a user to give an operation instruction to the image processing apparatus 111. The display unit 110 is, for example, a liquid crystal display or the like installed on the back surface of the image processing apparatus 111, and displays a screen for assisting operations during shooting, an image stored in the recording unit 105, and the like. Is.

  Next, a detailed description of the image processing unit 104 will be given with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the image processing unit 104 according to the present embodiment.

  The image processing unit 104 includes a synchronization processing unit 201, a white balance correction processing unit 202, a luminance / color signal generation unit 203, a color conversion processing unit 204, a contour enhancement processing unit 205, a color gamma processing unit 206, and a luminance gamma processing unit 207. , And a color difference signal generation unit 208. The image processing unit 104 further includes a blur / blur correction unit 209, a main subject detection unit 210, and a motion vector detection unit 211.

  The digital image signal output from the A / D conversion unit 103 is input to the synchronization processing unit 201 provided in the image processing unit 104. The synchronization processing unit 201 performs a synchronization process that is a process of calculating the color of each point by interpolating a spatial shift of the color signal associated with the color filter array. The synchronization processing unit 201 performs synchronization processing on the Bayer RGB image data output from the A / D conversion unit 103 to generate color signals R, G, and B. The color signals R, G, and B generated by the synchronization processing unit 201 are input to the white balance correction processing unit 202. The white balance correction processing unit 202 adjusts the white balance by applying a gain to the color signals R, G, and B based on the white balance gain value calculated by the control unit 106. The color signals R, G, and B whose white balance has been adjusted are output to the luminance / color signal generation unit 203. The luminance / color signal generation unit 203 generates a luminance signal Y from the color signals R, G, and B output from the white balance correction processing unit 202, and outputs the generated luminance signal Y to the contour enhancement processing unit 205. Further, the luminance / color signal generation unit 203 outputs the color signals R, G, and B to the color conversion processing unit 204.

  The color conversion processing unit 204 performs color conversion by matrix calculation or the like, and obtains color signals R, G, and B having a desired color balance. The color gamma processing unit 206 performs gamma correction on the color signals R, G, and B. The color difference signal generation unit 208 generates color difference signals RY and BY based on the color signals R, G, and B. The color difference signals RY and BY generated by the color difference signal generation unit 208 are output to the blur / blur correction unit 209.

  The contour enhancement processing unit 205 performs contour enhancement processing on the luminance signal Y and outputs the luminance signal Y subjected to the contour enhancement processing to the luminance gamma processing unit 207. The luminance gamma processing unit 207 performs gamma correction on the luminance signal Y, and the luminance signal Y subjected to the gamma correction is converted into a blur / blur correction unit 209, a main subject detection unit 210, a motion vector detection unit 211, and Output to the image memory 108.

  Based on the luminance signal Y, the main subject detection unit 210 detects the position and size of the main subject that is the main subject. Examples of the main subject include a human face. The main subject detection unit 210 detects the position and size of the face by a known method. The main subject is not limited to a person's face, and may be a person's torso, for example. The main subject is not limited to a person, and may be an animal other than a person or an object.

  The motion vector detection unit (motion information detection unit) 211 detects a motion vector based on the luminance signal Y output from the luminance gamma processing unit 207 and the luminance signal Y of the image taken immediately before. Note that the image taken immediately before is recorded in the image memory 108. As a motion vector detection method, a known technique can be used. Specifically, the image is divided into blocks of a predetermined size, and template matching is performed within a predetermined range for each block. Template matching refers to finding an image area that matches a small part of an image area called a template. A motion vector in each block can be obtained based on the coordinates of the template and the coordinates of the image area matched with the template.

  Note that the blur / blur correction unit 209 applies blur / blur to the color difference signals RY and BY output from the color difference signal generation unit 208 and the luminance signal Y output from the luminance gamma processing unit 207. A correction process is performed. The details of the blur / blur correction process performed in the blur / blur correction unit 209 will be described later. The image signal that has been subjected to the blur / blur correction process by the blur / blur correction unit 209 is output to the recording unit 105 and recorded in the recording unit 105.

  Next, blur / blur correction processing performed in the image processing apparatus 111 according to the present embodiment will be described in detail with reference to FIGS. 3 and 4.

  FIG. 3 is a block diagram illustrating a configuration of the blur / blur correction unit 209. As shown in FIG. 3, the blur / blur correction unit 209 includes an RGB signal conversion unit 301, a similar region detection unit 302, a first local region extraction unit 303, a second local region extraction unit 304, and a first feature amount. A calculation unit 305 and a second feature amount calculation unit 306 are provided. The blur / blur correction unit 209 further includes an area determination unit 307, an area selection unit 308, a PSF estimation unit 309, a correction processing unit 310, and a luminance / color difference signal generation unit 311.

  FIG. 4 is a flowchart showing a blur / blur correction process performed in the image processing apparatus 111 according to the present embodiment.

  In step S <b> 401, the similar region detection unit 302 detects the position and size of the main subject region detected by the main subject detection unit 210, the motion vector for each block detected by the motion vector detection unit 211, and the distance measurement unit 107. Get the generated distance map. In step S401, the similar region detection unit 302 detects a similar region in which blur / blur similar to the blur / blur generated in the main subject occurs.

  The similar area is detected for the following reason. That is, for example, when the image of the main subject area includes many saturated pixels, there is a high possibility that the image is a flat area, and there is a high possibility that information necessary for PSF estimation cannot be acquired sufficiently. On the other hand, a good PSF may be obtained from a similar region. When a good PSF can be obtained from a similar region, it is possible to correct blur and blur well using the PSF obtained from the similar region. For this reason, a similar region is detected.

  In order to obtain a PSF that can satisfactorily correct blur and blurring that occurs in the main subject, blurring similar to that occurring in the main subject has occurred, and the same blurring as that occurring in the main subject has occurred. It is preferable to detect a region where the out-of-focus is generated as a similar region. For this reason, at least the following two processes are performed in the similar area detection process. One is a process of detecting a region having a blur similar to the blur generated in the main subject. Such processing can be performed based on, for example, a motion vector for each block. The other is a process for detecting a region where a focal blur similar to the focal blur occurring in the main subject has occurred. Such processing can be performed based on a distance map, for example. Based on the results of these processes, a blur similar to the blur generated in the main subject is generated, and a similar blur is a region in which the same blur is generated as the focal blur generated in the main subject. A region is detected.

  Processing for detecting a region where a blur similar to the blur occurring in the main subject has occurred will be described in more detail. It is considered that the same blur occurs in the region where the same movement occurs within the unit time. It can be determined based on the motion vector whether or not the same motion has occurred within the unit time. For this reason, by detecting a motion vector region similar to the motion vector (motion information) in the main subject region, it is possible to detect a region where a blur similar to the blur occurring in the main subject has occurred. . Note that whether or not the motion vectors are similar can be determined based on the direction and magnitude of the motion vector. Thus, a motion information area close to the motion information in the main subject is detected.

  In addition, a process for detecting a region where a focal blur similar to the focal blur occurring in the main subject is detected will be described in more detail. It is considered that the same out-of-focus blur occurs in the subjects having the same distance from the imaging device. Whether or not the distance from the imaging device is the same can be determined based on the distance map. For this reason, by using the distance map, it is possible to detect a region where a focal blur similar to the focal blur occurring in the main subject has occurred. For example, the distance map is subjected to block integration in a block of a predetermined size, and a block area located at a distance equivalent to the distance to the main subject area is detected based on the result of block integration. In this way, it is possible to detect a region where a focal blur similar to the focal blur occurring in the main subject has occurred.

  In this way, the similar region detection unit 302 detects a region where blurring similar to the blurring occurring in the main subject has occurred and the same focal blurring as that occurring in the main subject is occurring. It can be detected as a similar region. Then, the similar region detection unit 302 outputs the detected position and size of the similar region to the second local region extraction unit 304.

  By the way, even if the distance from the imaging device is the same, if a region where the optical characteristics such as chromatic aberration of magnification are significantly different is used as a similar region, blur and blur in the main subject region can be corrected well. It is also conceivable that PSF cannot be obtained. For this reason, it is preferable to detect an area having optical characteristics equivalent to the optical characteristics in the main subject area as a similar area. Whether or not the optical characteristics are equivalent can be determined using image height information, for example. If a blur similar to the blur on the main subject is generated, a focal blur similar to the focus blur on the main subject is generated, and an area having optical characteristics equivalent to the main subject area is detected as a similar area, It is possible to obtain a PSF that can correct blurring and blurring of the subject area more favorably. As an area having the same optical characteristics as the main subject, an area having an image height position similar to that of the main subject area can be cited. An area having an image height lower than that of the main subject area may be detected as a similar area. This is because the optical characteristics deteriorate as the image height increases, so it is preferable to use a region with a low image height rather than a region with a high image height.

  In step S402, based on the position and size of the main subject detected by the main subject detection unit 210 and the luminance signal Y output from the luminance gamma processing unit 207, the first local region extraction unit 303 performs the main subject region. To extract. The first local region extraction unit 303 outputs the extracted luminance signal Y of the main subject region to the first feature amount calculation unit 305 and the region selection unit 308.

  In step S <b> 403, the second local region extraction unit 304 determines the similar region based on the position and size of the similar region detected by the similar region detection unit 302 and the luminance signal Y output from the luminance gamma processing unit 207. To extract. The second local region extraction unit 304 outputs the extracted luminance signal Y of the similar region to the second feature amount calculation unit 306 and the region selection unit 308.

  In step S404, the feature amount in the luminance signal Y of the main subject region output from the first local region extraction unit 303 is calculated by the first feature amount calculation unit 305, and the calculated feature amount is sent to the region determination unit 307. Is output. This feature amount is used to determine whether or not the luminance signal Y of the main subject area, that is, the input image is suitable for PSF estimation. The feature amount is calculated based on the edge strength of the input image and the number of saturated pixels in the input image.

  Here, the first feature amount calculation unit 305 will be described with reference to FIG. FIG. 5 is a block diagram illustrating a configuration of the first feature amount calculation unit 305. The configuration of the second feature quantity calculation unit 306 is the same as that of the first feature quantity calculation unit 305 shown in FIG.

  In the first feature quantity calculation unit 305, first, the horizontal edge strength in the input image and the vertical edge strength in the input image are obtained. Specifically, the horizontal band-pass filter unit 501 performs filtering on the input image, and the first absolute value calculation unit 503 calculates the edge strength. Based on the edge strength in the horizontal direction of all pixels in the input image, the average value of edge strength in the horizontal direction (edge strength average value) is calculated by the first average value calculation unit 505. In addition, the vertical band-pass filter unit 502 performs filtering on the input image, and the second absolute value calculation unit 504 calculates the edge strength. Based on the edge strength in the vertical direction of all pixels of the input image, the average value of the edge strength in the vertical direction is obtained by the second average value calculation unit 506. The smaller one of the edge strength average value in the horizontal direction and the edge strength average value in the vertical direction is selected by the minimum value selection unit 507, and the minimum value of the selected edge strength average value is output from the minimum value selection unit 507. The In this way, edge information that is a value corresponding to the edge strength is obtained. When the input image itself has few edges, or when there are few edges in the horizontal or vertical direction, the signal output from the minimum value selection unit 507 is small.

  On the other hand, the saturated pixel ratio detection unit 508 calculates the ratio of saturated pixels in the input image, that is, the saturated pixel ratio. Whether or not the pixel is a saturated pixel is determined by whether or not the pixel value exceeds a predetermined threshold value. When the ratio of saturated pixels in the input image is 100%, the saturated pixel ratio is 1, and when the ratio of saturated pixels in the input image is 0%, the saturated pixel ratio is 0. Therefore, the value range of the saturated pixel ratio is 0-1. Based on the saturated pixel ratio obtained by the saturated pixel ratio detection unit 508, a gain is calculated by a gain calculation unit (gain setting unit) 509. The gain range is 0-1. Multiplier 510 multiplies the value output from minimum value selector 507 by the gain calculated by gain calculator 509. The multiplication value thus obtained, that is, the feature amount is output from the first feature amount calculation unit 305 to the region determination unit 307.

  Note that the minimum value selection unit 507 selects the smaller one of the average edge strength value in the horizontal direction and the average edge strength value in the vertical direction instead of the larger one in the region that is not suitable for acquiring the PSF. This is to more strictly prevent the image from being used for acquiring the PSF.

  FIG. 6 is a graph showing the relationship between the saturated pixel ratio and the gain. FIG. 6A is a graph illustrating an example of the relationship between the saturated pixel ratio and the gain set by the gain calculation unit 509. FIG. 6A shows that when the ratio of saturated pixels is 0, the gain is 1, and when the ratio of saturated pixels is 1, the gain is 0. As the ratio of saturated pixels increases, the gain increases linearly. The case where it falls is shown. As the saturated pixel ratio increases, the gain decreases and the feature amount output from the first feature amount calculation unit 305 decreases. If the image of the main subject area is not suitable for PSF estimation, the feature quantity output from the first feature quantity calculation unit 305 is sufficiently small. Is not selected. Therefore, when the image of the main subject area is not suitable for PSF estimation, the image of the main subject area is not used for PSF estimation.

  The relationship between the saturated pixel ratio and the gain is not limited to the relationship shown in FIG. For example, the relationship between the ratio of saturated pixels and the gain may be set as shown in FIG. FIG. 6B is a graph showing another example of the relationship between the ratio of saturated pixels and the gain. As shown in FIG. 6B, the gain is set to 1 when the ratio of saturated pixels is equal to or less than the threshold Th1. On the other hand, when the ratio of saturated pixels is equal to or greater than the threshold Th2, the gain is set to zero. When the saturated pixel ratio is in the range from the threshold Th1 to the threshold Th2, the gain decreases linearly as the saturated pixel ratio increases. When the ratio of saturated pixels in the input image is equal to or less than Th1, it is considered that the PSF can be acquired favorably from the input image. Therefore, in this case, there is no particular problem even if the gain is set to 1 and the input image can be easily used for PSF estimation. Therefore, when the ratio of saturated pixels is equal to or less than the threshold Th1, the gain is set to 1. On the other hand, when the ratio of saturated pixels in the input image is equal to or greater than the threshold Th2, it is considered that the PSF cannot be estimated well from the input image. Therefore, in this case, it is preferable to set the gain to 0 so that the input image is not easily used for PSF estimation. Therefore, when the ratio of saturated pixels is equal to or greater than the threshold value Th2, the gain is set to zero.

  In step S405, the feature amount in the image of the similar region output from the second local region extraction unit 304 is calculated by the second feature amount calculation unit 306 in the same manner as the first feature amount calculation unit 305. The calculated feature amount is output to the region determination unit 307. As described above, the configuration of the second feature quantity calculation unit 306 is the same as the configuration of the first feature quantity calculation unit 305 described above. Therefore, a detailed description of the second feature amount calculation unit 306 is omitted here.

  In step S <b> 406, the region determination unit 307 compares the feature amount of the main subject region output from the first feature amount calculation unit 305 with the feature amount of the similar region output from the second feature amount calculation unit 306. Is done. It can be considered that the larger the feature amount, the better the PSF estimation. Therefore, the region determination unit 307 determines a region having a larger feature amount of the main subject region and the similar region as a region suitable for PSF estimation, and determines the region as a region used for PSF estimation. . The region determination unit 307 outputs information indicating a region used for PSF estimation, that is, a region having a larger feature of the main subject region and the similar region to the region selection unit 308.

  In step S407, the region selection unit 308 selects the luminance signal of the region determined by the region determination unit 307, and outputs the selected luminance signal to the PSF estimation unit (point spread function estimation unit) 309. That is, when the feature amount of the main subject region is larger than the feature amount of the similar region, the luminance signal of the main subject region extracted by the first local region extraction unit 303 is selected by the region selection unit 308, and the main subject region is extracted. The luminance signal of the subject area is output to the PSF estimation unit 309. On the other hand, when the feature amount of the similar region is larger than the feature amount of the main subject region, the luminance signal of the similar region extracted by the second local region extraction unit 304 is selected by the region selection unit 308, and the similar region Are output to the PSF estimation unit 309.

In step S408, the PSF estimation unit 309 estimates the PSF based on the luminance signal output from the region selection unit 308. The PSF can be estimated by a known method, for example. A known PSF estimation method will be described below. When an image input to the PSF estimation unit 309 is M, a PSF is K, and a latent image that is an image without blurring or blurring is L, the following equation (1) is established.
M = K * L (1)
Note that * is a symbol representing a convolution operation.

  K indicating PSF and L indicating latent image are unknowns. However, the provisional PSF can be estimated by appropriately setting the initial value of the latent image L. A temporary latent image is estimated using the temporary PSF, and a temporary PSF is estimated using the temporary latent image. By repeating the PSF estimation and the latent image estimation in this manner and repeating the update, it is possible to improve the estimation accuracy of the PSF and the latent image, and the PSF that satisfies a certain condition is set as the final PSF. be able to.

式（２）、（３）に示すエネルギー関数を最小化する手法としては、例えば共役勾配法が挙げられる。こうして、ＰＳＦを推定することが可能である。 The PSF and the latent image can be estimated by minimizing an energy function including a term representing a difference between both sides of the equation (1). Such an energy function is represented by the following equations (2) and (3), for example. In equations (2) and (3), σ represents a regularization term. Examples of the regularization term σ include an L2 norm that is a sum of squares of elements when an image or PSF is handled as a vector.

As a method for minimizing the energy function shown in the equations (2) and (3), for example, a conjugate gradient method can be cited. In this way, it is possible to estimate the PSF.

By the way, it is known that PSF can be estimated with high accuracy by using edge information of an image (Non-Patent Document 2). Therefore, in this embodiment, the PSF is estimated by minimizing an energy function such as Expression (4) using the edge information M ′ of the input image M and the edge information L ′ of the latent image L.

  The initial value of the edge information L ′ of the latent image L when the PSF is first obtained can be obtained as follows, for example. That is, by applying a shock filter to the input image M, the edge is enhanced while leaving a flat portion. The edge portion of the image thus obtained is used as the initial value of the edge information L ′.

  In step S409, the RGB signal conversion unit 301 converts the color difference signals RY and BY and the luminance signal Y input to the blur / blur correction unit 209 into color signals R, G, and B. Color signals R, G, and B are output to the correction processing unit 310.

ここで、Ｆはフーリエ変換、Ｆ^−１は逆フーリエ変換、￣は共役な複素数、λはブレやボケのない画像に含まれるノイズ成分の比率（Ｓ／Ｎ）である。 In step S410, the correction processing unit 310 performs blur / blur correction processing on each of the color signals R, G, and B using the PSF output from the PSF estimation unit 309. For example, the blur / blur correction process can be performed by the deconvolution method, more specifically, the winner deconvolution method as follows. That is, when the PSF estimated by the PSF estimation unit 309 is K, the captured image is M, and the latent image is L, the following equation (5) is established.

Here, F is a Fourier transform, F ⁻¹ is an inverse Fourier transform, ￣ is a conjugate complex number, and λ is a ratio (S / N) of a noise component included in an image free from blur or blur.

Note that the method used when performing the blur / blur correction process is not limited to the deconvolution method. For example, blur / blur correction processing may be performed using an inverse filter. Such an inverse filter is represented by the following equation (6).

  Further, the method used when performing the blur / blur correction process is not limited to these. For example, a filter for edge emphasis may be created from the estimated PSF, and blurring and blurring may be corrected by emphasizing the edge by processing using the filter.

  The color signals R, G, and B that have been subjected to the blur / blur correction process by the correction processing unit 310 as described above are output to the luminance / color difference signal generation unit 311. The luminance / color difference signal generation unit 311 generates a luminance signal Y and color difference signals RY and RB, and outputs the generated luminance signal Y and color difference signals RY and RB. The luminance signal Y and the color difference signals RY and RB output from the blur / blur correction unit 209 are recorded in the recording unit 105.

  Thus, according to the present embodiment, even if the image of the main subject region is not suitable for PSF estimation, the PSF can be estimated from a region similar to the main subject region. For this reason, according to the present embodiment, it is possible to obtain an image processing apparatus that can acquire PSF satisfactorily.

[Second Embodiment]
An image processing apparatus and an image processing method according to the second embodiment of the present invention will be described with reference to FIGS. The same components as those in the image processing apparatus and the image processing method according to the first embodiment shown in FIGS. 1 to 6 are denoted by the same reference numerals, and description thereof is omitted or simplified.

  When estimating the PSF from the image, an estimation error may occur. Such an estimation error depends on a subject used for PSF estimation. By combining PSFs estimated from a plurality of subjects, the dependence on the subjects can be relaxed, and estimation errors can be suppressed. Therefore, in this embodiment, the PSF is estimated by appropriately combining PSFs obtained from images of a plurality of regions. By the way, a PSF obtained from a region having a low feature amount is likely to include a large estimation error. For this reason, when the PSF estimated from the main subject area and the PSF estimated from the similar area are combined, the combination is performed based on the feature amount calculated from each area.

  FIG. 7 is a block diagram illustrating a configuration of the blur / blur correction unit 209 of the image processing apparatus 111 according to the present embodiment. In the present embodiment, the region determination unit 307, the region selection unit 308, and the PSF estimation unit 309 described above in the first embodiment with reference to FIG. 3 are not provided in the blur / blur correction unit 209. On the other hand, in the present embodiment, a first PSF estimation unit 701, a second PSF estimation unit 702, a synthesis ratio determination unit 703, and a synthesis unit 704 are provided in the blur / blur correction unit 209.

  FIG. 8 is a flowchart showing a blur / blur correction process performed in the image processing apparatus 111 according to the present embodiment.

  First, step S801 to step S805 are the same as step S401 to step S405 in the first embodiment, and thus description thereof will be omitted.

  In step S806, the first PSF estimation unit 701 estimates the PSF (first PSF) from the image information of the main subject region extracted by the first local region extraction unit 303, and combines the estimated PSF with the synthesis unit 704. Output to.

  In step S807, the second PSF estimation unit 702 estimates the PSF (second PSF) from the image information of the similar region extracted by the second local region extraction unit 304, and sends the estimated PSF to the synthesis unit 704. Output.

In step S <b> 808, the composition ratio determination unit 703 converts the main subject feature amount a <b> 1 output from the first feature amount calculation unit 305 and the similar region feature amount a <b> 2 output from the second feature amount calculation unit 306. Based on this, the composition ratio m is determined. Such a synthesis ratio m is obtained by the following equation (7), for example. The composition ratio determination unit 703 outputs the determined composition ratio m to the composition unit 704. Note that the range of the composition ratio m is 0 to 1.
m = a1 / (a1 + a2) (7)

  If a1 <a2 and a1 is equal to or smaller than a predetermined threshold value, it may be determined that the main subject area is not extremely suitable for PSF estimation, and m = 0 may be set. Further, when a1> a2 and a2 is equal to or smaller than a predetermined threshold, it is determined that the similar region is not extremely suitable for PSF estimation, and m = 1 may be set.

In step S809, the PSF estimated by the first PSF estimation unit 701 and the PSF estimated by the second PSF estimation unit 702 are combined based on the combination ratio m determined by the combination ratio determination unit 703. Unit 704 synthesizes. Since the PSF is expressed by a two-dimensional array, the elements of the PSF are synthesized based on the synthesis ratio m. First, the center of the first PSF and the center of the second PSF are aligned, and the area where the first PSF and the second PSF overlap is defined as the size (N × M) of the combined PSF. Determine. Assuming that the combined PSF is Km (i, j), the first PSF is K1 (i, j), and the second PSF is K2 (i, j), the combined PSF is expressed by the following formula ( 8). Note that (i, j) indicates coordinates.
Km (i, j) = m * K1 (i, j) + (1-m) * K2 (i, j) (8)
By performing the calculation based on the equation (8), the combined PSF is obtained. The range of i is 0 to N-1, and the range of j is 0 to M-1.

  In step S810, the correction processing unit 310 performs correction processing on each of the color signals R, G, and B output from the RGB signal conversion unit 301 based on the PSF combined by the combining unit 704. The correction processing unit 310 outputs the color signals R, G, and B subjected to the correction processing to the luminance / color difference signal generation unit 311. The luminance / color difference signal generation unit 311 generates the color difference signals RY, BY and the luminance signal Y based on the color signals R, G, B, and outputs these generated signals.

  As described above, according to the present embodiment, since the PSF is generated by combining the PSF estimated from the main subject area and the PSF estimated from the similar area, the dependence on the subject can be reduced. . Therefore, according to the present embodiment, it is possible to reduce the estimation error and obtain a better PSF.

[Modified Embodiment]
The present invention is not limited to the above embodiment, and various modifications are possible.
For example, in the above-described embodiment, an example has been described in which the region having the larger feature amount of the main subject region and the similar region is used for estimating the PSF. However, the present invention is not limited to this. For example, when the feature amount in the main subject region is equal to or greater than a predetermined threshold, the main subject region may be used for PSF estimation regardless of the feature amount in the similar region. This is because when the feature amount in the main subject area is equal to or larger than a predetermined threshold, the PSF can be estimated well from the image of the main subject area.

  In the above embodiment, the case where only one similar region is extracted has been described as an example. However, a plurality of similar regions may be extracted. In this case, the region having the largest feature amount is used for estimating the PSF.

  In addition, the feature amount calculation processing performed in the first feature amount calculation unit 305 and the second feature amount calculation unit 306 is not limited to the calculation processing as described above in the embodiment. That is, in the above embodiment, the feature amount is calculated based on the edge strength in the two directions of the horizontal direction and the vertical direction and the ratio of the saturated pixels, but the present invention is not limited to this. For example, the feature amount may be calculated based on the edge strength in four directions and the ratio of saturated pixels. Examples of the four directions include a horizontal direction, a vertical direction, an oblique 45 degree direction, and an oblique 135 degree direction.

  In the above embodiment, the case where the edge information used for calculating the feature amount is the edge strength itself has been described as an example. However, the edge information used for calculating the feature amount is not limited to the edge strength itself. . A value corresponding to the edge strength can be appropriately used as edge information used for calculating the feature amount. For example, the average value of the edge strength is calculated in each of a plurality of directions, the number of directions in which the average value is equal to or greater than a predetermined threshold is calculated, and the feature amount is calculated based on the calculated number of directions. Also good. That is, the edge information used for calculating the feature amount may be the number of directions in which the value based on the edge strength is equal to or greater than a predetermined threshold among the plurality of directions in which the value based on the edge strength is calculated. . In this case, the feature amount is calculated by multiplying the edge information calculated as described above by a gain corresponding to the ratio of saturated pixels. Alternatively, the average value of the edge strength may be calculated in each of a plurality of directions, and the feature amount may be calculated based on the calculated variance value of the average edge strength value. Further, the feature amount may be calculated based on the edge information calculated using a means that can acquire the edge strength without depending on the edge direction. As a means for acquiring the edge strength without depending on the edge direction, for example, a Laplacian filter or the like can be cited. The feature amount is calculated by multiplying the edge information obtained in this way by a gain corresponding to the ratio of saturated pixels.

  In the second embodiment, the first PSF obtained from the image information of the main subject area and the second PSF obtained from the image information of the similar area are synthesized based on the synthesis ratio m, so that However, the present invention is not limited to this. For example, the first information calculated based on the first PSF and the second information calculated based on the second PSF are combined based on the combination ratio m, thereby combining the combined information. May be obtained. Then, blurring and blurring may be corrected using the synthesized information. Examples of information calculated based on the PSF include an edge enhancement filter.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

111... Image processing device 308... Area selection unit 309... PSF estimation unit

Claims (17)

A first region detection unit for detecting a first region from an image;
A second region detecting unit for detecting a second region similar to the first region from the image;
A feature amount calculation unit that calculates a feature amount in at least one of the first region and the second region;
A region selection unit that selects the first region or the second region based on the feature amount calculated by the feature amount calculation unit;
An image processing apparatus comprising: a point spread function estimating unit that estimates a point spread function from the first region or the second region selected by the region selecting unit. A first region detection unit for detecting a first region from an image;
A second region detecting unit for detecting a second region similar to the first region from the image;
A feature amount calculation unit that calculates a feature amount in at least one of the first region and the second region;
A point spread function estimator that estimates a first point spread function from the first region and estimates a second point spread function from the second region;
A composition ratio determining unit that determines a composition ratio based on the feature amount calculated by the feature amount calculating unit;
Combining the first point spread function and the second point spread function based on the combination ratio; or information based on the first point spread function and information based on the second point spread function; An image processing apparatus comprising: a combining unit configured to combine the image processing unit based on the combining ratio. The image processing apparatus according to claim 1, wherein the first region is a main subject region that is a region of a main subject. A distance information acquisition unit that acquires distance information indicating a distance to the subject;
The image processing apparatus according to claim 1, wherein the second area detection unit detects the second area based on the distance information. A motion information detector that detects motion information indicating the motion of the subject;
The image processing apparatus according to claim 1, wherein the second area detection unit detects the second area based on the motion information. The image processing apparatus according to claim 1, wherein the second area detection unit detects the second area based on an image height. The feature amount calculation unit calculates the feature amount in the first region based on edge information in the first region, and the feature in the second region based on edge information in the second region. The image processing apparatus according to claim 1, wherein an amount is calculated. The image processing apparatus according to claim 7, wherein the edge information is a value corresponding to an edge strength. The edge information is a number of directions in which a value corresponding to the edge strength is equal to or greater than a predetermined threshold among a plurality of directions in which values corresponding to the edge strength are respectively calculated. Item 8. The image processing device according to Item 7. The feature amount calculating unit calculates the feature amount in the first region based further on the ratio of saturated pixels in the first region, and further based on the proportion of saturated pixels in the second region. The image processing apparatus according to claim 1, wherein the feature amount in the area of 2 is calculated. The image processing apparatus according to claim 1, wherein the region selection unit selects a region having the largest feature amount from the first region and the second region. When the feature amount in the first region is equal to or greater than a predetermined threshold, the region selection unit selects the first region regardless of the feature amount in the second region. The image processing apparatus according to claim 1. The correction means based on the first point spread function is a first filter generated based on the first point spread function;
The image processing apparatus according to claim 2, wherein the information based on the second point spread function is a second filter generated based on the second point spread function. Detecting a first region from the image;
Detecting a second region similar to the first region from the image;
Calculating a feature amount in at least one of the first region and the second region;
Selecting the first region or the second region based on the feature amount calculated in the step of calculating the feature amount;
And a step of estimating a point spread function from the first region or the second region selected in the step of selecting the first region or the second region. Detecting a first region from the image;
Detecting a second region similar to the first region from the image;
Calculating a feature amount in at least one of the first region and the second region;
Estimating a first point spread function from the first region and estimating a second point spread function from the second region;
A composition ratio determining unit that determines a composition ratio based on the feature amount calculated in the step of calculating the feature amount;
Combining the first point spread function and the second point spread function based on the combination ratio; or information based on the first point spread function and information based on the second point spread function; Combining the image processing method based on the combining ratio. On the computer,
Detecting a first region from the image;
Detecting a second region similar to the first region from the image;
Calculating a feature amount in at least one of the first region and the second region;
Selecting the first region or the second region based on the feature amount calculated in the step of calculating the feature amount;
A program for executing a step of estimating a point spread function from the first region or the second region selected in the step of selecting the first region or the second region. On the computer,
Detecting a first region from the image;
Detecting a second region similar to the first region from the image;
Calculating a feature amount in at least one of the first region and the second region;
Estimating a first point spread function from the first region and estimating a second point spread function from the second region;
A composition ratio determining unit that determines a composition ratio based on the feature amount calculated in the step of calculating the feature amount;
Combining the first point spread function and the second point spread function based on the combination ratio; or information based on the first point spread function and information based on the second point spread function; A program for executing the step of synthesizing based on the synthesis ratio.

Images