JP2017146882A - Image processing apparatus, image processing method, image processing program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to sharpen an image with blur and unsharpness and reduce an influence of an estimated error of an estimated point spread function (PSF).SOLUTION: An image processing apparatus comprises: a main subject detection part 209 that detects information on a subject from an input image; a PSF estimation part 302 that estimates a point spread function on the basis of the information on the subject; a smoothness processing part 303 that corrects the point spread function on the basis of the state of blur of the input image; and an edge emphasis processing part 304 that performs edge emphasis processing on the input image on the basis of the corrected point spread function.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a storage medium.

  Conventionally, a technique for sharpening a captured image using a point spread function (PSF) is known. For example, Patent Document 1 discloses a technique for sharpening an image that has deteriorated due to camera shake or the like by estimating PSF from camera motion information at the time of shooting and performing edge enhancement based on the estimated PSF. . Further, in Patent Document 2, an approximate solution of PSF is estimated from an image by the method described in Non-Patent Document 1, and image blurring and blurring are sharpened by deconvolution of the estimated approximate solution of PSF and an input image. Technology is disclosed.

Japanese Patent No. 5191224 JP2015-64890A

Qi Shan, Jiaja Jia, and Aseme Agarwala, “High-quality Motion Deblurring from a Single Image”, SIGGRAPH 2008

  However, since the technique disclosed in Patent Document 1 cannot estimate the PSF indicating the defocus of the subject, it cannot be sharpened in consideration of the defocus. Further, with the technique disclosed in Patent Document 2, it is possible to sharpen the subject's defocus by estimating the PSF from the image. However, since the approximate solution of the PSF is estimated, there is a problem caused by the estimation error. It can happen. In particular, since the deconvolution process is performed in the frequency space, there is a tendency that the harmful effect is prominent due to the Gibbs phenomenon or the like.

  In view of such a problem, the present invention enables an image processing apparatus, an image processing method, and an image processing program capable of sharpening an image with blurring or blurring and reducing the influence of an estimated PSF estimation error. And a storage medium.

  An image processing apparatus according to one aspect of the present invention includes a detection unit that detects subject information from an input image, an estimation unit that estimates a point spread function based on the subject information, and a blurring state of the input image. And a correction unit that corrects the point spread function based on the correction and a processing unit that performs an edge enhancement process on the input image based on the corrected point spread function.

  An image processing method according to another aspect of the present invention includes a step of detecting subject information from an input image, a step of estimating a point spread function based on the subject information, and a blurring state of the input image. And a step of correcting the point spread function based on the above and a step of performing an edge enhancement process on the input image based on the corrected point spread function.

  According to the present invention, it is possible to provide an image processing device, an image processing method, an image processing program, and a storage medium that can sharpen an image with blurring or blurring and reduce the influence of an estimated PSF estimation error. can do.

1 is a block diagram of a digital camera according to an embodiment of the present invention. It is a block diagram of an image processing part. FIG. 3 is a block diagram of a sharpening processing unit (Example 1). It is a flowchart which shows the sharpening process (Example 1). It is a block diagram of a smoothing process part (Example 1). FIG. 3 is a block diagram of an edge enhancement processing unit (first embodiment). FIG. 6 is a relationship diagram between a distance difference from a main subject and edge enhancement strength (Example 1). It is a block diagram of a sharpening process part (Example 2). It is a block diagram of a PSF intensity | strength change part (Example 2). It is a gain characteristic figure (Example 2).

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram of a digital camera according to an embodiment of the present invention. The digital camera includes an optical system 101, an image sensor 102, an A / D conversion unit 103, an image processing unit 104, a recording unit 105, a control unit 106, a focus detection unit 107, a shake state determination unit 108, an operation unit 109, and a display unit. 110.

  The optical system 101 includes a focus lens, a diaphragm, and a shutter. The optical system 101 adjusts the amount of exposure by driving the focus lens to focus the subject at the time of shooting, and controlling the aperture and shutter. The imaging element 102 is a photoelectric conversion element such as a CCD or CMOS that converts an object image formed in the optical system 101 into an electric signal by photoelectric conversion. The A / D conversion unit 103 digitizes the input electrical signal and outputs it to the image processing unit 104. The image processing unit 104 performs a synchronization process, a white balance correction process, a color conversion process, a gamma process, a sharpening process, and the like on the signal output from the A / D conversion unit 103, and the processed signal Is output to the recording unit 105. In addition, the image processing unit 104 executes main subject detection processing. The recording unit 105 converts the image information output from the image processing unit 104 into an image format such as JPEG and records it. The control unit 106 controls the operation of the entire digital camera. For example, the control unit 106 calculates a predetermined evaluation value based on the photographed image, and determines parameters for image processing performed by the image processing unit 104. The focus detection unit 107 acquires distance information to the subject at the time of shooting and generates a distance map. The distance map is a two-dimensional array indicating distance information to the subject in pixel units of the captured image. The shake state determination unit 108 determines the direction and size of the shake of the digital camera at the time of shooting using a motion sensor mounted on the camera. The operation unit 109 is a part where the user gives an operation instruction. The display unit 110 is, for example, a liquid crystal display or the like provided on the back of the camera, and displays a screen for assisting an operation during shooting, an image stored in the recording unit 105, and the like.

  The image processing unit 104 will be described in detail with reference to FIG. FIG. 2 is a block diagram of the image processing unit 104. The image processing unit 104 includes a synchronization processing unit 201, a white balance correction processing unit 202, and a luminance / color signal generation unit 203. The image processing unit 104 includes a color conversion processing unit 204, a color gamma processing unit 205, a luminance gamma processing unit 206, a color difference signal generation unit 207, a sharpening processing unit 208, and a main subject detection unit (detection unit) 209. . Note that the image processing executed by the image processing unit 104 of the present embodiment is executed according to an image processing program as a computer program operating on software and hardware. The image processing program may be recorded on a computer-readable recording medium, for example.

  The synchronization processing unit 201 performs synchronization processing on the input Bayer RGB image data to generate color signals R, G, and B. The white balance correction processing unit 202 adjusts the white balance by applying a gain to the color signal RGB generated by the synchronization processing unit 201 based on the white balance gain value calculated by the control unit 106. The luminance / color signal generation unit 203 generates a luminance signal Y based on the color signal RGB whose white balance has been adjusted by the white balance correction processing unit 202. The luminance / color signal generation unit 203 outputs the generated luminance signal Y to the luminance gamma processing unit 206 and outputs the color signal RGB to the color conversion processing unit 204.

  The color conversion processing unit 204 converts the color signal RGB into a desired color balance by performing a matrix operation on the color signals RGB. The color gamma processing unit 205 performs gamma correction on the color signals RGB. The color difference signal generation unit 207 generates RY and BY signals from the color signal RGB, and outputs the generated RY and BY signals to the recording unit 105.

  The luminance gamma processing unit 206 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the sharpening processing unit 208 and the main subject detection unit 209. The sharpening processing unit 208 performs a sharpening process on the luminance signal Y input from the luminance gamma processing unit 206. The luminance signal Y subjected to the sharpening process is output to the recording unit 105 and recorded. The main subject detection unit 209 detects the position and size of the main subject based on the luminance signal Y. The main subject refers to the face of a person in this embodiment, but may be a human body or a part thereof.

  With reference to FIGS. 3 and 4, the sharpening processing unit 208 of the present embodiment will be described. FIG. 3 is a block diagram of the sharpening processing unit 208 of the present embodiment. FIG. 4 is a flowchart showing the sharpening process. The sharpening processing unit 208 includes a local region extraction unit 301, a PSF estimation unit (estimation unit) 302, a smoothing processing unit (correction unit) 303, and an edge enhancement processing unit (processing unit) 304.

  In step S401, the local area extraction unit 301 extracts the luminance signal of the main subject from the luminance signal Y by using the luminance signal Y input to the sharpening processing unit 208 and the position and size information of the main subject. And output to the PSF estimation unit 302.

  In step S <b> 402, the PSF estimation unit 302 estimates a point spread function (hereinafter referred to as PSF) using the luminance signal output from the local region extraction unit 301, and outputs the estimated PSF to the smoothing processing unit 303.

Here, the PSF estimation method will be described. The image B input to the PSF estimation unit 302 is expressed by a convolution operation of K indicating the PSF to be estimated and L indicating an image (latent image) without blurring or blurring, as shown in Expression (1). it can.

Represents a convolution operation.

  Although K indicating PSF and the latent image L are unknown numbers, provisional PSF estimation is possible by setting an initial value of the latent image L. The provisional latent image L is estimated using the provisional PSF, and the provisional PSF is estimated using the provisional latent image L. In this way, the PSF and the latent image L are repeatedly estimated and updated. Thereby, the PSF and the estimation accuracy of the latent image L are improved, and a PSF that satisfies a certain condition is set as a final PSF.

  A specific estimation method of the PSF and the latent image L is to minimize an energy function including a term representing a difference between both sides of the equation (1). Each energy function is represented by the formula (2) and the formula (3). In the equations (2) and (3), σ represents a regularization term. As an example of the regularization term σ, there is an L2 norm that takes the sum of squares of elements when an image and a PSF are handled as vectors. As a method for minimizing the energy function, for example, there is a method using a conjugate gradient method.

  In step S <b> 403, the smoothing processing unit 303 smoothes the PSF estimated by the PSF estimation unit 302 based on the blur direction and magnitude determined by the blur state determination unit 108. When the main subject is out of focus and the amount of blur is small, the smoothing processing unit 303 performs smoothing evenly regardless of the direction of blur. In addition, when the main subject has a blur, the smoothing processing unit 303 performs smoothing so as to extend in the blur direction. Thereby, the influence of the PSF estimation error can be effectively reduced, and the edge enhancement processing unit 304 can achieve good sharpening.

  The PSF smoothing method will be described in detail with reference to FIG. FIG. 5 is a block diagram of the smoothing processing unit 303. The smoothing processing unit 303 includes a PSF rotation unit 501, a horizontal low-pass filter unit 502, a vertical low-pass filter unit 503, and a PSF reverse rotation unit 504.

  The PSF rotation unit 501 rotates the PSF based on the PSF and blur direction input to the smoothing processing unit 303. Specifically, the PSF rotating unit 501 rotates the PSF so that the blur direction is horizontal.

  The horizontal low-pass filter unit 502 applies a low-pass filter to the PSF rotated by the PSF rotation unit 501 based on the amount of blur input to the smoothing processing unit 303. For example, when the number of taps is N in the low-pass filter used in this embodiment, the amount of smoothing increases as the number of taps increases as shown in Expression (4).

  The number of taps of the low-pass filter increases as the blur magnitude increases. However, the number of taps is larger than a preset minimum value T_MIN.

  The vertical low-pass filter unit 503 applies a low-pass filter to the PSF filtered by the horizontal low-pass filter unit 502 in the vertical direction. The number of taps is T_MIN. Thereby, smoothing is possible like an ellipse extending in the blur direction by using a multi-tap smoothing filter from the direction orthogonal to the blur direction.

  The PSF reverse rotation unit 504 rotates the PSF filtered by the vertical low-pass filter unit 503 based on the blur direction input to the smoothing processing unit 303. The rotation direction is opposite to the rotation direction by the PSF rotation unit 501, and the direction characteristic of the PSF output from the PSF reverse rotation unit 504 is the same as the direction characteristic of the PSF before being input to the smoothing processing unit 303. The PSF reverse rotation unit 504 outputs the PSF whose direction characteristics have been returned to the edge enhancement processing unit 304.

  In step S404, the edge enhancement processing unit 304 uses the PSF smoothed by the smoothing processing unit 303, the distance map acquired by the focus detection unit 107, and the position and size of the main subject detected by the main subject detection unit 209. Thus, the brightness signal Y is sharpened.

  The edge enhancement process will be described with reference to FIG. FIG. 6 is a block diagram of the edge enhancement processing unit 304. The edge enhancement processing unit 304 includes a convolution unit 601, an intensity calculation unit 602, a subtraction unit 603, a multiplication unit 604, and an addition unit 605.

  The convolution unit 601 performs a convolution operation on the luminance signal Y input to the edge enhancement processing unit 304 and the smoothed PSF, and outputs the calculated luminance signal Y ′ to the subtraction unit 603. The subtraction unit 603 generates a difference signal Y−Y ′ between the luminance signal Y and the luminance signal Y ′, and outputs the difference signal Y−Y ′ to the multiplication unit 604. The multiplier 604 multiplies the difference signal Y-Y ′ by the intensity information output from the intensity calculator 602, and outputs the multiplied signal to the adder 605. The adder 605 performs edge enhancement by adding the signal multiplied by the multiplier 604 and the luminance signal Y together. The edge enhancement processing unit 304 extracts a high-frequency component from the difference between the luminance signal Y and the luminance signal Y ′ to which blur is applied based on the characteristics of blur and blur by convolving the smoothed PSF. Then, an edge emphasis signal that matches the characteristics of blur and blur is generated. Therefore, the edge enhancement processing unit 304 can realize edge enhancement processing that matches the characteristics of blur and blur by adding the difference signal Y−Y ′ to the luminance signal Y.

  The processing of the intensity calculation unit 602 will be described in detail. The intensity calculator 602 calculates intensity information using the distance map, the main subject position and size information, and the focus area position and size information input to the edge enhancement processing unit 304. First, the intensity calculating unit 602 refers to the distance map and calculates the distance average D1 of the main subject and the distance average D2 of the attention area. Next, an absolute value D_abs of a difference between the distance average D1 and the distance average D2 is calculated, and intensity information is determined based on the absolute value D_abs. At this time, if edge enhancement processing is performed on the focused subject, image quality degradation may occur. Therefore, when the absolute value D_abs is larger than the average distance Dp of the focused region, the intensity is 0 as shown in FIG. To do. The maximum value of the intensity information is a preset value.

  As described above, the image processing unit 104 according to the present exemplary embodiment can sharpen an image while reducing the influence of the estimated PSF estimation error.

  In the first embodiment, the sharpening processing unit 208 uses the smoothing processing unit 303 to reduce the influence of the PSF estimation error, but the present invention is not limited to this. FIG. 8 is a block diagram of the sharpening processing unit 208 of the present embodiment. In this embodiment, the sharpening processing unit 208 includes a PSF intensity changing unit (correction unit) 801 instead of the smoothing processing unit 303.

  The PSF intensity changing unit 801 will be described with reference to FIG. FIG. 9 is a block diagram of the PSF intensity changing unit 801. The PSF intensity changing unit 801 includes a gain setting unit 901, a gain shaping unit 902, and a multiplication unit 903. The PSF intensity changing unit 801 performs gain processing on the PSF based on the PSF estimated by the PSF estimating unit 302 and the blur direction and magnitude determined by the blur state determining unit 108 to change the intensity. The PSF whose intensity has been changed is output to the edge enhancement processing unit 304.

  The gain setting unit 901 sets a gain of 0.0 to 1.0 based on the PSF output from the PSF estimation unit 302. Since the gain is applied to the two-dimensional PSF, it is expressed in two dimensions. FIG. 10A is a gain characteristic diagram of the gain setting unit 901. The horizontal axis indicates the distance from the center of the PSF, and the vertical axis indicates the gain. As shown in FIG. 10A, the greater the distance from the center, the lower the gain applied to the PSF, and the gain becomes 0 when a preset threshold Th is exceeded. Thereby, a gain spreading in a circular shape can be set, and the set gain is output to the gain shaping unit 902.

  The gain shaping unit 902 performs shaping using the gain output from the gain setting unit 901 and the blur direction output from the blur state determination unit 108. Specifically, the gain shaping unit 902 performs shaping by multiplying a gain of 0 to 1 in a direction orthogonal to the blur direction so that the gain in the blur direction is higher than the gain in the direction orthogonal to the blur direction. To do. FIG. 10B is a gain characteristic diagram of the gain shaping unit 902. The horizontal axis indicates the distance from the straight line passing through the center of the PSF and extending in the blur direction, and the vertical axis indicates the gain. As shown in FIG. 10B, the gain decreases as the distance in the direction orthogonal to the blur direction increases, and the gain becomes 0 when a preset threshold value Th_min is exceeded. The threshold value Th_min is set to be smaller than the threshold value Th. As a result, the gain intensity in the blur direction becomes higher than the gain in the direction orthogonal to the blur direction. The set gain is output to the multiplier 903.

  Multiplication section 903 multiplies the PSF input to PSF intensity changing section 801 by the gain output from gain shaping section 902 to change the intensity of PSF.

  As described above, the image processing unit 104 according to the present exemplary embodiment can sharpen an image while reducing the influence of the estimated PSF estimation error.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

104 Image processing unit (image processing apparatus)
209 Main subject detection unit (detection unit)
302 PSF estimation unit (estimation unit)
303 Smoothing processing unit (correction unit)
304 Edge enhancement processing unit (processing unit)
801 PSF intensity change unit (correction unit)

Claims (10)

A detection unit for detecting subject information from the input image;
An estimation unit that estimates a point spread function based on the subject information;
A correction unit that corrects the point spread function based on a blurring state of the input image;
An image processing apparatus comprising: a processing unit that performs edge enhancement processing on the input image based on the corrected point spread function.   The image processing apparatus according to claim 1, wherein the correction unit corrects the point spread function based on a blur direction and a magnitude of the input image.   The image processing apparatus according to claim 2, wherein the correction unit smoothes the point spread function based on a blur direction and a magnitude of the input image.   The image processing apparatus according to claim 2, wherein the correction unit changes the intensity of the point spread function based on a blur direction and a magnitude of the input image.   The said process part performs the said edge emphasis process based on the image which carried out the convolution process of the said point spread function to the said input image, The one in any one of Claim 1 to 4 characterized by the above-mentioned. Image processing device.   The said process part performs the said edge emphasis process based on the difference of the distance of the to-be-photographed object detected by the said detection part, and the distance of another object, The any one of Claim 1 to 5 characterized by the above-mentioned. Image processing apparatus.   The said processing part sets the gain of the said edge emphasis process so that the gain with respect to the difference of the distance of the to-be-focused subject with the distance of the to-be-photographed object detected by the said detection part may become the minimum. The image processing apparatus according to any one of 1 to 6. Detecting subject information from the input image;
Estimating a point spread function based on the subject information;
Correcting the point spread function based on a blurring state of the input image;
Performing an edge enhancement process on the input image based on the corrected point spread function.   An image processing program for causing a computer to execute each step of the image processing method according to claim 8. A computer-readable recording medium for recording the image processing program according to claim 9.