JP6873815B2 - Image processing equipment, imaging equipment, image processing methods, programs, and storage media - Google Patents

Description

The present invention relates to an image processing device that corrects deterioration such as blurring and blurring of a captured image.

Conventionally, the point spread function (PSF), which represents the characteristics of the subject blur and blur included in the captured image, is estimated, and the captured image is corrected based on the estimated PSF to prevent deterioration of the image such as blur and blur. The technique of correction is known. For example, Patent Document 1 describes a method in which an image is divided into a plurality of small regions, PSFs are estimated for each of the plurality of small regions, and small regions having PSFs having similar characteristics are grouped to correct blurring of the image. It is disclosed.

Japanese Unexamined Patent Publication No. 2012-155456

By the way, when there are subject areas in the image that have different deterioration characteristics such as blurring characteristics, if the correction based on PSF is applied uniformly in the image, the correction will be erroneous depending on the area, such as ringing. May cause harmful effects. In this case, if the method of Patent Document 1 is used, it is possible to perform appropriate correction for each subject area according to the deterioration characteristics of the subject area. However, in the method of Patent Document 1, it is necessary to estimate the PSF for each of the plurality of small regions in which the image is divided, which increases the amount of calculation of the PSF estimation process.

Therefore, an object of the present invention is to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium capable of appropriately correcting an image having different deterioration characteristics for each region with a small amount of calculation. And.

The image processing apparatus according to one aspect of the present invention, by performing the estimation tough from estimated area as partial area of the image you estimate the point spread function, the correction process of the image based on the point spread function A correction processing unit that generates a corrected image, a deterioration characteristic estimation unit that estimates the deterioration characteristics of the image for each of a plurality of regions of the image different from the estimation regions, and a composite ratio based on the deterioration characteristics. and a synthesizing unit synthesizing the corrected image and the image, the combining unit, based on said deterioration characteristic for each of the plurality of areas of said deterioration characteristic and the image for the previous Ki推constant region of the image , The synthesis ratio in each of the plurality of regions is calculated .

An imaging device as another aspect of the present invention includes an imaging unit that photoelectrically converts an optical image formed via an imaging optical system and outputs an image signal, and the image processing device.

The image processing method as another aspect of the present invention, the corrected image by performing the steps of estimating a point spread function from the estimated region of the partial region of the image, the correction process of the image based on the point spread function The image and the corrected image are combined by a step of generating the image, a step of estimating the deterioration characteristic of the image for each of a plurality of regions of the image different from the estimation region, and a composition ratio based on the deterioration characteristic. and a step of the synthesis ratio, based on said deterioration characteristic for each of the plurality of areas of said deterioration characteristic and the image for the previous Ki推constant region of the image, in each of the plurality of regions It is calculated .

A program as another aspect of the present invention causes a computer to execute the image processing method.

A storage medium as another aspect of the present invention stores the program.

Other objects and features of the present invention will be described in the following embodiments.

According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium capable of appropriately correcting an image having different deterioration characteristics for each region with a small amount of calculation. it can.

It is a block diagram of the digital camera in each embodiment. It is a block diagram of the image processing part in 1st Embodiment. It is a flowchart which shows the image processing method in 1st Embodiment. It is explanatory drawing of the characteristic estimation processing and synthesis ratio calculation processing in 1st Embodiment. It is a block diagram of the image processing part in 2nd Embodiment. It is a flowchart which shows the image processing method in 2nd Embodiment. It is explanatory drawing of the selection method of PSF in 2nd Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment)
First, the configuration of the image pickup apparatus in the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram of an imaging device (digital camera) 100 according to the present embodiment. The present embodiment relates to a correction process applied to a captured image, more specifically, a method of estimating a PSF from an image signal and correcting blurring or blurring of the image based on the estimated PSF. Therefore, the configuration related to photography and recording in the image pickup apparatus 100 is not essential. In the present embodiment, "blurring" includes blurring (focus blurring) due to out-of-focus of the image, and "blurring" includes, but is not limited to, camera shake and subject blurring.

In FIG. 1, the lens group 101 is a zoom lens (imaging optical system) including a focus lens. In the image pickup device 100 of the present embodiment, the image pickup optical system and the image pickup device main body are integrally configured. However, the present embodiment is not limited to this, and can be applied to an imaging system including an imaging device main body and an imaging optical system (interchangeable lens) that can be attached to and detached from the imaging device main body.

The shutter 102 has an aperture function and is provided between the lens group 101 and the imaging unit 103. The image pickup unit 103 has an image pickup element such as a CMOS sensor or a CCD sensor, and converts an optical image formed on the image pickup surface via the lens group 101 into an electric signal (analog signal) in pixel units (photoelectric conversion). ). The A / D converter 104 converts the analog signal output from the imaging unit 103 into a digital signal (image data).

The image processing unit (image processing device) 105 performs various image processing such as color interpolation (demosaic), white balance adjustment, and γ correction on the image data output from the A / D converter 104. Further, the image processing unit 105 performs correction processing for correcting blurring and blurring of the captured image. The image memory 106 temporarily stores image data. The memory control unit 107 controls reading and writing of the image memory 106. The D / A converter 108 converts the image data into an analog signal. The display unit 109 has a display device such as an LCD or an organic EL display, and displays various GUIs, live view images, images read from the recording medium 112, and the like. The recording medium 112 is a removable recording means such as a semiconductor memory card or a card-type hard disk. The codec unit 110 encodes the image data stored in the image memory 106 by a predetermined method for recording on the recording medium 112, or displays the coded image data included in the image file, for example, for display. Decrypt. The interface (I / F) 111 mechanically and electrically connects the recording medium 112 to the image pickup apparatus 100.

The system control unit 50 has a programmable processor such as a CPU and an MPU. The system control unit 50 realizes each function of the image pickup apparatus 100 by, for example, executing a program stored in the non-volatile memory 121 or the built-in non-volatile memory to control necessary blocks and circuits. The face detection unit 113 detects a face region included in the captured image, and obtains face information such as position, size, and reliability for each of the detected face regions. The face detection unit 113 uses learning represented by a neural network, searches for characteristic parts such as eyes, nose, and mouth from an image area using template matching, and regards the face as a face when the degree of similarity is high. , The face region can be detected using any method.

The motion vector detection unit 124 uses the image signal output from the image processing unit 105 and the image signal of the immediately captured image recorded in the image memory 106 to generate a motion vector in each region of the captured image. Is detected (calculated). As a method for calculating the motion vector, a known technique can be used. The motion vector can be calculated, for example, as follows. First, the captured image is divided into blocks of a predetermined size. Subsequently, the area corresponding to each block of the captured image is detected from the image captured immediately before by using the template matching method. Then, the vector connecting the corresponding regions is used as the motion vector in each block.

The operation unit 120 is an input device such as a button or a switch for the user to input various instructions to the image pickup device 100. When the display unit 109 is a touch display, the operation unit 120 includes a touch panel. Further, the operation unit 120 may include an input device of a type such as voice input or line-of-sight input that inputs instructions in a non-contact manner. The non-volatile memory 121 is an EEPROM or the like that can be electrically erased and recorded. The non-volatile memory 121 stores various setting values and GUI data. Further, the non-volatile memory 121 stores a program for execution by the system control unit 50 when the system control unit 50 is an MPU or a CPU. The system memory 122 is used to develop constants and variables for the operation of the system control unit 50, a program read from the non-volatile memory 121, and the like. The distance measuring unit 123 is a distance measuring means for measuring the distance from the imaging device 100 to the subject.

Next, the basic operation when the subject is photographed by the imaging device 100 will be described. With the shutter 102 open, the imaging unit 103 photoelectrically converts a subject image (optical image) formed on the imaging surface via the lens group 101 and outputs the analog image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data), and outputs the image data to the image processing unit 105.

The image processing unit 105 performs various image processing such as simultaneous processing (demosaic processing) and γ correction on the image data from the A / D converter 104 or the image data from the memory control unit 107. Further, the image processing unit 105 performs predetermined arithmetic processing related to brightness, contrast, and the like using the image data obtained by photographing, and the system control unit 50 performs focus adjustment and exposure control based on the obtained arithmetic result. The detection results of the face detection unit 113 and the distance measuring unit 123 may be used for focus adjustment and exposure control. The image pickup apparatus 100 of the present embodiment performs TTL (through-the-lens) AF (autofocus) processing and AE (autoexposure) processing. Further, the image processing unit 105 performs blur and blur correction processing based on PSF by using a method described later.

The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109. The D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 displays on a display device such as an LCD according to the analog signal from the D / A converter 108. The codec unit 110 encodes the image data recorded in the image memory 106 based on a standard such as JPEG or MPEG. The system control unit 50 adds a predetermined header or the like to the encoded image data to form an image file, and records the encoded image data on the recording medium 112 via the interface 111.

The image pickup apparatus 100 can make the display unit 109 function as an electronic viewfinder (EVF) by taking a moving image in the shooting standby state and continuing to display the shot moving image on the display unit 109. In this case, the shutter 102 is in the open state, and the so-called electronic shutter of the imaging unit 103 is used to take a picture at, for example, 30 frames / second. Then, when the shutter button included in the operation unit 120 is pressed halfway, the system control unit 50 performs the above-mentioned AF control and AE control. When the shutter button included in the operation unit 120 is fully pressed, the system control unit 50 executes still image shooting for recording by the main shooting and records the shot image on the recording medium 112. Further, when the moving image shooting is instructed by the moving image shooting button or the like, the system control unit 50 starts recording the moving image on the recording medium 112.

Next, with reference to FIGS. 2 and 3, a function related to blur and blur correction processing by the image processing unit 105 will be described. FIG. 2 is a block diagram of the image processing unit 105. FIG. 3 is a flowchart of correction processing (image processing method of the present embodiment) by the image processing unit 105. Each step in FIG. 3 is executed by each part of the image processing unit 105.

At least one of the functional blocks shown in FIG. 2 is realized by a combination of a microprocessor and software. Alternatively, at least one of the functional blocks may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device). PLD includes, but is not limited to, FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array), and the like.

The blur or blur correction process is performed on an image taken with the execution of the blur or blur correction process specified, or on a recording medium 112 instructed to perform the blur or blur correction process from a menu screen or the like. This can be done for recorded images. It should be noted that the information at the time of shooting necessary for the blur and blur correction processing can be read from the non-volatile memory 121 or the system memory 122, or can be acquired from the header or metadata of the image file.

As shown in FIG. 2, the image processing unit 105 includes an image signal generation unit 201, a PSF estimation area extraction unit 202, a PSF estimation unit 203, an image correction unit 204, an image composition unit 205, a deterioration characteristic estimation unit 206, and It has a synthesis ratio calculation unit 207. In the present embodiment, the image correction unit 204, the image composition unit 205, and the composition ratio calculation unit 207 constitute a correction processing unit.

First, in step S301 of FIG. 3, the image signal generation unit 201 performs simultaneous processing (demosaic processing) on an image signal having information of one color (any one of R, G, and B) per pixel. Each pixel generates an image signal having information of three colors (RGB). The image signal generation unit 201 outputs the generated image signal to the PSF estimation area extraction unit 202, the image correction unit 204, and the image composition unit 205.

Subsequently, in step S302, the PSF estimation area extraction unit 202 selects a PSF estimation area as a partial area to be used for the PSF estimation process from the image signals (input images) output from the image signal generation unit 201. Then, the PSF estimation area extraction unit 202 extracts the image signal of the selected PSF estimation area from the input image. As a method of selecting the PSF estimation area, a method of detecting the main subject area in the input image and selecting the subject detection area can be used. Alternatively, the user may be made to specify the PSF estimation area via the operation unit 120. The PSF estimation area extraction unit 202 outputs the extracted image signal to the PSF estimation unit 203. Further, the PSF estimation area extraction unit 202 outputs information regarding the position of the selected PSF estimation area to the synthesis ratio calculation unit 207.

Subsequently, in step S303, the PSF estimation unit 203 estimates the PSF (point spread function) representing the characteristics (deterioration characteristics) related to blurring and blurring in the PSF estimation region based on the input image signal in the PSF estimation region (point spread function). PSF estimation process). That is, the PSF estimation unit 203 estimates the PSF from the PSF estimation region as a partial region of the image. Then, the PSF estimation unit 203 outputs the information regarding the estimated PSF to the image correction unit 204. In the present embodiment, a known technique can be used for the PSF estimation process. Hereinafter, as an example, an outline of the PSF estimation process using the iterative method will be described.

Of the captured images, the image signal in the PSF estimation region is B, the PSF to be estimated is K, and the image (latent image) without blurring or blurring in the PSF estimation region is L, and the noise of the captured image is ignored. The relationship of Eq. (1) holds.

In equation (1), Conv represents a convolution operation. Equation (1) cannot be solved analytically because both K and L are unknown. Therefore, the following estimation is performed.

First, an appropriate initial value of the latent image L is determined. As the initial value of the latent image L, an image obtained by subjecting the captured image to edge enhancement processing using a shock filter or the like can be used.

Subsequently, in the energy function E (K) represented by the following equation (2), K is calculated by fixing L and setting K as an unknown number to minimize the energy function E (K).

In equation (2), σ represents a regularization term, and here the L2 norm that takes the sum of squares of each element of PSF is used.

Subsequently, using the calculated K, in the energy function E (L) represented by the following equation (3), K is fixed, L is an unknown variable, and L is calculated to minimize the energy function E (L). To do.

In the equation (3), σ represents a regularization term, and here, the L2 norm that takes the sum of squares of each pixel value of the latent image L is used.

Then, the calculated K is used to calculate the K that minimizes the energy function E (K), and the calculated K is used to calculate the L that minimizes the energy function E (L). PSF is estimated by iteratively until it converges to a constant value. The above is the outline of the PSF estimation method using the iterative method.

Subsequently, in step S304 of FIG. 3, the image correction unit 204 corrects deterioration such as blurring and blurring of the image signal generated by the image signal generation unit 201 based on the input information regarding the PSF. Perform (image correction processing). Then, the image correction unit 204 outputs the corrected image signal to the image composition unit 205. In the present embodiment, a known technique can be used as the correction method based on PSF. As an example, an outline of the correction process using the winner deconvolution will be described. Assuming that the estimated PSF is K, the captured image is B, and the latent image is L, the latent image L can be expressed by the following equation (4).

In equation (4), f () indicates the inverse Fourier transform, * indicates the complex conjugate, and λ indicates the ratio (SN) of the noise component contained in the image without blurring or blurring. In addition to the above-mentioned winner deconvolution, a method using an inverse filter represented by the following equation (5) may be adopted.

Further, a correction method for emphasizing the edge by creating an edge enhancement filter from the estimated PSF and performing the filter processing may be used.

Subsequently, in step S305 of FIG. 3, the deterioration characteristic estimation unit (blurring characteristic estimation unit) 206 estimates the deterioration characteristics (blurring characteristics) such as blurring and blurring for each of the plurality of regions of the captured image, and the deterioration characteristic amount. (Blur blur characteristic amount) is calculated. The details of this characteristic estimation process will be described later. The deterioration characteristic estimation unit 206 outputs an evaluation value (deterioration characteristic amount) related to the estimated deterioration characteristic to the synthesis ratio calculation unit 207.

Subsequently, in step S306, the composition ratio calculation unit 207 calculates the image composition ratio in each region of the captured image. The image composition ratio in each region of the captured image includes an evaluation value (deterioration characteristic evaluation value) that represents the characteristics of blurring and blurring estimated by the deterioration characteristic estimation unit 206 and a PSF estimation region selected by the PSF estimation area extraction unit 202. It can be calculated based on the information about the position of. Here, the image composition ratio refers to the captured image before the correction process output from the image signal generation unit 201 and the image after the correction process (correction image) output from the image correction unit 204 by the image composition unit 205. Represents the composition ratio when synthesizing. The details of the calculation method of the synthesis ratio will be described later. The composition ratio calculation unit 207 outputs the calculated composition ratio to the image composition unit 205.

Subsequently, in step S307, the image synthesizing unit 205 calculates the image signal output from the image signal generation unit 201 and the corrected image signal output from the image correction unit 204 by the synthesizing ratio calculation unit 207. A composite image is generated by synthesizing based on the composition ratio. Here, the image signal of the composite image is I, the image signal generated by the image signal generation unit 201 is I ₀ , the corrected image signal generated by the image correction unit 204 is I ₁ , the composition ratio is m, and the image The position shall be represented by (x, y). The image synthesizing unit 205 synthesizes an image signal using the following equation (6).

When the image synthesizing unit 205 outputs the synthesized image signal, the synthesized image signal is recorded on the recording medium 112 based on the control by the system control unit 50. The above is the flow of the correction process in the present embodiment.

Next, with reference to FIG. 4, the deterioration characteristic estimation unit 206 estimates the characteristics of blurring and blurring in each region of the captured image (characteristic estimation processing), and the composition ratio calculation unit 207 calculates the image composition ratio. The details of the processing (composite ratio calculation processing) to be performed will be described. FIG. 4 is an explanatory diagram of the characteristic estimation process and the composite ratio calculation process.

The deterioration characteristic estimation unit 206 estimates the magnitude of blur (focus blur) in each region of the image, the magnitude of blur, and the direction of blur, and outputs an evaluation value regarding deterioration characteristics such as blur and blur. Here, the blur characteristic (deterioration characteristic related to blur) can be calculated based on the information of the distance between the image pickup device 100 and the subject in each region of the image measured by the distance measuring unit 123. Specifically, the distance information regarding the in-focus position determined as the in-focus position in the AF control performed at the time of shooting is compared with the distance information in each region of the image. When the distance information in each region of the image is close to the distance information in the in-focus point, the degree of defocus is small, and the different the distance information, the greater the degree of defocus. Therefore, the deterioration characteristic estimation unit 206 normalizes the difference value between the subject distance in each region of the image and the subject distance in the in-focus range by a predetermined normalization coefficient, and outputs the blur characteristic evaluation value.

The magnitude and direction of blur (camera shake or subject blur) can be estimated based on the motion vector detected by the motion vector detection unit 124. Specifically, the direction of the motion vector represents the direction in which the blur occurs, and the magnitude of the motion vector represents the magnitude of the blur. Therefore, the deterioration characteristic estimation unit 206 normalizes the magnitude and direction of the motion vector in each region of the image with a predetermined normalization coefficient, and outputs the characteristic evaluation value regarding the magnitude and direction of the blur.

Next, a method in which the composition ratio calculation unit 207 calculates the image composition ratio will be described. Here, the composite ratio calculation unit 207 is based on the comparison result between the deterioration characteristics such as blurring and blurring related to the PSF estimation region and the deterioration characteristics such as blurring and blurring related to each of the plurality of regions of the image, for each region of the image. Calculate the composite ratio. Of a plurality of regions of the image, a specific region having deterioration characteristics close to the deterioration characteristics related to the PSF estimation region can be corrected based on the PSF estimated by the PSF estimation region, and thus the ratio of the corrected images. To be high. On the other hand, for a specific region having deterioration characteristics different from the deterioration characteristics related to the PSF estimation region, the correction based on the estimated PSF may make an erroneous correction. Reduce the harmful effects that can occur due to the correction.

Specifically, it is calculated as follows. FIG. 4A is a diagram showing the characteristics (deterioration characteristics) of blurring and blurring estimated by the deterioration characteristic estimation unit 206. In FIG. 4A, the α-axis indicates the magnitude of blur, the β-axis indicates the magnitude of blur, and the γ-axis indicates the direction of blur. Further, the point P0 represents a characteristic evaluation value in the PSF estimation region, and the coordinates of the point P0 are (α0, β0, γ0). The point P1 represents a characteristic evaluation value in the target region for which the synthesis ratio is calculated, and the coordinates of the point P1 are (α1, β1, γ1). The synthesis ratio calculation unit 207 calculates the difference Δ between the deterioration characteristics such as blurring and blurring in the PSF estimation region and the deterioration characteristics in the synthesis ratio calculation target region using the following equation (7).

In the formula (7), a, b, and c are the magnitude of the blur, the magnitude of the blur, and the weight with respect to the direction of the blur, respectively. When the direction of blur is different, the shape of the corresponding PSF is significantly different from the case where the magnitude of blur or the magnitude of blur is different. Therefore, it is preferable that the weight c with respect to the difference in the blur direction takes a large value with respect to the weights a and b.

Next, a method of calculating the synthesis ratio will be described with reference to FIG. 4 (b). In FIG. 4B, the horizontal axis represents the difference Δ of the characteristic evaluation values of blurring and blurring, and the vertical axis represents the image composition ratio m. As shown in FIG. 4B, the larger the difference Δ of the characteristic evaluation values of blurring and blurring, the smaller the influence of the correction processing, that is, the lower the image composition ratio is set. Note that FIG. 4B shows a case where the composite ratio m is set to 1 when the difference Δ of the characteristic evaluation values is equal to or less than a predetermined threshold value, but the calculation method of the composite ratio m is limited to this. It's not something. For example, the composite ratio m may be set based on the characteristics of the curve set so that the composite ratio m becomes a lower value as the difference Δ of the characteristic evaluation values increases.

The composition ratio calculation unit 207 sequentially executes the above-mentioned calculation process of the composition ratio m for each region of the image, and outputs the calculated result to the image composition unit 205. As described above, the image composition unit 205 performs image composition processing based on the input composition ratio.

In the present embodiment, the image composition ratio in each region of the image is set based on the result of comparing the characteristics of blur and blur in each region of the image with the characteristics of blur and blur in the PSF estimation region. As a result, the corrected image is not used in the region where erroneous correction is performed when the correction based on the estimated PSF is applied. As a result, it is possible to reduce the harmful effects of the correction.

In the present embodiment, the case where three characteristic evaluation values of the size of the blur, the size of the blur, and the direction of the blur are used as the evaluation values representing the characteristics (deterioration characteristics) related to the blur and the blur has been described. The evaluation method of deterioration characteristics is not limited to this. For example, the PSF may be estimated in each region of the image, and the deterioration characteristics may be estimated based on the estimated shape of the PSF. In this case, the estimated shapes of the PSFs are compared, and if the shapes are close, an image corrected based on the PSF is used. On the other hand, when the shapes are different, the composition ratio is controlled so that the corrected image is not used. Here, since the PSF calculated in each region of the image is used for evaluating the deterioration characteristics, the estimation accuracy may be lower than in the case of estimating the PSF used for correcting the deterioration. Therefore, the amount of calculation may be reduced by relaxing the conditions for the convergence of PSF used in the iterative method.

Further, as an amount representing the blur characteristic, the edge signal included in the image signal may be focused on, and the deterioration characteristic may be estimated based on the gradient information of the change of the image signal at the edge portion. It can be estimated that the more gradual the change in the image signal at the edge portion, the greater the deterioration such as blurring and blurring. As a result, it is possible to appropriately control the correction of image deterioration even when it is not possible to acquire information on the distance information and the motion vector. Further, the deterioration characteristics of each of the plurality of regions of the image may be estimated based on the evaluation value for evaluating the in-focus state used in the AF control at the time of shooting the image.

Further, in order to evaluate the characteristics of the blur, the blur detection result (information about the blur) by the gyro (blurring detection unit) provided in the image pickup apparatus 100 may be used. If the detected blur is a blur that shifts in one direction, it can be estimated that the direction of the blur is uniform in the image. If the detected blur is a rotational blur, it can be estimated that the direction of the blur is non-uniform in the image. When the direction of blur is non-uniform, the correction process is controlled so that the correction based on PSF is not applied to the image area at a position different from the PSF estimation area.

In the present embodiment, the case where the image composition ratio is determined based on the amount (deterioration characteristic evaluation value) representing the characteristics (deterioration characteristics) such as blurring and blurring has been described, but the control method for correcting the deterioration characteristics is based on this. Not limited. For example, when correcting an image based on PSF, it may be determined whether or not to execute the correction process for each region based on the deterioration characteristics in each region of the image. In this case, it is sufficient to control so that the correction process is not executed in the region having the deterioration characteristic different from the deterioration characteristic in the PSF estimation region. Further, instead of synthesizing the captured image and the corrected image, a method of synthesizing the estimated PSF or a method of synthesizing the correction intensity or the correction amount determined based on the PSF can also be used. In this case, the composition ratio may be determined so that the correction based on PSF is not applied to the region having deterioration characteristics different from the PSF estimation region.

Although the present embodiment has described the case where the PSF estimation region is one location in the image, the method of PSF estimation processing is not limited to this. That is, the PSF may be estimated in a plurality of regions in the image. In this case, at the time of image composition, the PSF estimation region having the deterioration characteristic closest to the deterioration characteristic in the region to be processed is selected from the plurality of PSF estimation regions. Then, the corrected image corrected based on the estimated PSF in the selected PSF estimation region and the captured image before the correction may be combined. Alternatively, the PSFs may be estimated in a plurality of regions, and the corrected images corrected based on the respective PSFs may be combined. In this case, the synthesis ratio can be calculated as follows. First, among the plurality of PSF estimation regions, the PSF estimation region having the deterioration characteristics closest to the deterioration characteristics in the region to be processed is selected. Then, the compositing ratio may be set so that the compositing ratio of the corrected image corresponding to the selected PSF estimation region becomes high.

In the present embodiment, the case where the synthesis ratio is determined based on the absolute value of the difference of the deterioration characteristics (deterioration characteristic evaluation value) has been described, but the calculation method of the synthesis ratio is not limited to this. For example, in addition to the absolute value of the difference between the deterioration characteristic evaluation values, information on the magnitude relation can also be used. Specifically, when the deterioration characteristic evaluation value in the area to be processed is larger than the deterioration characteristic evaluation value in the PSF estimation area, the composition ratio with respect to the corrected image is set high, and when it is small, the composition ratio with respect to the correction image is set. It may be set low. This is because when a correction based on PSF is applied to a region where deterioration such as blurring and blurring is smaller than the PSF estimation region, there are many cases where noticeable adverse effects such as ringing occur.

Further, in addition to deterioration characteristics such as blurring and blurring, the composition ratio may be calculated using the result of detecting the subject in the image. Specifically, the correction image composition ratio is set low for a region in which the difference between the deterioration characteristic evaluation values is equal to or greater than a predetermined threshold value and belongs to a subject different from the PSF estimation region. As a result, it is possible to reduce the occurrence of adverse effects due to erroneous correction at the boundary between subjects.

Further, the composition ratio may be calculated by using the information of the edge signal included in each region of the image. Specifically, the composition ratio of the corrected image is set low in the region where the difference between the deterioration characteristic evaluation values is equal to or more than a predetermined threshold value and a large number of edge signals having a large amplitude are included. As a result, it is possible to reduce the occurrence of conspicuous adverse effects in the region containing many edges.

(Second Embodiment)
Next, with reference to FIGS. 5 and 6, a function related to correction processing for image deterioration such as blurring and blurring by the image processing unit 105a will be described. FIG. 5 is a block diagram of the image processing unit 105a. FIG. 6 is a flowchart of correction processing (image processing method of the present embodiment) by the image processing unit 105a. Each step of FIG. 6 is executed by each part of the image processing unit 105a.

The first embodiment has described the case where the image corrected based on the PSF and the captured image before the correction are combined. On the other hand, in the present embodiment, a plurality of PSFs relating to a plurality of regions are estimated, and an image correction process is performed using one of the estimated plurality of PSFs based on deterioration characteristics such as blurring and blurring of the image. explain. The image pickup apparatus (digital camera) of the present embodiment is different from the image pickup apparatus 100 of the first embodiment in that it has an image processing unit 105a instead of the image processing unit 105. Since other configurations of the image pickup apparatus in this embodiment are the same as those of the image pickup apparatus 100 described in the first embodiment with reference to FIG. 1, the description thereof will be omitted.

As shown in FIG. 5, the image processing unit 105a includes an image signal generation unit 201, an image correction unit 204, a PSF estimation area extraction unit 501a, 501b, a PSF estimation unit 502a, 502b, and a PSF selection unit 503. In the present embodiment, the image correction unit 204 and the PSF selection unit 503 constitute a correction processing unit.

In step S601 of FIG. 6, the image signal generation unit 201 generates image signals R, G, and B in the same manner as in step S301 of FIG. The image signal generation unit 201 outputs the generated image signal to the PSF estimation area extraction units 501a and 501b and the image correction unit 204.

Subsequently, in step S602, the PSF estimation region extraction units 501a and 501b select the PSF estimation regions a and b, respectively. As shown in FIG. 5, the image processing unit 105a has a plurality of PSF estimation units 502a and 502b. Therefore, PSF estimation processing can be performed for a plurality of locations. For example, when the captured image includes a foreground region (main subject region) and a background region, which are the main subjects, the PSF estimation region is selected so as to estimate the PSF from each of the main subject region and the background region. Alternatively, when a plurality of main subjects exist as in the case where a plurality of people are lined up, the PSF estimation area may be selected from each main subject area. Alternatively, when the user selects the subject to be corrected via the operation unit 120, a plurality of parts of the subject area selected by the user may be selected as the PSF estimation area. For example, when the subject selected by the user is a person, the PSF estimation area is selected from each of the face area and the body area of the person. The PSF estimation region extraction units 501a and 501b extract the image signals of the selected PSF estimation regions a and b and output them to the PSF estimation regions 502a and 502b, respectively. Further, the PSF estimation area extraction units 501a and 501b output information on the position of the selected PSF estimation area to the PSF selection unit 503.

Subsequently, in step S603, the PSF estimation units 502a and 502b estimate the PSF based on the image signals of the PSF estimation area output from the PSF estimation area extraction units 501a and 501b, respectively (PSF estimation process). Then, the PSF estimation units 502a and 502b output the estimated PSFs to the image correction unit 204. Since the method of PSF estimation processing in this embodiment is the same as that of the first embodiment described in step S303 of FIG. 3, the description thereof will be omitted.

Subsequently, in step S604, the deterioration characteristic estimation unit 206 estimates the deterioration characteristics such as blurring and blurring in each region of the captured image, and calculates the deterioration characteristic amount (evaluation value related to the deterioration characteristics). Then, the deterioration characteristic estimation unit 206 outputs the calculated deterioration characteristic evaluation value to the PSF selection unit 503. Since the method of calculating the deterioration characteristic value is the same as that of the first embodiment described in step S305 of FIG. 3, the description thereof will be omitted.

Subsequently, in step S605, the PSF selection unit 503 was estimated by the PSF estimation units 502a and 502b as the PSF used for correction for each region of the image based on the deterioration characteristic evaluation value output from the deterioration characteristic estimation unit 206. Select one of the PSFs. Hereinafter, a method of selecting PSF will be described with reference to FIG. 7.

FIG. 7 is a diagram showing the characteristics of blurring and blurring (deterioration characteristic evaluation value) estimated by the deterioration characteristic estimation unit 206. In FIG. 7, the α-axis indicates the magnitude of blur, the β-axis indicates the magnitude of blur, and the γ-axis indicates the direction of blur. In FIG. 7, points Pa and Pb indicate characteristic evaluation values of blurring and blurring in the PSF estimation region selected by the PSF estimation region extraction units 501a and 501b, respectively. Point P1 indicates a characteristic evaluation value of blurring or blurring in the area to be processed.

The PSF selection unit 503 calculates the difference Δa between the points P1 and Pa and the difference Δb between the points P1 and Pb by the same method as described with reference to step S306 of FIG. Then, when Δa <Δb, the PSF selection unit 503 selects the PSF estimated by the PSF estimation unit 502a. On the other hand, when Δb <Δa, the PSF selection unit 503 selects the PSF estimated by the PSF estimation unit 502b. In the present embodiment, when both the differences Δa and Δb exceed a predetermined threshold value, the PSF selection unit 503 does not select any PSF so as not to perform the correction process based on the PSF. The PSF selection unit 503 sequentially executes the above-mentioned selection process for each area of the image, and outputs the result to the image correction unit 204.

Subsequently, in step S606 of FIG. 6, the image correction unit 204 calculates the image composition ratio in each region of the captured image. Subsequently, in step S607 of FIG. 6, the image correction unit 204 performs correction processing on the image signal generated by the image signal generation unit 201 based on the information regarding the PSF selected by the PSF selection unit 503. Then, the image correction unit 204 outputs the corrected image signal to the image composition unit 205. Since the method of correction processing is the same as that of the first embodiment described in step S304 of FIG. 3, the description thereof will be omitted.

In the present embodiment, the PSF used for the correction process in each region of the image is selected based on the result of comparing the characteristics of blur and blur in each region of the image with the characteristics of blur and blur in the PSF estimation region. This makes it possible to select and correct a PSF suitable for each region of the image when the characteristics of blurring and blurring in the image are not uniform. As a result, it is possible to reduce the harmful effects of the correction.

In the present embodiment, a method of selecting a plurality of PSF estimation regions and selecting the PSF estimated in any one of the PSF estimation regions has been described, but the PSF selection method is not limited to this. Absent. For example, in order to make the boundary for switching the selection of PSF inconspicuous, images corrected based on different PSFs may be combined in the boundary region and controlled so that the boundary becomes inconspicuous.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

According to each embodiment, an image processing device, an imaging device, an image processing method, a program, and a storage medium capable of appropriately correcting images having different deterioration characteristics for each region with a small amount of calculation are provided. Can be done.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

105 Image processing unit (image processing device)
203 PSF estimation unit 204 image correction unit (correction processing unit)
205 Image composition unit (correction processing unit)
206 Deterioration characteristic estimation unit 207 Composite ratio calculation unit (correction processing unit)

Claims (11)

And estimation tough you estimate the point spread function from the estimated region of the partial region of the image,
A correction processing unit that generates a corrected image by performing correction processing of the image based on the point spread function , and a correction processing unit.
A deterioration characteristic estimation unit that estimates the deterioration characteristics of the image for each of a plurality of regions of the image , which is different from the estimation region.
It has a compositing unit that synthesizes the image and the corrected image at a compositing ratio based on the deterioration characteristics.
The synthesis unit, based on said deterioration characteristic for each of the plurality of areas of said deterioration characteristic and the image for the previous Ki推constant region of said image, to calculate the composite ratio in each of the plurality of regions An image processing device characterized by. The image processing apparatus according to claim 1 , wherein the deterioration characteristic of the image is a characteristic related to blurring or blurring of the image. The deterioration characteristic estimation unit includes information on the distance to the subject included in the image, information on the motion vector of the subject, information on blurring from the blur detection unit, and the point spread estimated for each region of the image. The image processing apparatus according to claim 1 or 2 , wherein the deterioration characteristic is estimated based on at least one of the information about the function. The combining unit, among the plurality of regions, the region with the deterioration characteristic difference between the degradation characteristics relating to the estimated area is equal to or less than a predetermined threshold value, and 1, the mixing ratio of the corrected image, The image processing apparatus according to any one of claims 1 to 3 , wherein the composition ratio of the images is set to 0. The combining unit, among the plurality of regions, claim 1, wherein the higher the region having said deterioration characteristic difference is large between the degradation characteristics relating to estimation region, to reduce the mixing ratio of the corrected image The image processing apparatus according to any one of 3 to 3. The combining unit, out of the previous SL plurality of areas, the degradation, the mixing ratio of the corrected image of the degradation characteristics is less than the degradation characteristics before Ki推constant region regions, the degradation properties in the estimation region The image processing apparatus according to any one of claims 1 to 5 , wherein the image processing apparatus is made smaller than a region larger than the characteristic. The combining unit, out of the previous SL plurality of regions, the more rich region an edge signal, the image processing according to any one of claims 1 to 6, characterized in that to reduce the mixing ratio of the corrected image apparatus. An imaging unit that photoelectrically converts an optical image formed via an imaging optical system and outputs an image signal,
And estimation tough you estimate the point spread function from the estimated region of the partial region of the image based on the image signal,
A correction processing unit that generates a corrected image by performing correction processing of the image based on the point spread function , and a correction processing unit.
A deterioration characteristic estimation unit that estimates the deterioration characteristics of the image for each of a plurality of regions of the image , which is different from the estimation region.
It has a compositing unit that synthesizes the image and the corrected image at a compositing ratio based on the deterioration characteristics.
The synthesis unit, based on said deterioration characteristic for each of the plurality of areas of said deterioration characteristic and the image for the previous Ki推constant region of said image, calculating a synthesis ratio in each of the plurality of regions An imaging device as a feature. Estimating a point spread function from the estimated region of the partial region of the image,
A step of generating a corrected image by performing correction processing of the image based on the point spread function, and
A step of estimating the deterioration characteristics of the image for each of a plurality of regions of the image , which is different from the estimation region.
It has a step of synthesizing the image and the corrected image at a composition ratio based on the deterioration characteristics.
The synthesis ratio, and characterized in that based on said deterioration characteristic for each of the plurality of areas of said deterioration characteristic and the image for the previous Ki推constant region of the image, is calculated in each of the plurality of regions Image processing method to be performed. Estimating a point spread function from the estimated region of the partial region of the image,
A step of generating a corrected image by performing correction processing of the image based on the point spread function, and
A step of estimating the deterioration characteristics of the image for each of a plurality of regions of the image , which is different from the estimation region.
A program that causes a computer to execute a step of synthesizing the image and the corrected image at a composition ratio based on the deterioration characteristics.
The synthesis ratio, and characterized in that based on said deterioration characteristic for each of the plurality of areas of said deterioration characteristic and the image for the previous Ki推constant region of the image, is calculated in each of the plurality of regions Program to do. A storage medium for storing the program according to claim 10.

Images