JP2017010092A - Image processing apparatus, imaging device, image processing method, image processing program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that, when estimating blur from an area where the amount of change in signal in a blur image is small, can suppress a reduction in estimation accuracy.SOLUTION: An image processing apparatus comprises acquisition means that acquires a blur image and creation means that acquires at least part of a first blur estimation area in the blur image and creates estimated blur. The creation means creates the estimated blur by performing repeated arithmetic processing of repeating correction processing of determining a second blur estimation area on the basis of the amount of change in signal in the first blur estimation area and correcting blur included in information on a signal in the second blur estimation area to create information on a correction signal, and estimation processing of estimating blur on the basis of information on the signal and information on the correction signal.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing method for estimating blur, which is a degradation component acting on a blurred image, from a single blurred image.

  In recent years, with higher definition of display devices, higher quality of captured images is desired. In the photographed image, information on the subject space is lost due to deterioration factors such as aberration and diffraction of an optical system used for photographing or camera shake during photographing. For this reason, conventionally, a method has been proposed in which deterioration of a captured image based on these factors is corrected to obtain a higher quality image. Examples of such a method include a Wiener filter and a Richardson-Lucy method. However, with these methods, when the deterioration component (blur) acting on the image is not known, a high correction effect cannot be obtained.

  On the other hand, conventionally, a method for estimating a camera shake component from one image deteriorated due to camera shake has been proposed. Patent Document 1 discloses a method for estimating a camera shake component by using statistical information related to a known intensity gradient distribution of a natural image from a single image deteriorated by camera shake. Here, the natural image means an image that is seen naturally by a modern human being. For this reason, the natural image is not limited to an image showing a tree or an animal, but includes an image showing a person, a building, an electronic device, or the like. As a nature of a natural image, it is known that a histogram related to the intensity gradient of a signal (intensity gradient histogram) follows a heavy tailed distribution according to the intensity of the gradient. Patent Document 1 discloses a method for estimating a camera shake correction image from only a camera shake image by constraining the intensity gradient histogram of the image with the camera shake corrected to follow a heavy distribution. Then, a shake component is estimated based on a comparison result between the camera shake correction image and the camera shake image. At this time, the blur component estimation accuracy increases as more edges are included in the hand shake image.

U.S. Pat. No. 7,616,826

  However, the amount of edges in the blurred image depends on the structure of the subject being photographed. For this reason, there is not always an edge amount sufficient for estimation in the image. In particular, when the blur shape changes according to the area in the image (referred to as shift-variant), it is necessary to estimate the blur using only a specific partial area in the blurred image, so the edge amount is insufficient. It's easy to do. In regions where there are few signal changes such as edges and textures, the blur estimation accuracy is extremely reduced. When the estimation method of Patent Document 1 is used for such a region, a large error occurs in the estimated camera shake component. The same applies to diffraction, aberration, defocus, or disturbance (generally referred to as “blur” of these deterioration factors) other than camera shake.

  Therefore, the present invention provides an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can suppress a decrease in estimation accuracy when blur is estimated from an area where there is little signal change in a blurred image. .

  An image processing apparatus according to one aspect of the present invention includes an acquisition unit that acquires a blur image, and a generation unit that acquires at least a part of a first blur estimation region in the blur image and generates an estimated blur. The generation means determines a second blur estimation area based on the amount of signal change in the first blur estimation area, and corrects blur included in the information regarding the signal in the second blur estimation area. The estimation blur is generated by performing an iterative calculation process that repeats a correction process for generating information on the correction signal and an estimation process for estimating the blur based on the information on the signal and the information on the correction signal.

  An imaging apparatus according to another aspect of the present invention includes an imaging element that photoelectrically converts an optical image formed via an optical system and outputs an image signal, and an acquisition unit that acquires a blurred image based on the image signal. Generating at least a part of the first blur estimation area in the blurred image and generating an estimated blur, wherein the generation means is based on the amount of signal change in the first blur estimation area. Correction processing for determining a second blur estimation area, correcting blur included in the information regarding the signal in the second blur estimation area to generate information regarding the correction signal, information regarding the signal, and the correction signal The estimated blur is generated by performing an iterative calculation process that repeats the estimation process for estimating the blur based on the information on the information.

  An image processing method according to another aspect of the present invention includes a step of acquiring a blur image, and acquiring at least a part of a first blur estimation region in the blur image and generating an estimated blur. In the step of generating the estimated blur, a second blur estimation area is determined based on an amount of signal change in the first blur estimation area, and the blur included in the information regarding the signal in the second blur estimation area is determined. The estimation blur is performed by performing an iterative calculation process that repeats the correction process for generating the information related to the correction signal by correcting the correction and the estimation process for estimating the blur based on the information related to the signal and the information related to the correction signal. Generate.

  A program according to another aspect of the present invention is configured to cause a computer to execute the image processing method.

  A storage medium according to another aspect of the present invention stores the program.

  Other objects and features of the present invention are illustrated in the following examples.

According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can suppress a decrease in estimation accuracy when blur is estimated from an area where a signal change in a blurred image is small. can do.

6 is a flowchart illustrating an image processing method according to the first and third embodiments. 1 is a block diagram of an image processing system in Embodiment 1. FIG. 1 is an external view of an image processing system in Embodiments 1 and 2. FIG. 6 is a flowchart illustrating a method for generating estimated blur in the first and third embodiments. In Examples 1-3, it is a figure which shows the relationship between a 1st blur estimation area | region and a 2nd blur estimation area | region. In Examples 1-3, it is a figure which shows the relationship between the 1st blur estimation area | region at the time of estimating a defocus, and a 2nd blur estimation area | region. 6 is a block diagram of an image processing system in Embodiment 2. FIG. 10 is a flowchart illustrating an image processing method according to the second exemplary embodiment. FIG. 6 is a block diagram of an imaging system in Embodiment 3. FIG. 6 is an external view of an imaging system in Embodiment 3.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  In the present embodiment, the factors that degrade the information of the subject space are collectively referred to as “blur”, but the factors of blur include diffraction, aberration, defocus, blurring, and disturbance. Before describing the present embodiment, these blurring factors will be described in detail.

  Diffraction is degradation caused by diffraction, which occurs in the optical system of the imaging device that captured the image. This occurs because the aperture diameter of the optical system is finite.

  Aberration is deterioration caused by deviation from an ideal wavefront generated in an optical system. Aberrations that occur due to design values of the optical system are called design aberrations, and aberrations that occur due to optical system manufacturing errors and environmental changes are called error aberrations. The environmental change indicates changes in temperature, humidity, atmospheric pressure, etc., and the performance of the optical system also changes in accordance with these changes. When referring simply to aberrations, both design aberrations and error aberrations are included.

  Defocus is deterioration caused by the fact that the focus of the optical system and the subject do not match. A case where defocusing extends over the entire image is called out-of-focus, and a case where defocusing is only part of the image is called defocusing blur. Specifically, defocus blur refers to a component that degrades background information when the main subject and background at different distances exist in the image and the main subject is in focus. The magnitude of this degradation varies depending on how far the background is from the main subject in the depth direction. Defocus blur is used, for example, as an expression method that makes a main subject in an image stand out. In the case of simply defocusing, both defocusing and defocus blur are included.

  The blur is deterioration caused by a change in the relative relationship (position and angle) between the subject and the imaging apparatus during exposure during shooting. Among the blurs, those that degrade the entire image are called camera shakes, and those that partially degrade the image are called subject blurs. In the case of simply blurring, both camera shake and subject blur are included.

  Disturbance is deterioration caused by fluctuation of a substance existing between a subject and an imaging device during photographing. For example, atmospheric fluctuations and water fluctuations in underwater photography can be mentioned. When disturbance occurs during short-second exposure, the straight line in the subject space becomes a curved curve in the blurred image. In order to correct this bending, a plurality of frame (or continuous shooting) images having different disturbance components may be synthesized. However, in such a synthesis process, even if the edge curvature can be corrected, the deterioration of the frequency component remains. The blurred image handled here is not only an image (long-second exposure image) in which the degradation of the frequency component due to the disturbance occurred during one exposure, but also a composite of multiple frames (or continuous shots) as described above. Includes images.

  In the present embodiment, the point image intensity distribution is referred to as PSF (Point Spread Function). In this embodiment, processing for a one-dimensional (monochrome) image will be described. However, the present invention can be similarly applied to a multi-dimensional (for example, RGB) image, and processing may be performed for each of RGB channels. If there is no difference between channels or the blur is negligible, the number of channels may be reduced (eg, RGB is converted into monochrome) and processed. When each channel acquires a different wavelength, blurring is an example of a blur that has no difference between channels.

  On the other hand, the aberration, diffraction, defocus, and disturbance change in blur depending on the wavelength. Here, even when the defocus is far away from the focus position, the spread of blur varies depending on the wavelength due to the influence of axial chromatic aberration. However, even for these blurs having wavelength dependency, if the performance difference between channels is sufficiently small with respect to the sampling frequency of the image sensor used for imaging, it may be considered that the difference between channels can be ignored. However, when estimating blur where the difference between these channels can be ignored, it is preferable to estimate one blur with a plurality of channels, rather than performing the above-described process of reducing the number of channels. Since blur estimation is performed using information related to the signal gradient of the image, the estimation accuracy improves as the information increases. In other words, signal gradient information increases when estimating with multiple images without reducing the number of channels (however, if the images of each channel match their proportional multiples, the signal gradient information does not increase) The blur can be estimated with higher accuracy. In the present embodiment, an example in which wavelengths with different channels are photographed is given, but other parameters (polarized light, etc.) may be considered in the same manner.

  Here, before describing specific examples, a brief overview of the present embodiment will be described. At the time of blur estimation, a blur estimation area for estimating blur from a blur image is obtained, and by comparing a signal gradient in the blur estimation area with a corrected signal gradient in which the blur of the signal gradient is corrected, Generate an estimated blur. At this time, if the amount of signal change (edge, texture, etc.) (signal change amount) in the blur estimation region is small, the estimation accuracy of the estimated blur is lowered. Therefore, first, the first blur estimation area is acquired from the blur image, and attention is paid to the amount of signal change included in the first blur estimation area. When the amount of signal change is small, the estimation accuracy decreases, and therefore, a second blur estimation area having a larger signal change amount than the first blur estimation area is acquired. The first blur estimation area and the second blur estimation area are different from each other in at least one of position and shape. By using the second blur estimation region having a large amount of signal change for blur estimation, it is possible to improve blur estimation accuracy.

  Here, when the blur changes according to the position in the blur image (in the case of Shift-variant), the estimated blur is the same as the original blur in the first blur estimation region by changing the region used for blur estimation. Is different. Shift-variant blur includes diffraction, aberration, defocus, blurring with rotation or position change in the depth direction, and disturbance when vignetting cannot be ignored. However, most of the shift-variant blur has a property that the blur continuously changes with respect to the position change on the blur image. For this reason, when the second blur estimation area is close to the first blur estimation area, there is no significant difference regarding the estimated blur. Since this difference is smaller than the estimation error that occurs when the signal change amount is insufficient, the error of the estimation blur is reduced as a result by using the second blur estimation region.

  If the signal change amount in the first blur estimation area is sufficient for blur estimation, the first blur estimation area may be set as the second estimation area as it is. For this reason, the signal change amount in the second blur estimation region is always greater than or equal to the signal change amount in the first blur estimation region.

  First, the image processing system according to the first embodiment of the present invention will be described with reference to FIGS. FIG. 2 is a block diagram of the image processing system 100 in the present embodiment. FIG. 3 is an external view of the image processing system 100.

  The image processing system 100 includes an imaging device 101, a recording medium 102, a display device 103, an output device 104, and an image processing device 105. The image processing apparatus 105 includes a communication unit 106, a storage unit 107, and a blur correction unit 108 (image processing unit). The blur correction unit 108 includes an acquisition unit 1081 (acquisition unit), a generation unit 1082 (generation unit), and a correction unit 1083 (correction unit). Each unit of the blur correction unit 108 executes the image processing method of the present embodiment as will be described later. Note that the image processing apparatus 105 (blur correction unit 108) may be provided inside the imaging apparatus 101.

  The imaging apparatus 101 includes an optical system 1011 (imaging optical system) and an imaging element 1012. The optical system 1011 forms an image of light rays from the subject space on the image sensor 1012. The image sensor 1012 has a plurality of pixels, photoelectrically converts an optical image (subject image) formed via the optical system 1011, and outputs an image signal. The imaging device 101 generates a captured image (blurred image) based on the image signal output from the image sensor 1012. The blurred image obtained by the imaging device 101 is output to the image processing device 105 via the communication unit 106. Regarding the blurred image, information on the subject space is deteriorated by the action of at least one of a plurality of types of blur as described above.

  The storage unit 107 stores a blurred image input to the image processing apparatus 105 and information regarding shooting conditions when the blurred image is shot. Here, the shooting conditions are the focal length, aperture, shutter speed, ISO sensitivity, and the like of the imaging apparatus 101 at the time of shooting. The blur correction unit 108 estimates and corrects a specific blur component based on the blur image, and generates a blur correction image. The blur correction image is output to any one or more of the display device 103, the recording medium 102, and the output device 104 via the communication unit 106. The display device 103 is, for example, a liquid crystal display or a projector. The user can perform work while confirming an image being processed via the display device 103. The recording medium 102 is a semiconductor memory, a hard disk, a server on a network, or the like. The output device 104 is a printer or the like. The image processing apparatus 105 may have a function of performing development processing and other image processing as necessary.

  In order to implement the image processing method of the present embodiment, software (image processing program) can be supplied to the image processing apparatus 105 via a network or a storage medium such as a CD-ROM. At this time, the image processing program is read by the computer (or CPU, MPU, etc.) of the image processing apparatus 105 and executes the function of the blur correction unit 108.

  The blur correction unit 108 performs image processing shown in the flowcharts of FIGS. 1 and 4. Before explaining the image processing, a method for correcting the blur of the signal gradient in the blur estimation region (at least a part of the blur image) (correction processing) and a method for estimating the blur (estimation processing) will be described. In the present embodiment, the signal gradient is information related to the signal in the blur estimation region, and information related to the image itself (the luminance distribution of the image) or information related to the n-th derivative (n is an integer including 0) (related to the differential value). Information).

  In this embodiment, the blur (deterioration component) is expressed in the form of PSF, but is not limited to this, and may be expressed in the form of, for example, OTF (Optical Transfer Function). First, an appropriate initial value of PSF (Gaussian distribution, one-dimensional line, etc.) is given, and the signal gradient in the second blur estimation region is corrected with the PSF to generate a corrected signal gradient. Subsequently, the PSF is estimated based on the signal gradient and the correction signal gradient. Using the estimated PSF, the signal gradient in the second blur estimation region is corrected again to generate a new corrected signal gradient, and the PSF is estimated. By repeating this (loop processing), it is possible to estimate the blur only from the blur image (more precisely, the second blur estimation region that is at least a part of the blur image).

  Next, the signal gradient blur correction method (correction process) will be specifically described. The relationship between the second blur estimation area and the blur is expressed by the following equation (1).

In the formula (1), _b-i signal distribution of the second blur estimation region in the i-th loop, _{k i} is blurred, _{a i} no degradation due to blur _{k i} signal distribution, _{n i} is the noise . “*” Represents a convolution operation.

By solving the optimization problem is represented by the following equation (2), _{(corresponding} to a correction signal distribution) estimate _d-i of a _i in the i-th loop process to estimate.

In Expression (2), L is a loss function, Φ is a regularization term for d _i , and specific examples of each will be described later. The loss function L has an effect of fitting the solution to a model (here, the equation (1) is indicated). The regularization term Φ has the effect of converging the solution to a plausible value. For the regularization term, a property that the solution (a _i ) called prior knowledge should have is used. Moreover, regularization term has a role of preventing excessive fitting experience when considering only loss function (to the influence of noise n _i will be reflected to d _i). When the second blur estimation area is acquired from a plurality of channels (for example, RGB), Equation (2) is solved for each channel.

  In this embodiment, the correction signal gradient is obtained by solving Equation (2), but other methods may be used. For example, a method using an inverse filter such as a Wiener filter or a super-resolution method such as a Richardson-Lucy method can be used. Further, a correction signal gradient may be generated by applying a sharpening filter such as a shock filter in combination with an inverse filter or a super-resolution technique. Furthermore, when a sharpening filter is used in combination, it is preferable to suppress noise and ringing by using a smoothing filter that maintains edge resolution such as a bilateral filter or a guided filter.

  Next, specific examples of the loss function L and the regularization term Φ in Expression (2) will be described. As the loss function L, a function represented by the following equation (3) is conceivable.

In the expression (3), a symbol represented as the following expression (4) represents a p-order average norm, and in the case of p = 2, represents a Euclidean norm.

  As an example of the regularization term Φ, there is a first-order average norm represented by the following equation (5).

In Equation (5), λ is a parameter that represents the weight of the regularization term Φ, and ψ is a function that represents a basis transformation for the image, examples of which include wavelet transformation and discrete cosine transformation. The regularization term Φ in equation (5) is expressed by a smaller number of signals because the image component is subjected to base transformation such as wavelet transformation or discrete cosine transformation, so that the signal component becomes sparse. Based on the nature of being able to. This is described, for example, in “Richard G. Baraniuk,“ Compressive Sensing ”, IEEE SIGNAL PROCESSING MAGAZINE [118] JULY 2007”. In addition, as examples of other regularization terms, a Tikhonov regularization term, a TV (Total Variation) norm regularization term, or the like may be used.

  In order to solve the estimation equation represented by Expression (2), which is an optimization problem, a technique using an iterative operation is used. For example, when a Tikhonov regularization term is employed, a conjugate gradient method or the like may be used. . In addition, when adopting the formula (5) or the TV norm regularization term, TwIST (Two-step Iterative Shrinkage / Thresholding) or the like may be used. Regarding TwIST, “J. M. Bioucas-Dias, et al.,“ A new TwIST: two-step iterative shrinkage / thresholding algorithms for image restoration ”, IEEE Trave. Explained.

  In addition, when performing these repetitive calculations, parameters such as regularization weights may be updated each time it is repeated. The expressions (1) and (2) are described for the image (signal distribution), but the same holds true for the differentiation of the image. Therefore, in place of the image, blur correction may be performed on the differentiation of the image (which is expressed as a signal gradient including both).

  Next, the blur estimation method (estimation process) will be specifically described. PSF can also be estimated using the following equation (6), as in equation (2).

When the second blur estimation area is acquired from a plurality of channels (the blur due to the channel is not changed or can be ignored), the formula (6) is transformed as shown in the following formula (6a). The

In the formula (6a), H denotes the total number of channels included in the second blur estimation _{region, d i, h} is _d at h-th channel i, _{v h} is the weight. In the case of signal gradient correction, equation (2) may be solved for each channel, but in blur estimation, the form is summarized for all channels as in equation (6a). This is because the target to be estimated is different for each channel in Equation (2), but is common in Equation (6a) (the same PSF).

  As the loss function L of the equations (6) and (6a), the following equation (7) can be considered.

In Expression (7), _{ｊ j} represents a differential operator. _{０ 0} (j = 0) is an identity operator, and ∂ _x and ∂ _y represent the horizontal and vertical differentiations of the image, respectively. Further, the higher-order differentiation is expressed as ∂ _xx or ∂ _xyy , for example. Note that the signal gradient in the blur estimation area includes all these j (j = 0, x, y, xx, xy, yy, yx, xxx,...). In this embodiment, j = x, y Think only. u _j is a weight.

  As the regularization term Φ in the equation (6), one that conforms to the property of the PSF to be estimated can be used. For example, when the PSF is in a collapsed form such as defocus, it is preferable to use TV norm regularization represented by the following equation (8).

In equation (8), ζ is the weight of the regularization term.

  In addition, when the PSF has a linear shape such as blurring, it is preferable to use a first-order average norm regularization represented by the following formula (9).

  Further, Tikhonov regularization may be used as a regularization term that is not dependent on the shape of the PSF, ensures a certain degree of accuracy, and can be solved at high speed.

  Next, the image processing method of the present embodiment will be described. In the present embodiment, the second blur estimation region is acquired according to the amount of signal change (edge, texture, or gradation) (signal change amount) in the first blur estimation region. If the amount of signal change in the first blur estimation area is an amount sufficient to perform blur estimation, the first blur estimation area is directly used as the second blur estimation area. On the other hand, when the amount of signal change in the first blur estimation region is insufficient, that is, when the amount of signal change is insufficient to perform blur estimation, the second amount of signal change is larger than that in the first blur estimation region. Get the blur estimation area. At this time, the second blur estimation area is different from the first blur estimation area in at least one of position and shape (including both shape and size). However, in the second blur estimation area, the blur acting in the second blur estimation area does not change so much from the blur acting in the first blur estimation area (substantially does not change). To be acquired). An estimated blur corresponding to the first blur estimation area is obtained by generating a correction signal gradient (correction process) and estimating a blur (estimation process) on the second blur estimation area.

Here, the effect of the amount of signal change in the blur estimation area for estimating blur on the blur estimation accuracy will be described in detail. As can be seen from the equation (1), the blur k _i that is an estimation target in blur estimation is expressed by a linear equation. Since this linear equation exists for the number of pixels in the blur estimation region b _i , estimation of k _i corresponds to a problem of solving simultaneous equations having the number of pixels of b _i . For this reason, if the number of independent linear equations is larger than the number of unknowns, the solution is uniquely determined. However, in general, the number of independent linear equations does not increase beyond the number of unknowns. This is because the blur k _i as well, the noise n _i also unknown (when determining the blur, in order to assign the value of d _i estimated in a _i, a _i is known). That is, the number of unknowns is equivalent to the sum of the number of components of the blur k _i, the number of components in the noise n _i and (equal to the number of pixels b _i), always (number of pixels b _i) number of linear equations is more than. Therefore, in order to determine the blur k _i, estimation processing is required.

Here, to suppress the influence of noise n _i, considered method of obtaining the blur k _i with high accuracy. First, Equation (1) is considered that noise is included in each of the simultaneous linear equations (the number is the number of pixels of b _i ) of the unknown number k _i (the number is the number of components of k _i ). However, the number of pixels of the _b-i is the larger number of components in k _i. If no noise is included, the number of simultaneous linear equations is larger than the number of unknowns, and some linear equations always have a dependency. That is, there are a plurality of the same equations. However, since there is actually noise, these equations are shifted by the amount of noise. On the other hand, noise can be suppressed by averaging signals having the same original signal but different noise values. For this reason, if there are many dependent linear equations with different ways of generating noise, blur estimation with the influence of noise suppressed by the effect of averaging becomes possible. When the noise suppression effect by this averaging is sufficiently acting, the regularization term Φ in the equations (6) and (6a) may be set to zero.

Also, to estimate the blur k _i with high accuracy, further conditions are necessary. That is, there are as many independent equations as possible in the simultaneous linear equations excluding noise. The number of independent equations than the number of components of the blur k _i (i.e. the number of unknowns) is too small, estimated it can not be obtained with sufficient accuracy. Since the coefficient of the simultaneous linear equations is the correction signal gradient d _i , the correction signal gradient d _i needs to have various combinations in order to increase the number of independent equations. In other words, the estimation accuracy improves as signal changes (edges, textures, or gradations) of various directions or intensities exist in the blur estimation area. For example, if it is assumed that the signal values of the blur estimation region b _i and the correction signal gradient d _i are constant (in the image, a completely flat portion), the simultaneous linear equations of Equation (1) are (excluding noise) ) Since all are the same, the blur k _i cannot be estimated.

  Summarizing the above discussion, if there are many signal changes with various directions and intensities in the blur estimation area, and there are multiple signals that have the same signal change but different noise appearance, the blur estimation accuracy Will improve. In order to realize this, in this embodiment, when the signal change (signal change amount) in the first blur estimation area is small, the second blur position is different from the first blur estimation area in the blur image. Get the blur estimation area. The second blur estimation area includes more signal changes such as edges than the first blur estimation area. Further, when the area of the second blur estimation area is larger than the area of the first blur estimation area, the number of linear equations (equations with different noise appearances) increases, so that a noise suppression effect can also be obtained. . However, as described above, when the blur is Shift-variant, the estimated blur is slightly different from the original blur due to a change in the blur estimation area (however, the estimation accuracy when the signal change amount is insufficient) The error is smaller than the decrease in. For this reason, the second blur estimation region having a different position or shape is acquired only when the signal change amount in the first blur estimation region is small. On the other hand, in other cases (when there is a sufficient signal change amount in the first blur estimation area), the first blur estimation area itself is set as the second blur estimation area.

  Next, image processing performed by the blur correction unit 108 will be described with reference to FIG. FIG. 1 is a flowchart illustrating an image processing method according to the present exemplary embodiment. Each step in FIG. 1 is executed by the acquisition unit 1081, the generation unit 1082, and the correction unit 1083 of the blur correction unit 108.

  First, in step S101, the acquisition unit 1081 of the blur correction unit 108 acquires a blurred image (captured image). Subsequently, in step S102, the generation unit 1082 of the blur correction unit 108 acquires distribution information regarding the amount of signal change (signal change amount) in the blurred image. The distribution information relating to the amount of signal change is information indicating in which position or region in the blurred image a signal change such as an edge exists and to what extent. As a method of acquiring distribution information regarding the amount of signal change, there is a method of acquiring a differential value of a blurred image (luminance distribution). As the absolute value of the differential value increases, the position where a large signal change occurs is indicated, and the accuracy of estimation is improved when the position is included in the blur estimation region.

  Subsequently, in step S103, the generation unit 1082 acquires a first blur estimation region from the blurred image. The first blur estimation region is a region where blur is to be estimated. The first blur estimation area may be an area specified by the user or automatically acquired. In this embodiment, a case where the blur is a shift-variant is handled. At this time, the first blur estimation area is acquired by, for example, dividing the blurred image into a plurality of partial areas and selecting one partial area from the plurality of divided partial areas. On the other hand, when the blur is constant in the entire blur image (referred to as Shift-invariant), the entire area of the blur image is set as the first blur estimation area. However, in this case, it is not possible to acquire from the blurred image the second blurred estimated area having a larger signal variation amount than the first blurred estimated area. For this reason, in this embodiment, a shift-variant blur will be described.

  Subsequently, in step S104, the generation unit 1082 acquires a second blur estimation area. The second blur estimation area is an area where the blur is actually calculated (estimated), and the estimated blur obtained in the second blur estimation area is output as blur acting on the first blur estimation area. . In the blur estimation, it is assumed that the same blur is acting in the second blur estimation area. The second blur estimation area is determined based on the signal change amount in the first blur estimation area. When the amount of signal change in the first blur estimation area is sufficient for blur estimation, the first blur estimation area is directly used as the second blur estimation area. On the other hand, when the amount of signal change in the first blur estimation area is insufficient for blur estimation, the second blur estimation is different in position or shape from the first blur estimation area so that the signal change amount increases. Get the area. Whether or not the signal change amount is sufficient for blur estimation can be determined, for example, based on whether or not the signal change amount is larger than a preset threshold value (first threshold value). This threshold is preferably determined from the number of blur components to be estimated (the number of unknowns). As the signal change amount in the first blur estimation region, for example, distribution information relating to the signal change amount calculated in step S102 may be used. Of this distribution information, only the value of the portion corresponding to the first blur estimation area is extracted. In order to compare this distribution information value (signal change amount) with a threshold value (first threshold value), it is necessary to convert the signal change amount from the distribution information to a scalar value. Examples thereof will be described below, but the present embodiment is not limited to these examples.

  The simplest method is to take the sum of the squares of differential values (may be absolute values) in the first blur estimation region. Preferably, the square of the differential value or the absolute value is quantized, and then the sum thereof is taken. When the sum is taken without quantization, it cannot be clearly distinguished whether there are many signal changes or only a very small amount of signal change. In the former case, there are many variations in the direction and strength of edges and textures. On the other hand, in the latter case, there are few such variations. If there are few variations such as edges and textures, the number of independent equations among the simultaneous linear equations expressed by Equation (1) becomes small, so that the blur estimation accuracy decreases. For this reason, it is preferable to be able to distinguish whether there are signal changes in a plurality of regions within the first blur estimation region or only a small amount of very large signal changes. This distinction can be realized by performing quantization. For example, when the square (or absolute value) of the differential amount is larger than a predetermined threshold (third threshold), it is 1, and when it is smaller than the predetermined threshold (in the case of a flat portion), it is 0. Take the sum after binarization. Thereby, it can be distinguished whether there are many edges and textures. Here, a binarization technique is used as quantization, but quantization may be performed with a larger number of bits.

  The second blur estimation area having a signal change amount larger than that of the first blur estimation area is acquired as shown in FIG. 5, for example. FIG. 5 is a diagram illustrating a relationship between the first blur estimation area and the second blur estimation area. FIG. 5A shows a state in which a second blur estimation region 203 a having a position different from that of the first blur estimation region 202 is acquired in the blur image 201. As to which position in the blurred image 201 the second blur estimation region 203a is acquired to obtain a larger amount of signal change, the distribution information relating to the signal change amount of the blurred image acquired in step S102 in FIG. It can be determined on the basis of. However, if the blur acting on the second blur estimation area is significantly different from the blur of the first blur estimation area, the accuracy of the estimation blur is lowered. For this reason, the second blur estimation area is acquired so that the blurs in the first blur estimation area and the second blur estimation area are substantially the same. Most blurs change continuously according to their position on the blurred image. For this reason, it is preferable that the second blur estimation area is acquired so as to include half or more of the area of the first blur estimation area. Thereby, a change in blur due to a position change in the blurred image can be suppressed. Therefore, the distribution information relating to the amount of signal change used when determining the second blur estimation area does not need to be acquired from the entire blur image as in step S102, and only around the first blur estimation area. You can get about.

  FIG. 5B shows a state where a second blur estimation area 203b having a shape (both shape and size) different from that of the first blur estimation area 202 is acquired in the blur image 201. . It can be determined from the distribution information regarding the amount of signal change in the blurred image acquired in step S102 as to what shape the second blur estimation region 203b can be made to obtain a larger amount of signal change. As described above, as the blur estimation area is larger (the number of linear equations is larger), it is possible to suppress a decrease in the accuracy of blur estimation. For this reason, as shown in FIG. 5B, the second blur estimation region 203b is preferably acquired so that the area thereof is larger than the area of the first blur estimation region 202. At this time, it is more preferable that the second blur estimation area is acquired so as to include all the first blur estimation areas so that blurs in the first blur estimation area and the second blur estimation area are substantially the same. . More preferably, the center of the second blur estimation area and the center of the first blur estimation area coincide with each other. Thereby, the deviation of the blur in the first blur estimation area and the second blur estimation area can be suppressed.

  In FIGS. 5A and 5B, the case where the position and the shape are different is shown individually. However, the second blur estimation area is the first blur estimation area with respect to both the position and the shape. And may be different. Further, the positions and shapes of the first blur estimation area and the second blur estimation area are not limited to those shown in FIGS. 5 (A) and 5 (B).

  Next, of the shift-variant blur, a second blur estimation area in defocusing with low continuity due to a position change in the blurred image will be described. Although the defocus blur continuously changes in the depth (depth) direction of the subject space, generally, the depth does not continuously change for the subject group within the shooting angle of view. In particular, the boundary between subjects existing at different depths has a large discontinuity. For this reason, when defocusing is estimated, it is preferable to obtain depth information (distance information) of the subject space and use it to determine the second blur estimation region. The depth information can be acquired by providing the imaging apparatus 101 with a distance measuring unit using a multi-view imaging system such as a multi-view camera, a TOF (Time of Flight) sensor, or a laser.

  Here, with reference to FIG. 6, a method for obtaining the second blur estimation area will be described. FIG. 6 is a diagram illustrating the relationship between the first blur estimation area and the second blur estimation area when defocusing is estimated. In FIG. 6, the depth information 204 represents the depth distribution of the subject space corresponding to the blurred image in shades. In FIG. 6, the first blur estimation area 202 and the second blur estimation area 203c are drawn in the depth information 204, but both are actually acquired from the blur image. The depth information 204 is used to determine the position and shape of the second blur estimation area 203c. As shown in FIG. 6, when the first blur estimation area 202 is set, the second blur estimation area 203c is set so as to include only an area whose depth is close to that of the first blur estimation area 202. By excluding a region where the depth changes greatly (including the boundary of the subject), it is possible to reduce a change in blur in the first blur estimation region and the second blur estimation region.

  The defocus blur shape is determined according to the pupil of the optical system 1011 in the imaging apparatus 101. In FIG. 6, the second blur estimation area 203 c is limited only to the vicinity of the first blur estimation area 202 in consideration of the vignetting effect of the optical system 1011. However, when the influence of the vignetting of the optical system 1011 can be ignored, the defocus blur changes only in accordance with the depth. In this case, all of the depth regions close to the first blur estimation region can be set as the second blur estimation region. Alternatively, if the depth is close to the first blur estimation area, the second blur estimation area may be set so as not to include the area at all.

  As a method for determining whether or not the blur of the target to be estimated is defocused, for example, there are the following methods. First, the user himself / herself designates the estimation target as defocus. For automatic determination, information on the frequency characteristics of the blurred image and the shooting conditions of the imaging apparatus 101 is used. When the F value is very large, it is considered that the user intends a pan-focus image, so defocus blur is estimated. Further, when the high-frequency component of the blurred image is isotropically missing, it can be determined that a focus shift has occurred.

  Subsequently, in step S105 of FIG. 1, the generation unit 1082 generates an estimated blur from the second blur estimation area. Details of step S105 are shown in the flowchart of FIG. 4, and the description thereof will be described later.

  Subsequently, in step S106, the generation unit 1082 determines whether or not the generation of the estimated blur has been completed for a predetermined region (for example, the entire blurred image) in the blurred image. If the generation of the estimated blur has been completed, the process proceeds to step S107. On the other hand, if the generation of the estimated blur has not been completed, the process returns to step S103. Then, the generation unit 1082 acquires a partial area where no estimated blur is generated from a predetermined area in the blurred image as a new first blur estimated area.

  Subsequently, in step S107, the generation unit 1082 outputs the generated estimated blur. When there are a plurality of blur estimation areas as the first blur estimation area, since there are a plurality of estimation blurs, all of them are output. In this embodiment, the estimated blur is PSF data. However, the present embodiment is not limited to this, and may be output in the form of coefficient data obtained by fitting OTF, PSF or OTF with a predetermined basis, or an image obtained by converting PSF or OTF into image data. Good.

  Subsequently, in step S108, the correction unit 1083 of the blur correction unit 108 corrects the blur of the blurred image using the output estimated blur. For the correction, for example, a method using an inverse filter such as a Wiener filter or a super-resolution method such as a Richardson-Lucy method can be used. When there are a plurality of estimated blurs, the correction unit 1083 corrects the first blur estimation region from which each corresponding estimated blur is obtained using the estimated blur. Further, before all the estimated blurs are output, the blur of the first blur estimation area is sequentially corrected using the estimated blur generated in step S105, and finally, the blur is corrected by combining them. An image may be generated.

  Next, the estimated blur generation process (step S105) will be described in detail with reference to FIG. FIG. 4 is a flowchart illustrating a method for generating estimated blur. Each step in FIG. 4 is executed by the generation unit 1082 of the blur correction unit 108.

  First, in step S201, the generation unit 1082 performs denoising of the second blur estimation area. The denoising of the second blur estimation region is performed in order to reduce deterioration in blur estimation accuracy due to the presence of noise in the second blur estimation region. Instead of step S201, a step of denoising the entire blurred image may be inserted before step S103 in FIG. As a denoising method, there is a method using a bilateral filter, an NLM (Non Local Means) filter, or the like.

  Preferably, the generation unit 1082 denoises the second blur estimation area (or blur image) by the following method. First, the generation unit 1082 generates a frequency-resolved blur estimation region by performing frequency decomposition on the second blur estimation region. Then, the generation unit 1082 denoises the frequency-resolved blur estimation area based on the amount of noise in the second blur estimation area. Next, the generation unit 1082 obtains a second blur estimation area with reduced noise by recombining the frequency-resolved blur estimation area. In general, the denoising process has a problem of reducing noise from an image and blurring the image. If the second blur estimation area is blurred by denoising, a PSF in which the blur that originally deteriorated the image and the blur due to denoising is mixed is estimated when performing the subsequent estimation process. For this reason, it is preferable to use a denoising technique with a small blur on the image. As such a technique, a denoising process using frequency decomposition of an image is applied. Here, an example using wavelet transform as frequency decomposition will be described. The details are described in “Donoho DL,“ De-noising by soft-thresholding ”, IEEE Trans. On Inf. Theory, 41, 3, pp. 613-627.

  The wavelet transform is a transform in which a frequency analysis is performed for each position of an image using a localized small wave (wavelet) and a signal is decomposed into a high frequency component and a low frequency component. In the wavelet transform of an image, the wavelet transform is performed on the horizontal direction of the image to decompose it into a low frequency component and a high frequency component. Do. By the wavelet transform, the image is divided into four and is frequency-resolved into four subband images having different frequency bands. At this time, the subband image of the upper left low frequency band component (scaling coefficient) is LL1, and the subband image of the lower right high frequency band component (wavelet coefficient) is HH1. The upper right (HL1) and lower left (LH1) subband images are obtained by taking a high frequency band component in the horizontal direction and taking out a low frequency band component in the vertical direction, and taking a low frequency band component in the horizontal direction, respectively. A high frequency band component is extracted in the vertical direction.

  Further, when the subband image LL1 is wavelet transformed, the image size can be halved and decomposed into subband images LL2, HL2, LH2, and HH2, and the conversion level for the subband image LL obtained by the decomposition can be reduced. It can be disassembled as many times as possible.

Thresholding is known as a method for performing noise reduction processing using wavelet transform. In this method, a component having an amount smaller than a set threshold value is regarded as noise, and the noise is reduced. The threshold processing on the wavelet space is performed on the subband images other than the subband image LL, and the wavelet coefficient w having an absolute value equal to or smaller than the threshold is represented by the following equation (10). _Denoising is performed by replacing _subband (x, y) with 0.

In Equation (10), x and y are the vertical and horizontal coordinates of the image, ρ _subband is a weight parameter, and σ is a standard deviation of noise. The amount of noise σ included in the second blur estimation area is obtained by measuring or estimating from the second blur estimation area. When the noise is white Gaussian noise that is uniform in real space and frequency space, there is a technique for estimating the noise in the second blur estimation region from MAD (Media Absolute Deviation) as shown in the following equation (11). Are known.

The MAD is obtained by using the median (median value) of the wavelet coefficient w _HH1 in the subband image HH1 obtained by wavelet transform of the second blur estimation region. Since the standard deviation and the MAD have the relationship shown in the following formula (12), the standard deviation of the noise component can be estimated.

In place of the equations (11) and (12), the noise amount σ may be acquired based on the shooting condition (ISO sensitivity at the time of shooting).

  Subsequently, in step S202, the generation unit 1082 reduces the resolution of the second blur estimation area and generates a low-resolution blur estimation area. By reducing the resolution of the second blur estimation area, the resolution of the blur to be estimated is similarly reduced. As a result, the convergence of the blur estimation described later can be improved, and the possibility that the estimation result falls into a local solution different from the optimal solution can be reduced. The rate of lowering the resolution of the second blur estimation area is determined according to the downsampling parameter. Step S202 is executed a plurality of times by loop processing (repetitive calculation), but a predetermined downsampling parameter is used at the first execution. In the second and subsequent executions, a low-resolution blur estimation region is generated using the downsampling parameters set in step S208 described later. Each time the loop is repeated, the amount of decrease in resolution decreases, and the resolution of the low-resolution blur estimation area gradually approaches the second blur estimation area. That is, at first, low-resolution blur is estimated, and the estimation result is used as a new initial value, and the estimation is repeated while gradually increasing the resolution. Thereby, it is possible to estimate the optimum blur while avoiding the local solution. Note that the resolution of the low-resolution blur estimation area is equal to or lower than the resolution of the second blur estimation area, and the resolutions of both may be the same.

  Subsequently, in step S203, the generation unit 1082 corrects the blur of the signal gradient (information about the signal) in the low-resolution blur estimation region, and generates a correction signal gradient (information about the correction signal) (correction processing). As described above, the correction using the equation (2), other super-resolution, an inverse filter, and the like are used to generate the correction signal gradient. The PSF used for blur correction uses the result (blur) estimated in step S204 in the previous loop. At the first time of the loop, an appropriately shaped PSF (Gauss distribution, one-dimensional line, etc.) is used. Subsequently, in step S204, the generation unit 1082 estimates blur based on the signal gradient and the correction signal gradient in the low-resolution blur estimation region (blur estimation process). For the estimation of the blur, optimization by the formula (6) or the formula (6a) is used.

  Subsequently, in step S205, the generation unit 1082 estimates the blur estimated in step S204 (second estimation process) (second blur) and the previous loop in step S204 (first estimation process). Compare with blur (first blur). Specifically, the generation unit 1082 determines whether or not the difference between the first blur and the second blur is equal to or less than a predetermined value (second threshold). If this difference is less than or equal to the predetermined value, the process proceeds to step S207. On the other hand, if the difference is greater than the predetermined value, the process proceeds to step S206. As described in step S202, in the present embodiment, the correction signal gradient generation (correction processing) and blur estimation (estimation processing) are looped while changing the resolution in the second blur estimation region. Since the low-resolution blur estimation region at the beginning of the loop has a small number of pixels, the blur estimation result is likely to converge. On the other hand, if the number of pixels in the low-resolution blur estimation area increases by repeating the loop, the result becomes difficult to converge. Further, if the signal change such as the edge is insufficient, the estimated blur is greatly deviated from the correct value. In order to avoid this, the generation unit 1082 evaluates the difference between the blur estimated in the current loop and the blur in the previous loop. Here, one of the blur in the current loop and the previous loop is resized to the same number of components as the other by bilinear interpolation, and the difference is evaluated by taking the sum of squares of each component. However, the present embodiment is not limited to this, and other methods may be used. When the difference between the two blurs is larger than a predetermined value, it is considered that the estimation accuracy is lowered due to insufficient signal change information necessary for blur estimation in the current loop. For this reason, the generation unit 1082 expands the second blur estimation area in step S206 and performs blur estimation again. In the first loop, since there is no estimated blur of the previous loop, the process proceeds to step S207 without executing step S205.

  In step S206, the generation unit 1082 increases the area of the second blur estimation region. The reason is as described above. Then, returning to step S202, the generation unit 1082 generates a low-resolution blur estimation region in the new second blur estimation region, and generates a correction signal gradient. At this time, it is preferable to perform the denoising described in step S <b> 201 also on the second blur estimation region having an enlarged area.

  In step S207, the generation unit 1082 determines whether or not the iterative (loop) calculation is completed, that is, whether or not the resolution difference between the second blur estimation area and the low-resolution blur estimation area is smaller than a predetermined value. judge. This determination is performed by comparing the resolution of the second blur estimation area with the resolution of the low-resolution blur estimation area. When the resolution difference between the two is smaller than a predetermined value, the iterative calculation is terminated. Then, the blur estimated in step S204 is regarded as the final estimated blur, and the process proceeds to step S209. Whether or not the resolution difference between the two is smaller than a predetermined value is not limited to whether or not the absolute value of the resolution difference between the two is smaller than the predetermined value, and the absolute value of the difference in the number of pixels between the two is a predetermined value. It may be determined based on whether or not the ratio is smaller, or whether the ratio of the number of pixels is closer to 1 than a predetermined value. When the predetermined condition is not satisfied (for example, when the absolute value of the resolution difference between the two is larger than the predetermined value), the estimated blur resolution is still insufficient, so the process proceeds to step S208, and the iterative calculation is performed.

  In step S208, the generation unit 1082 sets downsampling parameters used in step S202. In order to increase the resolution in the process of repeating steps S202 to S206, parameters are set so that the degree of downsampling is weaker than that of the previous loop (so that the amount of resolution decrease is reduced). In the iterative calculation, the blur estimated in step S204 of the previous loop is used as a new initial solution. At this time, it is necessary to increase the resolution of the blur. In order to improve the resolution, it is preferable to use bilinear interpolation or bicubic interpolation.

  In step S209, the generation unit 1082 performs denoising of the blur (estimated blur) estimated in step S204. As shown in Expression (1), since noise exists in the blurred image (second blur estimation region), noise is also generated in the estimated PSF due to the influence. Therefore, the generation unit 1082 performs a denoising process. When performing the denoising process, it is preferable to use a thresholding process, an opening process for removing isolated points, or a denoising process using a smoothing filter or the above-described frequency decomposition.

  According to the present embodiment, it is possible to provide an image processing system capable of suppressing a decrease in estimation accuracy when blur is estimated from an area where a signal change of a blurred image is small.

  Next, an image processing system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 7 is a block diagram of the image processing system 300 in the present embodiment.

  The image processing system 300 according to the present embodiment includes an image processing device 305 having an estimated blur generation unit 308 instead of the image processing device 105 having the blur correction unit 108. Is different. The image processing system 300 according to the present exemplary embodiment can perform blur estimation by a simpler process than the image processing system 100 according to the first exemplary embodiment. The estimated blur generation unit 308 estimates and outputs a blur component from the blur image captured by the imaging device 101. Since other parts of the image processing system 300 are the same as those of the image processing system 100 of the first embodiment, description thereof is omitted.

  Next, image processing performed by the estimated blur generation unit 308 will be described with reference to FIG. FIG. 8 is a flowchart showing the image processing method of the present embodiment. Each step in FIG. 8 is executed by the acquisition unit 3081 (acquisition unit) and the generation unit 3082 (generation unit) of the estimated blur generation unit 308.

  First, in step S301, the acquisition unit 3081 of the estimated blur generation unit 308 acquires a blurred image. Note that step S301 is the same as step S101 of the first embodiment described with reference to FIG. Subsequently, in step S302, the generation unit 3082 of the estimated blur generation unit 308 acquires a first blur estimation region and a signal change amount (signal change amount) in the first blur estimation region from the blur image. A specific description of the amount of signal change is the same as that in step S104 in the first embodiment.

  Subsequently, in step S303, the generation unit 3082 acquires a second blur estimation area from the blur image. If the signal change amount acquired in step S302 is equal to or greater than the threshold (first threshold), the generation unit 3082 sets the first blur estimation area as the second blur estimation area. On the other hand, when the signal change amount is smaller than the threshold value, the generation unit 3082 acquires a second blur estimation area having a larger area than the first blur estimation area. In the present embodiment, the center position of the second blur estimation area is always the same as the center position of the first blur estimation area, and its area is proportionally multiplied by a predetermined value compared to the first blur estimation area. This is the area that has been This means that the second blur estimation area is acquired as in the second blur estimation area 203b shown in FIG.

  Subsequently, in step S304, the generation unit 3082 corrects the signal gradient (information related to the signal) in the second blur estimation region, and generates a correction signal gradient (information related to the correction signal) (correction processing). This correction process is the same as the correction process in step S203 of the first embodiment. Subsequently, in step S305, the generation unit 3082 estimates blur based on the signal gradient in the second blur estimation region and the correction signal gradient generated in step S304 (blur estimation process). This estimation process is the same as the estimation process in step S204 of the first embodiment.

  Subsequently, in step S306, the generation unit 3082 determines whether the blur estimated in step S305 has converged. If the blur has converged, the process proceeds to step S307. On the other hand, if the blur has not converged, the process returns to step S304. When the processing returns to step S304, the generation unit 3082 newly generates a correction signal gradient in step S304 using the blur estimated in step S305. The method of determining whether or not the estimated blur has converged is, for example, obtaining a difference between the value obtained by degrading the correction signal gradient due to the estimated blur and the signal gradient of the second blur estimation region, or a ratio thereof. What is necessary is just to compare with a predetermined value. Alternatively, when blur is estimated by iterative calculation such as the conjugate gradient method in step S305, determination may be made based on whether or not the update amount of blur by this iterative calculation is smaller than a predetermined value. When the blur converges, the generation unit 3082 sets the blur estimated in step S305 as the final estimated blur in the first blur estimation region.

  Subsequently, in step S307, the generation unit 3082 determines whether or not generation of estimated blur has been completed for a predetermined region (for example, the entire blurred image) in the blurred image. If the generation of the estimated blur has been completed, the process proceeds to step S308. On the other hand, if the generation of the estimated blur has not been completed, the process returns to step S302. Step S307 is the same as step S106 in the first embodiment.

  In step S308, the generation unit 3082 outputs the generated estimated blur. When a plurality of first blur estimation regions are included in a predetermined region in the blurred image, the generation unit 3082 outputs a plurality of estimated blurs. At this time, you may output sequentially in the middle of calculating | requiring several estimation blur. The output estimated blur can be used for correcting a blurred image, measuring optical performance of a photographed optical system, or analyzing camera shake during photographing.

  According to the present embodiment, it is possible to provide an image processing system capable of suppressing a decrease in estimation accuracy when blur is estimated from an area where a signal change of a blurred image is small.

  Next, an imaging system according to Embodiment 3 of the present invention will be described with reference to FIGS. FIG. 9 is a block diagram of the imaging system 400 in the present embodiment. FIG. 10 is an external view of the imaging system 400.

  The imaging system 400 includes an imaging device 401, a network 402, and a server 403 (image processing device). The imaging apparatus 401 and the server 403 are connected wirelessly, and an image from the imaging apparatus 401 is transferred to the server 403, and the server 403 performs blur estimation and correction.

  The server 403 includes a communication unit 404, a storage unit 405, and a blur correction unit 406 (image processing unit). A communication unit 404 of the server 403 is connected to the imaging device 401 via the network 402. In this embodiment, the imaging device 401 and the server 403 are connected wirelessly, but the present invention is not limited to this, and may be connected by wire. The communication unit 404 of the server 403 is configured to receive a blurred image from the imaging device 401. When shooting is performed by the imaging device 401, a blurred image (input image or captured image) is automatically or manually input to the server 403 and sent to the storage unit 405 and the blur correction unit 406. The storage unit 405 stores information regarding a blurred image and shooting conditions for shooting the blurred image. The blur correction unit 406 estimates blur (estimated blur) based on the blur image. Then, the blur correction unit 406 generates a blur correction image based on the estimated blur. The blur correction image is stored in the storage unit 405 or sent to the imaging device 401 via the communication unit 404.

  In order to realize the image processing method of the present embodiment, software (image processing program) can be supplied to the server 403 via a network or a storage medium such as a CD-ROM. At this time, the image processing program is read by the computer (or CPU, MPU, etc.) of the server 403 and executes the function of the server 403.

  Note that the processing performed by the blur correction unit 406 is the same as the image processing method according to the first embodiment described with reference to FIGS. 1 and 4, and thus description thereof is omitted. According to the present embodiment, it is possible to provide an imaging system capable of suppressing a decrease in estimation accuracy when blur is estimated from an area where the amount of signal change in a blurred image is small.

  As described above, in each embodiment, the image processing apparatus (the image processing apparatuses 105 and 305 and the server 403) includes an acquisition unit (acquisition units 1081 and 3081) and a generation unit (generation units 1082 and 3082). The acquisition unit acquires a blurred image (a captured image or an input image). The generation unit acquires at least a part of the first blur estimation region in the blur image and generates an estimated blur. In addition, the generation unit determines the second blur estimation area based on the amount of signal change in the first blur estimation area. The generation unit estimates the blur based on the correction processing for correcting the blur included in the information on the signal in the second blur estimation region to generate the information on the correction signal, and the information on the signal and the information on the correction signal. An iterative calculation process that repeats the estimation process is performed. Thereby, the generation means generates an estimated blur.

  Preferably, the amount of signal change in the second blur estimation region is equal to or greater than the amount of signal change in the first blur estimation region. Preferably, the generating unit determines the first blur estimation area as the second blur estimation area when the amount of signal change in the first blur estimation area is larger than the first threshold. In addition, when the amount of signal change in the first blur estimation region is smaller than the first threshold, the generation unit determines a second blur estimation region in which the amount of signal change is larger than that of the first blur estimation region. More preferably, when the amount of signal change in the first blur estimation area is smaller than the first threshold, the generation unit determines a second blur estimation area having a position or shape different from that of the first blur estimation area. To do. Further preferably, when the amount of signal change in the first blur estimation area is smaller than the first threshold, the generation unit determines a second blur estimation area having an area larger than that of the first blur estimation area. Preferably, the second blur estimation area includes at least half of the first blur estimation area. More preferably, the second blur estimation area includes all of the first blur estimation area. Preferably, the generation unit acquires distance information of the subject space, and determines a second blur estimation area based on the distance information of the subject space.

  Preferably, the generation unit performs the second estimation process after the first estimation process and the first estimation process as the estimation process of the iterative calculation process. The generation unit determines a difference between the first blur estimated in the first estimation process and the second blur estimated in the second estimation process. When the difference is larger than the second threshold, the generation unit increases the area of the second blur estimation region and performs the estimation process again (S205 and S206).

  Preferably, the generation unit generates a low-resolution blur estimation area by reducing the resolution of the second blur estimation area. Further, the generation unit estimates the blur based on the correction processing for generating the information on the correction signal by correcting the blur included in the information on the signal in the low-resolution blur estimation region, and the information on the signal and the information on the correction signal. Iterative calculation processing is performed to repeat the estimation processing to be performed. Thereby, the generation means generates an estimated blur. Further, the generation means brings the resolution of the low-resolution blur estimation area closer to the resolution of the second blur estimation area in the process of the iterative calculation process (S208).

  Preferably, the generation unit acquires the amount of noise included in the second blur estimation area, and generates a frequency resolution blur estimation area by performing frequency decomposition on the second blur estimation area. Then, the generation unit performs denoising processing on the frequency resolution blur estimation region based on the amount of noise, and re-synthesizes the frequency resolution blur estimation region after the denoising processing (S201).

  Preferably, the image processing apparatus includes a correction unit (correction unit 1083) that corrects at least a part of the blurred image using the estimated blur. Further preferably, the amount of signal change in the first blur estimation region is an amount corresponding to the sum of absolute values or the sum of squares of the differential values of the luminance distribution in the first blur estimation region. Preferably, the information related to the signal is information related to a luminance distribution or a differential value of the luminance distribution in the second blur estimation region.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  According to each embodiment, an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can suppress a decrease in estimation accuracy when blur is estimated from an area where a signal change in a blurred image is small. Can be provided.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

105, 305 Image processing apparatus 1081, 3081 Acquisition unit (acquisition means)
1082, 3082 Generation unit (generation means)

Claims (18)

An acquisition means for acquiring a blurred image;
Generating means for acquiring at least a part of a first blur estimation region in the blur image and generating an estimated blur;
The generating means includes
Determining a second blur estimation region based on the amount of signal change in the first blur estimation region;
Correction processing for correcting the blur included in the information regarding the signal in the second blur estimation region to generate information regarding the correction signal, and estimation processing for estimating the blur based on the information regarding the signal and the information regarding the correction signal And generating the estimated blur by performing an iterative calculation process that repeats the above.   The image processing apparatus according to claim 1, wherein the amount of signal change in the second blur estimation region is equal to or greater than the amount of signal change in the first blur estimation region. The generating means includes
If the amount of signal change in the first blur estimation area is greater than a first threshold, determine the first blur estimation area as the second blur estimation area;
When the amount of signal change in the first blur estimation region is smaller than the first threshold, determine the second blur estimation region having a larger amount of signal change than the first blur estimation region; The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   When the signal change amount in the first blur estimation area is smaller than the first threshold, the generation unit generates the second blur estimation area at a position or shape different from that of the first blur estimation area. The image processing apparatus according to claim 3, wherein:   The generation unit determines the second blur estimation area having an area larger than that of the first blur estimation area when the amount of the signal change in the first blur estimation area is smaller than the first threshold. The image processing apparatus according to claim 3, wherein:   The image processing apparatus according to claim 1, wherein the second blur estimation area includes half or more of the first blur estimation area.   The image processing apparatus according to claim 6, wherein the second blur estimation area includes all of the first blur estimation area. The generating means includes
Get distance information of subject space,
The image processing apparatus according to claim 1, wherein the second blur estimation area is determined based on distance information of the subject space. The generating means includes
As the estimation process of the iterative calculation process, a first estimation process and a second estimation process after the first estimation process are performed,
Determining a difference between the first blur estimated in the first estimation process and the second blur estimated in the second estimation process;
10. The method according to claim 1, wherein when the difference is larger than a second threshold value, the area of the second blur estimation region is increased and the estimation process is performed again. 10. Image processing apparatus. The generating means includes
Reducing the resolution of the second blur estimation region to generate a low-resolution blur estimation region;
In the low-resolution blur estimation region, blur is estimated based on correction processing for correcting blur included in the information on the signal to generate information on the correction signal, and information on the signal and information on the correction signal. The estimation blur is generated by performing an iterative calculation process that repeats the estimation process to be performed,
10. The image according to claim 1, wherein the resolution of the low-resolution blur estimation region is made closer to the resolution of the second blur estimation region in the process of the iterative calculation process. Processing equipment. The generating means includes
Obtaining an amount of noise included in the second blur estimation region;
Frequency-resolving the second blur estimation region to generate a frequency-resolved blur estimation region;
Based on the amount of noise, performing a denoising process on the frequency-resolved blur estimation region,
The image processing apparatus according to claim 1, wherein the frequency-resolved blur estimation region after the denoising process is re-synthesized.   The image processing apparatus according to claim 1, further comprising a correction unit that corrects at least a part of the blurred image using the estimated blur.   The amount of the signal change in the first blur estimation region is an amount corresponding to an absolute value sum or a square sum of a differential value of a luminance distribution in the first blur estimation region. 13. The image processing apparatus according to any one of items 12.   The image processing apparatus according to claim 1, wherein the information regarding the signal is information regarding a luminance distribution in the second blur estimation region or a differential value of the luminance distribution. An image sensor that photoelectrically converts an optical image formed through the optical system and outputs an image signal;
Obtaining means for obtaining a blurred image based on the image signal;
Generating means for acquiring at least a part of a first blur estimation region in the blur image and generating an estimated blur;
The generating means includes
Determining a second blur estimation region based on the amount of signal change in the first blur estimation region;
Correction processing for correcting the blur included in the information regarding the signal in the second blur estimation region to generate information regarding the correction signal, and estimation processing for estimating the blur based on the information regarding the signal and the information regarding the correction signal And generating the estimated blur by performing an iterative calculation process that repeats. Obtaining a blurred image;
Obtaining at least a portion of a first blur estimation region in the blur image and generating an estimated blur;
Generating the estimated blur;
Determining a second blur estimation region based on the amount of signal change in the first blur estimation region;
Correction processing for correcting the blur included in the information regarding the signal in the second blur estimation region to generate information regarding the correction signal, and estimation processing for estimating the blur based on the information regarding the signal and the information regarding the correction signal And generating the estimated blur by performing an iterative calculation process that repeats the above. Obtaining a blurred image;
An image processing program configured to cause a computer to execute at least a part of a first blur estimation area in the blur image and generate an estimated blur,
Generating the estimated blur;
Determining a second blur estimation region based on the amount of signal change in the first blur estimation region;
Correction processing for correcting the blur included in the information regarding the signal in the second blur estimation region to generate information regarding the correction signal, and estimation processing for estimating the blur based on the information regarding the signal and the information regarding the correction signal An image processing program characterized by generating the estimated blur by performing an iterative calculation process that repeats.   A storage medium storing the image processing program according to claim 17.